How can I text-mine full-text articles from PubMed? [closed]

Question

I want to do a text mining study using full-text versions of articles I find on PubMed. My intended search protocol will be roughly as follows:

1. Search PubMed using a gene name (and any alternate names) as the query, all matching papers are subjected to Step 2; my understanding is that this will return articles that mention the gene in their abstract
2. Search full-text for any and all matches from a list of keywords; assign each paper a score based on the number of matching keywords; any papers that match have to be read by a human but the most relevant papers will have a higher score and get read first

The two-step search needs to be repeated many times with different genes in Step1 so an automated approach is probably worth the time it will take to develop. I know enough about programming that I could write a script to do Step2 if I had the paper as a plain-text document (I program in Perl but I also know a little Python) but I have no idea how I could automate the process of searching for papers, downloading them, converting them to plain-text documents that my program could work on.

I considered posting this in StackOverflow but have opted for this site because I have not ruled out the possibility that this can be done without doing my programming.

UPDATE: I have found one tool that might be very useful for exactly this problem. Unfortunately, I am not in a position to ask for a free trial so I cannot evaluate it. Even if it is an appropriate tool, I will most likely not be able to use it for my study.

Answer 1

Below is a Python script that might help you to get started (apologies if it fails the Pythonic test - it works!). It uses the Entrez part of the Biopython library. The script sets up a query, in this case yeast AND Saccharomyces against the pmc database. Also note that this script uses the 2 step process that NCBI likes you to use - the first part of the fetchByQuery function gets a set of results then the second part uses those results to actually obtain the data.

The output is an xml file which you get to parse with your favourite tools. In your case you will need to get out the text sections and do your token analysis. If you use Python for that I recommend the Natural Language Toolkit (NLTK).

In your case you could just set up search terms as a Python list and loop through writing each dataset to a file named from the search term.

from Bio import Entrez

import urllib 
import urllib2
import sys

def fetchByQuery(query,days):
    Entrez.email = "xxx" # you must give NCBI an email address
    searchHandle=Entrez.esearch(db="pmc", reldate=days, term=query, usehistory="y")
    searchResults=Entrez.read(searchHandle)
    searchHandle.close()
    webEnv=searchResults["WebEnv"]
    queryKey=searchResults["QueryKey"]
    batchSize=10
    try:
        fetchHandle = Entrez.efetch(db="pmc", retmax=100, retmode="xml", webenv=webEnv, query_key=queryKey)
        data=fetchHandle.read()
        fetchHandle.close()
        return data
    except:
        return None

days=100 #looking for papers in the last 100 days
termList=["yeast","Saccharomyces"] 

query=" AND ".join(termList)
xml_data=fetchByQuery(query,days)
if xml_data==None: 
    print 80*"*"+"\n"
    print "This search returned no hits"

else:
    f=open("pmcXml.txt" ,"w")
    f.write(xml_data)
    f.close()

Answer 2

I think it's possible with Entrez Direct

You'd start with something like:

esearch -db pubmed -query "atp6"

And then pipe that to maybe efetch -format ?? and then continue with gnu coreutils. It might be that only abstracts are available as text, in this case, you'd have to extract e.g. pubmed ids and then come up with a way to batch fetch those somehow..