I want to do a text mining study using full-text versions of articles I find on PubMed. My intended search protocol will be roughly as follows:

  1. Search PubMed using a gene name (and any alternate names) as the query, all matching papers are subjected to Step 2; my understanding is that this will return articles that mention the gene in their abstract
  2. Search full-text for any and all matches from a list of keywords; assign each paper a score based on the number of matching keywords; any papers that match have to be read by a human but the most relevant papers will have a higher score and get read first

The two-step search needs to be repeated many times with different genes in Step1 so an automated approach is probably worth the time it will take to develop. I know enough about programming that I could write a script to do Step2 if I had the paper as a plain-text document (I program in Perl but I also know a little Python) but I have no idea how I could automate the process of searching for papers, downloading them, converting them to plain-text documents that my program could work on.

I considered posting this in StackOverflow but have opted for this site because I have not ruled out the possibility that this can be done without doing my programming.

UPDATE: I have found one tool that might be very useful for exactly this problem. Unfortunately, I am not in a position to ask for a free trial so I cannot evaluate it. Even if it is an appropriate tool, I will most likely not be able to use it for my study.

  • $\begingroup$ Thats interesting. Would be interesting so see if there is a ready solution for it. $\endgroup$
    – Karolas
    Commented Sep 17, 2014 at 15:31
  • $\begingroup$ Do you mean all papers indexed in PubMed, or only those in PubMedCentral that have free full text available. $\endgroup$ Commented Sep 17, 2014 at 15:43
  • $\begingroup$ All papers, @MadScientist $\endgroup$
    – Slavatron
    Commented Sep 17, 2014 at 18:31
  • $\begingroup$ Note that bulk-downloading papers without permission may violate the terms of use of the repositories or journals where they are hosted. Journals could revoke your subscription (or your institution's) or threaten legal action. $\endgroup$ Commented Sep 17, 2014 at 21:01
  • 4
    $\begingroup$ I'm voting to close this question as off-topic because this is a question about mining and sifting data not Biology. $\endgroup$
    – dustin
    Commented Apr 20, 2015 at 22:42

2 Answers 2


Below is a Python script that might help you to get started (apologies if it fails the Pythonic test - it works!). It uses the Entrez part of the Biopython library. The script sets up a query, in this case yeast AND Saccharomyces against the pmc database. Also note that this script uses the 2 step process that NCBI likes you to use - the first part of the fetchByQuery function gets a set of results then the second part uses those results to actually obtain the data.

The output is an xml file which you get to parse with your favourite tools. In your case you will need to get out the text sections and do your token analysis. If you use Python for that I recommend the Natural Language Toolkit (NLTK).

In your case you could just set up search terms as a Python list and loop through writing each dataset to a file named from the search term.

from Bio import Entrez

import urllib 
import urllib2
import sys

def fetchByQuery(query,days):
    Entrez.email = "xxx" # you must give NCBI an email address
    searchHandle=Entrez.esearch(db="pmc", reldate=days, term=query, usehistory="y")
        fetchHandle = Entrez.efetch(db="pmc", retmax=100, retmode="xml", webenv=webEnv, query_key=queryKey)
        return data
        return None

days=100 #looking for papers in the last 100 days

query=" AND ".join(termList)
if xml_data==None: 
    print 80*"*"+"\n"
    print "This search returned no hits"

    f=open("pmcXml.txt" ,"w")

I think it's possible with Entrez Direct

You'd start with something like:

esearch -db pubmed -query "atp6"

And then pipe that to maybe efetch -format ?? and then continue with gnu coreutils. It might be that only abstracts are available as text, in this case, you'd have to extract e.g. pubmed ids and then come up with a way to batch fetch those somehow..


Not the answer you're looking for? Browse other questions tagged .