I want to do a text mining study using full-text versions of articles I find on PubMed. My intended search protocol will be roughly as follows:
- Search PubMed using a gene name (and any alternate names) as the query, all matching papers are subjected to Step 2; my understanding is that this will return articles that mention the gene in their abstract
- Search full-text for any and all matches from a list of keywords; assign each paper a score based on the number of matching keywords; any papers that match have to be read by a human but the most relevant papers will have a higher score and get read first
The two-step search needs to be repeated many times with different genes in Step1 so an automated approach is probably worth the time it will take to develop. I know enough about programming that I could write a script to do Step2 if I had the paper as a plain-text document (I program in Perl but I also know a little Python) but I have no idea how I could automate the process of searching for papers, downloading them, converting them to plain-text documents that my program could work on.
I considered posting this in StackOverflow but have opted for this site because I have not ruled out the possibility that this can be done without doing my programming.
UPDATE: I have found one tool that might be very useful for exactly this problem. Unfortunately, I am not in a position to ask for a free trial so I cannot evaluate it. Even if it is an appropriate tool, I will most likely not be able to use it for my study.