0
$\begingroup$

I'm actually trying to download all the virus protein sequences available in the NCBI database.

I guess the link is this one: ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/viral/

How could I get all the protein sequences in fasta format into one file ? Does someone have codes to do that easily?

Thank you very much.

$\endgroup$
  • 1
    $\begingroup$ I believe that this question belongs to StackOverflow, as it's not about biology or related interdisciplinary subjects. Also for such tasks I recommend using Scrapy for Python $\endgroup$ – Maxim Kuleshov Jan 4 at 17:04
  • 1
    $\begingroup$ @MaximKuleshov — Better on SE Bioinformatics than Stack Overflow as nobody on SO will know anything about the file format of Genbank files. And leave it at that. Recommending a package in a particular language is not appropriate in a comment and not much use anyway if the poster doesn't use the language. $\endgroup$ – David Jan 4 at 22:01
5
$\begingroup$

As with most things, there are multiple ways to do this. Here are two of them:

Download from the NCBI Assembly portal

  • Go to https://www.ncbi.nlm.nih.gov/assembly/ and search for 'Viruses[Organism]'. For our purposes, I am going to restrict this search to Virus assemblies that are latest RefSeq and reference genomes. The query for this will be: "Viruses"[Organism] AND "latest refseq"[filter] AND "reference genome"[filter]. It will return 46 results.
  • To download the protein FASTA sequences for all of these, click on the big blue Download Assemblies button, choose RefSeq as source and Protein FASTA as File Type.
  • Click the Download button and a tarball with FASTA files - one for each assembly - will be created for you to download. You will then have to combine them all into a single file. enter image description here

Download using wget

  • Go to ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/viral and download the assembly_summary.txt file. This is a tab-delimited file with information about every RefSeq viral assembly located in that FTP path. Column 20 has the FTP path for the assembly which you can parse out and use wget to download the protein FASTA file for each assembly, followed by combining them into a single gzipped file as shown below:

    ## download the assembly_summary.txt file
    wget ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/viral/assembly_summary.txt
    ## download the protein FASTA files
    grep -v '^#' assembly_summary.txt \
        | cut -f20 \
        | while read -r url ; do wget $url/*_protein.faa.gz ; done
    ## combine all files into a single one
    zcat *_protein.faa.gz | gzip > all_viral_proteins.faa.gz
    
$\endgroup$
  • $\begingroup$ Hi, thank you for your help, it did work. Unfortunalty when I try zcat *_protein.faa.gz | gzip > all_viral_proteins.faa.gz I get : -bash: /usr/bin/zcat: Argument list too long $\endgroup$ – Grendel Jan 6 at 20:19
  • $\begingroup$ Finally I found I way by doing : for i in $(ls | grep "_protein.faa.gz");do cat $i >> all_viral_proteins.faa.gz;done $\endgroup$ – Grendel Jan 6 at 20:24
  • $\begingroup$ Interesting. I never had to concatenate so many files! Is there a reason why you want such a large number of sequences in a single file? Can you not work with multiple files? Presumably, the single file would be very big and may not be the easiest to handle for downstream usage. $\endgroup$ – vkkodali Jan 6 at 21:50
  • $\begingroup$ It is because I'll make a blast with those sequences as queries, so I'll make a blast database with them. $\endgroup$ – Grendel Jan 7 at 8:26
  • $\begingroup$ Maybe you can help me again, I would like to download a list of genomes, I have their ID number (list with one ID number by line) and I would like to get all the genomes in .fa format do you know if it is possible with the same script? $\endgroup$ – Grendel Jan 7 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.