2
$\begingroup$

So, fastq-dump has the ability to be run on just an SRA file accession number, such that the SRA is converted to FASTQ on-the-fly, and the SRA doesn't have to be written to disk.

I'm curious whether it would be possible to use fastq-dump to write to a named pipe (using mkfifo) and feed that into another program, for example Trinity, to run an assembly on the FASTQ file without ever having to write all that data to disk. For large datasets, this could actually save quite a bit of time in aggregate.

Has anyone done something similar? I am going to try and experiment with the technique soon, but I a) don't know much about the mkfifo process to begin with and b) am unsure of how this procedure would work for paired-end data where fastq-dump is splitting the SRA file as it goes. How would one specify which output would go to which pipe?

EDIT: A (hypothetical) example:

mkfifo fileStream

fastq-dump SRR123456 > fileStream

Trinity --single fileStream misc_args

That should take the fastq-dump data and stream it into the named pipe "fileStream", which can then be used to stream data into Trinity. I don't know enough about all of the commands, however, to know if this makes any sense.

I would welcome any thoughts from more experienced users!

EDIT: Added update as an answer below.

$\endgroup$
  • $\begingroup$ Can you give an example of the kind of piping you are trying to do? Frequently I just try something like bin1 infile | bin2 - and see what happens. $\endgroup$ – kmm Dec 10 '14 at 23:21
  • $\begingroup$ I you don't get an answer here you should try asking at stackoverflow.com or unix.stackexchange.com, since this deals a lot with tools and piping in Bash. $\endgroup$ – fileunderwater Dec 12 '14 at 8:50
  • $\begingroup$ I agree with @fileunderwater ... this should be migrated to stack overflow (I just don't have the power to do so!) $\endgroup$ – TanMath Dec 12 '14 at 19:17
  • $\begingroup$ @fileunderwater, @T Abraham: I am not super familiar with the culture on here, and I am fine with having this question go wherever. It's possible, however, that with the updates this post might prove useful to someone else in bioinformatics using Trinity. $\endgroup$ – glarue Dec 12 '14 at 19:24
  • $\begingroup$ @glarue You should post the last update as an answer instead, since that is what it is. It is perfectly fine to post answers to your own questions, and then the community can decide (by voting) on which answer is the best. And for the record, with regards to my suggestion to move/crosspost; I don't think the question is off-topic here, since it clearly deals with methods in bioinformatics. $\endgroup$ – fileunderwater Dec 12 '14 at 19:36
6
$\begingroup$

fastq-dump can write to stdout ; the -Z option allows you to do that. This, you can pipe to any downstream process. [An example]. Also see the SRA-toolkit manual.

I guess someone asked this question in biostar too.

$\endgroup$
2
$\begingroup$

As WYSIWYG pointed out, fastq-dump can output to STDOUT, so if you want to combine it with named pipes, you'd do:

mkfifo mydata.fifo
fastq-dump -Z mydata.sra > mydata.fifo &
cat mydata.fifo # or whatever
rm mydata.fifo

If you try to catch the output from fastq-dump without -Z it fails with the Illegal seek(29) error:

mkfifo mydata.fastq
fastq-dump mydata.sra &

By the way, be careful with your downstream analyses: if the input is to be read more than once, respective program will fail to re-read from your pipe. For single-end data and Trinity that's probably not the case, but I suspect that for paired-end data Trinity will attempt to read the input files at more than one step.

$\endgroup$
  • $\begingroup$ Thanks for the additional info. I am now much more familiar with the commands and have gotten it working. I also appreciate the note on PE data; I believe for Trinity however it's not a problem because the first thing Trinity does is write the data to FASTA files (which it later concatenates). The trick with PE is to deal with the convolved data such that it gets properly split into separate streams for feeding into Trinity. I will update post with a working example soon. $\endgroup$ – glarue Dec 12 '14 at 17:46
1
$\begingroup$

Okay, so just for others who might stumble upon this, here is a brief description of one implementation of this technique to run with paired-end RNA-seq data:

fastq-dump SRA_file --split-files -I -Z | tee >(grep '@.*\.1\s' -A3 --no-group-separator > namedPipe_1) >(grep '@.*\.2\s' -A3 --no-group-separator > namedPipe_2) >/dev/null

This first requires the creation of two named pipes using mkfifo. For paired-end data, the -Z flag becomes problematic because it forces the data into a single stream. There are many ways to regain the two pairs, but the way I've elected to do it is to use --split-files to break up the stream beforehand, -I to append either ".1" or ".2" to the end of each header, and then use tee to duplicate the stream plus grep with a regex to parse the info from each pair back out into separate pipes for downstream use.

I have tested this with Trinity, running on each named pipe just as I would with a FASTQ file, and it seems to be working fine. While I am not 100% sure that Trinity won't try and go back to the original FASTQ files, the first thing Trinity does is take those FASTQ files and parse them into FASTA format, which it later concatenates into "both.fa", and so I'm pretty confident that this will work.

UPDATE: This implementation will not work (at least, not as far as I can tell) with the Trimmomatic plugin for Trinity. Something seems to get screwed up because of the way Trimmomatic pulls from the streams - just a cautionary note, I haven't figured out the cause.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.