fastq-dump using mkfifo to stream data

Question

So, fastq-dump has the ability to be run on just an SRA file accession number, such that the SRA is converted to FASTQ on-the-fly, and the SRA doesn't have to be written to disk.

I'm curious whether it would be possible to use fastq-dump to write to a named pipe (using mkfifo) and feed that into another program, for example Trinity, to run an assembly on the FASTQ file without ever having to write all that data to disk. For large datasets, this could actually save quite a bit of time in aggregate.

Has anyone done something similar? I am going to try and experiment with the technique soon, but I a) don't know much about the mkfifo process to begin with and b) am unsure of how this procedure would work for paired-end data where fastq-dump is splitting the SRA file as it goes. How would one specify which output would go to which pipe?

EDIT: A (hypothetical) example:

mkfifo fileStream

fastq-dump SRR123456 > fileStream

Trinity --single fileStream misc_args

That should take the fastq-dump data and stream it into the named pipe "fileStream", which can then be used to stream data into Trinity. I don't know enough about all of the commands, however, to know if this makes any sense.

I would welcome any thoughts from more experienced users!

EDIT: Added update as an answer below.

Answer 1

fastq-dump can write to stdout ; the -Z option allows you to do that. This, you can pipe to any downstream process. [An example]. Also see the SRA-toolkit manual.

I guess someone asked this question in biostar too.

Answer 2

As WYSIWYG pointed out, fastq-dump can output to STDOUT, so if you want to combine it with named pipes, you'd do:

mkfifo mydata.fifo
fastq-dump -Z mydata.sra > mydata.fifo &
cat mydata.fifo # or whatever
rm mydata.fifo

If you try to catch the output from fastq-dump without -Z it fails with the Illegal seek(29) error:

mkfifo mydata.fastq
fastq-dump mydata.sra &

By the way, be careful with your downstream analyses: if the input is to be read more than once, respective program will fail to re-read from your pipe. For single-end data and Trinity that's probably not the case, but I suspect that for paired-end data Trinity will attempt to read the input files at more than one step.

Answer 3

Okay, so just for others who might stumble upon this, here is a brief description of one implementation of this technique to run with paired-end RNA-seq data:

fastq-dump SRA_file --split-files -I -Z | tee >(grep '@.*\.1\s' -A3 --no-group-separator > namedPipe_1) >(grep '@.*\.2\s' -A3 --no-group-separator > namedPipe_2) >/dev/null

This first requires the creation of two named pipes using mkfifo. For paired-end data, the -Z flag becomes problematic because it forces the data into a single stream. There are many ways to regain the two pairs, but the way I've elected to do it is to use --split-files to break up the stream beforehand, -I to append either ".1" or ".2" to the end of each header, and then use tee to duplicate the stream plus grep with a regex to parse the info from each pair back out into separate pipes for downstream use.

I have tested this with Trinity, running on each named pipe just as I would with a FASTQ file, and it seems to be working fine. While I am not 100% sure that Trinity won't try and go back to the original FASTQ files, the first thing Trinity does is take those FASTQ files and parse them into FASTA format, which it later concatenates into "both.fa", and so I'm pretty confident that this will work.

UPDATE: This implementation will not work (at least, not as far as I can tell) with the Trimmomatic plugin for Trinity. Something seems to get screwed up because of the way Trimmomatic pulls from the streams - just a cautionary note, I haven't figured out the cause.