Tell me more ×
Biology Stack Exchange is a question and answer site for biology researchers, academics, and students. It's 100% free, no registration required.
up vote 1 down vote favorite

I have a large amount of Agilent data (580 text files that are ~200 MB in size each) that I'd like to normalize. I've run into a problem in that my machine (with 8 GB of memory) still runs out of memory when I try to read the files using marray in order to geneerate an expression set for normalization. I don't need the expression set, just the normalized output--is there an analogous program for Agilent data like justRMA is used for Affy data?

share|improve this question
This question would be better suited to biostars since it is a completely technical problem related only to computers and not to the biology side of bioinformatics. That said, you could always run your R script in a loop, once for each file couldn't you? Something like for n in *.data; do Rscript script.R $n > $n.norm; done. – terdon Jan 31 at 18:03
See the "Large memory and out-of-memory data" section of cran.r-project.org/web/views/HighPerformanceComputing.html – kmm♦ Jan 31 at 18:29
You could also perhaps ask here... stats.stackexchange.com – Ben Jan 31 at 19:20
@terdon: that wouldn't work, "the usual" array normalization methods pretty much require working on all of the data together (so it needs to be loaded in memory). That having been said, this question should definitely be asked on the bioconductor mailing-lsit, biostars.org. – Steve Lianoglou Jan 31 at 19:52
Would a "normalization of normals" work? Normalize 30 or so files at a time, then normalize the ~20 resulting normals. You may also be able to pick certain families of data points to normalize against each other, so you're only working with a fraction of the data at a time. Putting your data into a Hadoop (or similar) database is also an option. – MattDMo Jan 31 at 20:05
show 1 more comment

closed as off topic by terdon, jonsca, Rory M♦ Feb 1 at 16:03

Questions on Biology Stack Exchange are expected to relate to biology within the scope defined in the FAQ. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about closed questions here.

1 Answer

up vote 1 down vote

You might want to consider using a one-sample-at-a-time normalization scheme like frozenRMA: http://www.bioconductor.org/packages/2.11/bioc/html/frmaTools.html

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.