Take the 2-minute tour ×
Biology Stack Exchange is a question and answer site for biology researchers, academics, and students. It's 100% free, no registration required.

I have a large amount of Agilent data (580 text files that are ~200 MB in size each) that I'd like to normalize. I've run into a problem in that my machine (with 8 GB of memory) still runs out of memory when I try to read the files using marray in order to geneerate an expression set for normalization. I don't need the expression set, just the normalized output--is there an analogous program for Agilent data like justRMA is used for Affy data?

share|improve this question

closed as off topic by terdon, jonsca, Rory M Feb 1 '13 at 16:03

Questions on Biology Stack Exchange are expected to relate to biology within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

    
This question would be better suited to biostars since it is a completely technical problem related only to computers and not to the biology side of bioinformatics. That said, you could always run your R script in a loop, once for each file couldn't you? Something like for n in *.data; do Rscript script.R $n > $n.norm; done. –  terdon Jan 31 '13 at 18:03
    
See the "Large memory and out-of-memory data" section of cran.r-project.org/web/views/HighPerformanceComputing.html –  kmm Jan 31 '13 at 18:29
    
You could also perhaps ask here... stats.stackexchange.com –  Ben Jan 31 '13 at 19:20
    
@terdon: that wouldn't work, "the usual" array normalization methods pretty much require working on all of the data together (so it needs to be loaded in memory). That having been said, this question should definitely be asked on the bioconductor mailing-lsit, biostars.org. –  Steve Lianoglou Jan 31 '13 at 19:52
    
Would a "normalization of normals" work? Normalize 30 or so files at a time, then normalize the ~20 resulting normals. You may also be able to pick certain families of data points to normalize against each other, so you're only working with a fraction of the data at a time. Putting your data into a Hadoop (or similar) database is also an option. –  MattDMo Jan 31 '13 at 20:05
show 1 more comment

1 Answer 1

You might want to consider using a one-sample-at-a-time normalization scheme like frozenRMA: http://www.bioconductor.org/packages/2.11/bioc/html/frmaTools.html

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.