Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Streaming value of (200MB) from a SequenceFile


Copy link to this message
-
Streaming value of (200MB) from a SequenceFile
Jerry Lam 2013-03-30, 17:52
Hi everyone,

I'm having a problem to stream individual key-value pair of 200MB to 1GB
from a MapFile.
I need to stream the large value to an outputstream instead of reading the
entire value before processing because it potentially uses too much memory.

I read the API for MapFile, the next(WritableComparable key, Writable val)
does not return an input stream.

How can I accomplish this?

Thanks,

Jerry