Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Binary Files With No Record Begin and End


Copy link to this message
-
Binary Files With No Record Begin and End
Hi,

The input of my map reduce is a binary file with no record begin and
end marker. The only thing is that each record is a fixed 180bytes
size in the binary file. How do I make Hadoop to properly find the
record in the splits when a record overlap two splits. I was thinking
to make the splits size to be a multiple of 180 but was wondering if
there is anything else that I can do?  Please note that my files are
not sequence file and just a custom binary file.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB