Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Input splits for sequence file input


Copy link to this message
-
Re: Input splits for sequence file input
Hi Jeff,

This has been asked several times before (check out
http://search-hadoop.com please).

The answer is (3) for SequenceFiles (due to no notion of records) and
(2) as a general thought (i.e. text files, etc.).

On Mon, Dec 3, 2012 at 3:33 AM, Jeff LI <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I was reading on the relationship between input splits and HDFS blocks and a
> question came up to me:
>
> If a logical record crosses HDFS block boundary, let's say block#1 and
> block#2, does the mapper assigned with this input split asks for (1) both
> blocks, or (2) block#1 and just the part of block#2 that this logical record
> extends to, or (3) block#1 and part of block#2 up to some sync point that
> covers this particular logical record?  Note the input is sequence file.
>
> I guess my question really is: does Hadoop operate on a block basis or does
> it respect some sort of logical structure within a block when it's trying to
> feed the mappers with input data.
>
> Cheers
>
> Jeff
>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB