Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Sync-marker in uncompressed sequenceFile


Copy link to this message
-
Re: Sync-marker in uncompressed sequenceFile
It's used to align input splits of the SequenceFile. A reader can
start at an arbitrary offset, then find the boundary of the next block
of records by looking for the sync marker defined in the header. -C

On Mon, Mar 21, 2011 at 7:40 AM, Weishung Chung <[EMAIL PROTECTED]> wrote:
> Hello my fellow Hadoop users/developers,
>
> I'm reading the SequenceFile source code, and there is a checkAndWriteSync()
> method that writes a sync marker every so many bytes. I was wondering what's
> the use of the sync marker. I know one can use it to designate the end of a
> header, but it's also used in the process of writing the uncompressed data.
>  I wish I could have figured it out :(
>
> Thank you so much
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB