Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Sync-marker in uncompressed sequenceFile


Copy link to this message
-
Re: Sync-marker in uncompressed sequenceFile
It's used to align input splits of the SequenceFile. A reader can
start at an arbitrary offset, then find the boundary of the next block
of records by looking for the sync marker defined in the header. -C

On Mon, Mar 21, 2011 at 7:40 AM, Weishung Chung <[EMAIL PROTECTED]> wrote:
> Hello my fellow Hadoop users/developers,
>
> I'm reading the SequenceFile source code, and there is a checkAndWriteSync()
> method that writes a sync marker every so many bytes. I was wondering what's
> the use of the sync marker. I know one can use it to designate the end of a
> header, but it's also used in the process of writing the uncompressed data.
>  I wish I could have figured it out :(
>
> Thank you so much
>