Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Sync and Data Replication


Copy link to this message
-
Re: Sync and Data Replication
On Sun, Jun 10, 2012 at 9:39 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Mohit,
>
> On Sat, Jun 9, 2012 at 11:11 PM, Mohit Anchlia <[EMAIL PROTECTED]>
> wrote:
> > Thanks Harsh for detailed info. It clears things up. Only thing from
> those
> > page is concerning is what happens when client crashes. It says you could
> > lose upto a block worth of information. Is this still true given that NN
> > would auto close the file?
>
> Where does it say this exactly? It is true that immediate readers will
> not get the last block (as it remains open and uncommitted), but once
> the lease recovery kicks in the file is closed successfully and the
> last block is indeed made available, so there's no 'data loss'.
>

I saw it in "Coherency Model" -> "consequences of application design"
paragraph.

Thanks for the information. It at least helps me in that I don't have to
worry about the data loss when sync is not closed.

>
> > Is it a good practice to reduce NN default value so that it auto-closes
> > before 1 hr.
>
> I've not seen people do this/need to do this. Most don't run into such
> a situation and it is vital to properly close() files or sync() on
> file streams before making it available to readers. HBase manages open
> files during WAL-recovery using lightweight recoverLease APIs that
> were added for its benefit, so it doesn't need to wait for an hour for
> WALs to close and recover data.
>
> --
> Harsh J
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB