Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Replication - some timestamps off by 1 ms


+
Patrick Schless 2013-07-11, 17:23
Copy link to this message
-
Re: Replication - some timestamps off by 1 ms
Are those incremented cells?

J-D

On Thu, Jul 11, 2013 at 10:23 AM, Patrick Schless
<[EMAIL PROTECTED]> wrote:
> I have had replication running for about a week now, and have had a lot of
> data flowing to our slave cluster over that time. Now, I'm running the
> verifyrep MR job over a 1-hour period a couple days ago (which should be
> fully replicated), and I'm seeing a small number of "BADROWS".
> Spot-checking a few of them, the issue seems to be that the rows are
> present, and have the same values, but a single cell in the row will be off
> by 1ms.
>
> For instance, the log reports this error:
> java.lang.Exception: This result was different:
> keyvalues={01e581745c6a43aba01adf105af4e4a92013071015/data:!\xDF\xE0\x01/1373470622986/Put/vlen=8,
> 01e581745c6a43aba01adf105af4e4a92013071015/data:&s\xC0\x01/1373470923084/Put/vlen=8,
> 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223717/Put/vlen=8,
> 01e581745c6a43aba01adf105af4e4a92013071015/data:/\x9B\x80\x01/1373471523316/Put/vlen=8,
> 01e581745c6a43aba01adf105af4e4a92013071015/data:4/`\x01/1373471822913/Put/vlen=8}
> compared to
> keyvalues={01e581745c6a43aba01adf105af4e4a92013071015/data:!\xDF\xE0\x01/1373470622986/Put/vlen=8,
> 01e581745c6a43aba01adf105af4e4a92013071015/data:&s\xC0\x01/1373470923084/Put/vlen=8,
> 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223716/Put/vlen=8,
> 01e581745c6a43aba01adf105af4e4a92013071015/data:/\x9B\x80\x01/1373471523316/Put/vlen=8,
> 01e581745c6a43aba01adf105af4e4a92013071015/data:4/`\x01/1373471822913/Put/vlen=8}
>
> Some diffing reduces the issue down to:
> 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223717/Put/vlen=8
> compared to
> 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223716/Put/vlen=8.
>
> I'm assuming that the value before "/Put" is the cell's timestamp, which
> means that the copies are off by 1ms.
>
> Any idea what could cause this? So far (the job is still running), the
> problem seems rare (about 0.05% of rows).
>
> Thanks,
> Patrick
+
Patrick Schless 2013-07-11, 19:19
+
Jean-Daniel Cryans 2013-07-11, 20:44
+
lars hofhansl 2013-07-11, 21:04
+
Patrick Schless 2013-07-11, 21:36
+
Jean-Daniel Cryans 2013-07-11, 21:48
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB