Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Replication - some timestamps off by 1 ms


Copy link to this message
-
Re: Replication - some timestamps off by 1 ms
Yeah verifyrep is a pretty basic tool, there's tons of room for
improvement. For the moment I guess you can ignore the 8 bytes cells
that aren't printable strings. Feel free to hack around that MR job
and maybe contribute back?

The use case for which I built it had loads of tables and the ones
that had ICVs pretty much only had that, so it was easy to verify just
a couple of tables to have a good idea of how it was doing.

J-D

On Thu, Jul 11, 2013 at 2:36 PM, Patrick Schless
<[EMAIL PROTECTED]> wrote:
> Interesting (thanks for the info). I don't suppose there's an easy way to
> filter those incremented cells out, so the response from verifyRep is
> meaningful? :)
>
>
> On Thu, Jul 11, 2013 at 3:44 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> Yeah increments won't work. I guess the warning isn't really visible
>> but one place you can see it is:
>>
>> $ ./bin/hadoop jar ../hbase/hbase.jar
>> An example program must be given as the first argument.
>> Valid program names are:
>>   CellCounter: Count cells in HBase table
>>   completebulkload: Complete a bulk data load.
>>   copytable: Export a table from local cluster to peer cluster
>>   export: Write table data to HDFS.
>>   import: Import data written by Export.
>>   importtsv: Import data in TSV format.
>>   rowcounter: Count rows in HBase table
>> vvvv
>>   verifyrep: Compare the data from tables in two different clusters.
>> WARNING: It doesn't work for incrementColumnValues'd cells since the
>> timestamp is changed after being appended to the log.
>> ^^^^
>>
>> The problem is that increments' timestamps are different in the WAL
>> and in the final KV that's stored in HBase.
>>
>> J-D
>>
>> On Thu, Jul 11, 2013 at 12:19 PM, Patrick Schless
>> <[EMAIL PROTECTED]> wrote:
>> > It's possible, but I'm not sure. This is a live system, and we do use
>> > increment, and it's a smaller portion of our writes into HBase. I can try
>> > to duplicate it, but I can't say how these specific cells got written.
>> >
>> > Would incremented cells not get replicated correctly?
>> >
>> >
>> > On Thu, Jul 11, 2013 at 12:53 PM, Jean-Daniel Cryans <
>> [EMAIL PROTECTED]>wrote:
>> >
>> >> Are those incremented cells?
>> >>
>> >> J-D
>> >>
>> >> On Thu, Jul 11, 2013 at 10:23 AM, Patrick Schless
>> >> <[EMAIL PROTECTED]> wrote:
>> >> > I have had replication running for about a week now, and have had a
>> lot
>> >> of
>> >> > data flowing to our slave cluster over that time. Now, I'm running the
>> >> > verifyrep MR job over a 1-hour period a couple days ago (which should
>> be
>> >> > fully replicated), and I'm seeing a small number of "BADROWS".
>> >> > Spot-checking a few of them, the issue seems to be that the rows are
>> >> > present, and have the same values, but a single cell in the row will
>> be
>> >> off
>> >> > by 1ms.
>> >> >
>> >> > For instance, the log reports this error:
>> >> > java.lang.Exception: This result was different:
>> >> >
>> >>
>> keyvalues={01e581745c6a43aba01adf105af4e4a92013071015/data:!\xDF\xE0\x01/1373470622986/Put/vlen=8,
>> >> >
>> >>
>> 01e581745c6a43aba01adf105af4e4a92013071015/data:&s\xC0\x01/1373470923084/Put/vlen=8,
>> >> >
>> >>
>> 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223717/Put/vlen=8,
>> >> >
>> >>
>> 01e581745c6a43aba01adf105af4e4a92013071015/data:/\x9B\x80\x01/1373471523316/Put/vlen=8,
>> >> >
>> >>
>> 01e581745c6a43aba01adf105af4e4a92013071015/data:4/`\x01/1373471822913/Put/vlen=8}
>> >> > compared to
>> >> >
>> >>
>> keyvalues={01e581745c6a43aba01adf105af4e4a92013071015/data:!\xDF\xE0\x01/1373470622986/Put/vlen=8,
>> >> >
>> >>
>> 01e581745c6a43aba01adf105af4e4a92013071015/data:&s\xC0\x01/1373470923084/Put/vlen=8,
>> >> >
>> >>
>> 01e581745c6a43aba01adf105af4e4a92013071015/data:+\x07\xA0\x01/1373471223716/Put/vlen=8,
>> >> >
>> >>
>> 01e581745c6a43aba01adf105af4e4a92013071015/data:/\x9B\x80\x01/1373471523316/Put/vlen=8,
>> >> >
>> >>
>> 01e581745c6a43aba01adf105af4e4a92013071015/data:4/`\x01/1373471822913/Put/vlen=8}
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB