Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - ycsb and hbase 0.95.1  hadoop2


+
Paul Baclace 2013-07-29, 07:48
+
Nick Dimiduk 2013-07-29, 16:04
+
Nicolas Liochon 2013-07-29, 16:29
+
Paul Baclace 2013-07-29, 18:50
+
Nicolas Liochon 2013-07-29, 19:18
+
Paul Baclace 2013-07-29, 20:11
+
Ted Yu 2013-07-29, 20:14
+
Nicolas Liochon 2013-07-29, 20:34
Copy link to this message
-
Re: ycsb and hbase 0.95.1 hadoop2
Paul Baclace 2013-08-19, 03:14
I eventually determined that bin/ycsb (a python script) was causing most
of the linkage problems, so I recommend using your own bash script to
start ycsb for HBase. The problem is that the ordering of paths in the
classpath is completely uncontrolled: bin/ycsb does not put the
hbase-binding/conf first, so it can pick up a default hbase-site.xml
found in a jar, and it will recursively include any jars in any
subdirectories under hbase-binding/ (a surprise when I mv'ed some files
to a subdir like dont_use_these and lsof showed them to be in use).

I ended up using hbase 0.95.1 hadoop1 on Apache Hadoop 1.0.4 for my
benchmarking.
On 20130729 13:34 , Nicolas Liochon wrote:
> 0.95.2 is not yet finished. As Ted suggested, you can build HBase locally
> from the sources (the 0.95 branch is usually good as the patches are tested
> before being committed).
>
> When 0.95.2 will be out I will redo the integration with ycsb. Hopefully it
> will be simple to make it work with all hadoop versions on the standards
> maven repo...
>
>
> On Mon, Jul 29, 2013 at 10:14 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> You can perform the following command in 0.95 workspace:
>>
>> mvn clean install -DskipTests
>>
>> Cheers
>>
>> On Mon, Jul 29, 2013 at 1:11 PM, Paul Baclace <[EMAIL PROTECTED]
>>> wrote:
>>> On 20130729 12:18 , Nicolas Liochon wrote:
>>>
>>>> No, I've got nothing in my setting.xml
>>>> I build HBase locally (mvn install -Dhadoop.profile=2.0 -DskipTests), so
>>>> yes, it goes into my .m2
>>>>
>>>>
>>>> I've just rechecked with the last 0.97 as of now, it does compile. It
>>>> worked as well 2 weeks ago with the 0.95 branch. I'm not sure about the
>>>> 95.1. I remember some nasty maven issues.
>>>>
>>> ((I rather be attacked by angry ants than nasty mavens.))
>>>
>>>
>>>> If you're using the maven repo for the 0.95.1, IIRC there is no version
>>>> available that works well with hadoop 2 with a maven-built client (it's
>> ok
>>>> if you build hbase locally with the right hadoop version).
>>>>
>>> Is there an experimental or nightly 0.95.2 maven repo I can connect to? I
>>> can probably use 0.95.2 if all the tests passed; it looks like
>>> https://builds.apache.org/job/**HBase-0.95/373<
>> https://builds.apache.org/job/HBase-0.95/373>passed.
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jul 29, 2013 at 8:50 PM, Paul Baclace <[EMAIL PROTECTED]
>>>>> wrote:
>>>>   Thanks for your help, Nicolas. I adjusted the params and I get a
>>>>> ycsb/db/HBaseClient.java compile error (below) without any missing
>>>>> package
>>>>> errors, so I'm wondering how it can work for you. I am building by
>> adding
>>>>> artifacts from 0.95.1 release to my .m2; and you are building a
>> snapshot
>>>>> and "installing" that to .m2 (or other repo), I assume.
>>>>>
>>>>> It appears to be missing hbase-common which I added to .m2 and the
>>>>> compiler debug shows a classpath that only contains
>>>>> hbase-server-0.95.1-hadoop2.****jar (plus ycsb and general jars). Does
>>>>> your
>>>>> mvn -X compile cmd line for ycsb/db/HBaseClient.java contain more hbase
>>>>> or
>>>>> hadoop jars? Do you have a settings.xml that makes mvn pull in more
>>>>> dependencies?
>>>>>
>>>>>
>>>>> [ERROR] YCSB/hbase/src/main/java/com/****yahoo/ycsb/db/HBaseClient.****
>>>>> java:[34,30]
>>>>> package org.apache.hadoop.conf does not exist
>>>>> [ERROR]
>> /YCSB/hbase/src/main/java/com/****yahoo/ycsb/db/HBaseClient.****
>>>>> java:[35,31]
>>>>> cannot find symbol
>>>>> symbol  : class KeyValue
>>>>>
>>>>>
>>>>>
>>>>> On 20130729 9:29 , Nicolas Liochon wrote:
>>>>>
>>>>>   You can use the branch master here:
>> https://github.com/nkeywal/****YCSB<https://github.com/nkeywal/**YCSB>
>>>>>> <https://github.com/**nkeywal/YCSB <https://github.com/nkeywal/YCSB>>
>>>>>> Built it with:
>>>>>> mvn clean package -DskipTests -Dhbase-96 -Dhbase.version=0.97.0-**
>>>>>> SNAPSHOT
>>>>>> -Dhadoop.profile=2.0 -Dhadoop-two.version=2.0.5
>>>>>> (obviously, change the hbase.version & hadoop-two.version as you