Hi,

 

I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
trunk) and following the tutorial.

 

This line loads the sample file from HDFS successfully:

raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user, time,
query);

 

This line seems to work:

T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;

 

Because when I do:

DUMP T;

I get the response I expected.

 

But if I then do:

                STORE T INTO 'excite' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');

I get (in the log):

Pig Stack Trace

---------------

ERROR 2998: Unhandled internal error.
org/apache/hadoop/hbase/filter/WritableByteArrayComparable

 

java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/filter/WritableByteArrayComparable

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:247)

        at
org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)

        at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)

        at
org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
.java:723)

        at
org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
va:712)

        at
org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
java:4340)

        at
org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
.java:5956)

        at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
va:1122)

        at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
rator.java:683)

        at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
va:483)

        at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
69)

        at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)

        at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)

        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)

        at org.apache.pig.PigServer.registerQuery(PigServer.java:584)

        at
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)

        at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
java:386)

        at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
)

        at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
)

        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)

        at org.apache.pig.Main.run(Main.java:495)

        at org.apache.pig.Main.main(Main.java:111)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.filter.WritableByteArrayComparable

        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

        ... 28 more

 

Grateful for any help with this.

 

Thanks,

Royston

"Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.filter.WritableByteArrayComparable" indicates that
you don't have HBase on your classpath, or that the version of HBase you
are testing against moved this class someplace else. We've tested against
the 0.90 series, but not 0.92+... did they refactor something?

D

On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
[EMAIL PROTECTED]> wrote:

> Hi,
>
>
>
> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
> trunk) and following the tutorial.
>
>
>
> This line loads the sample file from HDFS successfully:
>
> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user, time,
> query);
>
>
>
> This line seems to work:
>
> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
>
>
>
> Because when I do:
>
> DUMP T;
>
> I get the response I expected.
>
>
>
> But if I then do:
>
>                STORE T INTO 'excite' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
>
> I get (in the log):
>
> Pig Stack Trace
>
> ---------------
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>
>
>
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>
>        at java.lang.Class.forName0(Native Method)
>
>        at java.lang.Class.forName(Class.java:247)
>
>        at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
>
>        at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
>
>        at
>
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
> .java:723)
>
>        at
>
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
> va:712)
>
>        at
>
> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
> java:4340)
>
>        at
>
> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
> .java:5956)
>
>        at
>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
> va:1122)
>
>        at
>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
> rator.java:683)
>
>        at
>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
> va:483)
>
>        at
>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
> 69)
>
>        at
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
>
>        at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
>
>        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>
>        at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>
>        at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>
>        at
>
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
> java:386)
>
>        at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
> )
>
>        at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
> )
>
>        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>
>        at org.apache.pig.Main.run(Main.java:495)
>
>        at org.apache.pig.Main.main(Main.java:111)
>
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>        at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
>
>        at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>        at java.lang.reflect.Method.invoke(Method.java:597)
>
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
>
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>
>        at java.security.AccessController.doPrivileged(Native Method)
>
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

Thanks for your reply. WritableByteArrayComparable is in the same place in HBase 0.93. I also registered my HBase jar from within Pig i.e register /opt/hbase/hbase-0.93.jar and that command returned without error. I also edited the line in the pig startup script that specifies the path to HBase.
Is there some other config file that controls how Pig finds HBase jar? Is there a default location Pig searches for jars that I could copy my jar into?

Cheers,
Royston

On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> "Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.filter.WritableByteArrayComparable" indicates that
> you don't have HBase on your classpath, or that the version of HBase you
> are testing against moved this class someplace else. We've tested against
> the 0.90 series, but not 0.92+... did they refactor something?
>
> D
>
> On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>>
>>
>> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
>> trunk) and following the tutorial.
>>
>>
>>
>> This line loads the sample file from HDFS successfully:
>>
>> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user, time,
>> query);
>>
>>
>>
>> This line seems to work:
>>
>> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
>>
>>
>>
>> Because when I do:
>>
>> DUMP T;
>>
>> I get the response I expected.
>>
>>
>>
>> But if I then do:
>>
>>               STORE T INTO 'excite' USING
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
>>
>> I get (in the log):
>>
>> Pig Stack Trace
>>
>> ---------------
>>
>> ERROR 2998: Unhandled internal error.
>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>>
>>
>>
>> java.lang.NoClassDefFoundError:
>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>>
>>       at java.lang.Class.forName0(Native Method)
>>
>>       at java.lang.Class.forName(Class.java:247)
>>
>>       at
>> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
>>
>>       at
>> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
>>
>>       at
>>
>> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
>> .java:723)
>>
>>       at
>>
>> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
>> va:712)
>>
>>       at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
>> java:4340)
>>
>>       at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
>> .java:5956)
>>
>>       at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
>> va:1122)
>>
>>       at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
>> rator.java:683)
>>
>>       at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
>> va:483)
>>
>>       at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
>> 69)
>>
>>       at
>> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
>>
>>       at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
>>
>>       at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>>
>>       at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>>
>>       at
>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>>
>>       at
>>
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
>> java:386)
>>
>>       at
>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
>> )
>>
>>       at
>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
>> )
>>
>>       at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>>
>>       at org.apache.pig.Main.run(Main.java:495)
>>
>>       at org.apache.pig.Main.main(Main.java:111)

Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath
Also you can try editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just
Add your Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig
script.
Don't know whether these methods are the best method, but they work for me
:-)

Thanks,
Praveenesh

On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman <
[EMAIL PROTECTED]> wrote:

> Thanks for your reply. WritableByteArrayComparable is in the same place in
> HBase 0.93. I also registered my HBase jar from within Pig i.e register
> /opt/hbase/hbase-0.93.jar and that command returned without error. I also
> edited the line in the pig startup script that specifies the path to HBase.
> Is there some other config file that controls how Pig finds HBase jar? Is
> there a default location Pig searches for jars that I could copy my jar
> into?
>
> Cheers,
> Royston
>
>
>
> On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:
>
> > "Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.hbase.filter.WritableByteArrayComparable" indicates
> that
> > you don't have HBase on your classpath, or that the version of HBase you
> > are testing against moved this class someplace else. We've tested against
> > the 0.90 series, but not 0.92+... did they refactor something?
> >
> > D
> >
> > On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >>
> >>
> >> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
> >> trunk) and following the tutorial.
> >>
> >>
> >>
> >> This line loads the sample file from HDFS successfully:
> >>
> >> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user,
> time,
> >> query);
> >>
> >>
> >>
> >> This line seems to work:
> >>
> >> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
> >>
> >>
> >>
> >> Because when I do:
> >>
> >> DUMP T;
> >>
> >> I get the response I expected.
> >>
> >>
> >>
> >> But if I then do:
> >>
> >>               STORE T INTO 'excite' USING
> >> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
> >>
> >> I get (in the log):
> >>
> >> Pig Stack Trace
> >>
> >> ---------------
> >>
> >> ERROR 2998: Unhandled internal error.
> >> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>
> >>
> >>
> >> java.lang.NoClassDefFoundError:
> >> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>
> >>       at java.lang.Class.forName0(Native Method)
> >>
> >>       at java.lang.Class.forName(Class.java:247)
> >>
> >>       at
> >> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
> >>
> >>       at
> >>
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
> >>
> >>       at
> >>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
> >> .java:723)
> >>
> >>       at
> >>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
> >> va:712)
> >>
> >>       at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
> >> java:4340)
> >>
> >>       at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
> >> .java:5956)
> >>
> >>       at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
> >> va:1122)
> >>
> >>       at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
> >> rator.java:683)
> >>
> >>       at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
> >> va:483)
> >>
> >>       at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
> >> 69)
> >>
> >>       at
> >>
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
> >>
> >>       at
> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
> >>
> >>       at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)

Hi Praveenesh,

Thanks for your suggestion but I checked and the HBase/ZK jars and dirs are correct in our CLASSPATH and in PIG_CLASSPATH.

A few more details might help someone to point out where we are going wrong:

We have the following script called 'hbase_sample.pig'...

raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
   listing_id: chararray,
   fname: chararray,
   lname: chararray );

STORE raw_data INTO 'hbase://hello_world' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage (
   'info:fname info:lname');

We have created the 'hello_world' table in HBase with the following command:

create 'hello_world','info'

There is an input file called 'sample_data.csv' containing the following:

1, John, Smith
2, Jane, Doe
3, George, Washington
4, Ben, Franklin

We have stored 'sample_data.csv' in the same directory as the 'hbase_sample.pig' script and we have also stored a copy of the CSV file on HDFS under the home directory for the current user "hdfs://user/hadoop1/sample_data.csv".

We ran the script in local mode SUCCESSFULLY with the following command:

pig -x local -f hbase_sample.pig

We then ran the script using the mapreduce mode (default for pig):

pig -x mapreduce -f hbase_sample.pig

The jobtracker shows the following error(repeatedly) in the log output for both map and reduce tasks:

Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.filter.WritableByteArrayComparable
       at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
       at java.lang.Class.forName0(Native Method)
       at java.lang.Class.forName(Class.java:247)
       at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
       at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
       at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
       at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:87)
       at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:69)
       at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
       at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
       at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:396)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
       at org.apache.hadoop.mapred.Child.main(Child.java:249)
In addition, we see the following errors from pig (on the stdout):

Input(s):
Failed to read data from "hdfs://10.0.0.235:8020/user/hadoop1/sample_data.csv"

Output(s):
Failed to produce result in "hbase://hello_world"
So we have established that we can run a pig script successfully in local mode to store data in HBase (and we have verified that this has actually worked), however we cannot seem to get the same script to work with HBase in MR mode.

Here is the complete output from pig:

http://pastebin.com/vVvsCHXv

Thanks,
Royston

On 3 Feb 2012, at 14:38, praveenesh kumar wrote:

> Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath
> Also you can try editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just
> Add your Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig
> script.
> Don't know whether these methods are the best method, but they work for me
> :-)

Did you explicitly "register" the hbase jar?

D

On Mon, Feb 6, 2012 at 3:37 AM, Royston Sellman <
[EMAIL PROTECTED]> wrote:

> Hi Praveenesh,
>
> Thanks for your suggestion but I checked and the HBase/ZK jars and dirs
> are correct in our CLASSPATH and in PIG_CLASSPATH.
>
> A few more details might help someone to point out where we are going
> wrong:
>
> We have the following script called 'hbase_sample.pig'...
>
> raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
>   listing_id: chararray,
>   fname: chararray,
>   lname: chararray );
>
> STORE raw_data INTO 'hbase://hello_world' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage (
>   'info:fname info:lname');
>
> We have created the 'hello_world' table in HBase with the following
> command:
>
> create 'hello_world','info'
>
> There is an input file called 'sample_data.csv' containing the following:
>
> 1, John, Smith
> 2, Jane, Doe
> 3, George, Washington
> 4, Ben, Franklin
>
> We have stored 'sample_data.csv' in the same directory as the
> 'hbase_sample.pig' script and we have also stored a copy of the CSV file on
> HDFS under the home directory for the current user
> "hdfs://user/hadoop1/sample_data.csv".
>
> We ran the script in local mode SUCCESSFULLY with the following command:
>
> pig -x local -f hbase_sample.pig
>
> We then ran the script using the mapreduce mode (default for pig):
>
> pig -x mapreduce -f hbase_sample.pig
>
> The jobtracker shows the following error(repeatedly) in the log output for
> both map and reduce tasks:
>
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>        at java.lang.Class.forName0(Native Method)
>       at java.lang.Class.forName(Class.java:247)
>       at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
>       at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
>        at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
>       at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:87)
>       at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:69)
>       at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
>       at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
>       at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:396)
>       at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>       at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> In addition, we see the following errors from pig (on the stdout):
>
> Input(s):
> Failed to read data from "hdfs://
> 10.0.0.235:8020/user/hadoop1/sample_data.csv"
>
> Output(s):
> Failed to produce result in "hbase://hello_world"
>
>
> So we have established that we can run a pig script successfully in local
> mode to store data in HBase (and we have verified that this has actually
> worked), however we cannot seem to get the same script to work with HBase
> in MR mode.
>
> Here is the complete output from pig:
>
> http://pastebin.com/vVvsCHXv
>
> Thanks,
> Royston
>
> On 3 Feb 2012, at 14:38, praveenesh kumar wrote:
>
> > Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath

[resending as previous attempt did not make it through]

Hi Praveenesh,

Thanks for your suggestion but I checked and the HBase/ZK jars and dirs are
correct in our CLASSPATH and in PIG_CLASSPATH.

A few more details might help someone to point out where we are going wrong:

We have the following script called 'hbase_sample.pig'...

raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
   listing_id: chararray,
   fname: chararray,
   lname: chararray );

STORE raw_data INTO 'hbase://hello_world' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage (
   'info:fname info:lname');

We have created the 'hello_world' table in HBase with the following command:

create 'hello_world','info'

There is an input file called 'sample_data.csv' containing the following:

1, John, Smith
2, Jane, Doe
3, George, Washington
4, Ben, Franklin

We have stored 'sample_data.csv' in the same directory as the
'hbase_sample.pig' script and we have also stored a copy of the CSV file on
HDFS under the home directory for the current user
"hdfs://user/hadoop1/sample_data.csv".

We ran the script in local mode SUCCESSFULLY with the following command:

pig -x local -f hbase_sample.pig

We then ran the script using the mapreduce mode (default for pig):

pig -x mapreduce -f hbase_sample.pig

The jobtracker shows the following error(repeatedly) in the log output for
both map and reduce tasks:

Error: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.filter.WritableByteArrayComparable
       at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
       at java.lang.Class.forName0(Native Method)
       at java.lang.Class.forName(Class.java:247)
       at
org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
       at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
       at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperat
ors.POStore.getStoreFunc(POStore.java:232)
       at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.getCommitters(PigOutputCommitter.java:87)
       at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.<init>(PigOutputCommitter.java:69)
       at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat
.getOutputCommitter(PigOutputFormat.java:279)
       at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
       at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:396)
       at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1083)
       at org.apache.hadoop.mapred.Child.main(Child.java:249)
In addition, we see the following errors from pig (on the stdout):

Input(s):
Failed to read data from
"hdfs://10.0.0.235:8020/user/hadoop1/sample_data.csv"

Output(s):
Failed to produce result in "hbase://hello_world"
So we have established that we can run a pig script successfully in local
mode to store data in HBase (and we have verified that this has actually
worked), however we cannot seem to get the same script to work with HBase in
MR mode.

Here is the complete output from pig:

http://pastebin.com/vVvsCHXv

Thanks,
Royston

-----Original Message-----
From: praveenesh kumar [mailto:[EMAIL PROTECTED]]
Sent: 03 February 2012 14:38
To: [EMAIL PROTECTED]
Subject: Re: Cannot STORE from Pig to HBase 0.9x

Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath Also you can try
editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just Add your
Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig script.
Don't know whether these methods are the best method, but they work for me

Thanks,
Praveenesh

On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman <
[EMAIL PROTECTED]> wrote:

specifies the path to HBase.
something?
org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.

Hi Dmitriy,

Well thanks, that fixed it.

A few days ago I WAS registering the HBase jar and getting the same error.
It turns out that you also need the jar in the PIG_CLASSPATH and I didn't.
So I was never at a point where both things were simultaneously true. Many
thanks for the reminder.

I think it would be helpful if the sample Pig code in the javadoc for
HBaseStorage included something like "register etc/hbase/hbase-0.92.jar"
before the LOAD and STORE examples as this is one of the first places one
naturally looks for help on HBaseStorage.

Many of the other examples that turn up in a Google search are also missing
the "register" statement. So is the HBase book (not your problem). I only
stumbled across it by luck.

Of course, none of the above means I'm not an idiot.

Regards,
Royston

-----Original Message-----
From: Dmitriy Ryaboy [mailto:[EMAIL PROTECTED]]
Sent: 06 February 2012 21:22
To: [EMAIL PROTECTED]
Subject: Re: Cannot STORE from Pig to HBase 0.9x

Did you explicitly "register" the hbase jar?

D

On Mon, Feb 6, 2012 at 3:37 AM, Royston Sellman <
[EMAIL PROTECTED]> wrote:

> Hi Praveenesh,
>
> Thanks for your suggestion but I checked and the HBase/ZK jars and
> dirs are correct in our CLASSPATH and in PIG_CLASSPATH.
>
> A few more details might help someone to point out where we are going
> wrong:
>
> We have the following script called 'hbase_sample.pig'...
>
> raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
>   listing_id: chararray,
>   fname: chararray,
>   lname: chararray );
>
> STORE raw_data INTO 'hbase://hello_world' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage (
>   'info:fname info:lname');
>
> We have created the 'hello_world' table in HBase with the following
> command:
>
> create 'hello_world','info'
>
> There is an input file called 'sample_data.csv' containing the following:
>
> 1, John, Smith
> 2, Jane, Doe
> 3, George, Washington
> 4, Ben, Franklin
>
> We have stored 'sample_data.csv' in the same directory as the
> 'hbase_sample.pig' script and we have also stored a copy of the CSV
> file on HDFS under the home directory for the current user
> "hdfs://user/hadoop1/sample_data.csv".
>
> We ran the script in local mode SUCCESSFULLY with the following command:
>
> pig -x local -f hbase_sample.pig
>
> We then ran the script using the mapreduce mode (default for pig):
>
> pig -x mapreduce -f hbase_sample.pig
>
> The jobtracker shows the following error(repeatedly) in the log output
> for both map and reduce tasks:
>
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>        at java.lang.Class.forName0(Native Method)
>       at java.lang.Class.forName(Class.java:247)
>       at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
>       at
>
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
>        at
>
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperat
ors.POStore.getStoreFunc(POStore.java:232)
>       at
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.getCommitters(PigOutputCommitter.java:87)
>       at
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.<init>(PigOutputCommitter.java:69)
>       at
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat
.getOutputCommitter(PigOutputFormat.java:279)
>       at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
>       at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1083)
org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.