Hi, I'am trying to read hbase key-values with pipes(pydoop). As hadoop
is unable to find the hbase jar files. I get

Exception in thread "main" java.lang.RuntimeException:
java.lang.RuntimeException: class
org.apache.hadoop.hbase.mapreduce.TableInputFormat not
org.apache.hadoop.mapred.InputFormat

have added export
HADOOP_CLASSPATH=/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar to my
hadoop-env.sh

According to the doc from cloudera,
https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-UsingMapReducewithHBase
TableMapReduceUtil.addDependencyJars(job); can be used as an
alternative. But is that possible with pipes?

-Håvard

Hi,

Per:

> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
org.apache.hadoop.mapred.InputFormat

Pydoop seems to be expecting you to pass it an old API class for
InputFormat/etc. but you've passed in the newer class. I am unsure
what part of your code exactly may be at fault since I do not have
access to it, but you probably want to use the deprecated
org.apache.hadoop.hbase.mapred.* package classes such as
org.apache.hadoop.hbase.mapred.TableInputFormat, and not the
org.apache.hadoop.hbase.mapreduce.* classes, as you are using at the
moment.

HTH!

On Wed, Aug 15, 2012 at 2:39 AM, Håvard Wahl Kongsgård
<[EMAIL PROTECTED]> wrote:
> Hi, I'am trying to read hbase key-values with pipes(pydoop). As hadoop
> is unable to find the hbase jar files. I get
>
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: class
> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
> org.apache.hadoop.mapred.InputFormat
>
> have added export
> HADOOP_CLASSPATH=/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar to my
> hadoop-env.sh
>
> According to the doc from cloudera,
> https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-UsingMapReducewithHBase
> TableMapReduceUtil.addDependencyJars(job); can be used as an
> alternative. But is that possible with pipes?
>
> -Håvard

--
Harsh J

Hi, my job config is

<property>
<name>mapred.input.format.class</name>
<value>org.apache.hadoop.hbase.mapred.TableInputFormat</value>
</property>

<property>
  <name>hadoop.pipes.java.recordreader</name>
  <value>true</value>
</property>
Exception in thread "main" java.lang.RuntimeException: Error in
configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:596)
        at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:977)
        at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:969)
        at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1248)
        at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:248)
        at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
        at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
        ... 17 more
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hbase.mapred.TableInputFormat.configure(TableInputFormat.java:51)
should I included the col names? according to the api it's deprecated?
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapred/TableInputFormat.html
-Håvard
On Tue, Aug 14, 2012 at 11:17 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Per:
>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
> org.apache.hadoop.mapred.InputFormat
>
> Pydoop seems to be expecting you to pass it an old API class for
> InputFormat/etc. but you've passed in the newer class. I am unsure
> what part of your code exactly may be at fault since I do not have
> access to it, but you probably want to use the deprecated
> org.apache.hadoop.hbase.mapred.* package classes such as
> org.apache.hadoop.hbase.mapred.TableInputFormat, and not the
> org.apache.hadoop.hbase.mapreduce.* classes, as you are using at the
> moment.
>
> HTH!
>
> On Wed, Aug 15, 2012 at 2:39 AM, Håvard Wahl Kongsgård
> <[EMAIL PROTECTED]> wrote:
>> Hi, I'am trying to read hbase key-values with pipes(pydoop). As hadoop
>> is unable to find the hbase jar files. I get
>>
>> Exception in thread "main" java.lang.RuntimeException:
>> java.lang.RuntimeException: class
>> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
>> org.apache.hadoop.mapred.InputFormat
>>
>> have added export
>> HADOOP_CLASSPATH=/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar to my
>> hadoop-env.sh
>>
>> According to the doc from cloudera,
>> https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-UsingMapReducewithHBase
>> TableMapReduceUtil.addDependencyJars(job); can be used as an
>> alternative. But is that possible with pipes?
>>
>> -Håvard
>
>
>
> --
> Harsh J

Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU

http://havard.security-review.net/

Hi, needed to add this as well
<property>
<name>hbase.mapred.tablecolumns</name>
<value>col_fam:name</value>
</property>

-Håvard
On Wed, Aug 15, 2012 at 9:42 AM, Håvard Wahl Kongsgård
<[EMAIL PROTECTED]> wrote:
> Hi, my job config is
>
> <property>
> <name>mapred.input.format.class</name>
> <value>org.apache.hadoop.hbase.mapred.TableInputFormat</value>
> </property>
>
> <property>
>   <name>hadoop.pipes.java.recordreader</name>
>   <value>true</value>
> </property>
>
>
> Exception in thread "main" java.lang.RuntimeException: Error in
> configuring object
>         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>         at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>         at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:596)
>         at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:977)
>         at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:969)
>         at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
>         at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1248)
>         at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:248)
>         at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
>         at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>         ... 17 more
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.hbase.mapred.TableInputFormat.configure(TableInputFormat.java:51)
>
>
> should I included the col names? according to the api it's deprecated?
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapred/TableInputFormat.html
>
>
> -Håvard
>
>
> On Tue, Aug 14, 2012 at 11:17 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> Per:
>>
>>> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
>> org.apache.hadoop.mapred.InputFormat
>>
>> Pydoop seems to be expecting you to pass it an old API class for
>> InputFormat/etc. but you've passed in the newer class. I am unsure
>> what part of your code exactly may be at fault since I do not have
>> access to it, but you probably want to use the deprecated
>> org.apache.hadoop.hbase.mapred.* package classes such as
>> org.apache.hadoop.hbase.mapred.TableInputFormat, and not the
>> org.apache.hadoop.hbase.mapreduce.* classes, as you are using at the
>> moment.
>>
>> HTH!
>>
>> On Wed, Aug 15, 2012 at 2:39 AM, Håvard Wahl Kongsgård
>> <[EMAIL PROTECTED]> wrote:
>>> Hi, I'am trying to read hbase key-values with pipes(pydoop). As hadoop
>>> is unable to find the hbase jar files. I get
>>>
>>> Exception in thread "main" java.lang.RuntimeException:
>>> java.lang.RuntimeException: class
>>> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
>>> org.apache.hadoop.mapred.InputFormat
>>>
>>> have added export
>>> HADOOP_CLASSPATH=/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar to my

however, when run hadoop pipes -conf myconf_job.conf -input
name_of_table -output /tmp/out

I don't get any error, hadoop just stalls with

12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.5-cdh3u4--1, built on 05/07/2012
21:08 GMT
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:host.name=kongs1
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_31
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.31/jre
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/usr/lib/hadoop-0.20/conf:/usr/lib/jvm/java-6-sun//lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar:/usr/lib/zookeeper/zookeeper-3.3.5-cdh3u4.jar
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-41-server
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:user.home=/usr/lib/hadoop-0.20
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/home/havard/d/graph
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=180000
watcher=hconnection
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Opening socket connection
to server localhost/127.0.0.1:2181
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Socket connection
established to localhost/127.0.0.1:2181, initiating session
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Session establishment
complete on server localhost/127.0.0.1:2181, sessionid 0x139266be8b90004, negotiated timeout = 40000
-Håvard
On Wed, Aug 15, 2012 at 10:01 AM, Håvard Wahl Kongsgård
<[EMAIL PROTECTED]> wrote:

Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU

http://havard.security-review.net/