Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Hadoop pipes with Hbase as input


Copy link to this message
-
Hadoop pipes with Hbase as input
Hi, when I attempt to use a hbase table as input hadoop just seems to
stall. I don't get any errors.

I can't find any examples where pipes are used with hbase, is it
possible at all? Streaming seems to be an alternative
http://dumbotics.com/2009/07/31/dumbo-over-hbase/

hadoop pipes -conf myconf_job.conf -input name_of_table -output /tmp/out

12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.5-cdh3u4--1, built on 05/07/2012
21:08 GMT
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:host.name=kongs1
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_31
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.31/jre
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/usr/lib/hadoop-0.20/conf:/usr/lib/jvm/java-6-sun//lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar:/usr/lib/zookeeper/zookeeper-3.3.5-cdh3u4.jar
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-41-server
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:user.home=/usr/lib/hadoop-0.20
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/home/havard/d/graph
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=180000
watcher=hconnection
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Opening socket connection
to server localhost/127.0.0.1:2181
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Socket connection
established to localhost/127.0.0.1:2181, initiating session
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Session establishment
complete on server localhost/127.0.0.1:2181, sessionid 0x139266be8b90004, negotiated timeout = 40000

my job conf

<property>
<name>mapred.input.format.class</name>
<value>org.apache.hadoop.hbase.mapred.TableInputFormat</value>
</property>

<property>
  <name>hadoop.pipes.java.recordreader</name>
  <value>true</value>
</property>

<property>
<name>hbase.mapred.tablecolumns</name>
<value>col_fam:name</value>
</property>
-Håvard
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB