Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> weird hbase sink behavior


Copy link to this message
-
Re: weird hbase sink behavior
Yeah, now I remembered, I was using Regex serializer...

The comment in the code is actually confusing:
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleHbaseEventSerializer.java

It gives me the impression that incrementColumn can be skipped...

On Tue, Jan 8, 2013 at 11:39 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> payloadColumn or incrementColumn must be specified. This should give
> you a better message and the open JIRA is
> https://issues.apache.org/jira/browse/FLUME-1757
>
> Brock
>
> On Tue, Jan 8, 2013 at 10:19 PM, Xu (Simon) Chen <[EMAIL PROTECTED]> wrote:
>> Hi folks,
>>
>> I am using flume to sink into hbase. I had the set up working
>> yesterday, but when I restarted my flume agent, it for some reason
>> cannot dump to hbase anymore (too bad I don't have a version-control
>> for my config yet.)
>>
>> Here is the configuration and execution output. The agent essentially
>> listens for objects via avro and dump to console for debugging and to
>> hbase. The last portion of the console output shows that an object is
>> indeed received, but for some reason "scan" on the hbase doesn't show
>> anything new. I tried Regex/Simple serializer and HBase and
>> AsyncHbase, none works now...
>>
>> From the log I don't see anything obviously wrong. I did a tcpdump on
>> the agent, and saw it established a couple of connections to the
>> regionservers, but closed them a few seconds later.
>>
>> Any info on how to correctly configure this or troubleshoot would be
>> appreciated.
>>
>> Thanks!
>> -Simon
>>
>>
>>
>> root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
>> a1.channels = c1 c2
>> a1.sinks = k1 k2
>> a1.sources = r1
>>
>> a1.sources.r1.type = avro
>> a1.sources.r1.bind = 0.0.0.0
>> a1.sources.r1.port = 8080
>>
>> a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
>> a1.sinks.k1.table = foo_table
>> a1.sinks.k1.columnFamily = bar_cf
>> a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
>>
>> a1.sinks.k2.type = logger
>>
>> #a1.channels.c1.type = file
>> #a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
>> #a1.channels.c1.dataDirs = /mnt/flume/data
>> a1.channels.c1.type = memory
>> a1.channels.c2.type = memory
>>
>> a1.sources.r1.channels = c1 c2
>> a1.sinks.k1.channel = c1
>> a1.sinks.k2.channel = c2
>>
>> root@flume-agent1:/usr/local/lib/flume# bin/flume-ng agent --name a1
>> --conf conf --conf-file conf/test.conf
>> -Dflume.root.logger=INFO,console
>> Info: Including Hadoop libraries found via
>> (/usr/local/hadoop/bin/hadoop) for HDFS access
>> Warning: $HADOOP_HOME is deprecated.
>>
>> Warning: $HADOOP_HOME is deprecated.
>>
>> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
>> from classpath
>> Info: Excluding
>> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>> classpath
>> Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase)
>> for HBASE access
>> Info: Excluding /usr/local/hbase/lib/slf4j-api-1.4.3.jar from classpath
>> Info: Excluding /usr/local/hbase/lib/slf4j-log4j12-1.4.3.jar from classpath
>> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
>> from classpath
>> Info: Excluding
>> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>> classpath
>> + exec /usr/lib/jvm/default-java/bin/java -Xmx20m
>> -Dflume.root.logger=INFO,console -cp
>> '/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/*:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB