Hi folks,

I am using flume to sink into hbase. I had the set up working
yesterday, but when I restarted my flume agent, it for some reason
cannot dump to hbase anymore (too bad I don't have a version-control
for my config yet.)

Here is the configuration and execution output. The agent essentially
listens for objects via avro and dump to console for debugging and to
hbase. The last portion of the console output shows that an object is
indeed received, but for some reason "scan" on the hbase doesn't show
anything new. I tried Regex/Simple serializer and HBase and
AsyncHbase, none works now...

>From the log I don't see anything obviously wrong. I did a tcpdump on
the agent, and saw it established a couple of connections to the
regionservers, but closed them a few seconds later.

Any info on how to correctly configure this or troubleshoot would be
appreciated.

Thanks!
-Simon

root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
a1.channels = c1 c2
a1.sinks = k1 k2
a1.sources = r1

a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 8080

a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
a1.sinks.k1.table = foo_table
a1.sinks.k1.columnFamily = bar_cf
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer

a1.sinks.k2.type = logger

#a1.channels.c1.type = file
#a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
#a1.channels.c1.dataDirs = /mnt/flume/data
a1.channels.c1.type = memory
a1.channels.c2.type = memory

a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2

root@flume-agent1:/usr/local/lib/flume# bin/flume-ng agent --name a1
--conf conf --conf-file conf/test.conf
-Dflume.root.logger=INFO,console
Info: Including Hadoop libraries found via
(/usr/local/hadoop/bin/hadoop) for HDFS access
Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
from classpath
Info: Excluding
/usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
classpath
Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase)
for HBASE access
Info: Excluding /usr/local/hbase/lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /usr/local/hbase/lib/slf4j-log4j12-1.4.3.jar from classpath
Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
from classpath
Info: Excluding
/usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
classpath
+ exec /usr/lib/jvm/default-java/bin/java -Xmx20m
-Dflume.root.logger=INFO,console -cp
'/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/*:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/lib/commons-cli-1.2.jar:/usr/local/hbase/lib/commons-codec-1.4.jar:/usr/local/hbase/lib/commons-collections-3.2.1.jar:/usr/local/hbase/lib/commons-configuration-1.6.jar:/usr/local/hbase/lib/commons-digester-1.8.jar:/usr/local/hbase/lib/commons-el-1.0.jar:/usr/local/hbase/lib/commons-httpclient-3.1.jar:/usr/local/hbase/lib/commons-io-2.1.jar:

payloadColumn or incrementColumn must be specified. This should give
you a better message and the open JIRA is
https://issues.apache.org/jira/browse/FLUME-1757

Brock

On Tue, Jan 8, 2013 at 10:19 PM, Xu (Simon) Chen <[EMAIL PROTECTED]> wrote:
> Hi folks,
>
> I am using flume to sink into hbase. I had the set up working
> yesterday, but when I restarted my flume agent, it for some reason
> cannot dump to hbase anymore (too bad I don't have a version-control
> for my config yet.)
>
> Here is the configuration and execution output. The agent essentially
> listens for objects via avro and dump to console for debugging and to
> hbase. The last portion of the console output shows that an object is
> indeed received, but for some reason "scan" on the hbase doesn't show
> anything new. I tried Regex/Simple serializer and HBase and
> AsyncHbase, none works now...
>
> From the log I don't see anything obviously wrong. I did a tcpdump on
> the agent, and saw it established a couple of connections to the
> regionservers, but closed them a few seconds later.
>
> Any info on how to correctly configure this or troubleshoot would be
> appreciated.
>
> Thanks!
> -Simon
>
>
>
> root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
> a1.channels = c1 c2
> a1.sinks = k1 k2
> a1.sources = r1
>
> a1.sources.r1.type = avro
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 8080
>
> a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
> a1.sinks.k1.table = foo_table
> a1.sinks.k1.columnFamily = bar_cf
> a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
>
> a1.sinks.k2.type = logger
>
> #a1.channels.c1.type = file
> #a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
> #a1.channels.c1.dataDirs = /mnt/flume/data
> a1.channels.c1.type = memory
> a1.channels.c2.type = memory
>
> a1.sources.r1.channels = c1 c2
> a1.sinks.k1.channel = c1
> a1.sinks.k2.channel = c2
>
> root@flume-agent1:/usr/local/lib/flume# bin/flume-ng agent --name a1
> --conf conf --conf-file conf/test.conf
> -Dflume.root.logger=INFO,console
> Info: Including Hadoop libraries found via
> (/usr/local/hadoop/bin/hadoop) for HDFS access
> Warning: $HADOOP_HOME is deprecated.
>
> Warning: $HADOOP_HOME is deprecated.
>
> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
> from classpath
> Info: Excluding
> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
> classpath
> Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase)
> for HBASE access
> Info: Excluding /usr/local/hbase/lib/slf4j-api-1.4.3.jar from classpath
> Info: Excluding /usr/local/hbase/lib/slf4j-log4j12-1.4.3.jar from classpath
> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
> from classpath
> Info: Excluding
> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
> classpath
> + exec /usr/lib/jvm/default-java/bin/java -Xmx20m
> -Dflume.root.logger=INFO,console -cp
> '/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/*:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/usr/local/hbase/conf:/usr/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/hbase-0.94.3.jar:/usr/local/hbase/hbase-0.94.3-tests.jar:/usr/local/hbase/lib/activation-1.1.jar:/usr/local/hbase/lib/asm-3.1.jar:/usr/local/hbase/lib/avro-1.5.3.jar:/usr/local/hbase/lib/avro-ipc-1.5.3.jar:/usr/local/hbase/lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/lib/commons-be

Yeah, now I remembered, I was using Regex serializer...

The comment in the code is actually confusing:
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleHbaseEventSerializer.java

It gives me the impression that incrementColumn can be skipped...

On Tue, Jan 8, 2013 at 11:39 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> payloadColumn or incrementColumn must be specified. This should give
> you a better message and the open JIRA is
> https://issues.apache.org/jira/browse/FLUME-1757
>
> Brock
>
> On Tue, Jan 8, 2013 at 10:19 PM, Xu (Simon) Chen <[EMAIL PROTECTED]> wrote:
>> Hi folks,
>>
>> I am using flume to sink into hbase. I had the set up working
>> yesterday, but when I restarted my flume agent, it for some reason
>> cannot dump to hbase anymore (too bad I don't have a version-control
>> for my config yet.)
>>
>> Here is the configuration and execution output. The agent essentially
>> listens for objects via avro and dump to console for debugging and to
>> hbase. The last portion of the console output shows that an object is
>> indeed received, but for some reason "scan" on the hbase doesn't show
>> anything new. I tried Regex/Simple serializer and HBase and
>> AsyncHbase, none works now...
>>
>> From the log I don't see anything obviously wrong. I did a tcpdump on
>> the agent, and saw it established a couple of connections to the
>> regionservers, but closed them a few seconds later.
>>
>> Any info on how to correctly configure this or troubleshoot would be
>> appreciated.
>>
>> Thanks!
>> -Simon
>>
>>
>>
>> root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
>> a1.channels = c1 c2
>> a1.sinks = k1 k2
>> a1.sources = r1
>>
>> a1.sources.r1.type = avro
>> a1.sources.r1.bind = 0.0.0.0
>> a1.sources.r1.port = 8080
>>
>> a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
>> a1.sinks.k1.table = foo_table
>> a1.sinks.k1.columnFamily = bar_cf
>> a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
>>
>> a1.sinks.k2.type = logger
>>
>> #a1.channels.c1.type = file
>> #a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
>> #a1.channels.c1.dataDirs = /mnt/flume/data
>> a1.channels.c1.type = memory
>> a1.channels.c2.type = memory
>>
>> a1.sources.r1.channels = c1 c2
>> a1.sinks.k1.channel = c1
>> a1.sinks.k2.channel = c2
>>
>> root@flume-agent1:/usr/local/lib/flume# bin/flume-ng agent --name a1
>> --conf conf --conf-file conf/test.conf
>> -Dflume.root.logger=INFO,console
>> Info: Including Hadoop libraries found via
>> (/usr/local/hadoop/bin/hadoop) for HDFS access
>> Warning: $HADOOP_HOME is deprecated.
>>
>> Warning: $HADOOP_HOME is deprecated.
>>
>> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
>> from classpath
>> Info: Excluding
>> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>> classpath
>> Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase)
>> for HBASE access
>> Info: Excluding /usr/local/hbase/lib/slf4j-api-1.4.3.jar from classpath
>> Info: Excluding /usr/local/hbase/lib/slf4j-log4j12-1.4.3.jar from classpath
>> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
>> from classpath
>> Info: Excluding
>> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>> classpath
>> + exec /usr/lib/jvm/default-java/bin/java -Xmx20m
>> -Dflume.root.logger=INFO,console -cp
>> '/usr/local/lib/flume/conf:/usr/local/lib/flume/lib/*:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.0.4.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib

Tried this new config, still doesn't work...

root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
a1.channels = c1 c2
a1.sinks = k1 k2
a1.sources = r1

a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 8080

a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
a1.sinks.k1.table = foo_table
a1.sinks.k1.columnFamily = bar_cf
a1.sinks.k1.payloadColumn = test
a1.sinks.k1.incrementColumn = count
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer

a1.sinks.k2.type = logger

#a1.channels.c1.type = file
#a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
#a1.channels.c1.dataDirs = /mnt/flume/data
a1.channels.c1.type = memory
a1.channels.c2.type = memory

a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
On Tue, Jan 8, 2013 at 11:43 PM, Xu (Simon) Chen <[EMAIL PROTECTED]> wrote:
> Yeah, now I remembered, I was using Regex serializer...
>
> The comment in the code is actually confusing:
> https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleHbaseEventSerializer.java
>
> It gives me the impression that incrementColumn can be skipped...
>
> On Tue, Jan 8, 2013 at 11:39 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>> payloadColumn or incrementColumn must be specified. This should give
>> you a better message and the open JIRA is
>> https://issues.apache.org/jira/browse/FLUME-1757
>>
>> Brock
>>
>> On Tue, Jan 8, 2013 at 10:19 PM, Xu (Simon) Chen <[EMAIL PROTECTED]> wrote:
>>> Hi folks,
>>>
>>> I am using flume to sink into hbase. I had the set up working
>>> yesterday, but when I restarted my flume agent, it for some reason
>>> cannot dump to hbase anymore (too bad I don't have a version-control
>>> for my config yet.)
>>>
>>> Here is the configuration and execution output. The agent essentially
>>> listens for objects via avro and dump to console for debugging and to
>>> hbase. The last portion of the console output shows that an object is
>>> indeed received, but for some reason "scan" on the hbase doesn't show
>>> anything new. I tried Regex/Simple serializer and HBase and
>>> AsyncHbase, none works now...
>>>
>>> From the log I don't see anything obviously wrong. I did a tcpdump on
>>> the agent, and saw it established a couple of connections to the
>>> regionservers, but closed them a few seconds later.
>>>
>>> Any info on how to correctly configure this or troubleshoot would be
>>> appreciated.
>>>
>>> Thanks!
>>> -Simon
>>>
>>>
>>>
>>> root@flume-agent1:/usr/local/lib/flume# cat conf/test.conf
>>> a1.channels = c1 c2
>>> a1.sinks = k1 k2
>>> a1.sources = r1
>>>
>>> a1.sources.r1.type = avro
>>> a1.sources.r1.bind = 0.0.0.0
>>> a1.sources.r1.port = 8080
>>>
>>> a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink
>>> a1.sinks.k1.table = foo_table
>>> a1.sinks.k1.columnFamily = bar_cf
>>> a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
>>>
>>> a1.sinks.k2.type = logger
>>>
>>> #a1.channels.c1.type = file
>>> #a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
>>> #a1.channels.c1.dataDirs = /mnt/flume/data
>>> a1.channels.c1.type = memory
>>> a1.channels.c2.type = memory
>>>
>>> a1.sources.r1.channels = c1 c2
>>> a1.sinks.k1.channel = c1
>>> a1.sinks.k2.channel = c2
>>>
>>> root@flume-agent1:/usr/local/lib/flume# bin/flume-ng agent --name a1
>>> --conf conf --conf-file conf/test.conf
>>> -Dflume.root.logger=INFO,console
>>> Info: Including Hadoop libraries found via
>>> (/usr/local/hadoop/bin/hadoop) for HDFS access
>>> Warning: $HADOOP_HOME is deprecated.
>>>
>>> Warning: $HADOOP_HOME is deprecated.
>>>
>>> Info: Excluding /usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar
>>> from classpath
>>> Info: Excluding
>>> /usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>>> classpath
>>> Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase)
>>> for HBASE access
>>> Info: Excluding /usr/local/hbase/lib/slf4j-api-1.4.3.jar from classpath

incrementColumn column can be skipped. It's either payloadColumn or
incrementColumn.

On Tue, Jan 8, 2013 at 10:50 PM, Xu (Simon) Chen <[EMAIL PROTECTED]> wrote:
> a1.sinks.k1.payloadColumn = test

I think that should be:

a1.sinks.k1.serializer.payloadColumn = test
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Silly me... it works now, thanks a lot!

On Wed, Jan 9, 2013 at 12:10 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
> incrementColumn column can be skipped. It's either payloadColumn or
> incrementColumn.
>
> On Tue, Jan 8, 2013 at 10:50 PM, Xu (Simon) Chen <[EMAIL PROTECTED]> wrote:
>> a1.sinks.k1.payloadColumn = test
>
> I think that should be:
>
> a1.sinks.k1.serializer.payloadColumn = test
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/