Is there also a sink available for Hbase?

On Thu, Jul 12, 2012 at 11:05:47AM -0700, Mohit Anchlia wrote:
> Is there also a sink available for Hbase?

Yes, version 1.2.0 that is currently under vote and should be released soon do contain Hbase sink.

Jarcec

Hi,

http://mapredit.blogspot.de/2012/06/apache-flume-12x-and-hbase.html

Short writeup.

cheers
- Alex
On Jul 12, 2012, at 8:10 PM, Jarek Jarcec Cecho wrote:

> On Thu, Jul 12, 2012 at 11:05:47AM -0700, Mohit Anchlia wrote:
>> Is there also a sink available for Hbase?
>
> Yes, version 1.2.0 that is currently under vote and should be released soon do contain Hbase sink.
>
> Jarcec
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Hi all,

Newbie here...I am trying to get flume ng to do a simple hbase sink. I have downloaded/installed flume-ng and configured it based oninfo found on the net. I don't see flume connecting to hbase.I am wondering what I could be missing?  - Thanks
Here is my configuration:
hbase-agent.sources = tailhbase-agent.sinks = sink1hbase-agent.channels = ch1
hbase-agent.sources.tail.type = exechbase-agent.sources.tail.bind = pseudohbase-agent.sources.tail.command = tail -F /tmp/test-flume.enshbase-agent.sources.tail.channels = ch1
hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSinkhbase-agent.sinks.sink1.channel = ch1hbase-agent.sinks.sink1.table = enshbase-agent.sinks.sink1.columnFamily = rawhbase-agent.sinks.sink1.column = eventhbase-agent.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializerhbase-agent.channels.ch1.type = memory

     

Hi George,

         We need to add the payload column as well.Add the following
line in your config file :
hbase-agent.sinks.sink1.serializer.payloadColumn = col_name

Regards,
    Mohammad Tariq
On Tue, Jul 17, 2012 at 8:23 PM, George Forman
<[EMAIL PROTECTED]> wrote:
> Hi all,
>
> Newbie here...I am trying to get flume ng to do a simple hbase sink.
> I have downloaded/installed flume-ng and configured it based on
> info found on the net. I don't see flume connecting to hbase.
> I am wondering what I could be missing?  - Thanks
>
> Here is my configuration:
>
> hbase-agent.sources = tail
> hbase-agent.sinks = sink1
> hbase-agent.channels = ch1
>
> hbase-agent.sources.tail.type = exec
> hbase-agent.sources.tail.bind = pseudo
> hbase-agent.sources.tail.command = tail -F /tmp/test-flume.ens
> hbase-agent.sources.tail.channels = ch1
>
> hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> hbase-agent.sinks.sink1.channel = ch1
> hbase-agent.sinks.sink1.table = ens
> hbase-agent.sinks.sink1.columnFamily = raw
> hbase-agent.sinks.sink1.column = event
> hbase-agent.sinks.sink1.serializer > org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> hbase-agent.channels.ch1.type = memory
>
>

Hi Users,

I have followed the below link for move the sample data to hbase sink. i
then executed the below command, I am getting following error output.
pasted the flume.log log file.

Please suggest and help on this issue.

http://cloudfront.blogspot.in/2012/06/how-to-move-data-into-hbase-table-using.html

command
==========bin/flume-ng agent -n hbase-agent -c conf/ -f conf/hbase-agent.conf
hbase-agent.conf
=================hbase-agent.sources = tail
hbase-agent.sinks = sink1
hbase-agent.channels = ch1
hbase-agent.sources.tail.type = exec
hbase-agent.sources.tail.command = tail -F /usr/local/demo.txt
hbase-agent.sources.tail.channels = ch1
hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
hbase-agent.sinks.sink1.channel = ch1
hbase-agent.sinks.sink1.table = demo
hbase-agent.sinks.sink1.columnFamily = cf
hbase-agent.sinks.sink1.serializer org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
hbase-agent.sinks.sink1.serializer.payloadColumn = col1
hbase-agent.sinks.sink1.serializer.keyType = timestamp
hbase-agent.sinks.sink1.serializer.rowPrefix = 1
hbase-agent.sinks.sink1.serializer.suffix = timestamp
hbase-agent.channels.ch1.type=memory
flume.log
================/flume/flume-1.2.0-incubating-SNAPSHOT# more flume.log
2012-07-19 18:24:58,007 INFO lifecycle.LifecycleSupervisor: Starting
lifecycle supervisor 1
2012-07-19 18:24:58,008 INFO node.FlumeNode: Flume node starting -
hbase-agent
2012-07-19 18:24:58,011 INFO nodemanager.DefaultLogicalNodeManager: Node
manager starting
2012-07-19 18:24:58,011 INFO lifecycle.LifecycleSupervisor: Starting
lifecycle supervisor 10
2012-07-19 18:24:58,011 INFO
properties.PropertiesFileConfigurationProvider: Configuration provider
starting
2012-07-19 18:24:58,013 INFO
properties.PropertiesFileConfigurationProvider: Reloading configuration
file:conf/hbase-agent.conf
2012-07-19 18:24:58,019 INFO conf.FlumeConfiguration: Processing:sink1
2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Added sinks: sink1
Agent: hbase-agent
2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
2012-07-19 18:24:58,021 INFO conf.FlumeConfiguration: Processing:sink1
2012-07-19 18:24:58,021 INFO conf.FlumeConfiguration: Processing:sink1
2012-07-19 18:24:58,034 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration  for agents: [hbase-agent]
2012-07-19 18:24:58,034 INFO
properties.PropertiesFileConfigurationProvider: Creating channels
2012-07-19 18:24:58,038 INFO
properties.PropertiesFileConfigurationProvider: created channel ch1
2012-07-19 18:24:58,046 INFO sink.DefaultSinkFactory: Creating instance of
sink sink1 typeorg.apache.flume.sink.hbase.HBaseSink
2012-07-19 18:24:58,051 ERROR
properties.PropertiesFileConfigurationProvider: Failed to start agent
because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
        at org.apache.flume.sink.hbase.HBaseSink.<init>(HBaseSink.java:94)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:355)
        at java.lang.Class.newInstance(Class.java:308)
        at
org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:103)
        at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:371)

        at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
        at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
        at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
        at
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

Thanks,
Prabhu.

Hi Prabhu,

   Have you exported HADOOP_HOME variable??

Regards,
    Mohammad Tariq
On Thu, Jul 19, 2012 at 6:34 PM, prabhu k <[EMAIL PROTECTED]> wrote:
> Hi Users,
>
> I have followed the below link for move the sample data to hbase sink. i
> then executed the below command, I am getting following error output.
> pasted the flume.log log file.
>
> Please suggest and help on this issue.
>
> http://cloudfront.blogspot.in/2012/06/how-to-move-data-into-hbase-table-using.html
>
> command
> ==========> bin/flume-ng agent -n hbase-agent -c conf/ -f conf/hbase-agent.conf
>
>
> hbase-agent.conf
> =================> hbase-agent.sources = tail
> hbase-agent.sinks = sink1
> hbase-agent.channels = ch1
> hbase-agent.sources.tail.type = exec
> hbase-agent.sources.tail.command = tail -F /usr/local/demo.txt
> hbase-agent.sources.tail.channels = ch1
> hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> hbase-agent.sinks.sink1.channel = ch1
> hbase-agent.sinks.sink1.table = demo
> hbase-agent.sinks.sink1.columnFamily = cf
> hbase-agent.sinks.sink1.serializer > org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> hbase-agent.sinks.sink1.serializer.payloadColumn = col1
> hbase-agent.sinks.sink1.serializer.keyType = timestamp
> hbase-agent.sinks.sink1.serializer.rowPrefix = 1
> hbase-agent.sinks.sink1.serializer.suffix = timestamp
> hbase-agent.channels.ch1.type=memory
>
>
> flume.log
> ================> /flume/flume-1.2.0-incubating-SNAPSHOT# more flume.log
> 2012-07-19 18:24:58,007 INFO lifecycle.LifecycleSupervisor: Starting
> lifecycle supervisor 1
> 2012-07-19 18:24:58,008 INFO node.FlumeNode: Flume node starting -
> hbase-agent
> 2012-07-19 18:24:58,011 INFO nodemanager.DefaultLogicalNodeManager: Node
> manager starting
> 2012-07-19 18:24:58,011 INFO lifecycle.LifecycleSupervisor: Starting
> lifecycle supervisor 10
> 2012-07-19 18:24:58,011 INFO properties.PropertiesFileConfigurationProvider:
> Configuration provider starting
> 2012-07-19 18:24:58,013 INFO properties.PropertiesFileConfigurationProvider:
> Reloading configuration file:conf/hbase-agent.conf
> 2012-07-19 18:24:58,019 INFO conf.FlumeConfiguration: Processing:sink1
> 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Added sinks: sink1
> Agent: hbase-agent
> 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> 2012-07-19 18:24:58,021 INFO conf.FlumeConfiguration: Processing:sink1
> 2012-07-19 18:24:58,021 INFO conf.FlumeConfiguration: Processing:sink1
> 2012-07-19 18:24:58,034 INFO conf.FlumeConfiguration: Post-validation flume
> configuration contains configuration  for agents: [hbase-agent]
> 2012-07-19 18:24:58,034 INFO properties.PropertiesFileConfigurationProvider:
> Creating channels
> 2012-07-19 18:24:58,038 INFO properties.PropertiesFileConfigurationProvider:
> created channel ch1
> 2012-07-19 18:24:58,046 INFO sink.DefaultSinkFactory: Creating instance of
> sink sink1 typeorg.apache.flume.sink.hbase.HBaseSink
> 2012-07-19 18:24:58,051 ERROR
> properties.PropertiesFileConfigurationProvider: Failed to start agent
> because dependencies were not found in classpath. Error follows.
> java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
>         at org.apache.flume.sink.hbase.HBaseSink.<init>(HBaseSink.java:94)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

Thanks for the quick response,

Yes, i have exported HADOOP_HOME variable, anything needs to do?

On Thu, Jul 19, 2012 at 6:41 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Hi Prabhu,
>
>    Have you exported HADOOP_HOME variable??
>
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Jul 19, 2012 at 6:34 PM, prabhu k <[EMAIL PROTECTED]> wrote:
> > Hi Users,
> >
> > I have followed the below link for move the sample data to hbase sink. i
> > then executed the below command, I am getting following error output.
> > pasted the flume.log log file.
> >
> > Please suggest and help on this issue.
> >
> >
> http://cloudfront.blogspot.in/2012/06/how-to-move-data-into-hbase-table-using.html
> >
> > command
> > ==========> > bin/flume-ng agent -n hbase-agent -c conf/ -f conf/hbase-agent.conf
> >
> >
> > hbase-agent.conf
> > =================> > hbase-agent.sources = tail
> > hbase-agent.sinks = sink1
> > hbase-agent.channels = ch1
> > hbase-agent.sources.tail.type = exec
> > hbase-agent.sources.tail.command = tail -F /usr/local/demo.txt
> > hbase-agent.sources.tail.channels = ch1
> > hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> > hbase-agent.sinks.sink1.channel = ch1
> > hbase-agent.sinks.sink1.table = demo
> > hbase-agent.sinks.sink1.columnFamily = cf
> > hbase-agent.sinks.sink1.serializer > > org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> > hbase-agent.sinks.sink1.serializer.payloadColumn = col1
> > hbase-agent.sinks.sink1.serializer.keyType = timestamp
> > hbase-agent.sinks.sink1.serializer.rowPrefix = 1
> > hbase-agent.sinks.sink1.serializer.suffix = timestamp
> > hbase-agent.channels.ch1.type=memory
> >
> >
> > flume.log
> > ================> > /flume/flume-1.2.0-incubating-SNAPSHOT# more flume.log
> > 2012-07-19 18:24:58,007 INFO lifecycle.LifecycleSupervisor: Starting
> > lifecycle supervisor 1
> > 2012-07-19 18:24:58,008 INFO node.FlumeNode: Flume node starting -
> > hbase-agent
> > 2012-07-19 18:24:58,011 INFO nodemanager.DefaultLogicalNodeManager: Node
> > manager starting
> > 2012-07-19 18:24:58,011 INFO lifecycle.LifecycleSupervisor: Starting
> > lifecycle supervisor 10
> > 2012-07-19 18:24:58,011 INFO
> properties.PropertiesFileConfigurationProvider:
> > Configuration provider starting
> > 2012-07-19 18:24:58,013 INFO
> properties.PropertiesFileConfigurationProvider:
> > Reloading configuration file:conf/hbase-agent.conf
> > 2012-07-19 18:24:58,019 INFO conf.FlumeConfiguration: Processing:sink1
> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Added sinks: sink1
> > Agent: hbase-agent
> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
> > 2012-07-19 18:24:58,021 INFO conf.FlumeConfiguration: Processing:sink1
> > 2012-07-19 18:24:58,021 INFO conf.FlumeConfiguration: Processing:sink1
> > 2012-07-19 18:24:58,034 INFO conf.FlumeConfiguration: Post-validation
> flume
> > configuration contains configuration  for agents: [hbase-agent]
> > 2012-07-19 18:24:58,034 INFO
> properties.PropertiesFileConfigurationProvider:
> > Creating channels
> > 2012-07-19 18:24:58,038 INFO
> properties.PropertiesFileConfigurationProvider:
> > created channel ch1
> > 2012-07-19 18:24:58,046 INFO sink.DefaultSinkFactory: Creating instance
> of
> > sink sink1 typeorg.apache.flume.sink.hbase.HBaseSink
> > 2012-07-19 18:24:58,051 ERROR
> > properties.PropertiesFileConfigurationProvider: Failed to start agent
> > because dependencies were not found in classpath. Error follows.
> > java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/HBaseConfiguration
> >         at
> org.apache.flume.sink.hbase.HBaseSink.<init>(HBaseSink.java:94)

Configuration looks fine. HBASE_HOME and CLASSPATH are also important.
Please check these once.

Regards,
    Mohammad Tariq
On Thu, Jul 19, 2012 at 6:50 PM, prabhu k <[EMAIL PROTECTED]> wrote:
> Thanks for the quick response,
>
> Yes, i have exported HADOOP_HOME variable, anything needs to do?
>
> On Thu, Jul 19, 2012 at 6:41 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>
>> Hi Prabhu,
>>
>>    Have you exported HADOOP_HOME variable??
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Jul 19, 2012 at 6:34 PM, prabhu k <[EMAIL PROTECTED]> wrote:
>> > Hi Users,
>> >
>> > I have followed the below link for move the sample data to hbase sink. i
>> > then executed the below command, I am getting following error output.
>> > pasted the flume.log log file.
>> >
>> > Please suggest and help on this issue.
>> >
>> >
>> > http://cloudfront.blogspot.in/2012/06/how-to-move-data-into-hbase-table-using.html
>> >
>> > command
>> > ==========>> > bin/flume-ng agent -n hbase-agent -c conf/ -f conf/hbase-agent.conf
>> >
>> >
>> > hbase-agent.conf
>> > =================>> > hbase-agent.sources = tail
>> > hbase-agent.sinks = sink1
>> > hbase-agent.channels = ch1
>> > hbase-agent.sources.tail.type = exec
>> > hbase-agent.sources.tail.command = tail -F /usr/local/demo.txt
>> > hbase-agent.sources.tail.channels = ch1
>> > hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
>> > hbase-agent.sinks.sink1.channel = ch1
>> > hbase-agent.sinks.sink1.table = demo
>> > hbase-agent.sinks.sink1.columnFamily = cf
>> > hbase-agent.sinks.sink1.serializer >> > org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
>> > hbase-agent.sinks.sink1.serializer.payloadColumn = col1
>> > hbase-agent.sinks.sink1.serializer.keyType = timestamp
>> > hbase-agent.sinks.sink1.serializer.rowPrefix = 1
>> > hbase-agent.sinks.sink1.serializer.suffix = timestamp
>> > hbase-agent.channels.ch1.type=memory
>> >
>> >
>> > flume.log
>> > ================>> > /flume/flume-1.2.0-incubating-SNAPSHOT# more flume.log
>> > 2012-07-19 18:24:58,007 INFO lifecycle.LifecycleSupervisor: Starting
>> > lifecycle supervisor 1
>> > 2012-07-19 18:24:58,008 INFO node.FlumeNode: Flume node starting -
>> > hbase-agent
>> > 2012-07-19 18:24:58,011 INFO nodemanager.DefaultLogicalNodeManager: Node
>> > manager starting
>> > 2012-07-19 18:24:58,011 INFO lifecycle.LifecycleSupervisor: Starting
>> > lifecycle supervisor 10
>> > 2012-07-19 18:24:58,011 INFO
>> > properties.PropertiesFileConfigurationProvider:
>> > Configuration provider starting
>> > 2012-07-19 18:24:58,013 INFO
>> > properties.PropertiesFileConfigurationProvider:
>> > Reloading configuration file:conf/hbase-agent.conf
>> > 2012-07-19 18:24:58,019 INFO conf.FlumeConfiguration: Processing:sink1
>> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
>> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
>> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Added sinks: sink1
>> > Agent: hbase-agent
>> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
>> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
>> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
>> > 2012-07-19 18:24:58,020 INFO conf.FlumeConfiguration: Processing:sink1
>> > 2012-07-19 18:24:58,021 INFO conf.FlumeConfiguration: Processing:sink1
>> > 2012-07-19 18:24:58,021 INFO conf.FlumeConfiguration: Processing:sink1
>> > 2012-07-19 18:24:58,034 INFO conf.FlumeConfiguration: Post-validation
>> > flume
>> > configuration contains configuration  for agents: [hbase-agent]
>> > 2012-07-19 18:24:58,034 INFO
>> > properties.PropertiesFileConfigurationProvider:
>> > Creating channels
>> > 2012-07-19 18:24:58,038 INFO
>> > properties.PropertiesFileConfigurationProvider:
>> > created channel ch1
>> > 2012-07-19 18:24:58,046 INFO sink.DefaultSinkFactory: Creating instance
>> > of
>> > sink sink1 typeorg.apache.flume.sink.hbase.HBaseSink
>> > 2012-07-19 18:24:58,051 ERROR

Hi Mohammad,

After i export HBASE_HOME and CLASSPATH  flume-ng executed again, it seems
keep on running, unexpectedly i have disconnected the session,
and i checked the hbase demo table, i hit the command like

scan 'demo'  getting like below.

hbase(main):001:0> scan 'demo'
ROW                                         COLUMN+CELL
table view also keep on running, it's not exit the command. I then did
ctrl+c.
WARN client.ZooKeeperSaslClient: SecurityException:
java.lang.SecurityException: Unable to locate a login configuration
occurred when trying to find JAAS configuration.

WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper
exception: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master

WARN client.ZooKeeperSaslClient: SecurityException:
java.lang.SecurityException: Unable to locate a login configuration
occurred when trying to find JAAS configuration.
 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper
exception: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master

WARN client.ZooKeeperSaslClient: SecurityException:
java.lang.SecurityException: Unable to locate a login configuration
occurred when trying to find JAAS configuration.

ERROR zookeeper.ZooKeeperWatcher: hconnection Received unexpected
KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master

ERROR client.HConnectionManager$HConnectionImplementation: Unexpected
exception during initialization, aborting
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master

attached the flume.log file, Please guide me how to fix this error.

Thanks,
Prabhu.

On Thu, Jul 19, 2012 at 7:05 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Configuration looks fine. HBASE_HOME and CLASSPATH are also important.
> Please check these once.
>
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Jul 19, 2012 at 6:50 PM, prabhu k <[EMAIL PROTECTED]> wrote:
> > Thanks for the quick response,
> >
> > Yes, i have exported HADOOP_HOME variable, anything needs to do?
> >
> > On Thu, Jul 19, 2012 at 6:41 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >>
> >> Hi Prabhu,
> >>
> >>    Have you exported HADOOP_HOME variable??
> >>
> >> Regards,
> >>     Mohammad Tariq
> >>
> >>
> >> On Thu, Jul 19, 2012 at 6:34 PM, prabhu k <[EMAIL PROTECTED]>
> wrote:
> >> > Hi Users,
> >> >
> >> > I have followed the below link for move the sample data to hbase
> sink. i
> >> > then executed the below command, I am getting following error output.
> >> > pasted the flume.log log file.
> >> >
> >> > Please suggest and help on this issue.
> >> >
> >> >
> >> >
> http://cloudfront.blogspot.in/2012/06/how-to-move-data-into-hbase-table-using.html
> >> >
> >> > command
> >> > ==========> >> > bin/flume-ng agent -n hbase-agent -c conf/ -f conf/hbase-agent.conf
> >> >
> >> >
> >> > hbase-agent.conf
> >> > =================> >> > hbase-agent.sources = tail
> >> > hbase-agent.sinks = sink1
> >> > hbase-agent.channels = ch1
> >> > hbase-agent.sources.tail.type = exec
> >> > hbase-agent.sources.tail.command = tail -F /usr/local/demo.txt
> >> > hbase-agent.sources.tail.channels = ch1
> >> > hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> >> > hbase-agent.sinks.sink1.channel = ch1
> >> > hbase-agent.sinks.sink1.table = demo
> >> > hbase-agent.sinks.sink1.columnFamily = cf
> >> > hbase-agent.sinks.sink1.serializer > >> > org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> >> > hbase-agent.sinks.sink1.serializer.payloadColumn = col1
> >> > hbase-agent.sinks.sink1.serializer.keyType = timestamp
> >> > hbase-agent.sinks.sink1.serializer.rowPrefix = 1
> >> > hbase-agent.sinks.sink1.serializer.suffix = timestamp
> >> > hbase-agent.channels.ch1.type=memory
> >> >
> >> >
> >> > flume.log
> >> > ================> >> > /flume/flume-1.2.0-incubating-SNAPSHOT# more flume.log

Hello Prabhu,

           Sorry for the late response. Is your zookeeper running
properly? Is it where your shell expects it to be? Can you access
HBase's web ui on port 60010?

Regards,
    Mohammad Tariq
On Thu, Jul 19, 2012 at 8:17 PM, prabhu k <[EMAIL PROTECTED]> wrote:
> Hi Mohammad,
>
> After i export HBASE_HOME and CLASSPATH  flume-ng executed again, it seems
> keep on running, unexpectedly i have disconnected the session,
> and i checked the hbase demo table, i hit the command like
>
> scan 'demo'  getting like below.
>
> hbase(main):001:0> scan 'demo'
> ROW                                         COLUMN+CELL
> table view also keep on running, it's not exit the command. I then did
> ctrl+c.
>
>
> WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration occurred
> when trying to find JAAS configuration.
>
> WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration occurred
> when trying to find JAAS configuration.
>
>
>  WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper
> exception: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration occurred
> when trying to find JAAS configuration.
>
> ERROR zookeeper.ZooKeeperWatcher: hconnection Received unexpected
> KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> ERROR client.HConnectionManager$HConnectionImplementation: Unexpected
> exception during initialization, aborting
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> attached the flume.log file, Please guide me how to fix this error.
>
> Thanks,
> Prabhu.
>
> On Thu, Jul 19, 2012 at 7:05 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>
>> Configuration looks fine. HBASE_HOME and CLASSPATH are also important.
>> Please check these once.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Jul 19, 2012 at 6:50 PM, prabhu k <[EMAIL PROTECTED]> wrote:
>> > Thanks for the quick response,
>> >
>> > Yes, i have exported HADOOP_HOME variable, anything needs to do?
>> >
>> > On Thu, Jul 19, 2012 at 6:41 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> Hi Prabhu,
>> >>
>> >>    Have you exported HADOOP_HOME variable??
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >> On Thu, Jul 19, 2012 at 6:34 PM, prabhu k <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Hi Users,
>> >> >
>> >> > I have followed the below link for move the sample data to hbase
>> >> > sink. i
>> >> > then executed the below command, I am getting following error output.
>> >> > pasted the flume.log log file.
>> >> >
>> >> > Please suggest and help on this issue.
>> >> >
>> >> >
>> >> >
>> >> > http://cloudfront.blogspot.in/2012/06/how-to-move-data-into-hbase-table-using.html
>> >> >
>> >> > command
>> >> > ==========>> >> > bin/flume-ng agent -n hbase-agent -c conf/ -f conf/hbase-agent.conf
>> >> >
>> >> >
>> >> > hbase-agent.conf
>> >> > =================>> >> > hbase-agent.sources = tail
>> >> > hbase-agent.sinks = sink1
>> >> > hbase-agent.channels = ch1
>> >> > hbase-agent.sources.tail.type = exec
>> >> > hbase-agent.sources.tail.command = tail -F /usr/local/demo.txt
>> >> > hbase-agent.sources.tail.channels = ch1
>> >> > hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
>> >> > hbase-agent.sinks.sink1.channel = ch1
>> >> > hbase-agent.sinks.sink1.table = demo
>> >> > hbase-agent.sinks.sink1.columnFamily = cf

I am looking at example in
http://flume.apache.org/FlumeUserGuide.html#hbasesinks but this doesn't
show me how to generate row keys. My row keys are timeseries data.

Hi,

You might want to write a class which implements HbaseEventSerializer. To
test, SimpleHbaseEventSerializer might fit your needs. Using timestamps as
keys will create hotspots. As such it's recommended you "bucket" so that
each write is hitting say 1 of 32 buckets. Each agent would be configured
as follows:

...rowPrefix = bucket
...suffix = timestamp

Brock

On Sun, Jul 29, 2012 at 11:15 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:

> I am looking at example in
> http://flume.apache.org/FlumeUserGuide.html#hbasesinks but this doesn't
> show me how to generate row keys. My row keys are timeseries data.
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

On Sun, Jul 29, 2012 at 9:32 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Hi,
>
> You might want to write a class which implements HbaseEventSerializer. To
> test, SimpleHbaseEventSerializer might fit your needs. Using timestamps as
> keys will create hotspots. As such it's recommended you "bucket" so that
> each write is hitting say 1 of 32 buckets. Each agent would be configured
> as follows:
>
> ...rowPrefix = bucket
> ...suffix = timestamp
>
Yes my row keys are bucketed and regions would be split per bucket. I just
wanted to see how the row keys are generated. Looks like this needs to be
implemented, which probably makes sense. But it would be good to have some
support for timeseries data as well. Let me see if I can share my
EventSerializer after I get it working :) I'll take a look at
SimpleHbaseEventSerializer

>  Brock
>
> On Sun, Jul 29, 2012 at 11:15 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:
>
>> I am looking at example in
>> http://flume.apache.org/FlumeUserGuide.html#hbasesinks but this doesn't
>> show me how to generate row keys. My row keys are timeseries data.
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

Hi Mohammad,

I'm able to access Hbase web ui and i have run the script again today,
Please find the attached flume.log file.

Please suggest and help me on this issue.

Thanks&Regards,
Prabhu.
On Fri, Jul 20, 2012 at 1:26 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Hello Prabhu,
>
>            Sorry for the late response. Is your zookeeper running
> properly? Is it where your shell expects it to be? Can you access
> HBase's web ui on port 60010?
>
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Jul 19, 2012 at 8:17 PM, prabhu k <[EMAIL PROTECTED]> wrote:
> > Hi Mohammad,
> >
> > After i export HBASE_HOME and CLASSPATH  flume-ng executed again, it
> seems
> > keep on running, unexpectedly i have disconnected the session,
> > and i checked the hbase demo table, i hit the command like
> >
> > scan 'demo'  getting like below.
> >
> > hbase(main):001:0> scan 'demo'
> > ROW                                         COLUMN+CELL
> > table view also keep on running, it's not exit the command. I then did
> > ctrl+c.
> >
> >
> > WARN client.ZooKeeperSaslClient: SecurityException:
> > java.lang.SecurityException: Unable to locate a login configuration
> occurred
> > when trying to find JAAS configuration.
> >
> > WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper
> exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> >
> > WARN client.ZooKeeperSaslClient: SecurityException:
> > java.lang.SecurityException: Unable to locate a login configuration
> occurred
> > when trying to find JAAS configuration.
> >
> >
> >  WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper
> > exception: org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> >
> > WARN client.ZooKeeperSaslClient: SecurityException:
> > java.lang.SecurityException: Unable to locate a login configuration
> occurred
> > when trying to find JAAS configuration.
> >
> > ERROR zookeeper.ZooKeeperWatcher: hconnection Received unexpected
> > KeeperException, re-throwing exception
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> >
> > ERROR client.HConnectionManager$HConnectionImplementation: Unexpected
> > exception during initialization, aborting
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> >
> > attached the flume.log file, Please guide me how to fix this error.
> >
> > Thanks,
> > Prabhu.
> >
> > On Thu, Jul 19, 2012 at 7:05 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >>
> >> Configuration looks fine. HBASE_HOME and CLASSPATH are also important.
> >> Please check these once.
> >>
> >> Regards,
> >>     Mohammad Tariq
> >>
> >>
> >> On Thu, Jul 19, 2012 at 6:50 PM, prabhu k <[EMAIL PROTECTED]>
> wrote:
> >> > Thanks for the quick response,
> >> >
> >> > Yes, i have exported HADOOP_HOME variable, anything needs to do?
> >> >
> >> > On Thu, Jul 19, 2012 at 6:41 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> >> > wrote:
> >> >>
> >> >> Hi Prabhu,
> >> >>
> >> >>    Have you exported HADOOP_HOME variable??
> >> >>
> >> >> Regards,
> >> >>     Mohammad Tariq
> >> >>
> >> >>
> >> >> On Thu, Jul 19, 2012 at 6:34 PM, prabhu k <[EMAIL PROTECTED]>
> >> >> wrote:
> >> >> > Hi Users,
> >> >> >
> >> >> > I have followed the below link for move the sample data to hbase
> >> >> > sink. i
> >> >> > then executed the below command, I am getting following error
> output.
> >> >> > pasted the flume.log log file.
> >> >> >
> >> >> > Please suggest and help on this issue.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> http://cloudfront.blogspot.in/2012/06/how-to-move-data-into-hbase-table-using.html
> >> >> >
> >> >> > command
> >> >> > ==========> >> >> > bin/flume-ng agent -n hbase-agent -c conf/ -f conf/hbase-agent.conf
> >> >> >
> >> >> >
> >> >> > hbase-agent.conf
> >> >> > =================> >> >> > hbase-agent.sources = tail

below email contd...

the flume-ng script got stuck, and when i see the demo table no rows
inserted.

hbase(main):001:0> scan 'demo'
ROW                                         COLUMN+CELL
 11347427525582                             column=cf:col1,
timestamp=1347427517764, value1 row(s) in 0.3530 seconds
On Wed, Sep 12, 2012 at 11:24 AM, prabhu k <[EMAIL PROTECTED]> wrote:

> Hi Mohammad,
>
> I'm able to access Hbase web ui and i have run the script again today,
> Please find the attached flume.log file.
>
> Please suggest and help me on this issue.
>
> Thanks&Regards,
> Prabhu.
>
>
> On Fri, Jul 20, 2012 at 1:26 AM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>
>> Hello Prabhu,
>>
>>            Sorry for the late response. Is your zookeeper running
>> properly? Is it where your shell expects it to be? Can you access
>> HBase's web ui on port 60010?
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Jul 19, 2012 at 8:17 PM, prabhu k <[EMAIL PROTECTED]> wrote:
>> > Hi Mohammad,
>> >
>> > After i export HBASE_HOME and CLASSPATH  flume-ng executed again, it
>> seems
>> > keep on running, unexpectedly i have disconnected the session,
>> > and i checked the hbase demo table, i hit the command like
>> >
>> > scan 'demo'  getting like below.
>> >
>> > hbase(main):001:0> scan 'demo'
>> > ROW                                         COLUMN+CELL
>> > table view also keep on running, it's not exit the command. I then did
>> > ctrl+c.
>> >
>> >
>> > WARN client.ZooKeeperSaslClient: SecurityException:
>> > java.lang.SecurityException: Unable to locate a login configuration
>> occurred
>> > when trying to find JAAS configuration.
>> >
>> > WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper
>> exception:
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase/master
>> >
>> > WARN client.ZooKeeperSaslClient: SecurityException:
>> > java.lang.SecurityException: Unable to locate a login configuration
>> occurred
>> > when trying to find JAAS configuration.
>> >
>> >
>> >  WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper
>> > exception: org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase/master
>> >
>> > WARN client.ZooKeeperSaslClient: SecurityException:
>> > java.lang.SecurityException: Unable to locate a login configuration
>> occurred
>> > when trying to find JAAS configuration.
>> >
>> > ERROR zookeeper.ZooKeeperWatcher: hconnection Received unexpected
>> > KeeperException, re-throwing exception
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase/master
>> >
>> > ERROR client.HConnectionManager$HConnectionImplementation: Unexpected
>> > exception during initialization, aborting
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase/master
>> >
>> > attached the flume.log file, Please guide me how to fix this error.
>> >
>> > Thanks,
>> > Prabhu.
>> >
>> > On Thu, Jul 19, 2012 at 7:05 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> Configuration looks fine. HBASE_HOME and CLASSPATH are also important.
>> >> Please check these once.
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >> On Thu, Jul 19, 2012 at 6:50 PM, prabhu k <[EMAIL PROTECTED]>
>> wrote:
>> >> > Thanks for the quick response,
>> >> >
>> >> > Yes, i have exported HADOOP_HOME variable, anything needs to do?
>> >> >
>> >> > On Thu, Jul 19, 2012 at 6:41 PM, Mohammad Tariq <[EMAIL PROTECTED]>
>> >> > wrote:
>> >> >>
>> >> >> Hi Prabhu,
>> >> >>
>> >> >>    Have you exported HADOOP_HOME variable??
>> >> >>
>> >> >> Regards,
>> >> >>     Mohammad Tariq
>> >> >>
>> >> >>
>> >> >> On Thu, Jul 19, 2012 at 6:34 PM, prabhu k <[EMAIL PROTECTED]>
>> >> >> wrote:
>> >> >> > Hi Users,
>> >> >> >
>> >> >> > I have followed the below link for move the sample data to hbase
>