Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Moving from Flume OG to Flume NG


+
Kumar, Deepak8 2012-10-05, 11:19
+
Brock Noland 2012-10-05, 14:35
+
Kumar, Deepak8 2012-10-05, 15:08
+
Brock Noland 2012-10-05, 15:17
Copy link to this message
-
RE: Moving from Flume OG to Flume NG
Kumar, Deepak8 2012-10-05, 15:28
Brock,

Apologize miss spelled your name last time. Hbase.jar as well as hbase-site.xml is there in /opt/sponge/fluming/flume/lib directory. Could you please guide me, how to provide hbase command while starting flume?

Regards,

Deepak

-----Original Message-----
From: Brock Noland [mailto:[EMAIL PROTECTED]]
Sent: Friday, October 05, 2012 11:17 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: Moving from Flume OG to Flume NG

Does either of

'/opt/sponge/flumeNG/flume/conf/deepakConf:/opt/sponge/flumeNG/flume/lib/*'

contain the hbase jars? Generally you want the hbase command to be in

the path of the user starting flume. In the hbase conf folder would be

the hbase-site.xml which would get picked up by the flume agent.

Brock

On Fri, Oct 5, 2012 at 10:08 AM, Kumar, Deepak8 <[EMAIL PROTECTED]> wrote:

> Broke, this is the command it prints.

> [root@vm-93ff-5f31 flume]# bin/flume-ng agent --conf ./conf/deepakConf/ -f ./conf/deepakConf/flume.conf -name hbase-agent

> + exec /opt/sponge/java//bin/java -Xmx20m -cp '/opt/sponge/flumeNG/flume/conf/deepakConf:/opt/sponge/flumeNG/flume/lib/*' -Djava.library.path= org.apache.flume.node.Application -f ./conf/deepakConf/flume.conf -name hbase-agent

>

> Regards,

> Deepak

>

> -----Original Message-----

> From: Brock Noland [mailto:[EMAIL PROTECTED]]

> Sent: Friday, October 05, 2012 10:35 AM

> To: [EMAIL PROTECTED]

> Cc: [EMAIL PROTECTED]

> Subject: Re: Moving from Flume OG to Flume NG

>

> Hi,

>

> When start the flume agent it prints out the command it is running to

> start the agent. Can you send that along?

>

> Brock

>

> On Fri, Oct 5, 2012 at 6:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]> wrote:

>> Hi Adam,

>>

>> I am just trying to use hbase sink with flume. It is a sort of Hello World.

>> My configuration file looks something like this. When I execute the node for

>> hbase-agent, the log shows following error:

>>

>>

>>

>> 2012-10-05 06:55:05,330 ERROR hbase.HBaseSink: Could not load table, test

>> from HBase

>>

>>

>>

>> It seems, it looks for hbase at localhost, but the hbase is running on

>> another VM. How should I provide the hbase VM details here? For testing

>> purpose, I have created a hbase-site.xml at location flume/lib/ but it

>> doesn't makes any different. My Hbase version is 0.92.1-cdh4.0.1 & Flume is

>> 1.2.

>>

>>

>>

>>

>>

>> Configuration File:

>>

>>

>>

>> hbase-agent.sources = tail

>>

>> hbase-agent.sinks = sink1

>>

>> hbase-agent.channels = ch1

>>

>> hbase-agent.sources.tail.type = exec

>>

>> hbase-agent.sources.tail.command = tail -F /opt/sponge/flumeNG/jongkook.txt

>>

>> hbase-agent.sources.tail.channels = ch1

>>

>> hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink

>>

>> hbase-agent.sinks.sink1.channel = ch1

>>

>> hbase-agent.sinks.sink1.table = test

>>

>> hbase-agent.sinks.sink1.columnFamily = f1

>>

>> hbase-agent.sinks.sink1.serializer
>> org.apache.flume.sink.hbase.SimpleHbaseEventSerializer

>>

>> hbase-agent.sinks.sink1.serializer.payloadColumn = col1

>>

>> hbase-agent.sinks.sink1.serializer.keyType = timestamp

>>

>> hbase-agent.sinks.sink1.serializer.rowPrefix = 1

>>

>> hbase-agent.sinks.sink1.serializer.suffix = timestamp

>>

>> hbase-agent.channels.ch1.type=memory

>>

>>

>>

>> Regards,

>>

>> Deepak

>

>

>

> --

> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

--

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Brock Noland 2012-10-08, 16:02
+
Kumar, Deepak8 2012-10-09, 06:03
+
Brock Noland 2012-10-09, 14:51
+
Kumar, Deepak8 2012-09-18, 18:02
+
Brock Noland 2012-09-18, 18:05
+
Shumin Wu 2012-09-18, 18:10
+
Kumar, Deepak8 2012-10-03, 10:42
+
Shumin Wu 2012-10-03, 14:10