Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Moving from Flume OG to Flume NG


+
Kumar, Deepak8 2012-10-05, 11:19
+
Brock Noland 2012-10-05, 14:35
+
Kumar, Deepak8 2012-10-05, 15:08
+
Brock Noland 2012-10-05, 15:17
+
Kumar, Deepak8 2012-10-05, 15:28
+
Brock Noland 2012-10-08, 16:02
Copy link to this message
-
RE: Moving from Flume OG to Flume NG
Brock,
I copied hbase-site.xml in conf & is working fine now.

Thanks,

Deepak

-----Original Message-----
From: Brock Noland [mailto:[EMAIL PROTECTED]]
Sent: Monday, October 08, 2012 12:03 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: Moving from Flume OG to Flume NG

Deepak,

What you have should work. Could you post your full log file on
pastebin.com or send it to me privately? (List doesn't take it)

Brock

On Fri, Oct 5, 2012 at 10:28 AM, Kumar, Deepak8 <[EMAIL PROTECTED]> wrote:
> Brock,
>
> Apologize miss spelled your name last time. Hbase.jar as well as
> hbase-site.xml is there in /opt/sponge/fluming/flume/lib directory. Could
> you please guide me, how to provide hbase command while starting flume?
>
> Regards,
>
> Deepak
>
>
>
> -----Original Message-----
> From: Brock Noland [mailto:[EMAIL PROTECTED]]
> Sent: Friday, October 05, 2012 11:17 AM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: Moving from Flume OG to Flume NG
>
>
>
> Does either of
>
>
>
> '/opt/sponge/flumeNG/flume/conf/deepakConf:/opt/sponge/flumeNG/flume/lib/*'
>
>
>
> contain the hbase jars? Generally you want the hbase command to be in
>
> the path of the user starting flume. In the hbase conf folder would be
>
> the hbase-site.xml which would get picked up by the flume agent.
>
>
>
> Brock
>
>
>
> On Fri, Oct 5, 2012 at 10:08 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>
> wrote:
>
>> Broke, this is the command it prints.
>
>> [root@vm-93ff-5f31 flume]# bin/flume-ng agent --conf ./conf/deepakConf/ -f
>> ./conf/deepakConf/flume.conf -name hbase-agent
>
>> + exec /opt/sponge/java//bin/java -Xmx20m -cp
>> '/opt/sponge/flumeNG/flume/conf/deepakConf:/opt/sponge/flumeNG/flume/lib/*'
>> -Djava.library.path= org.apache.flume.node.Application -f
>> ./conf/deepakConf/flume.conf -name hbase-agent
>
>>
>
>> Regards,
>
>> Deepak
>
>>
>
>> -----Original Message-----
>
>> From: Brock Noland [mailto:[EMAIL PROTECTED]]
>
>> Sent: Friday, October 05, 2012 10:35 AM
>
>> To: [EMAIL PROTECTED]
>
>> Cc: [EMAIL PROTECTED]
>
>> Subject: Re: Moving from Flume OG to Flume NG
>
>>
>
>> Hi,
>
>>
>
>> When start the flume agent it prints out the command it is running to
>
>> start the agent. Can you send that along?
>
>>
>
>> Brock
>
>>
>
>> On Fri, Oct 5, 2012 at 6:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>
>> wrote:
>
>>> Hi Adam,
>
>>>
>
>>> I am just trying to use hbase sink with flume. It is a sort of Hello
>>> World.
>
>>> My configuration file looks something like this. When I execute the node
>>> for
>
>>> hbase-agent, the log shows following error:
>
>>>
>
>>>
>
>>>
>
>>> 2012-10-05 06:55:05,330 ERROR hbase.HBaseSink: Could not load table, test
>
>>> from HBase
>
>>>
>
>>>
>
>>>
>
>>> It seems, it looks for hbase at localhost, but the hbase is running on
>
>>> another VM. How should I provide the hbase VM details here? For testing
>
>>> purpose, I have created a hbase-site.xml at location flume/lib/ but it
>
>>> doesn't makes any different. My Hbase version is 0.92.1-cdh4.0.1 & Flume
>>> is
>
>>> 1.2.
>
>>>
>
>>>
>
>>>
>
>>>
>
>>>
>
>>> Configuration File:
>
>>>
>
>>>
>
>>>
>
>>> hbase-agent.sources = tail
>
>>>
>
>>> hbase-agent.sinks = sink1
>
>>>
>
>>> hbase-agent.channels = ch1
>
>>>
>
>>> hbase-agent.sources.tail.type = exec
>
>>>
>
>>> hbase-agent.sources.tail.command = tail -F
>>> /opt/sponge/flumeNG/jongkook.txt
>
>>>
>
>>> hbase-agent.sources.tail.channels = ch1
>
>>>
>
>>> hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
>
>>>
>
>>> hbase-agent.sinks.sink1.channel = ch1
>
>>>
>
>>> hbase-agent.sinks.sink1.table = test
>
>>>
>
>>> hbase-agent.sinks.sink1.columnFamily = f1
>
>>>
>
>>> hbase-agent.sinks.sink1.serializer >
>>> org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
>
>>>
>
>>> hbase-agent.sinks.sink1.serializer.payloadColumn = col1
>
>>>
>
>>> hbase-agent.sinks.sink1.serializer.keyType = timestamp
>
>>>
>
>>> hbase-agent.sinks.sink1.serializer.rowPrefix = 1
>
>>>
>
>>> hbase-agent.sinks.sink1.serializer.suffix = timestamp

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Brock Noland 2012-10-09, 14:51
+
Kumar, Deepak8 2012-09-18, 18:02
+
Brock Noland 2012-09-18, 18:05
+
Shumin Wu 2012-09-18, 18:10
+
Kumar, Deepak8 2012-10-03, 10:42
+
Shumin Wu 2012-10-03, 14:10