Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Help on HDFS sink hangs


Copy link to this message
-
Re: Help on HDFS sink hangs
Can you get a thread dump to see what is going on? That'll give us an idea
of what is going on.
On Wed, Jan 8, 2014 at 8:38 PM, Chen Wang <[EMAIL PROTECTED]>wrote:

> David,
> I am using flume-ng 1.4.0.
> Thanks,
> Chen
>
>
> On Wed, Jan 8, 2014 at 5:26 PM, David Sinclair <
> [EMAIL PROTECTED]> wrote:
>
>> What version of flume are you using?
>>
>>
>> On Wed, Jan 8, 2014 at 8:01 PM, Chen Wang <[EMAIL PROTECTED]>wrote:
>>
>>> Hey, Guys,
>>> First time using flume. It goes all well, however, I found that in my
>>> custom source when I am calling
>>>
>>> Event event = EventBuilder.withBody(mySignal.toString().getBytes(),
>>> headers);
>>>
>>>   try {
>>>
>>>   logger.debug("submitting to channel");
>>>
>>>   channel.processEvent(event);
>>>
>>> }catch (ChannelException e) {
>>>
>>>   logger.debug("failed to submit to channel, exception {}",e);
>>>
>>>   throw e;
>>>
>>>   }
>>>
>>>
>>> It just hangs there on "submitting to channel", and hdfs file apparently
>>> is not created. Just wondering how can I trouble shoot this? I am running
>>> it as
>>>
>>> bin/flume-ng agent -c conf -f conf/flume.conf -n SocketAgent
>>> -Dflume.root.logger=DEBUG,console
>>>
>>> and my config file looks like:
>>>
>>> SocketAgent.sinks.HDFS.channel = MemChannel
>>>
>>> SocketAgent.sinks.HDFS.type = hdfs
>>>
>>> SocketAgent.sinks.HDFS.hdfs.path >>> maprfs:///hive/myname.db/targeting_storm/%{epoch}
>>>
>>> SocketAgent.sinks.HDFS.hdfs.fileType = SequenceFile
>>>
>>> SocketAgent.sinks.HDFS.hdfs.writeFormat = Text
>>>
>>> SocketAgent.sinks.HDFS.hdfs.batchSize = 1000
>>>
>>> SocketAgent.sinks.HDFS.hdfs.rollSize = 0
>>>
>>> SocketAgent.sinks.HDFS.hdfs.rollCount = 10000
>>>
>>> SocketAgent.sinks.HDFS.hdfs.rollInterval = 600
>>>
>>>
>>> SocketAgent.channels.MemChannel.type = memory
>>>
>>> SocketAgent.channels.MemChannel.capacity = 10000
>>>
>>> SocketAgent.channels.MemChannel.transactionCapacity = 100
>>>
>>>
>>> Thank you very much!!
>>>
>>> Chen
>>>
>>
>>
>