-
Re: Error in Upload the log file into hdfsAlexander Alten-Lorenz 2012-11-29, 19:14
Hi,
As Brock wrote, in the upcoming stable release 1.3.0 we implemented SpoolDirectorySource, which will do that for you. The appropriate Jira:
https://issues.apache.org/jira/browse/FLUME-1425
Cheers,
Alex
On Nov 29, 2012, at 6:03 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> Yes Flume-1.3.0 supports a source called Spool Directory Source which does
> this.
>
> On Thu, Nov 29, 2012 at 10:59 AM, kashif khan <[EMAIL PROTECTED]>wrote:
>
>> Thanks Alexander
>>
>> Yeah you right, I was not read the document properly, sorry for that. once
>> read the documents, I have done and now uploading the data.
>>
>> I have just question about Flume 1.3. (which to be release very soon) Is
>> it support to automatically detect the file in folder and upload into HDFS?
>>
>> Many thanks
>>
>> Best regards
>>
>>
>> On Thu, Nov 29, 2012 at 3:39 PM, Alexander Alten-Lorenz <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> agent.channels.memoryChannel.transactionCapacity=1000
>>>
>>> Is wrong. You cant have equal or more transaction capacity in a channel
>>> as configured capacity. Use the opposite, when you want to use it.
>>> from our Guide:
>>>
>>> capacity NUM The max number of events stored in the
>>> channel
>>> transactionCapacity NUM The max number of events stored in the
>>> channel per transaction
>>>
>>> Try this:
>>>
>>> agent.channels.memoryChannel.capacity = 1000
>>> agent.channels.memoryChannel.transactionCapacity=10
>>>
>>> cheers
>>> - Alex
>>>
>>>
>>> On Nov 29, 2012, at 1:03 PM, kashif khan <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am just struggling to learn the flume and doing some testing. I am
>>>> running two agents (agent, agent1). The agent used to upload the log
>>> data
>>>> into hdfs and agent1 used as logger. The configuration of two agents as:
>>>>
>>>> agent.sources = tail
>>>> agent.channels = memoryChannel
>>>> agent.sinks = hdfs-clusterSink
>>>>
>>>> agent.sources.tail.type = exec
>>>> agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
>>>> agent.sources.tail.channels = memoryChannel
>>>>
>>>> agent.sinks.hdfs-clusterSink.
>>>> channel = memoryChannel
>>>> agent.sinks.hdfs-clusterSink.type = hdfs
>>>> agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
>>>> hadoop1.example.com/user/root/Test/
>>>>
>>>>
>>>> agent.channels.memoryChannel.type = memory
>>>> agent.channels.memoryChannel.transactionCapacity=1000
>>>> agent.channels.memoryChannel.capacity = 100
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sources = source1
>>>> agent1.sinks = sink1
>>>> agent1.channels = channel1
>>>>
>>>> # Describe/configure source1
>>>> agent1.sources.source1.type = netcat
>>>> agent1.sources.source1.bind = localhost
>>>> agent1.sources.source1.port = 44444
>>>>
>>>> # Describe sink1
>>>> agent1.sinks.sink1.type = logger
>>>>
>>>> # Use a channel which buffers events in memory
>>>> agent1.channels.channel1.type = memory
>>>> agent1.channels.channel1.capacity = 1000
>>>> agent1.channels.channel1.transactionCapactiy = 100
>>>>
>>>> # Bind the source and sink to the channel
>>>> agent1.sources.source1.channels = channel1
>>>> agent1.sinks.sink1.channel = channel1
>>>>
>>>>
>>>> I dont know why it does not upload the log file into hdfs. where I am
>>> doing
>>>> mistake . If anyone who have solution please let me know.
>>>>
>>>>
>>>> The log file as:
>>>>
>>>>
>>>> 29 Nov 2012 11:49:13,046 INFO [main]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 1
>>>> 29 Nov 2012 11:49:13,050 INFO [main]
>>>> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
>>> agent
>>>> 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>>> -
>>>> Node manager starting
>>>> 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 10
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
As Brock wrote, in the upcoming stable release 1.3.0 we implemented SpoolDirectorySource, which will do that for you. The appropriate Jira:
https://issues.apache.org/jira/browse/FLUME-1425
Cheers,
Alex
On Nov 29, 2012, at 6:03 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> Yes Flume-1.3.0 supports a source called Spool Directory Source which does
> this.
>
> On Thu, Nov 29, 2012 at 10:59 AM, kashif khan <[EMAIL PROTECTED]>wrote:
>
>> Thanks Alexander
>>
>> Yeah you right, I was not read the document properly, sorry for that. once
>> read the documents, I have done and now uploading the data.
>>
>> I have just question about Flume 1.3. (which to be release very soon) Is
>> it support to automatically detect the file in folder and upload into HDFS?
>>
>> Many thanks
>>
>> Best regards
>>
>>
>> On Thu, Nov 29, 2012 at 3:39 PM, Alexander Alten-Lorenz <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> agent.channels.memoryChannel.transactionCapacity=1000
>>>
>>> Is wrong. You cant have equal or more transaction capacity in a channel
>>> as configured capacity. Use the opposite, when you want to use it.
>>> from our Guide:
>>>
>>> capacity NUM The max number of events stored in the
>>> channel
>>> transactionCapacity NUM The max number of events stored in the
>>> channel per transaction
>>>
>>> Try this:
>>>
>>> agent.channels.memoryChannel.capacity = 1000
>>> agent.channels.memoryChannel.transactionCapacity=10
>>>
>>> cheers
>>> - Alex
>>>
>>>
>>> On Nov 29, 2012, at 1:03 PM, kashif khan <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am just struggling to learn the flume and doing some testing. I am
>>>> running two agents (agent, agent1). The agent used to upload the log
>>> data
>>>> into hdfs and agent1 used as logger. The configuration of two agents as:
>>>>
>>>> agent.sources = tail
>>>> agent.channels = memoryChannel
>>>> agent.sinks = hdfs-clusterSink
>>>>
>>>> agent.sources.tail.type = exec
>>>> agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
>>>> agent.sources.tail.channels = memoryChannel
>>>>
>>>> agent.sinks.hdfs-clusterSink.
>>>> channel = memoryChannel
>>>> agent.sinks.hdfs-clusterSink.type = hdfs
>>>> agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
>>>> hadoop1.example.com/user/root/Test/
>>>>
>>>>
>>>> agent.channels.memoryChannel.type = memory
>>>> agent.channels.memoryChannel.transactionCapacity=1000
>>>> agent.channels.memoryChannel.capacity = 100
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sources = source1
>>>> agent1.sinks = sink1
>>>> agent1.channels = channel1
>>>>
>>>> # Describe/configure source1
>>>> agent1.sources.source1.type = netcat
>>>> agent1.sources.source1.bind = localhost
>>>> agent1.sources.source1.port = 44444
>>>>
>>>> # Describe sink1
>>>> agent1.sinks.sink1.type = logger
>>>>
>>>> # Use a channel which buffers events in memory
>>>> agent1.channels.channel1.type = memory
>>>> agent1.channels.channel1.capacity = 1000
>>>> agent1.channels.channel1.transactionCapactiy = 100
>>>>
>>>> # Bind the source and sink to the channel
>>>> agent1.sources.source1.channels = channel1
>>>> agent1.sinks.sink1.channel = channel1
>>>>
>>>>
>>>> I dont know why it does not upload the log file into hdfs. where I am
>>> doing
>>>> mistake . If anyone who have solution please let me know.
>>>>
>>>>
>>>> The log file as:
>>>>
>>>>
>>>> 29 Nov 2012 11:49:13,046 INFO [main]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 1
>>>> 29 Nov 2012 11:49:13,050 INFO [main]
>>>> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
>>> agent
>>>> 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>>> -
>>>> Node manager starting
>>>> 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 10
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
