Flume, mail # user - Re: Error in Upload the log file into hdfs

Error in Upload the log file into hdfs

kashif khan 2012-11-29, 12:03

Hi,

I am just struggling to learn the flume and doing some testing. I am
running two agents (agent, agent1). The agent used to upload the log data
into hdfs and agent1 used as logger. The configuration of two agents as:

agent.sources = tail
agent.channels = memoryChannel
agent.sinks = hdfs-clusterSink

agent.sources.tail.type = exec
agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
agent.sources.tail.channels = memoryChannel

agent.sinks.hdfs-clusterSink.
channel = memoryChannel
agent.sinks.hdfs-clusterSink.type = hdfs
agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
hadoop1.example.com/user/root/Test/
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.transactionCapacity=1000
agent.channels.memoryChannel.capacity = 100
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1

# Describe/configure source1
agent1.sources.source1.type = netcat
agent1.sources.source1.bind = localhost
agent1.sources.source1.port = 44444

# Describe sink1
agent1.sinks.sink1.type = logger

# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapactiy = 100

# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
I dont know why it does not upload the log file into hdfs. where I am doing
mistake . If anyone who have solution please let me know.
The log file as:
29 Nov 2012 11:49:13,046 INFO [main]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 1
29 Nov 2012 11:49:13,050 INFO [main]
(org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) -
Node manager starting
29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 10
29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) -
Configuration provider starting
29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
- Reloading configuration file:/etc/flume-ng/conf/flume.conf
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: hdfs-clusterSink Agent: agent
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: sink1 Agent: agent1
29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,087 INFO [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122) -
Post-validation flume configuration contains configuration for agents:
[agent, agent1]
29 Nov 2012 11:49:13,088 INFO [conf-file-poller-0]
(org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
- Creating channels
29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
- Failed to load configuration data. Exception follows.
java.lang.IllegalStateException
   at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
   at
org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
   at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
   at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
   at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
   at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
   at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
   at
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
   at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
   at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
29 Nov 2012 11:49:22,157 INFO [main]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
lifecycle supervisor 1
29 Nov 2012 11:49:22,161 INFO [m

Re: Error in Upload the log file into hdfs

Alexander Alten-Lorenz 2012-11-29, 15:39

Hi,

agent.channels.memoryChannel.transactionCapacity=1000

Is wrong. You cant have equal or more transaction capacity in a channel as configured capacity. Use the opposite, when you want to use it.
from our Guide:

capacity NUM The max number of events stored in the channel
transactionCapacity NUM The max number of events stored in the channel per transaction

Try this:

agent.channels.memoryChannel.capacity = 1000
agent.channels.memoryChannel.transactionCapacity=10

cheers
- Alex
On Nov 29, 2012, at 1:03 PM, kashif khan <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am just struggling to learn the flume and doing some testing. I am
> running two agents (agent, agent1). The agent used to upload the log data
> into hdfs and agent1 used as logger. The configuration of two agents as:
>
> agent.sources = tail
> agent.channels = memoryChannel
> agent.sinks = hdfs-clusterSink
>
> agent.sources.tail.type = exec
> agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
> agent.sources.tail.channels = memoryChannel
>
> agent.sinks.hdfs-clusterSink.
> channel = memoryChannel
> agent.sinks.hdfs-clusterSink.type = hdfs
> agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
> hadoop1.example.com/user/root/Test/
>
>
> agent.channels.memoryChannel.type = memory
> agent.channels.memoryChannel.transactionCapacity=1000
> agent.channels.memoryChannel.capacity = 100
>
>
>
>
> agent1.sources = source1
> agent1.sinks = sink1
> agent1.channels = channel1
>
> # Describe/configure source1
> agent1.sources.source1.type = netcat
> agent1.sources.source1.bind = localhost
> agent1.sources.source1.port = 44444
>
> # Describe sink1
> agent1.sinks.sink1.type = logger
>
> # Use a channel which buffers events in memory
> agent1.channels.channel1.type = memory
> agent1.channels.channel1.capacity = 1000
> agent1.channels.channel1.transactionCapactiy = 100
>
> # Bind the source and sink to the channel
> agent1.sources.source1.channels = channel1
> agent1.sinks.sink1.channel = channel1
>
>
> I dont know why it does not upload the log file into hdfs. where I am doing
> mistake . If anyone who have solution please let me know.
>
>
> The log file as:
>
>
> 29 Nov 2012 11:49:13,046 INFO [main]
> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> lifecycle supervisor 1
> 29 Nov 2012 11:49:13,050 INFO [main]
> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
> 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203) -
> Node manager starting
> 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> lifecycle supervisor 10
> 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67) -
> Configuration provider starting
> 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> - Added sinks: hdfs-clusterSink Agent: agent
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:hdfs-clusterSink
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:sink1
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:hdfs-clusterSink
> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> - Processing:sink1
> 29 Nov 2012 11:49:13,058 INFO [conf-file-poller-0]

Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Error in Upload the log file into hdfs

kashif khan 2012-11-29, 16:59

Thanks Alexander

Yeah you right, I was not read the document properly, sorry for that. once
read the documents, I have done and now uploading the data.

I have just question about Flume 1.3. (which to be release very soon) Is
it support to automatically detect the file in folder and upload into HDFS?

Many thanks

Best regards

On Thu, Nov 29, 2012 at 3:39 PM, Alexander Alten-Lorenz <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> agent.channels.memoryChannel.transactionCapacity=1000
>
> Is wrong. You cant have equal or more transaction capacity in a channel as
> configured capacity. Use the opposite, when you want to use it.
> from our Guide:
>
> capacity NUM The max number of events stored in the
> channel
> transactionCapacity NUM The max number of events stored in the
> channel per transaction
>
> Try this:
>
> agent.channels.memoryChannel.capacity = 1000
> agent.channels.memoryChannel.transactionCapacity=10
>
> cheers
> - Alex
>
>
> On Nov 29, 2012, at 1:03 PM, kashif khan <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I am just struggling to learn the flume and doing some testing. I am
> > running two agents (agent, agent1). The agent used to upload the log data
> > into hdfs and agent1 used as logger. The configuration of two agents as:
> >
> > agent.sources = tail
> > agent.channels = memoryChannel
> > agent.sinks = hdfs-clusterSink
> >
> > agent.sources.tail.type = exec
> > agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
> > agent.sources.tail.channels = memoryChannel
> >
> > agent.sinks.hdfs-clusterSink.
> > channel = memoryChannel
> > agent.sinks.hdfs-clusterSink.type = hdfs
> > agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
> > hadoop1.example.com/user/root/Test/
> >
> >
> > agent.channels.memoryChannel.type = memory
> > agent.channels.memoryChannel.transactionCapacity=1000
> > agent.channels.memoryChannel.capacity = 100
> >
> >
> >
> >
> > agent1.sources = source1
> > agent1.sinks = sink1
> > agent1.channels = channel1
> >
> > # Describe/configure source1
> > agent1.sources.source1.type = netcat
> > agent1.sources.source1.bind = localhost
> > agent1.sources.source1.port = 44444
> >
> > # Describe sink1
> > agent1.sinks.sink1.type = logger
> >
> > # Use a channel which buffers events in memory
> > agent1.channels.channel1.type = memory
> > agent1.channels.channel1.capacity = 1000
> > agent1.channels.channel1.transactionCapactiy = 100
> >
> > # Bind the source and sink to the channel
> > agent1.sources.source1.channels = channel1
> > agent1.sinks.sink1.channel = channel1
> >
> >
> > I dont know why it does not upload the log file into hdfs. where I am
> doing
> > mistake . If anyone who have solution please let me know.
> >
> >
> > The log file as:
> >
> >
> > 29 Nov 2012 11:49:13,046 INFO [main]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 1
> > 29 Nov 2012 11:49:13,050 INFO [main]
> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
> > 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
> -
> > Node manager starting
> > 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 10
> > 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
> -
> > Configuration provider starting
> > 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: hdfs-clusterSink Agent: agent
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)

Re: Error in Upload the log file into hdfs

Brock Noland 2012-11-29, 17:03

Yes Flume-1.3.0 supports a source called Spool Directory Source which does
this.

On Thu, Nov 29, 2012 at 10:59 AM, kashif khan <[EMAIL PROTECTED]>wrote:

> Thanks Alexander
>
> Yeah you right, I was not read the document properly, sorry for that. once
> read the documents, I have done and now uploading the data.
>
> I have just question about Flume 1.3. (which to be release very soon) Is
> it support to automatically detect the file in folder and upload into HDFS?
>
> Many thanks
>
> Best regards
>
>
> On Thu, Nov 29, 2012 at 3:39 PM, Alexander Alten-Lorenz <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> agent.channels.memoryChannel.transactionCapacity=1000
>>
>> Is wrong. You cant have equal or more transaction capacity in a channel
>> as configured capacity. Use the opposite, when you want to use it.
>> from our Guide:
>>
>> capacity NUM The max number of events stored in the
>> channel
>> transactionCapacity NUM The max number of events stored in the
>> channel per transaction
>>
>> Try this:
>>
>> agent.channels.memoryChannel.capacity = 1000
>> agent.channels.memoryChannel.transactionCapacity=10
>>
>> cheers
>> - Alex
>>
>>
>> On Nov 29, 2012, at 1:03 PM, kashif khan <[EMAIL PROTECTED]> wrote:
>>
>> > Hi,
>> >
>> > I am just struggling to learn the flume and doing some testing. I am
>> > running two agents (agent, agent1). The agent used to upload the log
>> data
>> > into hdfs and agent1 used as logger. The configuration of two agents as:
>> >
>> > agent.sources = tail
>> > agent.channels = memoryChannel
>> > agent.sinks = hdfs-clusterSink
>> >
>> > agent.sources.tail.type = exec
>> > agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
>> > agent.sources.tail.channels = memoryChannel
>> >
>> > agent.sinks.hdfs-clusterSink.
>> > channel = memoryChannel
>> > agent.sinks.hdfs-clusterSink.type = hdfs
>> > agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
>> > hadoop1.example.com/user/root/Test/
>> >
>> >
>> > agent.channels.memoryChannel.type = memory
>> > agent.channels.memoryChannel.transactionCapacity=1000
>> > agent.channels.memoryChannel.capacity = 100
>> >
>> >
>> >
>> >
>> > agent1.sources = source1
>> > agent1.sinks = sink1
>> > agent1.channels = channel1
>> >
>> > # Describe/configure source1
>> > agent1.sources.source1.type = netcat
>> > agent1.sources.source1.bind = localhost
>> > agent1.sources.source1.port = 44444
>> >
>> > # Describe sink1
>> > agent1.sinks.sink1.type = logger
>> >
>> > # Use a channel which buffers events in memory
>> > agent1.channels.channel1.type = memory
>> > agent1.channels.channel1.capacity = 1000
>> > agent1.channels.channel1.transactionCapactiy = 100
>> >
>> > # Bind the source and sink to the channel
>> > agent1.sources.source1.channels = channel1
>> > agent1.sinks.sink1.channel = channel1
>> >
>> >
>> > I dont know why it does not upload the log file into hdfs. where I am
>> doing
>> > mistake . If anyone who have solution please let me know.
>> >
>> >
>> > The log file as:
>> >
>> >
>> > 29 Nov 2012 11:49:13,046 INFO [main]
>> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>> > lifecycle supervisor 1
>> > 29 Nov 2012 11:49:13,050 INFO [main]
>> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
>> agent
>> > 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
>> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>> -
>> > Node manager starting
>> > 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
>> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>> > lifecycle supervisor 10
>> > 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
>> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>> -
>> > Configuration provider starting
>> > 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
>> >
>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
>> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Re: Error in Upload the log file into hdfs

Alexander Alten-Lorenz 2012-11-29, 19:14

Hi,

As Brock wrote, in the upcoming stable release 1.3.0 we implemented SpoolDirectorySource, which will do that for you. The appropriate Jira:
https://issues.apache.org/jira/browse/FLUME-1425

Cheers,
Alex
On Nov 29, 2012, at 6:03 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Yes Flume-1.3.0 supports a source called Spool Directory Source which does
> this.
>
> On Thu, Nov 29, 2012 at 10:59 AM, kashif khan <[EMAIL PROTECTED]>wrote:
>
>> Thanks Alexander
>>
>> Yeah you right, I was not read the document properly, sorry for that. once
>> read the documents, I have done and now uploading the data.
>>
>> I have just question about Flume 1.3. (which to be release very soon) Is
>> it support to automatically detect the file in folder and upload into HDFS?
>>
>> Many thanks
>>
>> Best regards
>>
>>
>> On Thu, Nov 29, 2012 at 3:39 PM, Alexander Alten-Lorenz <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> agent.channels.memoryChannel.transactionCapacity=1000
>>>
>>> Is wrong. You cant have equal or more transaction capacity in a channel
>>> as configured capacity. Use the opposite, when you want to use it.
>>> from our Guide:
>>>
>>> capacity NUM The max number of events stored in the
>>> channel
>>> transactionCapacity NUM The max number of events stored in the
>>> channel per transaction
>>>
>>> Try this:
>>>
>>> agent.channels.memoryChannel.capacity = 1000
>>> agent.channels.memoryChannel.transactionCapacity=10
>>>
>>> cheers
>>> - Alex
>>>
>>>
>>> On Nov 29, 2012, at 1:03 PM, kashif khan <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am just struggling to learn the flume and doing some testing. I am
>>>> running two agents (agent, agent1). The agent used to upload the log
>>> data
>>>> into hdfs and agent1 used as logger. The configuration of two agents as:
>>>>
>>>> agent.sources = tail
>>>> agent.channels = memoryChannel
>>>> agent.sinks = hdfs-clusterSink
>>>>
>>>> agent.sources.tail.type = exec
>>>> agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
>>>> agent.sources.tail.channels = memoryChannel
>>>>
>>>> agent.sinks.hdfs-clusterSink.
>>>> channel = memoryChannel
>>>> agent.sinks.hdfs-clusterSink.type = hdfs
>>>> agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
>>>> hadoop1.example.com/user/root/Test/
>>>>
>>>>
>>>> agent.channels.memoryChannel.type = memory
>>>> agent.channels.memoryChannel.transactionCapacity=1000
>>>> agent.channels.memoryChannel.capacity = 100
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sources = source1
>>>> agent1.sinks = sink1
>>>> agent1.channels = channel1
>>>>
>>>> # Describe/configure source1
>>>> agent1.sources.source1.type = netcat
>>>> agent1.sources.source1.bind = localhost
>>>> agent1.sources.source1.port = 44444
>>>>
>>>> # Describe sink1
>>>> agent1.sinks.sink1.type = logger
>>>>
>>>> # Use a channel which buffers events in memory
>>>> agent1.channels.channel1.type = memory
>>>> agent1.channels.channel1.capacity = 1000
>>>> agent1.channels.channel1.transactionCapactiy = 100
>>>>
>>>> # Bind the source and sink to the channel
>>>> agent1.sources.source1.channels = channel1
>>>> agent1.sinks.sink1.channel = channel1
>>>>
>>>>
>>>> I dont know why it does not upload the log file into hdfs. where I am
>>> doing
>>>> mistake . If anyone who have solution please let me know.
>>>>
>>>>
>>>> The log file as:
>>>>
>>>>
>>>> 29 Nov 2012 11:49:13,046 INFO [main]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 1
>>>> 29 Nov 2012 11:49:13,050 INFO [main]
>>>> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting -
>>> agent
>>>> 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>>> -
>>>> Node manager starting
>>>> 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>>> lifecycle supervisor 10

Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Error in Upload the log file into hdfs

Brock Noland 2012-11-29, 15:54

Alex is right and our error message there needs much improvement. I have
created a JIRA here https://issues.apache.org/jira/browse/FLUME-1744

On Thu, Nov 29, 2012 at 9:39 AM, Alexander Alten-Lorenz <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> agent.channels.memoryChannel.transactionCapacity=1000
>
> Is wrong. You cant have equal or more transaction capacity in a channel as
> configured capacity. Use the opposite, when you want to use it.
> from our Guide:
>
> capacity NUM The max number of events stored in the
> channel
> transactionCapacity NUM The max number of events stored in the
> channel per transaction
>
> Try this:
>
> agent.channels.memoryChannel.capacity = 1000
> agent.channels.memoryChannel.transactionCapacity=10
>
> cheers
> - Alex
>
>
> On Nov 29, 2012, at 1:03 PM, kashif khan <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I am just struggling to learn the flume and doing some testing. I am
> > running two agents (agent, agent1). The agent used to upload the log data
> > into hdfs and agent1 used as logger. The configuration of two agents as:
> >
> > agent.sources = tail
> > agent.channels = memoryChannel
> > agent.sinks = hdfs-clusterSink
> >
> > agent.sources.tail.type = exec
> > agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
> > agent.sources.tail.channels = memoryChannel
> >
> > agent.sinks.hdfs-clusterSink.
> > channel = memoryChannel
> > agent.sinks.hdfs-clusterSink.type = hdfs
> > agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
> > hadoop1.example.com/user/root/Test/
> >
> >
> > agent.channels.memoryChannel.type = memory
> > agent.channels.memoryChannel.transactionCapacity=1000
> > agent.channels.memoryChannel.capacity = 100
> >
> >
> >
> >
> > agent1.sources = source1
> > agent1.sinks = sink1
> > agent1.channels = channel1
> >
> > # Describe/configure source1
> > agent1.sources.source1.type = netcat
> > agent1.sources.source1.bind = localhost
> > agent1.sources.source1.port = 44444
> >
> > # Describe sink1
> > agent1.sinks.sink1.type = logger
> >
> > # Use a channel which buffers events in memory
> > agent1.channels.channel1.type = memory
> > agent1.channels.channel1.capacity = 1000
> > agent1.channels.channel1.transactionCapactiy = 100
> >
> > # Bind the source and sink to the channel
> > agent1.sources.source1.channels = channel1
> > agent1.sinks.sink1.channel = channel1
> >
> >
> > I dont know why it does not upload the log file into hdfs. where I am
> doing
> > mistake . If anyone who have solution please let me know.
> >
> >
> > The log file as:
> >
> >
> > 29 Nov 2012 11:49:13,046 INFO [main]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 1
> > 29 Nov 2012 11:49:13,050 INFO [main]
> > (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
> > 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
> -
> > Node manager starting
> > 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
> > lifecycle supervisor 10
> > 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
> -
> > Configuration provider starting
> > 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: hdfs-clusterSink Agent: agent
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Re: Error in Upload the log file into hdfs

Alexander Alten-Lorenz 2012-11-29, 19:05

Brock, thats right. Will do over the weekend. We should add these into the Userguide, too.

Cheers,
Alex

On Nov 29, 2012, at 4:54 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Alex is right and our error message there needs much improvement. I have
> created a JIRA here https://issues.apache.org/jira/browse/FLUME-1744
>
> On Thu, Nov 29, 2012 at 9:39 AM, Alexander Alten-Lorenz <[EMAIL PROTECTED]
>> wrote:
>
>> Hi,
>>
>> agent.channels.memoryChannel.transactionCapacity=1000
>>
>> Is wrong. You cant have equal or more transaction capacity in a channel as
>> configured capacity. Use the opposite, when you want to use it.
>> from our Guide:
>>
>> capacity NUM The max number of events stored in the
>> channel
>> transactionCapacity NUM The max number of events stored in the
>> channel per transaction
>>
>> Try this:
>>
>> agent.channels.memoryChannel.capacity = 1000
>> agent.channels.memoryChannel.transactionCapacity=10
>>
>> cheers
>> - Alex
>>
>>
>> On Nov 29, 2012, at 1:03 PM, kashif khan <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> I am just struggling to learn the flume and doing some testing. I am
>>> running two agents (agent, agent1). The agent used to upload the log data
>>> into hdfs and agent1 used as logger. The configuration of two agents as:
>>>
>>> agent.sources = tail
>>> agent.channels = memoryChannel
>>> agent.sinks = hdfs-clusterSink
>>>
>>> agent.sources.tail.type = exec
>>> agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
>>> agent.sources.tail.channels = memoryChannel
>>>
>>> agent.sinks.hdfs-clusterSink.
>>> channel = memoryChannel
>>> agent.sinks.hdfs-clusterSink.type = hdfs
>>> agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
>>> hadoop1.example.com/user/root/Test/
>>>
>>>
>>> agent.channels.memoryChannel.type = memory
>>> agent.channels.memoryChannel.transactionCapacity=1000
>>> agent.channels.memoryChannel.capacity = 100
>>>
>>>
>>>
>>>
>>> agent1.sources = source1
>>> agent1.sinks = sink1
>>> agent1.channels = channel1
>>>
>>> # Describe/configure source1
>>> agent1.sources.source1.type = netcat
>>> agent1.sources.source1.bind = localhost
>>> agent1.sources.source1.port = 44444
>>>
>>> # Describe sink1
>>> agent1.sinks.sink1.type = logger
>>>
>>> # Use a channel which buffers events in memory
>>> agent1.channels.channel1.type = memory
>>> agent1.channels.channel1.capacity = 1000
>>> agent1.channels.channel1.transactionCapactiy = 100
>>>
>>> # Bind the source and sink to the channel
>>> agent1.sources.source1.channels = channel1
>>> agent1.sinks.sink1.channel = channel1
>>>
>>>
>>> I dont know why it does not upload the log file into hdfs. where I am
>> doing
>>> mistake . If anyone who have solution please let me know.
>>>
>>>
>>> The log file as:
>>>
>>>
>>> 29 Nov 2012 11:49:13,046 INFO [main]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>> lifecycle supervisor 1
>>> 29 Nov 2012 11:49:13,050 INFO [main]
>>> (org.apache.flume.node.FlumeNode.start:54) - Flume node starting - agent
>>> 29 Nov 2012 11:49:13,051 INFO [lifecycleSupervisor-1-0]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>> -
>>> Node manager starting
>>> 29 Nov 2012 11:49:13,053 INFO [lifecycleSupervisor-1-0]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.start:67) - Starting
>>> lifecycle supervisor 10
>>> 29 Nov 2012 11:49:13,052 INFO [lifecycleSupervisor-1-2]
>>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>> -
>>> Configuration provider starting
>>> 29 Nov 2012 11:49:13,054 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
>>> - Reloading configuration file:/etc/flume-ng/conf/flume.conf
>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]
>>>
>> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
>>> - Added sinks: hdfs-clusterSink Agent: agent
>>> 29 Nov 2012 11:49:13,057 INFO [conf-file-poller-0]

Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF