Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Error in Upload the log file into hdfs


Copy link to this message
-
Error in Upload the log file into hdfs
Hi,

I am just struggling to learn the flume and doing some testing. I am
running two agents (agent, agent1). The agent used to upload the log data
into hdfs and agent1 used as logger. The configuration of two agents as:

agent.sources = tail
agent.channels = memoryChannel
agent.sinks = hdfs-clusterSink

agent.sources.tail.type = exec
agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
agent.sources.tail.channels = memoryChannel

agent.sinks.hdfs-clusterSink.
channel = memoryChannel
agent.sinks.hdfs-clusterSink.type = hdfs
agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
hadoop1.example.com/user/root/Test/
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.transactionCapacity=1000
agent.channels.memoryChannel.capacity = 100
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1

# Describe/configure source1
agent1.sources.source1.type = netcat
agent1.sources.source1.bind = localhost
agent1.sources.source1.port = 44444

# Describe sink1
agent1.sinks.sink1.type = logger

# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapactiy = 100

# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
I dont know why it does not upload the log file into hdfs. where I am doing
mistake . If anyone who have solution please let me know.
The log file as:
29 Nov 2012 11:49:13,046 INFO  [main]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67)  - Starting
lifecycle supervisor 1
29 Nov 2012 11:49:13,050 INFO  [main]
(org.apache.flume.node.FlumeNode.start:54)  - Flume node starting - agent
29 Nov 2012 11:49:13,051 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)  -
Node manager starting
29 Nov 2012 11:49:13,053 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67)  - Starting
lifecycle supervisor 10
29 Nov 2012 11:49:13,052 INFO  [lifecycleSupervisor-1-2]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)  -
Configuration provider starting
29 Nov 2012 11:49:13,054 INFO  [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
- Reloading configuration file:/etc/flume-ng/conf/flume.conf
29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: hdfs-clusterSink Agent: agent
29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:sink1
29 Nov 2012 11:49:13,058 INFO  [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
- Added sinks: sink1 Agent: agent1
29 Nov 2012 11:49:13,058 INFO  [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
- Processing:hdfs-clusterSink
29 Nov 2012 11:49:13,087 INFO  [conf-file-poller-0]
(org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122)  -
Post-validation flume configuration contains configuration  for agents:
[agent, agent1]
29 Nov 2012 11:49:13,088 INFO  [conf-file-poller-0]
(org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
- Creating channels
29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
- Failed to load configuration data. Exception follows.
java.lang.IllegalStateException
    at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
    at
org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
    at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
    at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
    at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
    at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
    at
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
29 Nov 2012 11:49:22,157 INFO  [main]
(org.apache.flume.lifecycle.LifecycleSupervisor.start:67)  - Starting
lifecycle supervisor 1
29 Nov 2012 11:49:22,161 INFO  [m
+
Alexander Alten-Lorenz 2012-11-29, 15:39
+
Brock Noland 2012-11-29, 15:54
+
Alexander Alten-Lorenz 2012-11-29, 19:05
+
kashif khan 2012-11-29, 16:59
+
Brock Noland 2012-11-29, 17:03
+
Alexander Alten-Lorenz 2012-11-29, 19:14
+
kashif khan 2012-11-30, 10:47
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB