Jagadish Bihani
2012-08-08, 07:45
alo alt
2012-08-08, 07:49
Patrick Wendell
2012-08-08, 16:49
Hari Shreedharan
2012-08-08, 16:57
Jagadish Bihani
2012-08-10, 10:00
Jagadish Bihani
2012-08-11, 09:09
Patrick Wendell
2012-08-11, 20:49
Jagadish Bihani
2012-08-14, 13:11
ashutosh
2012-08-09, 08:48
Hari Shreedharan
2012-08-10, 17:07
-
flume-ng agent startup problemJagadish Bihani 2012-08-08, 07:45
Hi
I have downloaded the tarball of latest flume-ng 1.2.0.
I have JAVA_HOME properly set.
To begin with I have followed the instructions in "
https://cwiki.apache.org/FLUME/getting-started.html"
as it is. And even for that basic example:
My flume agent stucks printing the following output and then nothing
happens:
* bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
+ exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
'/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
-Djava.library.path= org.apache.flume.node.Application -f
conf/flume.conf -n agent1*
Please guide me what might be the cause.
Regards,
Jagadish
I have downloaded the tarball of latest flume-ng 1.2.0.
I have JAVA_HOME properly set.
To begin with I have followed the instructions in "
https://cwiki.apache.org/FLUME/getting-started.html"
as it is. And even for that basic example:
My flume agent stucks printing the following output and then nothing
happens:
* bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
+ exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
'/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
-Djava.library.path= org.apache.flume.node.Application -f
conf/flume.conf -n agent1*
Please guide me what might be the cause.
Regards,
Jagadish
+
Jagadish Bihani 2012-08-08, 07:45
-
Re: flume-ng agent startup problemalo alt 2012-08-08, 07:49
HI,
This page is outdated. Please follow:
http://flume.apache.org/FlumeUserGuide.html#setup
Since you haven't attached logs or error messages it's hard to say what
happen.
best
- Alex
Jagadish Bihani wrote:
>
> Hi
>
> I have downloaded the tarball of latest flume-ng 1.2.0.
> I have JAVA_HOME properly set.
>
> To begin with I have followed the instructions in "
> https://cwiki.apache.org/FLUME/getting-started.html"
> as it is. And even for that basic example:
>
> My flume agent stucks printing the following output and then nothing
> happens:
>
> *bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
>
> -Djava.library.path= org.apache.flume.node.Application -f
> conf/flume.conf -n agent1*
>
> Please guide me what might be the cause.
>
> Regards,
> Jagadish
> Jagadish Bihani <mailto:[EMAIL PROTECTED]>
> August 8, 2012 9:45 AM
> Hi
>
> I have downloaded the tarball of latest flume-ng 1.2.0.
> I have JAVA_HOME properly set.
>
> To begin with I have followed the instructions in "
> https://cwiki.apache.org/FLUME/getting-started.html"
> as it is. And even for that basic example:
>
> My flume agent stucks printing the following output and then nothing
> happens:
>
> * bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
> -Djava.library.path= org.apache.flume.node.Application -f
> conf/flume.conf -n agent1*
>
> Please guide me what might be the cause.
>
> Regards,
> Jagadish
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
This page is outdated. Please follow:
http://flume.apache.org/FlumeUserGuide.html#setup
Since you haven't attached logs or error messages it's hard to say what
happen.
best
- Alex
Jagadish Bihani wrote:
>
> Hi
>
> I have downloaded the tarball of latest flume-ng 1.2.0.
> I have JAVA_HOME properly set.
>
> To begin with I have followed the instructions in "
> https://cwiki.apache.org/FLUME/getting-started.html"
> as it is. And even for that basic example:
>
> My flume agent stucks printing the following output and then nothing
> happens:
>
> *bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
>
> -Djava.library.path= org.apache.flume.node.Application -f
> conf/flume.conf -n agent1*
>
> Please guide me what might be the cause.
>
> Regards,
> Jagadish
> Jagadish Bihani <mailto:[EMAIL PROTECTED]>
> August 8, 2012 9:45 AM
> Hi
>
> I have downloaded the tarball of latest flume-ng 1.2.0.
> I have JAVA_HOME properly set.
>
> To begin with I have followed the instructions in "
> https://cwiki.apache.org/FLUME/getting-started.html"
> as it is. And even for that basic example:
>
> My flume agent stucks printing the following output and then nothing
> happens:
>
> * bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
> -Djava.library.path= org.apache.flume.node.Application -f
> conf/flume.conf -n agent1*
>
> Please guide me what might be the cause.
>
> Regards,
> Jagadish
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
+
alo alt 2012-08-08, 07:49
-
Re: flume-ng agent startup problemPatrick Wendell 2012-08-08, 16:49
Jagadish,
If you use the new page and it still doesn't work, can you send me the
output of jstack against the flume pid? I've noticed some similar issues in
the past (silent startup failures) and I'm also trying to debug.
- Patrick
On Wed, Aug 8, 2012 at 12:49 AM, alo alt <[EMAIL PROTECTED]> wrote:
> HI,
>
> This page is outdated. Please follow:
> http://flume.apache.org/FlumeUserGuide.html#setup
>
> Since you haven't attached logs or error messages it's hard to say what
> happen.
>
> best
> - Alex
>
>
> Jagadish Bihani wrote:
>
>
> Hi
>
> I have downloaded the tarball of latest flume-ng 1.2.0.
> I have JAVA_HOME properly set.
>
> To begin with I have followed the instructions in "
> https://cwiki.apache.org/FLUME/getting-started.html"
> as it is. And even for that basic example:
>
> My flume agent stucks printing the following output and then nothing
> happens:
>
> *bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
>
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
> -Djava.library.path= org.apache.flume.node.Application -f
> conf/flume.conf -n agent1*
>
>
> Please guide me what might be the cause.
>
> Regards,
> Jagadish
>
>
>
> Jagadish Bihani <[EMAIL PROTECTED]>
> August 8, 2012 9:45 AM
> Hi
>
> I have downloaded the tarball of latest flume-ng 1.2.0.
> I have JAVA_HOME properly set.
>
> To begin with I have followed the instructions in "
> https://cwiki.apache.org/FLUME/getting-started.html"
> as it is. And even for that basic example:
>
> My flume agent stucks printing the following output and then nothing
> happens:
>
> * bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
> -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf
> -n agent1*
>
> Please guide me what might be the cause.
>
> Regards,
> Jagadish
>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
If you use the new page and it still doesn't work, can you send me the
output of jstack against the flume pid? I've noticed some similar issues in
the past (silent startup failures) and I'm also trying to debug.
- Patrick
On Wed, Aug 8, 2012 at 12:49 AM, alo alt <[EMAIL PROTECTED]> wrote:
> HI,
>
> This page is outdated. Please follow:
> http://flume.apache.org/FlumeUserGuide.html#setup
>
> Since you haven't attached logs or error messages it's hard to say what
> happen.
>
> best
> - Alex
>
>
> Jagadish Bihani wrote:
>
>
> Hi
>
> I have downloaded the tarball of latest flume-ng 1.2.0.
> I have JAVA_HOME properly set.
>
> To begin with I have followed the instructions in "
> https://cwiki.apache.org/FLUME/getting-started.html"
> as it is. And even for that basic example:
>
> My flume agent stucks printing the following output and then nothing
> happens:
>
> *bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
>
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
> -Djava.library.path= org.apache.flume.node.Application -f
> conf/flume.conf -n agent1*
>
>
> Please guide me what might be the cause.
>
> Regards,
> Jagadish
>
>
>
> Jagadish Bihani <[EMAIL PROTECTED]>
> August 8, 2012 9:45 AM
> Hi
>
> I have downloaded the tarball of latest flume-ng 1.2.0.
> I have JAVA_HOME properly set.
>
> To begin with I have followed the instructions in "
> https://cwiki.apache.org/FLUME/getting-started.html"
> as it is. And even for that basic example:
>
> My flume agent stucks printing the following output and then nothing
> happens:
>
> * bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
> -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf
> -n agent1*
>
> Please guide me what might be the cause.
>
> Regards,
> Jagadish
>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
+
Patrick Wendell 2012-08-08, 16:49
-
Re: flume-ng agent startup problemHari Shreedharan 2012-08-08, 16:57
Could you share your configuration with us? Your class path is quite bare, and remember that Hadoop does not ship with Flume. Can you take a look at the logs and see if you notice any errors?
Note that Flume "not printing" stuff is quite normal - flume is not exactly interactive. You should look for logs - it does not print stuff on the console. It will print to the console, only the exec command it uses to start up flume - which is what you are seeing. The logs should give you more details. Did you check the logs to make sure Flume is not working at all?
Hari
--
Hari Shreedharan
On Wednesday, August 8, 2012 at 9:49 AM, Patrick Wendell wrote:
> Jagadish,
>
> If you use the new page and it still doesn't work, can you send me the output of jstack against the flume pid? I've noticed some similar issues in the past (silent startup failures) and I'm also trying to debug.
>
> - Patrick
>
> On Wed, Aug 8, 2012 at 12:49 AM, alo alt <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > HI,
> >
> > This page is outdated. Please follow:
> > http://flume.apache.org/FlumeUserGuide.html#setup
> >
> > Since you haven't attached logs or error messages it's hard to say what happen.
> >
> > best
> > - Alex
> >
> >
> > Jagadish Bihani wrote:
> > >
> > > Hi
> > >
> > > I have downloaded the tarball of latest flume-ng 1.2.0.
> > > I have JAVA_HOME properly set.
> > >
> > > To begin with I have followed the instructions in "
> > > https://cwiki.apache.org/FLUME/getting-started.html"
> > > as it is. And even for that basic example:
> > >
> > > My flume agent stucks printing the following output and then nothing
> > > happens:
> > >
> > > *bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> > >
> > > + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> > > '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
> > > -Djava.library.path= org.apache.flume.node.Application -f
> > > conf/flume.conf -n agent1*
> > >
> > >
> > > Please guide me what might be the cause.
> > >
> > > Regards,
> > > Jagadish
> >
> > > Jagadish Bihani (mailto:[EMAIL PROTECTED])
> > > August 8, 2012 9:45 AM
> > >
> > >
> > > Hi
> > >
> > > I have downloaded the tarball of latest flume-ng 1.2.0.
> > > I have JAVA_HOME properly set.
> > >
> > > To begin with I have followed the instructions in " https://cwiki.apache.org/FLUME/getting-started.html"
> > > as it is. And even for that basic example:
> > >
> > > My flume agent stucks printing the following output and then nothing happens:
> > >
> > > bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> > > + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*' -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf -n agent1
> > >
> > > Please guide me what might be the cause.
> > >
> > > Regards,
> > > Jagadish
> >
> > --
> > Alexander Alten-Lorenz
> > http://mapredit.blogspot.com
> > German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Note that Flume "not printing" stuff is quite normal - flume is not exactly interactive. You should look for logs - it does not print stuff on the console. It will print to the console, only the exec command it uses to start up flume - which is what you are seeing. The logs should give you more details. Did you check the logs to make sure Flume is not working at all?
Hari
--
Hari Shreedharan
On Wednesday, August 8, 2012 at 9:49 AM, Patrick Wendell wrote:
> Jagadish,
>
> If you use the new page and it still doesn't work, can you send me the output of jstack against the flume pid? I've noticed some similar issues in the past (silent startup failures) and I'm also trying to debug.
>
> - Patrick
>
> On Wed, Aug 8, 2012 at 12:49 AM, alo alt <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > HI,
> >
> > This page is outdated. Please follow:
> > http://flume.apache.org/FlumeUserGuide.html#setup
> >
> > Since you haven't attached logs or error messages it's hard to say what happen.
> >
> > best
> > - Alex
> >
> >
> > Jagadish Bihani wrote:
> > >
> > > Hi
> > >
> > > I have downloaded the tarball of latest flume-ng 1.2.0.
> > > I have JAVA_HOME properly set.
> > >
> > > To begin with I have followed the instructions in "
> > > https://cwiki.apache.org/FLUME/getting-started.html"
> > > as it is. And even for that basic example:
> > >
> > > My flume agent stucks printing the following output and then nothing
> > > happens:
> > >
> > > *bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> > >
> > > + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp
> > > '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*'
> > > -Djava.library.path= org.apache.flume.node.Application -f
> > > conf/flume.conf -n agent1*
> > >
> > >
> > > Please guide me what might be the cause.
> > >
> > > Regards,
> > > Jagadish
> >
> > > Jagadish Bihani (mailto:[EMAIL PROTECTED])
> > > August 8, 2012 9:45 AM
> > >
> > >
> > > Hi
> > >
> > > I have downloaded the tarball of latest flume-ng 1.2.0.
> > > I have JAVA_HOME properly set.
> > >
> > > To begin with I have followed the instructions in " https://cwiki.apache.org/FLUME/getting-started.html"
> > > as it is. And even for that basic example:
> > >
> > > My flume agent stucks printing the following output and then nothing happens:
> > >
> > > bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1
> > > + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m -cp '/root/flume/apache-flume-1.2.0/conf:/root/flume/apache-flume-1.2.0/lib/*' -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf -n agent1
> > >
> > > Please guide me what might be the cause.
> > >
> > > Regards,
> > > Jagadish
> >
> > --
> > Alexander Alten-Lorenz
> > http://mapredit.blogspot.com
> > German Hadoop LinkedIn Group: http://goo.gl/N8pCF
+
Hari Shreedharan 2012-08-08, 16:57
-
Re: flume-ng agent startup problemJagadish Bihani 2012-08-10, 10:00
Hi
Thanks all for the inputs. After the initial problem I was able to start
flume except in one scenario in
which I use HDFS as sink.
I have a production machine with hadoop-0.20.1 installed. I have
installed latest flume 1.2.0.
It works fine for all the configurations (at least which I tried) except
when HDFS sink is used.
Test:
---------
I used both netcat listener as the source of the agent and HDFS is sink.
Then I start the agent using
the command *"bin/flume-ng agent -n agent1 -c conf -f
conf/flume_hdfs.conf --classpath
/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
-Dflume.root.logger=DEBUG,console"*
with DEBUG logging mode enabled. I don't get any error/exception. I use
*"/usr/sbin/lsof -i:<port_no>"* command to check whether the source
is actually bound to that port and it doesn't return any port. But when
I use *file sink instead of HDFS sink* and run lsof it correctly shows
me the port on which
it is listening.
Thus when HDFS sink is used even source part of agent doesn't work and
it doesn't give any exception. And nothing is written to
HDFS sink.
P.S. I have checked the user,permission details of HDFS. They are fine.
I have run flume on my other machines with different version of hadoop
(0.23 & 1.0). It has run HDFS sink properly there.
Does flume support hadoop-0.20.1 or there is something I am missing???
This is my Configuration:
-----------------------------------------
agent1.sources = sequencer
agent1.sinks =hdfsSink fileSink
agent1.sinks =fileSink
agent1.channels =memoryChannel fileChannel
agent1.sources.sequencer.type=seq
agent1.sinks.hdfsSink.type=hdfs
agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume
agent1.sinks.fileSink.type=file_roll
agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output
agent1.channels.memoryChannel.type=memory
agent1.channels.memoryChannel.capacity=10000
agent1.channels.memoryChannel.transactionCapacity=100
agent1.channels.fileChannel.type=file
agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir
agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir
agent1.sources.sequencer.channels = fileChannel
agent1.sinks.hdfsSink.channel = fileChannel
This is the log which I get:
----------------------------------------------------------
bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf --classpath
/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop
-0.20.1-core.jar -Dflume.root.logger=DEBUG,console
+ exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
-Dflume.root.logger=DEBUG,console -cp
'/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/li
b/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar'
-Djava.library.path= org.apache.flume.node.Application -n agent1 -f
conf/flume_hdfs.conf
2012-08-10 10:56:50,604 (main) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervisor 1
2012-08-10 10:56:50,607 (main) [INFO -
org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
starting - agent1
2012-08-10 10:56:50,611 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
Configuration provider starting
2012-08-10 10:56:50,611 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
Node ma
nager starting
2012-08-10 10:56:50,613 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervis
or 9
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]
Node manager started
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
] Configuration provider started
2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv
ider.java:188)] Checking file:conf/flume_hdfs.conf for changes
2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi
der.java:195)] Reloading configuration file:conf/flume_hdfs.conf
2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:fi
leSink
2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for fileSink: sink.directory
2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for hdfsSink: type
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]
Node m
anager started
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
] Configuration provider started
2012
Thanks all for the inputs. After the initial problem I was able to start
flume except in one scenario in
which I use HDFS as sink.
I have a production machine with hadoop-0.20.1 installed. I have
installed latest flume 1.2.0.
It works fine for all the configurations (at least which I tried) except
when HDFS sink is used.
Test:
---------
I used both netcat listener as the source of the agent and HDFS is sink.
Then I start the agent using
the command *"bin/flume-ng agent -n agent1 -c conf -f
conf/flume_hdfs.conf --classpath
/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
-Dflume.root.logger=DEBUG,console"*
with DEBUG logging mode enabled. I don't get any error/exception. I use
*"/usr/sbin/lsof -i:<port_no>"* command to check whether the source
is actually bound to that port and it doesn't return any port. But when
I use *file sink instead of HDFS sink* and run lsof it correctly shows
me the port on which
it is listening.
Thus when HDFS sink is used even source part of agent doesn't work and
it doesn't give any exception. And nothing is written to
HDFS sink.
P.S. I have checked the user,permission details of HDFS. They are fine.
I have run flume on my other machines with different version of hadoop
(0.23 & 1.0). It has run HDFS sink properly there.
Does flume support hadoop-0.20.1 or there is something I am missing???
This is my Configuration:
-----------------------------------------
agent1.sources = sequencer
agent1.sinks =hdfsSink fileSink
agent1.sinks =fileSink
agent1.channels =memoryChannel fileChannel
agent1.sources.sequencer.type=seq
agent1.sinks.hdfsSink.type=hdfs
agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume
agent1.sinks.fileSink.type=file_roll
agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output
agent1.channels.memoryChannel.type=memory
agent1.channels.memoryChannel.capacity=10000
agent1.channels.memoryChannel.transactionCapacity=100
agent1.channels.fileChannel.type=file
agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir
agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir
agent1.sources.sequencer.channels = fileChannel
agent1.sinks.hdfsSink.channel = fileChannel
This is the log which I get:
----------------------------------------------------------
bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf --classpath
/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop
-0.20.1-core.jar -Dflume.root.logger=DEBUG,console
+ exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
-Dflume.root.logger=DEBUG,console -cp
'/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/li
b/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar'
-Djava.library.path= org.apache.flume.node.Application -n agent1 -f
conf/flume_hdfs.conf
2012-08-10 10:56:50,604 (main) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervisor 1
2012-08-10 10:56:50,607 (main) [INFO -
org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
starting - agent1
2012-08-10 10:56:50,611 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
Configuration provider starting
2012-08-10 10:56:50,611 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
Node ma
nager starting
2012-08-10 10:56:50,613 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervis
or 9
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]
Node manager started
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
] Configuration provider started
2012-08-10 10:56:50,614 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProv
ider.java:188)] Checking file:conf/flume_hdfs.conf for changes
2012-08-10 10:56:50,614 (conf-file-poller-0) [INFO -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvi
der.java:195)] Reloading configuration file:conf/flume_hdfs.conf
2012-08-10 10:56:50,618 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:fi
leSink
2012-08-10 10:56:50,619 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for fileSink: sink.directory
2012-08-10 10:56:50,619 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)]
Created cont
ext for hdfsSink: type
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,620 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hd
fsSink
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-0) [DEBUG -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)]
Node m
anager started
2012-08-10 10:56:50,614 (lifecycleSupervisor-1-2) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)
] Configuration provider started
2012
+
Jagadish Bihani 2012-08-10, 10:00
-
Re: flume-ng agent startup problemJagadish Bihani 2012-08-11, 09:09
Hi
In my case flume is not transferring data to HDFS with my hadoop version
being 0.20.1 and it doesn't show any error even in DEBUG log mode.
It works fine for other sinks.
Is there any known compatibility problem with hadoop 0.20.1 ? OR
can there be a problem due to an particular hadoop version?
(I know its an old version but it is on production machine and cant upgrade
as of now...)
Details of configuration and log records are in the following mail
Thanks ,
Jagadish
On 08/10/2012 03:30 PM, Jagadish Bihani wrote:
> Hi
>
> Thanks all for the inputs. After the initial problem I was able to
> start flume except in one scenario in
> which I use HDFS as sink.
>
> I have a production machine with hadoop-0.20.1 installed. I have
> installed latest flume 1.2.0.
> It works fine for all the configurations (at least which I tried)
> except when HDFS sink is used.
>
> Test:
> ---------
> I used both netcat listener as the source of the agent and HDFS is
> sink. Then I start the agent using
> the command *"bin/flume-ng agent -n agent1 -c conf -f
> conf/flume_hdfs.conf --classpath
> /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
> -Dflume.root.logger=DEBUG,console"*
> with DEBUG logging mode enabled. I don't get any error/exception. I
> use *"/usr/sbin/lsof -i:<port_no>"* command to check whether the source
> is actually bound to that port and it doesn't return any port. But
> when I use *file sink instead of HDFS sink* and run lsof it correctly
> shows me the port on which
> it is listening.
> Thus when HDFS sink is used even source part of agent doesn't work and
> it doesn't give any exception. And nothing is written to
> HDFS sink.
>
> P.S. I have checked the user,permission details of HDFS. They are fine.
>
> I have run flume on my other machines with different version of hadoop
> (0.23 & 1.0). It has run HDFS sink properly there.
> Does flume support hadoop-0.20.1 or there is something I am missing???
>
> This is my Configuration:
> -----------------------------------------
> agent1.sources = sequencer
> agent1.sinks =hdfsSink fileSink
> agent1.sinks =fileSink
> agent1.channels =memoryChannel fileChannel
>
> agent1.sources.sequencer.type=seq
>
> agent1.sinks.hdfsSink.type=hdfs
> agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume
>
> agent1.sinks.fileSink.type=file_roll
> agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output
>
>
> agent1.channels.memoryChannel.type=memory
> agent1.channels.memoryChannel.capacity=10000
> agent1.channels.memoryChannel.transactionCapacity=100
>
>
> agent1.channels.fileChannel.type=file
> agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir
> agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir
>
> agent1.sources.sequencer.channels = fileChannel
> agent1.sinks.hdfsSink.channel = fileChannel
>
> This is the log which I get:
> ----------------------------------------------------------
>
> bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf
> --classpath /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop
> -0.20.1-core.jar -Dflume.root.logger=DEBUG,console
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
> -Dflume.root.logger=DEBUG,console -cp
> '/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/li
> b/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar'
> -Djava.library.path= org.apache.flume.node.Application -n agent1 -f
> conf/flume_hdfs.conf
> 2012-08-10 10:56:50,604 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1
> 2012-08-10 10:56:50,607 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent1
> 2012-08-10 10:56:50,611 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting
In my case flume is not transferring data to HDFS with my hadoop version
being 0.20.1 and it doesn't show any error even in DEBUG log mode.
It works fine for other sinks.
Is there any known compatibility problem with hadoop 0.20.1 ? OR
can there be a problem due to an particular hadoop version?
(I know its an old version but it is on production machine and cant upgrade
as of now...)
Details of configuration and log records are in the following mail
Thanks ,
Jagadish
On 08/10/2012 03:30 PM, Jagadish Bihani wrote:
> Hi
>
> Thanks all for the inputs. After the initial problem I was able to
> start flume except in one scenario in
> which I use HDFS as sink.
>
> I have a production machine with hadoop-0.20.1 installed. I have
> installed latest flume 1.2.0.
> It works fine for all the configurations (at least which I tried)
> except when HDFS sink is used.
>
> Test:
> ---------
> I used both netcat listener as the source of the agent and HDFS is
> sink. Then I start the agent using
> the command *"bin/flume-ng agent -n agent1 -c conf -f
> conf/flume_hdfs.conf --classpath
> /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
> -Dflume.root.logger=DEBUG,console"*
> with DEBUG logging mode enabled. I don't get any error/exception. I
> use *"/usr/sbin/lsof -i:<port_no>"* command to check whether the source
> is actually bound to that port and it doesn't return any port. But
> when I use *file sink instead of HDFS sink* and run lsof it correctly
> shows me the port on which
> it is listening.
> Thus when HDFS sink is used even source part of agent doesn't work and
> it doesn't give any exception. And nothing is written to
> HDFS sink.
>
> P.S. I have checked the user,permission details of HDFS. They are fine.
>
> I have run flume on my other machines with different version of hadoop
> (0.23 & 1.0). It has run HDFS sink properly there.
> Does flume support hadoop-0.20.1 or there is something I am missing???
>
> This is my Configuration:
> -----------------------------------------
> agent1.sources = sequencer
> agent1.sinks =hdfsSink fileSink
> agent1.sinks =fileSink
> agent1.channels =memoryChannel fileChannel
>
> agent1.sources.sequencer.type=seq
>
> agent1.sinks.hdfsSink.type=hdfs
> agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume
>
> agent1.sinks.fileSink.type=file_roll
> agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output
>
>
> agent1.channels.memoryChannel.type=memory
> agent1.channels.memoryChannel.capacity=10000
> agent1.channels.memoryChannel.transactionCapacity=100
>
>
> agent1.channels.fileChannel.type=file
> agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir
> agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir
>
> agent1.sources.sequencer.channels = fileChannel
> agent1.sinks.hdfsSink.channel = fileChannel
>
> This is the log which I get:
> ----------------------------------------------------------
>
> bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf
> --classpath /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop
> -0.20.1-core.jar -Dflume.root.logger=DEBUG,console
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
> -Dflume.root.logger=DEBUG,console -cp
> '/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/li
> b/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar'
> -Djava.library.path= org.apache.flume.node.Application -n agent1 -f
> conf/flume_hdfs.conf
> 2012-08-10 10:56:50,604 (main) [INFO -
> org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
> Starting lifecycle supervisor 1
> 2012-08-10 10:56:50,607 (main) [INFO -
> org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node
> starting - agent1
> 2012-08-10 10:56:50,611 (lifecycleSupervisor-1-2) [INFO -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
> Configuration provider starting
+
Jagadish Bihani 2012-08-11, 09:09
-
Re: flume-ng agent startup problemPatrick Wendell 2012-08-11, 20:49
Jagadish,
One possibility is that flume is not finding the Hadoop classpath correctly
and silently failing when trying to create the HDFS sink. I've run into
something like this before and thought we had fixed it.
Do you have HADOOP_HOME set in your environment? If you run "$> hadoop
classpath" on the command line does it correctly print out the hadoop
classpath? Flume uses these to try and find out the correct hadoop
directories to include in the classpath.
Also, can you run ./flume-ng with the -d option to print out the classpath
that is being used to launch flume? You want to verify that your hadoop
directory is in there.
- Patrick
On Sat, Aug 11, 2012 at 2:09 AM, Jagadish Bihani <
[EMAIL PROTECTED]> wrote:
> Hi
>
> In my case flume is not transferring data to HDFS with my hadoop version
> being 0.20.1 and it doesn't show any error even in DEBUG log mode.
> It works fine for other sinks.
>
> Is there any known compatibility problem with hadoop 0.20.1 ? OR
> can there be a problem due to an particular hadoop version?
> (I know its an old version but it is on production machine and cant upgrade
> as of now...)
>
> Details of configuration and log records are in the following mail
>
> Thanks ,
> Jagadish
>
>
> On 08/10/2012 03:30 PM, Jagadish Bihani wrote:
>
> Hi
>
> Thanks all for the inputs. After the initial problem I was able to start
> flume except in one scenario in
> which I use HDFS as sink.
>
> I have a production machine with hadoop-0.20.1 installed. I have installed
> latest flume 1.2.0.
> It works fine for all the configurations (at least which I tried) except
> when HDFS sink is used.
>
> Test:
> ---------
> I used both netcat listener as the source of the agent and HDFS is sink.
> Then I start the agent using
> the command *"bin/flume-ng agent -n agent1 -c conf -f
> conf/flume_hdfs.conf --classpath
> /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
> -Dflume.root.logger=DEBUG,console"*
> with DEBUG logging mode enabled. I don't get any error/exception. I use *"/usr/sbin/lsof
> -i:<port_no>"* command to check whether the source
> is actually bound to that port and it doesn't return any port. But when I
> use *file sink instead of HDFS sink* and run lsof it correctly shows me
> the port on which
> it is listening.
> Thus when HDFS sink is used even source part of agent doesn't work and it
> doesn't give any exception. And nothing is written to
> HDFS sink.
>
> P.S. I have checked the user,permission details of HDFS. They are fine.
>
> I have run flume on my other machines with different version of hadoop
> (0.23 & 1.0). It has run HDFS sink properly there.
> Does flume support hadoop-0.20.1 or there is something I am missing???
>
> This is my Configuration:
> -----------------------------------------
> agent1.sources = sequencer
> agent1.sinks =hdfsSink fileSink
> agent1.sinks =fileSink
> agent1.channels =memoryChannel fileChannel
>
> agent1.sources.sequencer.type=seq
>
> agent1.sinks.hdfsSink.type=hdfs
> agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume
>
> agent1.sinks.fileSink.type=file_roll
> agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output
>
>
> agent1.channels.memoryChannel.type=memory
> agent1.channels.memoryChannel.capacity=10000
> agent1.channels.memoryChannel.transactionCapacity=100
>
>
> agent1.channels.fileChannel.type=file
>
> agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir
> agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir
>
> agent1.sources.sequencer.channels = fileChannel
> agent1.sinks.hdfsSink.channel = fileChannel
>
> This is the log which I get:
> ----------------------------------------------------------
>
> bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf --classpath
> /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop
> -0.20.1-core.jar -Dflume.root.logger=DEBUG,console
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
> -Dflume.root.logger=DEBUG,console -cp
One possibility is that flume is not finding the Hadoop classpath correctly
and silently failing when trying to create the HDFS sink. I've run into
something like this before and thought we had fixed it.
Do you have HADOOP_HOME set in your environment? If you run "$> hadoop
classpath" on the command line does it correctly print out the hadoop
classpath? Flume uses these to try and find out the correct hadoop
directories to include in the classpath.
Also, can you run ./flume-ng with the -d option to print out the classpath
that is being used to launch flume? You want to verify that your hadoop
directory is in there.
- Patrick
On Sat, Aug 11, 2012 at 2:09 AM, Jagadish Bihani <
[EMAIL PROTECTED]> wrote:
> Hi
>
> In my case flume is not transferring data to HDFS with my hadoop version
> being 0.20.1 and it doesn't show any error even in DEBUG log mode.
> It works fine for other sinks.
>
> Is there any known compatibility problem with hadoop 0.20.1 ? OR
> can there be a problem due to an particular hadoop version?
> (I know its an old version but it is on production machine and cant upgrade
> as of now...)
>
> Details of configuration and log records are in the following mail
>
> Thanks ,
> Jagadish
>
>
> On 08/10/2012 03:30 PM, Jagadish Bihani wrote:
>
> Hi
>
> Thanks all for the inputs. After the initial problem I was able to start
> flume except in one scenario in
> which I use HDFS as sink.
>
> I have a production machine with hadoop-0.20.1 installed. I have installed
> latest flume 1.2.0.
> It works fine for all the configurations (at least which I tried) except
> when HDFS sink is used.
>
> Test:
> ---------
> I used both netcat listener as the source of the agent and HDFS is sink.
> Then I start the agent using
> the command *"bin/flume-ng agent -n agent1 -c conf -f
> conf/flume_hdfs.conf --classpath
> /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
> -Dflume.root.logger=DEBUG,console"*
> with DEBUG logging mode enabled. I don't get any error/exception. I use *"/usr/sbin/lsof
> -i:<port_no>"* command to check whether the source
> is actually bound to that port and it doesn't return any port. But when I
> use *file sink instead of HDFS sink* and run lsof it correctly shows me
> the port on which
> it is listening.
> Thus when HDFS sink is used even source part of agent doesn't work and it
> doesn't give any exception. And nothing is written to
> HDFS sink.
>
> P.S. I have checked the user,permission details of HDFS. They are fine.
>
> I have run flume on my other machines with different version of hadoop
> (0.23 & 1.0). It has run HDFS sink properly there.
> Does flume support hadoop-0.20.1 or there is something I am missing???
>
> This is my Configuration:
> -----------------------------------------
> agent1.sources = sequencer
> agent1.sinks =hdfsSink fileSink
> agent1.sinks =fileSink
> agent1.channels =memoryChannel fileChannel
>
> agent1.sources.sequencer.type=seq
>
> agent1.sinks.hdfsSink.type=hdfs
> agent1.sinks.hdfsSink.hdfs.path=hdfs://MLNameNode2001:54310/flume
>
> agent1.sinks.fileSink.type=file_roll
> agent1.sinks.fileSink.sink.directory=/home/hadoop/flume/output
>
>
> agent1.channels.memoryChannel.type=memory
> agent1.channels.memoryChannel.capacity=10000
> agent1.channels.memoryChannel.transactionCapacity=100
>
>
> agent1.channels.fileChannel.type=file
>
> agent1.channels.fileChannel.checkpointDir=/home/hadoop/flume/channel/checkpointDir
> agent1.channels.fileChannel.dataDirs=/home/hadoop/flume/channel/dataDir
>
> agent1.sources.sequencer.channels = fileChannel
> agent1.sinks.hdfsSink.channel = fileChannel
>
> This is the log which I get:
> ----------------------------------------------------------
>
> bin/flume-ng agent -n agent1 -c conf -f conf/flume_hdfs.conf --classpath
> /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop
> -0.20.1-core.jar -Dflume.root.logger=DEBUG,console
> + exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
> -Dflume.root.logger=DEBUG,console -cp
+
Patrick Wendell 2012-08-11, 20:49
-
Re: flume-ng agent startup problemJagadish Bihani 2012-08-14, 13:11
Hi Patrick
I verified that flume is finding hadoop directory in the classpath.
And HADOOP_HOME is also set.
When I run the flume command it prints:
apache-flume-1.2.0]$ bin/flume-ng agent -n agent -c conf -f
conf/flume_socket.conf --classpath
/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
-Dflume.root.logger=DEBUG,console
*Info: Including Hadoop libraries found via
(/MachineLearning/OTFA/hadoop-0.20.1-cluster1/bin/hadoop) for HDFS access*
+ exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
-Dflume.root.logger=DEBUG,console -cp
'/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/lib/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar'
-Djava.library.path=:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/bin/../lib/native/Linux-i386-32
org.apache.flume.node.Application -n agent -f conf/flume_socket.conf
So it finds Hadoop libraries. I also tried adding other jars like
commons-cli & commons-codec in the classpath.
But still HDFS sink is not working.
Regards,
Jagadish
On 08/12/2012 02:19 AM, Patrick Wendell wrote:
> Jagadish,
>
> One possibility is that flume is not finding the Hadoop classpath
> correctly and silently failing when trying to create the HDFS sink.
> I've run into something like this before and thought we had fixed it.
>
> Do you have HADOOP_HOME set in your environment? If you run "$> hadoop
> classpath" on the command line does it correctly print out the hadoop
> classpath? Flume uses these to try and find out the correct hadoop
> directories to include in the classpath.
>
> Also, can you run ./flume-ng with the -d option to print out the
> classpath that is being used to launch flume? You want to verify that
> your hadoop directory is in there.
>
> - Patrick
>
> On Sat, Aug 11, 2012 at 2:09 AM, Jagadish Bihani
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> wrote:
>
> Hi
>
> In my case flume is not transferring data to HDFS with my hadoop
> version
> being 0.20.1 and it doesn't show any error even in DEBUG log mode.
> It works fine for other sinks.
>
> Is there any known compatibility problem with hadoop 0.20.1 ? OR
> can there be a problem due to an particular hadoop version?
> (I know its an old version but it is on production machine and
> cant upgrade
> as of now...)
>
> Details of configuration and log records are in the following mail
>
> Thanks ,
> Jagadish
>
>
> On 08/10/2012 03:30 PM, Jagadish Bihani wrote:
>> Hi
>>
>> Thanks all for the inputs. After the initial problem I was able
>> to start flume except in one scenario in
>> which I use HDFS as sink.
>>
>> I have a production machine with hadoop-0.20.1 installed. I have
>> installed latest flume 1.2.0.
>> It works fine for all the configurations (at least which I tried)
>> except when HDFS sink is used.
>>
>> Test:
>> ---------
>> I used both netcat listener as the source of the agent and HDFS
>> is sink. Then I start the agent using
>> the command *"bin/flume-ng agent -n agent1 -c conf -f
>> conf/flume_hdfs.conf --classpath
>> /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
>> -Dflume.root.logger=DEBUG,console"*
>> with DEBUG logging mode enabled. I don't get any
>> error/exception. I use *"/usr/sbin/lsof -i:<port_no>"* command to
>> check whether the source
>> is actually bound to that port and it doesn't return any port.
>> But when I use *file sink instead of HDFS sink* and run lsof it
>> correctly shows me the port on which
>> it is listening.
>> Thus when HDFS sink is used even source part of agent doesn't
>> work and it doesn't give any exception. And nothing is written to
>> HDFS sink.
>>
>> P.S. I have checked the user,permission details of HDFS. They are
>> fine.
>>
>> I have run flume on my other machines with different version of
>> hadoop (0.23 & 1.0). It has run HDFS sink properly there.
I verified that flume is finding hadoop directory in the classpath.
And HADOOP_HOME is also set.
When I run the flume command it prints:
apache-flume-1.2.0]$ bin/flume-ng agent -n agent -c conf -f
conf/flume_socket.conf --classpath
/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
-Dflume.root.logger=DEBUG,console
*Info: Including Hadoop libraries found via
(/MachineLearning/OTFA/hadoop-0.20.1-cluster1/bin/hadoop) for HDFS access*
+ exec /usr/java/jdk1.6.0_12/bin/java -Xmx20m
-Dflume.root.logger=DEBUG,console -cp
'/home/hadoop/flume/apache-flume-1.2.0/conf:/home/hadoop/flume/apache-flume-1.2.0/lib/*:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar'
-Djava.library.path=:/MachineLearning/OTFA/hadoop-0.20.1-cluster1/bin/../lib/native/Linux-i386-32
org.apache.flume.node.Application -n agent -f conf/flume_socket.conf
So it finds Hadoop libraries. I also tried adding other jars like
commons-cli & commons-codec in the classpath.
But still HDFS sink is not working.
Regards,
Jagadish
On 08/12/2012 02:19 AM, Patrick Wendell wrote:
> Jagadish,
>
> One possibility is that flume is not finding the Hadoop classpath
> correctly and silently failing when trying to create the HDFS sink.
> I've run into something like this before and thought we had fixed it.
>
> Do you have HADOOP_HOME set in your environment? If you run "$> hadoop
> classpath" on the command line does it correctly print out the hadoop
> classpath? Flume uses these to try and find out the correct hadoop
> directories to include in the classpath.
>
> Also, can you run ./flume-ng with the -d option to print out the
> classpath that is being used to launch flume? You want to verify that
> your hadoop directory is in there.
>
> - Patrick
>
> On Sat, Aug 11, 2012 at 2:09 AM, Jagadish Bihani
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> wrote:
>
> Hi
>
> In my case flume is not transferring data to HDFS with my hadoop
> version
> being 0.20.1 and it doesn't show any error even in DEBUG log mode.
> It works fine for other sinks.
>
> Is there any known compatibility problem with hadoop 0.20.1 ? OR
> can there be a problem due to an particular hadoop version?
> (I know its an old version but it is on production machine and
> cant upgrade
> as of now...)
>
> Details of configuration and log records are in the following mail
>
> Thanks ,
> Jagadish
>
>
> On 08/10/2012 03:30 PM, Jagadish Bihani wrote:
>> Hi
>>
>> Thanks all for the inputs. After the initial problem I was able
>> to start flume except in one scenario in
>> which I use HDFS as sink.
>>
>> I have a production machine with hadoop-0.20.1 installed. I have
>> installed latest flume 1.2.0.
>> It works fine for all the configurations (at least which I tried)
>> except when HDFS sink is used.
>>
>> Test:
>> ---------
>> I used both netcat listener as the source of the agent and HDFS
>> is sink. Then I start the agent using
>> the command *"bin/flume-ng agent -n agent1 -c conf -f
>> conf/flume_hdfs.conf --classpath
>> /MachineLearning/OTFA/hadoop-0.20.1-cluster1/hadoop-0.20.1-core.jar
>> -Dflume.root.logger=DEBUG,console"*
>> with DEBUG logging mode enabled. I don't get any
>> error/exception. I use *"/usr/sbin/lsof -i:<port_no>"* command to
>> check whether the source
>> is actually bound to that port and it doesn't return any port.
>> But when I use *file sink instead of HDFS sink* and run lsof it
>> correctly shows me the port on which
>> it is listening.
>> Thus when HDFS sink is used even source part of agent doesn't
>> work and it doesn't give any exception. And nothing is written to
>> HDFS sink.
>>
>> P.S. I have checked the user,permission details of HDFS. They are
>> fine.
>>
>> I have run flume on my other machines with different version of
>> hadoop (0.23 & 1.0). It has run HDFS sink properly there.
+
Jagadish Bihani 2012-08-14, 13:11
-
RE: flume-ng agent startup problemashutosh 2012-08-09, 08:48
Hi ,
I am also facing the same issue, agent stucks. But, the flume events are transferring to the downstream agent and there is no warning and error printed in flume.log.
I don’t know whether it’s a expected behavior of the flume agent. Please let me know if you have any idea.
Patrick: Please see the below output of the jstack -l <pid> command as you suggested:
2012-08-09 17:32:50
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):
"Attach Listener" daemon prio=10 tid=0x000000004fb5e800 nid=0x77a5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00002aaab4255800 nid=0x7765 waiting on condition [0x0000000042f2f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fec04490> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"SinkRunner-PollingRunner-DefaultSinkProcessor" prio=10 tid=0x000000004fcb6800 nid=0x7764 waiting on condition [0x00000000414ba000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fff853c0> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:72)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
at org.apache.flume.sink.AvroSink.process(AvroSink.java:287)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"New I/O client worker #1-1" daemon prio=10 tid=0x00002aaab423b000 nid=0x7763 runnable [0x000000004104a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000fff9d540> (a sun.nio.ch.Util$2)
- locked <0x00000000fff9d530> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000fff9aae0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:165)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x00000000fff9a970> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00002aaab4257800 nid=0x7762 waiting on condition [0x00000000408be000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fff9a380> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"StderrReader-[tail -F /logs01/tx.log]" daemon prio=10 tid=0x000000004fb52000 nid=0x7761 runnable [0x0000000042e2e000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
- locked <0x00000000fff9eac0> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamR
I am also facing the same issue, agent stucks. But, the flume events are transferring to the downstream agent and there is no warning and error printed in flume.log.
I don’t know whether it’s a expected behavior of the flume agent. Please let me know if you have any idea.
Patrick: Please see the below output of the jstack -l <pid> command as you suggested:
2012-08-09 17:32:50
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):
"Attach Listener" daemon prio=10 tid=0x000000004fb5e800 nid=0x77a5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00002aaab4255800 nid=0x7765 waiting on condition [0x0000000042f2f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fec04490> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"SinkRunner-PollingRunner-DefaultSinkProcessor" prio=10 tid=0x000000004fcb6800 nid=0x7764 waiting on condition [0x00000000414ba000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fff853c0> (a java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:72)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
at org.apache.flume.sink.AvroSink.process(AvroSink.java:287)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"New I/O client worker #1-1" daemon prio=10 tid=0x00002aaab423b000 nid=0x7763 runnable [0x000000004104a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000fff9d540> (a sun.nio.ch.Util$2)
- locked <0x00000000fff9d530> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000fff9aae0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:165)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x00000000fff9a970> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00002aaab4257800 nid=0x7762 waiting on condition [0x00000000408be000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fff9a380> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"StderrReader-[tail -F /logs01/tx.log]" daemon prio=10 tid=0x000000004fb52000 nid=0x7761 runnable [0x0000000042e2e000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
- locked <0x00000000fff9eac0> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamR
+
ashutosh 2012-08-09, 08:48
-
Re: flume-ng agent startup problemHari Shreedharan 2012-08-10, 17:07
If flume is transferring data it is doing its job. It is not going to print stuff to the console all the time. Everything just gets written to the log file. I don't know why you would expect Flume to write stuff out to console continuously. Flume starts up and prints to the console the command it is executing - after which everything is simply logged to the file. I don't see anything wrong with that. If nothing is printed to console does not mean Flume is not doing its job. Flume's job is to transfer data - if it is being transferred, it is doing its job :-)
Hari
--
Hari Shreedharan
On Thursday, August 9, 2012 at 1:48 AM, ashutosh(오픈��랫폼개발팀) wrote:
> Hi ,
>
> I am also facing the same issue, agent stucks. But, the flume events are transferring to the downstream agent and there is no warning and error printed in flume.log.
>
> I don’t know whether it’s a expected behavior of the flume agent. Please let me know if you have any idea.
>
> Patrick: Please see the below output of the jstack –l <pid> command as you suggested:
> 2012-08-09 17:32:50
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x000000004fb5e800 nid=0x77a5 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> Locked ownable synchronizers:
> - None
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00002aaab4255800 nid=0x7765 waiting on condition [0x0000000042f2f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fec04490> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "SinkRunner-PollingRunner-DefaultSinkProcessor" prio=10 tid=0x000000004fcb6800 nid=0x7764 waiting on condition [0x00000000414ba000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fff853c0> (a java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)
> at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:72)
> at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:287)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "New I/O client worker #1-1" daemon prio=10 tid=0x00002aaab423b000 nid=0x7763 runnable [0x000000004104a000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
Hari
--
Hari Shreedharan
On Thursday, August 9, 2012 at 1:48 AM, ashutosh(오픈��랫폼개발팀) wrote:
> Hi ,
>
> I am also facing the same issue, agent stucks. But, the flume events are transferring to the downstream agent and there is no warning and error printed in flume.log.
>
> I don’t know whether it’s a expected behavior of the flume agent. Please let me know if you have any idea.
>
> Patrick: Please see the below output of the jstack –l <pid> command as you suggested:
> 2012-08-09 17:32:50
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x000000004fb5e800 nid=0x77a5 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> Locked ownable synchronizers:
> - None
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00002aaab4255800 nid=0x7765 waiting on condition [0x0000000042f2f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fec04490> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "SinkRunner-PollingRunner-DefaultSinkProcessor" prio=10 tid=0x000000004fcb6800 nid=0x7764 waiting on condition [0x00000000414ba000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fff853c0> (a java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)
> at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:72)
> at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:287)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "New I/O client worker #1-1" daemon prio=10 tid=0x00002aaab423b000 nid=0x7763 runnable [0x000000004104a000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
+
Hari Shreedharan 2012-08-10, 17:07
