Parag Hukeri
2012-11-02, 18:15
Josh West
2012-11-02, 18:48
Parag Hukeri
2012-11-03, 00:21
Alexander Lorenz
2012-11-03, 06:55
Parag Hukeri
2012-11-05, 13:39
Brock Noland
2012-11-05, 13:51
Alexander Lorenz
2012-11-06, 07:01
Parag Hukeri
2012-11-06, 18:38
-
ExecSource tail -F stops after a whileParag Hukeri 2012-11-02, 18:15
Hi
I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
-
Re: ExecSource tail -F stops after a whileJosh West 2012-11-02, 18:48
This happened to me consistently when buffering tail -F contents into
the Memory Channel. Try switching to the File Channel. This solved the
issue for me.
On 11/02/2012 07:15 PM, Parag Hukeri wrote:
> Hi
> I am using apache-flume-1.2.0. I have set up an ExecSource that tails
> a log file & gets the entries from this log file. The log file size is
> approx 4 GB (approx 12 million lines written over a period of 12 hrs).
> As the process starts everything seems to work fine for approx 9.5
> hrs. (approx 9.5 million lines of log data is successfully retrieved
> by ExecSource). It then stops reading the output from tail. I don't
> see any exceptions being thrown. I don't think the problem is with the
> actual tail command. I was able to successfully manually tail the log
> file & all 12 million records were retrieved by tail. Running on a
> Linux system. Not sure what the issue might be with ExecSource? Any
> help would be appreciated. Thanks.
>
--
Josh West
Lead Systems Administrator
One.com, [EMAIL PROTECTED]
the Memory Channel. Try switching to the File Channel. This solved the
issue for me.
On 11/02/2012 07:15 PM, Parag Hukeri wrote:
> Hi
> I am using apache-flume-1.2.0. I have set up an ExecSource that tails
> a log file & gets the entries from this log file. The log file size is
> approx 4 GB (approx 12 million lines written over a period of 12 hrs).
> As the process starts everything seems to work fine for approx 9.5
> hrs. (approx 9.5 million lines of log data is successfully retrieved
> by ExecSource). It then stops reading the output from tail. I don't
> see any exceptions being thrown. I don't think the problem is with the
> actual tail command. I was able to successfully manually tail the log
> file & all 12 million records were retrieved by tail. Running on a
> Linux system. Not sure what the issue might be with ExecSource? Any
> help would be appreciated. Thanks.
>
--
Josh West
Lead Systems Administrator
One.com, [EMAIL PROTECTED]
-
Re: ExecSource tail -F stops after a whileParag Hukeri 2012-11-03, 00:21
I tried to use the FileChannel; but am getting an exception -
org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
Thanks
--- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED]> wrote:
From: Josh West <[EMAIL PROTECTED]>
Subject: Re: ExecSource tail -F stops after a while
To: [EMAIL PROTECTED]
Date: Friday, November 2, 2012, 1:48 PM
This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
On 11/02/2012 07:15 PM, Parag Hukeri wrote:
Hi
I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
--
Josh West
Lead Systems Administrator
One.com, [EMAIL PROTECTED]
org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
Thanks
--- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED]> wrote:
From: Josh West <[EMAIL PROTECTED]>
Subject: Re: ExecSource tail -F stops after a while
To: [EMAIL PROTECTED]
Date: Friday, November 2, 2012, 1:48 PM
This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
On 11/02/2012 07:15 PM, Parag Hukeri wrote:
Hi
I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
--
Josh West
Lead Systems Administrator
One.com, [EMAIL PROTECTED]
-
Re: ExecSource tail -F stops after a whileAlexander Lorenz 2012-11-03, 06:55
Hi,
http://flume.apache.org/FlumeUserGuide.html#exec-source
Please see the WARNING ;)
Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
FileChannel is available in from up to 1.3
http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
cheers
- Alex
On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED]> wrote:
> I tried to use the FileChannel; but am getting an exception -
>
> org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
> Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
>
> I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
>
> Thanks
>
> --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED]> wrote:
>
>
> From: Josh West <[EMAIL PROTECTED]>
> Subject: Re: ExecSource tail -F stops after a while
> To: [EMAIL PROTECTED]
> Date: Friday, November 2, 2012, 1:48 PM
>
>
>
> This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
>
>
> On 11/02/2012 07:15 PM, Parag Hukeri wrote:
>
>
>
>
>
>
> Hi
>
> I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
> --
> Josh West
> Lead Systems Administrator
> One.com, [EMAIL PROTECTED]
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
http://flume.apache.org/FlumeUserGuide.html#exec-source
Please see the WARNING ;)
Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
FileChannel is available in from up to 1.3
http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
cheers
- Alex
On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED]> wrote:
> I tried to use the FileChannel; but am getting an exception -
>
> org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
> Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
>
> I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
>
> Thanks
>
> --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED]> wrote:
>
>
> From: Josh West <[EMAIL PROTECTED]>
> Subject: Re: ExecSource tail -F stops after a while
> To: [EMAIL PROTECTED]
> Date: Friday, November 2, 2012, 1:48 PM
>
>
>
> This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
>
>
> On 11/02/2012 07:15 PM, Parag Hukeri wrote:
>
>
>
>
>
>
> Hi
>
> I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
> --
> Josh West
> Lead Systems Administrator
> One.com, [EMAIL PROTECTED]
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
-
Re: ExecSource tail -F stops after a whileParag Hukeri 2012-11-05, 13:39
While starting my flume-agent I did use these options:- java -Xms512m -Xmx2048m. But that did not help. What should I tweak in flume-env.sh? Not sure?
I noticed that the source for fileChannel was included in flume 1.2. So I tried compiling that & using that jar; but got the exact same results with fileChannel - the process stopped after approx 9.5 hrs (after having written approx 9.5 million records).
Any help would be appreciated. Thanks.
--- On Sat, 11/3/12, Alexander Lorenz <[EMAIL PROTECTED]> wrote:
From: Alexander Lorenz <[EMAIL PROTECTED]>
Subject: Re: ExecSource tail -F stops after a while
To: [EMAIL PROTECTED]
Date: Saturday, November 3, 2012, 1:55 AM
Hi,
http://flume.apache.org/FlumeUserGuide.html#exec-source
Please see the WARNING ;)
Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
FileChannel is available in from up to 1.3
http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
cheers
- Alex
On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED]> wrote:
> I tried to use the FileChannel; but am getting an exception -
>
> org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
> Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
>
> I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
>
> Thanks
>
> --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED]> wrote:
>
>
> From: Josh West <[EMAIL PROTECTED]>
> Subject: Re: ExecSource tail -F stops after a while
> To: [EMAIL PROTECTED]
> Date: Friday, November 2, 2012, 1:48 PM
>
>
>
> This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
>
>
> On 11/02/2012 07:15 PM, Parag Hukeri wrote:
>
>
>
>
>
>
> Hi
>
> I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
> --
> Josh West
> Lead Systems Administrator
> One.com, [EMAIL PROTECTED]
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
I noticed that the source for fileChannel was included in flume 1.2. So I tried compiling that & using that jar; but got the exact same results with fileChannel - the process stopped after approx 9.5 hrs (after having written approx 9.5 million records).
Any help would be appreciated. Thanks.
--- On Sat, 11/3/12, Alexander Lorenz <[EMAIL PROTECTED]> wrote:
From: Alexander Lorenz <[EMAIL PROTECTED]>
Subject: Re: ExecSource tail -F stops after a while
To: [EMAIL PROTECTED]
Date: Saturday, November 3, 2012, 1:55 AM
Hi,
http://flume.apache.org/FlumeUserGuide.html#exec-source
Please see the WARNING ;)
Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
FileChannel is available in from up to 1.3
http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
cheers
- Alex
On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED]> wrote:
> I tried to use the FileChannel; but am getting an exception -
>
> org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
> Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
>
> I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
>
> Thanks
>
> --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED]> wrote:
>
>
> From: Josh West <[EMAIL PROTECTED]>
> Subject: Re: ExecSource tail -F stops after a while
> To: [EMAIL PROTECTED]
> Date: Friday, November 2, 2012, 1:48 PM
>
>
>
> This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
>
>
> On 11/02/2012 07:15 PM, Parag Hukeri wrote:
>
>
>
>
>
>
> Hi
>
> I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
> --
> Josh West
> Lead Systems Administrator
> One.com, [EMAIL PROTECTED]
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
-
Re: ExecSource tail -F stops after a whileBrock Noland 2012-11-05, 13:51
Can you take a thread dump, using kill -3 not jstack, when you hit that point?
On Monday, November 5, 2012 at 7:39 AM, Parag Hukeri wrote:
> While starting my flume-agent I did use these options:- java -Xms512m -Xmx2048m. But that did not help. What should I tweak in flume-env.sh? Not sure?
> I noticed that the source for fileChannel was included in flume 1.2. So I tried compiling that & using that jar; but got the exact same results with fileChannel - the process stopped after approx 9.5 hrs (after having written approx 9.5 million records).
>
> Any help would be appreciated. Thanks.
>
> --- On Sat, 11/3/12, Alexander Lorenz <[EMAIL PROTECTED]> wrote:
> >
> > From: Alexander Lorenz <[EMAIL PROTECTED]>
> > Subject: Re: ExecSource tail -F stops after a while
> > To: [EMAIL PROTECTED]
> > Date: Saturday, November 3, 2012, 1:55 AM
> >
> > Hi,
> >
> > http://flume.apache.org/FlumeUserGuide.html#exec-source
> > Please see the WARNING ;)
> >
> > Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
> >
> > FileChannel is available in from up to 1.3
> > http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
> >
> > cheers
> > - Alex
> >
> >
> > On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])> wrote:
> >
> > > I tried to use the FileChannel; but am getting an exception -
> > >
> > > org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
> > > Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
> > >
> > > I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
> > >
> > > Thanks
> > >
> > > --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])> wrote:
> > >
> > >
> > > From: Josh West <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])>
> > > Subject: Re: ExecSource tail -F stops after a while
> > > To: [EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])
> > > Date: Friday, November 2, 2012, 1:48 PM
> > >
> > >
> > >
> > > This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
> > >
> > >
> > > On 11/02/2012 07:15 PM, Parag Hukeri wrote:
> > >
> > >
> > >
> > >
> > >
> > >
> > > Hi
> > >
> > > I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
> > > --
> > > Josh West
> > > Lead Systems Administrator
> > > One.com, [EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])
> >
> > --
> > Alexander Alten-Lorenz
> > http://mapredit.blogspot.com (http://mapredit.blogspot.com/)
> > German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >
On Monday, November 5, 2012 at 7:39 AM, Parag Hukeri wrote:
> While starting my flume-agent I did use these options:- java -Xms512m -Xmx2048m. But that did not help. What should I tweak in flume-env.sh? Not sure?
> I noticed that the source for fileChannel was included in flume 1.2. So I tried compiling that & using that jar; but got the exact same results with fileChannel - the process stopped after approx 9.5 hrs (after having written approx 9.5 million records).
>
> Any help would be appreciated. Thanks.
>
> --- On Sat, 11/3/12, Alexander Lorenz <[EMAIL PROTECTED]> wrote:
> >
> > From: Alexander Lorenz <[EMAIL PROTECTED]>
> > Subject: Re: ExecSource tail -F stops after a while
> > To: [EMAIL PROTECTED]
> > Date: Saturday, November 3, 2012, 1:55 AM
> >
> > Hi,
> >
> > http://flume.apache.org/FlumeUserGuide.html#exec-source
> > Please see the WARNING ;)
> >
> > Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
> >
> > FileChannel is available in from up to 1.3
> > http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
> >
> > cheers
> > - Alex
> >
> >
> > On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])> wrote:
> >
> > > I tried to use the FileChannel; but am getting an exception -
> > >
> > > org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
> > > Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
> > >
> > > I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
> > >
> > > Thanks
> > >
> > > --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])> wrote:
> > >
> > >
> > > From: Josh West <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])>
> > > Subject: Re: ExecSource tail -F stops after a while
> > > To: [EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])
> > > Date: Friday, November 2, 2012, 1:48 PM
> > >
> > >
> > >
> > > This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
> > >
> > >
> > > On 11/02/2012 07:15 PM, Parag Hukeri wrote:
> > >
> > >
> > >
> > >
> > >
> > >
> > > Hi
> > >
> > > I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
> > > --
> > > Josh West
> > > Lead Systems Administrator
> > > One.com, [EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])
> >
> > --
> > Alexander Alten-Lorenz
> > http://mapredit.blogspot.com (http://mapredit.blogspot.com/)
> > German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >
-
Re: ExecSource tail -F stops after a whileAlexander Lorenz 2012-11-06, 07:01
Did you remove the older build completely? Are you sure that you've no other FLUME libs installed on your system? How did you configured your sink?
- Alex
On Nov 5, 2012, at 2:51 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> Can you take a thread dump, using kill -3 not jstack, when you hit that point?
>
>
> On Monday, November 5, 2012 at 7:39 AM, Parag Hukeri wrote:
>
>> While starting my flume-agent I did use these options:- java -Xms512m -Xmx2048m. But that did not help. What should I tweak in flume-env.sh? Not sure?
>> I noticed that the source for fileChannel was included in flume 1.2. So I tried compiling that & using that jar; but got the exact same results with fileChannel - the process stopped after approx 9.5 hrs (after having written approx 9.5 million records).
>>
>> Any help would be appreciated. Thanks.
>>
>> --- On Sat, 11/3/12, Alexander Lorenz <[EMAIL PROTECTED]> wrote:
>>>
>>> From: Alexander Lorenz <[EMAIL PROTECTED]>
>>> Subject: Re: ExecSource tail -F stops after a while
>>> To: [EMAIL PROTECTED]
>>> Date: Saturday, November 3, 2012, 1:55 AM
>>>
>>> Hi,
>>>
>>> http://flume.apache.org/FlumeUserGuide.html#exec-source
>>> Please see the WARNING ;)
>>>
>>> Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
>>>
>>> FileChannel is available in from up to 1.3
>>> http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
>>>
>>> cheers
>>> - Alex
>>>
>>>
>>> On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])> wrote:
>>>
>>>> I tried to use the FileChannel; but am getting an exception -
>>>>
>>>> org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
>>>>
>>>> I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
>>>>
>>>> Thanks
>>>>
>>>> --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])> wrote:
>>>>
>>>>
>>>> From: Josh West <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])>
>>>> Subject: Re: ExecSource tail -F stops after a while
>>>> To: [EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])
>>>> Date: Friday, November 2, 2012, 1:48 PM
>>>>
>>>>
>>>>
>>>> This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
>>>>
>>>>
>>>> On 11/02/2012 07:15 PM, Parag Hukeri wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>> I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
>>>> --
>>>> Josh West
>>>> Lead Systems Administrator
>>>> One.com, [EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])
>>>
>>> --
>>> Alexander Alten-Lorenz
>>> http://mapredit.blogspot.com (http://mapredit.blogspot.com/)
>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
- Alex
On Nov 5, 2012, at 2:51 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
> Can you take a thread dump, using kill -3 not jstack, when you hit that point?
>
>
> On Monday, November 5, 2012 at 7:39 AM, Parag Hukeri wrote:
>
>> While starting my flume-agent I did use these options:- java -Xms512m -Xmx2048m. But that did not help. What should I tweak in flume-env.sh? Not sure?
>> I noticed that the source for fileChannel was included in flume 1.2. So I tried compiling that & using that jar; but got the exact same results with fileChannel - the process stopped after approx 9.5 hrs (after having written approx 9.5 million records).
>>
>> Any help would be appreciated. Thanks.
>>
>> --- On Sat, 11/3/12, Alexander Lorenz <[EMAIL PROTECTED]> wrote:
>>>
>>> From: Alexander Lorenz <[EMAIL PROTECTED]>
>>> Subject: Re: ExecSource tail -F stops after a while
>>> To: [EMAIL PROTECTED]
>>> Date: Saturday, November 3, 2012, 1:55 AM
>>>
>>> Hi,
>>>
>>> http://flume.apache.org/FlumeUserGuide.html#exec-source
>>> Please see the WARNING ;)
>>>
>>> Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
>>>
>>> FileChannel is available in from up to 1.3
>>> http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
>>>
>>> cheers
>>> - Alex
>>>
>>>
>>> On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])> wrote:
>>>
>>>> I tried to use the FileChannel; but am getting an exception -
>>>>
>>>> org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
>>>>
>>>> I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
>>>>
>>>> Thanks
>>>>
>>>> --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])> wrote:
>>>>
>>>>
>>>> From: Josh West <[EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])>
>>>> Subject: Re: ExecSource tail -F stops after a while
>>>> To: [EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])
>>>> Date: Friday, November 2, 2012, 1:48 PM
>>>>
>>>>
>>>>
>>>> This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
>>>>
>>>>
>>>> On 11/02/2012 07:15 PM, Parag Hukeri wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>> I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
>>>> --
>>>> Josh West
>>>> Lead Systems Administrator
>>>> One.com, [EMAIL PROTECTED] (http://us.mc1634.mail.yahoo.com/mc/compose?[EMAIL PROTECTED])
>>>
>>> --
>>> Alexander Alten-Lorenz
>>> http://mapredit.blogspot.com (http://mapredit.blogspot.com/)
>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
-
Re: ExecSource tail -F stops after a whileParag Hukeri 2012-11-06, 18:38
Took a threaddump when it hit that point. Looks like the issue might be with some custom interceptor code that we have written. Looking into it. Thanks for pointing in the right direction.
--- On Mon, 11/5/12, Brock Noland <[EMAIL PROTECTED]> wrote:
From: Brock Noland <[EMAIL PROTECTED]>
Subject: Re: ExecSource tail -F stops after a while
To: [EMAIL PROTECTED]
Date: Monday, November 5, 2012, 7:51 AM
Can you take a thread dump, using kill -3 not jstack, when you hit that point?
On Monday, November 5, 2012 at 7:39 AM, Parag Hukeri wrote:
While starting my flume-agent I did use these options:- java -Xms512m -Xmx2048m. But that did not help. What should I tweak in flume-env.sh? Not sure?
I noticed that the source for fileChannel was included in flume 1.2. So I tried compiling that & using that jar; but got the exact same results with fileChannel - the process stopped after approx 9.5 hrs (after having written approx 9.5 million records).
Any help would be appreciated. Thanks.
--- On Sat, 11/3/12, Alexander Lorenz <[EMAIL PROTECTED]> wrote:
From: Alexander Lorenz <[EMAIL PROTECTED]>
Subject: Re: ExecSource tail -F stops after a while
To: [EMAIL PROTECTED]
Date: Saturday, November 3, 2012, 1:55 AM
Hi,
http://flume.apache.org/FlumeUserGuide.html#exec-source
Please see the WARNING ;)
Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
FileChannel is available in from up to 1.3
http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
cheers
- Alex
On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED]> wrote:
> I tried to use the FileChannel; but am getting an exception -
>
> org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
> Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
>
> I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
>
> Thanks
>
> --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED]> wrote:
>
>
> From: Josh West <[EMAIL PROTECTED]>
> Subject: Re: ExecSource tail -F stops after a while
> To: [EMAIL PROTECTED]
> Date: Friday, November 2, 2012, 1:48 PM
>
>
>
> This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
>
>
> On 11/02/2012 07:15 PM, Parag Hukeri wrote:
>
>
>
>
>
>
> Hi
>
> I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
> --
> Josh West
> Lead Systems Administrator
> One.com, [EMAIL PROTECTED]
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
--- On Mon, 11/5/12, Brock Noland <[EMAIL PROTECTED]> wrote:
From: Brock Noland <[EMAIL PROTECTED]>
Subject: Re: ExecSource tail -F stops after a while
To: [EMAIL PROTECTED]
Date: Monday, November 5, 2012, 7:51 AM
Can you take a thread dump, using kill -3 not jstack, when you hit that point?
On Monday, November 5, 2012 at 7:39 AM, Parag Hukeri wrote:
While starting my flume-agent I did use these options:- java -Xms512m -Xmx2048m. But that did not help. What should I tweak in flume-env.sh? Not sure?
I noticed that the source for fileChannel was included in flume 1.2. So I tried compiling that & using that jar; but got the exact same results with fileChannel - the process stopped after approx 9.5 hrs (after having written approx 9.5 million records).
Any help would be appreciated. Thanks.
--- On Sat, 11/3/12, Alexander Lorenz <[EMAIL PROTECTED]> wrote:
From: Alexander Lorenz <[EMAIL PROTECTED]>
Subject: Re: ExecSource tail -F stops after a while
To: [EMAIL PROTECTED]
Date: Saturday, November 3, 2012, 1:55 AM
Hi,
http://flume.apache.org/FlumeUserGuide.html#exec-source
Please see the WARNING ;)
Anyhow, this error indicates that you've to less memory configured for too many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
FileChannel is available in from up to 1.3
http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
cheers
- Alex
On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[EMAIL PROTECTED]> wrote:
> I tried to use the FileChannel; but am getting an exception -
>
> org.apache.flume.FlumeException: Unable to load channel type: file, class: org.apache.flume.channel.file.FileChannel
> Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
>
> I am using apache-flume-1.2.0. What version of flume is needed for FileChannel?
>
> Thanks
>
> --- On Fri, 11/2/12, Josh West <[EMAIL PROTECTED]> wrote:
>
>
> From: Josh West <[EMAIL PROTECTED]>
> Subject: Re: ExecSource tail -F stops after a while
> To: [EMAIL PROTECTED]
> Date: Friday, November 2, 2012, 1:48 PM
>
>
>
> This happened to me consistently when buffering tail -F contents into the Memory Channel. Try switching to the File Channel. This solved the issue for me.
>
>
> On 11/02/2012 07:15 PM, Parag Hukeri wrote:
>
>
>
>
>
>
> Hi
>
> I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log file & gets the entries from this log file. The log file size is approx 4 GB (approx 12 million lines written over a period of 12 hrs). As the process starts everything seems to work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved by ExecSource). It then stops reading the output from tail. I don't see any exceptions being thrown. I don't think the problem is with the actual tail command. I was able to successfully manually tail the log file & all 12 million records were retrieved by tail. Running on a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated. Thanks.
> --
> Josh West
> Lead Systems Administrator
> One.com, [EMAIL PROTECTED]
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
