|
|
lancexxx 2012-10-30, 02:37
why have not log header via flume-ng with avrosource to collecting web log and system log (/var/log/message)? -- lancexxx
-
Re: about the log header
Brock Noland 2012-10-30, 15:17
This is because the default is BodyTextEventSerializer, if you use FlumeEventAvroEventSerializer the headers are persisted. http://flume.apache.org/FlumeUserGuide.html#hdfs-sinkOn Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED]> wrote: > why have not log header via flume-ng with avrosource to collecting web log > and system log (/var/log/message)? > > > -- > lancexxx > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
-
Re: about the log header
lancexxx 2012-10-30, 15:35
oh,sorry,brock,I use syslogudp source and there is the log on the agent is not whole relative to the log of localhost. the same,i found at http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/browserhttps://issues.apache.org/jira/browse/FLUME-1277http://mail-archives.apache.org/mod_mbox/incubator-flume-dev/201206.mbox/%3C1798418757.10377.1339553622911.JavaMail.jiratomcat@issues-vm%3Ebut now I don't understand the issue had not solved of flume-ng 1.3.0 trunk?if yes,how to use the path above? sorry for my basic question agin。 -- lancexxx On 2012年10月30日Tuesday at 下午11:17, Brock Noland wrote: > This is because the default is BodyTextEventSerializer, if you use > FlumeEventAvroEventSerializer the headers are persisted. > > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > On Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > why have not log header via flume-ng with avrosource to collecting web log > > and system log (/var/log/message)? > > > > > > -- > > lancexxx > > > > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> >
-
Re: about the log header
Brock Noland 2012-10-30, 15:38
https://issues.apache.org/jira/browse/FLUME-1277 is not committed to any branch. I do think it will make the 1.3 release though as I am working on it right this minute. On Tue, Oct 30, 2012 at 10:35 AM, lancexxx <[EMAIL PROTECTED]> wrote: > oh,sorry,brock,I use syslogudp source and there is the log on the agent is > not whole relative to the log of localhost. > the same,i found at > http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/browser> https://issues.apache.org/jira/browse/FLUME-1277> http://mail-archives.apache.org/mod_mbox/incubator-flume-dev/201206.mbox/%3C1798418757.10377.1339553622911.JavaMail.jiratomcat@issues-vm%3E> but now I don't understand the issue had not solved of flume-ng 1.3.0 > trunk?if yes,how to use the path above? > sorry for my basic question agin。 > > -- > lancexxx > > On 2012年10月30日Tuesday at 下午11:17, Brock Noland wrote: > > This is because the default is BodyTextEventSerializer, if you use > FlumeEventAvroEventSerializer the headers are persisted. > > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > On Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED]> wrote: > > why have not log header via flume-ng with avrosource to collecting web log > and system log (/var/log/message)? > > > -- > lancexxx > > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
-
Re: about the log header
lancexxx 2012-10-30, 15:53
sorry,what should I do for solving the issue I met? Do you have any suggestions? thanks. -- lancexxx On 2012年10月30日Tuesday at 下午11:38, Brock Noland wrote: > https://issues.apache.org/jira/browse/FLUME-1277 is not committed to > any branch. I do think it will make the 1.3 release though as I am > working on it right this minute. > > On Tue, Oct 30, 2012 at 10:35 AM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > oh,sorry,brock,I use syslogudp source and there is the log on the agent is > > not whole relative to the log of localhost. > > the same,i found at > > http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/browser> > https://issues.apache.org/jira/browse/FLUME-1277> > http://mail-archives.apache.org/mod_mbox/incubator-flume-dev/201206.mbox/%3C1798418757.10377.1339553622911.JavaMail.jiratomcat@issues-vm%3E> > but now I don't understand the issue had not solved of flume-ng 1.3.0 > > trunk?if yes,how to use the path above? > > sorry for my basic question agin。 > > > > -- > > lancexxx > > > > On 2012年10月30日Tuesday at 下午11:17, Brock Noland wrote: > > > > This is because the default is BodyTextEventSerializer, if you use > > FlumeEventAvroEventSerializer the headers are persisted. > > > > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > > > On Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > > > why have not log header via flume-ng with avrosource to collecting web log > > and system log (/var/log/message)? > > > > > > -- > > lancexxx > > > > > > > > > > -- > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> >
-
Re: about the log header
Brock Noland 2012-10-30, 15:56
I would wait for the 1.3 release or apply the patch and build it yourself.. Brock On Tue, Oct 30, 2012 at 10:53 AM, lancexxx <[EMAIL PROTECTED]> wrote: > sorry,what should I do for solving the issue I met? > Do you have any suggestions? > thanks. > -- > lancexxx > > On 2012年10月30日Tuesday at 下午11:38, Brock Noland wrote: > > https://issues.apache.org/jira/browse/FLUME-1277 is not committed to > any branch. I do think it will make the 1.3 release though as I am > working on it right this minute. > > On Tue, Oct 30, 2012 at 10:35 AM, lancexxx <[EMAIL PROTECTED]> wrote: > > oh,sorry,brock,I use syslogudp source and there is the log on the agent is > not whole relative to the log of localhost. > the same,i found at > http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/browser> https://issues.apache.org/jira/browse/FLUME-1277> http://mail-archives.apache.org/mod_mbox/incubator-flume-dev/201206.mbox/%3C1798418757.10377.1339553622911.JavaMail.jiratomcat@issues-vm%3E> but now I don't understand the issue had not solved of flume-ng 1.3.0 > trunk?if yes,how to use the path above? > sorry for my basic question agin。 > > -- > lancexxx > > On 2012年10月30日Tuesday at 下午11:17, Brock Noland wrote: > > This is because the default is BodyTextEventSerializer, if you use > FlumeEventAvroEventSerializer the headers are persisted. > > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > On Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED]> wrote: > > why have not log header via flume-ng with avrosource to collecting web log > and system log (/var/log/message)? > > > -- > lancexxx > > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
-
Re: about the log header
lancexxx 2012-10-30, 16:13
I met the issue is same as this well, I will try it tomorrow (may be I do not know how to path it . Quietly tell you ,I worship you!:) -- lancexxx On 2012年10月30日Tuesday at 下午11:56, Brock Noland wrote: > I would wait for the 1.3 release or apply the patch and build it yourself.. > > Brock > > On Tue, Oct 30, 2012 at 10:53 AM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > sorry,what should I do for solving the issue I met? > > Do you have any suggestions? > > thanks. > > -- > > lancexxx > > > > On 2012年10月30日Tuesday at 下午11:38, Brock Noland wrote: > > > > https://issues.apache.org/jira/browse/FLUME-1277 is not committed to > > any branch. I do think it will make the 1.3 release though as I am > > working on it right this minute. > > > > On Tue, Oct 30, 2012 at 10:35 AM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > > > oh,sorry,brock,I use syslogudp source and there is the log on the agent is > > not whole relative to the log of localhost. > > the same,i found at > > http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/browser> > https://issues.apache.org/jira/browse/FLUME-1277> > http://mail-archives.apache.org/mod_mbox/incubator-flume-dev/201206.mbox/%3C1798418757.10377.1339553622911.JavaMail.jiratomcat@issues-vm%3E> > but now I don't understand the issue had not solved of flume-ng 1.3.0 > > trunk?if yes,how to use the path above? > > sorry for my basic question agin。 > > > > -- > > lancexxx > > > > On 2012年10月30日Tuesday at 下午11:17, Brock Noland wrote: > > > > This is because the default is BodyTextEventSerializer, if you use > > FlumeEventAvroEventSerializer the headers are persisted. > > > > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > > > On Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > > > why have not log header via flume-ng with avrosource to collecting web log > > and system log (/var/log/message)? > > > > > > -- > > lancexxx > > > > > > > > > > -- > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > > > > > > -- > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> >
-
Re: about the log header
Brock Noland 2012-10-30, 16:19
I see, that is this issue: https://issues.apache.org/jira/browse/FLUME-1666The items are added as headers which could be logged with AVRO_EVENT serialization. Brock On Tue, Oct 30, 2012 at 11:13 AM, lancexxx <[EMAIL PROTECTED]> wrote: > I met the issue is same as this > > well, I will try it tomorrow (may be I do not know how to path it . > Quietly tell you ,I worship you!:) > > -- > lancexxx > > On 2012年10月30日Tuesday at 下午11:56, Brock Noland wrote: > > I would wait for the 1.3 release or apply the patch and build it yourself.. > > Brock > > On Tue, Oct 30, 2012 at 10:53 AM, lancexxx <[EMAIL PROTECTED]> > wrote: > > sorry,what should I do for solving the issue I met? > Do you have any suggestions? > thanks. > -- > lancexxx > > On 2012年10月30日Tuesday at 下午11:38, Brock Noland wrote: > > https://issues.apache.org/jira/browse/FLUME-1277 is not committed to > any branch. I do think it will make the 1.3 release though as I am > working on it right this minute. > > On Tue, Oct 30, 2012 at 10:35 AM, lancexxx <[EMAIL PROTECTED]> > wrote: > > oh,sorry,brock,I use syslogudp source and there is the log on the agent is > not whole relative to the log of localhost. > the same,i found at > http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/browser> https://issues.apache.org/jira/browse/FLUME-1277> > http://mail-archives.apache.org/mod_mbox/incubator-flume-dev/201206.mbox/%3C1798418757.10377.1339553622911.JavaMail.jiratomcat@issues-vm%3E> but now I don't understand the issue had not solved of flume-ng 1.3.0 > trunk?if yes,how to use the path above? > sorry for my basic question agin。 > > -- > lancexxx > > On 2012年10月30日Tuesday at 下午11:17, Brock Noland wrote: > > This is because the default is BodyTextEventSerializer, if you use > FlumeEventAvroEventSerializer the headers are persisted. > > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > On Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED]> wrote: > > why have not log header via flume-ng with avrosource to collecting web log > and system log (/var/log/message)? > > > -- > lancexxx > > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/> > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/> > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/> > > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
-
Re: about the log header
lancexxx 2012-10-31, 12:44
I apply the patch (patch -p1 < FLUME-1666-SyslogTextSerializer.patch ) https://issues.apache.org/jira/browse/FLUME-1666and anew mvn compile && mvn install to flume-1.3 but the issue as before。 Is it any wrong with my operating? -- lancexxx On 2012年10月31日Wednesday at 上午12:19, Brock Noland wrote: > I see, that is this issue: > > https://issues.apache.org/jira/browse/FLUME-1666> > The items are added as headers which could be logged with AVRO_EVENT serialization. > > Brock > > On Tue, Oct 30, 2012 at 11:13 AM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > I met the issue is same as this > > > > > > well, I will try it tomorrow (may be I do not know how to path it . > > Quietly tell you ,I worship you!:) > > > > -- > > lancexxx > > > > > > On 2012年10月30日Tuesday at 下午11:56, Brock Noland wrote: > > > > > I would wait for the 1.3 release or apply the patch and build it yourself.. > > > > > > Brock > > > > > > On Tue, Oct 30, 2012 at 10:53 AM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > > > sorry,what should I do for solving the issue I met? > > > > Do you have any suggestions? > > > > thanks. > > > > -- > > > > lancexxx > > > > > > > > On 2012年10月30日Tuesday at 下午11:38, Brock Noland wrote: > > > > > > > > https://issues.apache.org/jira/browse/FLUME-1277 is not committed to > > > > any branch. I do think it will make the 1.3 release though as I am > > > > working on it right this minute. > > > > > > > > On Tue, Oct 30, 2012 at 10:35 AM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > > > > > > > oh,sorry,brock,I use syslogudp source and there is the log on the agent is > > > > not whole relative to the log of localhost. > > > > the same,i found at > > > > http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/browser> > > > https://issues.apache.org/jira/browse/FLUME-1277> > > > http://mail-archives.apache.org/mod_mbox/incubator-flume-dev/201206.mbox/%3C1798418757.10377.1339553622911.JavaMail.jiratomcat@issues-vm%3E> > > > but now I don't understand the issue had not solved of flume-ng 1.3.0 > > > > trunk?if yes,how to use the path above? > > > > sorry for my basic question agin。 > > > > > > > > -- > > > > lancexxx > > > > > > > > On 2012年10月30日Tuesday at 下午11:17, Brock Noland wrote: > > > > > > > > This is because the default is BodyTextEventSerializer, if you use > > > > FlumeEventAvroEventSerializer the headers are persisted. > > > > > > > > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > > > > > > > On Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > > > > > > > why have not log header via flume-ng with avrosource to collecting web log > > > > and system log (/var/log/message)? > > > > > > > > > > > > -- > > > > lancexxx > > > > > > > > > > > > > > > > > > > > -- > > > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > > > > > > > > > > > > > > > > -- > > > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > > > > > > > > > > > > > > > -- > > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > > > > > > > > > > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
-
Re: about the log header
lancexxx 2012-11-01, 09:44
the mean only can use hdfs sink with syslogtcp/udp source? other sink have no sink.serializer properties except file roll sink. -- lancexxx On 2012年10月31日Wednesday at 上午12:19, Brock Noland wrote: > I see, that is this issue: > > https://issues.apache.org/jira/browse/FLUME-1666> > The items are added as headers which could be logged with AVRO_EVENT serialization. > > Brock > > On Tue, Oct 30, 2012 at 11:13 AM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > I met the issue is same as this > > > > > > well, I will try it tomorrow (may be I do not know how to path it . > > Quietly tell you ,I worship you!:) > > > > -- > > lancexxx > > > > > > On 2012年10月30日Tuesday at 下午11:56, Brock Noland wrote: > > > > > I would wait for the 1.3 release or apply the patch and build it yourself.. > > > > > > Brock > > > > > > On Tue, Oct 30, 2012 at 10:53 AM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > > > sorry,what should I do for solving the issue I met? > > > > Do you have any suggestions? > > > > thanks. > > > > -- > > > > lancexxx > > > > > > > > On 2012年10月30日Tuesday at 下午11:38, Brock Noland wrote: > > > > > > > > https://issues.apache.org/jira/browse/FLUME-1277 is not committed to > > > > any branch. I do think it will make the 1.3 release though as I am > > > > working on it right this minute. > > > > > > > > On Tue, Oct 30, 2012 at 10:35 AM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > > > > > > > oh,sorry,brock,I use syslogudp source and there is the log on the agent is > > > > not whole relative to the log of localhost. > > > > the same,i found at > > > > http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/browser> > > > https://issues.apache.org/jira/browse/FLUME-1277> > > > http://mail-archives.apache.org/mod_mbox/incubator-flume-dev/201206.mbox/%3C1798418757.10377.1339553622911.JavaMail.jiratomcat@issues-vm%3E> > > > but now I don't understand the issue had not solved of flume-ng 1.3.0 > > > > trunk?if yes,how to use the path above? > > > > sorry for my basic question agin。 > > > > > > > > -- > > > > lancexxx > > > > > > > > On 2012年10月30日Tuesday at 下午11:17, Brock Noland wrote: > > > > > > > > This is because the default is BodyTextEventSerializer, if you use > > > > FlumeEventAvroEventSerializer the headers are persisted. > > > > > > > > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > > > > > > > On Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: > > > > > > > > why have not log header via flume-ng with avrosource to collecting web log > > > > and system log (/var/log/message)? > > > > > > > > > > > > -- > > > > lancexxx > > > > > > > > > > > > > > > > > > > > -- > > > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > > > > > > > > > > > > > > > > -- > > > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > > > > > > > > > > > > > > > -- > > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/> > > > > > > > > > > > > > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
-
Re: about the log header
Brock Noland 2012-11-01, 11:31
No hdfs sink can be used with any source. On Thu, Nov 1, 2012 at 4:44 AM, lancexxx <[EMAIL PROTECTED]> wrote: > the mean only can use hdfs sink with syslogtcp/udp source? other sink > have no sink.serializer properties except file roll sink. > > -- > lancexxx > > On 2012年10月31日Wednesday at 上午12:19, Brock Noland wrote: > > I see, that is this issue: > > https://issues.apache.org/jira/browse/FLUME-1666> > The items are added as headers which could be logged with > AVRO_EVENT serialization. > > Brock > > On Tue, Oct 30, 2012 at 11:13 AM, lancexxx <[EMAIL PROTECTED]>wrote: > > I met the issue is same as this > > well, I will try it tomorrow (may be I do not know how to path it . > Quietly tell you ,I worship you!:) > > -- > lancexxx > > On 2012年10月30日Tuesday at 下午11:56, Brock Noland wrote: > > I would wait for the 1.3 release or apply the patch and build it yourself.. > > Brock > > On Tue, Oct 30, 2012 at 10:53 AM, lancexxx <[EMAIL PROTECTED]> > wrote: > > sorry,what should I do for solving the issue I met? > Do you have any suggestions? > thanks. > -- > lancexxx > > On 2012年10月30日Tuesday at 下午11:38, Brock Noland wrote: > > https://issues.apache.org/jira/browse/FLUME-1277 is not committed to > any branch. I do think it will make the 1.3 release though as I am > working on it right this minute. > > On Tue, Oct 30, 2012 at 10:35 AM, lancexxx <[EMAIL PROTECTED]> > wrote: > > oh,sorry,brock,I use syslogudp source and there is the log on the agent > is > not whole relative to the log of localhost. > the same,i found at > http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/browser> https://issues.apache.org/jira/browse/FLUME-1277> > http://mail-archives.apache.org/mod_mbox/incubator-flume-dev/201206.mbox/%3C1798418757.10377.1339553622911.JavaMail.jiratomcat@issues-vm%3E> but now I don't understand the issue had not solved of flume-ng 1.3.0 > trunk?if yes,how to use the path above? > sorry for my basic question agin。 > > -- > lancexxx > > On 2012年10月30日Tuesday at 下午11:17, Brock Noland wrote: > > This is because the default is BodyTextEventSerializer, if you use > FlumeEventAvroEventSerializer the headers are persisted. > > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > On Mon, Oct 29, 2012 at 9:37 PM, lancexxx <[EMAIL PROTECTED]> wrote: > > why have not log header via flume-ng with avrosource to collecting web log > and system log (/var/log/message)? > > > -- > lancexxx > > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/> > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/> > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/> > > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/> > > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
|
|