Ashutoshsharma
2012-11-08, 01:39
Alexander Lorenz
2012-11-08, 07:20
Ashutoshsharma
2012-11-08, 07:37
Juhani Connolly
2012-11-08, 08:07
Ashutoshsharma
2012-11-08, 08:18
Ashutoshsharma
2012-11-09, 08:43
Nitin Pawar
2012-11-09, 08:48
Juhani Connolly
2012-11-09, 10:08
Roshan Naik
2012-11-09, 11:46
Ashutoshsharma
2012-11-12, 02:05
Ashutoshsharma
2012-11-12, 02:07
-
multiple agentsAshutoshsharma 2012-11-08, 01:39
Hi,
I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent -n agent1” command, so two flume process.
Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
Please share your suggestions and thoughts.
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent -n agent1” command, so two flume process.
Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
Please share your suggestions and thoughts.
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
-
Re: multiple agentsAlexander Lorenz 2012-11-08, 07:20
Hi,
you can use one config-file and define the agents with flow1, flow2 and so on. I assume, the sources of the logs are different types, isn't it? When you let grab them from one agent you could create multiple flows with (http://flume.apache.org/FlumeUserGuide.html#multiplexing-channel-selector).
Please note, you've to tweak the settings in flume-env.sh to reflect to the needed memory consumption (rename flume-env.sh.template into flume-env.sh and comment the Xmx parameters out).
cheers,
Alex
On Nov 8, 2012, at 2:39 AM, Ashutoshsharma(오픈플랫폼개발팀) <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent -n agent1” command, so two flume process.
>
> Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
>
> Please share your suggestions and thoughts.
>
> ----------------------------------------
> Thanks & Regards,
> Ashutosh Sharma
> ----------------------------------------
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
you can use one config-file and define the agents with flow1, flow2 and so on. I assume, the sources of the logs are different types, isn't it? When you let grab them from one agent you could create multiple flows with (http://flume.apache.org/FlumeUserGuide.html#multiplexing-channel-selector).
Please note, you've to tweak the settings in flume-env.sh to reflect to the needed memory consumption (rename flume-env.sh.template into flume-env.sh and comment the Xmx parameters out).
cheers,
Alex
On Nov 8, 2012, at 2:39 AM, Ashutoshsharma(오픈플랫폼개발팀) <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent -n agent1” command, so two flume process.
>
> Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
>
> Please share your suggestions and thoughts.
>
> ----------------------------------------
> Thanks & Regards,
> Ashutosh Sharma
> ----------------------------------------
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
-
RE: multiple agentsAshutoshsharma 2012-11-08, 07:37
Hi Alex,
Thank you for your response and inputs. Yes, we have different types of logs sources. Even, I was thinking the same solution and it's well explained in the document. Just wanted to hear from experts about the ideal way to define the agents.
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
-----Original Message-----
From: Alexander Lorenz [mailto:[EMAIL PROTECTED]]
Sent: Thursday, November 08, 2012 4:20 PM
To: [EMAIL PROTECTED]
Subject: Re: multiple agents
Hi,
you can use one config-file and define the agents with flow1, flow2 and so on. I assume, the sources of the logs are different types, isn't it? When you let grab them from one agent you could create multiple flows with (http://flume.apache.org/FlumeUserGuide.html#multiplexing-channel-selector).
Please note, you've to tweak the settings in flume-env.sh to reflect to the needed memory consumption (rename flume-env.sh.template into flume-env.sh and comment the Xmx parameters out).
cheers,
Alex
On Nov 8, 2012, at 2:39 AM, Ashutoshsharma(오픈플랫폼개발팀) <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent -n agent1” command, so two flume process.
>
> Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
>
> Please share your suggestions and thoughts.
>
> ----------------------------------------
> Thanks & Regards,
> Ashutosh Sharma
> ----------------------------------------
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
Thank you for your response and inputs. Yes, we have different types of logs sources. Even, I was thinking the same solution and it's well explained in the document. Just wanted to hear from experts about the ideal way to define the agents.
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
-----Original Message-----
From: Alexander Lorenz [mailto:[EMAIL PROTECTED]]
Sent: Thursday, November 08, 2012 4:20 PM
To: [EMAIL PROTECTED]
Subject: Re: multiple agents
Hi,
you can use one config-file and define the agents with flow1, flow2 and so on. I assume, the sources of the logs are different types, isn't it? When you let grab them from one agent you could create multiple flows with (http://flume.apache.org/FlumeUserGuide.html#multiplexing-channel-selector).
Please note, you've to tweak the settings in flume-env.sh to reflect to the needed memory consumption (rename flume-env.sh.template into flume-env.sh and comment the Xmx parameters out).
cheers,
Alex
On Nov 8, 2012, at 2:39 AM, Ashutoshsharma(오픈플랫폼개발팀) <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent -n agent1” command, so two flume process.
>
> Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
>
> Please share your suggestions and thoughts.
>
> ----------------------------------------
> Thanks & Regards,
> Ashutosh Sharma
> ----------------------------------------
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
-
Re: multiple agentsJuhani Connolly 2012-11-08, 08:07
Hi Ashutosh,
as was pointed out, one configuration will work fine.
There is nothing stopping you running multiple background tasks, but
that won't be possible with the service scripts that come with the flume
packaged in cdh, you'd have to write your own service scripts. But
really I can't think of a use case where you would want multiple processes
On 11/08/2012 10:39 AM, Ashutoshsharma(占쏙옙占쏙옙占시뤄옙占쏙옙占쏙옙占� wrote:
>
> Hi,
>
> I have sources to collect multiple types of logs(mainly three types).
> Most of them generate at least two types of logs. That mean, a server
> generates two types of log. For my use case, I created two separate
> agents running on a server to collect the logs. I am running these
> agents in foreground using 占쏙옙flume-ng agent --n agent1占쏙옙 command, so two
> flume process.
>
> Now, I have doubt, should I merge these two agents configuration into
> single flume.conf file? If I want to continue to use two different
> conf file, then how can I run the two agents in background? Is there
> any known performance issue with any of these approaches?
>
> Please share your suggestions and thoughts.
>
> ----------------------------------------
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
> ----------------------------------------
>
>
> 占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쏙옙占싸몌옙占쏙옙 占쏙옙占쏙옙 占쌜쇽옙占실억옙占쏙옙占쏙옙, 占쌩울옙占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쌜깍옙占쏙옙
> 占쏙옙占쏙옙占싹곤옙 占쏙옙占쏙옙 占쏙옙 占쌍쏙옙占싹댐옙. 占쏘떠占쏙옙 占쏙옙占쏙옙 占쏙옙占쏙옙, 占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쌉듸옙 占쏙옙占쏙옙占쏙옙
> 占쏙옙占�占실댐옙 占싹부몌옙 占쏙옙占쏙옙占쏙옙占쏙옙 占쏙옙3占쌘울옙占쏙옙 占쏙옙, 占쏙옙占쏙옙, 占쏙옙占쏙옙 占실댐옙 占쏙옙占쏙옙求占�占쏙옙
> 占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쏙옙占쌌니댐옙. 占쏙옙占쏙옙, 占쏙옙 占쏙옙占쏙옙占쏙옙 占쌩몌옙 占쏙옙滂占�占쏙옙占� 占쌩쏙옙占쏙옙 占실댐옙 占쏙옙
> 占썹에 占싯뤄옙占쌍시곤옙, 占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占�占쏙옙占쏙옙占싹울옙 占쌍시깍옙 占쌕띰옙求占�
> This E-mail may contain confidential information and/or copyright
> material. This email is intended for the use of the addressee only. If
> you receive this email by mistake, please either delete it without
> reproducing, distributing or retaining copies thereof or notify the
> sender immediately.
>
as was pointed out, one configuration will work fine.
There is nothing stopping you running multiple background tasks, but
that won't be possible with the service scripts that come with the flume
packaged in cdh, you'd have to write your own service scripts. But
really I can't think of a use case where you would want multiple processes
On 11/08/2012 10:39 AM, Ashutoshsharma(占쏙옙占쏙옙占시뤄옙占쏙옙占쏙옙占� wrote:
>
> Hi,
>
> I have sources to collect multiple types of logs(mainly three types).
> Most of them generate at least two types of logs. That mean, a server
> generates two types of log. For my use case, I created two separate
> agents running on a server to collect the logs. I am running these
> agents in foreground using 占쏙옙flume-ng agent --n agent1占쏙옙 command, so two
> flume process.
>
> Now, I have doubt, should I merge these two agents configuration into
> single flume.conf file? If I want to continue to use two different
> conf file, then how can I run the two agents in background? Is there
> any known performance issue with any of these approaches?
>
> Please share your suggestions and thoughts.
>
> ----------------------------------------
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
> ----------------------------------------
>
>
> 占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쏙옙占싸몌옙占쏙옙 占쏙옙占쏙옙 占쌜쇽옙占실억옙占쏙옙占쏙옙, 占쌩울옙占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쌜깍옙占쏙옙
> 占쏙옙占쏙옙占싹곤옙 占쏙옙占쏙옙 占쏙옙 占쌍쏙옙占싹댐옙. 占쏘떠占쏙옙 占쏙옙占쏙옙 占쏙옙占쏙옙, 占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쌉듸옙 占쏙옙占쏙옙占쏙옙
> 占쏙옙占�占실댐옙 占싹부몌옙 占쏙옙占쏙옙占쏙옙占쏙옙 占쏙옙3占쌘울옙占쏙옙 占쏙옙, 占쏙옙占쏙옙, 占쏙옙占쏙옙 占실댐옙 占쏙옙占쏙옙求占�占쏙옙
> 占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占쏙옙占쌌니댐옙. 占쏙옙占쏙옙, 占쏙옙 占쏙옙占쏙옙占쏙옙 占쌩몌옙 占쏙옙滂占�占쏙옙占� 占쌩쏙옙占쏙옙 占실댐옙 占쏙옙
> 占썹에 占싯뤄옙占쌍시곤옙, 占쏙옙 占쏙옙占쏙옙占쏙옙 占쏙옙占�占쏙옙占쏙옙占싹울옙 占쌍시깍옙 占쌕띰옙求占�
> This E-mail may contain confidential information and/or copyright
> material. This email is intended for the use of the addressee only. If
> you receive this email by mistake, please either delete it without
> reproducing, distributing or retaining copies thereof or notify the
> sender immediately.
>
-
RE: multiple agentsAshutoshsharma 2012-11-08, 08:18
Thank you Juhani for your thoughts!
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
From: Juhani Connolly [mailto:[EMAIL PROTECTED]]
Sent: Thursday, November 08, 2012 5:07 PM
To: [EMAIL PROTECTED]
Subject: Re: multiple agents
Hi Ashutosh,
as was pointed out, one configuration will work fine.
There is nothing stopping you running multiple background tasks, but that won't be possible with the service scripts that come with the flume packaged in cdh, you'd have to write your own service scripts. But really I can't think of a use case where you would want multiple processes
On 11/08/2012 10:39 AM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
Hi,
I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent –n agent1” command, so two flume process.
Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
Please share your suggestions and thoughts.
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
From: Juhani Connolly [mailto:[EMAIL PROTECTED]]
Sent: Thursday, November 08, 2012 5:07 PM
To: [EMAIL PROTECTED]
Subject: Re: multiple agents
Hi Ashutosh,
as was pointed out, one configuration will work fine.
There is nothing stopping you running multiple background tasks, but that won't be possible with the service scripts that come with the flume packaged in cdh, you'd have to write your own service scripts. But really I can't think of a use case where you would want multiple processes
On 11/08/2012 10:39 AM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
Hi,
I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent –n agent1” command, so two flume process.
Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
Please share your suggestions and thoughts.
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
-
RE: multiple agentsAshutoshsharma 2012-11-09, 08:43
Hi,
Can I define the multiple flows with different sources, sinks and channels as below:
agent.sources = tx-avro dev-avro web-avro
agent.sinks = tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink web-es-sink web-hdfs-sink
agent.channels = tx-mem-channel tx-file-channel dev-mem-channel dev-file-channel web-mem-channel web-file-channel
##### Flow1 - Start #################################
## Define Avro source
agent.sources.tx-avro.type = avro
agent.sources.tx-avro.bind = 0.0.0.0
agent.sources.tx-avro.port = 35853
agent.sources.tx-avro.channels = tx-mem-channel tx-file-channel
agent.sources.tx-avro.selector.type = replicating
## Define HDFS sink
agent.sinks.tx-hdfs-sink.type = hdfs
agent.sinks.tx-hdfs-sink.hdfs.path = hdfs://…/%{hostname}/%Y-%m-%d
agent.sinks.tx-hdfs-sink.hdfs.fileType = DataStream
agent.sinks.tx-hdfs-sink.hdfs.writeFormat = Text
agent.sinks.tx-hdfs-sink.hdfs.filePrefix = transaction
agent.sinks.tx-hdfs-sink.channel = tx-file-channel
agent.sinks.tx-hdfs-sink.hdfs.rollCount = 0
agent.sinks.tx-hdfs-sink.hdfs.rollSize = 0
agent.sinks.tx-hdfs-sink.hdfs.rollInterval = 600
## Define es sink
agent.sinks.tx-es-sink.type = org.flume.sink.ESSink
agent.sinks.tx-es-sink.indexName = txlog
agent.sinks.tx-es-sink.typeName = tx
agent.sinks.tx-es-sink.cluster = es-cluster
agent.sinks.tx-es-sink.host = 9.127.216.198
agent.sinks.tx-es-sink.channel = tx-mem-channel
## Define the memory channel
agent.channels.tx-mem-channel.type = memory
agent.channels.tx-mem-channel.capacity = 10000
agent.channels.tx-mem-channel.transactionCapacity = 20
## Define the file channel
agent.channels.tx-file-channel.type = FILE
agent.channels.tx-file-channel.checkpointDir = /flume/agent/tx-file-channel/checkpoint
agent.channels.tx-file-channel.dataDirs = /flume/agent/tx-file-channel/data
Same as flow1 #### Flow2 #####....#### Flow3 ######....... is defined with different port for avro source. Here I am using flow1, flow2 and flow3 for three different types of logs and store separately i.e. different location.
When I defined the flume.conf(collector) as mentioned above, agents failed to connect to the avro sources. It returns the RPC connection error. However, I checked that the agent is able to send the events to the collector if I specify only one avro source.
So, the question is, can I define the mentioned configuration to have multiple agents(flows) as mentioned above?
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
From: Juhani Connolly [mailto:[EMAIL PROTECTED]]
Sent: Thursday, November 08, 2012 5:07 PM
To: [EMAIL PROTECTED]
Subject: Re: multiple agents
Hi Ashutosh,
as was pointed out, one configuration will work fine.
There is nothing stopping you running multiple background tasks, but that won't be possible with the service scripts that come with the flume packaged in cdh, you'd have to write your own service scripts. But really I can't think of a use case where you would want multiple processes
On 11/08/2012 10:39 AM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
Hi,
I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent –n agent1” command, so two flume process.
Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
Please share your suggestions and thoughts.
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
Can I define the multiple flows with different sources, sinks and channels as below:
agent.sources = tx-avro dev-avro web-avro
agent.sinks = tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink web-es-sink web-hdfs-sink
agent.channels = tx-mem-channel tx-file-channel dev-mem-channel dev-file-channel web-mem-channel web-file-channel
##### Flow1 - Start #################################
## Define Avro source
agent.sources.tx-avro.type = avro
agent.sources.tx-avro.bind = 0.0.0.0
agent.sources.tx-avro.port = 35853
agent.sources.tx-avro.channels = tx-mem-channel tx-file-channel
agent.sources.tx-avro.selector.type = replicating
## Define HDFS sink
agent.sinks.tx-hdfs-sink.type = hdfs
agent.sinks.tx-hdfs-sink.hdfs.path = hdfs://…/%{hostname}/%Y-%m-%d
agent.sinks.tx-hdfs-sink.hdfs.fileType = DataStream
agent.sinks.tx-hdfs-sink.hdfs.writeFormat = Text
agent.sinks.tx-hdfs-sink.hdfs.filePrefix = transaction
agent.sinks.tx-hdfs-sink.channel = tx-file-channel
agent.sinks.tx-hdfs-sink.hdfs.rollCount = 0
agent.sinks.tx-hdfs-sink.hdfs.rollSize = 0
agent.sinks.tx-hdfs-sink.hdfs.rollInterval = 600
## Define es sink
agent.sinks.tx-es-sink.type = org.flume.sink.ESSink
agent.sinks.tx-es-sink.indexName = txlog
agent.sinks.tx-es-sink.typeName = tx
agent.sinks.tx-es-sink.cluster = es-cluster
agent.sinks.tx-es-sink.host = 9.127.216.198
agent.sinks.tx-es-sink.channel = tx-mem-channel
## Define the memory channel
agent.channels.tx-mem-channel.type = memory
agent.channels.tx-mem-channel.capacity = 10000
agent.channels.tx-mem-channel.transactionCapacity = 20
## Define the file channel
agent.channels.tx-file-channel.type = FILE
agent.channels.tx-file-channel.checkpointDir = /flume/agent/tx-file-channel/checkpoint
agent.channels.tx-file-channel.dataDirs = /flume/agent/tx-file-channel/data
Same as flow1 #### Flow2 #####....#### Flow3 ######....... is defined with different port for avro source. Here I am using flow1, flow2 and flow3 for three different types of logs and store separately i.e. different location.
When I defined the flume.conf(collector) as mentioned above, agents failed to connect to the avro sources. It returns the RPC connection error. However, I checked that the agent is able to send the events to the collector if I specify only one avro source.
So, the question is, can I define the mentioned configuration to have multiple agents(flows) as mentioned above?
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
From: Juhani Connolly [mailto:[EMAIL PROTECTED]]
Sent: Thursday, November 08, 2012 5:07 PM
To: [EMAIL PROTECTED]
Subject: Re: multiple agents
Hi Ashutosh,
as was pointed out, one configuration will work fine.
There is nothing stopping you running multiple background tasks, but that won't be possible with the service scripts that come with the flume packaged in cdh, you'd have to write your own service scripts. But really I can't think of a use case where you would want multiple processes
On 11/08/2012 10:39 AM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
Hi,
I have sources to collect multiple types of logs(mainly three types). Most of them generate at least two types of logs. That mean, a server generates two types of log. For my use case, I created two separate agents running on a server to collect the logs. I am running these agents in foreground using “flume-ng agent –n agent1” command, so two flume process.
Now, I have doubt, should I merge these two agents configuration into single flume.conf file? If I want to continue to use two different conf file, then how can I run the two agents in background? Is there any known performance issue with any of these approaches?
Please share your suggestions and thoughts.
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
-
Re: multiple agentsNitin Pawar 2012-11-09, 08:48
you can have the conf in same file.
when you start the agent you give the name of agent and start it.
you can always start a process in background in linux with &
This does not hamper any performance as long as I know but experts may
point out if I am wrong
On Thu, Nov 8, 2012 at 7:09 AM, Ashutoshsharma(오픈플랫폼개발팀) <
[EMAIL PROTECTED]> wrote:
> Hi,
>
>
>
> I have sources to collect multiple types of logs(mainly three types). Most
> of them generate at least two types of logs. That mean, a server generates
> two types of log. For my use case, I created two separate agents running on
> a server to collect the logs. I am running these agents in foreground using
> "flume-ng agent -n agent1" command, so two flume process.
>
>
>
> Now, I have doubt, should I merge these two agents configuration into
> single flume.conf file? If I want to continue to use two different conf
> file, then how can I run the two agents in background? Is there any known
> performance issue with any of these approaches?
>
>
>
> Please share your suggestions and thoughts.
>
>
>
> ----------------------------------------
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
> ----------------------------------------
>
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에
> 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못
> 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright
> material. This email is intended for the use of the addressee only. If you
> receive this email by mistake, please either delete it without reproducing,
> distributing or retaining copies thereof or notify the sender immediately.
>
--
Nitin Pawar
when you start the agent you give the name of agent and start it.
you can always start a process in background in linux with &
This does not hamper any performance as long as I know but experts may
point out if I am wrong
On Thu, Nov 8, 2012 at 7:09 AM, Ashutoshsharma(오픈플랫폼개발팀) <
[EMAIL PROTECTED]> wrote:
> Hi,
>
>
>
> I have sources to collect multiple types of logs(mainly three types). Most
> of them generate at least two types of logs. That mean, a server generates
> two types of log. For my use case, I created two separate agents running on
> a server to collect the logs. I am running these agents in foreground using
> "flume-ng agent -n agent1" command, so two flume process.
>
>
>
> Now, I have doubt, should I merge these two agents configuration into
> single flume.conf file? If I want to continue to use two different conf
> file, then how can I run the two agents in background? Is there any known
> performance issue with any of these approaches?
>
>
>
> Please share your suggestions and thoughts.
>
>
>
> ----------------------------------------
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
> ----------------------------------------
>
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에
> 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못
> 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright
> material. This email is intended for the use of the addressee only. If you
> receive this email by mistake, please either delete it without reproducing,
> distributing or retaining copies thereof or notify the sender immediately.
>
--
Nitin Pawar
-
Re: multiple agentsJuhani Connolly 2012-11-09, 10:08
I can't see any obvious problem with your config.
When you start up, check your logs if all the components were correctly
configured and started. You may need to adjust the log4j configurations
in your conf directory.
Are all your file channels configured to write to different directories?
If they have the same place setup, things aren't going to work well.
On 11/09/2012 05:43 PM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
>
> Hi,
>
> Can I define the multiple flows with different sources, sinks and
> channels as below:
>
> agent.sources = tx-avro dev-avro web-avro
>
> agent.sinks = tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink
> web-es-sink web-hdfs-sink
>
> agent.channels = tx-mem-channel tx-file-channel dev-mem-channel
> dev-file-channel web-mem-channel web-file-channel
>
> ##### Flow1 - Start #################################
>
> ## Define Avro source
>
> agent.sources.tx-avro.type = avro
>
> agent.sources.tx-avro.bind = 0.0.0.0
>
> agent.sources.tx-avro.port = 35853
>
> agent.sources.tx-avro.channels = tx-mem-channel tx-file-channel
>
> agent.sources.tx-avro.selector.type = replicating
>
> ## Define HDFS sink
>
> agent.sinks.tx-hdfs-sink.type = hdfs
>
> agent.sinks.tx-hdfs-sink.hdfs.path = hdfs://…/%{hostname}/%Y-%m-%d
>
> agent.sinks.tx-hdfs-sink.hdfs.fileType = DataStream
>
> agent.sinks.tx-hdfs-sink.hdfs.writeFormat = Text
>
> agent.sinks.tx-hdfs-sink.hdfs.filePrefix = transaction
>
> agent.sinks.tx-hdfs-sink.channel = tx-file-channel
>
> agent.sinks.tx-hdfs-sink.hdfs.rollCount = 0
>
> agent.sinks.tx-hdfs-sink.hdfs.rollSize = 0
>
> agent.sinks.tx-hdfs-sink.hdfs.rollInterval = 600
>
> ## Define es sink
>
> agent.sinks.tx-es-sink.type = org.flume.sink.ESSink
>
> agent.sinks.tx-es-sink.indexName = txlog
>
> agent.sinks.tx-es-sink.typeName = tx
>
> agent.sinks.tx-es-sink.cluster = es-cluster
>
> agent.sinks.tx-es-sink.host = 9.127.216.198
>
> agent.sinks.tx-es-sink.channel = tx-mem-channel
>
> ## Define the memory channel
>
> agent.channels.tx-mem-channel.type = memory
>
> agent.channels.tx-mem-channel.capacity = 10000
>
> agent.channels.tx-mem-channel.transactionCapacity = 20
>
> ## Define the file channel
>
> agent.channels.tx-file-channel.type = FILE
>
> agent.channels.tx-file-channel.checkpointDir =
> /flume/agent/tx-file-channel/checkpoint
>
> agent.channels.tx-file-channel.dataDirs =
> /flume/agent/tx-file-channel/data
>
> Same as flow1 #### Flow2 #####....#### Flow3 ######....... is defined
> with different port for avro source. Here I am using flow1, flow2 and
> flow3 for three different types of logs and store separately i.e.
> different location.
>
> When I defined the flume.conf(collector) as mentioned above, agents
> failed to connect to the avro sources. It returns the RPC connection
> error. However, I checked that the agent is able to send the events to
> the collector if I specify only one avro source.
>
> So, the question is, can I define the mentioned configuration to have
> multiple agents(flows) as mentioned above?
>
> ----------------------------------------
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
> ----------------------------------------
>
> *From:*Juhani Connolly [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, November 08, 2012 5:07 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: multiple agents
>
> Hi Ashutosh,
>
> as was pointed out, one configuration will work fine.
>
> There is nothing stopping you running multiple background tasks, but
> that won't be possible with the service scripts that come with the
> flume packaged in cdh, you'd have to write your own service scripts.
> But really I can't think of a use case where you would want multiple
> processes
>
> On 11/08/2012 10:39 AM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
>
> Hi,
>
> I have sources to collect multiple types of logs(mainly three
> types). Most of them generate at least two types of logs. That
> mean, a server generates two types of log. For my use case, I
> created two separate agents running on a server to collect the
When you start up, check your logs if all the components were correctly
configured and started. You may need to adjust the log4j configurations
in your conf directory.
Are all your file channels configured to write to different directories?
If they have the same place setup, things aren't going to work well.
On 11/09/2012 05:43 PM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
>
> Hi,
>
> Can I define the multiple flows with different sources, sinks and
> channels as below:
>
> agent.sources = tx-avro dev-avro web-avro
>
> agent.sinks = tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink
> web-es-sink web-hdfs-sink
>
> agent.channels = tx-mem-channel tx-file-channel dev-mem-channel
> dev-file-channel web-mem-channel web-file-channel
>
> ##### Flow1 - Start #################################
>
> ## Define Avro source
>
> agent.sources.tx-avro.type = avro
>
> agent.sources.tx-avro.bind = 0.0.0.0
>
> agent.sources.tx-avro.port = 35853
>
> agent.sources.tx-avro.channels = tx-mem-channel tx-file-channel
>
> agent.sources.tx-avro.selector.type = replicating
>
> ## Define HDFS sink
>
> agent.sinks.tx-hdfs-sink.type = hdfs
>
> agent.sinks.tx-hdfs-sink.hdfs.path = hdfs://…/%{hostname}/%Y-%m-%d
>
> agent.sinks.tx-hdfs-sink.hdfs.fileType = DataStream
>
> agent.sinks.tx-hdfs-sink.hdfs.writeFormat = Text
>
> agent.sinks.tx-hdfs-sink.hdfs.filePrefix = transaction
>
> agent.sinks.tx-hdfs-sink.channel = tx-file-channel
>
> agent.sinks.tx-hdfs-sink.hdfs.rollCount = 0
>
> agent.sinks.tx-hdfs-sink.hdfs.rollSize = 0
>
> agent.sinks.tx-hdfs-sink.hdfs.rollInterval = 600
>
> ## Define es sink
>
> agent.sinks.tx-es-sink.type = org.flume.sink.ESSink
>
> agent.sinks.tx-es-sink.indexName = txlog
>
> agent.sinks.tx-es-sink.typeName = tx
>
> agent.sinks.tx-es-sink.cluster = es-cluster
>
> agent.sinks.tx-es-sink.host = 9.127.216.198
>
> agent.sinks.tx-es-sink.channel = tx-mem-channel
>
> ## Define the memory channel
>
> agent.channels.tx-mem-channel.type = memory
>
> agent.channels.tx-mem-channel.capacity = 10000
>
> agent.channels.tx-mem-channel.transactionCapacity = 20
>
> ## Define the file channel
>
> agent.channels.tx-file-channel.type = FILE
>
> agent.channels.tx-file-channel.checkpointDir =
> /flume/agent/tx-file-channel/checkpoint
>
> agent.channels.tx-file-channel.dataDirs =
> /flume/agent/tx-file-channel/data
>
> Same as flow1 #### Flow2 #####....#### Flow3 ######....... is defined
> with different port for avro source. Here I am using flow1, flow2 and
> flow3 for three different types of logs and store separately i.e.
> different location.
>
> When I defined the flume.conf(collector) as mentioned above, agents
> failed to connect to the avro sources. It returns the RPC connection
> error. However, I checked that the agent is able to send the events to
> the collector if I specify only one avro source.
>
> So, the question is, can I define the mentioned configuration to have
> multiple agents(flows) as mentioned above?
>
> ----------------------------------------
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
> ----------------------------------------
>
> *From:*Juhani Connolly [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, November 08, 2012 5:07 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: multiple agents
>
> Hi Ashutosh,
>
> as was pointed out, one configuration will work fine.
>
> There is nothing stopping you running multiple background tasks, but
> that won't be possible with the service scripts that come with the
> flume packaged in cdh, you'd have to write your own service scripts.
> But really I can't think of a use case where you would want multiple
> processes
>
> On 11/08/2012 10:39 AM, Ashutoshsharma(오픈플랫폼개발팀) wrote:
>
> Hi,
>
> I have sources to collect multiple types of logs(mainly three
> types). Most of them generate at least two types of logs. That
> mean, a server generates two types of log. For my use case, I
> created two separate agents running on a server to collect the
-
Re: multiple agentsRoshan Naik 2012-11-09, 11:46
Ashutosh,
You didnt mention.. but I assume you have named each agent differently.
Can you provide the INFO or WARN level logs ?
-roshan
You didnt mention.. but I assume you have named each agent differently.
Can you provide the INFO or WARN level logs ?
-roshan
-
RE: multiple agentsAshutoshsharma 2012-11-12, 02:05
Hi All,
Thank you for your responses!
I already mentioned that I am running multiple flow from flume configuration file. The configuration file is working fine with the apache release “1.2”. But when I use this configuration file with CDH4 release it doesn’t work. I don’t understand this behavior and there is very less information in flume.log file to identify problem. It seems to me that the agent is stuck somewhere. Below is my log file content:
2012-11-12 09:20:40,237 INFO node.FlumeNode: Flume node stopping - agent
2012-11-12 09:20:40,237 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-11-12 09:20:40,238 INFO properties.PropertiesFileConfigurationProvider: Configuration provider stopping
2012-11-12 09:20:40,238 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider stopped
2012-11-12 09:20:40,238 INFO nodemanager.DefaultLogicalNodeManager: Node manager stopping
2012-11-12 09:20:40,238 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-11-12 09:20:40,238 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager stopped
2012-11-12 09:21:41,681 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
2012-11-12 09:21:41,683 INFO node.FlumeNode: Flume node starting - agent
2012-11-12 09:21:41,688 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
2012-11-12 09:21:41,688 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
2012-11-12 09:21:41,690 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
2012-11-12 09:21:41,691 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager started
2012-11-12 09:21:41,692 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider started
2012-11-12 09:21:41,693 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:/etc/flume-ng/conf/flume.conf for changes
2012-11-12 09:21:41,693 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:/etc/flume-ng/conf/flume.conf
2012-11-12 09:21:41,698 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,698 DEBUG conf.FlumeConfiguration: Created context for tx-es-sink: cluster
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for tx-hdfs-sink: hdfs.fileType
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for web-hdfs-sink: hdfs.rollCount
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for dev-es-sink: type
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for dev-hdfs-sink: hdfs.path
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 DEBUG conf.FlumeConfiguration: Created context for web-es-sink: typeName
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Added sinks: tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink web-es-sink web-hdfs-sink Agent: agent
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,701
Thank you for your responses!
I already mentioned that I am running multiple flow from flume configuration file. The configuration file is working fine with the apache release “1.2”. But when I use this configuration file with CDH4 release it doesn’t work. I don’t understand this behavior and there is very less information in flume.log file to identify problem. It seems to me that the agent is stuck somewhere. Below is my log file content:
2012-11-12 09:20:40,237 INFO node.FlumeNode: Flume node stopping - agent
2012-11-12 09:20:40,237 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-11-12 09:20:40,238 INFO properties.PropertiesFileConfigurationProvider: Configuration provider stopping
2012-11-12 09:20:40,238 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider stopped
2012-11-12 09:20:40,238 INFO nodemanager.DefaultLogicalNodeManager: Node manager stopping
2012-11-12 09:20:40,238 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-11-12 09:20:40,238 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager stopped
2012-11-12 09:21:41,681 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
2012-11-12 09:21:41,683 INFO node.FlumeNode: Flume node starting - agent
2012-11-12 09:21:41,688 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
2012-11-12 09:21:41,688 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
2012-11-12 09:21:41,690 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
2012-11-12 09:21:41,691 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager started
2012-11-12 09:21:41,692 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider started
2012-11-12 09:21:41,693 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:/etc/flume-ng/conf/flume.conf for changes
2012-11-12 09:21:41,693 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:/etc/flume-ng/conf/flume.conf
2012-11-12 09:21:41,698 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,698 DEBUG conf.FlumeConfiguration: Created context for tx-es-sink: cluster
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for tx-hdfs-sink: hdfs.fileType
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for web-hdfs-sink: hdfs.rollCount
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for dev-es-sink: type
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for dev-hdfs-sink: hdfs.path
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 DEBUG conf.FlumeConfiguration: Created context for web-es-sink: typeName
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Added sinks: tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink web-es-sink web-hdfs-sink Agent: agent
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,701
-
RE: multiple agentsAshutoshsharma 2012-11-12, 02:07
Hi All,
Thank you for your responses!
I already mentioned that I am running multiple flow from flume configuration file. The configuration file is working fine with the apache release “1.2”. But when I use this configuration file with CDH4 release it doesn’t work. I don’t understand this behavior and there is very less information in flume.log file to identify problem. It seems to me that the agent is stuck somewhere. Below is my log file content:
2012-11-12 09:20:40,237 INFO node.FlumeNode: Flume node stopping - agent
2012-11-12 09:20:40,237 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-11-12 09:20:40,238 INFO properties.PropertiesFileConfigurationProvider: Configuration provider stopping
2012-11-12 09:20:40,238 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider stopped
2012-11-12 09:20:40,238 INFO nodemanager.DefaultLogicalNodeManager: Node manager stopping
2012-11-12 09:20:40,238 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-11-12 09:20:40,238 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager stopped
2012-11-12 09:21:41,681 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
2012-11-12 09:21:41,683 INFO node.FlumeNode: Flume node starting - agent
2012-11-12 09:21:41,688 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
2012-11-12 09:21:41,688 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
2012-11-12 09:21:41,690 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
2012-11-12 09:21:41,691 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager started
2012-11-12 09:21:41,692 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider started
2012-11-12 09:21:41,693 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:/etc/flume-ng/conf/flume.conf for changes
2012-11-12 09:21:41,693 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:/etc/flume-ng/conf/flume.conf
2012-11-12 09:21:41,698 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,698 DEBUG conf.FlumeConfiguration: Created context for tx-es-sink: cluster
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for tx-hdfs-sink: hdfs.fileType
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for web-hdfs-sink: hdfs.rollCount
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for dev-es-sink: type
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for dev-hdfs-sink: hdfs.path
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 DEBUG conf.FlumeConfiguration: Created context for web-es-sink: typeName
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Added sinks: tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink web-es-sink web-hdfs-sink Agent: agent
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,701
Thank you for your responses!
I already mentioned that I am running multiple flow from flume configuration file. The configuration file is working fine with the apache release “1.2”. But when I use this configuration file with CDH4 release it doesn’t work. I don’t understand this behavior and there is very less information in flume.log file to identify problem. It seems to me that the agent is stuck somewhere. Below is my log file content:
2012-11-12 09:20:40,237 INFO node.FlumeNode: Flume node stopping - agent
2012-11-12 09:20:40,237 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-11-12 09:20:40,238 INFO properties.PropertiesFileConfigurationProvider: Configuration provider stopping
2012-11-12 09:20:40,238 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider stopped
2012-11-12 09:20:40,238 INFO nodemanager.DefaultLogicalNodeManager: Node manager stopping
2012-11-12 09:20:40,238 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
2012-11-12 09:20:40,238 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager stopped
2012-11-12 09:21:41,681 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
2012-11-12 09:21:41,683 INFO node.FlumeNode: Flume node starting - agent
2012-11-12 09:21:41,688 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
2012-11-12 09:21:41,688 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
2012-11-12 09:21:41,690 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
2012-11-12 09:21:41,691 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager started
2012-11-12 09:21:41,692 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider started
2012-11-12 09:21:41,693 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:/etc/flume-ng/conf/flume.conf for changes
2012-11-12 09:21:41,693 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:/etc/flume-ng/conf/flume.conf
2012-11-12 09:21:41,698 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,698 DEBUG conf.FlumeConfiguration: Created context for tx-es-sink: cluster
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for tx-hdfs-sink: hdfs.fileType
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for web-hdfs-sink: hdfs.rollCount
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for dev-es-sink: type
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,699 DEBUG conf.FlumeConfiguration: Created context for dev-hdfs-sink: hdfs.path
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,699 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 DEBUG conf.FlumeConfiguration: Created context for web-es-sink: typeName
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Added sinks: tx-es-sink tx-hdfs-sink dev-es-sink dev-hdfs-sink web-es-sink web-hdfs-sink Agent: agent
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-es-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,700 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:dev-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:tx-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-hdfs-sink
2012-11-12 09:21:41,701 INFO conf.FlumeConfiguration: Processing:web-es-sink
2012-11-12 09:21:41,701
