Mohammad Tariq
2011-11-04, 12:39
AD
2011-11-04, 13:19
Mohammad Tariq
2011-11-04, 13:48
AD
2011-11-04, 14:02
Bill Graham
2011-11-05, 00:08
Mohammad Tariq
2011-11-10, 13:13
AD
2011-11-10, 15:00
Mohammad Tariq
2011-11-10, 19:42
Mohammad Tariq
2011-11-10, 19:46
Mohammad Tariq
2011-11-10, 19:53
AD
2011-11-10, 21:38
TARIQ
2011-11-10, 21:51
Eric Yang
2011-11-10, 22:08
Bill Graham
2011-11-10, 22:08
AD
2011-11-10, 22:18
Mohammad Tariq
2011-11-11, 07:26
Mohammad Tariq
2011-11-11, 13:03
Mohammad Tariq
2011-11-11, 13:15
Eric Yang
2011-11-11, 16:45
-
How to begin with ChukwaMohammad Tariq 2011-11-04, 12:39
Hello list,
I am totally new to Chukwa and I have to learn and start using it as soon
soon as possible as a part of my new project. Could any one let me know how
to begin with?? It would be a great favor to me. I know Flume and have used
it already.Also I would like to know how different is Chukwa from Flume??
Many thanks in advance.
Regards,
Mohammad Tariq
I am totally new to Chukwa and I have to learn and start using it as soon
soon as possible as a part of my new project. Could any one let me know how
to begin with?? It would be a great favor to me. I know Flume and have used
it already.Also I would like to know how different is Chukwa from Flume??
Many thanks in advance.
Regards,
Mohammad Tariq
-
Re: How to begin with ChukwaAD 2011-11-04, 13:19
I think it helps to read he wiki and understand the basic concepts, like
agents and collectors. What kind of data do you want to parse? I have
used both Flume and Chukwa myself and can say for sure Chukwa is miles
easier to work with and configure. Flume may have some more advanced
features, but so far i found Chukwa both easy to extend from a development
standpoint, and much easier to configure.
Have you read the wiki pages ?
On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello list,
>
> I am totally new to Chukwa and I have to learn and start using it as
> soon soon as possible as a part of my new project. Could any one let me
> know how to begin with?? It would be a great favor to me. I know Flume and
> have used it already.Also I would like to know how different is Chukwa from
> Flume??
> Many thanks in advance.
>
> Regards,
> Mohammad Tariq
>
>
agents and collectors. What kind of data do you want to parse? I have
used both Flume and Chukwa myself and can say for sure Chukwa is miles
easier to work with and configure. Flume may have some more advanced
features, but so far i found Chukwa both easy to extend from a development
standpoint, and much easier to configure.
Have you read the wiki pages ?
On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello list,
>
> I am totally new to Chukwa and I have to learn and start using it as
> soon soon as possible as a part of my new project. Could any one let me
> know how to begin with?? It would be a great favor to me. I know Flume and
> have used it already.Also I would like to know how different is Chukwa from
> Flume??
> Many thanks in advance.
>
> Regards,
> Mohammad Tariq
>
>
-
Re: How to begin with ChukwaMohammad Tariq 2011-11-04, 13:48
Hello AD,
Yes, I have read the wiki and right now I am trying to set up a
Chukwa cluster using the wiki itself. Are there any other links you
are aware of that can help me learn and use Chukwa in a better
fashion?? I am trying to use Chukwa for data in "The Wellsite
Information Transfer Standard Markup Language (WITSML)" format.
Regards,
Mohammad Tariq
On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
>
> I think it helps to read he wiki and understand the basic concepts, like agents and collectors. What kind of data do you want to parse? I have used both Flume and Chukwa myself and can say for sure Chukwa is miles easier to work with and configure. Flume may have some more advanced features, but so far i found Chukwa both easy to extend from a development standpoint, and much easier to configure.
> Have you read the wiki pages ?
>
> On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>
>> Hello list,
>> I am totally new to Chukwa and I have to learn and start using it as soon soon as possible as a part of my new project. Could any one let me know how to begin with?? It would be a great favor to me. I know Flume and have used it already.Also I would like to know how different is Chukwa from Flume??
>> Many thanks in advance.
>>
>> Regards,
>> Mohammad Tariq
>
Yes, I have read the wiki and right now I am trying to set up a
Chukwa cluster using the wiki itself. Are there any other links you
are aware of that can help me learn and use Chukwa in a better
fashion?? I am trying to use Chukwa for data in "The Wellsite
Information Transfer Standard Markup Language (WITSML)" format.
Regards,
Mohammad Tariq
On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
>
> I think it helps to read he wiki and understand the basic concepts, like agents and collectors. What kind of data do you want to parse? I have used both Flume and Chukwa myself and can say for sure Chukwa is miles easier to work with and configure. Flume may have some more advanced features, but so far i found Chukwa both easy to extend from a development standpoint, and much easier to configure.
> Have you read the wiki pages ?
>
> On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>
>> Hello list,
>> I am totally new to Chukwa and I have to learn and start using it as soon soon as possible as a part of my new project. Could any one let me know how to begin with?? It would be a great favor to me. I know Flume and have used it already.Also I would like to know how different is Chukwa from Flume??
>> Many thanks in advance.
>>
>> Regards,
>> Mohammad Tariq
>
-
Re: How to begin with ChukwaAD 2011-11-04, 14:02
Are you running the agent and collector on the same machine? Off the cuff
its something like
1 - set your CHUKWA_HOME env variable
2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars look
good
3 - edit conf/initial_adaptors with the adaptors you need. note here that
the dataType
4 - edit conf/collectors and add the hostname/ip of the collectors
5 - edit conf/agents and add ip/hostname of the agents
6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
for your dataType (name from initial_adaptors)
On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
On collector node run bin/chukwa collector , tail
${CHUKWA_LOGS]/collector.log
On collector node run bin/chukwa Demux , tail ${CHUKWA_LOGS}/Demux.log
On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello AD,
> Yes, I have read the wiki and right now I am trying to set up a
> Chukwa cluster using the wiki itself. Are there any other links you
> are aware of that can help me learn and use Chukwa in a better
> fashion?? I am trying to use Chukwa for data in "The Wellsite
> Information Transfer Standard Markup Language (WITSML)" format.
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
> >
> > I think it helps to read he wiki and understand the basic concepts, like
> agents and collectors. What kind of data do you want to parse? I have
> used both Flume and Chukwa myself and can say for sure Chukwa is miles
> easier to work with and configure. Flume may have some more advanced
> features, but so far i found Chukwa both easy to extend from a development
> standpoint, and much easier to configure.
> > Have you read the wiki pages ?
> >
> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >>
> >> Hello list,
> >> I am totally new to Chukwa and I have to learn and start using it as
> soon soon as possible as a part of my new project. Could any one let me
> know how to begin with?? It would be a great favor to me. I know Flume and
> have used it already.Also I would like to know how different is Chukwa from
> Flume??
> >> Many thanks in advance.
> >>
> >> Regards,
> >> Mohammad Tariq
> >
>
its something like
1 - set your CHUKWA_HOME env variable
2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars look
good
3 - edit conf/initial_adaptors with the adaptors you need. note here that
the dataType
4 - edit conf/collectors and add the hostname/ip of the collectors
5 - edit conf/agents and add ip/hostname of the agents
6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
for your dataType (name from initial_adaptors)
On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
On collector node run bin/chukwa collector , tail
${CHUKWA_LOGS]/collector.log
On collector node run bin/chukwa Demux , tail ${CHUKWA_LOGS}/Demux.log
On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello AD,
> Yes, I have read the wiki and right now I am trying to set up a
> Chukwa cluster using the wiki itself. Are there any other links you
> are aware of that can help me learn and use Chukwa in a better
> fashion?? I am trying to use Chukwa for data in "The Wellsite
> Information Transfer Standard Markup Language (WITSML)" format.
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
> >
> > I think it helps to read he wiki and understand the basic concepts, like
> agents and collectors. What kind of data do you want to parse? I have
> used both Flume and Chukwa myself and can say for sure Chukwa is miles
> easier to work with and configure. Flume may have some more advanced
> features, but so far i found Chukwa both easy to extend from a development
> standpoint, and much easier to configure.
> > Have you read the wiki pages ?
> >
> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >>
> >> Hello list,
> >> I am totally new to Chukwa and I have to learn and start using it as
> soon soon as possible as a part of my new project. Could any one let me
> know how to begin with?? It would be a great favor to me. I know Flume and
> have used it already.Also I would like to know how different is Chukwa from
> Flume??
> >> Many thanks in advance.
> >>
> >> Regards,
> >> Mohammad Tariq
> >
>
-
Re: How to begin with ChukwaBill Graham 2011-11-05, 00:08
One difference between Chukwa and Flume is that in Chukwa you configure
each component individually to know how to handle different datatypes. This
makes it fairly straight-forward to set up, but can be painful to manage
for very large installs (so I've heard). Flume centralized the configs of
"flows" which helps when managing larger installs, at the cost of
additional complexity.
On Fri, Nov 4, 2011 at 7:02 AM, AD <[EMAIL PROTECTED]> wrote:
> Are you running the agent and collector on the same machine? Off the cuff
> its something like
>
> 1 - set your CHUKWA_HOME env variable
> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars look
> good
> 3 - edit conf/initial_adaptors with the adaptors you need. note here that
> the dataType
> 4 - edit conf/collectors and add the hostname/ip of the collectors
> 5 - edit conf/agents and add ip/hostname of the agents
> 6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
> for your dataType (name from initial_adaptors)
>
> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
>
> On collector node run bin/chukwa collector , tail
> ${CHUKWA_LOGS]/collector.log
>
> On collector node run bin/chukwa Demux , tail ${CHUKWA_LOGS}/Demux.log
>
>
>
> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>
>> Hello AD,
>> Yes, I have read the wiki and right now I am trying to set up a
>> Chukwa cluster using the wiki itself. Are there any other links you
>> are aware of that can help me learn and use Chukwa in a better
>> fashion?? I am trying to use Chukwa for data in "The Wellsite
>> Information Transfer Standard Markup Language (WITSML)" format.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
>> >
>> > I think it helps to read he wiki and understand the basic concepts,
>> like agents and collectors. What kind of data do you want to parse? I
>> have used both Flume and Chukwa myself and can say for sure Chukwa is miles
>> easier to work with and configure. Flume may have some more advanced
>> features, but so far i found Chukwa both easy to extend from a development
>> standpoint, and much easier to configure.
>> > Have you read the wiki pages ?
>> >
>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> Hello list,
>> >> I am totally new to Chukwa and I have to learn and start using it as
>> soon soon as possible as a part of my new project. Could any one let me
>> know how to begin with?? It would be a great favor to me. I know Flume and
>> have used it already.Also I would like to know how different is Chukwa from
>> Flume??
>> >> Many thanks in advance.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >
>>
>
>
each component individually to know how to handle different datatypes. This
makes it fairly straight-forward to set up, but can be painful to manage
for very large installs (so I've heard). Flume centralized the configs of
"flows" which helps when managing larger installs, at the cost of
additional complexity.
On Fri, Nov 4, 2011 at 7:02 AM, AD <[EMAIL PROTECTED]> wrote:
> Are you running the agent and collector on the same machine? Off the cuff
> its something like
>
> 1 - set your CHUKWA_HOME env variable
> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars look
> good
> 3 - edit conf/initial_adaptors with the adaptors you need. note here that
> the dataType
> 4 - edit conf/collectors and add the hostname/ip of the collectors
> 5 - edit conf/agents and add ip/hostname of the agents
> 6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
> for your dataType (name from initial_adaptors)
>
> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
>
> On collector node run bin/chukwa collector , tail
> ${CHUKWA_LOGS]/collector.log
>
> On collector node run bin/chukwa Demux , tail ${CHUKWA_LOGS}/Demux.log
>
>
>
> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>
>> Hello AD,
>> Yes, I have read the wiki and right now I am trying to set up a
>> Chukwa cluster using the wiki itself. Are there any other links you
>> are aware of that can help me learn and use Chukwa in a better
>> fashion?? I am trying to use Chukwa for data in "The Wellsite
>> Information Transfer Standard Markup Language (WITSML)" format.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
>> >
>> > I think it helps to read he wiki and understand the basic concepts,
>> like agents and collectors. What kind of data do you want to parse? I
>> have used both Flume and Chukwa myself and can say for sure Chukwa is miles
>> easier to work with and configure. Flume may have some more advanced
>> features, but so far i found Chukwa both easy to extend from a development
>> standpoint, and much easier to configure.
>> > Have you read the wiki pages ?
>> >
>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> Hello list,
>> >> I am totally new to Chukwa and I have to learn and start using it as
>> soon soon as possible as a part of my new project. Could any one let me
>> know how to begin with?? It would be a great favor to me. I know Flume and
>> have used it already.Also I would like to know how different is Chukwa from
>> Flume??
>> >> Many thanks in advance.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >
>>
>
>
-
Re: How to begin with ChukwaMohammad Tariq 2011-11-10, 13:13
Hello AD,
I am sorry for leaving the discussion in between, I was busy with
my exams.Now I am back and need your help.I started with what you have
suggested and read about the concepts (agents and collectors etc). Now
I am trying to collect the data from a file on the local file system
and trying to dump it into the HDFS and I am running the agent and
collector on the same machine.I was able to start the agent but when I
tried to start the collector I got the following line at the terminal
and the terminal got stuck there itself -
tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO: Logging
to STDERR via org.mortbay.log.StdErrLog
2011-11-10 18:42:09.544::INFO: jetty-6.1.11
When I issued jps, I could see only the agent.
Regards,
Mohammad Tariq
On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> One difference between Chukwa and Flume is that in Chukwa you configure each
> component individually to know how to handle different datatypes. This makes
> it fairly straight-forward to set up, but can be painful to manage for very
> large installs (so I've heard). Flume centralized the configs of "flows"
> which helps when managing larger installs, at the cost of additional
> complexity.
>
> On Fri, Nov 4, 2011 at 7:02 AM, AD <[EMAIL PROTECTED]> wrote:
>>
>> Are you running the agent and collector on the same machine? Off the cuff
>> its something like
>> 1 - set your CHUKWA_HOME env variable
>> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars look
>> good
>> 3 - edit conf/initial_adaptors with the adaptors you need. note here that
>> the dataType
>> 4 - edit conf/collectors and add the hostname/ip of the collectors
>> 5 - edit conf/agents and add ip/hostname of the agents
>> 6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
>> for your dataType (name from initial_adaptors)
>> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
>> On collector node run bin/chukwa collector , tail
>> ${CHUKWA_LOGS]/collector.log
>> On collector node run bin/chukwa Demux , tail ${CHUKWA_LOGS}/Demux.log
>>
>>
>> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>>
>>> Hello AD,
>>> Yes, I have read the wiki and right now I am trying to set up a
>>> Chukwa cluster using the wiki itself. Are there any other links you
>>> are aware of that can help me learn and use Chukwa in a better
>>> fashion?? I am trying to use Chukwa for data in "The Wellsite
>>> Information Transfer Standard Markup Language (WITSML)" format.
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
>>> >
>>> > I think it helps to read he wiki and understand the basic concepts,
>>> > like agents and collectors. What kind of data do you want to parse? I have
>>> > used both Flume and Chukwa myself and can say for sure Chukwa is miles
>>> > easier to work with and configure. Flume may have some more advanced
>>> > features, but so far i found Chukwa both easy to extend from a development
>>> > standpoint, and much easier to configure.
>>> > Have you read the wiki pages ?
>>> >
>>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> > wrote:
>>> >>
>>> >> Hello list,
>>> >> I am totally new to Chukwa and I have to learn and start using it as
>>> >> soon soon as possible as a part of my new project. Could any one let me know
>>> >> how to begin with?? It would be a great favor to me. I know Flume and have
>>> >> used it already.Also I would like to know how different is Chukwa from
>>> >> Flume??
>>> >> Many thanks in advance.
>>> >>
>>> >> Regards,
>>> >> Mohammad Tariq
>>> >
>>
>
>
I am sorry for leaving the discussion in between, I was busy with
my exams.Now I am back and need your help.I started with what you have
suggested and read about the concepts (agents and collectors etc). Now
I am trying to collect the data from a file on the local file system
and trying to dump it into the HDFS and I am running the agent and
collector on the same machine.I was able to start the agent but when I
tried to start the collector I got the following line at the terminal
and the terminal got stuck there itself -
tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO: Logging
to STDERR via org.mortbay.log.StdErrLog
2011-11-10 18:42:09.544::INFO: jetty-6.1.11
When I issued jps, I could see only the agent.
Regards,
Mohammad Tariq
On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> One difference between Chukwa and Flume is that in Chukwa you configure each
> component individually to know how to handle different datatypes. This makes
> it fairly straight-forward to set up, but can be painful to manage for very
> large installs (so I've heard). Flume centralized the configs of "flows"
> which helps when managing larger installs, at the cost of additional
> complexity.
>
> On Fri, Nov 4, 2011 at 7:02 AM, AD <[EMAIL PROTECTED]> wrote:
>>
>> Are you running the agent and collector on the same machine? Off the cuff
>> its something like
>> 1 - set your CHUKWA_HOME env variable
>> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars look
>> good
>> 3 - edit conf/initial_adaptors with the adaptors you need. note here that
>> the dataType
>> 4 - edit conf/collectors and add the hostname/ip of the collectors
>> 5 - edit conf/agents and add ip/hostname of the agents
>> 6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
>> for your dataType (name from initial_adaptors)
>> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
>> On collector node run bin/chukwa collector , tail
>> ${CHUKWA_LOGS]/collector.log
>> On collector node run bin/chukwa Demux , tail ${CHUKWA_LOGS}/Demux.log
>>
>>
>> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>>
>>> Hello AD,
>>> Yes, I have read the wiki and right now I am trying to set up a
>>> Chukwa cluster using the wiki itself. Are there any other links you
>>> are aware of that can help me learn and use Chukwa in a better
>>> fashion?? I am trying to use Chukwa for data in "The Wellsite
>>> Information Transfer Standard Markup Language (WITSML)" format.
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
>>> >
>>> > I think it helps to read he wiki and understand the basic concepts,
>>> > like agents and collectors. What kind of data do you want to parse? I have
>>> > used both Flume and Chukwa myself and can say for sure Chukwa is miles
>>> > easier to work with and configure. Flume may have some more advanced
>>> > features, but so far i found Chukwa both easy to extend from a development
>>> > standpoint, and much easier to configure.
>>> > Have you read the wiki pages ?
>>> >
>>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]>
>>> > wrote:
>>> >>
>>> >> Hello list,
>>> >> I am totally new to Chukwa and I have to learn and start using it as
>>> >> soon soon as possible as a part of my new project. Could any one let me know
>>> >> how to begin with?? It would be a great favor to me. I know Flume and have
>>> >> used it already.Also I would like to know how different is Chukwa from
>>> >> Flume??
>>> >> Many thanks in advance.
>>> >>
>>> >> Regards,
>>> >> Mohammad Tariq
>>> >
>>
>
>
-
Re: How to begin with ChukwaAD 2011-11-10, 15:00
what happens when you tail the collector.log file?
can you share your initial_adaptors and collectors file in conf ?
On Thu, Nov 10, 2011 at 8:13 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello AD,
> I am sorry for leaving the discussion in between, I was busy with
> my exams.Now I am back and need your help.I started with what you have
> suggested and read about the concepts (agents and collectors etc). Now
> I am trying to collect the data from a file on the local file system
> and trying to dump it into the HDFS and I am running the agent and
> collector on the same machine.I was able to start the agent but when I
> tried to start the collector I got the following line at the terminal
> and the terminal got stuck there itself -
> tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO: Logging
> to STDERR via org.mortbay.log.StdErrLog
> 2011-11-10 18:42:09.544::INFO: jetty-6.1.11
>
> When I issued jps, I could see only the agent.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> > One difference between Chukwa and Flume is that in Chukwa you configure
> each
> > component individually to know how to handle different datatypes. This
> makes
> > it fairly straight-forward to set up, but can be painful to manage for
> very
> > large installs (so I've heard). Flume centralized the configs of "flows"
> > which helps when managing larger installs, at the cost of additional
> > complexity.
> >
> > On Fri, Nov 4, 2011 at 7:02 AM, AD <[EMAIL PROTECTED]> wrote:
> >>
> >> Are you running the agent and collector on the same machine? Off the
> cuff
> >> its something like
> >> 1 - set your CHUKWA_HOME env variable
> >> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars
> look
> >> good
> >> 3 - edit conf/initial_adaptors with the adaptors you need. note here
> that
> >> the dataType
> >> 4 - edit conf/collectors and add the hostname/ip of the collectors
> >> 5 - edit conf/agents and add ip/hostname of the agents
> >> 6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
> >> for your dataType (name from initial_adaptors)
> >> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
> >> On collector node run bin/chukwa collector , tail
> >> ${CHUKWA_LOGS]/collector.log
> >> On collector node run bin/chukwa Demux , tail ${CHUKWA_LOGS}/Demux.log
> >>
> >>
> >> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>> Hello AD,
> >>> Yes, I have read the wiki and right now I am trying to set up a
> >>> Chukwa cluster using the wiki itself. Are there any other links you
> >>> are aware of that can help me learn and use Chukwa in a better
> >>> fashion?? I am trying to use Chukwa for data in "The Wellsite
> >>> Information Transfer Standard Markup Language (WITSML)" format.
> >>>
> >>> Regards,
> >>> Mohammad Tariq
> >>>
> >>>
> >>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
> >>> >
> >>> > I think it helps to read he wiki and understand the basic concepts,
> >>> > like agents and collectors. What kind of data do you want to parse?
> I have
> >>> > used both Flume and Chukwa myself and can say for sure Chukwa is
> miles
> >>> > easier to work with and configure. Flume may have some more advanced
> >>> > features, but so far i found Chukwa both easy to extend from a
> development
> >>> > standpoint, and much easier to configure.
> >>> > Have you read the wiki pages ?
> >>> >
> >>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> >>> > wrote:
> >>> >>
> >>> >> Hello list,
> >>> >> I am totally new to Chukwa and I have to learn and start using it
> as
> >>> >> soon soon as possible as a part of my new project. Could any one
> let me know
> >>> >> how to begin with?? It would be a great favor to me. I know Flume
> and have
> >>> >> used it already.Also I would like to know how different is Chukwa
> from
> >>> >> Flume??
can you share your initial_adaptors and collectors file in conf ?
On Thu, Nov 10, 2011 at 8:13 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello AD,
> I am sorry for leaving the discussion in between, I was busy with
> my exams.Now I am back and need your help.I started with what you have
> suggested and read about the concepts (agents and collectors etc). Now
> I am trying to collect the data from a file on the local file system
> and trying to dump it into the HDFS and I am running the agent and
> collector on the same machine.I was able to start the agent but when I
> tried to start the collector I got the following line at the terminal
> and the terminal got stuck there itself -
> tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO: Logging
> to STDERR via org.mortbay.log.StdErrLog
> 2011-11-10 18:42:09.544::INFO: jetty-6.1.11
>
> When I issued jps, I could see only the agent.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> > One difference between Chukwa and Flume is that in Chukwa you configure
> each
> > component individually to know how to handle different datatypes. This
> makes
> > it fairly straight-forward to set up, but can be painful to manage for
> very
> > large installs (so I've heard). Flume centralized the configs of "flows"
> > which helps when managing larger installs, at the cost of additional
> > complexity.
> >
> > On Fri, Nov 4, 2011 at 7:02 AM, AD <[EMAIL PROTECTED]> wrote:
> >>
> >> Are you running the agent and collector on the same machine? Off the
> cuff
> >> its something like
> >> 1 - set your CHUKWA_HOME env variable
> >> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars
> look
> >> good
> >> 3 - edit conf/initial_adaptors with the adaptors you need. note here
> that
> >> the dataType
> >> 4 - edit conf/collectors and add the hostname/ip of the collectors
> >> 5 - edit conf/agents and add ip/hostname of the agents
> >> 6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
> >> for your dataType (name from initial_adaptors)
> >> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
> >> On collector node run bin/chukwa collector , tail
> >> ${CHUKWA_LOGS]/collector.log
> >> On collector node run bin/chukwa Demux , tail ${CHUKWA_LOGS}/Demux.log
> >>
> >>
> >> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>> Hello AD,
> >>> Yes, I have read the wiki and right now I am trying to set up a
> >>> Chukwa cluster using the wiki itself. Are there any other links you
> >>> are aware of that can help me learn and use Chukwa in a better
> >>> fashion?? I am trying to use Chukwa for data in "The Wellsite
> >>> Information Transfer Standard Markup Language (WITSML)" format.
> >>>
> >>> Regards,
> >>> Mohammad Tariq
> >>>
> >>>
> >>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[EMAIL PROTECTED]> wrote:
> >>> >
> >>> > I think it helps to read he wiki and understand the basic concepts,
> >>> > like agents and collectors. What kind of data do you want to parse?
> I have
> >>> > used both Flume and Chukwa myself and can say for sure Chukwa is
> miles
> >>> > easier to work with and configure. Flume may have some more advanced
> >>> > features, but so far i found Chukwa both easy to extend from a
> development
> >>> > standpoint, and much easier to configure.
> >>> > Have you read the wiki pages ?
> >>> >
> >>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> >>> > wrote:
> >>> >>
> >>> >> Hello list,
> >>> >> I am totally new to Chukwa and I have to learn and start using it
> as
> >>> >> soon soon as possible as a part of my new project. Could any one
> let me know
> >>> >> how to begin with?? It would be a great favor to me. I know Flume
> and have
> >>> >> used it already.Also I would like to know how different is Chukwa
> from
> >>> >> Flume??
-
Re: How to begin with ChukwaMohammad Tariq 2011-11-10, 19:42
Hello AD,
this my initial_adaptors file -
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
60 /usr/bin/iostat -x -k 55 2 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
/bin/df -l 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
/usr/bin/sar -q -r -n ALL 55 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
/usr/bin/top -b -n 1 -c 0
and this is the collectors file -
localhost
I am trying to run both agent and collector on the same machine.
The collector.log file looks like this -
2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
/home/prashant/chukwa-0.4.0/bin/../conf
2011-11-11 01:04:25,434 INFO main root - initing servletCollector
2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
pipelined writers, pipe length is 2
2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is 300000
2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is /chukwa/logs/
2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is
hdfs://localhost:9000/
2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type from
core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
2011-11-11 01:04:25,455 INFO Thread-6 SocketTeeWriter - listen thread started
2011-11-11 01:04:25,593 ERROR main SeqFileWriter - can't connect to
HDFS, trying default file system instead (likely to be local)
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:222)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.configuration.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 29 more
Regards,
Mohammad Tariq
On Thu, Nov 10, 2011 at 8:30 PM, AD <[EMAIL PROTECTED]> wrote:
this my initial_adaptors file -
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
60 /usr/bin/iostat -x -k 55 2 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
/bin/df -l 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
/usr/bin/sar -q -r -n ALL 55 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
/usr/bin/top -b -n 1 -c 0
and this is the collectors file -
localhost
I am trying to run both agent and collector on the same machine.
The collector.log file looks like this -
2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
/home/prashant/chukwa-0.4.0/bin/../conf
2011-11-11 01:04:25,434 INFO main root - initing servletCollector
2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
pipelined writers, pipe length is 2
2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is 300000
2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is /chukwa/logs/
2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is
hdfs://localhost:9000/
2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type from
core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
2011-11-11 01:04:25,455 INFO Thread-6 SocketTeeWriter - listen thread started
2011-11-11 01:04:25,593 ERROR main SeqFileWriter - can't connect to
HDFS, trying default file system instead (likely to be local)
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:222)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.configuration.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 29 more
Regards,
Mohammad Tariq
On Thu, Nov 10, 2011 at 8:30 PM, AD <[EMAIL PROTECTED]> wrote:
-
Re: How to begin with ChukwaMohammad Tariq 2011-11-10, 19:46
This is the output on my terminal when i issue
"prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa agent" -
prashant@ubuntu:~/chukwa-0.4.0$ java.io.IOException: Cannot run
program "/usr/bin/iostat": java.io.IOException: error=2, No such file
or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:328)
at org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
at org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: java.io.IOException: error=2, No such
file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 7 more
java.io.IOException: Cannot run program "/usr/bin/sar":
java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:328)
at org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
at org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: java.io.IOException: error=2, No such
file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 7 more
and when i issue "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa
collector" i get this -
prashant@ubuntu:~/chukwa-0.4.0$ 2011-11-11 01:04:25.851::INFO:
Logging to STDERR via org.mortbay.log.StdErrLog
2011-11-11 01:04:25.894::INFO: jetty-6.1.11
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 1:12 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello AD,
> this my initial_adaptors file -
>
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
> 60 /usr/bin/iostat -x -k 55 2 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
> /bin/df -l 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
> /usr/bin/sar -q -r -n ALL 55 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
> /usr/bin/top -b -n 1 -c 0
>
> and this is the collectors file -
> localhost
>
> I am trying to run both agent and collector on the same machine.
>
> The collector.log file looks like this -
> 2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
> /home/prashant/chukwa-0.4.0/bin/../conf
> 2011-11-11 01:04:25,434 INFO main root - initing servletCollector
> 2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
> pipelined writers, pipe length is 2
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is 300000
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is /chukwa/logs/
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is
> hdfs://localhost:9000/
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2011-11-11 01:04:25,455 INFO Thread-6 SocketTeeWriter - listen thread started
> 2011-11-11 01:04:25,593 ERROR main SeqFileWriter - can't connect to
> HDFS, trying default file system instead (likely to be local)
> java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
> at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
"prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa agent" -
prashant@ubuntu:~/chukwa-0.4.0$ java.io.IOException: Cannot run
program "/usr/bin/iostat": java.io.IOException: error=2, No such file
or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:328)
at org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
at org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: java.io.IOException: error=2, No such
file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 7 more
java.io.IOException: Cannot run program "/usr/bin/sar":
java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:328)
at org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
at org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: java.io.IOException: error=2, No such
file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 7 more
and when i issue "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa
collector" i get this -
prashant@ubuntu:~/chukwa-0.4.0$ 2011-11-11 01:04:25.851::INFO:
Logging to STDERR via org.mortbay.log.StdErrLog
2011-11-11 01:04:25.894::INFO: jetty-6.1.11
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 1:12 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello AD,
> this my initial_adaptors file -
>
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
> 60 /usr/bin/iostat -x -k 55 2 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
> /bin/df -l 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
> /usr/bin/sar -q -r -n ALL 55 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
> /usr/bin/top -b -n 1 -c 0
>
> and this is the collectors file -
> localhost
>
> I am trying to run both agent and collector on the same machine.
>
> The collector.log file looks like this -
> 2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
> /home/prashant/chukwa-0.4.0/bin/../conf
> 2011-11-11 01:04:25,434 INFO main root - initing servletCollector
> 2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
> pipelined writers, pipe length is 2
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is 300000
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is /chukwa/logs/
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is
> hdfs://localhost:9000/
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2011-11-11 01:04:25,455 INFO Thread-6 SocketTeeWriter - listen thread started
> 2011-11-11 01:04:25,593 ERROR main SeqFileWriter - can't connect to
> HDFS, trying default file system instead (likely to be local)
> java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
> at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
-
Re: How to begin with ChukwaMohammad Tariq 2011-11-10, 19:53
Can you share 'configured' Chukwa configuration files that are
necessary in order to run Chukwa agent and collector, if you have any
and if possible for you so that I can observe them and figure out what
I am missing???
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 1:16 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> This is the output on my terminal when i issue
> "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa agent" -
> prashant@ubuntu:~/chukwa-0.4.0$ java.io.IOException: Cannot run
> program "/usr/bin/iostat": java.io.IOException: error=2, No such file
> or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at java.lang.Runtime.exec(Runtime.java:593)
> at java.lang.Runtime.exec(Runtime.java:431)
> at java.lang.Runtime.exec(Runtime.java:328)
> at org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> at org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such
> file or directory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 7 more
> java.io.IOException: Cannot run program "/usr/bin/sar":
> java.io.IOException: error=2, No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at java.lang.Runtime.exec(Runtime.java:593)
> at java.lang.Runtime.exec(Runtime.java:431)
> at java.lang.Runtime.exec(Runtime.java:328)
> at org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> at org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such
> file or directory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 7 more
>
> and when i issue "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa
> collector" i get this -
> prashant@ubuntu:~/chukwa-0.4.0$ 2011-11-11 01:04:25.851::INFO:
> Logging to STDERR via org.mortbay.log.StdErrLog
> 2011-11-11 01:04:25.894::INFO: jetty-6.1.11
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 11, 2011 at 1:12 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> Hello AD,
>> this my initial_adaptors file -
>>
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
>> 60 /usr/bin/iostat -x -k 55 2 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
>> /bin/df -l 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
>> /usr/bin/sar -q -r -n ALL 55 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
>> /usr/bin/top -b -n 1 -c 0
>>
>> and this is the collectors file -
>> localhost
>>
>> I am trying to run both agent and collector on the same machine.
>>
>> The collector.log file looks like this -
>> 2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
>> /home/prashant/chukwa-0.4.0/bin/../conf
>> 2011-11-11 01:04:25,434 INFO main root - initing servletCollector
>> 2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
>> pipelined writers, pipe length is 2
>> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is 300000
>> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is /chukwa/logs/
>> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is
>> hdfs://localhost:9000/
>> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type from
necessary in order to run Chukwa agent and collector, if you have any
and if possible for you so that I can observe them and figure out what
I am missing???
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 1:16 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> This is the output on my terminal when i issue
> "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa agent" -
> prashant@ubuntu:~/chukwa-0.4.0$ java.io.IOException: Cannot run
> program "/usr/bin/iostat": java.io.IOException: error=2, No such file
> or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at java.lang.Runtime.exec(Runtime.java:593)
> at java.lang.Runtime.exec(Runtime.java:431)
> at java.lang.Runtime.exec(Runtime.java:328)
> at org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> at org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such
> file or directory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 7 more
> java.io.IOException: Cannot run program "/usr/bin/sar":
> java.io.IOException: error=2, No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at java.lang.Runtime.exec(Runtime.java:593)
> at java.lang.Runtime.exec(Runtime.java:431)
> at java.lang.Runtime.exec(Runtime.java:328)
> at org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> at org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such
> file or directory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 7 more
>
> and when i issue "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa
> collector" i get this -
> prashant@ubuntu:~/chukwa-0.4.0$ 2011-11-11 01:04:25.851::INFO:
> Logging to STDERR via org.mortbay.log.StdErrLog
> 2011-11-11 01:04:25.894::INFO: jetty-6.1.11
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 11, 2011 at 1:12 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> Hello AD,
>> this my initial_adaptors file -
>>
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
>> 60 /usr/bin/iostat -x -k 55 2 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
>> /bin/df -l 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
>> /usr/bin/sar -q -r -n ALL 55 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
>> /usr/bin/top -b -n 1 -c 0
>>
>> and this is the collectors file -
>> localhost
>>
>> I am trying to run both agent and collector on the same machine.
>>
>> The collector.log file looks like this -
>> 2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
>> /home/prashant/chukwa-0.4.0/bin/../conf
>> 2011-11-11 01:04:25,434 INFO main root - initing servletCollector
>> 2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
>> pipelined writers, pipe length is 2
>> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is 300000
>> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is /chukwa/logs/
>> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is
>> hdfs://localhost:9000/
>> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type from
-
Re: How to begin with ChukwaAD 2011-11-10, 21:38
can you share the collector.log file?
Also are you setting CHUKWA_HOME variable properly ?
the iostat error is a bit concerning.
On Thu, Nov 10, 2011 at 2:53 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Can you share 'configured' Chukwa configuration files that are
> necessary in order to run Chukwa agent and collector, if you have any
> and if possible for you so that I can observe them and figure out what
> I am missing???
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 11, 2011 at 1:16 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> > This is the output on my terminal when i issue
> > "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa agent" -
> > prashant@ubuntu:~/chukwa-0.4.0$ java.io.IOException: Cannot run
> > program "/usr/bin/iostat": java.io.IOException: error=2, No such file
> > or directory
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> > at java.lang.Runtime.exec(Runtime.java:593)
> > at java.lang.Runtime.exec(Runtime.java:431)
> > at java.lang.Runtime.exec(Runtime.java:328)
> > at
> org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> > at
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> > at java.util.TimerThread.mainLoop(Timer.java:512)
> > at java.util.TimerThread.run(Timer.java:462)
> > Caused by: java.io.IOException: java.io.IOException: error=2, No such
> > file or directory
> > at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> > at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> > ... 7 more
> > java.io.IOException: Cannot run program "/usr/bin/sar":
> > java.io.IOException: error=2, No such file or directory
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> > at java.lang.Runtime.exec(Runtime.java:593)
> > at java.lang.Runtime.exec(Runtime.java:431)
> > at java.lang.Runtime.exec(Runtime.java:328)
> > at
> org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> > at
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> > at java.util.TimerThread.mainLoop(Timer.java:512)
> > at java.util.TimerThread.run(Timer.java:462)
> > Caused by: java.io.IOException: java.io.IOException: error=2, No such
> > file or directory
> > at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> > at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> > ... 7 more
> >
> > and when i issue "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa
> > collector" i get this -
> > prashant@ubuntu:~/chukwa-0.4.0$ 2011-11-11 01:04:25.851::INFO:
> > Logging to STDERR via org.mortbay.log.StdErrLog
> > 2011-11-11 01:04:25.894::INFO: jetty-6.1.11
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> >
> > On Fri, Nov 11, 2011 at 1:12 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >> Hello AD,
> >> this my initial_adaptors file -
> >>
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
> >> 60 /usr/bin/iostat -x -k 55 2 0
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
> >> /bin/df -l 0
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
> >> /usr/bin/sar -q -r -n ALL 55 0
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
> >> /usr/bin/top -b -n 1 -c 0
> >>
> >> and this is the collectors file -
> >> localhost
> >>
> >> I am trying to run both agent and collector on the same machine.
> >>
> >> The collector.log file looks like this -
> >> 2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
> >> /home/prashant/chukwa-0.4.0/bin/../conf
> >> 2011-11-11 01:04:25,434 INFO main root - initing servletCollector
> >> 2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
Also are you setting CHUKWA_HOME variable properly ?
the iostat error is a bit concerning.
On Thu, Nov 10, 2011 at 2:53 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Can you share 'configured' Chukwa configuration files that are
> necessary in order to run Chukwa agent and collector, if you have any
> and if possible for you so that I can observe them and figure out what
> I am missing???
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 11, 2011 at 1:16 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> > This is the output on my terminal when i issue
> > "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa agent" -
> > prashant@ubuntu:~/chukwa-0.4.0$ java.io.IOException: Cannot run
> > program "/usr/bin/iostat": java.io.IOException: error=2, No such file
> > or directory
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> > at java.lang.Runtime.exec(Runtime.java:593)
> > at java.lang.Runtime.exec(Runtime.java:431)
> > at java.lang.Runtime.exec(Runtime.java:328)
> > at
> org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> > at
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> > at java.util.TimerThread.mainLoop(Timer.java:512)
> > at java.util.TimerThread.run(Timer.java:462)
> > Caused by: java.io.IOException: java.io.IOException: error=2, No such
> > file or directory
> > at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> > at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> > ... 7 more
> > java.io.IOException: Cannot run program "/usr/bin/sar":
> > java.io.IOException: error=2, No such file or directory
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> > at java.lang.Runtime.exec(Runtime.java:593)
> > at java.lang.Runtime.exec(Runtime.java:431)
> > at java.lang.Runtime.exec(Runtime.java:328)
> > at
> org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> > at
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> > at java.util.TimerThread.mainLoop(Timer.java:512)
> > at java.util.TimerThread.run(Timer.java:462)
> > Caused by: java.io.IOException: java.io.IOException: error=2, No such
> > file or directory
> > at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> > at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> > ... 7 more
> >
> > and when i issue "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa
> > collector" i get this -
> > prashant@ubuntu:~/chukwa-0.4.0$ 2011-11-11 01:04:25.851::INFO:
> > Logging to STDERR via org.mortbay.log.StdErrLog
> > 2011-11-11 01:04:25.894::INFO: jetty-6.1.11
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> >
> > On Fri, Nov 11, 2011 at 1:12 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
> >> Hello AD,
> >> this my initial_adaptors file -
> >>
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
> >> 60 /usr/bin/iostat -x -k 55 2 0
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
> >> /bin/df -l 0
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
> >> /usr/bin/sar -q -r -n ALL 55 0
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
> >> /usr/bin/top -b -n 1 -c 0
> >>
> >> and this is the collectors file -
> >> localhost
> >>
> >> I am trying to run both agent and collector on the same machine.
> >>
> >> The collector.log file looks like this -
> >> 2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
> >> /home/prashant/chukwa-0.4.0/bin/../conf
> >> 2011-11-11 01:04:25,434 INFO main root - initing servletCollector
> >> 2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
-
Re: How to begin with ChukwaTARIQ 2011-11-10, 21:51
Here is the collector.log file -
2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
/home/prashant/chukwa-0.4.0/bin/../conf
2011-11-11 03:20:38,960 INFO main root - initing servletCollector
2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
pipelined writers, pipe length is 2
2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread started
2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is /chukwa/logs/
2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
hdfs://localhost:9000/
2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
HDFS, trying default file system instead (likely to be local)
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:222)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.configuration.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 29 more
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 3:09 AM, AD [via Apache Chukwa]
<ml-node+[EMAIL PROTECTED]> wrote:
View this message in context: http://apache-chukwa.679492.n3.nabble.com/How-to-begin-with-Chukwa-tp3479982p3498068.html
Sent from the Chukwa - Users mailing list archive at Nabble.com.
2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
/home/prashant/chukwa-0.4.0/bin/../conf
2011-11-11 03:20:38,960 INFO main root - initing servletCollector
2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
pipelined writers, pipe length is 2
2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread started
2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is /chukwa/logs/
2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
hdfs://localhost:9000/
2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
HDFS, trying default file system instead (likely to be local)
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:222)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.configuration.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 29 more
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 3:09 AM, AD [via Apache Chukwa]
<ml-node+[EMAIL PROTECTED]> wrote:
View this message in context: http://apache-chukwa.679492.n3.nabble.com/How-to-begin-with-Chukwa-tp3479982p3498068.html
Sent from the Chukwa - Users mailing list archive at Nabble.com.
-
Re: How to begin with ChukwaEric Yang 2011-11-10, 22:08
There are two possible cause.
1. Make sure chukwa-collector-conf.xml has writer.hdfs.filesystem maps to your hdfs://namenode:8020/chukwa
2. Commons Configuration jar file is not on Chukwa collector class path. This happens when the Hadoop cluster is 0.20.203+, and commons-configuration*.jar should be copied from hadoop lib directory to Chukwa class path.
Hope this helps.
regards,
Eric
On Nov 10, 2011, at 1:51 PM, TARIQ wrote:
> Here is the collector.log file -
>
> 2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
> /home/prashant/chukwa-0.4.0/bin/../conf
> 2011-11-11 03:20:38,960 INFO main root - initing servletCollector
> 2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
> pipelined writers, pipe length is 2
> 2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread started
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is /chukwa/logs/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
> hdfs://localhost:9000/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
> HDFS, trying default file system instead (likely to be local)
> java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
> at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
> at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
> at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
> at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
> at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
> at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
> at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
> at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
> at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
> at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
> at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> at org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
1. Make sure chukwa-collector-conf.xml has writer.hdfs.filesystem maps to your hdfs://namenode:8020/chukwa
2. Commons Configuration jar file is not on Chukwa collector class path. This happens when the Hadoop cluster is 0.20.203+, and commons-configuration*.jar should be copied from hadoop lib directory to Chukwa class path.
Hope this helps.
regards,
Eric
On Nov 10, 2011, at 1:51 PM, TARIQ wrote:
> Here is the collector.log file -
>
> 2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
> /home/prashant/chukwa-0.4.0/bin/../conf
> 2011-11-11 03:20:38,960 INFO main root - initing servletCollector
> 2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
> pipelined writers, pipe length is 2
> 2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread started
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is /chukwa/logs/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
> hdfs://localhost:9000/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
> HDFS, trying default file system instead (likely to be local)
> java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
> at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
> at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
> at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
> at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
> at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
> at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
> at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
> at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
> at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
> at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
> at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> at org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
-
Re: How to begin with ChukwaBill Graham 2011-11-10, 22:08
Unfortunately, the conf/collectors config file is used in different ways
(with different expected formats) by the agent and the collector. See this
discussion:
http://lucene.472066.n3.nabble.com/Chukwa-setup-issues-tt2764763.html#a2765143
If you try to run both processes frmo the same configs, you'll run into
issues.
On Thu, Nov 10, 2011 at 1:51 PM, TARIQ <[EMAIL PROTECTED]> wrote:
> Here is the collector.log file -
>
> 2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
> /home/prashant/chukwa-0.4.0/bin/../conf
> 2011-11-11 03:20:38,960 INFO main root - initing servletCollector
> 2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
> pipelined writers, pipe length is 2
> 2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread
> started
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is
> /chukwa/logs/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
> hdfs://localhost:9000/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
> HDFS, trying default file system instead (likely to be local)
> java.lang.NoClassDefFoundError:
> org/apache/commons/configuration/Configuration
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
>
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
>
> at
> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
>
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>
> at
> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
>
> at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> at
> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
>
> at
> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
>
> at
> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
>
> at
> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at
> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at
> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
>
> at
> org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at
> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
(with different expected formats) by the agent and the collector. See this
discussion:
http://lucene.472066.n3.nabble.com/Chukwa-setup-issues-tt2764763.html#a2765143
If you try to run both processes frmo the same configs, you'll run into
issues.
On Thu, Nov 10, 2011 at 1:51 PM, TARIQ <[EMAIL PROTECTED]> wrote:
> Here is the collector.log file -
>
> 2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
> /home/prashant/chukwa-0.4.0/bin/../conf
> 2011-11-11 03:20:38,960 INFO main root - initing servletCollector
> 2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
> pipelined writers, pipe length is 2
> 2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread
> started
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is
> /chukwa/logs/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
> hdfs://localhost:9000/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
> HDFS, trying default file system instead (likely to be local)
> java.lang.NoClassDefFoundError:
> org/apache/commons/configuration/Configuration
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
>
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
>
> at
> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
>
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>
> at
> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
>
> at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> at
> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
>
> at
> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
>
> at
> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
>
> at
> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at
> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at
> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
>
> at
> org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at
> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
-
Re: How to begin with ChukwaAD 2011-11-10, 22:18
do you have a /chukwa/logs directory? If not you may want to update in
collector-conf.xml
<property>
<name>chukwaCollector.outputDir</name>
<value>/chukwa/logs/</value>
<description>Chukwa data sink directory</description>
</property>
Also try commenting out
<property>
<name>writer.hdfs.filesystem</name>
<value>@TODO-COLLECTORS-NAMENODE@</value>
<description>HDFS to dump to</description>
</property>
To avoid the HDFS error.
On Thu, Nov 10, 2011 at 4:51 PM, TARIQ <[EMAIL PROTECTED]> wrote:
> Here is the collector.log file -
>
> 2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
> /home/prashant/chukwa-0.4.0/bin/../conf
> 2011-11-11 03:20:38,960 INFO main root - initing servletCollector
> 2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
> pipelined writers, pipe length is 2
> 2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread
> started
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is
> /chukwa/logs/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
> hdfs://localhost:9000/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
> HDFS, trying default file system instead (likely to be local)
> java.lang.NoClassDefFoundError:
> org/apache/commons/configuration/Configuration
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
>
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
>
> at
> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
>
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>
> at
> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
>
> at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> at
> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
>
> at
> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
>
> at
> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
>
> at
> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at
> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at
> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
>
> at
> org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at
> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
collector-conf.xml
<property>
<name>chukwaCollector.outputDir</name>
<value>/chukwa/logs/</value>
<description>Chukwa data sink directory</description>
</property>
Also try commenting out
<property>
<name>writer.hdfs.filesystem</name>
<value>@TODO-COLLECTORS-NAMENODE@</value>
<description>HDFS to dump to</description>
</property>
To avoid the HDFS error.
On Thu, Nov 10, 2011 at 4:51 PM, TARIQ <[EMAIL PROTECTED]> wrote:
> Here is the collector.log file -
>
> 2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
> /home/prashant/chukwa-0.4.0/bin/../conf
> 2011-11-11 03:20:38,960 INFO main root - initing servletCollector
> 2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
> pipelined writers, pipe length is 2
> 2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread
> started
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is
> /chukwa/logs/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
> hdfs://localhost:9000/
> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
> HDFS, trying default file system instead (likely to be local)
> java.lang.NoClassDefFoundError:
> org/apache/commons/configuration/Configuration
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
>
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
>
> at
> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
>
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>
> at
> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
>
> at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> at
> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
>
> at
> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
>
> at
> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
>
> at
> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at
> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at
> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
>
> at
> org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at
> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
-
Re: How to begin with ChukwaMohammad Tariq 2011-11-11, 07:26
Hi bill,
I have edited my collectors file..it contains following two lines now -
localhost
http://localhost:9999/
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 3:38 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> Unfortunately, the conf/collectors config file is used in different ways
> (with different expected formats) by the agent and the collector. See this
> discussion:
> http://lucene.472066.n3.nabble.com/Chukwa-setup-issues-tt2764763.html#a2765143
> If you try to run both processes frmo the same configs, you'll run into
> issues.
>
> On Thu, Nov 10, 2011 at 1:51 PM, TARIQ <[EMAIL PROTECTED]> wrote:
>>
>> Here is the collector.log file -
>>
>> 2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
>> /home/prashant/chukwa-0.4.0/bin/../conf
>> 2011-11-11 03:20:38,960 INFO main root - initing servletCollector
>> 2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
>> pipelined writers, pipe length is 2
>> 2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread
>> started
>> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
>> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is
>> /chukwa/logs/
>> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
>> hdfs://localhost:9000/
>> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
>> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
>> 2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
>> HDFS, trying default file system instead (likely to be local)
>> java.lang.NoClassDefFoundError:
>> org/apache/commons/configuration/Configuration
>> at
>> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
>> at
>> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
>> at
>> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
>> at
>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
>> at
>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>> at
>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>> at
>> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
>> at
>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
>> at
>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>> at
>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>> at
>> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
>> at
>> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
>> at
>> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
>> at
>> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
>> at
>> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
>> at
>> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
>> at
>> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
>> at
>> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
>> at
>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
>> at
>> org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
>> at
>> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
I have edited my collectors file..it contains following two lines now -
localhost
http://localhost:9999/
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 3:38 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> Unfortunately, the conf/collectors config file is used in different ways
> (with different expected formats) by the agent and the collector. See this
> discussion:
> http://lucene.472066.n3.nabble.com/Chukwa-setup-issues-tt2764763.html#a2765143
> If you try to run both processes frmo the same configs, you'll run into
> issues.
>
> On Thu, Nov 10, 2011 at 1:51 PM, TARIQ <[EMAIL PROTECTED]> wrote:
>>
>> Here is the collector.log file -
>>
>> 2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is
>> /home/prashant/chukwa-0.4.0/bin/../conf
>> 2011-11-11 03:20:38,960 INFO main root - initing servletCollector
>> 2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using
>> pipelined writers, pipe length is 2
>> 2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread
>> started
>> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000
>> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is
>> /chukwa/logs/
>> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is
>> hdfs://localhost:9000/
>> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from
>> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
>> 2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to
>> HDFS, trying default file system instead (likely to be local)
>> java.lang.NoClassDefFoundError:
>> org/apache/commons/configuration/Configuration
>> at
>> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
>> at
>> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
>> at
>> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
>> at
>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
>> at
>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>> at
>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>> at
>> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
>> at
>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
>> at
>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
>> at
>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
>> at
>> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
>> at
>> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
>> at
>> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
>> at
>> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
>> at
>> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
>> at
>> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
>> at
>> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
>> at
>> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
>> at
>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
>> at
>> org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
>> at
>> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
-
Re: How to begin with ChukwaMohammad Tariq 2011-11-11, 13:03
I removed everything and started again from the scratch :
1 - First of all I modified HADOOP_HOME and HADOOP_CONF_DIR -
export HADOOP_HOME="/home/tariq/hadoop-0.20.203.0"
export HADOOP_CONF_DIR="/home/tariq/hadoop-0.20.203.0"
2 - My collectors and agents files have a single line i. e -
localhost
3 - This is the content of my chukwa-collector-conf.xml file -
<property>
<name>writer.hdfs.filesystem</name>
<value>hdfs://localhost:9000/</value>
<description>HDFS to dump to</description>
</property>
<property>
<name>chukwaCollector.outputDir</name>
<value>/chukwa/logs/</value>
<description>Chukwa data sink directory</description>
</property>
<property>
<name>chukwaCollector.rotateInterval</name>
<value>300000</value>
<description>Chukwa rotate interval (ms)</description>
</property>
<property>
<name>chukwaCollector.http.port</name>
<value>8080</value>
<description>The HTTP port number the collector will listen on</description>
</property>
4 - This is the initial_adaptors file -
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
60 /usr/bin/iostat -x -k 55 2 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
/bin/df -l 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
/usr/bin/sar -q -r -n ALL 55 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
/usr/bin/top -b -n 1 -c 0
5 - This is my chukwa-agent-conf.xml file -
<property>
<name>chukwaAgent.tags</name>
<value>cluster="chukwa"</value>
<description>The cluster's name for this agent</description>
</property>
<property>
<name>chukwaAgent.control.port</name>
<value>9093</value>
<description>The socket port number the agent's control interface
can be contacted at.</description>
</property>
<property>
<name>chukwaAgent.hostname</name>
<value>localhost</value>
<description>The hostname of the agent on this node. Usually
localhost, this is used by the chukwa instrumentation agent-control
interface library</description>
</property>
<property>
<name>chukwaAgent.checkpoint.name</name>
<value>chukwa_agent_checkpoint</value>
<description>the prefix to to prepend to the agent's checkpoint
file(s)</description>
</property>
<property>
<name>chukwaAgent.checkpoint.dir</name>
<value>${CHUKWA_LOG_DIR}/</value>
<description>the location to put the agent's checkpoint
file(s)</description>
</property>
<property>
<name>chukwaAgent.checkpoint.interval</name>
<value>5000</value>
<description>the frequency interval for the agent to do
checkpoints, in milliseconds</description>
</property>
<property>
<name>chukwaAgent.sender.fastRetries</name>
<value>4</value>
<description>the number of post attempts to make to a single
collector, before marking it failed</description>
</property>
<property>
<name>chukwaAgent.collector.retries</name>
<value>144000</value>
<description>the number of attempts to find a working
collector</description>
</property>
<property>
<name>chukwaAgent.collector.retryInterval</name>
<value>20000</value>
<description>the number of milliseconds to wait between searches
for a collector</description>
</property>
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 12:56 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hi bill,
> I have edited my collectors file..it contains following two lines now -
>
> localhost
> http://localhost:9999/
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 11, 2011 at 3:38 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
>> Unfortunately, the conf/collectors config file is used in different ways
>> (with different expected formats) by the agent and the collector. See this
>> discussion:
>> http://lucene.472066.n3.nabble.com/Chukwa-setup-issues-tt2764763.html#a2765143
>> If you try to run both processes frmo the same configs, you'll run into
>> issues.
1 - First of all I modified HADOOP_HOME and HADOOP_CONF_DIR -
export HADOOP_HOME="/home/tariq/hadoop-0.20.203.0"
export HADOOP_CONF_DIR="/home/tariq/hadoop-0.20.203.0"
2 - My collectors and agents files have a single line i. e -
localhost
3 - This is the content of my chukwa-collector-conf.xml file -
<property>
<name>writer.hdfs.filesystem</name>
<value>hdfs://localhost:9000/</value>
<description>HDFS to dump to</description>
</property>
<property>
<name>chukwaCollector.outputDir</name>
<value>/chukwa/logs/</value>
<description>Chukwa data sink directory</description>
</property>
<property>
<name>chukwaCollector.rotateInterval</name>
<value>300000</value>
<description>Chukwa rotate interval (ms)</description>
</property>
<property>
<name>chukwaCollector.http.port</name>
<value>8080</value>
<description>The HTTP port number the collector will listen on</description>
</property>
4 - This is the initial_adaptors file -
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
60 /usr/bin/iostat -x -k 55 2 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
/bin/df -l 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
/usr/bin/sar -q -r -n ALL 55 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
/usr/bin/top -b -n 1 -c 0
5 - This is my chukwa-agent-conf.xml file -
<property>
<name>chukwaAgent.tags</name>
<value>cluster="chukwa"</value>
<description>The cluster's name for this agent</description>
</property>
<property>
<name>chukwaAgent.control.port</name>
<value>9093</value>
<description>The socket port number the agent's control interface
can be contacted at.</description>
</property>
<property>
<name>chukwaAgent.hostname</name>
<value>localhost</value>
<description>The hostname of the agent on this node. Usually
localhost, this is used by the chukwa instrumentation agent-control
interface library</description>
</property>
<property>
<name>chukwaAgent.checkpoint.name</name>
<value>chukwa_agent_checkpoint</value>
<description>the prefix to to prepend to the agent's checkpoint
file(s)</description>
</property>
<property>
<name>chukwaAgent.checkpoint.dir</name>
<value>${CHUKWA_LOG_DIR}/</value>
<description>the location to put the agent's checkpoint
file(s)</description>
</property>
<property>
<name>chukwaAgent.checkpoint.interval</name>
<value>5000</value>
<description>the frequency interval for the agent to do
checkpoints, in milliseconds</description>
</property>
<property>
<name>chukwaAgent.sender.fastRetries</name>
<value>4</value>
<description>the number of post attempts to make to a single
collector, before marking it failed</description>
</property>
<property>
<name>chukwaAgent.collector.retries</name>
<value>144000</value>
<description>the number of attempts to find a working
collector</description>
</property>
<property>
<name>chukwaAgent.collector.retryInterval</name>
<value>20000</value>
<description>the number of milliseconds to wait between searches
for a collector</description>
</property>
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 12:56 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hi bill,
> I have edited my collectors file..it contains following two lines now -
>
> localhost
> http://localhost:9999/
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 11, 2011 at 3:38 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
>> Unfortunately, the conf/collectors config file is used in different ways
>> (with different expected formats) by the agent and the collector. See this
>> discussion:
>> http://lucene.472066.n3.nabble.com/Chukwa-setup-issues-tt2764763.html#a2765143
>> If you try to run both processes frmo the same configs, you'll run into
>> issues.
-
Re: How to begin with ChukwaMohammad Tariq 2011-11-11, 13:15
I strange thing I have noticed is that when I issued " bin/chukwa
collector " I got this message - " $ 2011-11-11 18:40:36.367::INFO:
Logging to STDERR via org.mortbay.log.StdErrLog
2011-11-11 18:40:36.391::INFO: jetty-6.1.11 "
But then I tried " bin/start-collectors.sh " I got this -
localhost: starting collector, logging to
/tmp/chukwa/log/chukwa-chukwa-collector-ubuntu.out
localhost: collector running as process 4570. Stop it first.
localhost: 2011-11-11 16:51:41.485::INFO: Logging to STDERR via
org.mortbay.log.StdErrLog
localhost: 2011-11-11 16:51:41.508::INFO: jetty-6.1.11
This means that collector is already running..but when tried " kill
4570 " it showed this -
bash: kill: (4570) - No such process...I am completely
clueless..Please help me out
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 6:33 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> I removed everything and started again from the scratch :
>
> 1 - First of all I modified HADOOP_HOME and HADOOP_CONF_DIR -
> export HADOOP_HOME="/home/tariq/hadoop-0.20.203.0"
> export HADOOP_CONF_DIR="/home/tariq/hadoop-0.20.203.0"
>
> 2 - My collectors and agents files have a single line i. e -
> localhost
>
> 3 - This is the content of my chukwa-collector-conf.xml file -
> <property>
> <name>writer.hdfs.filesystem</name>
> <value>hdfs://localhost:9000/</value>
> <description>HDFS to dump to</description>
> </property>
>
> <property>
> <name>chukwaCollector.outputDir</name>
> <value>/chukwa/logs/</value>
> <description>Chukwa data sink directory</description>
> </property>
>
> <property>
> <name>chukwaCollector.rotateInterval</name>
> <value>300000</value>
> <description>Chukwa rotate interval (ms)</description>
> </property>
>
> <property>
> <name>chukwaCollector.http.port</name>
> <value>8080</value>
> <description>The HTTP port number the collector will listen on</description>
> </property>
>
> 4 - This is the initial_adaptors file -
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
> 60 /usr/bin/iostat -x -k 55 2 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
> /bin/df -l 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
> /usr/bin/sar -q -r -n ALL 55 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
> /usr/bin/top -b -n 1 -c 0
>
> 5 - This is my chukwa-agent-conf.xml file -
>
> <property>
> <name>chukwaAgent.tags</name>
> <value>cluster="chukwa"</value>
> <description>The cluster's name for this agent</description>
> </property>
>
> <property>
> <name>chukwaAgent.control.port</name>
> <value>9093</value>
> <description>The socket port number the agent's control interface
> can be contacted at.</description>
> </property>
>
> <property>
> <name>chukwaAgent.hostname</name>
> <value>localhost</value>
> <description>The hostname of the agent on this node. Usually
> localhost, this is used by the chukwa instrumentation agent-control
> interface library</description>
> </property>
>
> <property>
> <name>chukwaAgent.checkpoint.name</name>
> <value>chukwa_agent_checkpoint</value>
> <description>the prefix to to prepend to the agent's checkpoint
> file(s)</description>
> </property>
>
> <property>
> <name>chukwaAgent.checkpoint.dir</name>
> <value>${CHUKWA_LOG_DIR}/</value>
> <description>the location to put the agent's checkpoint
> file(s)</description>
> </property>
>
> <property>
> <name>chukwaAgent.checkpoint.interval</name>
> <value>5000</value>
> <description>the frequency interval for the agent to do
> checkpoints, in milliseconds</description>
> </property>
>
> <property>
> <name>chukwaAgent.sender.fastRetries</name>
> <value>4</value>
> <description>the number of post attempts to make to a single
> collector, before marking it failed</description>
> </property>
>
> <property>
> <name>chukwaAgent.collector.retries</name>
> <value>144000</value>
collector " I got this message - " $ 2011-11-11 18:40:36.367::INFO:
Logging to STDERR via org.mortbay.log.StdErrLog
2011-11-11 18:40:36.391::INFO: jetty-6.1.11 "
But then I tried " bin/start-collectors.sh " I got this -
localhost: starting collector, logging to
/tmp/chukwa/log/chukwa-chukwa-collector-ubuntu.out
localhost: collector running as process 4570. Stop it first.
localhost: 2011-11-11 16:51:41.485::INFO: Logging to STDERR via
org.mortbay.log.StdErrLog
localhost: 2011-11-11 16:51:41.508::INFO: jetty-6.1.11
This means that collector is already running..but when tried " kill
4570 " it showed this -
bash: kill: (4570) - No such process...I am completely
clueless..Please help me out
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 6:33 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> I removed everything and started again from the scratch :
>
> 1 - First of all I modified HADOOP_HOME and HADOOP_CONF_DIR -
> export HADOOP_HOME="/home/tariq/hadoop-0.20.203.0"
> export HADOOP_CONF_DIR="/home/tariq/hadoop-0.20.203.0"
>
> 2 - My collectors and agents files have a single line i. e -
> localhost
>
> 3 - This is the content of my chukwa-collector-conf.xml file -
> <property>
> <name>writer.hdfs.filesystem</name>
> <value>hdfs://localhost:9000/</value>
> <description>HDFS to dump to</description>
> </property>
>
> <property>
> <name>chukwaCollector.outputDir</name>
> <value>/chukwa/logs/</value>
> <description>Chukwa data sink directory</description>
> </property>
>
> <property>
> <name>chukwaCollector.rotateInterval</name>
> <value>300000</value>
> <description>Chukwa rotate interval (ms)</description>
> </property>
>
> <property>
> <name>chukwaCollector.http.port</name>
> <value>8080</value>
> <description>The HTTP port number the collector will listen on</description>
> </property>
>
> 4 - This is the initial_adaptors file -
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
> 60 /usr/bin/iostat -x -k 55 2 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
> /bin/df -l 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
> /usr/bin/sar -q -r -n ALL 55 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
> /usr/bin/top -b -n 1 -c 0
>
> 5 - This is my chukwa-agent-conf.xml file -
>
> <property>
> <name>chukwaAgent.tags</name>
> <value>cluster="chukwa"</value>
> <description>The cluster's name for this agent</description>
> </property>
>
> <property>
> <name>chukwaAgent.control.port</name>
> <value>9093</value>
> <description>The socket port number the agent's control interface
> can be contacted at.</description>
> </property>
>
> <property>
> <name>chukwaAgent.hostname</name>
> <value>localhost</value>
> <description>The hostname of the agent on this node. Usually
> localhost, this is used by the chukwa instrumentation agent-control
> interface library</description>
> </property>
>
> <property>
> <name>chukwaAgent.checkpoint.name</name>
> <value>chukwa_agent_checkpoint</value>
> <description>the prefix to to prepend to the agent's checkpoint
> file(s)</description>
> </property>
>
> <property>
> <name>chukwaAgent.checkpoint.dir</name>
> <value>${CHUKWA_LOG_DIR}/</value>
> <description>the location to put the agent's checkpoint
> file(s)</description>
> </property>
>
> <property>
> <name>chukwaAgent.checkpoint.interval</name>
> <value>5000</value>
> <description>the frequency interval for the agent to do
> checkpoints, in milliseconds</description>
> </property>
>
> <property>
> <name>chukwaAgent.sender.fastRetries</name>
> <value>4</value>
> <description>the number of post attempts to make to a single
> collector, before marking it failed</description>
> </property>
>
> <property>
> <name>chukwaAgent.collector.retries</name>
> <value>144000</value>
-
Re: How to begin with ChukwaEric Yang 2011-11-11, 16:45
It looks like a bug in pid file management. run $JAVA_HOME/bin/jps, then kill the running CollectorStub.
regards,
Eric
On Nov 11, 2011, at 5:15 AM, Mohammad Tariq wrote:
> I strange thing I have noticed is that when I issued " bin/chukwa
> collector " I got this message - " $ 2011-11-11 18:40:36.367::INFO:
> Logging to STDERR via org.mortbay.log.StdErrLog
> 2011-11-11 18:40:36.391::INFO: jetty-6.1.11 "
>
> But then I tried " bin/start-collectors.sh " I got this -
> localhost: starting collector, logging to
> /tmp/chukwa/log/chukwa-chukwa-collector-ubuntu.out
> localhost: collector running as process 4570. Stop it first.
> localhost: 2011-11-11 16:51:41.485::INFO: Logging to STDERR via
> org.mortbay.log.StdErrLog
> localhost: 2011-11-11 16:51:41.508::INFO: jetty-6.1.11
>
> This means that collector is already running..but when tried " kill
> 4570 " it showed this -
> bash: kill: (4570) - No such process...I am completely
> clueless..Please help me out
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 11, 2011 at 6:33 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> I removed everything and started again from the scratch :
>>
>> 1 - First of all I modified HADOOP_HOME and HADOOP_CONF_DIR -
>> export HADOOP_HOME="/home/tariq/hadoop-0.20.203.0"
>> export HADOOP_CONF_DIR="/home/tariq/hadoop-0.20.203.0"
>>
>> 2 - My collectors and agents files have a single line i. e -
>> localhost
>>
>> 3 - This is the content of my chukwa-collector-conf.xml file -
>> <property>
>> <name>writer.hdfs.filesystem</name>
>> <value>hdfs://localhost:9000/</value>
>> <description>HDFS to dump to</description>
>> </property>
>>
>> <property>
>> <name>chukwaCollector.outputDir</name>
>> <value>/chukwa/logs/</value>
>> <description>Chukwa data sink directory</description>
>> </property>
>>
>> <property>
>> <name>chukwaCollector.rotateInterval</name>
>> <value>300000</value>
>> <description>Chukwa rotate interval (ms)</description>
>> </property>
>>
>> <property>
>> <name>chukwaCollector.http.port</name>
>> <value>8080</value>
>> <description>The HTTP port number the collector will listen on</description>
>> </property>
>>
>> 4 - This is the initial_adaptors file -
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
>> 60 /usr/bin/iostat -x -k 55 2 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
>> /bin/df -l 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
>> /usr/bin/sar -q -r -n ALL 55 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
>> /usr/bin/top -b -n 1 -c 0
>>
>> 5 - This is my chukwa-agent-conf.xml file -
>>
>> <property>
>> <name>chukwaAgent.tags</name>
>> <value>cluster="chukwa"</value>
>> <description>The cluster's name for this agent</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.control.port</name>
>> <value>9093</value>
>> <description>The socket port number the agent's control interface
>> can be contacted at.</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.hostname</name>
>> <value>localhost</value>
>> <description>The hostname of the agent on this node. Usually
>> localhost, this is used by the chukwa instrumentation agent-control
>> interface library</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.checkpoint.name</name>
>> <value>chukwa_agent_checkpoint</value>
>> <description>the prefix to to prepend to the agent's checkpoint
>> file(s)</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.checkpoint.dir</name>
>> <value>${CHUKWA_LOG_DIR}/</value>
>> <description>the location to put the agent's checkpoint
>> file(s)</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.checkpoint.interval</name>
>> <value>5000</value>
>> <description>the frequency interval for the agent to do
>> checkpoints, in milliseconds</description>
regards,
Eric
On Nov 11, 2011, at 5:15 AM, Mohammad Tariq wrote:
> I strange thing I have noticed is that when I issued " bin/chukwa
> collector " I got this message - " $ 2011-11-11 18:40:36.367::INFO:
> Logging to STDERR via org.mortbay.log.StdErrLog
> 2011-11-11 18:40:36.391::INFO: jetty-6.1.11 "
>
> But then I tried " bin/start-collectors.sh " I got this -
> localhost: starting collector, logging to
> /tmp/chukwa/log/chukwa-chukwa-collector-ubuntu.out
> localhost: collector running as process 4570. Stop it first.
> localhost: 2011-11-11 16:51:41.485::INFO: Logging to STDERR via
> org.mortbay.log.StdErrLog
> localhost: 2011-11-11 16:51:41.508::INFO: jetty-6.1.11
>
> This means that collector is already running..but when tried " kill
> 4570 " it showed this -
> bash: kill: (4570) - No such process...I am completely
> clueless..Please help me out
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 11, 2011 at 6:33 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> I removed everything and started again from the scratch :
>>
>> 1 - First of all I modified HADOOP_HOME and HADOOP_CONF_DIR -
>> export HADOOP_HOME="/home/tariq/hadoop-0.20.203.0"
>> export HADOOP_CONF_DIR="/home/tariq/hadoop-0.20.203.0"
>>
>> 2 - My collectors and agents files have a single line i. e -
>> localhost
>>
>> 3 - This is the content of my chukwa-collector-conf.xml file -
>> <property>
>> <name>writer.hdfs.filesystem</name>
>> <value>hdfs://localhost:9000/</value>
>> <description>HDFS to dump to</description>
>> </property>
>>
>> <property>
>> <name>chukwaCollector.outputDir</name>
>> <value>/chukwa/logs/</value>
>> <description>Chukwa data sink directory</description>
>> </property>
>>
>> <property>
>> <name>chukwaCollector.rotateInterval</name>
>> <value>300000</value>
>> <description>Chukwa rotate interval (ms)</description>
>> </property>
>>
>> <property>
>> <name>chukwaCollector.http.port</name>
>> <value>8080</value>
>> <description>The HTTP port number the collector will listen on</description>
>> </property>
>>
>> 4 - This is the initial_adaptors file -
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
>> 60 /usr/bin/iostat -x -k 55 2 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
>> /bin/df -l 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
>> /usr/bin/sar -q -r -n ALL 55 0
>> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
>> /usr/bin/top -b -n 1 -c 0
>>
>> 5 - This is my chukwa-agent-conf.xml file -
>>
>> <property>
>> <name>chukwaAgent.tags</name>
>> <value>cluster="chukwa"</value>
>> <description>The cluster's name for this agent</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.control.port</name>
>> <value>9093</value>
>> <description>The socket port number the agent's control interface
>> can be contacted at.</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.hostname</name>
>> <value>localhost</value>
>> <description>The hostname of the agent on this node. Usually
>> localhost, this is used by the chukwa instrumentation agent-control
>> interface library</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.checkpoint.name</name>
>> <value>chukwa_agent_checkpoint</value>
>> <description>the prefix to to prepend to the agent's checkpoint
>> file(s)</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.checkpoint.dir</name>
>> <value>${CHUKWA_LOG_DIR}/</value>
>> <description>the location to put the agent's checkpoint
>> file(s)</description>
>> </property>
>>
>> <property>
>> <name>chukwaAgent.checkpoint.interval</name>
>> <value>5000</value>
>> <description>the frequency interval for the agent to do
>> checkpoints, in milliseconds</description>
