Hello Guys,
                     I was previously speaking to Ari and he told me about
the need for more community knowledge of Chukwa. I think contributing an
article about Chukwa to IBM developerWorks would be a great way of informing
the general community about it. IBM developerWorks is a great source of
accurate and relevant information about many subjects that interest the IT
community in general, people have already contributed articles about Hadoop
and Mahout and they've got positive feedback, so I think the time has come
to publish one about Chukwa.

I am willing to co-author this article with anyone who's interested. What do
u think?

--
Ahmed Fathalla

Hi Ahmed,

This is a great idea.  I am interested.  However, Chukwa is going through some major changes in Chukwa 0.5.  The changes are taking a while to be completed.  It may be worth while to wait for the new changes to arrive before writing the article.  What do you think?

Regards,
Eric

On 9/2/10 3:23 AM, "Ahmed Fathalla" <[EMAIL PROTECTED]> wrote:

Hello Guys,
                     I was previously speaking to Ari and he told me about
the need for more community knowledge of Chukwa. I think contributing an
article about Chukwa to IBM developerWorks would be a great way of informing
the general community about it. IBM developerWorks is a great source of
accurate and relevant information about many subjects that interest the IT
community in general, people have already contributed articles about Hadoop
and Mahout and they've got positive feedback, so I think the time has come
to publish one about Chukwa.

I am willing to co-author this article with anyone who's interested. What do
u think?

--
Ahmed Fathalla

Hey Eric,
                    Thanks for your feedback. Publishing an artcile on
developerWorks takes some time, you have to submit an abstract, then if the
abstract is accepted you submit drafts which are reviewed by editors before
the final version is published. The process takes no less than 8 weeks.

So I suggest we get started while working on 0.5. I can start by submitting
an abstract and communicating with the editor, do you agree?

On Mon, Sep 6, 2010 at 5:15 AM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Hi Ahmed,
>
> This is a great idea.  I am interested.  However, Chukwa is going through
> some major changes in Chukwa 0.5.  The changes are taking a while to be
> completed.  It may be worth while to wait for the new changes to arrive
> before writing the article.  What do you think?
>
> Regards,
> Eric
>
> On 9/2/10 3:23 AM, "Ahmed Fathalla" <[EMAIL PROTECTED]> wrote:
>
> Hello Guys,
>                     I was previously speaking to Ari and he told me about
> the need for more community knowledge of Chukwa. I think contributing an
> article about Chukwa to IBM developerWorks would be a great way of
> informing
> the general community about it. IBM developerWorks is a great source of
> accurate and relevant information about many subjects that interest the IT
> community in general, people have already contributed articles about Hadoop
> and Mahout and they've got positive feedback, so I think the time has come
> to publish one about Chukwa.
>
> I am willing to co-author this article with anyone who's interested. What
> do
> u think?
>
> --
> Ahmed Fathalla
>
>
--
Ahmed Fathalla

Chukwa 0.5 is not going to be ready by end of this year.  There are
some major changes which I only work on them on long weekends.  I
think it's best for you to ask questions and you can choose to target
for Chukwa 0.4 or 0.5 on your own discretion.  For Chukwa 0.3, it was
prototyped to load data to HDFS and load materialized result to MySQL.
 For Chukwa 0.4, it is designed to load data to HDFS only.  MySQL path
is no longer supported.  However, there are hacks to make MySQL
components to work with Chukwa 0.4.  Chukwa 0.5 is designed to load
data to HBase for scalability reason.  However, there is a big chunk
of work post data loading for Chukwa 0.5 that are not written.  Hence,
it depends on which version that you are interested in writing
article.  I can provide pointers, but it is definitely going to take
much longer than 8 weeks for Chukwa 0.5.

regards,
Eric

On Sun, Sep 5, 2010 at 8:27 PM, Ahmed Fathalla <[EMAIL PROTECTED]> wrote:
> Hey Eric,
>                    Thanks for your feedback. Publishing an artcile on
> developerWorks takes some time, you have to submit an abstract, then if the
> abstract is accepted you submit drafts which are reviewed by editors before
> the final version is published. The process takes no less than 8 weeks.
>
> So I suggest we get started while working on 0.5. I can start by submitting
> an abstract and communicating with the editor, do you agree?
>
>
>
> On Mon, Sep 6, 2010 at 5:15 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Hi Ahmed,
>>
>> This is a great idea.  I am interested.  However, Chukwa is going through
>> some major changes in Chukwa 0.5.  The changes are taking a while to be
>> completed.  It may be worth while to wait for the new changes to arrive
>> before writing the article.  What do you think?
>>
>> Regards,
>> Eric
>>
>> On 9/2/10 3:23 AM, "Ahmed Fathalla" <[EMAIL PROTECTED]> wrote:
>>
>> Hello Guys,
>>                     I was previously speaking to Ari and he told me about
>> the need for more community knowledge of Chukwa. I think contributing an
>> article about Chukwa to IBM developerWorks would be a great way of
>> informing
>> the general community about it. IBM developerWorks is a great source of
>> accurate and relevant information about many subjects that interest the IT
>> community in general, people have already contributed articles about Hadoop
>> and Mahout and they've got positive feedback, so I think the time has come
>> to publish one about Chukwa.
>>
>> I am willing to co-author this article with anyone who's interested. What
>> do
>> u think?
>>
>> --
>> Ahmed Fathalla
>>
>>
>
>
> --
> Ahmed Fathalla
>

My plan for the article is to contain an introduction about the importance
of log processing frameworks like Chukwa, Scribe and Flume. Then provide a
description of Chukwa's architecture and finally a section on installing and
configuring Chukwa.

I can base the article on 0.4 and provide pointers about areas where users
can expect changes in 0.5 and later.

Also, you said MySQL is not supported in 0.4, does this mean that HICC
doesn't work for 0.4 out of the box?

On Tue, Sep 7, 2010 at 12:33 AM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Chukwa 0.5 is not going to be ready by end of this year.  There are
> some major changes which I only work on them on long weekends.  I
> think it's best for you to ask questions and you can choose to target
> for Chukwa 0.4 or 0.5 on your own discretion.  For Chukwa 0.3, it was
> prototyped to load data to HDFS and load materialized result to MySQL.
>  For Chukwa 0.4, it is designed to load data to HDFS only.  MySQL path
> is no longer supported.  However, there are hacks to make MySQL
> components to work with Chukwa 0.4.  Chukwa 0.5 is designed to load
> data to HBase for scalability reason.  However, there is a big chunk
> of work post data loading for Chukwa 0.5 that are not written.  Hence,
> it depends on which version that you are interested in writing
> article.  I can provide pointers, but it is definitely going to take
> much longer than 8 weeks for Chukwa 0.5.
>
> regards,
> Eric
>
> On Sun, Sep 5, 2010 at 8:27 PM, Ahmed Fathalla <[EMAIL PROTECTED]>
> wrote:
> > Hey Eric,
> >                    Thanks for your feedback. Publishing an artcile on
> > developerWorks takes some time, you have to submit an abstract, then if
> the
> > abstract is accepted you submit drafts which are reviewed by editors
> before
> > the final version is published. The process takes no less than 8 weeks.
> >
> > So I suggest we get started while working on 0.5. I can start by
> submitting
> > an abstract and communicating with the editor, do you agree?
> >
> >
> >
> > On Mon, Sep 6, 2010 at 5:15 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Ahmed,
> >>
> >> This is a great idea.  I am interested.  However, Chukwa is going
> through
> >> some major changes in Chukwa 0.5.  The changes are taking a while to be
> >> completed.  It may be worth while to wait for the new changes to arrive
> >> before writing the article.  What do you think?
> >>
> >> Regards,
> >> Eric
> >>
> >> On 9/2/10 3:23 AM, "Ahmed Fathalla" <[EMAIL PROTECTED]> wrote:
> >>
> >> Hello Guys,
> >>                     I was previously speaking to Ari and he told me
> about
> >> the need for more community knowledge of Chukwa. I think contributing an
> >> article about Chukwa to IBM developerWorks would be a great way of
> >> informing
> >> the general community about it. IBM developerWorks is a great source of
> >> accurate and relevant information about many subjects that interest the
> IT
> >> community in general, people have already contributed articles about
> Hadoop
> >> and Mahout and they've got positive feedback, so I think the time has
> come
> >> to publish one about Chukwa.
> >>
> >> I am willing to co-author this article with anyone who's interested.
> What
> >> do
> >> u think?
> >>
> >> --
> >> Ahmed Fathalla
> >>
> >>
> >
> >
> > --
> > Ahmed Fathalla
> >
>

--
Ahmed Fathalla

Sounds good.  You are correct.  HICC does not work out of the box for 0.4.

regards,
Eric

On Mon, Sep 6, 2010 at 3:23 PM, Ahmed Fathalla <[EMAIL PROTECTED]> wrote:
> My plan for the article is to contain an introduction about the importance
> of log processing frameworks like Chukwa, Scribe and Flume. Then provide a
> description of Chukwa's architecture and finally a section on installing and
> configuring Chukwa.
>
> I can base the article on 0.4 and provide pointers about areas where users
> can expect changes in 0.5 and later.
>
> Also, you said MySQL is not supported in 0.4, does this mean that HICC
> doesn't work for 0.4 out of the box?
>
>
>
> On Tue, Sep 7, 2010 at 12:33 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Chukwa 0.5 is not going to be ready by end of this year.  There are
>> some major changes which I only work on them on long weekends.  I
>> think it's best for you to ask questions and you can choose to target
>> for Chukwa 0.4 or 0.5 on your own discretion.  For Chukwa 0.3, it was
>> prototyped to load data to HDFS and load materialized result to MySQL.
>>  For Chukwa 0.4, it is designed to load data to HDFS only.  MySQL path
>> is no longer supported.  However, there are hacks to make MySQL
>> components to work with Chukwa 0.4.  Chukwa 0.5 is designed to load
>> data to HBase for scalability reason.  However, there is a big chunk
>> of work post data loading for Chukwa 0.5 that are not written.  Hence,
>> it depends on which version that you are interested in writing
>> article.  I can provide pointers, but it is definitely going to take
>> much longer than 8 weeks for Chukwa 0.5.
>>
>> regards,
>> Eric
>>
>> On Sun, Sep 5, 2010 at 8:27 PM, Ahmed Fathalla <[EMAIL PROTECTED]>
>> wrote:
>> > Hey Eric,
>> >                    Thanks for your feedback. Publishing an artcile on
>> > developerWorks takes some time, you have to submit an abstract, then if
>> the
>> > abstract is accepted you submit drafts which are reviewed by editors
>> before
>> > the final version is published. The process takes no less than 8 weeks.
>> >
>> > So I suggest we get started while working on 0.5. I can start by
>> submitting
>> > an abstract and communicating with the editor, do you agree?
>> >
>> >
>> >
>> > On Mon, Sep 6, 2010 at 5:15 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>> >
>> >> Hi Ahmed,
>> >>
>> >> This is a great idea.  I am interested.  However, Chukwa is going
>> through
>> >> some major changes in Chukwa 0.5.  The changes are taking a while to be
>> >> completed.  It may be worth while to wait for the new changes to arrive
>> >> before writing the article.  What do you think?
>> >>
>> >> Regards,
>> >> Eric
>> >>
>> >> On 9/2/10 3:23 AM, "Ahmed Fathalla" <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hello Guys,
>> >>                     I was previously speaking to Ari and he told me
>> about
>> >> the need for more community knowledge of Chukwa. I think contributing an
>> >> article about Chukwa to IBM developerWorks would be a great way of
>> >> informing
>> >> the general community about it. IBM developerWorks is a great source of
>> >> accurate and relevant information about many subjects that interest the
>> IT
>> >> community in general, people have already contributed articles about
>> Hadoop
>> >> and Mahout and they've got positive feedback, so I think the time has
>> come
>> >> to publish one about Chukwa.
>> >>
>> >> I am willing to co-author this article with anyone who's interested.
>> What
>> >> do
>> >> u think?
>> >>
>> >> --
>> >> Ahmed Fathalla
>> >>
>> >>
>> >
>> >
>> > --
>> > Ahmed Fathalla
>> >
>>
>
>
>
> --
> Ahmed Fathalla
>

Hello,
           I've been doing some thinking about this issue. I am very
inclined to switch the focus of this initial article from Chukwa
installaiton and administration to an overview of different distributed log
collection frameworks (Chukwa, Flume, Scirbe,Honu) with a discussion of each
one's architecture and design goals. I think this kind of article would be
much more useful for the IT community at large and given the significant
changes Chukwa is ongoing at the moment, an article or series of articles
specifically about Chukwa can be written once 0.5 is released.

What are your thoughts?

On Tue, Sep 7, 2010 at 5:00 AM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Sounds good.  You are correct.  HICC does not work out of the box for 0.4.
>
> regards,
> Eric
>
> On Mon, Sep 6, 2010 at 3:23 PM, Ahmed Fathalla <[EMAIL PROTECTED]>
> wrote:
> > My plan for the article is to contain an introduction about the
> importance
> > of log processing frameworks like Chukwa, Scribe and Flume. Then provide
> a
> > description of Chukwa's architecture and finally a section on installing
> and
> > configuring Chukwa.
> >
> > I can base the article on 0.4 and provide pointers about areas where
> users
> > can expect changes in 0.5 and later.
> >
> > Also, you said MySQL is not supported in 0.4, does this mean that HICC
> > doesn't work for 0.4 out of the box?
> >
> >
> >
> > On Tue, Sep 7, 2010 at 12:33 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> >
> >> Chukwa 0.5 is not going to be ready by end of this year.  There are
> >> some major changes which I only work on them on long weekends.  I
> >> think it's best for you to ask questions and you can choose to target
> >> for Chukwa 0.4 or 0.5 on your own discretion.  For Chukwa 0.3, it was
> >> prototyped to load data to HDFS and load materialized result to MySQL.
> >>  For Chukwa 0.4, it is designed to load data to HDFS only.  MySQL path
> >> is no longer supported.  However, there are hacks to make MySQL
> >> components to work with Chukwa 0.4.  Chukwa 0.5 is designed to load
> >> data to HBase for scalability reason.  However, there is a big chunk
> >> of work post data loading for Chukwa 0.5 that are not written.  Hence,
> >> it depends on which version that you are interested in writing
> >> article.  I can provide pointers, but it is definitely going to take
> >> much longer than 8 weeks for Chukwa 0.5.
> >>
> >> regards,
> >> Eric
> >>
> >> On Sun, Sep 5, 2010 at 8:27 PM, Ahmed Fathalla <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hey Eric,
> >> >                    Thanks for your feedback. Publishing an artcile on
> >> > developerWorks takes some time, you have to submit an abstract, then
> if
> >> the
> >> > abstract is accepted you submit drafts which are reviewed by editors
> >> before
> >> > the final version is published. The process takes no less than 8
> weeks.
> >> >
> >> > So I suggest we get started while working on 0.5. I can start by
> >> submitting
> >> > an abstract and communicating with the editor, do you agree?
> >> >
> >> >
> >> >
> >> > On Mon, Sep 6, 2010 at 5:15 AM, Eric Yang <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> >> Hi Ahmed,
> >> >>
> >> >> This is a great idea.  I am interested.  However, Chukwa is going
> >> through
> >> >> some major changes in Chukwa 0.5.  The changes are taking a while to
> be
> >> >> completed.  It may be worth while to wait for the new changes to
> arrive
> >> >> before writing the article.  What do you think?
> >> >>
> >> >> Regards,
> >> >> Eric
> >> >>
> >> >> On 9/2/10 3:23 AM, "Ahmed Fathalla" <[EMAIL PROTECTED]> wrote:
> >> >>
> >> >> Hello Guys,
> >> >>                     I was previously speaking to Ari and he told me
> >> about
> >> >> the need for more community knowledge of Chukwa. I think contributing
> an
> >> >> article about Chukwa to IBM developerWorks would be a great way of
> >> >> informing
> >> >> the general community about it. IBM developerWorks is a great source
> of
> >> >> accurate and relevant information about many subjects that interest

Ahmed Fathalla

Sounds good.

On Mon, Sep 13, 2010 at 2:16 PM, Ahmed Fathalla <[EMAIL PROTECTED]> wrote:
> Hello,
>           I've been doing some thinking about this issue. I am very
> inclined to switch the focus of this initial article from Chukwa
> installaiton and administration to an overview of different distributed log
> collection frameworks (Chukwa, Flume, Scirbe,Honu) with a discussion of each
> one's architecture and design goals. I think this kind of article would be
> much more useful for the IT community at large and given the significant
> changes Chukwa is ongoing at the moment, an article or series of articles
> specifically about Chukwa can be written once 0.5 is released.
>
> What are your thoughts?
>
> On Tue, Sep 7, 2010 at 5:00 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Sounds good.  You are correct.  HICC does not work out of the box for 0.4.
>>
>> regards,
>> Eric
>>
>> On Mon, Sep 6, 2010 at 3:23 PM, Ahmed Fathalla <[EMAIL PROTECTED]>
>> wrote:
>> > My plan for the article is to contain an introduction about the
>> importance
>> > of log processing frameworks like Chukwa, Scribe and Flume. Then provide
>> a
>> > description of Chukwa's architecture and finally a section on installing
>> and
>> > configuring Chukwa.
>> >
>> > I can base the article on 0.4 and provide pointers about areas where
>> users
>> > can expect changes in 0.5 and later.
>> >
>> > Also, you said MySQL is not supported in 0.4, does this mean that HICC
>> > doesn't work for 0.4 out of the box?
>> >
>> >
>> >
>> > On Tue, Sep 7, 2010 at 12:33 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>> >
>> >> Chukwa 0.5 is not going to be ready by end of this year.  There are
>> >> some major changes which I only work on them on long weekends.  I
>> >> think it's best for you to ask questions and you can choose to target
>> >> for Chukwa 0.4 or 0.5 on your own discretion.  For Chukwa 0.3, it was
>> >> prototyped to load data to HDFS and load materialized result to MySQL.
>> >>  For Chukwa 0.4, it is designed to load data to HDFS only.  MySQL path
>> >> is no longer supported.  However, there are hacks to make MySQL
>> >> components to work with Chukwa 0.4.  Chukwa 0.5 is designed to load
>> >> data to HBase for scalability reason.  However, there is a big chunk
>> >> of work post data loading for Chukwa 0.5 that are not written.  Hence,
>> >> it depends on which version that you are interested in writing
>> >> article.  I can provide pointers, but it is definitely going to take
>> >> much longer than 8 weeks for Chukwa 0.5.
>> >>
>> >> regards,
>> >> Eric
>> >>
>> >> On Sun, Sep 5, 2010 at 8:27 PM, Ahmed Fathalla <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Hey Eric,
>> >> >                    Thanks for your feedback. Publishing an artcile on
>> >> > developerWorks takes some time, you have to submit an abstract, then
>> if
>> >> the
>> >> > abstract is accepted you submit drafts which are reviewed by editors
>> >> before
>> >> > the final version is published. The process takes no less than 8
>> weeks.
>> >> >
>> >> > So I suggest we get started while working on 0.5. I can start by
>> >> submitting
>> >> > an abstract and communicating with the editor, do you agree?
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Sep 6, 2010 at 5:15 AM, Eric Yang <[EMAIL PROTECTED]>
>> wrote:
>> >> >
>> >> >> Hi Ahmed,
>> >> >>
>> >> >> This is a great idea.  I am interested.  However, Chukwa is going
>> >> through
>> >> >> some major changes in Chukwa 0.5.  The changes are taking a while to
>> be
>> >> >> completed.  It may be worth while to wait for the new changes to
>> arrive
>> >> >> before writing the article.  What do you think?
>> >> >>
>> >> >> Regards,
>> >> >> Eric
>> >> >>
>> >> >> On 9/2/10 3:23 AM, "Ahmed Fathalla" <[EMAIL PROTECTED]> wrote:
>> >> >>
>> >> >> Hello Guys,
>> >> >>                     I was previously speaking to Ari and he told me
>> >> about
>> >> >> the need for more community knowledge of Chukwa. I think contributing
>> an
>> >> >> article about Chukwa to IBM developerWorks would be a great way of

Ari Rabkin [EMAIL PROTECTED]
UC Berkeley Computer Science Department