Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # dev >> Experiment with Chukwa trunk


Copy link to this message
-
Re: Experiment with Chukwa trunk
I think I'm with Bill on this.

There are useful Chukwa deployments without HBase, and I don't want to
scare anybody away by saying "a full deployment has HBase."

I would say "you can use Chukwa on either HDFS or HBase. HBase has
lower latency, and requires Hadoop 0.20. Using HDFS+Mapred works with
0.18"

--Ari

On Mon, Oct 11, 2010 at 4:31 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
> For streaming data to Hbase 0.20.6, Hadoop 0.20 is required.  If you are
> using the sequence file and mapreduce, then Hadoop 0.18+ should work fine.
> I would like to move forward with map reduce on table instead of map reduce
> on file to reduce data processing latency.  Hence, I am making Hbase a
> required component for trunk for a full end to end deployment.  The setup
> may sound difficult to do, but it is trivial to setup.  In the long run,
> demux does not need to be a map reduce job, and demux will only get
> optimized for incremental updates, when developers start to adopt the table
> approach.  Considering this migration as planning for a better future.  :)
> For now, the existing Chukwa 0.4 approach is still works in trunk but I am
> not enhancing the batch model.
>
> Regards,
> Eric
>
> On 10/11/10 1:11 PM, "Bill Graham" <[EMAIL PROTECTED]> wrote:
>
>> Hi Eric,
>>
>> A read this over and have a few comments:
>>
>> - Prerequisites say that Hadoop 0.20+ is required, which I think isn't
>> entirely true. Agents, collectors and the data processor processes all
>> can run with 0.18.3. I though HICC was the only thing that required
>> 0.20, no? Either way, we should clarify which components require what
>> version.
>>
>> - You talk about a minimal Chukwa install including HBase, which also
>> seems misleading. Will HBase be required to use Chukwa going forward?
>> If not, no need to make the barrier for entry sound higher than it
>> needs to be IMO.
>>
>> thanks,
>> Bill
>>
>> On Sat, Oct 9, 2010 at 6:47 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>> Hi,
>>>
>>> I have written some instructions on how to deploy Chukwa 0.5
>>> pre-release system with HBase.  Instructions are available here:
>>>
>>> http://wiki.apache.org/hadoop/Chukwa_Quick_Start
>>>
>>> Suggestions and feedback are welcome.
>>>
>>> regards,
>>> Eric
>>>
>>
>
>

--
Ari Rabkin [EMAIL PROTECTED]
UC Berkeley Computer Science Department
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB