Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> How to process Data when Server Connected Resused?


Copy link to this message
-
Re: How to process Data when Server Connected Resused?
hi GuoWei,

The current offset of the adaptor is one way to track how much data are
sent.  If your data is not stored in local disk of the agent, then data can
be lost when sending buffer is full.  Collector buffers data in memory, it
does not store in collector disks by default.  There is a local writer
which buffers data in disk and upload to hdfs.  This is not turn on by
default.  Hence, you can utilize local writer or implement your own writer
to achieve the same result.
See: http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html for more
information about local writer.

We are planning to merge collector and agent together to provide ability to
spill to local disk, and reduce deployment complexity.  This is in:

https://issues.apache.org/jira/browse/CHUKWA-674

If you are interested.

regards,
Eric

On Mon, Dec 24, 2012 at 9:52 PM, GuoWei <[EMAIL PROTECTED]> wrote:

> Dear all,
>
> Recently, I write a my custom adaptor and map processor to store data to
> Hbase.  But when I shut down the Hbase, I found many many error logs in
> Collector.log. My questions are :
>
> 1. When my custom adaptor can not connect to collector, do I need to store
> data temporary myself ? Or chukwa can do this for me ?
>      2.  When my custom demux map processor can not connect to Hbase, do I
> need to store data   temporary my self at collector side ? Or chukwa can do
> this for me ?
>
> Thanks in advance.
>
> *Best Regards / **商祺**
> *郭伟 Guo Wei
> -----------------------------------------------------
> 南京西桥科技有限公司
> Western Bridge Tech Ltd.,  Nanjing
>
> 南京市玄武区花园路8号一号楼511
> No. 511, Building 1, No. 8, Hua Yuan Road
>
> Xuanwu District, Nanjing, PR China
>
> Email: [EMAIL PROTECTED]
>
> Tel: +86 25 8528 4900 (Operator)
> Mobile: +86 138 1589 8257
>
> Fax: +86 25 8528 4980
>
> Web: http://www.wbkit.com
>
> ----------------------------------------------------- *******
> **WesternBridge Tech: Professional software service provider.
> Professional is MANNER as well CAPABILITY.*
>
> *
> *
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB