Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # dev >> Improvement for Chukwa Agent and Collector


Copy link to this message
-
Re: Improvement for Chukwa Agent and Collector
I generally feel that all params should be able to be passed either entirely
in the body or entirely in the URI regardless which ones are
required/optional (with the exception of the asset id, which typically is in
the path regardless). I vote for passing them all in the body as a json blob
in this case (if Content-Type is set to application/json that is).

Thinking more about the base path to the API that I proposed, perhaps the
/v1.0 in the URL is overkill. I could go for removing that part. The /rest
path has value though to me though, because I could see keeping '/' or
'/tool' to potentially point to an HTML summary page or mini-UI at some
point.

On Mon, Aug 9, 2010 at 2:42 PM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Hi Bill,
>
> I like your design better.  +1 on the revised version.  RecordType and
> Adaptor are required parameters, would it make sense if we could put them
> on
> the path parameters for POST?
>
> Regards,
> Eric
>
> On 8/9/10 11:33 AM, "Bill Graham" <[EMAIL PROTECTED]> wrote:
>
> > I agree that we should implement the features you suggest. I've been
> > thinking about a REST API for the agents lately, as I'd also like to be
> able
> > to expose statistics to help with monitoring. Something similar to what
> the
> > collector does so you can attach monitoring to a URL see if the average
> data
> > rate suddenly drops.
> >
> > Regarding the proposed API protocol, I think we should use POST, GET and
> > DELETE to create, fetch and remove adaptors, similar to how you propose,
> but
> > the identifier in the rest resource should be the adaptor id, not the
> > filename. This is more RESTful since the adaptor is the thing being
> > accessed, not the file. Also, you could have more than one adaptor on a
> > given file and some adaptors (i.e., JMSAdaptor) don't have a file
> associated
> > with them.
> >
> > I propose something like this:
> >
> > - Add Adaptor:
> >
> > POST /rest/v1.0/adaptor HTTP/1.0
> > Accept: text/plain
> > Content-Type: application/json
> > { "RecordType" : "jvm", "Cluster": "demo", adaptor configs including
> offset,
> > other tags ... }
> >
> > Returns: adaptor metadata including id
> >
> > - Get Adaptor fcb0fe44e9dd6d2283962cb0e3b4ea0f:
> >
> > GET /rest/v1.0/adaptor/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
> >
> > - Remove Adaptor fcb0fe44e9dd6d2283962cb0e3b4ea0f:
> >
> > DELETE /rest/v1.0/adaptor/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
> >
> > - List all adaptors:
> > GET /rest/v1.0/adaptor HTTP/1.0
> >
> > - Help
> > GET /rest/v1.0/help HTTP/1.0
> >
> > - Statistics for all adaptors
> > GET /rest/v1.0/adaptorStats HTTP/1.0
> >
> > - Statistics for a single adaptor
> > GET /rest/v1.0/adaptorStats/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
> >
> > Thoughts?
> >
> > thanks,
> > Bill
> >
> > On Mon, Aug 9, 2010 at 10:01 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> >
> >> Hi all,
> >>
> >>  Chukwa Agent has a custom command protocol (port 9093).  The current
> >> protocol is not easy to modify to implement security related features
> such
> >> as authentication and authorization.  I would like to propose that we
> use
> >> web service REST like protocol to improve security and be more aligned
> with
> >> web standards.  Let¹s go through the use cases of Chukwa Agent command
> >> protocol:
> >>
> >> Start an adaptor:
> >>
> >> Current command: Add
> >>
> >>
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAd
> >> aptorUTF8NewLineEscaped
> >> /tmp/chukwa/var/log/metrics/chukwa-hdfs-jvm-1271121726962.log 0
> >>
> >> Proposed:
> >> POST /tmp/chukwa/var/log/metrics/chukwa-hdfs-jvm-1271121726962.log
> HTTP/1.0
> >> Accept: chukwa/UTF8NewLineEscaped (optional)
> >> Offset: 0 (optional)
> >> Content-Type: application/json
> >> { ³RecordType² : ³jvm², "Cluster": "demo", other tags ... }
> >>
> >> List adaptors:
> >>
> >> Current command: List
> >>
> >> Proposed:
> >> GET / HTTP/1.0
> >> Accept: text/html
> >> Get list of information about all streaming adatpors
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB