Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # dev >> Improvement for Chukwa Agent and Collector


Copy link to this message
-
Re: Improvement for Chukwa Agent and Collector
I generally feel that all params should be able to be passed either entirely
in the body or entirely in the URI regardless which ones are
required/optional (with the exception of the asset id, which typically is in
the path regardless). I vote for passing them all in the body as a json blob
in this case (if Content-Type is set to application/json that is).

Thinking more about the base path to the API that I proposed, perhaps the
/v1.0 in the URL is overkill. I could go for removing that part. The /rest
path has value though to me though, because I could see keeping '/' or
'/tool' to potentially point to an HTML summary page or mini-UI at some
point.

On Mon, Aug 9, 2010 at 2:42 PM, Eric Yang <[EMAIL PROTECTED]> wrote:

> Hi Bill,
>
> I like your design better.  +1 on the revised version.  RecordType and
> Adaptor are required parameters, would it make sense if we could put them
> on
> the path parameters for POST?
>
> Regards,
> Eric
>
> On 8/9/10 11:33 AM, "Bill Graham" <[EMAIL PROTECTED]> wrote:
>
> > I agree that we should implement the features you suggest. I've been
> > thinking about a REST API for the agents lately, as I'd also like to be
> able
> > to expose statistics to help with monitoring. Something similar to what
> the
> > collector does so you can attach monitoring to a URL see if the average
> data
> > rate suddenly drops.
> >
> > Regarding the proposed API protocol, I think we should use POST, GET and
> > DELETE to create, fetch and remove adaptors, similar to how you propose,
> but
> > the identifier in the rest resource should be the adaptor id, not the
> > filename. This is more RESTful since the adaptor is the thing being
> > accessed, not the file. Also, you could have more than one adaptor on a
> > given file and some adaptors (i.e., JMSAdaptor) don't have a file
> associated
> > with them.
> >
> > I propose something like this:
> >
> > - Add Adaptor:
> >
> > POST /rest/v1.0/adaptor HTTP/1.0
> > Accept: text/plain
> > Content-Type: application/json
> > { "RecordType" : "jvm", "Cluster": "demo", adaptor configs including
> offset,
> > other tags ... }
> >
> > Returns: adaptor metadata including id
> >
> > - Get Adaptor fcb0fe44e9dd6d2283962cb0e3b4ea0f:
> >
> > GET /rest/v1.0/adaptor/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
> >
> > - Remove Adaptor fcb0fe44e9dd6d2283962cb0e3b4ea0f:
> >
> > DELETE /rest/v1.0/adaptor/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
> >
> > - List all adaptors:
> > GET /rest/v1.0/adaptor HTTP/1.0
> >
> > - Help
> > GET /rest/v1.0/help HTTP/1.0
> >
> > - Statistics for all adaptors
> > GET /rest/v1.0/adaptorStats HTTP/1.0
> >
> > - Statistics for a single adaptor
> > GET /rest/v1.0/adaptorStats/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
> >
> > Thoughts?
> >
> > thanks,
> > Bill
> >
> > On Mon, Aug 9, 2010 at 10:01 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> >
> >> Hi all,
> >>
> >>  Chukwa Agent has a custom command protocol (port 9093).  The current
> >> protocol is not easy to modify to implement security related features
> such
> >> as authentication and authorization.  I would like to propose that we
> use
> >> web service REST like protocol to improve security and be more aligned
> with
> >> web standards.  Let¹s go through the use cases of Chukwa Agent command
> >> protocol:
> >>
> >> Start an adaptor:
> >>
> >> Current command: Add
> >>
> >>
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAd
> >> aptorUTF8NewLineEscaped
> >> /tmp/chukwa/var/log/metrics/chukwa-hdfs-jvm-1271121726962.log 0
> >>
> >> Proposed:
> >> POST /tmp/chukwa/var/log/metrics/chukwa-hdfs-jvm-1271121726962.log
> HTTP/1.0
> >> Accept: chukwa/UTF8NewLineEscaped (optional)
> >> Offset: 0 (optional)
> >> Content-Type: application/json
> >> { ³RecordType² : ³jvm², "Cluster": "demo", other tags ... }
> >>
> >> List adaptors:
> >>
> >> Current command: List
> >>
> >> Proposed:
> >> GET / HTTP/1.0
> >> Accept: text/html
> >> Get list of information about all streaming adatpors
>