Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # dev >> Improvement for Chukwa Agent and Collector


+
Eric Yang 2010-08-09, 17:01
+
Bill Graham 2010-08-09, 18:33
+
Eric Yang 2010-08-09, 21:42
+
Bill Graham 2010-08-09, 21:59
+
Eric Yang 2010-08-10, 00:03
Copy link to this message
-
Re: Improvement for Chukwa Agent and Collector
Proposal overall sounds useful.  I like versioning.

--Ari

On Mon, Aug 9, 2010 at 5:03 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
> I like to have /v1/ at least to identify the URL versioning.  Just to be
> safe, if we change URL in the future.  / and /tool to point to information
> UI make sense.
>
> Regards,
> Eric
>
> On 8/9/10 2:59 PM, "Bill Graham" <[EMAIL PROTECTED]> wrote:
>
>> I generally feel that all params should be able to be passed either entirely
>> in the body or entirely in the URI regardless which ones are required/optional
>> (with the exception of the asset id, which typically is in the path
>> regardless). I vote for passing them all in the body as a json blob in this
>> case (if Content-Type is set to application/json that is).
>>
>> Thinking more about the base path to the API that I proposed, perhaps the
>> /v1.0 in the URL is overkill. I could go for removing that part. The /rest
>> path has value though to me though, because I could see keeping '/' or '/tool'
>> to potentially point to an HTML summary page or mini-UI at some point.
>>
>>
>>
>> On Mon, Aug 9, 2010 at 2:42 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>> Hi Bill,
>>>
>>> I like your design better.  +1 on the revised version.  RecordType and
>>> Adaptor are required parameters, would it make sense if we could put them on
>>> the path parameters for POST?
>>>
>>> Regards,
>>> Eric
>>>
>>> On 8/9/10 11:33 AM, "Bill Graham" <[EMAIL PROTECTED]> wrote:
>>>
>>>> I agree that we should implement the features you suggest. I've been
>>>> thinking about a REST API for the agents lately, as I'd also like to be able
>>>> to expose statistics to help with monitoring. Something similar to what the
>>>> collector does so you can attach monitoring to a URL see if the average data
>>>> rate suddenly drops.
>>>>
>>>> Regarding the proposed API protocol, I think we should use POST, GET and
>>>> DELETE to create, fetch and remove adaptors, similar to how you propose, but
>>>> the identifier in the rest resource should be the adaptor id, not the
>>>> filename. This is more RESTful since the adaptor is the thing being
>>>> accessed, not the file. Also, you could have more than one adaptor on a
>>>> given file and some adaptors (i.e., JMSAdaptor) don't have a file associated
>>>> with them.
>>>>
>>>> I propose something like this:
>>>>
>>>> - Add Adaptor:
>>>>
>>>> POST /rest/v1.0/adaptor HTTP/1.0
>>>> Accept: text/plain
>>>> Content-Type: application/json
>>>> { "RecordType" : "jvm", "Cluster": "demo", adaptor configs including offset,
>>>> other tags ... }
>>>>
>>>> Returns: adaptor metadata including id
>>>>
>>>> - Get Adaptor fcb0fe44e9dd6d2283962cb0e3b4ea0f:
>>>>
>>>> GET /rest/v1.0/adaptor/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
>>>>
>>>> - Remove Adaptor fcb0fe44e9dd6d2283962cb0e3b4ea0f:
>>>>
>>>> DELETE /rest/v1.0/adaptor/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
>>>>
>>>> - List all adaptors:
>>>> GET /rest/v1.0/adaptor HTTP/1.0
>>>>
>>>> - Help
>>>> GET /rest/v1.0/help HTTP/1.0
>>>>
>>>> - Statistics for all adaptors
>>>> GET /rest/v1.0/adaptorStats HTTP/1.0
>>>>
>>>> - Statistics for a single adaptor
>>>> GET /rest/v1.0/adaptorStats/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
>>>>
>>>> Thoughts?
>>>>
>>>> thanks,
>>>> Bill
>>>>
>>>> On Mon, Aug 9, 2010 at 10:01 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>>  Chukwa Agent has a custom command protocol (port 9093).  The current
>>>>> protocol is not easy to modify to implement security related features such
>>>>> as authentication and authorization.  I would like to propose that we use
>>>>> web service REST like protocol to improve security and be more aligned with
>>>>> web standards.  Let¹s go through the use cases of Chukwa Agent command
>>>>> protocol:
>>>>>
>>>>> Start an adaptor:
>>>>>
>>>>> Current command: Add
>>>>>
>>>>>
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingA>>>>
> d
>>>>> aptorUTF8NewLineEscaped
>>>>> /tmp/chukwa/var/log/metrics/chukwa-hdfs-jvm-1271121726962.log 0

Ari Rabkin [EMAIL PROTECTED]
UC Berkeley Computer Science Department