Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # dev >> Improvement for Chukwa Agent and Collector


Copy link to this message
-
Re: Improvement for Chukwa Agent and Collector
Proposal overall sounds useful.  I like versioning.

--Ari

On Mon, Aug 9, 2010 at 5:03 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
> I like to have /v1/ at least to identify the URL versioning.  Just to be
> safe, if we change URL in the future.  / and /tool to point to information
> UI make sense.
>
> Regards,
> Eric
>
> On 8/9/10 2:59 PM, "Bill Graham" <[EMAIL PROTECTED]> wrote:
>
>> I generally feel that all params should be able to be passed either entirely
>> in the body or entirely in the URI regardless which ones are required/optional
>> (with the exception of the asset id, which typically is in the path
>> regardless). I vote for passing them all in the body as a json blob in this
>> case (if Content-Type is set to application/json that is).
>>
>> Thinking more about the base path to the API that I proposed, perhaps the
>> /v1.0 in the URL is overkill. I could go for removing that part. The /rest
>> path has value though to me though, because I could see keeping '/' or '/tool'
>> to potentially point to an HTML summary page or mini-UI at some point.
>>
>>
>>
>> On Mon, Aug 9, 2010 at 2:42 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>> Hi Bill,
>>>
>>> I like your design better.  +1 on the revised version.  RecordType and
>>> Adaptor are required parameters, would it make sense if we could put them on
>>> the path parameters for POST?
>>>
>>> Regards,
>>> Eric
>>>
>>> On 8/9/10 11:33 AM, "Bill Graham" <[EMAIL PROTECTED]> wrote:
>>>
>>>> I agree that we should implement the features you suggest. I've been
>>>> thinking about a REST API for the agents lately, as I'd also like to be able
>>>> to expose statistics to help with monitoring. Something similar to what the
>>>> collector does so you can attach monitoring to a URL see if the average data
>>>> rate suddenly drops.
>>>>
>>>> Regarding the proposed API protocol, I think we should use POST, GET and
>>>> DELETE to create, fetch and remove adaptors, similar to how you propose, but
>>>> the identifier in the rest resource should be the adaptor id, not the
>>>> filename. This is more RESTful since the adaptor is the thing being
>>>> accessed, not the file. Also, you could have more than one adaptor on a
>>>> given file and some adaptors (i.e., JMSAdaptor) don't have a file associated
>>>> with them.
>>>>
>>>> I propose something like this:
>>>>
>>>> - Add Adaptor:
>>>>
>>>> POST /rest/v1.0/adaptor HTTP/1.0
>>>> Accept: text/plain
>>>> Content-Type: application/json
>>>> { "RecordType" : "jvm", "Cluster": "demo", adaptor configs including offset,
>>>> other tags ... }
>>>>
>>>> Returns: adaptor metadata including id
>>>>
>>>> - Get Adaptor fcb0fe44e9dd6d2283962cb0e3b4ea0f:
>>>>
>>>> GET /rest/v1.0/adaptor/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
>>>>
>>>> - Remove Adaptor fcb0fe44e9dd6d2283962cb0e3b4ea0f:
>>>>
>>>> DELETE /rest/v1.0/adaptor/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
>>>>
>>>> - List all adaptors:
>>>> GET /rest/v1.0/adaptor HTTP/1.0
>>>>
>>>> - Help
>>>> GET /rest/v1.0/help HTTP/1.0
>>>>
>>>> - Statistics for all adaptors
>>>> GET /rest/v1.0/adaptorStats HTTP/1.0
>>>>
>>>> - Statistics for a single adaptor
>>>> GET /rest/v1.0/adaptorStats/fcb0fe44e9dd6d2283962cb0e3b4ea0f HTTP/1.0
>>>>
>>>> Thoughts?
>>>>
>>>> thanks,
>>>> Bill
>>>>
>>>> On Mon, Aug 9, 2010 at 10:01 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>>  Chukwa Agent has a custom command protocol (port 9093).  The current
>>>>> protocol is not easy to modify to implement security related features such
>>>>> as authentication and authorization.  I would like to propose that we use
>>>>> web service REST like protocol to improve security and be more aligned with
>>>>> web standards.  Let¹s go through the use cases of Chukwa Agent command
>>>>> protocol:
>>>>>
>>>>> Start an adaptor:
>>>>>
>>>>> Current command: Add
>>>>>
>>>>>
> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingA>>>>
> d
>>>>> aptorUTF8NewLineEscaped
>>>>> /tmp/chukwa/var/log/metrics/chukwa-hdfs-jvm-1271121726962.log 0

Ari Rabkin [EMAIL PROTECTED]
UC Berkeley Computer Science Department
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB