Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: Flume adopt message from existing local Scribe


Copy link to this message
-
Re: Review Request: Flume adopt message from existing local Scribe


> On Aug. 3, 2012, 3:45 a.m., Hari Shreedharan wrote:
> > I am not sure exactly how scribe works. From what you explained, it seems like you need to write a client of some sort from scribe side or scribe can somehow be configured to write to send the messages to a specified host/port?
> >
> > Anyway, what I was asking was if you could add a section to the Flume User Guide on how to set up the ScribeSource on Flume side, and also how to set up scribe to write to this source. You can find the Flume User Guide here: flume-ng-doc/sphinx/FlumeUserGuide.rst. If you could add an example of how to configure both scribe and flume so that you can dump events to flume, that would be great.
> >
> > I'd like to commit this if I am able to test basic functionality, and improve and fix issues as they are noticed, since committing this will not affect the working of other components the remaining system.
> >
> > So if you can help me set it up - configuring this and configuring scribe to write to this, it would be great.
>
> Denny Ye wrote:
>     Hari, I posted ScribeSource guide patch into into bug https://issues.apache.org/jira/browse/FLUME-1382 (two patches). Please review it. Scribe is existing ingest system from Facebook. The configuration of Scribe was produced by Facebook.

While this uses the scribe protocol, there is nothing scribe specific about it other than that. Anything that can send thrift messages using the protocol will work. We have a program that we've been using to feed data to scribe using the thrift protocol until now and it works fine with this now
- Juhani
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6089/#review9582
-----------------------------------------------------------
On July 30, 2012, 2:49 a.m., Denny Ye wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6089/
> -----------------------------------------------------------
>
> (Updated July 30, 2012, 2:49 a.m.)
>
>
> Review request for Flume and Hari Shreedharan.
>
>
> Description
> -------
>
> There may someone like me that want to replace central Scribe with Flume to adopt existing ingest system, using smooth changes for application user.
> Here is the ScribeSource put into legacy folder without deserializing.
>
>
> This addresses bug https://issues.apache.org/jira/browse/FLUME-1382.
>     https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1382
>
>
> Diffs
> -----
>
>   trunk/flume-ng-dist/pom.xml 1363210
>   trunk/flume-ng-sources/flume-scribe-source/pom.xml PRE-CREATION
>   trunk/flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/LogEntry.java PRE-CREATION
>   trunk/flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ResultCode.java PRE-CREATION
>   trunk/flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java PRE-CREATION
>   trunk/flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java PRE-CREATION
>   trunk/flume-ng-sources/pom.xml PRE-CREATION
>   trunk/pom.xml 1363210
>
> Diff: https://reviews.apache.org/r/6089/diff/
>
>
> Testing
> -------
>
> I already used ScribeSource into local environment and tested in past week. It can use the existing local Scribe interface
>
>
> Thanks,
>
> Denny Ye
>
>