Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Scribe Source .thrift file


+
Hari Shreedharan 2013-02-03, 00:14
+
Juhani Connolly 2013-02-05, 02:30
+
Denny Ye 2013-02-05, 02:46
+
Hari Shreedharan 2013-02-05, 04:05
+
Hari Shreedharan 2013-02-05, 06:57
+
Denny Ye 2013-02-05, 07:42
Copy link to this message
-
Re: Scribe Source .thrift file
OK, no problem. I modified the Scribe Source to simply pick up the String and convert it to a byte array. Unfortunately, since we need to maintain the generated code even in between Thrift upgrades, we should probably not edit the generated code by hand. I am actually OK with modifying that class to use ByteBuffer, but just document that when we do a Thrift upgrade, we must change the LogEntry class manually - this is not a good idea, but if you think that makes it effective and gives better performance I won't complain. Do you think you could do this using a script (find/replace), so we can add ant-run plugin to the pom.xml to handle this case?
Thanks
Hari

--
Hari Shreedharan
On Monday, February 4, 2013 at 11:42 PM, Denny Ye wrote:

> hi Hari, the original version of generated code from that Thrift template
> has only support message with 'String' type, I think it's ineffective that
> deserialized from byte stream to String again. That's CPU sensitive work
> and useless in progress of data flow. Thus, I modified code of LogEntry to
> support byte array incoming from socket except deserializing.
>
> What should I do next?
>
>
> 2013/2/5 Hari Shreedharan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
>
> > Denny,
> >
> > I modified the file's namespace and the final file looks like this:
> >
> > namespace java org.apache.flume.source.scribe
> >
> > enum ResultCode
> > {
> > OK,
> > TRY_LATER
> > }
> >
> > struct LogEntry
> > {
> > 1: string category,
> > 2: string message
> > }
> >
> > service Scribe
> > {
> > ResultCode Log(1: list<LogEntry> messages);
> > }
> >
> >
> >
> > But it looks like the ScribeSource.java (line 154) expects
> > LogEntry.message to be "binary" and not "string." Since there are no unit
> > tests, I cannot verify that making message binary will not break something.
> > Could you give me some fix for this?
> >
> >
> > Thanks
> > Hari
> >
> > --
> > Hari Shreedharan
> >
> >
> > On Monday, February 4, 2013 at 8:05 PM, Hari Shreedharan wrote:
> >
> > > Hi Denny, Juhani,
> > >
> > > Thanks! I will add this file to the compileThrift profile, so we can
> > update the thrift generated code when we upgrade thrift.
> > >
> > > Thanks!
> > > Hari
> > >
> > > --
> > > Hari Shreedharan
> > >
> > >
> > > On Monday, February 4, 2013 at 6:46 PM, Denny Ye wrote:
> > >
> > > > Sorry for late response
> > > >
> > > > Here is the scribe.thrift file :
> > > > ************************* file separator *********************
> > > > namespace java scribe.thrift
> > > >
> > > > enum ResultCode
> > > > {
> > > > OK,
> > > > TRY_LATER
> > > > }
> > > >
> > > > struct LogEntry
> > > > {
> > > > 1: string category,
> > > > 2: string message
> > > > }
> > > >
> > > > service scribe
> > > > {
> > > > ResultCode Log(1: list<LogEntry> messages);
> > > > }
> > > > ************************* file separator *********************
> > > >
> > > > To Juhani, I haven't use the facebook template, there have so many
> > > > additional interfaces that I don't like it.
> > > >
> > > >
> > > >
> > > > 2013/2/5 Juhani Connolly <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) (mailto:
> > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>
> > > >
> > > > > This is the file we want:
> > > > >
> > > > > https://github.com/facebook/**scribe/blob/master/if/scribe.**thrift<
> > https://github.com/facebook/scribe/blob/master/if/scribe.thrift>
> > > > >
> > > > > It's apache 2 licensed but also has a facebooke copyright header. I
> > think
> > > > > this is why we didn't originally include it.
> > > > >
> > > > >
> > > > > On 02/03/2013 09:14 AM, Hari Shreedharan wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > The scribe source has some thrift generated code committed, but no
> > > > > > ".thrift" file. This makes upgrading to a new version of libthrift
> > > > > > impossible.
> > > > > >
> > > > > > I believe Denny Ye contributed this code, so Denny, it'd be nice
+
Denny Ye 2013-02-05, 08:08
+
Hari Shreedharan 2013-02-05, 08:10
+
Denny Ye 2013-02-05, 08:16
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB