Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Scribe Source .thrift file


Copy link to this message
-
Re: Scribe Source .thrift file
OK, I'm not sure which is better method to keep the effective code during
upgrading.
I will post it if I have a good idea.
2013/2/5 Hari Shreedharan <[EMAIL PROTECTED]>

> OK, no problem. I modified the Scribe Source to simply pick up the String
> and convert it to a byte array. Unfortunately, since we need to maintain
> the generated code even in between Thrift upgrades, we should probably not
> edit the generated code by hand. I am actually OK with modifying that class
> to use ByteBuffer, but just document that when we do a Thrift upgrade, we
> must change the LogEntry class manually - this is not a good idea, but if
> you think that makes it effective and gives better performance I won't
> complain. Do you think you could do this using a script (find/replace), so
> we can add ant-run plugin to the pom.xml to handle this case?
>
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
>
> On Monday, February 4, 2013 at 11:42 PM, Denny Ye wrote:
>
> > hi Hari, the original version of generated code from that Thrift template
> > has only support message with 'String' type, I think it's ineffective
> that
> > deserialized from byte stream to String again. That's CPU sensitive work
> > and useless in progress of data flow. Thus, I modified code of LogEntry
> to
> > support byte array incoming from socket except deserializing.
> >
> > What should I do next?
> >
> >
> > 2013/2/5 Hari Shreedharan <[EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED])>
> >
> > > Denny,
> > >
> > > I modified the file's namespace and the final file looks like this:
> > >
> > > namespace java org.apache.flume.source.scribe
> > >
> > > enum ResultCode
> > > {
> > > OK,
> > > TRY_LATER
> > > }
> > >
> > > struct LogEntry
> > > {
> > > 1: string category,
> > > 2: string message
> > > }
> > >
> > > service Scribe
> > > {
> > > ResultCode Log(1: list<LogEntry> messages);
> > > }
> > >
> > >
> > >
> > > But it looks like the ScribeSource.java (line 154) expects
> > > LogEntry.message to be "binary" and not "string." Since there are no
> unit
> > > tests, I cannot verify that making message binary will not break
> something.
> > > Could you give me some fix for this?
> > >
> > >
> > > Thanks
> > > Hari
> > >
> > > --
> > > Hari Shreedharan
> > >
> > >
> > > On Monday, February 4, 2013 at 8:05 PM, Hari Shreedharan wrote:
> > >
> > > > Hi Denny, Juhani,
> > > >
> > > > Thanks! I will add this file to the compileThrift profile, so we can
> > > update the thrift generated code when we upgrade thrift.
> > > >
> > > > Thanks!
> > > > Hari
> > > >
> > > > --
> > > > Hari Shreedharan
> > > >
> > > >
> > > > On Monday, February 4, 2013 at 6:46 PM, Denny Ye wrote:
> > > >
> > > > > Sorry for late response
> > > > >
> > > > > Here is the scribe.thrift file :
> > > > > ************************* file separator *********************
> > > > > namespace java scribe.thrift
> > > > >
> > > > > enum ResultCode
> > > > > {
> > > > > OK,
> > > > > TRY_LATER
> > > > > }
> > > > >
> > > > > struct LogEntry
> > > > > {
> > > > > 1: string category,
> > > > > 2: string message
> > > > > }
> > > > >
> > > > > service scribe
> > > > > {
> > > > > ResultCode Log(1: list<LogEntry> messages);
> > > > > }
> > > > > ************************* file separator *********************
> > > > >
> > > > > To Juhani, I haven't use the facebook template, there have so many
> > > > > additional interfaces that I don't like it.
> > > > >
> > > > >
> > > > >
> > > > > 2013/2/5 Juhani Connolly <[EMAIL PROTECTED](mailto:
> [EMAIL PROTECTED]) (mailto:
> > > [EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED]))>
> > > > >
> > > > > > This is the file we want:
> > > > > >
> > > > > >
> https://github.com/facebook/**scribe/blob/master/if/scribe.**thrift<
> > > https://github.com/facebook/scribe/blob/master/if/scribe.thrift>
> > > > > >
> > > > > > It's apache 2 licensed but also has a facebooke copyright
> header. I
> > > think
> > > > > > this is why we didn't originally include it.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB