Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Scribe Source .thrift file


+
Hari Shreedharan 2013-02-03, 00:14
+
Juhani Connolly 2013-02-05, 02:30
+
Denny Ye 2013-02-05, 02:46
+
Hari Shreedharan 2013-02-05, 04:05
+
Hari Shreedharan 2013-02-05, 06:57
+
Denny Ye 2013-02-05, 07:42
+
Hari Shreedharan 2013-02-05, 07:53
+
Denny Ye 2013-02-05, 08:08
+
Hari Shreedharan 2013-02-05, 08:10
Copy link to this message
-
Re: Scribe Source .thrift file
Denny Ye 2013-02-05, 08:16
Thanks Hari, no other better solution now, we only can document in source
code to make it clearly. Wish you can agree with the modification for that
generated code
2013/2/5 Hari Shreedharan <[EMAIL PROTECTED]>

> Thanks Denny. If you don't have an idea, I will document that in the
> pom.xml of the scribe source, and in the ScribeSource.java file, so the
> next time thrift is upgraded - the developer can manually change the string
> to a ByteBuffer. Does that make sense to you?
>
>
> Hari
>
> --
> Hari Shreedharan
>
>
> On Tuesday, February 5, 2013 at 12:08 AM, Denny Ye wrote:
>
> > OK, I'm not sure which is better method to keep the effective code during
> > upgrading.
> > I will post it if I have a good idea.
> >
> >
> > 2013/2/5 Hari Shreedharan <[EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED])>
> >
> > > OK, no problem. I modified the Scribe Source to simply pick up the
> String
> > > and convert it to a byte array. Unfortunately, since we need to
> maintain
> > > the generated code even in between Thrift upgrades, we should probably
> not
> > > edit the generated code by hand. I am actually OK with modifying that
> class
> > > to use ByteBuffer, but just document that when we do a Thrift upgrade,
> we
> > > must change the LogEntry class manually - this is not a good idea, but
> if
> > > you think that makes it effective and gives better performance I won't
> > > complain. Do you think you could do this using a script
> (find/replace), so
> > > we can add ant-run plugin to the pom.xml to handle this case?
> > >
> > >
> > > Thanks
> > > Hari
> > >
> > > --
> > > Hari Shreedharan
> > >
> > >
> > > On Monday, February 4, 2013 at 11:42 PM, Denny Ye wrote:
> > >
> > > > hi Hari, the original version of generated code from that Thrift
> template
> > > > has only support message with 'String' type, I think it's ineffective
> > > >
> > >
> > > that
> > > > deserialized from byte stream to String again. That's CPU sensitive
> work
> > > > and useless in progress of data flow. Thus, I modified code of
> LogEntry
> > > >
> > >
> > > to
> > > > support byte array incoming from socket except deserializing.
> > > >
> > > > What should I do next?
> > > >
> > > >
> > > > 2013/2/5 Hari Shreedharan <[EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED]) (mailto:
> > > [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]))>
> > > >
> > > > > Denny,
> > > > >
> > > > > I modified the file's namespace and the final file looks like this:
> > > > >
> > > > > namespace java org.apache.flume.source.scribe
> > > > >
> > > > > enum ResultCode
> > > > > {
> > > > > OK,
> > > > > TRY_LATER
> > > > > }
> > > > >
> > > > > struct LogEntry
> > > > > {
> > > > > 1: string category,
> > > > > 2: string message
> > > > > }
> > > > >
> > > > > service Scribe
> > > > > {
> > > > > ResultCode Log(1: list<LogEntry> messages);
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > > But it looks like the ScribeSource.java (line 154) expects
> > > > > LogEntry.message to be "binary" and not "string." Since there are
> no
> > > > >
> > > >
> > > >
> > >
> > > unit
> > > > > tests, I cannot verify that making message binary will not break
> > > >
> > >
> > > something.
> > > > > Could you give me some fix for this?
> > > > >
> > > > >
> > > > > Thanks
> > > > > Hari
> > > > >
> > > > > --
> > > > > Hari Shreedharan
> > > > >
> > > > >
> > > > > On Monday, February 4, 2013 at 8:05 PM, Hari Shreedharan wrote:
> > > > >
> > > > > > Hi Denny, Juhani,
> > > > > >
> > > > > > Thanks! I will add this file to the compileThrift profile, so we
> can
> > > > > update the thrift generated code when we upgrade thrift.
> > > > > >
> > > > > > Thanks!
> > > > > > Hari
> > > > > >
> > > > > > --
> > > > > > Hari Shreedharan
> > > > > >
> > > > > >
> > > > > > On Monday, February 4, 2013 at 6:46 PM, Denny Ye wrote:
> > > > > >
> > > > > > > Sorry for late response
> > > > > > >
> > > > > > > Here is the scribe.thrift file :
> > > > > > > ************************* file separator *********************