Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> HDFS escape sequence and use of timestamp header


+
Mohit Anchlia 2012-08-22, 00:52
+
Mohit Anchlia 2012-08-22, 18:50
+
Kathleen Ting 2012-08-22, 21:16
+
Mohit Anchlia 2012-08-23, 00:05
+
Kathleen Ting 2012-08-23, 00:52
Copy link to this message
-
Re: HDFS escape sequence and use of timestamp header
thanks that works

On Wed, Aug 22, 2012 at 5:52 PM, Kathleen Ting <[EMAIL PROTECTED]> wrote:

> Mohit,
>
> As it happens, both the example and the table are correct. You can
> specify either the fully qualified class name or the alias 'TIMESTAMP'
>
> Regards, Kathleen
>
> On Wed, Aug 22, 2012 at 5:05 PM, Mohit Anchlia <[EMAIL PROTECTED]>
> wrote:
> >
> >
> > On Wed, Aug 22, 2012 at 2:16 PM, Kathleen Ting <[EMAIL PROTECTED]>
> wrote:
> >>
> >> Hi Mohit,
> >>
> >> You can configure a timestamp interceptor onto your source as follows:
> >>
> >> agent.sources.src-0.interceptors.ts.type = TIMESTAMP
> >>
> >> Once the timestamp interceptor is in place, you can use, in sinks, the
> >> timestamp it writes. Here is an example of using the timestamp in an
> >> HDFS sink:
> >>
> >> agent.sinks.sink-0.hdfs.filePrefix = FlumeData.%Y-%m-%d
> >>
> >> Here is more info about the timestamp interceptor:
> >> http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor
> >
> > Thanks this helps. The type specified in the example is a fully qualified
> > class name, however in the table it says use type as TIMESTAMP. I guess
> > example is not correct and I should just use TIMESTAMP.
> >>
> >> Regards, Kathleen
> >>
> >> On Wed, Aug 22, 2012 at 11:50 AM, Mohit Anchlia <[EMAIL PROTECTED]
> >
> >> wrote:
> >> > I see this JIRA https://issues.apache.org/jira/browse/FLUME-1215 but
> how
> >> > do
> >> > I take advantage of it? I am using 1.2.0 but %Y %m doesn't work. I
> just
> >> > get
> >> > number format exceptions.
> >> >
> >> >
> >> > On Tue, Aug 21, 2012 at 5:52 PM, Mohit Anchlia <
> [EMAIL PROTECTED]>
> >> > wrote:
> >> >>
> >> >> I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence.
> Do
> >> >> I
> >> >> need to write some custom interceptor? Could you please point me to
> an
> >> >> example? Currently my AvroClient looks like this:
> >> >>
> >> >>
> >> >> public void sendDataToFlume(String data) {
> >> >>
> >> >> // Create flume event object
> >> >>
> >> >> Event event = EventBuilder.withBody(data, Charset.forName(
> >> >>
> >> >> "UTF-8"));
> >> >>
> >> >> Map<String,String> headers > >> >>
> >> >> new HashMap<String,String>();
> >> >>
> >> >> headers.put(
> >> >>
> >> >> "host", hostName);
> >> >>
> >> >> event.setHeaders(headers);
> >> >>
> >> >> try {
> >> >>
> >> >> rpcClient.append(event);
> >> >>
> >> >> }
> >> >>
> >> >> catch (EventDeliveryException e) {
> >> >>
> >> >> connect();
> >> >>
> >> >> }
> >> >>
> >> >> }
> >> >
> >> >
> >
> >
>