Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - HDFS escape sequence and use of timestamp header


Copy link to this message
-
Re: HDFS escape sequence and use of timestamp header
Kathleen Ting 2012-08-23, 00:52
Mohit,

As it happens, both the example and the table are correct. You can
specify either the fully qualified class name or the alias 'TIMESTAMP'

Regards, Kathleen

On Wed, Aug 22, 2012 at 5:05 PM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
>
>
> On Wed, Aug 22, 2012 at 2:16 PM, Kathleen Ting <[EMAIL PROTECTED]> wrote:
>>
>> Hi Mohit,
>>
>> You can configure a timestamp interceptor onto your source as follows:
>>
>> agent.sources.src-0.interceptors.ts.type = TIMESTAMP
>>
>> Once the timestamp interceptor is in place, you can use, in sinks, the
>> timestamp it writes. Here is an example of using the timestamp in an
>> HDFS sink:
>>
>> agent.sinks.sink-0.hdfs.filePrefix = FlumeData.%Y-%m-%d
>>
>> Here is more info about the timestamp interceptor:
>> http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor
>
> Thanks this helps. The type specified in the example is a fully qualified
> class name, however in the table it says use type as TIMESTAMP. I guess
> example is not correct and I should just use TIMESTAMP.
>>
>> Regards, Kathleen
>>
>> On Wed, Aug 22, 2012 at 11:50 AM, Mohit Anchlia <[EMAIL PROTECTED]>
>> wrote:
>> > I see this JIRA https://issues.apache.org/jira/browse/FLUME-1215 but how
>> > do
>> > I take advantage of it? I am using 1.2.0 but %Y %m doesn't work. I just
>> > get
>> > number format exceptions.
>> >
>> >
>> > On Tue, Aug 21, 2012 at 5:52 PM, Mohit Anchlia <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence. Do
>> >> I
>> >> need to write some custom interceptor? Could you please point me to an
>> >> example? Currently my AvroClient looks like this:
>> >>
>> >>
>> >> public void sendDataToFlume(String data) {
>> >>
>> >> // Create flume event object
>> >>
>> >> Event event = EventBuilder.withBody(data, Charset.forName(
>> >>
>> >> "UTF-8"));
>> >>
>> >> Map<String,String> headers >> >>
>> >> new HashMap<String,String>();
>> >>
>> >> headers.put(
>> >>
>> >> "host", hostName);
>> >>
>> >> event.setHeaders(headers);
>> >>
>> >> try {
>> >>
>> >> rpcClient.append(event);
>> >>
>> >> }
>> >>
>> >> catch (EventDeliveryException e) {
>> >>
>> >> connect();
>> >>
>> >> }
>> >>
>> >> }
>> >
>> >
>
>