Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> HDFS escape sequence and use of timestamp header


Copy link to this message
-
Re: HDFS escape sequence and use of timestamp header
Mohit,

As it happens, both the example and the table are correct. You can
specify either the fully qualified class name or the alias 'TIMESTAMP'

Regards, Kathleen

On Wed, Aug 22, 2012 at 5:05 PM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
>
>
> On Wed, Aug 22, 2012 at 2:16 PM, Kathleen Ting <[EMAIL PROTECTED]> wrote:
>>
>> Hi Mohit,
>>
>> You can configure a timestamp interceptor onto your source as follows:
>>
>> agent.sources.src-0.interceptors.ts.type = TIMESTAMP
>>
>> Once the timestamp interceptor is in place, you can use, in sinks, the
>> timestamp it writes. Here is an example of using the timestamp in an
>> HDFS sink:
>>
>> agent.sinks.sink-0.hdfs.filePrefix = FlumeData.%Y-%m-%d
>>
>> Here is more info about the timestamp interceptor:
>> http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor
>
> Thanks this helps. The type specified in the example is a fully qualified
> class name, however in the table it says use type as TIMESTAMP. I guess
> example is not correct and I should just use TIMESTAMP.
>>
>> Regards, Kathleen
>>
>> On Wed, Aug 22, 2012 at 11:50 AM, Mohit Anchlia <[EMAIL PROTECTED]>
>> wrote:
>> > I see this JIRA https://issues.apache.org/jira/browse/FLUME-1215 but how
>> > do
>> > I take advantage of it? I am using 1.2.0 but %Y %m doesn't work. I just
>> > get
>> > number format exceptions.
>> >
>> >
>> > On Tue, Aug 21, 2012 at 5:52 PM, Mohit Anchlia <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence. Do
>> >> I
>> >> need to write some custom interceptor? Could you please point me to an
>> >> example? Currently my AvroClient looks like this:
>> >>
>> >>
>> >> public void sendDataToFlume(String data) {
>> >>
>> >> // Create flume event object
>> >>
>> >> Event event = EventBuilder.withBody(data, Charset.forName(
>> >>
>> >> "UTF-8"));
>> >>
>> >> Map<String,String> headers >> >>
>> >> new HashMap<String,String>();
>> >>
>> >> headers.put(
>> >>
>> >> "host", hostName);
>> >>
>> >> event.setHeaders(headers);
>> >>
>> >> try {
>> >>
>> >> rpcClient.append(event);
>> >>
>> >> }
>> >>
>> >> catch (EventDeliveryException e) {
>> >>
>> >> connect();
>> >>
>> >> }
>> >>
>> >> }
>> >
>> >
>
>