Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG


Copy link to this message
-
Re: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG
Right, you can specify the FQCN, or the alias 'TIMESTAMP'

Cheers,
Will

On Jul 3, 2012, at 7:19 PM, Juhani Connolly <[EMAIL PROTECTED]> wrote:

> To use the date related escape sequences you need to provide a timestamp header.
>
> The easiest way to do this would be configuring a timestamp interceptor onto your source, ours looks something like:
>
> sender1.sources.tail.interceptors = ts
> sender1.sources.tail.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
>
>
> On 07/04/2012 11:00 AM, Shara Shi wrote:
>> HI
>>  
>> I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs.
>> I got following error messages
>>  
>> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>> java.lang.NumberFormatException: null
>>         at java.lang.Long.parseLong(Long.java:375)
>>         at java.lang.Long.valueOf(Long.java:525)
>>         at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>         at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
>> org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.NumberFormatException: null
>>         at java.lang.Long.parseLong(Long.java:375)
>>         at java.lang.Long.valueOf(Long.java:525)
>>         at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>         at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>         ... 3 more
>> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>> java.lang.NumberFormatException: null
>>         at java.lang.Long.parseLong(Long.java:375)
>>         at java.lang.Long.valueOf(Long.java:525)
>>         at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>         at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
>> org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.NumberFormatException: null
>>         at java.lang.Long.parseLong(Long.java:375)
>>         at java.lang.Long.valueOf(Long.java:525)
>>         at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>         at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>         ... 3 more
>> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB