Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG


Copy link to this message
-
Re: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG
To use the date related escape sequences you need to provide a timestamp
header.

The easiest way to do this would be configuring a timestamp interceptor
onto your source, ours looks something like:

sender1.sources.tail.interceptors = ts
sender1.sources.tail.interceptors.ts.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
On 07/04/2012 11:00 AM, Shara Shi wrote:
>
> HI
>
> I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs.
>
> I got following error messages
>
> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>
> java.lang.NumberFormatException: null
>
>         at java.lang.Long.parseLong(Long.java:375)
>
>         at java.lang.Long.valueOf(Long.java:525)
>
>         at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
>         at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
>
> org.apache.flume.EventDeliveryException:
> java.lang.NumberFormatException: null
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.lang.NumberFormatException: null
>
>         at java.lang.Long.parseLong(Long.java:375)
>
>         at java.lang.Long.valueOf(Long.java:525)
>
>         at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
>         at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
>         ... 3 more
>
> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>
> java.lang.NumberFormatException: null
>
>         at java.lang.Long.parseLong(Long.java:375)
>
>         at java.lang.Long.valueOf(Long.java:525)
>
>         at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
>         at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
>
> org.apache.flume.EventDeliveryException:
> java.lang.NumberFormatException: null
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.lang.NumberFormatException: null
>
>         at java.lang.Long.parseLong(Long.java:375)
>
>         at java.lang.Long.valueOf(Long.java:525)
>
>         at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
>         at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
>         ... 3 more
>
> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>