Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG


Copy link to this message
-
Re: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG
To use the date related escape sequences you need to provide a timestamp
header.

The easiest way to do this would be configuring a timestamp interceptor
onto your source, ours looks something like:

sender1.sources.tail.interceptors = ts
sender1.sources.tail.interceptors.ts.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
On 07/04/2012 11:00 AM, Shara Shi wrote:
>
> HI
>
> I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs.
>
> I got following error messages
>
> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>
> java.lang.NumberFormatException: null
>
>         at java.lang.Long.parseLong(Long.java:375)
>
>         at java.lang.Long.valueOf(Long.java:525)
>
>         at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
>         at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
>
> org.apache.flume.EventDeliveryException:
> java.lang.NumberFormatException: null
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.lang.NumberFormatException: null
>
>         at java.lang.Long.parseLong(Long.java:375)
>
>         at java.lang.Long.valueOf(Long.java:525)
>
>         at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
>         at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
>         ... 3 more
>
> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>
> java.lang.NumberFormatException: null
>
>         at java.lang.Long.parseLong(Long.java:375)
>
>         at java.lang.Long.valueOf(Long.java:525)
>
>         at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
>         at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
>
> org.apache.flume.EventDeliveryException:
> java.lang.NumberFormatException: null
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
>         at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.lang.NumberFormatException: null
>
>         at java.lang.Long.parseLong(Long.java:375)
>
>         at java.lang.Long.valueOf(Long.java:525)
>
>         at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
>         at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
>         ... 3 more
>
> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB