Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Problems with time variables in HDFS path


Copy link to this message
-
Problems with time variables in HDFS path
Hi,

we are running into a strange problem using Flume-NG 1.10 from CDH 4.0.1.

Setup:
Flume-NG opens a TCP syslog port, collects all messages and forwards them directly into HDFS. This works fine until the point where we want to forward MS IIS Logs in W3C format. The reason seems to be a " - " inside the log message. I could reproduce the problem using rsyslogd forwarding all syslog messages to flume:

logger "Hello this is a test" => Works fine :)

logger "hello - this will break" => breaks flume :(

If I remove the time variables from the HDFS path in our configuration (attached) everything is working fine...

Exception:

2012-07-11 11:08:18,292 ERROR hdfs.HDFSEventSink: process failed
java.lang.NumberFormatException: null
        at java.lang.Long.parseLong(Long.java:375)
        at java.lang.Long.valueOf(Long.java:525)
        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)
2012-07-11 11:08:18,294 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NumberFormatException: null
        at java.lang.Long.parseLong(Long.java:375)
        at java.lang.Long.valueOf(Long.java:525)
        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
        ... 3 more

I attached our configuration in case something is broken in there.

Best regards,

Christian Schroer

--
AutoScout24 GmbH
Dingolfinger Str. 1-15
81673 München

E-Mail    [EMAIL PROTECTED]
Internet www.autoscout24.de
________________________________________________________________________________________
Geschäftsführer: Alberto Sanz de Lama, André Stark
Vorsitzender des Beirats: Dr. Martin Enderle | Amtsgericht München HRB 128 701