Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Apache flume 1.4.0 - Spooling directory issue


Copy link to this message
-
Apache flume 1.4.0 - Spooling directory issue
Hi

I am using spooling directory source with apache flume 1.4.0 and having
the problem
that *same configuration works on some machines and doesn't work on some
machines.*

Configuration used to work with flume 1.3.1. (Only the properties
related to deserializer are changed).

*Configuration is:*
agent.sources.spooler.type = spooldir
agent.sources.spooler.spoolDir = /TRACKING_TSV_FILES/tracking_backup_flume
agent.sources.spooler.batchSize = 10000
agent.sources.spooler.channels = fileChannel
agent.sources.spooler.spooldir.deserializer= LINE
agent.sources.spooler.spooldir.deserializer.maxLineLength =  20000
*Exception stack trace is: **
*java.nio.charset.MalformedInputException: Input length = 1
     at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
     at
org.apache.flume.serialization.ResettableFileInputStream.readChar(ResettableFileInputStream.java:169)

     at
org.apache.flume.serialization.LineDeserializer.readLine(LineDeserializer.java:134)

     at
org.apache.flume.serialization.LineDeserializer.readEvent(LineDeserializer.java:72)

     at
org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:91)

     at
org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221)

     at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:160)

     at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

Is there any software/libraries dependency for it to work?

Regards,
Jagadish