Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Apache flume 1.4.0 - Spooling directory issue


Copy link to this message
-
Apache flume 1.4.0 - Spooling directory issue
Hi

I am using spooling directory source with apache flume 1.4.0 and having
the problem
that *same configuration works on some machines and doesn't work on some
machines.*

Configuration used to work with flume 1.3.1. (Only the properties
related to deserializer are changed).

*Configuration is:*
agent.sources.spooler.type = spooldir
agent.sources.spooler.spoolDir = /TRACKING_TSV_FILES/tracking_backup_flume
agent.sources.spooler.batchSize = 10000
agent.sources.spooler.channels = fileChannel
agent.sources.spooler.spooldir.deserializer= LINE
agent.sources.spooler.spooldir.deserializer.maxLineLength =  20000
*Exception stack trace is: **
*java.nio.charset.MalformedInputException: Input length = 1
     at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
     at
org.apache.flume.serialization.ResettableFileInputStream.readChar(ResettableFileInputStream.java:169)

     at
org.apache.flume.serialization.LineDeserializer.readLine(LineDeserializer.java:134)

     at
org.apache.flume.serialization.LineDeserializer.readEvent(LineDeserializer.java:72)

     at
org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:91)

     at
org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221)

     at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:160)

     at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

Is there any software/libraries dependency for it to work?

Regards,
Jagadish

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB