Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Fwd: Flume-NG 1.3.1 : Spooling dir source : java.io.IOException: Stream closed


Copy link to this message
-
Fwd: Flume-NG 1.3.1 : Spooling dir source : java.io.IOException: Stream closed
---------- Forwarded message ----------
From: NGuyen thi Kim Tuyen <[EMAIL PROTECTED]>
Date: Mon, Jan 28, 2013 at 2:14 PM
Subject: Flume-NG 1.3.1 : Spooling dir source : java.io.IOException: Stream
closed
To: [EMAIL PROTECTED]
Hi ,

Please help me .

I want to use Flume in the following case :
Spooling directory source --> FileChannel --> HBase sink . But I have some
problems with Spooling directory source :

Here is my test flume.conf :
t-game-db194.sources = test-hbase

t-game-db194.sinks = sink-hbase

t-game-db194.channels = hbase-channel

#source spoolDir
t-game-db194.sources.test-hbase.type = spooldir

t-game-db194.sources.test-hbase.spoolDir =/var/log/testhbase

t-game-db194.sources.test-hbase.fileHeader = true

t-game-db194.sources.test-hbase.channels = hbase-channel

#file Channel
t-game-db194.channels.hbase-channel.type = file

t-game-db194.channels.hbase-channel.checkpointDir /var/log/flume-ng/checkpoint

t-game-db194.channels.hbase-channel.dataDir = /var/log/flume-ng/filedata
#sink
t-game-db194.sinks.sink-hbase.type = logger

t-game-db194.sinks.sink-hbase.channel = hbase-channel

And I tested : echo "tuyen ssssssssss " >> "/var/log/testhbase/hbase_1.log"
. The first event is OK , but the next events are not work . Here is
flume.log

28 Jan 2013 13:16:47,424 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.source.SpoolDirectorySource.start:64)  -
SpoolDirectorySource source starting with directory:/var/log/testhbase
28 Jan 2013 13:16:47,732 INFO  [pool-7-thread-1]
(org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229)
 - Preparing to move file /var/log/testhbase/hbase_1.log to
/var/log/testhbase/hbase_1.log.COMPLETED
28 Jan 2013 13:16:48,436 INFO
 [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.LoggerSink.process:70)  - Event: {
headers:{file=/var/log/testhbase/hbase_1.log} body: 74 75 79 65 6E 20 73 73
73 73 73 73 73 73 73 73 tuyen ssssssssss }

28 Jan 2013 13:17:08,836 INFO  [pool-7-thread-1]
(org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229)
 - Preparing to move file /var/log/testhbase/hbase_1.log to
/var/log/testhbase/hbase_1.log.COMPLETED
28 Jan 2013 13:17:08,837 ERROR [pool-7-thread-1]
(org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148)
 - Uncaught exception in Runnable
java.lang.IllegalStateException: File name has been re-used with different
files. Spooling assumption violated for
/var/log/testhbase/hbase_1.log.COMPLETED
 at
org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:272)
 at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:185)
 at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
28 Jan 2013 13:17:09,340 ERROR [pool-7-thread-1]
(org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148)
 - Uncaught exception in Runnable
java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:97)
 at java.io.BufferedReader.readLine(BufferedReader.java:292)
 at java.io.BufferedReader.readLine(BufferedReader.java:362)
 at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180)
 at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
Are there more documents about Flume-ng spooling source  , beside
http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source ?

Could you please give me some advice ?

Nguyễn Thị Kim Tuyên
Computer Science Engineering
HCMC University Of Technology.

Nguyễn Thị Kim Tuyên
Computer Science Engineering
HCMC University Of Technology.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB