Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Spooling Directory Source Stuck in Exception [Serializer has been closed]


+
Snehal Nagmote 2013-10-28, 19:05
Copy link to this message
-
Re: Spooling Directory Source Stuck in Exception [Serializer has been closed]
Snehal,
  The Sppoling Dir source requires that the files in the directory it is
monitoring to consume are not be written/appended to. Basically once the
files are completed they need to be moved into the directory monitored by
SpoolingDir source. I suspect your setup is appending to the files that are
being consumed by this source.
-roshan
On Mon, Oct 28, 2013 at 12:05 PM, Snehal Nagmote
<[EMAIL PROTECTED]>wrote:

> Hi All,
>
> I am using spooling directory source With Flume 1.4.0 (HDP) . Nginx is set
> up in a way that it rotates logs into spool directory every 15 minutes.
>
> I am seeing this exception quite often with flume agents
>
> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:173)
>  - Uncaught exception in Runnable
> java.lang.IllegalStateException: Serializer has been closed
> at
> org.apache.flume.serialization.LineDeserializer.ensureOpen(LineDeserializer.java:124)
> at
> org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:88)
>  at
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:160)
>  at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>
> and it just get stuck and needs Flume restart . I found these two tickets
> describing same issue,
>
> https://issues.apache.org/jira/browse/FLUME-2160
>
> https://issues.apache.org/jira/browse/FLUME-1887
>
> Has anyone faced similar issues and is there any workaround to avoid this ?
>
>
>
> Thanks,
> Snehal
>
>
>
>
>
>
>
>
>

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
+
Snehal Nagmote 2013-10-28, 22:42
+
Christopher Surage 2013-10-30, 13:47
+
Snehal Nagmote 2013-10-30, 17:49
+
Brock Noland 2013-10-30, 18:07
+
Snehal Nagmote 2013-10-31, 18:19