Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> SpoolDir sub-directories


Copy link to this message
-
RE: SpoolDir sub-directories
Will do. Thanks!

 

From: Brock Noland [mailto:[EMAIL PROTECTED]]
Sent: Monday, February 4, 2013 11:14 AM
To: [EMAIL PROTECTED]
Subject: Re: SpoolDir sub-directories

 

Hi,
I don't believe sub directories are supported at this time. Perhaps you
should file a JIRA for this improvement?

 

Brock

 

On Mon, Feb 4, 2013 at 10:01 AM, Tzur Turkenitz <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> > wrote:

Hi,

I couldn't find any information about this in the Wiki / User guide.

Are sub directories for the Flume SpoolDir supported in 1.3x?

Thanks!

 

On Thu, Jan 31, 2013 at 11:56 AM, Tzur Turkenitz <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> > wrote:

Hi All,

 

I am experimenting with Flumes` 1.3 new SpoolDir adapter.

I was wondering is there an option to tell the adapter to walk down
directories and search for files.

 

Currently, if I have a directory within the source location I get java
exceptions.

13/01/30 17:41:30 WARN avro.SpoolingFileLineReader: Could not find file:
/home/fl/source/dir

java.io.FileNotFoundException: /home/fl/source/dir (Is a directory)

at java.io.FileInputStream.open(Native Method)

at java.io.FileInputStream.<init>(FileInputStream.java:120)

at java.io.FileReader.<init>(FileReader.java:55)

at
org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFile
LineReader.java:322)

at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLi
neReader.java:172)

at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(Spoo
lDirectorySource.java:135)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
101(ScheduledThreadPoolExecutor.java:98)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
odic(ScheduledThreadPoolExecutor.java:180)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:204)

at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

at java.lang.Thread.run(Thread.java:662)

Thanks!

 

 

 

--

Regards,

Tzur Turkenitz

Vision.BI

 <http://www.vision.bi/> http://www.vision.bi/

 

"Facts are stubborn things, but statistics are more pliable"

-Mark Twain

 

--

Regards,

Tzur Turkenitz

Vision.BI

 <http://www.vision.bi/> http://www.vision.bi/

 

"Facts are stubborn things, but statistics are more pliable"

-Mark Twain

 

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB