Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - SpoolDir sub-directories


Copy link to this message
-
RE: SpoolDir sub-directories
Tzur Turkenitz 2013-02-04, 16:41
Will do. Thanks!

 

From: Brock Noland [mailto:[EMAIL PROTECTED]]
Sent: Monday, February 4, 2013 11:14 AM
To: [EMAIL PROTECTED]
Subject: Re: SpoolDir sub-directories

 

Hi,
I don't believe sub directories are supported at this time. Perhaps you
should file a JIRA for this improvement?

 

Brock

 

On Mon, Feb 4, 2013 at 10:01 AM, Tzur Turkenitz <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> > wrote:

Hi,

I couldn't find any information about this in the Wiki / User guide.

Are sub directories for the Flume SpoolDir supported in 1.3x?

Thanks!

 

On Thu, Jan 31, 2013 at 11:56 AM, Tzur Turkenitz <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> > wrote:

Hi All,

 

I am experimenting with Flumes` 1.3 new SpoolDir adapter.

I was wondering is there an option to tell the adapter to walk down
directories and search for files.

 

Currently, if I have a directory within the source location I get java
exceptions.

13/01/30 17:41:30 WARN avro.SpoolingFileLineReader: Could not find file:
/home/fl/source/dir

java.io.FileNotFoundException: /home/fl/source/dir (Is a directory)

at java.io.FileInputStream.open(Native Method)

at java.io.FileInputStream.<init>(FileInputStream.java:120)

at java.io.FileReader.<init>(FileReader.java:55)

at
org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFile
LineReader.java:322)

at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLi
neReader.java:172)

at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(Spoo
lDirectorySource.java:135)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
101(ScheduledThreadPoolExecutor.java:98)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
odic(ScheduledThreadPoolExecutor.java:180)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:204)

at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

at java.lang.Thread.run(Thread.java:662)

Thanks!

 

 

 

--

Regards,

Tzur Turkenitz

Vision.BI

 <http://www.vision.bi/> http://www.vision.bi/

 

"Facts are stubborn things, but statistics are more pliable"

-Mark Twain

 

--

Regards,

Tzur Turkenitz

Vision.BI

 <http://www.vision.bi/> http://www.vision.bi/

 

"Facts are stubborn things, but statistics are more pliable"

-Mark Twain

 

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/