Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> SpoolDir sub-directories


Copy link to this message
-
Re: SpoolDir sub-directories
Hi,

I don't believe sub directories are supported at this time. Perhaps you
should file a JIRA for this improvement?

Brock
On Mon, Feb 4, 2013 at 10:01 AM, Tzur Turkenitz <[EMAIL PROTECTED]> wrote:

> Hi,
> I couldn't find any information about this in the Wiki / User guide.
> Are sub directories for the Flume SpoolDir supported in 1.3x?
> Thanks!
>
>
> On Thu, Jan 31, 2013 at 11:56 AM, Tzur Turkenitz <[EMAIL PROTECTED]> wrote:
>
>> Hi All,
>>
>> I am experimenting with Flumes` 1.3 new SpoolDir adapter.
>> I was wondering is there an option to tell the adapter to walk down
>> directories and search for files.
>>
>> Currently, if I have a directory within the source location I get java
>> exceptions.
>> 13/01/30 17:41:30 WARN avro.SpoolingFileLineReader: Could not find file:
>> /home/fl/source/dir
>> java.io.FileNotFoundException: /home/fl/source/dir (Is a directory)
>>  at java.io.FileInputStream.open(Native Method)
>> at java.io.FileInputStream.<init>(FileInputStream.java:120)
>>  at java.io.FileReader.<init>(FileReader.java:55)
>> at
>> org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
>>  at
>> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
>> at
>> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
>>  at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>> Thanks!
>>
>>
>>
>> --
>>  Regards,
>> Tzur Turkenitz
>> Vision.BI
>> http://www.vision.bi/
>>
>> "*Facts are stubborn things, but statistics are more pliable*"
>> -Mark Twain
>>
>
>
>
> --
> Regards,
> Tzur Turkenitz
> Vision.BI
> http://www.vision.bi/
>
> "*Facts are stubborn things, but statistics are more pliable*"
> -Mark Twain
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/