Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> SpoolDir sub-directories


+
Tzur Turkenitz 2013-01-31, 16:56
+
Tzur Turkenitz 2013-02-04, 16:01
Copy link to this message
-
Re: SpoolDir sub-directories
Hi,

I don't believe sub directories are supported at this time. Perhaps you
should file a JIRA for this improvement?

Brock
On Mon, Feb 4, 2013 at 10:01 AM, Tzur Turkenitz <[EMAIL PROTECTED]> wrote:

> Hi,
> I couldn't find any information about this in the Wiki / User guide.
> Are sub directories for the Flume SpoolDir supported in 1.3x?
> Thanks!
>
>
> On Thu, Jan 31, 2013 at 11:56 AM, Tzur Turkenitz <[EMAIL PROTECTED]> wrote:
>
>> Hi All,
>>
>> I am experimenting with Flumes` 1.3 new SpoolDir adapter.
>> I was wondering is there an option to tell the adapter to walk down
>> directories and search for files.
>>
>> Currently, if I have a directory within the source location I get java
>> exceptions.
>> 13/01/30 17:41:30 WARN avro.SpoolingFileLineReader: Could not find file:
>> /home/fl/source/dir
>> java.io.FileNotFoundException: /home/fl/source/dir (Is a directory)
>>  at java.io.FileInputStream.open(Native Method)
>> at java.io.FileInputStream.<init>(FileInputStream.java:120)
>>  at java.io.FileReader.<init>(FileReader.java:55)
>> at
>> org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
>>  at
>> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
>> at
>> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
>>  at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>> Thanks!
>>
>>
>>
>> --
>>  Regards,
>> Tzur Turkenitz
>> Vision.BI
>> http://www.vision.bi/
>>
>> "*Facts are stubborn things, but statistics are more pliable*"
>> -Mark Twain
>>
>
>
>
> --
> Regards,
> Tzur Turkenitz
> Vision.BI
> http://www.vision.bi/
>
> "*Facts are stubborn things, but statistics are more pliable*"
> -Mark Twain
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Tzur Turkenitz 2013-02-04, 16:41
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB