Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Flume and HDFS integration


+
Emile Kao 2012-11-29, 17:14
Copy link to this message
-
Re: Flume and HDFS integration
HI,

1) It's a sequence file, you can change it a text file if you want. See
FileType here http://flume.apache.org/FlumeUserGuide.html#hdfs-sink

2) The "tailDir(" syntax is Flume 0.9 and is no long used. The first
example has the correct syntax.

Brock

On Thu, Nov 29, 2012 at 11:14 AM, Emile Kao <[EMAIL PROTECTED]> wrote:

> Dear support,
> I would like to ask you some questions about issues I am facing trying to
> implement Flume in a customer environment.
>
> I am using following release of Flume: apache-flume-1.4.0-SNAPSHOT-bin
>
> Here are my questions:
>
> Question no.1
> I have define the following command in flume.conf:
> agent1.sources.tail.command = tail -F /opt/apache2/logs/access_log
>
> Now the resulting files (FlumeData.xxxxxxxxxxxxx) are not readable. At
> least for human being. I guess they are in binary format. Now , my question
> is: is there a way to make or convert those files in an ascii / readable
> format for human being?
>
>
>
> Question no.2
> I am trying to use the tailDir command without success. Here is the
> setting in flume,conf:
> agent1.sources.tail.command = tailDir("/opt/apache2/logs/")
>
>
> --> here is what I am getting as result...Can you help?
>
> 2012-11-29 16:48:17,548 (pool-6-thread-1) [ERROR -
> org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:284)]
> Failed while running command: tailDir("/opt/apache2/logs/")
> java.io.IOException: Cannot run program "tailDir("/opt/apache2/logs/")":
> java.io.IOException: error=2, No such file or directory
>     at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>     at
> org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:259)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such file
> or directory
>     at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>     at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>     at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>     ... 7 more
> 2012-11-29 16:48:17,549 (pool-6-thread-1) [INFO -
> org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:307)]
> Command [tailDir("/opt/apache2/logs/")] exited with -1073741824
>
>
> Many Thank!
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Roman Shaposhnik 2012-11-30, 01:17
+
Brock Noland 2012-11-30, 01:26
+
Emile Kao 2012-11-30, 08:51
+
Roman Shaposhnik 2012-12-01, 00:40
+
Brock Noland 2012-11-30, 12:40
+
Emile Kao 2012-12-03, 09:51