Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Flume and HDFS integration


Copy link to this message
-
Flume and HDFS integration
Emile Kao 2012-11-29, 17:14
Dear support,
I would like to ask you some questions about issues I am facing trying to implement Flume in a customer environment.

I am using following release of Flume: apache-flume-1.4.0-SNAPSHOT-bin

Here are my questions:  

Question no.1
I have define the following command in flume.conf:
agent1.sources.tail.command = tail -F /opt/apache2/logs/access_log

Now the resulting files (FlumeData.xxxxxxxxxxxxx) are not readable. At least for human being. I guess they are in binary format. Now , my question is: is there a way to make or convert those files in an ascii / readable format for human being?

Question no.2
I am trying to use the tailDir command without success. Here is the setting in flume,conf:
agent1.sources.tail.command = tailDir("/opt/apache2/logs/")
--> here is what I am getting as result...Can you help?

2012-11-29 16:48:17,548 (pool-6-thread-1) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:284)] Failed while running command: tailDir("/opt/apache2/logs/")
java.io.IOException: Cannot run program "tailDir("/opt/apache2/logs/")": java.io.IOException: error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:259)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(ProcessImpl.java:65)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
    ... 7 more
2012-11-29 16:48:17,549 (pool-6-thread-1) [INFO - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:307)] Command [tailDir("/opt/apache2/logs/")] exited with -1073741824
Many Thank!