Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Flume and HDFS integration


Copy link to this message
-
Flume and HDFS integration
Dear support,
I would like to ask you some questions about issues I am facing trying to implement Flume in a customer environment.

I am using following release of Flume: apache-flume-1.4.0-SNAPSHOT-bin

Here are my questions:  

Question no.1
I have define the following command in flume.conf:
agent1.sources.tail.command = tail -F /opt/apache2/logs/access_log

Now the resulting files (FlumeData.xxxxxxxxxxxxx) are not readable. At least for human being. I guess they are in binary format. Now , my question is: is there a way to make or convert those files in an ascii / readable format for human being?

Question no.2
I am trying to use the tailDir command without success. Here is the setting in flume,conf:
agent1.sources.tail.command = tailDir("/opt/apache2/logs/")
--> here is what I am getting as result...Can you help?

2012-11-29 16:48:17,548 (pool-6-thread-1) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:284)] Failed while running command: tailDir("/opt/apache2/logs/")
java.io.IOException: Cannot run program "tailDir("/opt/apache2/logs/")": java.io.IOException: error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:259)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(ProcessImpl.java:65)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
    ... 7 more
2012-11-29 16:48:17,549 (pool-6-thread-1) [INFO - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:307)] Command [tailDir("/opt/apache2/logs/")] exited with -1073741824
Many Thank!
+
Brock Noland 2012-11-29, 17:18
+
Roman Shaposhnik 2012-11-30, 01:17
+
Brock Noland 2012-11-30, 01:26
+
Emile Kao 2012-11-30, 08:51
+
Roman Shaposhnik 2012-12-01, 00:40
+
Brock Noland 2012-11-30, 12:40
+
Emile Kao 2012-12-03, 09:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB