Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Flume and HDFS integration


+
Emile Kao 2012-11-29, 17:14
Copy link to this message
-
Re: Flume and HDFS integration
HI,

1) It's a sequence file, you can change it a text file if you want. See
FileType here http://flume.apache.org/FlumeUserGuide.html#hdfs-sink

2) The "tailDir(" syntax is Flume 0.9 and is no long used. The first
example has the correct syntax.

Brock

On Thu, Nov 29, 2012 at 11:14 AM, Emile Kao <[EMAIL PROTECTED]> wrote:

> Dear support,
> I would like to ask you some questions about issues I am facing trying to
> implement Flume in a customer environment.
>
> I am using following release of Flume: apache-flume-1.4.0-SNAPSHOT-bin
>
> Here are my questions:
>
> Question no.1
> I have define the following command in flume.conf:
> agent1.sources.tail.command = tail -F /opt/apache2/logs/access_log
>
> Now the resulting files (FlumeData.xxxxxxxxxxxxx) are not readable. At
> least for human being. I guess they are in binary format. Now , my question
> is: is there a way to make or convert those files in an ascii / readable
> format for human being?
>
>
>
> Question no.2
> I am trying to use the tailDir command without success. Here is the
> setting in flume,conf:
> agent1.sources.tail.command = tailDir("/opt/apache2/logs/")
>
>
> --> here is what I am getting as result...Can you help?
>
> 2012-11-29 16:48:17,548 (pool-6-thread-1) [ERROR -
> org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:284)]
> Failed while running command: tailDir("/opt/apache2/logs/")
> java.io.IOException: Cannot run program "tailDir("/opt/apache2/logs/")":
> java.io.IOException: error=2, No such file or directory
>     at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>     at
> org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:259)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such file
> or directory
>     at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>     at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>     at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>     ... 7 more
> 2012-11-29 16:48:17,549 (pool-6-thread-1) [INFO -
> org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:307)]
> Command [tailDir("/opt/apache2/logs/")] exited with -1073741824
>
>
> Many Thank!
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Roman Shaposhnik 2012-11-30, 01:17
+
Brock Noland 2012-11-30, 01:26
+
Emile Kao 2012-11-30, 08:51
+
Roman Shaposhnik 2012-12-01, 00:40
+
Brock Noland 2012-11-30, 12:40
+
Emile Kao 2012-12-03, 09:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB