Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - using tail to HDFS sink


Copy link to this message
-
Re: using tail to HDFS sink
Mohammad Tariq 2012-07-12, 11:00
You can follow this link if you want to have a look on how to use
%{host} - http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html...I
was also struggling initially, so thought of writing down the things I
did.

Regards,
    Mohammad Tariq
On Thu, Jul 12, 2012 at 4:01 PM, Juhani Connolly
<[EMAIL PROTECTED]> wrote:
> Most people in the US are asleep at this time so I wouldn't expect a fast
> response. People will respond when they have time and are able to.
>
> As to your problem, as I said earlier on, unless you've changed your config
> since then, you don't have a host header defined. The logs don't seem to
> show anything wrong, so I suspect that it may be sending data to your hdfs
> without replacing the %{host} with your actual host. Remove the %{host} and
> replace it with a hardcoded string. If that works, you know that is your
> problem. Try using the HostInterceptor to add a host header:
> http://people.apache.org/~juhanic/flume-docs/FlumeUserGuide.html#host-interceptor
>
> There is also a pair of quotation marks in your path string... is that
> intended? agent2.sinks.HDFS.hdfs.path > hdfs://10.5.114.110:54310/user/flume/'%{host}'
>
> I can't remember what happens when you  try to use a header that doesn't
> exist... Probably gets replaced by an empty string as it's not throwing
> exceptions.
> Try checking hdfs in /user/flume and see if there is a poorly named
> directory there with your data.
>
>
> On 07/12/2012 06:41 PM, prabhu k wrote:
>
> can you please any one respond on the below issue?
>
> On Thu, Jul 12, 2012 at 12:57 PM, prabhu k <[EMAIL PROTECTED]> wrote:
>>
>> As per mohammad suggestion i have executed like below.
>>
>>
>> root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
>> bin/flume-ng agent -n agent2 -c /conf -f conf/agent2.conf
>>
>>
>> Info: Including Hadoop libraries found via
>> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
>> Info: Excluding
>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from
>> classpath
>> Info: Excluding
>> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
>> classpath
>> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp
>> '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/!
>> commons-da
>> emon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_!