Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Use of Flume for the sensor network data


Copy link to this message
-
Re: Use of Flume for the sensor network data
Mohammad Tariq 2012-07-23, 08:09
Hi mardan,

     You need to use the -c option with tour command to specify the
directory where your configuration file is kept. Just look at the the
other thread of yours.

Regards,
    Mohammad Tariq
On Mon, Jul 23, 2012 at 10:19 AM, mardan Khan <[EMAIL PROTECTED]> wrote:
> Dear Mohammad Tariq,
>
> Many thanks for your valuable information.
>
> For the testing purpose , i have installed flume on SuSE Linux system. when
> i have type the command $/etc/init.d/flume-ng-agent start. Then i have
> receive message that Starting Flume NG agent daemon (flume-ng-agent):
>
> I think this mean my flume agent is working properly. I have the following
> changes in configuration file according to your example. The configuration
> file as:
>
>
>
> agent.sources = seqGenSrc
> agent.channels = memoryChannel
> agent.sinks = loggerSink
>
> # For each one of the sources, the type is defined
> agent.sources.seqGenSrc.type = seq
>
> # The channel can be defined as follows.
> agent.sources.seqGenSrc.channels = memoryChannel
>
> # Each sink's type must be defined
> agent.sinks.loggerSink.type = logger
>
> #Specify the channel the sink should use
> agent.sinks.loggerSink.channel = memoryChannel
>
> # Each channel's type is defined.
> agent.channels.memoryChannel.type = memory
>
> # Other config values specific to each type of channel(sink or source)
> # can be defined as well
> # In this case, it specifies the capacity of the memory channel
> agent.channels.memoryChannel.capacity = 100
>
>
> agent1.sources = tail
> agent1.channels = MemoryChannel-2
> agent1.sinks = HDFS
>
> agent1.sources.tail.type = exec
> agent1.sources.tail.command = tail -F /var/log/flume-ng/flume-init.log
>
> agent1.sources.tail.channels = MemoryChannel-2
>
> agent1.sources.tail.interceptors = hostint
> agent1.sources.tail.interceptors.hostint.type > org.apache.flume.interceptor.HostInterceptor$Builder
> agent1.sources.tail.interceptors.hostint.preserverExisting = true
>
> agent1.sources.tail.interceptors.hostint.useIP = true
>
> agent1.sinks.HDFS.channel = MemoryChannel-2
> agent1.channels.MemoryChannel-2.type = memory
> agent1.sinks.HDFS.type =hdfs
> agent1.sinks.HDFS.hdfs.path = hdfs://134.83.35.24/user/mardan/
>
> agent1.sinks.HDFS.hdfs.file.Type = DataStream
> agent1.sinks.HDFS.hdfs.writeFormat = Text
>
>
>
> when i have type the following command
>
> $ /usr/bin/flume-ng agent -n agent1 -f /etc/flume-ng/conf/flume.conf
>
>
>
> I got the following warring / error  messages
>
> Warning: No configuration directory set! Use --conf <dir> to override.
> Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS access
> Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar from
> classpath
> Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE access
> Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/zookeeper/lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar from
> classpath
> Info: Excluding /usr/lib/hadoop/lib/slf4j-api-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar from classpath
> Info: Excluding /usr/lib/hadoop-hdfs/lib/slf4j-api-1.6.1.jar from classpath
> + exec /usr/java/jdk1.6.0_31/bin/java -Xmx20m -cp
> '/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/avro-1.5.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/commons-logging-api-1.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/hue-plugins-2.0.0-cdh4.0.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.