Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Use of Flume for the sensor network data


Copy link to this message
-
Re: Use of Flume for the sensor network data
Hello Mardan,

        In order to aggregate data into your Hadoop cluster you need
to set up a Flume agent first. In order to do that you have to write a
config file having desired properties. An example file would be
somewhat like this :

agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS

agent1.sources.tail.type = exec
agent1.sources.tail.command = tail -F /var/log/apache2/access.log
agent1.sources.tail.channels = MemoryChannel-2

agent1.sources.tail.interceptors = hostint
agent1.sources.tail.interceptors.hostint.type org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.tail.interceptors.hostint.preserveExisting = true
agent1.sources.tail.interceptors.hostint.useIP = true

agent1.sinks.HDFS.channel = MemoryChannel-2
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/%{host}
agent1.sinks.HDFS.hdfs.file.Type = DataStream
agent1.sinks.HDFS.hdfs.writeFormat = Text

agent1.channels.MemoryChannel-2.type = memory

You can visit this link as the starting point, if you want -
http://cloudfront.blogspot.in/2012/06/how-to-build-and-use-flume-ng.html

And, it is quite possible to run Flume-1.x o windows. Here is a great
post by Alex on how to do that -
http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html

Hope it helps.

Regards,
    Mohammad Tariq
On Mon, Jul 23, 2012 at 2:17 AM, mardan Khan <[EMAIL PROTECTED]> wrote:
> Yeah, my cluster is always running. But i dont know how to setup the flume
> that directly stream the data to hadoop. I have must install the flume agent
> on window machine. As per my study the flume version-0.9.4 agent can install
> on window machine. Can we install flume version 1.x on window machine?
> If any one have done, please let me guide.
>
>
>
> Many thanks
>
>
>
> On Sun, Jul 22, 2012 at 7:26 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>
>> NameNode and DataNode must be running if we need to write anything to the
>> Hdfs.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Sun, Jul 22, 2012 at 11:41 PM, Henry Larson <[EMAIL PROTECTED]>
>> wrote:
>> > You can have flume write to HDFS: however, do you have your hadoop
>> > cluster running all the time?
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB