Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Use of Flume for the sensor network data


Copy link to this message
-
Re: Use of Flume for the sensor network data
Mohammad Tariq 2012-07-22, 21:12
Hello Mardan,

        In order to aggregate data into your Hadoop cluster you need
to set up a Flume agent first. In order to do that you have to write a
config file having desired properties. An example file would be
somewhat like this :

agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS

agent1.sources.tail.type = exec
agent1.sources.tail.command = tail -F /var/log/apache2/access.log
agent1.sources.tail.channels = MemoryChannel-2

agent1.sources.tail.interceptors = hostint
agent1.sources.tail.interceptors.hostint.type org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.tail.interceptors.hostint.preserveExisting = true
agent1.sources.tail.interceptors.hostint.useIP = true

agent1.sinks.HDFS.channel = MemoryChannel-2
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/%{host}
agent1.sinks.HDFS.hdfs.file.Type = DataStream
agent1.sinks.HDFS.hdfs.writeFormat = Text

agent1.channels.MemoryChannel-2.type = memory

You can visit this link as the starting point, if you want -
http://cloudfront.blogspot.in/2012/06/how-to-build-and-use-flume-ng.html

And, it is quite possible to run Flume-1.x o windows. Here is a great
post by Alex on how to do that -
http://mapredit.blogspot.in/2012/07/run-flume-13x-on-windows.html

Hope it helps.

Regards,
    Mohammad Tariq
On Mon, Jul 23, 2012 at 2:17 AM, mardan Khan <[EMAIL PROTECTED]> wrote:
> Yeah, my cluster is always running. But i dont know how to setup the flume
> that directly stream the data to hadoop. I have must install the flume agent
> on window machine. As per my study the flume version-0.9.4 agent can install
> on window machine. Can we install flume version 1.x on window machine?
> If any one have done, please let me guide.
>
>
>
> Many thanks
>
>
>
> On Sun, Jul 22, 2012 at 7:26 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>>
>> NameNode and DataNode must be running if we need to write anything to the
>> Hdfs.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Sun, Jul 22, 2012 at 11:41 PM, Henry Larson <[EMAIL PROTECTED]>
>> wrote:
>> > You can have flume write to HDFS: however, do you have your hadoop
>> > cluster running all the time?
>
>