Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Flume and HDFS integration


+
Emile Kao 2012-11-29, 17:14
+
Brock Noland 2012-11-29, 17:18
+
Roman Shaposhnik 2012-11-30, 01:17
+
Brock Noland 2012-11-30, 01:26
+
Emile Kao 2012-11-30, 08:51
Copy link to this message
-
Re: Flume and HDFS integration
On Fri, Nov 30, 2012 at 12:51 AM, Emile Kao <[EMAIL PROTECTED]> wrote:
> Hello Brock,
> first of all thank you for answering my questions. I appreciate it since I am a real newbie in Flume / Hadoop , etc...
>
> But now I am confused. According to you statement, the filetype is the key here. Now just take a look on my flume.conf below:
> The filetype was from set to "DataStream".
> Now which is the right one now: SequenceFile, DataStream or CompressedStream?

Here's what works for me in the situation very similar to yours:

# Sink configuration
agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = /flume/cluster-logs
agent.sinks.sink1.hdfs.writeFormat = Text
agent.sinks.sink1.hdfs.fileType = DataStream
agent.sinks.sink1.hdfs.filePrefix = events-
agent.sinks.sink1.hdfs.round = true
agent.sinks.sink1.hdfs.roundValue = 10
agent.sinks.sink1.hdfs.roundUnit = minute
# agent.sinks.sink1.hdfs.serializer org.apache.flume.serialization.BodyTextEventSerializer

Thanks,
Roman.
+
Brock Noland 2012-11-30, 12:40
+
Emile Kao 2012-12-03, 09:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB