Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume and HDFS integration


Copy link to this message
-
Re: Flume and HDFS integration
On Fri, Nov 30, 2012 at 12:51 AM, Emile Kao <[EMAIL PROTECTED]> wrote:
> Hello Brock,
> first of all thank you for answering my questions. I appreciate it since I am a real newbie in Flume / Hadoop , etc...
>
> But now I am confused. According to you statement, the filetype is the key here. Now just take a look on my flume.conf below:
> The filetype was from set to "DataStream".
> Now which is the right one now: SequenceFile, DataStream or CompressedStream?

Here's what works for me in the situation very similar to yours:

# Sink configuration
agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = /flume/cluster-logs
agent.sinks.sink1.hdfs.writeFormat = Text
agent.sinks.sink1.hdfs.fileType = DataStream
agent.sinks.sink1.hdfs.filePrefix = events-
agent.sinks.sink1.hdfs.round = true
agent.sinks.sink1.hdfs.roundValue = 10
agent.sinks.sink1.hdfs.roundUnit = minute
# agent.sinks.sink1.hdfs.serializer org.apache.flume.serialization.BodyTextEventSerializer

Thanks,
Roman.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB