Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume and HDFS integration


Copy link to this message
-
Re: Flume and HDFS integration
On Fri, Nov 30, 2012 at 12:51 AM, Emile Kao <[EMAIL PROTECTED]> wrote:
> Hello Brock,
> first of all thank you for answering my questions. I appreciate it since I am a real newbie in Flume / Hadoop , etc...
>
> But now I am confused. According to you statement, the filetype is the key here. Now just take a look on my flume.conf below:
> The filetype was from set to "DataStream".
> Now which is the right one now: SequenceFile, DataStream or CompressedStream?

Here's what works for me in the situation very similar to yours:

# Sink configuration
agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = /flume/cluster-logs
agent.sinks.sink1.hdfs.writeFormat = Text
agent.sinks.sink1.hdfs.fileType = DataStream
agent.sinks.sink1.hdfs.filePrefix = events-
agent.sinks.sink1.hdfs.round = true
agent.sinks.sink1.hdfs.roundValue = 10
agent.sinks.sink1.hdfs.roundUnit = minute
# agent.sinks.sink1.hdfs.serializer org.apache.flume.serialization.BodyTextEventSerializer

Thanks,
Roman.