Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume/HDFS Encoding


Copy link to this message
-
Re: Flume/HDFS Encoding
Did you fix the parameter naming problem I described in the earlier message?

On Fri, Dec 14, 2012 at 2:59 PM, Cormier, Christopher
<[EMAIL PROTECTED]> wrote:
> Thanks Brock,
> When I run it as a DataStream I still get some weird characters between records.
>
> [DATA_HERE]ÿÿÿÿ×ùÎ0ÆÜ9Ig::¬                  ;)
>  [DATA_HERE]ÿÿÿÿ×ùÎ0ÆÜ9Ig::¬
>                                          ;)
>                                            Î[DATA_HERE]ÿÿÿÿ×ùÎ0ÆÜ9Ig::¬
>                                                                                                                                                                                                            ;0
>                                                                                                                                                                                                              ½[DATA_HERE]ÿÿÿÿ×ùÎ0ÆÜ9Ig::¬
> ;0
> :[DATA_HERE]
>
> I was hoping to avoid the ÿÿÿÿ and spaces (I'm assuming they're characters that are encoded such that -cat won't show them).
>
> Any thoughts?
>
> Thanks again,
>
> Chris
>
> -----Original Message-----
> From: Brock Noland [mailto:[EMAIL PROTECTED]]
> Sent: Friday, December 14, 2012 3:52 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Flume/HDFS Encoding
>
> Hi,
>
> On Fri, Dec 14, 2012 at 2:48 PM, Cormier, Christopher <[EMAIL PROTECTED]> wrote:
>> SEQ!org.apache.hadoop.io.LongWritableorg.apache.hadoop.io.TextY]
>> õpµ^R÷ﳬÕ
>>
>
> This is a SequenceFile.
>
>>
>> requestToHDFS.sinks.HDFS.hdfs.file.Type = DataStream
>>
>> # also tried...
>>
>> #requestToHDFS.sinks.HDFS.hdfs.file.Type = SequenceFile
>>
>
> The parameter is hdfs.fileType. See here:
>
> http://flume.apache.org/FlumeUserGuide.html#hdfs-sink
>
> It sounds like you want a text file so you should use DataStream.
>
> Brock

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/