Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - flume ng error while going for hdfs sink
Amit Handa
2012-07-05, 07:28
alo alt
2012-07-05, 07:52
Mike Percy
2012-07-05, 18:51
Amit Handa
2012-07-06, 09:44
ashutosh
2012-07-06, 10:08
Amit Handa
2012-07-06, 10:53
Will McQueen
2012-07-06, 11:18
Amit Handa
2012-07-06, 12:46
Copy link to this message
-
Re: flume ng error while going for hdfs sink
Will McQueen 2012-07-06, 11:21
Oops, I just noticed that this was already suggested by Ashutosh Sharma.

On Fri, Jul 6, 2012 at 4:18 AM, Will McQueen <[EMAIL PROTECTED]> wrote:

> Hi Amit,
>
> Try:
>
> *agent1.sinks.HDFS.hdfs.file.Type = DataStream
> ===change to==>
> **agent1.sinks.HDFS.hdfs.fileType = DataStream*
>
> Otherwise the fileType is SequenceFile by default.
>
> Cheers,
> Will
>
>
> On Fri, Jul 6, 2012 at 2:44 AM, Amit Handa <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> @Mike thanks for ur reply.
>>
>> 1) After executing Flume-ng agent, and avro client, File is created in
>> HDFS.
>> I used today same flume-ng setup with hadoop 1.0.1.
>> Now i m facing problem that through avro client i am sending normal text
>> file. But inside HDFS File content is coming like as shown below. I want in
>> HDFS this file content should be in normal text format
>> HDFS File Content:
>> *"SEQ^F!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable^@^@^@^@^@^@^UªG^Oòá~v¾­z/<87>^[~ð^@^@^@)^@^@^@^H^@^@^A8[<8e>)Ú^@^@^@^]We
>> are modifying the file now^@^@^@*
>>
>> Given txt file content through AvroClient is
>>                 * We are modifying the file now*
>>
>> Kindly provide ur inputs to resolve this issue.
>> my flume.conf file content is as folows:
>> *
>> # Define a memory channel called ch1 on agent1
>> agent1.channels.ch1.type = memory
>>
>>
>> # Define an Avro source called avro-source1 on agent1 and tell it
>> # to bind to 0.0.0.0:41414. Connect it to channel ch1.
>> agent1.sources.avro-source1.channels = ch1
>> agent1.sources.avro-source1.type = avro
>> agent1.sources.avro-source1.selector.type=replicating
>>
>> agent1.sources.avro-source1.bind = 0.0.0.0
>> agent1.sources.avro-source1.port = 41414
>>
>>
>> # Define a hdfs sink that simply logs all events it receives
>> # and connect it to the other end of the same channel.
>> agent1.sinks.HDFS.channel = ch1
>> agent1.sinks.HDFS.type = hdfs
>> agent1.sinks.HDFS.hdfs.path >> hdfs://localhost:54310/user/hadoop-node1/flumeTest
>> agent1.sinks.HDFS.hdfs.file.Type = DataStream
>> agent1.sinks.HDFS.hdfs.file.Format = Text
>>
>>
>> # Finally, now that we've defined all of our components, tell
>> # agent1 which ones we want to activate.
>> agent1.channels = ch1
>> agent1.sources = avro-source1
>> agent1.sinks = HDFS*
>>
>>
>> 2) AT Flume NG Side still i am getting security related IO Exception.
>> when i start flume-ng using above configuration file.
>> Exception log coming at flume-ng side is :
>> 2012-07-06 11:14:42,957 (conf-file-poller-0) [DEBUG -
>> org.apache.hadoop.security.Groups.<init>(Groups.java:59)] Group mapping
>> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
>> cacheTimeout=300000
>> 2012-07-06 11:14:42,961 (conf-file-poller-0) [DEBUG -
>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)]
>> java.io.IOException: config()
>>
>>     at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
>>     at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>>      at
>> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:212)
>>
>>     at
>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>>     at
>> org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:516)
>>     at
>> org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239)
>>     at
>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>>     at
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)