Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> flume ng error while going for hdfs sink


+
Amit Handa 2012-07-05, 07:28
+
alo alt 2012-07-05, 07:52
+
Mike Percy 2012-07-05, 18:51
+
Amit Handa 2012-07-06, 09:44
+
ashutosh 2012-07-06, 10:08
+
Amit Handa 2012-07-06, 10:53
+
Will McQueen 2012-07-06, 11:18
Copy link to this message
-
Re: flume ng error while going for hdfs sink
Oops, I just noticed that this was already suggested by Ashutosh Sharma.

On Fri, Jul 6, 2012 at 4:18 AM, Will McQueen <[EMAIL PROTECTED]> wrote:

> Hi Amit,
>
> Try:
>
> *agent1.sinks.HDFS.hdfs.file.Type = DataStream
> ===change to==>
> **agent1.sinks.HDFS.hdfs.fileType = DataStream*
>
> Otherwise the fileType is SequenceFile by default.
>
> Cheers,
> Will
>
>
> On Fri, Jul 6, 2012 at 2:44 AM, Amit Handa <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> @Mike thanks for ur reply.
>>
>> 1) After executing Flume-ng agent, and avro client, File is created in
>> HDFS.
>> I used today same flume-ng setup with hadoop 1.0.1.
>> Now i m facing problem that through avro client i am sending normal text
>> file. But inside HDFS File content is coming like as shown below. I want in
>> HDFS this file content should be in normal text format
>> HDFS File Content:
>> *"SEQ^F!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable^@^@^@^@^@^@^UªG^Oòá~v¾­z/<87>^[~ð^@^@^@)^@^@^@^H^@^@^A8[<8e>)Ú^@^@^@^]We
>> are modifying the file now^@^@^@*
>>
>> Given txt file content through AvroClient is
>>                 * We are modifying the file now*
>>
>> Kindly provide ur inputs to resolve this issue.
>> my flume.conf file content is as folows:
>> *
>> # Define a memory channel called ch1 on agent1
>> agent1.channels.ch1.type = memory
>>
>>
>> # Define an Avro source called avro-source1 on agent1 and tell it
>> # to bind to 0.0.0.0:41414. Connect it to channel ch1.
>> agent1.sources.avro-source1.channels = ch1
>> agent1.sources.avro-source1.type = avro
>> agent1.sources.avro-source1.selector.type=replicating
>>
>> agent1.sources.avro-source1.bind = 0.0.0.0
>> agent1.sources.avro-source1.port = 41414
>>
>>
>> # Define a hdfs sink that simply logs all events it receives
>> # and connect it to the other end of the same channel.
>> agent1.sinks.HDFS.channel = ch1
>> agent1.sinks.HDFS.type = hdfs
>> agent1.sinks.HDFS.hdfs.path >> hdfs://localhost:54310/user/hadoop-node1/flumeTest
>> agent1.sinks.HDFS.hdfs.file.Type = DataStream
>> agent1.sinks.HDFS.hdfs.file.Format = Text
>>
>>
>> # Finally, now that we've defined all of our components, tell
>> # agent1 which ones we want to activate.
>> agent1.channels = ch1
>> agent1.sources = avro-source1
>> agent1.sinks = HDFS*
>>
>>
>> 2) AT Flume NG Side still i am getting security related IO Exception.
>> when i start flume-ng using above configuration file.
>> Exception log coming at flume-ng side is :
>> 2012-07-06 11:14:42,957 (conf-file-poller-0) [DEBUG -
>> org.apache.hadoop.security.Groups.<init>(Groups.java:59)] Group mapping
>> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
>> cacheTimeout=300000
>> 2012-07-06 11:14:42,961 (conf-file-poller-0) [DEBUG -
>> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)]
>> java.io.IOException: config()
>>
>>     at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
>>     at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>>      at
>> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:212)
>>
>>     at
>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>>     at
>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>>     at
>> org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:516)
>>     at
>> org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239)
>>     at
>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>>     at
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
+
Amit Handa 2012-07-06, 12:46
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB