Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> flume ng error while going for hdfs sink


Copy link to this message
-
Re: flume ng error while going for hdfs sink
Hi Amit,

Try:

*agent1.sinks.HDFS.hdfs.file.Type = DataStream
===change to==>
**agent1.sinks.HDFS.hdfs.fileType = DataStream*

Otherwise the fileType is SequenceFile by default.

Cheers,
Will

On Fri, Jul 6, 2012 at 2:44 AM, Amit Handa <[EMAIL PROTECTED]> wrote:

> Hi,
>
> @Mike thanks for ur reply.
>
> 1) After executing Flume-ng agent, and avro client, File is created in
> HDFS.
> I used today same flume-ng setup with hadoop 1.0.1.
> Now i m facing problem that through avro client i am sending normal text
> file. But inside HDFS File content is coming like as shown below. I want in
> HDFS this file content should be in normal text format
> HDFS File Content:
> *"SEQ^F!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable^@^@^@^@^@^@^UªG^Oòá~v¾­z/<87>^[~ð^@^@^@)^@^@^@^H^@^@^A8[<8e>)Ú^@^@^@^]We
> are modifying the file now^@^@^@*
>
> Given txt file content through AvroClient is
>                 * We are modifying the file now*
>
> Kindly provide ur inputs to resolve this issue.
> my flume.conf file content is as folows:
> *
> # Define a memory channel called ch1 on agent1
> agent1.channels.ch1.type = memory
>
>
> # Define an Avro source called avro-source1 on agent1 and tell it
> # to bind to 0.0.0.0:41414. Connect it to channel ch1.
> agent1.sources.avro-source1.channels = ch1
> agent1.sources.avro-source1.type = avro
> agent1.sources.avro-source1.selector.type=replicating
>
> agent1.sources.avro-source1.bind = 0.0.0.0
> agent1.sources.avro-source1.port = 41414
>
>
> # Define a hdfs sink that simply logs all events it receives
> # and connect it to the other end of the same channel.
> agent1.sinks.HDFS.channel = ch1
> agent1.sinks.HDFS.type = hdfs
> agent1.sinks.HDFS.hdfs.path > hdfs://localhost:54310/user/hadoop-node1/flumeTest
> agent1.sinks.HDFS.hdfs.file.Type = DataStream
> agent1.sinks.HDFS.hdfs.file.Format = Text
>
>
> # Finally, now that we've defined all of our components, tell
> # agent1 which ones we want to activate.
> agent1.channels = ch1
> agent1.sources = avro-source1
> agent1.sinks = HDFS*
>
>
> 2) AT Flume NG Side still i am getting security related IO Exception. when
> i start flume-ng using above configuration file.
> Exception log coming at flume-ng side is :
> 2012-07-06 11:14:42,957 (conf-file-poller-0) [DEBUG -
> org.apache.hadoop.security.Groups.<init>(Groups.java:59)] Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> cacheTimeout=300000
> 2012-07-06 11:14:42,961 (conf-file-poller-0) [DEBUG -
> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)]
> java.io.IOException: config()
>
>     at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
>     at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
>     at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>     at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>     at
> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
>     at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:212)
>
>     at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>     at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>     at
> org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:516)
>     at
> org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239)
>     at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>     at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
>     at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
>     at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB