Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Flume 0.94 with CDH4


Copy link to this message
-
Flume 0.94 with CDH4
Kumar, Deepak8 2012-10-03, 06:26
Hi Hari,

I am trying to use Flume 0.94 with CDH4. I am getting following exception.

1.       Without hbase 0.9.2 jar file, flume throws following error

2012-10-01 13:29:13,958 ERROR com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode vm-ccf5-9333.nam.nsroot.net_synch-56 in error state KeywordSynchSource | ConsoleEventSink because Not a host:port pair: �^@^@^@![EMAIL PROTECTED],60020,1348696066287

It's a version conflict error if you try to access to hbsae 0.9.2 from hbsae 0.90.4 client.

2.       Hbase 0.9.2 + hadoop 0.20.2 jar files, flume throws following error.

Exception in thread "Heartbeat" java.lang.NoClassDefFoundError: org/apache/hadoop/net/SocketInputWrapper

       at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1035)

        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)

        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)

        at $Proxy7.getProtocolVersion(Unknown Source)

        at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:183)

        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)

        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)

        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)

        at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1278)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1235)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1222)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:918)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:915)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)

        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)

        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)

        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)

        at com.citi.sponge.flume.sink.ELFNotifyCacheSink$1.build(ELFNotifyCacheSink.java:117)

        at com.cloudera.flume.conf.SinkFactory$SinkBuilder.create(SinkFactory.java:42)

        at com.cloudera.flume.conf.SinkFactoryImpl.createSink(SinkFactoryImpl.java:318)

        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:500)

        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:531)

        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:513)

        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:544)

        at com.cloudera.flume.conf.FlumeBuilder.buildEventSink(FlumeBuilder.java:544)

        at com.cloudera.flume.conf.FlumeBuilder.buildSink(FlumeBuilder.java:302)

        at com.cloudera.flume.agent.LogicalNode.loadConfig(LogicalNode.java:215)

        at com.cloudera.flume.agent.LogicalNodeManager.spawn(LogicalNodeManager.java:104)

        at com.cloudera.flume.agent.LivenessManager.checkLogicalNodes(LivenessManager.java:141)

        at com.cloudera.flume.agent.LivenessManager.heartbeatChecks(LivenessManager.java:183)

        at com.cloudera.flume.agent.LivenessManager$HeartbeatThread.run(LivenessManager.java:233)

3.       Hbase 0.9.2 + Hadoop 2.0 + flume 0.9.4

flume script complains that hadoop-core.jar is not found. Flume 0.9.4 hardcodes the name of hadoop and the name of jar files are changed from hadoop 2.0

4.       Hbase 0.9.2 + Hadoop 2.0 + modify hadoop jar location on the script file + flume 0.9.4

Exception in thread "logicalNode vm-93ff-5f31.nam.nsroot.net_sa56647_1345135162189-22" java.lang.NoSuchMethodError: org.apache.hadoop.io.SequenceFile$Writer: method <init>()V not found

        at org.apache.hadoop.io.RawSequenceFileWriter.<init>(RawSequenceFileWriter.java:70)

        at org.apache.hadoop.io.RawSequenceFileWriter.createWriter(RawSequenceFileWriter.java:61)

        at com.cloudera.flume.handlers.hdfs.SeqfileEventSink.open(SeqfileEventSink.java:79)

        at com.cloudera.flume.core.EventSinkDecorator.open(EventSinkDecorator.java:75)

        at com.cloudera.flume.handlers.rolling.RollSink.open(RollSink.java:381)

        at com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.open(DiskFailoverDeco.java:196)

        at com.cloudera.flume.core.BackOffFailOverSink.open(BackOffFailOverSink.java:203)

        at com.cloudera.flume.core.EventSinkDecorator.open(EventSinkDecorator.java:75)

        at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)

It seems Flume has a package org.apache.hadoop.io and there is a class name RawSequenceFileWriter. Flume call a hadoop API (org.apache.hadoop.io.SequenceFile$Writer.init()) from this class and this API signature is changed on hadoop 2.0.

Could you please guide me, whether we can use Flume 0.94 with CDH4 or not?

Regards,
Deepak
+
Hari Shreedharan 2012-10-04, 00:36