Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Guidance for running MR jobs against trunk/0.95


+
Nick Dimiduk 2013-02-25, 22:37
+
Nick Dimiduk 2013-02-25, 22:58
+
Elliott Clark 2013-02-25, 23:10
+
Nick Dimiduk 2013-02-25, 23:19
+
Elliott Clark 2013-02-25, 23:27
+
Nick Dimiduk 2013-02-25, 23:54
Copy link to this message
-
Re: Guidance for running MR jobs against trunk/0.95
Right now only used at runtime on the server.  I guess we need a way to
prevent instantiation on the client.

If we allowed it as one of the client RPC encodings, then client would need
the module as well.  However, I don't think prefix-tree is a good candidate
for client RPC encoding anyway since RPC is a streaming activity, not
random access.  The existing delta encoders will offer higher performance
with similar bandwidth savings.
On Mon, Feb 25, 2013 at 3:54 PM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> Under what context is hbase-prefix-tree necessary? Currently, this
> dependency is scoped as 'runtime'. Adding it explicitly will require this
> be changed.
>
> Thanks,
> Nick
>
> On Mon, Feb 25, 2013 at 3:10 PM, Elliott Clark <[EMAIL PROTECTED]> wrote:
>
> > You'll currently need:
> >
> > hbase-server
> > hbase-common
> > hbase-protocol
> > hbase-prefix-tree  (after HBASE-7188)
> > hbase-client (after HBASE-7188)
> >
> > on your classpath to do any mapreduce jobs.
> >
> >
> > On Mon, Feb 25, 2013 at 2:37 PM, Nick Dimiduk <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi all,
> > >
> > > I'm seeing ClassNotFoundExceptions when running ImportTsv against
> trunk.
> > > Presumably the old instructions for classpath assembly need an update.
> I
> > > guess I could create a hadoop jar that ships the classpath entries in
> > lib,
> > > but IMHO that's not a viable solution for users who want to run one of
> > the
> > > jobs we distribute. Am I missing something?
> > >
> > > Thanks,
> > > Nick
> > >
> > > For you reference:
> > >
> > > $ HADOOP_CLASSPATH=$(./bin/hbase classpath) hadoop jar
> > > ./hbase-server/target/hbase-server-0.95-SNAPSHOT.jar importtsv
> > > -Dimporttsv.columns=HBASE_ROW_KEY,d:c1,d:c2
> > > -Dimporttsv.bulk.output=hdfs:///tmp/hfiles simple
> hdfs:///tmp/simple.tsv
> > >
> > > ...
> > >
> > > Error: java.lang.ClassNotFoundException:
> > >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Mutate$MutateType
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > >  at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > >  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.MutationSerialization$MutationSerializer.serialize(MutationSerialization.java:87)
> > >  at
> > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.MutationSerialization$MutationSerializer.serialize(MutationSerialization.java:70)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069)
> > >  at
> > >
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > >  at
> > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:151)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:37)
> > >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> > >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> > > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > >  at java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:396)
> > >  at
> > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> > >
> >
>
+
Nick Dimiduk 2013-02-26, 00:41