Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Guidance for running MR jobs against trunk/0.95


+
Nick Dimiduk 2013-02-25, 22:37
+
Nick Dimiduk 2013-02-25, 22:58
+
Elliott Clark 2013-02-25, 23:10
+
Nick Dimiduk 2013-02-25, 23:19
+
Elliott Clark 2013-02-25, 23:27
+
Nick Dimiduk 2013-02-25, 23:54
+
Matt Corgan 2013-02-26, 00:01
Copy link to this message
-
Re: Guidance for running MR jobs against trunk/0.95
Thanks for the replies everyone. Let's move conversation over to comments
on HBASE-7934.

Thanks,
Nick

On Mon, Feb 25, 2013 at 4:01 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:

> Right now only used at runtime on the server.  I guess we need a way to
> prevent instantiation on the client.
>
> If we allowed it as one of the client RPC encodings, then client would need
> the module as well.  However, I don't think prefix-tree is a good candidate
> for client RPC encoding anyway since RPC is a streaming activity, not
> random access.  The existing delta encoders will offer higher performance
> with similar bandwidth savings.
>
>
> On Mon, Feb 25, 2013 at 3:54 PM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:
>
> > Under what context is hbase-prefix-tree necessary? Currently, this
> > dependency is scoped as 'runtime'. Adding it explicitly will require this
> > be changed.
> >
> > Thanks,
> > Nick
> >
> > On Mon, Feb 25, 2013 at 3:10 PM, Elliott Clark <[EMAIL PROTECTED]>
> wrote:
> >
> > > You'll currently need:
> > >
> > > hbase-server
> > > hbase-common
> > > hbase-protocol
> > > hbase-prefix-tree  (after HBASE-7188)
> > > hbase-client (after HBASE-7188)
> > >
> > > on your classpath to do any mapreduce jobs.
> > >
> > >
> > > On Mon, Feb 25, 2013 at 2:37 PM, Nick Dimiduk <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm seeing ClassNotFoundExceptions when running ImportTsv against
> > trunk.
> > > > Presumably the old instructions for classpath assembly need an
> update.
> > I
> > > > guess I could create a hadoop jar that ships the classpath entries in
> > > lib,
> > > > but IMHO that's not a viable solution for users who want to run one
> of
> > > the
> > > > jobs we distribute. Am I missing something?
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > > > For you reference:
> > > >
> > > > $ HADOOP_CLASSPATH=$(./bin/hbase classpath) hadoop jar
> > > > ./hbase-server/target/hbase-server-0.95-SNAPSHOT.jar importtsv
> > > > -Dimporttsv.columns=HBASE_ROW_KEY,d:c1,d:c2
> > > > -Dimporttsv.bulk.output=hdfs:///tmp/hfiles simple
> > hdfs:///tmp/simple.tsv
> > > >
> > > > ...
> > > >
> > > > Error: java.lang.ClassNotFoundException:
> > > >
> > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Mutate$MutateType
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > > >  at java.security.AccessController.doPrivileged(Native Method)
> > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > > >  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > >  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.MutationSerialization$MutationSerializer.serialize(MutationSerialization.java:87)
> > > >  at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.MutationSerialization$MutationSerializer.serialize(MutationSerialization.java:70)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069)
> > > >  at
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > > >  at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:151)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:37)
> > > >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> > > >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> > > > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > > >  at java.security.AccessController.doPrivileged(Native Method)
> > > > at javax.security.auth.Subject.doAs(Subject.java:396)
> > > >  at
> > > >