Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Exception (possibly) due to type casting between bytearray to charray in HBaseStorage


Copy link to this message
-
Re: Exception (possibly) due to type casting between bytearray to charray in HBaseStorage
Shahab Yunus 2013-05-31, 14:46
Praveen,

Including those worked! Thanks.

But I have some questions:

1-Why do we even need to do this? If in a cluster, same nodes are being
used as DataNodes/TaskTrackers/RegionServers then shouldn't these libs be
on the classpath already? These are pretty basic libs? Had it been the case
that we have different RegionServer nodes from TaskTrackers then I could
understand. Or even if the same node/server is hosting
RegionServer/DataNode/TaskTracker, the TT daemon, when started, did not
have hbase jars on its classpath?

2-Why hbase security jar? Why not the main one? Is the security name
misleading in the context that it is not 'only' for hbase security but
hbase 'with' security?

3-Why zookeeper jar though? Is it because the hbase jar that is actually
needed here depends, downstream in the dependency chain?

4-Lastly, is there a way where we cannot hardcode these registered
libraries' path?

Or am I missing something here? Thanks for any explanation.

Regards,
Shahab
On Fri, May 31, 2013 at 7:44 AM, shashwat shriparv <
[EMAIL PROTECTED]> wrote:

> export HBASE_CLASS= jarfiles
> export HADOOP_CLASS= jarfiles
>
> *Thanks & Regards    *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Fri, May 31, 2013 at 4:49 PM, Shahab Yunus <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks, I will try and provide an update. Out of curiosity, did you ever
> > resolved it?
> >
> > Regards,
> > Shahab
> >
> >
> > On Fri, May 31, 2013 at 4:17 AM, Praveen Bysani <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Yes, it could be. Try registering these jar files
> > >
> > >
> >
> /opt/cloudera/parcels/CDH-<version>/lib/hbase/hbase-<version>-security.jar
> > >
> /opt/cloudera/parcels/CDH-<version>/lib/zookeeper/zookeeper-<version>.jar
> > >
> > > On 30 May 2013 20:14, Shahab Yunus <[EMAIL PROTECTED]> wrote:
> > >
> > > > I am not explicitly registering any of these jars in the script. The
> > > > cluster was setup through standard Cloudera installation (4.2.0).
> > Should
> > > I?
> > > > Is that the issue?
> > > >
> > > > Regards,
> > > > Shahab
> > > >
> > > >
> > > > On Wed, May 29, 2013 at 11:26 PM, Praveen Bysani <
> > > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Are you registering the hbase and zookeepr jar files in your pig
> > > script ?
> > > > >
> > > > > On 30 May 2013 06:24, Shahab Yunus <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > When loading data from a HBase table in Pig, using HBaseStorage,
> > if I
> > > > > > specify the type of the fields as chararray, I get an exception:
> > > > > >
> > > > > > 2013-05-29 16:18:56,557 INFO
> > > > > > org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
> > > > > > truncater with mapRetainSize=-1 and reduceRetainSize=-1
> > > > > > 2013-05-29 16:18:56,560 FATAL org.apache.hadoop.mapred.Child:
> Error
> > > > > > running child : java.lang.NoClassDefFoundError:
> > > > > > org/apache/hadoop/hbase/mapreduce/TableInputFormat
> > > > > >         at java.lang.ClassLoader.defineClass1(Native Method)
> > > > > >         at
> > > java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> > > > > >         at
> java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> > > > > >         at
> > > > > >
> > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > > > > >         at
> > > java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > > > > >         at
> > java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > > > > >         at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > > > > >         at java.security.AccessController.doPrivileged(Native
> > Method)
> > > > > >         at
> > java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > > > > >         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > > > >         at
> > > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > > > >         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)