Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Error with Pig (CDH4.0.0)
Dhaval Shah
2012-10-09, 22:29
Cheolsoo Park
2012-10-09, 22:57
Dhaval Shah
2012-10-09, 23:02
Cheolsoo Park
2012-10-09, 23:15
Cheolsoo Park
2012-10-09, 23:31
Cheolsoo Park
2012-10-09, 23:31
Russell Jurney
2012-10-09, 23:34
Cheolsoo Park
2012-10-09, 23:50
Russell Jurney
2012-10-09, 23:56
Dhaval Shah
2012-10-10, 14:57
Dhaval Shah
2012-10-11, 00:20
Copy link to this message
-
Re: Error with Pig (CDH4.0.0)
Cheolsoo Park 2012-10-10, 20:55
Hi Dhaval,

Thank you very much for sharing your analysis! Your explanation definitely
provides more insights. :-)

If you don't mind, I'd like to clarify a couple of things. I am just trying
to see if there's something to be fixed in terms of Pig packaging:

>> the root issue was that the pig libraries needed to be present in the
HADOOP_CLASSPATH and it needs to be specifically set in hadoop_env.sh..

I am not sure if missing dependency libraries is the root cause. Given the
following error in your call stack, I believe that you have a different
version of antlr in classpath.

java.lang.NoSuchFieldError: name
>         at org.apache.pig.parser.QueryParserStringStream.<init>
> (QueryParserStringStream.java:32)
If you look at QueryParserStringStream.java, it extends ANTLRStringStream,
and Pig 0.9 is compiled against antlr 3.4. Now if antlr is missing, you
should get ClassNotFoundError not NoSuchFiledError. Only possible reason
that I can think of is that there is a different version of antlr in
classpath at run-time.

In fact, there was a similar discussion on this mailing list a while ago
(Please note that he is reporting the same call stack as yours):
http://mail-archives.apache.org/mod_mbox/pig-user/201206.mbox/%3CCAOkn+GjPCgY1i+-zfNWF9cKz+[EMAIL PROTECTED]%3E

Antlr is a common tool, so it's possible that it's installed by other tools
such as Hive.

>> I tried setting PIG_CLASSPATH and HADOOP_CLASSPATH to have those
libraries in the shell and then start pig but that did not help either..

Assuming that a wrong version of antlr is present in classpath, this makes
perfect sense because PIG_CLASSPATH adds libraries to the end of CLASPATH:

# add user-specified CLASSPATH
if [ "$PIG_CLASSPATH" != "" ]; then
    CLASSPATH=${CLASSPATH}:${PIG_CLASSPATH}
fi

That is, as long as the wrong antlr is present before the correct one, the
wrong one be picked up at run-time.

>> 2. The error message is not super helpful.. If libraries are missing,
the pig shell/grunt should not open at all.. However, in my case, it did
start up and then the error message was in no ways intuitive or pointing to
the root issue..

I agree with you that the error message is not very intuitive here. But
errors caused by dependency libraries can be only caught when Pig makes
calls to methods of those libraries at run-time. Furthermore, if there are
two different versions of the same library in classpath, the root cause can
be more subtle.

The challenge is that a Hadoop distribution such as CDH bundles many
sub-projects, and they often depend on different versions of the same
libraries. We do our best to harmonize all the versions of dependency
libraries across the platform, but I admit that it is not always perfect.

Thanks,
Cheolsoo

On Wed, Oct 10, 2012 at 7:57 AM, Dhaval Shah <[EMAIL PROTECTED]>wrote:

> Alright I was eventually able to get the issue resolved.. For everyone's
> benefit, the root issue was that the pig libraries needed to be present in
> the HADOOP_CLASSPATH and it needs to be specifically set in hadoop_env.sh..
> I tried setting PIG_CLASSPATH and HADOOP_CLASSPATH to have those libraries
> in the shell and then start pig but that did not help either.. Setting that
> explicitly in hadoop_env.sh did the trick..
>
> As I see there are 2 issues here:
> 1. Somehow the pig executable seems broken.. As in setting
> HADOOP_CLASSPATH on the shell and then running pig should have the same
> effect as setting it in hadoop_env.sh.. Also having a * in the
> PIG_CLASSPATH/HADOOP_CLASSPATH (when set in the shell) makes the pig
> executable set HADOOP_CLASSPATH to empty before it actually runs the hadoop
> jar command inside the pig script.. In CDH3, I never needed to set the
> classpath to include pig libraries.. The pig executable did that for
> me..(and I am never overriding HADOOP_CLASSPATH.. I only append to it)
> 2. The error message is not super helpful.. If libraries are missing, the
> pig shell/grunt should not open at all.. However, in my case, it did start