Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Trouble doing the most basic thing in Pig with HCatalog (v. 0.11.0)


Copy link to this message
-
Re: Trouble doing the most basic thing in Pig with HCatalog (v. 0.11.0)
Timothy Potter 2013-10-09, 20:22
Having deja vu from when I did our HCatalog 0.4 install ... the issue is
the datanucleus jars are out-of-date. I upgraded the datanucleus JARs and
am now past this issue ... new issue is:

2013-10-09 20:20:36,411 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
recreate exception from backend error:
org.apache.hcatalog.common.HCatException : 2004 : HCatOutputFormat not
initialized, setOutput has to be called
at
org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getJobInfo(HCatBaseOutputFormat.java:111)
at
org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getJobInfo(HCatBaseOutputFormat.java:97)
at
org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getOutputFormat(HCatBaseOutputFormat.java:85)
at
org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:75)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecsHelper(PigOutputFormat.java:207)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:187)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:935)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:318)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.startReadyJobs(JobControl.java:238)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:269)
at java.lang.Thread.run(Thread.java:662)

On Wed, Oct 9, 2013 at 1:05 PM, Timothy Potter <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Long time user of HCatalog 0.4 and am testing out an upgrade to Hive /
> HCatalog 0.11.0 as we need windowing functions and ORC
>
> I'm testing the HCatLoader from Pig and am getting the exceptions below
> using this simple Pig script:
>
> sigs_in = load 'signals' using org.apache.hcatalog.pig.HCatLoader();
> describe sigs_in;
> sigs = filter sigs_in by datetime_partition == '2013-10-07_0000';
> ...
>
> The exceptions (see below) occur in the Pig front-end processing, trying
> to get the input paths. The Pig describe command returns the schema, so I
> know there's some communication going on between the LoadFunc and the
> metastore. Also, if I do: hcat -e "show partitions signals;" I get the
> list of expected partitions on that table.
>
> Any ideas on where to start troubleshooting this issue? I'm using Pig 0.10
> with Hive / HCatalog 0.11.0 running on Hadoop 2.0.0-cdh4.1.2.
>
> I built Hive/HCatalog from source using: *ant clean package
> -Dmvn.hadoop.profile=hadoop23 -Dhadoop.mr.rev=23*
>
> Exception:
>
> Caused by: java.io.IOException:
> org.shaded.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> at
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:87)
>  at
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:63)
> at org.apache.hcatalog.pig.HCatLoader.setLocation(HCatLoader.java:119)
>  at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:380)
> ... 17 more
> Caused by: org.shaded.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> at
> org.shaded.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>  at org.shaded.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.shaded.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>  at
> org.shaded.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
> org.shaded.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)