Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> alter table add partition error


+
Pradeep Kamath 2010-06-17, 00:22
+
yq he 2010-06-17, 00:54
+
Pradeep Kamath 2010-06-17, 16:09
+
Ashish Thusoo 2010-06-17, 18:15
+
Pradeep Kamath 2010-06-17, 20:24
+
Pradeep Kamath 2010-06-18, 17:19
+
Ning Zhang 2010-06-18, 17:49
+
Edward Capriolo 2010-06-18, 18:22
+
Pradeep Kamath 2010-06-18, 18:25
+
Ning Zhang 2010-06-18, 18:51
+
Paul Yang 2010-06-18, 18:59
+
Pradeep Kamath 2010-06-18, 19:18
+
Paul Yang 2010-06-18, 19:10
+
Pradeep Kamath 2010-06-18, 20:37
+
Paul Yang 2010-06-18, 21:19
+
Pradeep Kamath 2010-06-18, 21:51
+
Pradeep Kamath 2010-06-21, 21:11
+
Paul Yang 2010-06-21, 21:34
+
Pradeep Kamath 2010-06-22, 00:46
+
Paul Yang 2010-06-22, 01:25
Copy link to this message
-
Re: alter table add partition error
Hi Paul,
   I have attached a draft of a patch (incomplete). The problem I am
facing is that to make this change I seem to need to change the thrift
idl (hive_metastore.thrift). Looking at the code structure it seems like
the generate java files are checked in rather than generated through the
build (is this so that the build does not depend on the thrift
compiler?). I tried downloading and building thrift but hit issues in
the make install - wanted to check with you if the patch looks correct
so far or if I am on the wrong path. Also if the direction is right, I
need some guidance on generating the java files from the modified idl.

Thanks,
Pradeep

Paul Yang wrote:
>
> Well, the section of code you are referring to is automatically
> generated by the thrift compiler so we can't modify it. As the
> inability to return null is a limitation of Thrift, we can work around
> that issue by throwing an exception instead.
>
>  
>
> Check out this thread:
>
> http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
>
>  
>
>  
>
> *From:* Pradeep Kamath [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, June 21, 2010 5:46 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* RE: alter table add partition error
>
>  
>
> Hi Paul,
>
>   Yes I don't mind working on a patch though I have no knowledge of
> how thrift code works. I was also confused by your suggestion for a
> fix. I thought the fix would be in
> ThriftHiveMetaStore.Client.recv_get_partition() -- to first check
> if(result.o1 != null) throw result.o1, then to just return
> result.success and not throw the "unknown result" exception. I tried
> something along these lines and the alter table worked. Am I missing
> something?
>
>  
>
> Thanks,
>
> Pradeep
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Paul Yang [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, June 21, 2010 2:35 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* RE: alter table add partition error
>
>  
>
> Ah okay. So what's going on is that getPartition() is supposed to
> return null if the specified partition doesn't exist. But because
> Thrift can't handle null return values, we are seeing this exception.
> This isn't a problem with a local metastore because Thrift isn't used.
>
>  
>
> One solution to this problem would be modify
> ObjectStore.getPartition() to throw a NoSuchObjectException instead of
> returning null for non-existent partitions. Then Hive.getPartition()
> can catch the exception and return null to retain the original behavior.
>
>  
>
> Pradeep, are you interested in working on a patch for this problem?
>
>  
>
> *From:* Pradeep Kamath [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, June 21, 2010 2:11 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* RE: alter table add partition error
>
>  
>
> I tried debugging in code a little more. Here is what I found:
>
> The code in ThriftHiveMetaStore eventually makes a call --
> get_partition() passing the partition key values for the partition I
> am trying to add using alter table. I assume this is to check that the
> partition doesn't already exist.
>
> I added a debug line in the following code:
>
>   public Partition recv_get_partition() throws MetaException, TException
>
>     {
>
>       TMessage msg = iprot_.readMessageBegin();
>
>       if (msg.type == TMessageType.EXCEPTION) {
>
>         TApplicationException x = TApplicationException.read(iprot_);
>
>         iprot_.readMessageEnd();
>
>         throw x;
>
>       }
>
>       get_partition_result result = new get_partition_result();
>
>      
>
>       result.read(iprot_);
>
>       *System.err.println("XXX: result:" + result);*
>
>       iprot_.readMessageEnd();
>
>       if (result.isSetSuccess()) {
>
>         return result.success;
>
>       }
>
>       if (result.o1 != null) {
>
>         throw result.o1;
>
>       }
>
> *      throw new
> TApplicationException(TApplicationException.MISSING_RESULT,
+
Paul Yang 2010-06-23, 01:06
+
Pradeep Kamath 2010-06-23, 16:03
+
John Sichi 2010-06-23, 16:44
+
Pradeep Kamath 2010-06-25, 20:36