Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - Regression in trunk? (RE: Insert overwrite error using hive trunk)


+
Pradeep Kamath 2010-09-27, 16:10
+
Ning Zhang 2010-09-27, 16:25
+
Ashutosh Chauhan 2010-09-27, 16:25
+
Ning Zhang 2010-09-27, 16:31
+
yongqiang he 2010-09-27, 16:44
Copy link to this message
-
Re: Regression in trunk? (RE: Insert overwrite error using hive trunk)
Pradeep Kamath 2010-09-27, 17:38
Here is the output of explain:

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-4 depends on stages: Stage-1 , consists of Stage-3, Stage-2
  Stage-3
  Stage-0 depends on stages: Stage-3, Stage-2
  Stage-2

STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        numbers_text
          TableScan
            alias: numbers_text
            Select Operator
              expressions:
                    expr: id
                    type: int
                    expr: num
                    type: int
              outputColumnNames: _col0, _col1
              File Output Operator
                compressed: false
                GlobalTableId: 1
                table:
                    input format: org.apache.hadoop.mapred.TextInputFormat
                    output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                    serde:
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                    name: numbers_text_part

  Stage: Stage-4
    Conditional Operator

  Stage: Stage-3
    Move Operator
      files:
          hdfs directory: true
          destination:
hdfs://wilbur21.labs.corp.sp1.yahoo.com/tmp/hive-pradeepk/hive_2010-09-27_10-37-06_724_1678373180997754320/-ext-10000

  Stage: Stage-0
    Move Operator
      tables:
          partition:
            part p1
          replace: true
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: numbers_text_part

  Stage: Stage-2
    Map Reduce
      Alias -> Map Operator Tree:
        
hdfs://wilbur21.labs.corp.sp1.yahoo.com/tmp/hive-pradeepk/hive_2010-09-27_10-37-06_724_1678373180997754320/-ext-10002
            File Output Operator
              compressed: false
              GlobalTableId: 0
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                  name: numbers_text_part
yongqiang he wrote:
> There is one ticket for insert overwrite local directory:
> https://issues.apache.org/jira/browse/HIVE-1582
>
> On Mon, Sep 27, 2010 at 9:31 AM, Ning Zhang <[EMAIL PROTECTED]> wrote:
>  
>> Can you do explain your query after setting the parameter?
>>
>>
>> On Sep 27, 2010, at 9:25 AM, Ashutosh Chauhan wrote:
>>
>>    
>>> I suspected the same. But, even after setting this property, second MR
>>> job did get launched and then failed.
>>>
>>> Ashutosh
>>> On Mon, Sep 27, 2010 at 09:25, Ning Zhang <[EMAIL PROTECTED]> wrote:
>>>      
>>>> I'm guessing this is due to the merge task (the 2nd MR job that merges small
>>>> files together). You can try to 'set hive.merge.mapfiles=false;' before the
>>>> query and see if it succeeded.
>>>> If it is due to merge job, can you attach the plan and check the
>>>> mapper/reducer task log and see what errors/exceptions are there?
>>>>
>>>> On Sep 27, 2010, at 9:10 AM, Pradeep Kamath wrote:
>>>>
>>>> Hi,
>>>>
>>>>   Any help in debugging the issue I am seeing below will be greatly
>>>> appreciated. Unless I am doing something wrong, this seems to be a
>>>> regression in trunk.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Pradeep
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Pradeep Kamath [mailto:[EMAIL PROTECTED]]
>>>> Sent: Friday, September 24, 2010 1:41 PM
>>>> To: [EMAIL PROTECTED]
>>>> Subject: Insert overwrite error using hive trunk
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>    I am trying to insert overwrite into a partitioned table reading data
>>>> from a non partitioned table and seeing a failure in the second map reduce
>>>> job � wonder if I am doing something wrong � any pointers appreciated (I am
>>>> using latest trunk code against hadoop 20 cluster). Details below[1].
+
Ning Zhang 2010-09-27, 17:52
+
Pradeep Kamath 2010-09-27, 18:22
+
Ning Zhang 2010-09-27, 18:33
+
Pradeep Kamath 2010-09-27, 19:33
+
Steven Wong 2010-09-27, 20:11
+
Ning Zhang 2010-09-27, 20:37
+
Pradeep Kamath 2010-09-28, 00:58
+
Amareshwari Sri Ramadasu 2010-09-28, 08:03
+
Pradeep Kamath 2010-09-28, 16:31
+
Pradeep Kamath 2010-09-28, 17:30
+
Ning Zhang 2010-09-28, 18:23
+
Pradeep Kamath 2010-09-28, 19:30
+
Ning Zhang 2010-09-28, 20:43