Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Regression in trunk? (RE: Insert overwrite error using hive trunk)


Copy link to this message
-
Re: Regression in trunk? (RE: Insert overwrite error using hive trunk)
Here is the output of explain:

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-4 depends on stages: Stage-1 , consists of Stage-3, Stage-2
  Stage-3
  Stage-0 depends on stages: Stage-3, Stage-2
  Stage-2

STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        numbers_text
          TableScan
            alias: numbers_text
            Select Operator
              expressions:
                    expr: id
                    type: int
                    expr: num
                    type: int
              outputColumnNames: _col0, _col1
              File Output Operator
                compressed: false
                GlobalTableId: 1
                table:
                    input format: org.apache.hadoop.mapred.TextInputFormat
                    output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                    serde:
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                    name: numbers_text_part

  Stage: Stage-4
    Conditional Operator

  Stage: Stage-3
    Move Operator
      files:
          hdfs directory: true
          destination:
hdfs://wilbur21.labs.corp.sp1.yahoo.com/tmp/hive-pradeepk/hive_2010-09-27_10-37-06_724_1678373180997754320/-ext-10000

  Stage: Stage-0
    Move Operator
      tables:
          partition:
            part p1
          replace: true
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: numbers_text_part

  Stage: Stage-2
    Map Reduce
      Alias -> Map Operator Tree:
        
hdfs://wilbur21.labs.corp.sp1.yahoo.com/tmp/hive-pradeepk/hive_2010-09-27_10-37-06_724_1678373180997754320/-ext-10002
            File Output Operator
              compressed: false
              GlobalTableId: 0
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                  name: numbers_text_part
yongqiang he wrote:
> There is one ticket for insert overwrite local directory:
> https://issues.apache.org/jira/browse/HIVE-1582
>
> On Mon, Sep 27, 2010 at 9:31 AM, Ning Zhang <[EMAIL PROTECTED]> wrote:
>  
>> Can you do explain your query after setting the parameter?
>>
>>
>> On Sep 27, 2010, at 9:25 AM, Ashutosh Chauhan wrote:
>>
>>    
>>> I suspected the same. But, even after setting this property, second MR
>>> job did get launched and then failed.
>>>
>>> Ashutosh
>>> On Mon, Sep 27, 2010 at 09:25, Ning Zhang <[EMAIL PROTECTED]> wrote:
>>>      
>>>> I'm guessing this is due to the merge task (the 2nd MR job that merges small
>>>> files together). You can try to 'set hive.merge.mapfiles=false;' before the
>>>> query and see if it succeeded.
>>>> If it is due to merge job, can you attach the plan and check the
>>>> mapper/reducer task log and see what errors/exceptions are there?
>>>>
>>>> On Sep 27, 2010, at 9:10 AM, Pradeep Kamath wrote:
>>>>
>>>> Hi,
>>>>
>>>>   Any help in debugging the issue I am seeing below will be greatly
>>>> appreciated. Unless I am doing something wrong, this seems to be a
>>>> regression in trunk.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Pradeep
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Pradeep Kamath [mailto:[EMAIL PROTECTED]]
>>>> Sent: Friday, September 24, 2010 1:41 PM
>>>> To: [EMAIL PROTECTED]
>>>> Subject: Insert overwrite error using hive trunk
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>    I am trying to insert overwrite into a partitioned table reading data
>>>> from a non partitioned table and seeing a failure in the second map reduce
>>>> job � wonder if I am doing something wrong � any pointers appreciated (I am
>>>> using latest trunk code against hadoop 20 cluster). Details below[1].
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB