Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> enough alter tables in the same .q file eventually fail


Copy link to this message
-
Re: enough alter tables in the same .q file eventually fail
Typo in Ed's last email (table name):
echo "create external table if not exists ed*_*test ( dat string )
partitioned by (dummy string) location '/tmp/a';" > test.q

On Thu, Apr 8, 2010 at 3:14 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

>
>
>
> On Thu, Apr 8, 2010 at 5:22 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:
>
>>
>>
>> On Thu, Apr 8, 2010 at 5:14 PM, Paul Yang <[EMAIL PROTECTED]> wrote:
>>
>>> Seems to be fixed in 0.6. Here's what I got:
>>>
>>> test.q:
>>> alter table tmp_pyang_t ADD PARTITION (ds='2') LOCATION '/tmp/blah2';
>>> alter table tmp_pyang_t ADD PARTITION (ds='3') LOCATION '/tmp/blah2';
>>> alter table tmp_pyang_t ADD PARTITION (ds='4') LOCATION '/tmp/blah2';
>>> alter table tmp_pyang_t ADD PARTITION (ds='5') LOCATION '/tmp/blah2';
>>> alter table tmp_pyang_t ADD PARTITION (ds='6') LOCATION '/tmp/blah2';
>>> alter table tmp_pyang_t ADD PARTITION (ds='7') LOCATION '/tmp/blah2';
>>> alter table tmp_pyang_t ADD PARTITION (ds='8') LOCATION '/tmp/blah2';
>>>
>>>
>>> Hive history
>>> file=/tmp/pyang/hive_job_log_pyang_201004081410_378771152.txt
>>> OK
>>> Time taken: 4.101 seconds
>>> OK
>>> Time taken: 0.558 seconds
>>> OK
>>> Time taken: 0.453 seconds
>>> OK
>>> Time taken: 0.416 seconds
>>> OK
>>> Time taken: 0.378 seconds
>>> OK
>>> Time taken: 0.457 seconds
>>> OK
>>> Time taken: 0.454 seconds
>>>
>>>
>>> Can you the stack trace in /tmp/<username>/hive.log?
>>>
>>>
>>> -----Original Message-----
>>> From: Prasad Chakka [mailto:[EMAIL PROTECTED]]
>>> Sent: Thursday, April 08, 2010 1:03 PM
>>> To: [EMAIL PROTECTED]
>>> Subject: Re: enough alter tables in the same .q file eventually fail
>>>
>>> There was a bug that got fixed where each request was creating a separate
>>> metastore client. That could be it or something similar that hasn't gotten
>>> fixed.
>>>
>>> On Apr 8, 2010, at 11:47 AM, Edward Capriolo wrote:
>>>
>>> > Hive 5.0 mysql as metastore backend. Using external tables with
>>> location for partitions...
>>> >
>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid >>> '000843') LOCATION 'hit_date=20100329/mid=000843';
>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid >>> '000844') LOCATION 'hit_date=20100329/mid=000844';
>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid >>> '000849') LOCATION 'hit_date=20100329/mid=000849';
>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid >>> '000850') LOCATION 'hit_date=20100329/mid=000850';
>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid >>> '000851') LOCATION 'hit_date=20100329/mid=000851';
>>> > alter table XXXX_action ADD PARTITION (hit_date = '20100329' , mid >>> '000852') LOCATION 'hit_date=20100329/mid=000852';
>>> >
>>> > Eventually this fails after a number of entries.
>>> >
>>> > Time taken: 0.159 seconds
>>> > OK
>>> > Time taken: 0.17 seconds
>>> > OK
>>> > Time taken: 0.241 seconds
>>> > FAILED: Error in metadata: Unable to fetch table XXXXX_action
>>> > FAILED: Execution Error, return code 1 from
>>> org.apache.hadoop.hive.ql.exec.DDLTask
>>> >
>>> > Restarting the process after removing the already added tables works
>>> until it breaks again. Anyone ever dealt with this?
>>> >
>>> > Doing one hive -e per table always works but takes a lot longer ...3
>>> seconds a partition rather then ~.5 seconds.
>>> >
>>> >
>>>
>>>
>> It does not happen after 4 or 5 more like 100 or 1000+. I will try to
>> track this down a bit.
>>
>> Edward
>>
>
>
> Try this:
>
> echo "create external table if not exists edtest ( dat string ) partitioned
> by (dummy string) location '/tmp/a';" > test.q
>  for i in {1..3000} ; do echo "alter table ed_test add partition
> (dummy='${i}') location '/tmp/duh';" ; done >> test.q
> hive -f test.q
>
> On Hive 5.0 I get a failure mid way.
> Edward
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB