Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> alter table add partition error


Copy link to this message
-
RE: alter table add partition error
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:

 

cat create_dummy.sql

CREATE external TABLE if not exists dummy (

 

  partition_name string

  ,partition_id int

)

PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )

row format delimited

stored as textfile

location '/user/pradeepk/dummy';

 

hive -f create_dummy.sql

10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively

Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

OK

Time taken: 0.627 seconds

 

hive  -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"

10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively

Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result

FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask

 

hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"

10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively

Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

OK

Time taken: 5.849 seconds

 

Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.

 

Thanks,

Pradeep

 

 

________________________________

From: Pradeep Kamath [mailto:[EMAIL PROTECTED]]
Sent: Thursday, June 17, 2010 1:25 PM
To: [EMAIL PROTECTED]
Subject: RE: alter table add partition error

 

Here are the create table and alter table statements:

CREATE external TABLE if not exists mytable (

 

  bc string

  ,src_spaceid string

  ,srcpvid string

  ,dstpvid string

  ,dst_spaceid string

  ,page_params map<string, string>

  ,clickinfo map<string, string>

  ,viewinfo array<map<string, string>>

 

)

PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )

row format serde 'com.yahoo.mySerde'

stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

location '/user/pradeepk/mytable';

 

hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid '19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

 

I get the following error:

Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result

FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask

If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:

hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid '19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

 

However I get errors when I try to run a query:

[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"

10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively

Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

Total MapReduce jobs = 1

Launching Job 1 out of 1

Number of reduce tasks is set to 0 since there's no reduce operator

java.lang.IllegalArgumentException: Can not create a Path from an empty
string

        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

        at org.apache.hadoop.fs.Path.<init>(Path.java:90)

        at org.apache.hadoop.fs.Path.<init>(Path.java:50)

        at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

        at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)

        at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

        at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

        at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

        at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

        at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)

        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

 

Any help is much appreciated
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB