Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> dynamic Partition not splitting properly


Copy link to this message
-
Re: dynamic Partition not splitting properly
Im trying to create partition table (dynamically) from old non partitioned
table. the query is as follow

*INSERT OVERWRITE TABLE new_events_details Partition (event_date) SELECT
id, event_id, user_id, intval_1, intval_2, intval_3, intval_4, intval_5,
intval_6, intval_7, intval_8, intval_9, intval_10, intval_11, intval_12,
intval_13, intval_14, intval_15, intval_16, intval_17, intval_18,
intval_19, intval_20, intval_21, intval_22, intval_23, intval_24,
intval_25, intval_26 , to_date(event_date) FROM events_details;*

After waiting for more then 2 hours, following exceptions raised and
further executions stops

*spark.SparkException: Job failed: ResultTask(0, 1063) failed:
ExceptionFailure(org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/tmp/hive-hadoop/hive_2013-06-16_15-08-26_985_3160022698353542666/_task_tmp.-ext-10000/event_date=2013-02-22/_tmp.001063_0
could only be replicated to 0 nodes, instead of 1
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
)
    at
spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:529)
    at
spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:527)
    at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:527)
    at
spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:497)
    at spark.scheduler.DAGScheduler.run(DAGScheduler.scala:269)
    at spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:90)
FAILED: Execution Error, return code -101 from shark.execution.SparkTask
*
why it giving me exception?
On Fri, Jun 14, 2013 at 1:38 PM, Hamza Asad <[EMAIL PROTECTED]> wrote:

> ok .. got it.. Thanx :)
> p.s Nitin, have u any idea of indexes? i have emailed issue with subject "
> *index not working*", can u see it please?
>
>
> On Fri, Jun 14, 2013 at 1:30 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
>> thats a wrong query
>>
>> insert into table table_name partition (partition_names) select cols,
>> do_data(event_date) from table
>>
>> this is how it should look like
>> hive will take care of inserting into respective partitions after you
>> enable dynamic partitions
>>
>>
>> On Fri, Jun 14, 2013 at 1:21 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:
>>
>>> i 'm executing following command but it fail to recognize partition
>>> column
>>> *INSERT INTO TABLE rc_partition_table
>>> PARTITION (to_date(event_date))
>>> SELECT * FROM events_details*
>>>
>>>
>>> On Fri, Jun 14, 2013 at 12:41 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>>
>>>> just use hive split function for strings and get the value extracted.
>>>> by the way I am not sure why the to_date function is failing
>>>> stringto_date(string timestamp)Returns the date part of a timestamp
>>>> string: to_date("1970-01-01 00:00:00") = "1970-01-01"
>>>> also, I am sorry i might have misread your question
>>>> what do you mean by "partition column does not accepts
>>>> to_date(event_date) form "
>>>>
>>>>
>>>>
>>>> On Fri, Jun 14, 2013 at 1:04 PM, Hamza Asad <[EMAIL PROTECTED]>wrote:

*Muhammad Hamza Asad*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB