Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> How to execute query with timestamp type (Hbase/Hive integeration)


Copy link to this message
-
Re: How to execute query with timestamp type (Hbase/Hive integeration)
Hi Peyton,
I've been using Integer to store Unix timestamps in Hive and that seems to work fine.

Let me know if I misunderstood your question:-)

Mark
----- Original Message -----
From: "Peyton Peng" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Thursday, May 31, 2012 2:41:15 AM
Subject: Re: How to execute query with timestamp type (Hbase/Hive integeration)

Hi Mark,

thanks for your response, I tried with other data type, it seems the issue
occur while query the timestamp field only, not sure how the timestamp
mapping work...

>From the hbase, I seek the data and the value of timestamp(event_time) is:
Wed May 30 16:15:06 CST 2012, should I should the value as long type?

Regards,
Peyton

-----原始邮件-----
From: Mark Grover
Sent: Thursday, May 31, 2012 11:14 AM
To: [EMAIL PROTECTED]
Subject: Re: How to execute query with timestamp type (Hbase/Hive
integeration)

Hi Peyton,
It seems like something to do with timestamp mapping.
What happens if you change your Hive table definition to have the event_time
as int or string?

Mark

----- Original Message -----
From: "Peyton Peng" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, May 30, 2012 5:54:20 AM
Subject: Re: How to execute query with timestamp type (Hbase/Hive
integeration)
Actually I can execute the first sql and it works well, all the libs you
specified is under the hive lib folder,

I doubt if the issue is caused by the timestamp mapping between hbase with
hive..

Regards,
Peyton

From: shashwat shriparv
Sent: Wednesday, May 30, 2012 5:26 PM
To: [EMAIL PROTECTED]
Subject: Re: How to execute query with timestamp type (Hbase/Hive
integeration)

Add these file to hive lib folder

>>> hadoop-0.20-core.jar
>>> hive/lib/hive-exec-0.7.1.jar
>>> hive/lib/hive-jdbc-0.7.1.jar
>>> hive/lib/hive-metastore-0.7.1.jar
>>> hive/lib/hive-service-0.7.1.jar
>>> hive/lib/libfb303.jar
>>> lib/commons-logging-1.0.4.jar
>>> slf4j-api-1.6.1.jar
>>> slf4j-log4j12-1.6.1.jar

and then try

On Wed, May 30, 2012 at 2:23 PM, Peyton Peng < [EMAIL PROTECTED] > wrote:
Hi,

I build the hive table mapped with hbase table,

CREATE TABLE http_access(key string, client_ip string, client_port int,
request_method string, event_time timestamp)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.columns.mapping" =
":key,client:ip,client:port,request:method,request:event_time"
);

The data is store with hbase client.

I get an issue while query with hive for (hbase/hive integration), while I
execute sql: select ip, port, request_method from http_access , it works
well with no problem,

but while I execute below sql: select ip, port, event_time from http_access,
I got below exception.

The only difference between two sqls is: the event_time is timestamp type, I
can scan the corresponding hbase table and see the value of event_time is:
1338365792142 column=request:event_time, timestamp=1338365739818, value=Wed
May 30 16:15:06 CST 2012

Anyone who know what the issue is? (Not sure if I made a wrong mapping or
should I just store the timestamp value as long in hbase? currently I store
the value as java.util.Date)

Thank you very much....

Regards,
Peyton

Exception tracking:

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
****hdfs://Server:9000/user/hive/warehouse/http_access
Starting Job = job_201205291421_0008, Tracking URL =
http://Server:50030/jobdetails.jsp?jobid=job_201205291421_0008
Kill Command = /<here is the hadoop_home>/libexec/../bin/hadoop
job -Dmapred.job.tracker=Server:9001 –kill job_201205291421_0008
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 0
2012-05-30 16:28:01,572 Stage-1 map = 0%, reduce = 0%
2012-05-30 16:28:34,707 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201205291421_0008 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201205291421_0008_m_000002 (and more) from job
job_201205291421_0008
Exception in thread "Thread-211" java.lang.RuntimeException: Error while
reading from task log url
at
org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
at
org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebuggerjava:211)
at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:81)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for
URL:
http://Server:50060/tasklog?taskid=attempt_201205291421_0008_m_000000_1&start=-8193
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1305)
at java.net.URL.openStream(URL.java:1009)
at
org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
.. 3 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

∞ Shashwat Shriparv
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB