Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Run hive queries, and collect job information


+
Mathieu Despriee 2013-01-30, 10:03
+
Qiang Wang 2013-01-30, 10:25
Copy link to this message
-
Re: Run hive queries, and collect job information
for all the queries you run as user1 .. hive stores the hive cli history
into .hive_history file (please check the limits on how many queries it
stores)

For all the jobs hive cli runs, it keeps the details in /tmp/user.name/

all these values are configurable into hive-site.xml
On Wed, Jan 30, 2013 at 3:55 PM, Qiang Wang <[EMAIL PROTECTED]> wrote:

> Every hive query has a history file, and you can get these info from hive
> history file
>
> Following java code can be an example:
>
> https://github.com/anjuke/hwi/blob/master/src/main/java/org/apache/hadoop/hive/hwi/util/QueryUtil.java
>
> Regard,
> Qiang
>
>
> 2013/1/30 Mathieu Despriee <[EMAIL PROTECTED]>
>
>> Hi folks,
>>
>> I would like to run a list of generated HIVE queries. For each, I would
>> like to retrieve the MR job_id (or ids, in case of multiple stages). And
>> then, with this job_id, collect statistics from job tracker (cumulative
>> CPU, read bytes...)
>>
>> How can I send HIVE queries from a bash or python script, and retrieve
>> the job_id(s) ?
>>
>> For the 2nd part (collecting stats for the job), we're using a MRv1
>> Hadoop cluster, so I don't have the AppMaster REST API<http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html>.
>> I'm about to collect data from the jobtracker web UI. Any better idea ?
>>
>> Mathieu
>>
>>
>>
>
--
Nitin Pawar
+
Mathieu Despriee 2013-01-30, 13:52
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB