Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> Re: Review Request: HIVE-4513 - disable hivehistory logs by default

Copy link to this message
Re: Review Request: HIVE-4513 - disable hivehistory logs by default

This is an automatically generated e-mail. To reply, visit:

(Updated May 13, 2013, 10:12 p.m.)
Review request for hive.

Updating review with background of the changes.
Description (updated)

HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information.

There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space.
I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs.

The change :
A new config parameter hive.session.history.enabled controls if the history-log is enabled. By default it is set to false.
SessionState initializes the HiveHIstory object. When this config is set to false, it creates a Proxy object that does not do anything. I did this instead of having SessionState return null, because that would add null checks in too many places. This keeps the code cleaner and avoids possibility of NPE.
As the proxy only works against interfaces, i created a HiveHistory interface, moved the implementation to HiveHistoryImpl. static functions were moved to HiveHistoryUtil .
This addresses bug HIVE-4513.

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1672453
  conf/hive-default.xml.template 3a7d1dc
  data/conf/hive-site.xml 544ba35
  ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java e1c1ae3
  ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java fdd56db
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 3d43451
  ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java a783303

Diff: https://reviews.apache.org/r/11029/diff/

Thejas Nair