Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - Re: Review Request: HIVE-4513 - disable hivehistory logs by default


Copy link to this message
-
Re: Review Request: HIVE-4513 - disable hivehistory logs by default
Thejas Nair 2013-05-13, 22:12


> On May 9, 2013, 4:37 p.m., Brock Noland wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java, lines 71-73
> > <https://reviews.apache.org/r/11029/diff/1/?file=289274#file289274line71>
> >
> >     This is bad... I know it's not related to your change but can we fix this?

I have made things slightly better by  logging the error. I looked at throwing an exception, but that would need changes in other classes to handle the exception correctly (Such as hive web interface classes).  Since this code is unrelated to the patch, and it is not a 1-2 liner, I think we should address that separately.
- Thejas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11029/#review20380
-----------------------------------------------------------
On May 13, 2013, 10:12 p.m., Thejas Nair wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11029/
> -----------------------------------------------------------
>
> (Updated May 13, 2013, 10:12 p.m.)
>
>
> Review request for hive.
>
>
> Description
> -------
>
> HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information.
>
> There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space.
> I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs.
>
> The change :
> A new config parameter hive.session.history.enabled controls if the history-log is enabled. By default it is set to false.
> SessionState initializes the HiveHIstory object. When this config is set to false, it creates a Proxy object that does not do anything. I did this instead of having SessionState return null, because that would add null checks in too many places. This keeps the code cleaner and avoids possibility of NPE.
> As the proxy only works against interfaces, i created a HiveHistory interface, moved the implementation to HiveHistoryImpl. static functions were moved to HiveHistoryUtil .
>
>
> This addresses bug HIVE-4513.
>     https://issues.apache.org/jira/browse/HIVE-4513
>
>
> Diffs
> -----
>
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1672453
>   conf/hive-default.xml.template 3a7d1dc
>   data/conf/hive-site.xml 544ba35
>   ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java e1c1ae3
>   ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java PRE-CREATION
>   ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java PRE-CREATION
>   ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java PRE-CREATION
>   ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java fdd56db
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 3d43451
>   ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java a783303
>
> Diff: https://reviews.apache.org/r/11029/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Thejas Nair
>
>