+
Harsh J 2012-11-23, 07:42
+
Kartashov, Andy 2012-11-23, 21:44
+
Mohammad Tariq 2012-11-23, 07:54
-
Re: Log files occupy lot of Disk sizeshashwat shriparv 2012-11-23, 08:45
When you run a hive query it internally runs lot of map reduce tasks, which
intern generates lot of temporary files, so your disk uses grows, so can
you tell which folder is taking most of the spaces?
∞
Shashwat Shriparv
On Fri, Nov 23, 2012 at 1:24 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Harsh has got a point. I was thinking the same, but then I thought maybe
> you need all these log files. If not then do as Harsh has suggested. And
> deleting log files won't affect your Hdfs working, but it will not write
> logs for any operation until the next Hdfs restart.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Lower your log levels if you do not need all that verbosity. You can
>> control log retention, max sizes to keep, max number of files to keep,
>> and logging levels, etc. via each components' log4j.properties file.
>>
>> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
>> <[EMAIL PROTECTED]> wrote:
>> > If i delete the log file without stopping the cluster won't it
>> > terminate the session.
>> >
>> >
>> >
>> > On 11/23/12, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> >> Hi there,
>> >>
>> >> You can write a small job or some script which periodically checks
>> for
>> >> the log growth and performs the delete after certain threshold.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >>
>> >>
>> >>
>> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> >> [EMAIL PROTECTED]> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Everytime i query hbase or hive ,there is a significant growth in my
>> >>> log files and it consumes lot of space from my hard disk....(Approx 40
>> >>> gb)
>> >>> So i stop the cluster ,delete all the logs and free the space and then
>> >>> again start the cluster to start my work.
>> >>>
>> >>> Is there any other solution coz i cannot restart the cluster everyday.
>> >>>
>> >>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
intern generates lot of temporary files, so your disk uses grows, so can
you tell which folder is taking most of the spaces?
∞
Shashwat Shriparv
On Fri, Nov 23, 2012 at 1:24 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Harsh has got a point. I was thinking the same, but then I thought maybe
> you need all these log files. If not then do as Harsh has suggested. And
> deleting log files won't affect your Hdfs working, but it will not write
> logs for any operation until the next Hdfs restart.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Lower your log levels if you do not need all that verbosity. You can
>> control log retention, max sizes to keep, max number of files to keep,
>> and logging levels, etc. via each components' log4j.properties file.
>>
>> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
>> <[EMAIL PROTECTED]> wrote:
>> > If i delete the log file without stopping the cluster won't it
>> > terminate the session.
>> >
>> >
>> >
>> > On 11/23/12, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>> >> Hi there,
>> >>
>> >> You can write a small job or some script which periodically checks
>> for
>> >> the log growth and performs the delete after certain threshold.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >>
>> >>
>> >>
>> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> >> [EMAIL PROTECTED]> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Everytime i query hbase or hive ,there is a significant growth in my
>> >>> log files and it consumes lot of space from my hard disk....(Approx 40
>> >>> gb)
>> >>> So i stop the cluster ,delete all the logs and free the space and then
>> >>> again start the cluster to start my work.
>> >>>
>> >>> Is there any other solution coz i cannot restart the cluster everyday.
>> >>>
>> >>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
+
iwannaplay games 2012-11-23, 07:10
