Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Fw: pointing mapred.local.dir to a ramdisk


Copy link to this message
-
Re: pointing mapred.local.dir to a ramdisk
Joey Echeverria 2011-10-03, 18:49
Raj,

I just tried this on my CHD3u1 VM, and the ramdisk worked the first
time. So, it's possible you've hit a bug in CDH3b3 that was later
fixed. Can you enable debug logging in log4j.properties and then
repost your task tracker log? I think there might be more details that
it will print that will be helpful.

-Joey

On Mon, Oct 3, 2011 at 2:18 PM, Raj V <[EMAIL PROTECTED]> wrote:
> Edward
>
> I understand the size limitations - but for my experiment the ramdisk size I have created is large enough.
> I think there will be substantial benefits by putting the intermediate map outputs on a ramdisk - size permitting, ofcourse, but I can't provide any numbers to substantiate my claim  given that I can't get it to run.
>
> -best regards
>
> Raj
>
>
>
>>________________________________
>>From: Edward Capriolo <[EMAIL PROTECTED]>
>>To: [EMAIL PROTECTED]
>>Cc: Raj V <[EMAIL PROTECTED]>
>>Sent: Monday, October 3, 2011 10:36 AM
>>Subject: Re: pointing mapred.local.dir to a ramdisk
>>
>>This directory can get very large, in many cases I doubt it would fit on a
>>ram disk.
>>
>>Also RAM Disks tend to help most with random read/write, since hadoop is
>>doing mostly linear IO you may not see a great benefit from the RAM disk.
>>
>>
>>
>>On Mon, Oct 3, 2011 at 12:07 PM, Vinod Kumar Vavilapalli <
>>[EMAIL PROTECTED]> wrote:
>>
>>> Must be related to some kind of permissions problems.
>>>
>>> It will help if you can paste the corresponding source code for
>>> FileUtil.copy(). Hard to track it with different versions, so.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>>
>>> On Mon, Oct 3, 2011 at 9:28 PM, Raj V <[EMAIL PROTECTED]> wrote:
>>>
>>> > Eric
>>> >
>>> > Yes. The owner is hdfs and group is hadoop and the directory is group
>>> > writable(775).  This is tehe exact same configuration I have when I use
>>> real
>>> > disks.But let me give it a try again to see if I overlooked something.
>>> > Thanks
>>> >
>>> > Raj
>>> >
>>> > >________________________________
>>> > >From: Eric Caspole <[EMAIL PROTECTED]>
>>> > >To: [EMAIL PROTECTED]
>>> > >Sent: Monday, October 3, 2011 8:44 AM
>>> > >Subject: Re: pointing mapred.local.dir to a ramdisk
>>> > >
>>> > >Are you sure you have chown'd/chmod'd the ramdisk directory to be
>>> > writeable by your hadoop user? I have played with this in the past and it
>>> > should basically work.
>>> > >
>>> > >
>>> > >On Oct 3, 2011, at 10:37 AM, Raj V wrote:
>>> > >
>>> > >> Sending it to the hadoop mailing list - I think this is a hadoop
>>> related
>>> > problem and not related to Cloudera distribution.
>>> > >>
>>> > >> Raj
>>> > >>
>>> > >>
>>> > >> ----- Forwarded Message -----
>>> > >>> From: Raj V <[EMAIL PROTECTED]>
>>> > >>> To: CDH Users <[EMAIL PROTECTED]>
>>> > >>> Sent: Friday, September 30, 2011 5:21 PM
>>> > >>> Subject: pointing mapred.local.dir to a ramdisk
>>> > >>>
>>> > >>>
>>> > >>> Hi all
>>> > >>>
>>> > >>>
>>> > >>> I have been trying some experiments to improve performance. One of
>>> the
>>> > experiments involved pointing mapred.local.dir to a RAM disk. To this end
>>> I
>>> > created a 128MB RAM disk ( each of my map outputs are smaller than this)
>>> but
>>> > I have not been able to get the task tracker to start.
>>> > >>>
>>> > >>>
>>> > >>> I am running CDH3B3 ( hadoop-0.20.2+737) and here the error message
>>> > from the task tracker log.
>>> > >>>
>>> > >>>
>>> > >>> Tasktracker logs
>>> > >>>
>>> > >>>
>>> > >>> 2011-09-30 16:50:00,689 INFO org.mortbay.log: Logging to
>>> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> > org.mortbay.log.Slf4jLog
>>> > >>> 2011-09-30 16:50:00,930 INFO org.apache.hadoop.http.HttpServer: Added
>>> > global filtersafety
>>> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>> > >>> 2011-09-30 16:50:01,000 INFO org.apache.hadoop.http.HttpServer: Port
>>> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
>>> -1.
>>> > Opening the listener on 50060
>>> > >>> 2011-09-30 16
Joseph Echeverria
Cloudera, Inc.
443.305.9434