Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Profiling Hadoop Job


Copy link to this message
-
Re: Profiling Hadoop Job
The JobClient is trying to download the profile output to the local
directory. It seems like you don't have write permissions in the
current working directory where you are running the JobClient. Please
check that.

HTH.

+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Thu, Mar 8, 2012 at 3:13 PM, Leonardo Urbina <[EMAIL PROTECTED]> wrote:
> Does anyone have any idea how to solve this problem? Regardless of whether
> I'm using plain HPROF or profiling through Starfish, I am getting the same
> error:
>
> Exception in thread "main" java.io.FileNotFoundException:
> attempt_201203071311_0004_m_
> 000000_0.profile (Permission denied)
>        at java.io.FileOutputStream.open(Native Method)
>        at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
>        at java.io.FileOutputStream.<init>(FileOutputStream.java:84)
>        at
> org.apache.hadoop.mapred.JobClient.downloadProfile(JobClient.java:1226)
>        at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1302)
>        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1251)
>        at
> com.BitSight.hadoopAggregator.AggregatorDriver.run(AggregatorDriver.java:89)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at
> com.BitSight.hadoopAggregator.AggregatorDriver.main(AggregatorDriver.java:94)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> But I can't find what permissions to change to fix this issue. Any ideas?
> Thanks in advance,
>
> Best,
> -Leo
>
>
> On Wed, Mar 7, 2012 at 3:52 PM, Leonardo Urbina <[EMAIL PROTECTED]> wrote:
>
>> Thanks,
>> -Leo
>>
>>
>> On Wed, Mar 7, 2012 at 3:47 PM, Jie Li <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Leo,
>>>
>>> Thanks for pointing out the outdated README file.  Glad to tell you that
>>> we
>>> do support the old API in the latest version. See here:
>>>
>>> http://www.cs.duke.edu/starfish/previous.html
>>>
>>> Welcome to join our mailing list and your questions will reach more of our
>>> group members.
>>>
>>> Jie
>>>
>>> On Wed, Mar 7, 2012 at 3:37 PM, Leonardo Urbina <[EMAIL PROTECTED]> wrote:
>>>
>>> > Hi Jie,
>>> >
>>> > According to the Starfish README, the hadoop programs must be written
>>> using
>>> > the new Hadoop API. This is not my case (I am using MultipleInputs among
>>> > other non-new API supported features). Is there any way around this?
>>> > Thanks,
>>> >
>>> > -Leo
>>> >
>>> > On Wed, Mar 7, 2012 at 3:19 PM, Jie Li <[EMAIL PROTECTED]> wrote:
>>> >
>>> > > Hi Leonardo,
>>> > >
>>> > > You might want to try Starfish which supports the memory profiling as
>>> > well
>>> > > as cpu/disk/network profiling for the performance tuning.
>>> > >
>>> > > Jie
>>> > > ------------------
>>> > > Starfish is an intelligent performance tuning tool for Hadoop.
>>> > > Homepage: www.cs.duke.edu/starfish/
>>> > > Mailing list: http://groups.google.com/group/hadoop-starfish
>>> > >
>>> > >
>>> > > On Wed, Mar 7, 2012 at 2:36 PM, Leonardo Urbina <[EMAIL PROTECTED]>
>>> wrote:
>>> > >
>>> > > > Hello everyone,
>>> > > >
>>> > > > I have a Hadoop job that I run on several GBs of data that I am
>>> trying
>>> > to
>>> > > > optimize in order to reduce the memory consumption as well as
>>> improve
>>> > the
>>> > > > speed. I am following the steps outlined in Tom White's "Hadoop: The
>>> > > > Definitive Guide" for profiling using HPROF (p161), by setting the
>>> > > > following properties in the JobConf:
>>> > > >
>>> > > >        job.setProfileEnabled(true);
>>> > > >
>>> > > >
>>> job.setProfileParams("-agentlib:hprof=cpu=samples,heap=sites,depth=6,"
>>> > +
>>> > > >                "force=n,thread=y,verbose=n,file=%s");
>>> > > >        job.setProfileTaskRange(true, "0-2");