Hi,
I m trying to find out best way to add debugging in map- red code.
I have System.out.println() statements that I keep on commenting and uncommenting so as not to increase stdout size

But problem is anytime I need debug, I Hv to re-compile.

If there a way, I can define log levels using log4j in map-red code and define log level as conf option ?

Thanks,
JJ

Sent from my iPhone

Yes this is possible, and there's two ways to do this.

1. Use a distro/release that carries the
https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will let
you avoid work (see 2, which is same as your idea)

2. Configure your implementation's logger object's level in the
setup/setConf methods of the task, by looking at some conf prop to
decide the level. This will work just as well - and will also avoid
changing Hadoop's own Child log levels, unlike the (1) method.

On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]> wrote:
> Hi,
> I m trying to find out best way to add debugging in map- red code.
> I have System.out.println() statements that I keep on commenting and uncommenting so as not to increase stdout size
>
> But problem is anytime I need debug, I Hv to re-compile.
>
> If there a way, I can define log levels using log4j in map-red code and define log level as conf option ?
>
> Thanks,
> JJ
>
> Sent from my iPhone

--
Harsh J

I'm interested in this too, but could you tell me where to apply the patch
and is the following the right command to write it:

 <https://issues.apache.org/jira/secure/attachment/12416955/MAPREDUCE-336_0_20090818.patch>patch
< MAPREDUCE-336_0_20090818.patch<https://issues.apache.org/jira/secure/attachment/12416955/MAPREDUCE-336_0_20090818.patch>

Thank you,
Mark

On Fri, Apr 20, 2012 at 8:28 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Yes this is possible, and there's two ways to do this.
>
> 1. Use a distro/release that carries the
> https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will let
> you avoid work (see 2, which is same as your idea)
>
> 2. Configure your implementation's logger object's level in the
> setup/setConf methods of the task, by looking at some conf prop to
> decide the level. This will work just as well - and will also avoid
> changing Hadoop's own Child log levels, unlike the (1) method.
>
> On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> > I m trying to find out best way to add debugging in map- red code.
> > I have System.out.println() statements that I keep on commenting and
> uncommenting so as not to increase stdout size
> >
> > But problem is anytime I need debug, I Hv to re-compile.
> >
> > If there a way, I can define log levels using log4j in map-red code and
> define log level as conf option ?
> >
> > Thanks,
> > JJ
> >
> > Sent from my iPhone
>
>
>
> --
> Harsh J
>

Hi Harsh,

Does doing (ii) mess up with hadoop (i) level ?

Or does it happen in both the options anyways ?
Thanks,
-JJ

On Fri, Apr 20, 2012 at 8:28 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> Yes this is possible, and there's two ways to do this.
>
> 1. Use a distro/release that carries the
> https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will let
> you avoid work (see 2, which is same as your idea)
>
> 2. Configure your implementation's logger object's level in the
> setup/setConf methods of the task, by looking at some conf prop to
> decide the level. This will work just as well - and will also avoid
> changing Hadoop's own Child log levels, unlike the (1) method.
>
> On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> > I m trying to find out best way to add debugging in map- red code.
> > I have System.out.println() statements that I keep on commenting and
> uncommenting so as not to increase stdout size
> >
> > But problem is anytime I need debug, I Hv to re-compile.
> >
> > If there a way, I can define log levels using log4j in map-red code and
> define log level as conf option ?
> >
> > Thanks,
> > JJ
> >
> > Sent from my iPhone
>
>
>
> --
> Harsh J
>

Doing (ii) would be an isolated app-level config and wouldn't get
affected by the toggling of
(i). The feature from (i) is available already in CDH 4.0.0-b2 btw.

On Fri, May 4, 2012 at 4:58 AM, Mapred Learn <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>
> Does doing (ii) mess up with hadoop (i) level ?
>
> Or does it happen in both the options anyways ?
>
>
> Thanks,
> -JJ
>
> On Fri, Apr 20, 2012 at 8:28 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Yes this is possible, and there's two ways to do this.
>>
>> 1. Use a distro/release that carries the
>> https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will let
>> you avoid work (see 2, which is same as your idea)
>>
>> 2. Configure your implementation's logger object's level in the
>> setup/setConf methods of the task, by looking at some conf prop to
>> decide the level. This will work just as well - and will also avoid
>> changing Hadoop's own Child log levels, unlike the (1) method.
>>
>> On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> > I m trying to find out best way to add debugging in map- red code.
>> > I have System.out.println() statements that I keep on commenting and
>> uncommenting so as not to increase stdout size
>> >
>> > But problem is anytime I need debug, I Hv to re-compile.
>> >
>> > If there a way, I can define log levels using log4j in map-red code and
>> define log level as conf option ?
>> >
>> > Thanks,
>> > JJ
>> >
>> > Sent from my iPhone
>>
>>
>>
>> --
>> Harsh J
>>

--
Harsh J

(Shit, sorry about my last response, I got confused and thought this
were a cdh-user list. I deeply apologize.)

The (i) is in 0.23.1/2.x if you upgrade to that today.

On Fri, May 4, 2012 at 5:02 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> Doing (ii) would be an isolated app-level config and wouldn't get
> affected by the toggling of
> (i). The feature from (i) is available already in CDH 4.0.0-b2 btw.
>
> On Fri, May 4, 2012 at 4:58 AM, Mapred Learn <[EMAIL PROTECTED]> wrote:
>> Hi Harsh,
>>
>> Does doing (ii) mess up with hadoop (i) level ?
>>
>> Or does it happen in both the options anyways ?
>>
>>
>> Thanks,
>> -JJ
>>
>> On Fri, Apr 20, 2012 at 8:28 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> Yes this is possible, and there's two ways to do this.
>>>
>>> 1. Use a distro/release that carries the
>>> https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will let
>>> you avoid work (see 2, which is same as your idea)
>>>
>>> 2. Configure your implementation's logger object's level in the
>>> setup/setConf methods of the task, by looking at some conf prop to
>>> decide the level. This will work just as well - and will also avoid
>>> changing Hadoop's own Child log levels, unlike the (1) method.
>>>
>>> On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]>
>>> wrote:
>>> > Hi,
>>> > I m trying to find out best way to add debugging in map- red code.
>>> > I have System.out.println() statements that I keep on commenting and
>>> uncommenting so as not to increase stdout size
>>> >
>>> > But problem is anytime I need debug, I Hv to re-compile.
>>> >
>>> > If there a way, I can define log levels using log4j in map-red code and
>>> define log level as conf option ?
>>> >
>>> > Thanks,
>>> > JJ
>>> >
>>> > Sent from my iPhone
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>
>
>
> --
> Harsh J

--
Harsh J

Thanks Harsh.

Are they in cdh3 too ?

On Thu, May 3, 2012 at 4:32 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Doing (ii) would be an isolated app-level config and wouldn't get
> affected by the toggling of
> (i). The feature from (i) is available already in CDH 4.0.0-b2 btw.
>
> On Fri, May 4, 2012 at 4:58 AM, Mapred Learn <[EMAIL PROTECTED]>
> wrote:
> > Hi Harsh,
> >
> > Does doing (ii) mess up with hadoop (i) level ?
> >
> > Or does it happen in both the options anyways ?
> >
> >
> > Thanks,
> > -JJ
> >
> > On Fri, Apr 20, 2012 at 8:28 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >
> >> Yes this is possible, and there's two ways to do this.
> >>
> >> 1. Use a distro/release that carries the
> >> https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will let
> >> you avoid work (see 2, which is same as your idea)
> >>
> >> 2. Configure your implementation's logger object's level in the
> >> setup/setConf methods of the task, by looking at some conf prop to
> >> decide the level. This will work just as well - and will also avoid
> >> changing Hadoop's own Child log levels, unlike the (1) method.
> >>
> >> On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi,
> >> > I m trying to find out best way to add debugging in map- red code.
> >> > I have System.out.println() statements that I keep on commenting and
> >> uncommenting so as not to increase stdout size
> >> >
> >> > But problem is anytime I need debug, I Hv to re-compile.
> >> >
> >> > If there a way, I can define log levels using log4j in map-red code
> and
> >> define log level as conf option ?
> >> >
> >> > Thanks,
> >> > JJ
> >> >
> >> > Sent from my iPhone
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>
>
> --
> Harsh J
>

Hi Harsh,
Could you show one sample of how to do this ?

I have not seen/written  any mapper code where people use log4j logger or
log4j file to set the log level.

Thanks in advance
-JJ

On Thu, May 3, 2012 at 4:32 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Doing (ii) would be an isolated app-level config and wouldn't get
> affected by the toggling of
> (i). The feature from (i) is available already in CDH 4.0.0-b2 btw.
>
> On Fri, May 4, 2012 at 4:58 AM, Mapred Learn <[EMAIL PROTECTED]>
> wrote:
> > Hi Harsh,
> >
> > Does doing (ii) mess up with hadoop (i) level ?
> >
> > Or does it happen in both the options anyways ?
> >
> >
> > Thanks,
> > -JJ
> >
> > On Fri, Apr 20, 2012 at 8:28 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >
> >> Yes this is possible, and there's two ways to do this.
> >>
> >> 1. Use a distro/release that carries the
> >> https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will let
> >> you avoid work (see 2, which is same as your idea)
> >>
> >> 2. Configure your implementation's logger object's level in the
> >> setup/setConf methods of the task, by looking at some conf prop to
> >> decide the level. This will work just as well - and will also avoid
> >> changing Hadoop's own Child log levels, unlike the (1) method.
> >>
> >> On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi,
> >> > I m trying to find out best way to add debugging in map- red code.
> >> > I have System.out.println() statements that I keep on commenting and
> >> uncommenting so as not to increase stdout size
> >> >
> >> > But problem is anytime I need debug, I Hv to re-compile.
> >> >
> >> > If there a way, I can define log levels using log4j in map-red code
> and
> >> define log level as conf option ?
> >> >
> >> > Thanks,
> >> > JJ
> >> >
> >> > Sent from my iPhone
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>
>
> --
> Harsh J
>

here is a sample code from log4j documentation
if you want to specify a specific file where you want to write the log ..
you can have a log4j properties file and add it to the classpath

 import com.foo.Bar;

 // Import log4j classes.
 *import org.apache.log4j.Logger;
 import org.apache.log4j.BasicConfigurator;*

 public class MyApp {

   // Define a static logger variable so that it references the
   // Logger instance named "MyApp".
   *static* Logger logger = *Logger.getLogger(MyApp.class);*

   public static void main(String[] args) {

     // Set up a simple configuration that logs on the console.
     *BasicConfigurator.configure();*

     logger.info("Entering application.");
     Bar bar = new Bar();
     bar.doIt();
     logger.info("Exiting application.");
   }
 }
On Sat, May 5, 2012 at 3:40 AM, Mapred Learn <[EMAIL PROTECTED]> wrote:

> Hi Harsh,
> Could you show one sample of how to do this ?
>
> I have not seen/written  any mapper code where people use log4j logger or
> log4j file to set the log level.
>
> Thanks in advance
> -JJ
>
> On Thu, May 3, 2012 at 4:32 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
> > Doing (ii) would be an isolated app-level config and wouldn't get
> > affected by the toggling of
> > (i). The feature from (i) is available already in CDH 4.0.0-b2 btw.
> >
> > On Fri, May 4, 2012 at 4:58 AM, Mapred Learn <[EMAIL PROTECTED]>
> > wrote:
> > > Hi Harsh,
> > >
> > > Does doing (ii) mess up with hadoop (i) level ?
> > >
> > > Or does it happen in both the options anyways ?
> > >
> > >
> > > Thanks,
> > > -JJ
> > >
> > > On Fri, Apr 20, 2012 at 8:28 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> > >
> > >> Yes this is possible, and there's two ways to do this.
> > >>
> > >> 1. Use a distro/release that carries the
> > >> https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will
> let
> > >> you avoid work (see 2, which is same as your idea)
> > >>
> > >> 2. Configure your implementation's logger object's level in the
> > >> setup/setConf methods of the task, by looking at some conf prop to
> > >> decide the level. This will work just as well - and will also avoid
> > >> changing Hadoop's own Child log levels, unlike the (1) method.
> > >>
> > >> On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]
> >
> > >> wrote:
> > >> > Hi,
> > >> > I m trying to find out best way to add debugging in map- red code.
> > >> > I have System.out.println() statements that I keep on commenting and
> > >> uncommenting so as not to increase stdout size
> > >> >
> > >> > But problem is anytime I need debug, I Hv to re-compile.
> > >> >
> > >> > If there a way, I can define log levels using log4j in map-red code
> > and
> > >> define log level as conf option ?
> > >> >
> > >> > Thanks,
> > >> > JJ
> > >> >
> > >> > Sent from my iPhone
> > >>
> > >>
> > >>
> > >> --
> > >> Harsh J
> > >>
> >
> >
> >
> > --
> > Harsh J
> >
>

--
Nitin Pawar

Thanks Nitin but I was asking in context to mapper code..

Sent from my iPhone

On May 4, 2012, at 9:06 PM, Nitin Pawar <[EMAIL PROTECTED]> wrote:

> here is a sample code from log4j documentation
> if you want to specify a specific file where you want to write the log ..
> you can have a log4j properties file and add it to the classpath
>
> import com.foo.Bar;
>
> // Import log4j classes.
> *import org.apache.log4j.Logger;
> import org.apache.log4j.BasicConfigurator;*
>
> public class MyApp {
>
>   // Define a static logger variable so that it references the
>   // Logger instance named "MyApp".
>   *static* Logger logger = *Logger.getLogger(MyApp.class);*
>
>   public static void main(String[] args) {
>
>     // Set up a simple configuration that logs on the console.
>     *BasicConfigurator.configure();*
>
>     logger.info("Entering application.");
>     Bar bar = new Bar();
>     bar.doIt();
>     logger.info("Exiting application.");
>   }
> }
>
>
> On Sat, May 5, 2012 at 3:40 AM, Mapred Learn <[EMAIL PROTECTED]> wrote:
>
>> Hi Harsh,
>> Could you show one sample of how to do this ?
>>
>> I have not seen/written  any mapper code where people use log4j logger or
>> log4j file to set the log level.
>>
>> Thanks in advance
>> -JJ
>>
>> On Thu, May 3, 2012 at 4:32 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> Doing (ii) would be an isolated app-level config and wouldn't get
>>> affected by the toggling of
>>> (i). The feature from (i) is available already in CDH 4.0.0-b2 btw.
>>>
>>> On Fri, May 4, 2012 at 4:58 AM, Mapred Learn <[EMAIL PROTECTED]>
>>> wrote:
>>>> Hi Harsh,
>>>>
>>>> Does doing (ii) mess up with hadoop (i) level ?
>>>>
>>>> Or does it happen in both the options anyways ?
>>>>
>>>>
>>>> Thanks,
>>>> -JJ
>>>>
>>>> On Fri, Apr 20, 2012 at 8:28 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Yes this is possible, and there's two ways to do this.
>>>>>
>>>>> 1. Use a distro/release that carries the
>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will
>> let
>>>>> you avoid work (see 2, which is same as your idea)
>>>>>
>>>>> 2. Configure your implementation's logger object's level in the
>>>>> setup/setConf methods of the task, by looking at some conf prop to
>>>>> decide the level. This will work just as well - and will also avoid
>>>>> changing Hadoop's own Child log levels, unlike the (1) method.
>>>>>
>>>>> On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]
>>>
>>>>> wrote:
>>>>>> Hi,
>>>>>> I m trying to find out best way to add debugging in map- red code.
>>>>>> I have System.out.println() statements that I keep on commenting and
>>>>> uncommenting so as not to increase stdout size
>>>>>>
>>>>>> But problem is anytime I need debug, I Hv to re-compile.
>>>>>>
>>>>>> If there a way, I can define log levels using log4j in map-red code
>>> and
>>>>> define log level as conf option ?
>>>>>>
>>>>>> Thanks,
>>>>>> JJ
>>>>>>
>>>>>> Sent from my iPhone
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>
>
>
> --
> Nitin Pawar

I tried:
 
public class TestMapper extends Mapper<Text,BytesWritable, LongWritable , Text> {
private String numericRegEx = "(-|\\+)?[0-9]+";
 static Logger logger = Logger.getLogger(TestMapper.class);
 

 public void setup(Context context) throws IOException, InterruptedException {
  conf = context.getConfiguration();
  
  org.apache.log4j.Level level = org.apache.log4j.Level.INFO;
  
  logger.setLevel(level);
  
 logger.info("Err File Name: " + errFileName);
}
.
.
 
But this does not add anything to stdout task attempt log.
 
 
Any help on this is appreciated.
 
Thanks,
-JJ
 

Sent from my iPhone

On May 4, 2012, at 9:06 PM, Nitin Pawar <[EMAIL PROTECTED]> wrote:

> here is a sample code from log4j documentation
> if you want to specify a specific file where you want to write the log ..
> you can have a log4j properties file and add it to the classpath
>
> import com.foo.Bar;
>
> // Import log4j classes.
> *import org.apache.log4j.Logger;
> import org.apache.log4j.BasicConfigurator;*
>
> public class MyApp {
>
>   // Define a static logger variable so that it references the
>   // Logger instance named "MyApp".
>   *static* Logger logger = *Logger.getLogger(MyApp.class);*
>
>   public static void main(String[] args) {
>
>     // Set up a simple configuration that logs on the console.
>     *BasicConfigurator.configure();*
>
>     logger.info("Entering application.");
>     Bar bar = new Bar();
>     bar.doIt();
>     logger.info("Exiting application.");
>   }
> }
>
>
> On Sat, May 5, 2012 at 3:40 AM, Mapred Learn <[EMAIL PROTECTED]> wrote:
>
>> Hi Harsh,
>> Could you show one sample of how to do this ?
>>
>> I have not seen/written  any mapper code where people use log4j logger or
>> log4j file to set the log level.
>>
>> Thanks in advance
>> -JJ
>>
>> On Thu, May 3, 2012 at 4:32 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> Doing (ii) would be an isolated app-level config and wouldn't get
>>> affected by the toggling of
>>> (i). The feature from (i) is available already in CDH 4.0.0-b2 btw.
>>>
>>> On Fri, May 4, 2012 at 4:58 AM, Mapred Learn <[EMAIL PROTECTED]>
>>> wrote:
>>>> Hi Harsh,
>>>>
>>>> Does doing (ii) mess up with hadoop (i) level ?
>>>>
>>>> Or does it happen in both the options anyways ?
>>>>
>>>>
>>>> Thanks,
>>>> -JJ
>>>>
>>>> On Fri, Apr 20, 2012 at 8:28 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Yes this is possible, and there's two ways to do this.
>>>>>
>>>>> 1. Use a distro/release that carries the
>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-336 fix. This will
>> let
>>>>> you avoid work (see 2, which is same as your idea)
>>>>>
>>>>> 2. Configure your implementation's logger object's level in the
>>>>> setup/setConf methods of the task, by looking at some conf prop to
>>>>> decide the level. This will work just as well - and will also avoid
>>>>> changing Hadoop's own Child log levels, unlike the (1) method.
>>>>>
>>>>> On Fri, Apr 20, 2012 at 8:47 PM, Mapred Learn <[EMAIL PROTECTED]
>>>
>>>>> wrote:
>>>>>> Hi,
>>>>>> I m trying to find out best way to add debugging in map- red code.
>>>>>> I have System.out.println() statements that I keep on commenting and
>>>>> uncommenting so as not to increase stdout size
>>>>>>
>>>>>> But problem is anytime I need debug, I Hv to re-compile.
>>>>>>
>>>>>> If there a way, I can define log levels using log4j in map-red code
>>> and
>>>>> define log level as conf option ?
>>>>>>
>>>>>> Thanks,
>>>>>> JJ
>>>>>>
>>>>>> Sent from my iPhone
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>
>
>
> --
> Nitin Pawar