Pig, mail # user - Re: Removing local job traces in local mode

Removing local job traces in local mode

Vincent Barat 2011-06-27, 10:46

Re: Removing local job traces in local mode

Vincent 2011-06-27, 10:49

Re: Removing local job traces in local mode

Vincent Barat 2011-06-27, 11:21

Re: Removing local job traces in local mode

Vincent Barat 2011-08-26, 14:56

Here is how I solved this issue (it was only related to log4j
configuration):

   /* Deactivate most traces from Hadoop and PIG (keep ERROR) */
   props.setProperty("log4j.logger.org.apache.hadoop", "ERROR");
   props.setProperty("log4j.logger.org.apache.zookeeper", "ERROR");
   props.setProperty("log4j.logger.org.apache.pig", "ERROR");

   /*
   * Keep traces from M/R launcher (percentage of advancement)
and job statistics (nb of record
   * written etc.)
   */
   props.setProperty(

"log4j.logger.org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher",
   "INFO");

props.setProperty("log4j.logger.org.apache.pig.tools.pigstats.PigStats",
"INFO");
Le 27/06/11 13:21, Vincent Barat a ï¿½crit :
> Thanks, but it is actually not what I'm looking for.
>
> First, I use PIG from Java, not from the pig shell, so the "-d"
> option cannot be used.
> Second, I want to be able to choose between which trace I want,
> and which I don't, depending on the origin of the trace.
> I guess log4j can be configured for that, but I don't manage to
> configure it correctly.
>
> Any hint ?
>
> Le 27/06/11 12:49, Vincent a ï¿½crit :
>> You can try -d option:
>>
>> For example pig -d WARN -x local myscript.pig
>>
>> On Mon, Jun 27, 2011 at 2:46 PM, Vincent
>> Barat<[EMAIL PROTECTED]>wrote:
>>
>>> Hi,
>>>
>>> I'm using PIG in local mode during development phases, and I get
>>> bored
>>> about traces outputted by local jobs (the one attached).
>>> Is there a way to get rid of them?
>>>
>>> Thanks for your help.
>>>
>>> 2011-06-27 12:26:56,289 | WARN | Thread-55 | JobClient | Use
>>> GenericOptionsParser for parsing the arguments. Applications should
>>> implement Tool for the same.
>>> 2011-06-27 12:26:56,423 | INFO | Thread-55 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,427 | INFO | Thread-55 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat |
>>> Total input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total
>>> input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total
>>> input
>>> paths (combined) to process : 1
>>> 2011-06-27 12:26:56,614 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat |
>>> Total input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total
>>> input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total
>>> input
>>> paths (combined) to process : 1
>>> 2011-06-27 12:26:56,653 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,657 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask |
>>> io.sort.mb = 100
>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data
>>> buffer >>> 79691776/99614720
>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask |
>>> record buffer >>> 262144/327680
>>> 2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask |
>>> Starting flush
>>> of map output
>>> 2011-06-27 12:26:56,785 | INFO | Thread-64 | MapTask |
>>> Finished spill 0
>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | LocalJobRunner |
>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | TaskRunner | Task
>>> 'attempt_local_0004_m_000000_**0' done.
>>> 2011-06-27 12:26:56,795 | INFO | Thread-64 | JvmMetrics | Cannot

Re: Removing local job traces in local mode

Ashutosh Chauhan 2011-08-26, 16:14

Vincent,

Glad that you were able to solve the issue. Ideally, one should be able to
configure log4j externally through log4j.properties config file and not by
setting them explicitly in code. Did you try that?

Ashutosh
On Fri, Aug 26, 2011 at 07:56, Vincent Barat <[EMAIL PROTECTED]>wrote:

> Here is how I solved this issue (it was only related to log4j
> configuration):
>
> /* Deactivate most traces from Hadoop and PIG (keep ERROR) */
> props.setProperty("log4j.**logger.org.apache.hadoop", "ERROR");
> props.setProperty("log4j.**logger.org.apache.zookeeper", "ERROR");
> props.setProperty("log4j.**logger.org.apache.pig", "ERROR");
>
> /*
> * Keep traces from M/R launcher (percentage of advancement) and job
> statistics (nb of record
> * written etc.)
> */
> props.setProperty(
> "log4j.logger.org.apache.pig.**backend.hadoop.**executionengine.**
> mapReduceLayer.**MapReduceLauncher",
> "INFO");
> props.setProperty("log4j.**logger.org.apache.pig.tools.**pigstats.PigStats",
> "INFO");
>
>
> Le 27/06/11 13:21, Vincent Barat a écrit :
>
> Thanks, but it is actually not what I'm looking for.
>>
>> First, I use PIG from Java, not from the pig shell, so the "-d" option
>> cannot be used.
>> Second, I want to be able to choose between which trace I want, and which
>> I don't, depending on the origin of the trace.
>> I guess log4j can be configured for that, but I don't manage to configure
>> it correctly.
>>
>> Any hint ?
>>
>> Le 27/06/11 12:49, Vincent a écrit :
>>
>>> You can try -d option:
>>>
>>> For example pig -d WARN -x local myscript.pig
>>>
>>> On Mon, Jun 27, 2011 at 2:46 PM, Vincent Barat<[EMAIL PROTECTED]>*
>>> *wrote:
>>>
>>> Hi,
>>>>
>>>> I'm using PIG in local mode during development phases, and I get bored
>>>> about traces outputted by local jobs (the one attached).
>>>> Is there a way to get rid of them?
>>>>
>>>> Thanks for your help.
>>>>
>>>> 2011-06-27 12:26:56,289 | WARN | Thread-55 | JobClient | Use
>>>> GenericOptionsParser for parsing the arguments. Applications should
>>>> implement Tool for the same.
>>>> 2011-06-27 12:26:56,423 | INFO | Thread-55 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,427 | INFO | Thread-55 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat | Total
>>>> input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
>>>> paths (combined) to process : 1
>>>> 2011-06-27 12:26:56,614 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat | Total
>>>> input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
>>>> paths (combined) to process : 1
>>>> 2011-06-27 12:26:56,653 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,657 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask | io.sort.mb >>>> 100
>>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data buffer >>>> 79691776/99614720
>>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | record buffer
>>>> >>>> 262144/327680
>>>> 2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask | Starting
>>>> flush
>>>> of map output