Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Removing local job traces in local mode


+
Vincent Barat 2011-06-27, 10:46
+
Vincent 2011-06-27, 10:49
+
Vincent Barat 2011-06-27, 11:21
+
Vincent Barat 2011-08-26, 14:56
Copy link to this message
-
Re: Removing local job traces in local mode
Vincent,

Glad that you were able to solve the issue. Ideally, one should be able to
configure log4j externally through log4j.properties config file and not by
setting them explicitly in code. Did you try that?

Ashutosh
On Fri, Aug 26, 2011 at 07:56, Vincent Barat <[EMAIL PROTECTED]>wrote:

> Here is how I solved this issue (it was only related to log4j
> configuration):
>
>      /* Deactivate most traces from Hadoop and PIG (keep ERROR) */
>      props.setProperty("log4j.**logger.org.apache.hadoop", "ERROR");
>      props.setProperty("log4j.**logger.org.apache.zookeeper", "ERROR");
>      props.setProperty("log4j.**logger.org.apache.pig", "ERROR");
>
>      /*
>       * Keep traces from M/R launcher (percentage of advancement) and job
> statistics (nb of record
>       * written etc.)
>       */
>      props.setProperty(
>        "log4j.logger.org.apache.pig.**backend.hadoop.**executionengine.**
> mapReduceLayer.**MapReduceLauncher",
>        "INFO");
>      props.setProperty("log4j.**logger.org.apache.pig.tools.**pigstats.PigStats",
> "INFO");
>
>
> Le 27/06/11 13:21, Vincent Barat a écrit :
>
>  Thanks, but it is actually not what I'm looking for.
>>
>> First, I use PIG from Java, not from the pig shell, so the "-d" option
>> cannot be used.
>> Second, I want to be able to choose between which trace I want, and which
>> I don't, depending on the origin of the trace.
>> I guess log4j can be configured for that, but I don't manage to configure
>> it correctly.
>>
>> Any hint ?
>>
>> Le 27/06/11 12:49, Vincent a écrit :
>>
>>> You can try -d option:
>>>
>>> For example pig -d WARN -x local myscript.pig
>>>
>>> On Mon, Jun 27, 2011 at 2:46 PM, Vincent Barat<[EMAIL PROTECTED]>*
>>> *wrote:
>>>
>>>  Hi,
>>>>
>>>> I'm using PIG in local mode during development phases, and I get bored
>>>> about traces outputted by local jobs (the one attached).
>>>> Is there a way to get rid of them?
>>>>
>>>> Thanks for your help.
>>>>
>>>> 2011-06-27 12:26:56,289 | WARN  | Thread-55 | JobClient  | Use
>>>> GenericOptionsParser for parsing the arguments. Applications should
>>>> implement Tool for the same.
>>>> 2011-06-27 12:26:56,423 | INFO  | Thread-55 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,427 | INFO  | Thread-55 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,429 | INFO  | Thread-55 | FileInputFormat | Total
>>>> input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,429 | INFO  | Thread-55 | MapRedUtil | Total input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,429 | INFO  | Thread-55 | MapRedUtil | Total input
>>>> paths (combined) to process : 1
>>>> 2011-06-27 12:26:56,614 | INFO  | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,616 | INFO  | Thread-64 | FileInputFormat | Total
>>>> input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,616 | INFO  | Thread-64 | MapRedUtil | Total input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,616 | INFO  | Thread-64 | MapRedUtil | Total input
>>>> paths (combined) to process : 1
>>>> 2011-06-27 12:26:56,653 | INFO  | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,657 | INFO  | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,658 | INFO  | Thread-64 | MapTask    | io.sort.mb >>>> 100
>>>> 2011-06-27 12:26:56,751 | INFO  | Thread-64 | MapTask    | data buffer >>>> 79691776/99614720
>>>> 2011-06-27 12:26:56,751 | INFO  | Thread-64 | MapTask    | record buffer
>>>> >>>> 262144/327680
>>>> 2011-06-27 12:26:56,775 | INFO  | Thread-64 | MapTask    | Starting
>>>> flush
>>>> of map output