Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> how to get the detailed  warn or error log message from pig udfs


Copy link to this message
-
Re: how to get the detailed warn or error log message from pig udfs
Hi Lucas,

It seems that you are using org.apache.pig.EvalFunc.warn(String, Enum)
which acts differently. Check the code or Javadocs. It works through Hadoop
counters I guess. You can use a regular log4j warnings or just
Sysout.out.println. But keep in mind that your UDF is implemented on a
remote node, not on your client node. Usually you can find its outputs when
you look at the WEB UI of the job, when you find the specific task's logs.
But dropping too much WARN's (say one WARN per row) can be really hard for
Hadoop logs, as there can be millions of rows, so Hadoop counters are good
for such things.

Ruslan

On Wed, Apr 17, 2013 at 12:27 PM, 刘振川 <[EMAIL PROTECTED]> wrote:

>
> hey,guys,
>    is there any way to log the detailed warn or error message from my udfs
> ?  Apache Pig version 0.8.1-cdh3u1
>    i wrote the udf warn like this:
>
> IpRegion region = null;
> try {
> region = IPAnalyzeSingleton.getInstance().citySeek(ip);
> } catch (Exception e) {
> warn("invalid ip[" + ip + "],will return -1 .", PigWarning.UDF_WARNING_1);
> return -1;
> }
>
>    after run the script with this command:  pig -4 /tmp/log4j.properties
> test.pig
>
>    only got the count numbers of the warns,but i meaned to get the
> detailed warn info.
>    WARN main
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Encountered Warning UDF_WARNING_1 9261846 time(s).
>
>    the log4j configure as follows:
>
> log4j.rootLogger=WARN, B
>
> log4j.appender.B = org.apache.log4j.DailyRollingFileAppender
> log4j.appender.B.File = /tmp/pig/logs/log.log
> log4j.appender.B.DatePattern='.'yyyy-MM-dd
> log4j.appender.B.Append = true
> log4j.appender.B.Threshold = WARN
> log4j.appender.B.layout = org.apache.log4j.PatternLayout
> log4j.appender.B.layout.ConversionPattern = %p %t %c - %m%n
>
>
>
>
> lucas