Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Problems using Pig with Oozie


Copy link to this message
-
Problems using Pig with Oozie
All,
I have a Pig script that reads data from HBase using HBaseStorage, does
some manipulation with some Python UDFs and then writes it using
PigStorage.  It works fine when I run it as a standalone script, but will
not run in an Oozie workflow.  I can run normal Pig scripts using Oozie,
but run into problems when trying to run this script.  I believe I have
isolated the error to be with loading from HBaseStorage.  I stripped
everything out of my script except loading from HBaseStorage and outputting
to PigStorage.  The full script is below.

profiles = LOAD 'hbase://profile' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('e:*') as (columns:map[]);
limited = LIMIT profiles 200;
STORE limited into '/tmp/123456' using PigStorage();

The log files are not very helpful.  It gives me an error Launcher ERROR,
reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2].
 I also included the workflow and logfile below just in case.  Also, I am
running Cloudera 4.1.2.  I added all of the Oozie libraries to HDFS as
specified in the setup instructions.  I appreciate any help.

Thanks,
Shawn

<workflow-app xmlns="uri:oozie:workflow:0.3" name="simple-wf">
    <start to="pig-node"/>
    <action name="pig-node">
        <pig>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/${wf:user()}/tmp/65321"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>mapred.compress.map.output</name>
                    <value>true</value>
                </property>
            </configuration>
            <script>simple.pig</script>
        </pig>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Job failed,error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

2013-02-02 14:42:50,123 INFO
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@:start:] Start action
[0000035-130130192939212-oozie-oozi-W@:start:] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-02-02 14:42:50,123 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@:start:]
[***0000035-130130192939212-oozie-oozi-W@:start:***]Action status=DONE
2013-02-02 14:42:50,123 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@:start:]
[***0000035-130130192939212-oozie-oozi-W@:start:***]Action updated in DB!
2013-02-02 14:42:50,257 INFO
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] Start action
[0000035-130130192939212-oozie-oozi-W@pig-node] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-02-02 14:42:51,090 WARN
org.apache.oozie.action.hadoop.PigActionExecutor: USER[dev] GROUP[-]
TOKEN[] APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] credentials is null
for the action
2013-02-02 14:42:53,047 INFO
org.apache.oozie.action.hadoop.PigActionExecutor: USER[dev] GROUP[-]
TOKEN[] APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] checking action,
external ID [job_201301031449_0327] status [RUNNING]
2013-02-02 14:42:53,150 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node]
[***0000035-130130192939212-oozie-oozi-W@pig-node***]Action status=RUNNING
2013-02-02 14:42:53,151 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node]
[***0000035-130130192939212-oozie-oozi-W@pig-node***]Action updated in DB!
2013-02-02 14:53:12,231 INFO
org.apache.oozie.action.hadoop.PigActionExecutor: USER[dev] GROUP[-]
TOKEN[] APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] action completed,
external ID [job_201301031449_0327]
2013-02-02 14:53:12,272 WARN
org.apache.oozie.action.hadoop.PigActionExecutor: USER[dev] GROUP[-]
TOKEN[] APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] Launcher ERROR,
reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]
2013-02-02 14:53:12,433 INFO org.apache.oozie.command.wf.ActionEndXCommand:
USER[dev] GROUP[-] TOKEN[] APP[simple-wf]
JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] ERROR is considered
as FAILED for SLA
2013-02-02 14:53:12,480 INFO
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@fail] Start action
[0000035-130130192939212-oozie-oozi-W@fail] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-02-02 14:53:12,480 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@fail]
[***0000035-130130192939212-oozie-oozi-W@fail***]Action
+
Jonas Hartwig 2013-02-02, 20:50
+
Shawn Hermans 2013-02-02, 21:42
+
Prashant Kommireddi 2013-02-02, 22:21
+
Harish Krishnan 2013-02-02, 22:30
+
Shawn Hermans 2013-02-03, 21:51