Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Problems using Pig with Oozie


Copy link to this message
-
Problems using Pig with Oozie
All,
I have a Pig script that reads data from HBase using HBaseStorage, does
some manipulation with some Python UDFs and then writes it using
PigStorage.  It works fine when I run it as a standalone script, but will
not run in an Oozie workflow.  I can run normal Pig scripts using Oozie,
but run into problems when trying to run this script.  I believe I have
isolated the error to be with loading from HBaseStorage.  I stripped
everything out of my script except loading from HBaseStorage and outputting
to PigStorage.  The full script is below.

profiles = LOAD 'hbase://profile' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('e:*') as (columns:map[]);
limited = LIMIT profiles 200;
STORE limited into '/tmp/123456' using PigStorage();

The log files are not very helpful.  It gives me an error Launcher ERROR,
reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2].
 I also included the workflow and logfile below just in case.  Also, I am
running Cloudera 4.1.2.  I added all of the Oozie libraries to HDFS as
specified in the setup instructions.  I appreciate any help.

Thanks,
Shawn

<workflow-app xmlns="uri:oozie:workflow:0.3" name="simple-wf">
    <start to="pig-node"/>
    <action name="pig-node">
        <pig>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/${wf:user()}/tmp/65321"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>mapred.compress.map.output</name>
                    <value>true</value>
                </property>
            </configuration>
            <script>simple.pig</script>
        </pig>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Job failed,error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

2013-02-02 14:42:50,123 INFO
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@:start:] Start action
[0000035-130130192939212-oozie-oozi-W@:start:] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-02-02 14:42:50,123 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@:start:]
[***0000035-130130192939212-oozie-oozi-W@:start:***]Action status=DONE
2013-02-02 14:42:50,123 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@:start:]
[***0000035-130130192939212-oozie-oozi-W@:start:***]Action updated in DB!
2013-02-02 14:42:50,257 INFO
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] Start action
[0000035-130130192939212-oozie-oozi-W@pig-node] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-02-02 14:42:51,090 WARN
org.apache.oozie.action.hadoop.PigActionExecutor: USER[dev] GROUP[-]
TOKEN[] APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] credentials is null
for the action
2013-02-02 14:42:53,047 INFO
org.apache.oozie.action.hadoop.PigActionExecutor: USER[dev] GROUP[-]
TOKEN[] APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] checking action,
external ID [job_201301031449_0327] status [RUNNING]
2013-02-02 14:42:53,150 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node]
[***0000035-130130192939212-oozie-oozi-W@pig-node***]Action status=RUNNING
2013-02-02 14:42:53,151 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node]
[***0000035-130130192939212-oozie-oozi-W@pig-node***]Action updated in DB!
2013-02-02 14:53:12,231 INFO
org.apache.oozie.action.hadoop.PigActionExecutor: USER[dev] GROUP[-]
TOKEN[] APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] action completed,
external ID [job_201301031449_0327]
2013-02-02 14:53:12,272 WARN
org.apache.oozie.action.hadoop.PigActionExecutor: USER[dev] GROUP[-]
TOKEN[] APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] Launcher ERROR,
reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]
2013-02-02 14:53:12,433 INFO org.apache.oozie.command.wf.ActionEndXCommand:
USER[dev] GROUP[-] TOKEN[] APP[simple-wf]
JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@pig-node] ERROR is considered
as FAILED for SLA
2013-02-02 14:53:12,480 INFO
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@fail] Start action
[0000035-130130192939212-oozie-oozi-W@fail] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-02-02 14:53:12,480 WARN
org.apache.oozie.command.wf.ActionStartXCommand: USER[dev] GROUP[-] TOKEN[]
APP[simple-wf] JOB[0000035-130130192939212-oozie-oozi-W]
ACTION[0000035-130130192939212-oozie-oozi-W@fail]
[***0000035-130130192939212-oozie-oozi-W@fail***]Action
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB