Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> pig-0.10.0: TestJobSubmission failed with: Internal error creating job configuration.


+
lulynn_2008 2012-08-15, 07:56
+
Cheolsoo Park 2012-08-15, 17:30
+
lulynn_2008 2012-08-17, 06:02
+
Cheolsoo Park 2012-08-18, 06:26
+
lulynn_2008 2012-08-20, 03:01
+
lulynn_2008 2012-08-21, 07:24
+
Cheolsoo Park 2012-08-21, 08:25
Copy link to this message
-
Re: Re: Re: pig-0.10.0: TestJobSubmission failed with: Internal error creating job configuration.
OK, I got TestJobSubmission passing. Please apply the following diff to
your Pig:
diff --git test/org/apache/pig/test/TestJobSubmission.java
test/org/apache/pig/test/TestJobSubmission.java
index 89ac132..211c4ae 100644
--- test/org/apache/pig/test/TestJobSubmission.java
+++ test/org/apache/pig/test/TestJobSubmission.java
@@ -26,8 +26,10 @@ import java.util.Random;

 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.HBaseTestingUtility;
*+import org.apache.hadoop.hbase.HConstants;*
 import org.apache.hadoop.hbase.MiniHBaseCluster;
 import org.apache.hadoop.hbase.util.Bytes;
*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;*
 import org.apache.hadoop.mapred.jobcontrol.Job;
 import org.apache.hadoop.mapred.jobcontrol.JobControl;
 import org.apache.pig.ExecType;
@@ -541,7 +543,7 @@ public class TestJobSubmission {
         // use the estimation
         Configuration conf = cluster.getConfiguration();
         HBaseTestingUtility util = new HBaseTestingUtility(conf);
*-        util.startMiniZKCluster();*
*+        int clientPort = util.startMiniZKCluster().getClientPort();*
         util.startMiniHBaseCluster(1, 1);

         String query = "a = load '/passwd';" +
@@ -553,6 +555,7 @@ public class TestJobSubmission {

         pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
"100");
         pc.getConf().setProperty("pig.exec.reducers.max", "10");
*+        pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
Integer.toString(clientPort));*
         ConfigurationValidator.validatePigProperties(pc.getProperties());
         conf = ConfigurationUtil.toConfiguration(pc.getProperties());
         JobControlCompiler jcc = new JobControlCompiler(pc, conf);
There are several changes to MiniZooKeeperCluster in HBase 0.94 compared to
0.90. In particular, the ZK client port is no longer 21818 by default. In
fact, it is randomized in 0.94. So the client port has to be explicitly set
in PigContext at runtime.

Now I am able to run both TestJobSubmission and TestHBaseStorage. Please
let me know if this works for you. I am going to file a jira to capture
what I found, so we can make use of it when bumping the Hbase version in
the future.

Thanks,
Cheolsoo
On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <[EMAIL PROTECTED]>wrote:

> Hi,
>
> TestJobSubmission doesn't pass for me either. But it doesn't throw the
> error that you're seeing. Instead, it hangs at line 583
> in TestJobSubmission.java. I am not sure yet what's happening though.
>
> Thanks,
> Cheolsoo
>
> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <[EMAIL PROTECTED]> wrote:
>
>> Hi Cheolsoo,
>> Please help to check whether TestJobSunbmission passed in your
>> environment. Thanks
>>
>>
>> Hi Cheolsoo,
>> TestHBaseStorage passed now with your fix. Thanks.
>> But TestJobSubmission still failed with the same previous error with
>> TestHBaseStorage. Please help to check. Thanks
>>
>>
>>
>>
>>
>> At 2012-08-18 14:26:32,"Cheolsoo Park" <[EMAIL PROTECTED]> wrote:
>> >Hi,
>> >
>> >I tried the same version of libraries as you're using and was able to get
>> >TestHBaseStorage passing in trunk. Here is what I did:
>> >
>> >1) Updated the version of hadoop, hbase, and zookeeper in
>> >ivy/libraries.properties:
>> >
>> >hadoop-core.version=1.0.3
>> >> hadoop-test.version=1.0.3
>> >> hbase.version=0.94.0
>> >> zookeeper.version=3.4.3
>> >
>> >
>> >2) Updated TestHBaseStorage.java since it doesn't compile with
>> hbase-0.94.0:
>> >
>> >HTable table = new HTable(tableName); => HTable table = new HTable(conf,
>> >> tableName);
>> >
>> >
>> >There are 6 places to fix.
>> >
>> >3) Added the following lines to ivy.xml:
>> >
>> ><dependency org="com.github.stephenc.high-scale-lib"
>> name="high-scale-lib"
>> >> rev="${high-scale-lib.version}" conf="test->default"/>
>> >> <dependency org="com.google.protobuf" name="protobuf-java"
>> >> rev="${protobuf-java.version}" conf="test->default"/>
>> >
>> >
>> >I also added the following line to ivy/libraries.properties:
+
lulynn_2008 2012-08-22, 02:55
+
lulynn_2008 2012-08-17, 05:59