Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> pig-0.10.0: TestJobSubmission failed with: Internal error creating job configuration.


+
lulynn_2008 2012-08-15, 07:56
+
Cheolsoo Park 2012-08-15, 17:30
+
lulynn_2008 2012-08-17, 06:02
+
Cheolsoo Park 2012-08-18, 06:26
+
lulynn_2008 2012-08-20, 03:01
+
lulynn_2008 2012-08-21, 07:24
+
Cheolsoo Park 2012-08-21, 08:25
+
Cheolsoo Park 2012-08-21, 21:51
Copy link to this message
-
Re:Re: Re: Re: pig-0.10.0: TestJobSubmission failed with: Internal error creating job configuration.
Both UT passed now:-)
Thanks Cheolsoo.
At 2012-08-22 05:51:38,"Cheolsoo Park" <[EMAIL PROTECTED]> wrote:
>OK, I got TestJobSubmission passing. Please apply the following diff to
>your Pig:
>
>
>diff --git test/org/apache/pig/test/TestJobSubmission.java
>test/org/apache/pig/test/TestJobSubmission.java
>index 89ac132..211c4ae 100644
>--- test/org/apache/pig/test/TestJobSubmission.java
>+++ test/org/apache/pig/test/TestJobSubmission.java
>@@ -26,8 +26,10 @@ import java.util.Random;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.HBaseTestingUtility;
>*+import org.apache.hadoop.hbase.HConstants;*
> import org.apache.hadoop.hbase.MiniHBaseCluster;
> import org.apache.hadoop.hbase.util.Bytes;
>*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;*
> import org.apache.hadoop.mapred.jobcontrol.Job;
> import org.apache.hadoop.mapred.jobcontrol.JobControl;
> import org.apache.pig.ExecType;
>@@ -541,7 +543,7 @@ public class TestJobSubmission {
>         // use the estimation
>         Configuration conf = cluster.getConfiguration();
>         HBaseTestingUtility util = new HBaseTestingUtility(conf);
>*-        util.startMiniZKCluster();*
>*+        int clientPort = util.startMiniZKCluster().getClientPort();*
>         util.startMiniHBaseCluster(1, 1);
>
>         String query = "a = load '/passwd';" +
>@@ -553,6 +555,7 @@ public class TestJobSubmission {
>
>         pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
>"100");
>         pc.getConf().setProperty("pig.exec.reducers.max", "10");
>*+        pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
>Integer.toString(clientPort));*
>         ConfigurationValidator.validatePigProperties(pc.getProperties());
>         conf = ConfigurationUtil.toConfiguration(pc.getProperties());
>         JobControlCompiler jcc = new JobControlCompiler(pc, conf);
>
>
>There are several changes to MiniZooKeeperCluster in HBase 0.94 compared to
>0.90. In particular, the ZK client port is no longer 21818 by default. In
>fact, it is randomized in 0.94. So the client port has to be explicitly set
>in PigContext at runtime.
>
>Now I am able to run both TestJobSubmission and TestHBaseStorage. Please
>let me know if this works for you. I am going to file a jira to capture
>what I found, so we can make use of it when bumping the Hbase version in
>the future.
>
>Thanks,
>Cheolsoo
>
>
>On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> TestJobSubmission doesn't pass for me either. But it doesn't throw the
>> error that you're seeing. Instead, it hangs at line 583
>> in TestJobSubmission.java. I am not sure yet what's happening though.
>>
>> Thanks,
>> Cheolsoo
>>
>> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Cheolsoo,
>>> Please help to check whether TestJobSunbmission passed in your
>>> environment. Thanks
>>>
>>>
>>> Hi Cheolsoo,
>>> TestHBaseStorage passed now with your fix. Thanks.
>>> But TestJobSubmission still failed with the same previous error with
>>> TestHBaseStorage. Please help to check. Thanks
>>>
>>>
>>>
>>>
>>>
>>> At 2012-08-18 14:26:32,"Cheolsoo Park" <[EMAIL PROTECTED]> wrote:
>>> >Hi,
>>> >
>>> >I tried the same version of libraries as you're using and was able to get
>>> >TestHBaseStorage passing in trunk. Here is what I did:
>>> >
>>> >1) Updated the version of hadoop, hbase, and zookeeper in
>>> >ivy/libraries.properties:
>>> >
>>> >hadoop-core.version=1.0.3
>>> >> hadoop-test.version=1.0.3
>>> >> hbase.version=0.94.0
>>> >> zookeeper.version=3.4.3
>>> >
>>> >
>>> >2) Updated TestHBaseStorage.java since it doesn't compile with
>>> hbase-0.94.0:
>>> >
>>> >HTable table = new HTable(tableName); => HTable table = new HTable(conf,
>>> >> tableName);
>>> >
>>> >
>>> >There are 6 places to fix.
>>> >
>>> >3) Added the following lines to ivy.xml:
>>> >
>>> ><dependency org="com.github.stephenc.high-scale-lib"
>>> name="high-scale-lib"
>>> >> rev="${high-scale-lib.version}" conf="test->default"/>
+
lulynn_2008 2012-08-17, 05:59