Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Re:Re: Re: Re: pig-0.10.0: TestJobSubmission failed with: Internal error creating job configuration.


Copy link to this message
-
Re:Re: Re: Re: pig-0.10.0: TestJobSubmission failed with: Internal error creating job configuration.
Both UT passed now:-)
Thanks Cheolsoo.
At 2012-08-22 05:51:38,"Cheolsoo Park" <[EMAIL PROTECTED]> wrote:
>OK, I got TestJobSubmission passing. Please apply the following diff to
>your Pig:
>
>
>diff --git test/org/apache/pig/test/TestJobSubmission.java
>test/org/apache/pig/test/TestJobSubmission.java
>index 89ac132..211c4ae 100644
>--- test/org/apache/pig/test/TestJobSubmission.java
>+++ test/org/apache/pig/test/TestJobSubmission.java
>@@ -26,8 +26,10 @@ import java.util.Random;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.HBaseTestingUtility;
>*+import org.apache.hadoop.hbase.HConstants;*
> import org.apache.hadoop.hbase.MiniHBaseCluster;
> import org.apache.hadoop.hbase.util.Bytes;
>*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;*
> import org.apache.hadoop.mapred.jobcontrol.Job;
> import org.apache.hadoop.mapred.jobcontrol.JobControl;
> import org.apache.pig.ExecType;
>@@ -541,7 +543,7 @@ public class TestJobSubmission {
>         // use the estimation
>         Configuration conf = cluster.getConfiguration();
>         HBaseTestingUtility util = new HBaseTestingUtility(conf);
>*-        util.startMiniZKCluster();*
>*+        int clientPort = util.startMiniZKCluster().getClientPort();*
>         util.startMiniHBaseCluster(1, 1);
>
>         String query = "a = load '/passwd';" +
>@@ -553,6 +555,7 @@ public class TestJobSubmission {
>
>         pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
>"100");
>         pc.getConf().setProperty("pig.exec.reducers.max", "10");
>*+        pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
>Integer.toString(clientPort));*
>         ConfigurationValidator.validatePigProperties(pc.getProperties());
>         conf = ConfigurationUtil.toConfiguration(pc.getProperties());
>         JobControlCompiler jcc = new JobControlCompiler(pc, conf);
>
>
>There are several changes to MiniZooKeeperCluster in HBase 0.94 compared to
>0.90. In particular, the ZK client port is no longer 21818 by default. In
>fact, it is randomized in 0.94. So the client port has to be explicitly set
>in PigContext at runtime.
>
>Now I am able to run both TestJobSubmission and TestHBaseStorage. Please
>let me know if this works for you. I am going to file a jira to capture
>what I found, so we can make use of it when bumping the Hbase version in
>the future.
>
>Thanks,
>Cheolsoo
>
>
>On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> TestJobSubmission doesn't pass for me either. But it doesn't throw the
>> error that you're seeing. Instead, it hangs at line 583
>> in TestJobSubmission.java. I am not sure yet what's happening though.
>>
>> Thanks,
>> Cheolsoo
>>
>> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Cheolsoo,
>>> Please help to check whether TestJobSunbmission passed in your
>>> environment. Thanks
>>>
>>>
>>> Hi Cheolsoo,
>>> TestHBaseStorage passed now with your fix. Thanks.
>>> But TestJobSubmission still failed with the same previous error with
>>> TestHBaseStorage. Please help to check. Thanks
>>>
>>>
>>>
>>>
>>>
>>> At 2012-08-18 14:26:32,"Cheolsoo Park" <[EMAIL PROTECTED]> wrote:
>>> >Hi,
>>> >
>>> >I tried the same version of libraries as you're using and was able to get
>>> >TestHBaseStorage passing in trunk. Here is what I did:
>>> >
>>> >1) Updated the version of hadoop, hbase, and zookeeper in
>>> >ivy/libraries.properties:
>>> >
>>> >hadoop-core.version=1.0.3
>>> >> hadoop-test.version=1.0.3
>>> >> hbase.version=0.94.0
>>> >> zookeeper.version=3.4.3
>>> >
>>> >
>>> >2) Updated TestHBaseStorage.java since it doesn't compile with
>>> hbase-0.94.0:
>>> >
>>> >HTable table = new HTable(tableName); => HTable table = new HTable(conf,
>>> >> tableName);
>>> >
>>> >
>>> >There are 6 places to fix.
>>> >
>>> >3) Added the following lines to ivy.xml:
>>> >
>>> ><dependency org="com.github.stephenc.high-scale-lib"
>>> name="high-scale-lib"
>>> >> rev="${high-scale-lib.version}" conf="test->default"/>
+
lulynn_2008 2012-08-15, 07:56
+
Cheolsoo Park 2012-08-15, 17:30
+
lulynn_2008 2012-08-17, 06:02
+
Cheolsoo Park 2012-08-18, 06:26
+
lulynn_2008 2012-08-20, 03:01
+
lulynn_2008 2012-08-21, 07:24
+
Cheolsoo Park 2012-08-21, 08:25
+
lulynn_2008 2012-08-17, 05:59
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB