Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> [pig-0.10.0 failed to load hbase table]Re:Re: Re: Re: pig-0.10.0: TestJobSubmission failed with: Internal error creating job configuration.

Copy link to this message
[pig-0.10.0 failed to load hbase table]Re:Re: Re: Re: pig-0.10.0: TestJobSubmission failed with: Internal error creating job configuration.
Hi Cheolsoo,
Current TestJobSubmission and TestHBaseStorage passed. But if I run following scripts with hbase-0.94/zookeeper-3.4.3 cluster:

the same issue happened.

2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /
2012-09-12 00:30:07,199 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is [EMAIL PROTECTED]
2012-09-12 00:30:07,212 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-09-12 00:30:07,213 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-09-12 00:30:07,221 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)

Reproduce steps:

Create hbase table:
./hbase shell:
create 'employees', 'SN', 'department', 'address'
put 'employees', 'Hong', 'address:country', 'China'

Run following pig commands:
A = load 'hbase://employees' using org.apache.pig.backend.hadoop.hbase.HBaseStorage( 'address:country', '-loadKey true') as (SN:bytearray,country:bytearray);
B = filter A by SN == 'Hong';
dump B;

At 2012-08-22 05:51:38,"Cheolsoo Park" <[EMAIL PROTECTED]> wrote:
>OK, I got TestJobSubmission passing. Please apply the following diff to
>your Pig:
>diff --git test/org/apache/pig/test/TestJobSubmission.java
>index 89ac132..211c4ae 100644
>--- test/org/apache/pig/test/TestJobSubmission.java
>+++ test/org/apache/pig/test/TestJobSubmission.java
>@@ -26,8 +26,10 @@ import java.util.Random;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.HBaseTestingUtility;
>*+import org.apache.hadoop.hbase.HConstants;*
> import org.apache.hadoop.hbase.MiniHBaseCluster;
> import org.apache.hadoop.hbase.util.Bytes;
>*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;*
> import org.apache.hadoop.mapred.jobcontrol.Job;
> import org.apache.hadoop.mapred.jobcontrol.JobControl;
> import org.apache.pig.ExecType;
>@@ -541,7 +543,7 @@ public class TestJobSubmission {
>         // use the estimation
>         Configuration conf = cluster.getConfiguration();
>         HBaseTestingUtility util = new HBaseTestingUtility(conf);
>*-        util.startMiniZKCluster();*
>*+        int clientPort = util.startMiniZKCluster().getClientPort();*
>         util.startMiniHBaseCluster(1, 1);
>         String query = "a = load '/passwd';" +
>@@ -553,6 +555,7 @@ public class TestJobSubmission {
>         pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
>         pc.getConf().setProperty("pig.exec.reducers.max", "10");
>*+        pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
>         ConfigurationValidator.validatePigProperties(pc.getProperties());
>         conf = ConfigurationUtil.toConfiguration(pc.getProperties());
>         JobControlCompiler jcc = new JobControlCompiler(pc, conf);
>There are several changes to MiniZooKeeperCluster in HBase 0.94 compared to
>0.90. In particular, the ZK client port is no longer 21818 by default. In
>fact, it is randomized in 0.94. So the client port has to be explicitly set
>in PigContext at runtime.
>Now I am able to run both TestJobSubmission and TestHBaseStorage. Please