Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Help with HBaseTestingUtility / MiniDFSCluster


Copy link to this message
-
Re: Help with HBaseTestingUtility / MiniDFSCluster
Ted Yu 2013-07-14, 17:47
Here is one way of figuring out whether hdfs experiences the same issue is
the following:

check out http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1
use the command to find out which tests create MiniDFSCluster: find . -name
'*.java' -exec grep 'new MiniDFSCluster(' {} \; -print
run one of the tests: ant test-core -Dtestcase=TestCopyCommitter

See if there is similar exception.

On my Mac, I got:

    [junit] Running org.apache.hadoop.tools.distcp2.mapred.TestCopyCommitter
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 12.283 sec

On Sun, Jul 14, 2013 at 9:58 AM, David Williams
<[EMAIL PROTECTED]>wrote:

> Hi Ted,
>
> I'd be interested to know more about an hdfs issue.  I was go to look
> further into the line of reasoning you mentioned about the call
> to getAddress.
>
> MiniDFSCluster.java, line 426:
> String ipAddr = dn.getSelfAddr().getAddress().getHostAddress();
>
> So that one of getSelfAddr or getAddress returned NULL.   Is there
> something new about hdfs or are the two related?
>
>
>
>
>
> On Jul 13, 2013, at 9:41 PM, Ted Yu wrote:
>
> Most likely this is an hdfs issue.
>
> On Sat, Jul 13, 2013 at 4:08 PM, David Williams <[EMAIL PROTECTED]
> > wrote:
>
>> Hi Ted,
>>
>> I updated the dependencies and ran the tests again, and on my Mac OSX
>> they pass and on CentOS I get the same error:
>>
>>
>> $ lein test
>> Retrieving org/apache/hbase/hbase/0.94.9/hbase-0.94.9.pom from central
>> Retrieving org/apache/hbase/hbase/0.94.9/hbase-0.94.9.jar from central
>> Retrieving org/apache/hbase/hbase/0.94.9/hbase-0.94.9-tests.jar from
>> central
>>
>> lein test hbase.config-test
>>
>> lein test hbase.table-test
>> Starting DataNode 0 with dfs.data.dir:
>> /home/dwilliams/Desktop/Repos/hbase/target/test-data/1140edc6-7242-40cd-8ed8-05847fb14949/dfscluster_1e40ce89-1986-450b-ba6d-983caa9aeb78/dfs/data/data1,/home/dwilliams/Desktop/Repos/hbase/target/test-data/1140edc6-7242-40cd-8ed8-05847fb14949/dfscluster_1e40ce89-1986-450b-ba6d-983caa9aeb78/dfs/data/data2
>>
>> lein test :only hbase.table-test/create-table
>>
>>
>> ERROR in (create-table) (MiniDFSCluster.java:426)
>> Uncaught exception, not in assertion.
>> expected: nil
>>   actual: java.lang.NullPointerException: null
>>  at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes
>> (MiniDFSCluster.java:426)
>>     org.apache.hadoop.hdfs.MiniDFSCluster.<init> (MiniDFSCluster.java:284)
>>     org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster
>> (HBaseTestingUtility.java:451)
>>     org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
>> (HBaseTestingUtility.java:619)
>>     org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
>> (HBaseTestingUtility.java:575)
>>     org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster
>> (HBaseTestingUtility.java:562)
>>     hbase.table_test$test_config.doInvoke (table_test.clj:10)
>>     clojure.lang.RestFn.invoke (RestFn.java:397)
>>     hbase.table_test/fn (table_test.clj:19)
>>
>> On Jul 13, 2013, at 1:11 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>> Do you mind trying the following change to see if the problem persists
>> for latest 0.94 release ?
>>
>> Thanks
>>
>> diff --git a/project.clj b/project.clj
>> index 2554784..0d8be0e 100644
>> --- a/project.clj
>> +++ b/project.clj
>> @@ -5,8 +5,8 @@
>>         :dependencies [
>>                 [org.clojure/clojure "1.5.1"]
>>                 [org.apache.hadoop/hadoop-core "1.2.0"]
>> -               [org.apache.hbase/hbase "0.94.6.1"]
>> +               [org.apache.hbase/hbase "0.94.9"]
>>                 [org.apache.hadoop/hadoop-test "1.2.0"]
>> -               [org.apache.hbase/hbase "0.94.6.1" :classifier "tests"]]
>> +               [org.apache.hbase/hbase "0.94.9" :classifier "tests"]]
>>         :plugins [[lein-marginalia "0.7.1"]])
>>
>>
>> On Fri, Jul 12, 2013 at 10:22 PM, David Williams <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi Ted,
>>>
>>> In terms of versions, here are the jars I'm using, which come from Maven