Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> Re: Heads up: branch-2.1-beta


+
Roman Shaposhnik 2013-06-10, 16:54
+
Arpit Gupta 2013-06-12, 23:01
+
Arun C Murthy 2013-06-14, 20:17
+
Alejandro Abdelnur 2013-06-14, 21:02
+
Alejandro Abdelnur 2013-06-14, 23:34
+
Roman Shaposhnik 2013-06-15, 02:17
+
Alejandro Abdelnur 2013-06-15, 04:56
+
Alejandro Abdelnur 2013-06-15, 15:19
+
Ralph Castain 2013-06-15, 18:46
+
Karthik Kambatla 2013-06-15, 23:18
+
Ralph Castain 2013-06-16, 17:16
+
Roman Shaposhnik 2013-06-19, 23:29
+
Roman Shaposhnik 2013-06-19, 16:35
+
Steve Loughran 2013-06-19, 09:01
+
Roman Shaposhnik 2013-06-17, 23:04
Copy link to this message
-
Re: Heads up: branch-2.1-beta
Roman,

 Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?

thanks,
Arun

On Jun 16, 2013, at 4:50 PM, Arun C Murthy wrote:

> Responses inline:
>
> On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:
>
>> But there's a bit of bad news too (or at least the news that need to
>> be triaged). At
>> this point I don't know whether the Hadoop code is to blame or the
>> tests/components
>> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>>   1. HDFS append integration tests failed:
>>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>>       It seems that there's some incompatibility between the client code
>>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>>       and the current Hadoop 2.1.0.
>
> This is well known, we need to recompile against hadoop-2.1.0-beta.
>
>>
>>    2. Quite a few Sqoop tests ended up failing because of what seems
>> like AM not
>>    realizing that one of the tasks exited and waiting for it to
>> timeout. In the end
>>    the task is getting killed like this:
>>         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
>> after 600 secsContainer killed by the ApplicationMaster.
>>     but it takes a VERY long time (on the task side the log is
>> attached bellow).
>
> Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.
>
> We'll probably need another 2.1.1-beta anyway...
>
>>
>>     3. There's a couple of Hive tests (out of more than a dozen) that
>> failed in a pretty odd way
>>          (scroll to the very bottom of every page to see the excpetion):
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>>         What's peculiar here is that nothing has failed *before* or
>> *after* these particular
>>         tests. Hence I don't think that the state of the cluster
>> deployment is to blame.
>
> Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:
>
> Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(File /user/jenkins/.staging/job_1371348647940_0308/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.
>
>
>>
>>      4. All of the Mahout tests failed with the following:
>>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>>          This odd, and as I said -- all I know at this point is that
>> the very same
>>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
>
> Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in particular this was due to the compatibility work done via MAPREDUCE-5156 (part of MAPREDUCE-5108).
>
>
> Arun
>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
+
Roman Shaposhnik 2013-06-17, 23:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB