Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # dev - Re: Heads up: branch-2.1-beta


+
Roman Shaposhnik 2013-06-10, 16:54
+
Arpit Gupta 2013-06-12, 23:01
+
Arun C Murthy 2013-06-14, 20:17
+
Alejandro Abdelnur 2013-06-14, 21:02
+
Alejandro Abdelnur 2013-06-14, 23:34
+
Roman Shaposhnik 2013-06-15, 02:17
+
Alejandro Abdelnur 2013-06-15, 04:56
+
Alejandro Abdelnur 2013-06-15, 15:19
+
Ralph Castain 2013-06-15, 18:46
+
Karthik Kambatla 2013-06-15, 23:18
+
Ralph Castain 2013-06-16, 17:16
+
Roman Shaposhnik 2013-06-19, 23:29
+
Roman Shaposhnik 2013-06-19, 16:35
+
Steve Loughran 2013-06-19, 09:01
Copy link to this message
-
Re: Heads up: branch-2.1-beta
Roman Shaposhnik 2013-06-17, 23:04
Hi Arun,

thanks for chiming in!

On Sun, Jun 16, 2013 at 4:50 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
>> But there's a bit of bad news too (or at least the news that need to
>> be triaged). At
>> this point I don't know whether the Hadoop code is to blame or the
>> tests/components
>> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>>   1. HDFS append integration tests failed:
>>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>>       It seems that there's some incompatibility between the client code
>>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>>       and the current Hadoop 2.1.0.
>
> This is well known, we need to recompile against hadoop-2.1.0-beta.

That's what I suspected. Thanks for confirming. Like I said -- given
the tight schedule I didn't have the time to recompile the entire
Bigtop stack against Hadoop 2.1.0-beta (including tests).

> Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.
>
> We'll probably need another 2.1.1-beta anyway...

Agreed. There seems to be some weird interaction between
Sqoop and RM, though. But it definitely doesn't qualify as
a blocker from what I see in the logs and my debugger
and whether it ends up being a Sqoop issue or Hadoop
issue we can tackle it a tad later.

> Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:

Bigtop Hive tests don't use MiniHDFSCluster -- they are executed against
an honest fully distributed cluster. Like I said -- the weird part here is
that the failing two are in the middle of a whole bunch that worked
just fine. And I haven't really seen them fail before like that ever.
I wish I had more time to investigate.

>>      4. All of the Mahout tests failed with the following:
>>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>>          This odd, and as I said -- all I know at this point is that
>> the very same
>>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
>
> Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in
> particular this was due to the compatibility work done via MAPREDUCE-5156 (part of
> MAPREDUCE-5108).

Ah! Didn't know that.

Well, it sounds like we've got enough mileage out of of the quick
application of Bigtop stack/tests from Bigtop 0.6.0. At this point
I feel like the first level of smoke testing passed and if we want
to drill down we'd have to spend time and rebuild everything
against Hadoop 2.1.0. That'll take some time and should happen
sometimes this week.

Hope the amount of validation I was capable of was useful so far.

Thanks,
Roman.
+
Arun C Murthy 2013-06-17, 00:14
+
Roman Shaposhnik 2013-06-17, 23:06