Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> 1.4 unit test failing in Bamboo


Copy link to this message
-
1.4 unit test failing in Bamboo
The HDFS tests fail unless your umask is (from memory) 0022. Otherwise it
can't delete files it creates or something.

The exec source thing appears to be a race condition. I keep running the
build until i get success.

Yes, I agree, frustrating on Bamboo.

Mike said,
"I've done a small proof of concept on some flume sink customizations in
1.3.1.  I want to incorporate these customizations into our build server
(Bamboo).  I started by creating a local git repo of 1.4 and I'm able to
run a mvn clean install just fine.  However when I push the raw un-altered
code to our build server the build fails and not consistently.

Half the time TestExecSource fails in flume-ng-core:

build    01-Aug-2013 22:56:11    testMonitoredCounterGroup(org.
apache.flume.source.TestExecSource)  Time elapsed: 1264 sec  <<< FAILURE!
build    01-Aug-2013 22:56:11    org.junit.ComparisonFailure: Expected
Value: 5 expected:<[5]> but was:<[0]>
build    01-Aug-2013 22:56:11        at org.junit.Assert.assertEquals(
Assert.java:125)
build    01-Aug-2013 22:56:11        at org.apache.flume.source.
TestExecSource.testMonitoredCounterGroup(TestExecSource.java:215)

When that passes my build fails in flume-hdfs-sink
build    02-Aug-2013 00:03:30    simpleHDFSGZipCompressedTest(
org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster)  Time elapsed:
2676 sec  <<< ERROR!
build    02-Aug-2013 00:03:30    java.lang.NullPointerException
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.<init>(MiniDFSCluster.java:280)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.<init>(MiniDFSCluster.java:124)
build    02-Aug-2013 00:03:30        at org.apache.flume.sink.hdfs.
TestHDFSEventSinkOnMiniCluster.simpleHDFSGZipCompressedTest(
TestHDFSEventSinkOnMiniCluster.java:164)
...
build    02-Aug-2013 00:03:30    underReplicationTest(org.
apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster)  Time elapsed: 31
sec  <<< ERROR!
build    02-Aug-2013 00:03:30    java.io.IOException: Cannot lock storage
target/test/dfs/dfs/name1. The directory is already locked.
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
common.Storage$StorageDirectory.lock(Storage.java:602)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
namenode.FSImage.format(FSImage.java:1219)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
namenode.FSImage.format(FSImage.java:1237)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
namenode.NameNode.format(NameNode.java:1164)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
namenode.NameNode.format(NameNode.java:184)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.<init>(MiniDFSCluster.java:267)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.<init>(MiniDFSCluster.java:124)
I'm at a loss to explain the inconsistency of flume-ng-core and when it
does succeed how to solve the failure of HDFS-sink.  Any advice would be
appreciated.
Thanks,

Mike"
+
Mike Keane 2013-08-02, 00:14