Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - 1.4 unit test failing in Bamboo


Copy link to this message
-
1.4 unit test failing in Bamboo
Edward Sargisson 2013-08-03, 14:19
The HDFS tests fail unless your umask is (from memory) 0022. Otherwise it
can't delete files it creates or something.

The exec source thing appears to be a race condition. I keep running the
build until i get success.

Yes, I agree, frustrating on Bamboo.

Mike said,
"I've done a small proof of concept on some flume sink customizations in
1.3.1.  I want to incorporate these customizations into our build server
(Bamboo).  I started by creating a local git repo of 1.4 and I'm able to
run a mvn clean install just fine.  However when I push the raw un-altered
code to our build server the build fails and not consistently.

Half the time TestExecSource fails in flume-ng-core:

build    01-Aug-2013 22:56:11    testMonitoredCounterGroup(org.
apache.flume.source.TestExecSource)  Time elapsed: 1264 sec  <<< FAILURE!
build    01-Aug-2013 22:56:11    org.junit.ComparisonFailure: Expected
Value: 5 expected:<[5]> but was:<[0]>
build    01-Aug-2013 22:56:11        at org.junit.Assert.assertEquals(
Assert.java:125)
build    01-Aug-2013 22:56:11        at org.apache.flume.source.
TestExecSource.testMonitoredCounterGroup(TestExecSource.java:215)

When that passes my build fails in flume-hdfs-sink
build    02-Aug-2013 00:03:30    simpleHDFSGZipCompressedTest(
org.apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster)  Time elapsed:
2676 sec  <<< ERROR!
build    02-Aug-2013 00:03:30    java.lang.NullPointerException
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.<init>(MiniDFSCluster.java:280)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.<init>(MiniDFSCluster.java:124)
build    02-Aug-2013 00:03:30        at org.apache.flume.sink.hdfs.
TestHDFSEventSinkOnMiniCluster.simpleHDFSGZipCompressedTest(
TestHDFSEventSinkOnMiniCluster.java:164)
...
build    02-Aug-2013 00:03:30    underReplicationTest(org.
apache.flume.sink.hdfs.TestHDFSEventSinkOnMiniCluster)  Time elapsed: 31
sec  <<< ERROR!
build    02-Aug-2013 00:03:30    java.io.IOException: Cannot lock storage
target/test/dfs/dfs/name1. The directory is already locked.
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
common.Storage$StorageDirectory.lock(Storage.java:602)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
namenode.FSImage.format(FSImage.java:1219)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
namenode.FSImage.format(FSImage.java:1237)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
namenode.NameNode.format(NameNode.java:1164)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.server.
namenode.NameNode.format(NameNode.java:184)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.<init>(MiniDFSCluster.java:267)
build    02-Aug-2013 00:03:30        at org.apache.hadoop.hdfs.
MiniDFSCluster.<init>(MiniDFSCluster.java:124)
I'm at a loss to explain the inconsistency of flume-ng-core and when it
does succeed how to solve the failure of HDFS-sink.  Any advice would be
appreciated.
Thanks,

Mike"