Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop, mail # user - Translate this to english :)


Copy link to this message
-
Translate this to english :)
Jay Vyas 2013-08-28, 21:49
Hi flume  (cc'd bigtop also)....

I've been playing with the bigtop tests, and this snippet is particularly
difficult to
understand for a flume novice :

  private void compressionCommonTest(String id, String decompress, String
glob) {
    String node_config "node:text(\"events.txt\")|collectorSink(\"${hdfs_sink_dir}\",\"data\");";

    sh.exec("export FLUME_CONF_DIR=./${id}",
            "flume node_nowatch -s -1 -n node -c '${node_config}'");
    assertEquals("Flume failed to accept events",
                 0, sh.ret);

    sh.exec("hadoop fs -cat ${hdfs_sink_dir}/${glob} | ${decompress} | wc
-l");
    assertEquals("Wrong # of lines in output found at ${hdfs_sink_dir}",
                 "10000", sh.out[0]);
  }

Can someone explain the logic of it to me? In particular...

1) What is the difference between flume and flume ng - and does "flume_ng"
support these options (i.e. node_nowatch) ?

2) What is the "node_config |" syntax accomplishing?

The entire Flume bigtop smoke test can be found here:

https://github.com/apache/bigtop/tree/master/bigtop-tests/test-artifacts/flume/src/main/groovy/org/apache/bigtop/itest/flumesmoke

--
Jay Vyas
http://jayunit100.blogspot.com