Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # user >> Translate this to english :)

Copy link to this message
Translate this to english :)
Hi flume  (cc'd bigtop also)....

I've been playing with the bigtop tests, and this snippet is particularly
difficult to
understand for a flume novice :

  private void compressionCommonTest(String id, String decompress, String
glob) {
    String node_config "node:text(\"events.txt\")|collectorSink(\"${hdfs_sink_dir}\",\"data\");";

    sh.exec("export FLUME_CONF_DIR=./${id}",
            "flume node_nowatch -s -1 -n node -c '${node_config}'");
    assertEquals("Flume failed to accept events",
                 0, sh.ret);

    sh.exec("hadoop fs -cat ${hdfs_sink_dir}/${glob} | ${decompress} | wc
    assertEquals("Wrong # of lines in output found at ${hdfs_sink_dir}",
                 "10000", sh.out[0]);

Can someone explain the logic of it to me? In particular...

1) What is the difference between flume and flume ng - and does "flume_ng"
support these options (i.e. node_nowatch) ?

2) What is the "node_config |" syntax accomplishing?

The entire Flume bigtop smoke test can be found here:


Jay Vyas