Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # user >> Translate this to english :)


Copy link to this message
-
Translate this to english :)
Hi flume  (cc'd bigtop also)....

I've been playing with the bigtop tests, and this snippet is particularly
difficult to
understand for a flume novice :

  private void compressionCommonTest(String id, String decompress, String
glob) {
    String node_config "node:text(\"events.txt\")|collectorSink(\"${hdfs_sink_dir}\",\"data\");";

    sh.exec("export FLUME_CONF_DIR=./${id}",
            "flume node_nowatch -s -1 -n node -c '${node_config}'");
    assertEquals("Flume failed to accept events",
                 0, sh.ret);

    sh.exec("hadoop fs -cat ${hdfs_sink_dir}/${glob} | ${decompress} | wc
-l");
    assertEquals("Wrong # of lines in output found at ${hdfs_sink_dir}",
                 "10000", sh.out[0]);
  }

Can someone explain the logic of it to me? In particular...

1) What is the difference between flume and flume ng - and does "flume_ng"
support these options (i.e. node_nowatch) ?

2) What is the "node_config |" syntax accomplishing?

The entire Flume bigtop smoke test can be found here:

https://github.com/apache/bigtop/tree/master/bigtop-tests/test-artifacts/flume/src/main/groovy/org/apache/bigtop/itest/flumesmoke

--
Jay Vyas
http://jayunit100.blogspot.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB