Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> how can I know which file source from on collector sink?


Copy link to this message
-
how can I know which file source from on collector sink?
Hi,
I want gather different format and name log file from agent. I want to
write each file to hdfs with different file name prefix or different dir so
that I can recognize the files are from which.

source like:
config [MAgent-44, text("D:\\TKServer\\_BakLog\\20130207 655game.log"),
batch(1000) agentDFOSink("192.168.10.48", 35853)]
config [MAgent-44, text("D:\\TKServer\\_BakLog\\20130207 655user.log"),
batch(1000) agentDFOSink("192.168.10.48", 35853)]

config [MAgent-45, text("D:\\TKServer\\_BakLog\\20130207 655game.log"),
batch(1000) agentDFOSink("192.168.10.48", 35853)]
config [MAgent-45, text("D:\\TKServer\\_BakLog\\20130207 655user.log"),
batch(1000) agentDFOSink("192.168.10.48", 35853)]
collector like
config [co1, collectorSource( 35853 ), gunzip unbatch collectorSink(
"hdfs://hadoop48:54310/user/flume/%y%m/%d","%{host}-%{sourcefile}-")]

note:%{sourcefile} isn't existed.

results like:
-rw-r--r--   2 zhouhh supergroup    7309058 2013-02-07 16:14
/user/flume/1302/07/MAgent-44-game.log-20130207-161423054+0800.885170506522053.00035553
-rw-r--r--   2 zhouhh supergroup   17922102 2013-02-07 16:14
/user/flume/1302/07/MAgent-44-user.log-20130207-161453158+0800.885200610609053.00035551
-rw-r--r--   2 zhouhh supergroup   17854942 2013-02-07 16:15
/user/flume/1302/07/MAgent-45-game.log-20130207-161523249+0800.885230701184053.00035551
-rw-r--r--   2 zhouhh supergroup   17827087 2013-02-07 16:15
/user/flume/1302/07/MAgent-45-user.log-20130207-161553269+0800.885260721933053.00035551
-rw-r--r--   2 zhouhh supergroup   17820650 2013-02-07 16:16
/user/flume/1302/07/MAgent-44-game.log-20130207-161623290+0800.885290742023053.00035551

how can I do this?
can I using different ports of collector to map different source files?

Thanks.
Andy
+
hoo.smth 2013-02-16, 09:16
+
周梦想 2013-02-18, 03:16
+
Hari Shreedharan 2013-02-18, 05:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB