Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> how can I know which file source from on collector sink?


Copy link to this message
-
Re: how can I know which file source from on collector sink?
You may use flume-ng, then you can set information in header.

On Thu, Feb 7, 2013 at 4:28 PM, 周梦想 <[EMAIL PROTECTED]> wrote:

> Hi,
> I want gather different format and name log file from agent. I want to
> write each file to hdfs with different file name prefix or different dir so
> that I can recognize the files are from which.
>
> source like:
> config [MAgent-44, text("D:\\TKServer\\_BakLog\\20130207 655game.log"),
> batch(1000) agentDFOSink("192.168.10.48", 35853)]
> config [MAgent-44, text("D:\\TKServer\\_BakLog\\20130207 655user.log"),
> batch(1000) agentDFOSink("192.168.10.48", 35853)]
>
> config [MAgent-45, text("D:\\TKServer\\_BakLog\\20130207 655game.log"),
> batch(1000) agentDFOSink("192.168.10.48", 35853)]
> config [MAgent-45, text("D:\\TKServer\\_BakLog\\20130207 655user.log"),
> batch(1000) agentDFOSink("192.168.10.48", 35853)]
>
>
> collector like
> config [co1, collectorSource( 35853 ), gunzip unbatch collectorSink(
> "hdfs://hadoop48:54310/user/flume/%y%m/%d","%{host}-%{sourcefile}-")]
>
> note:%{sourcefile} isn't existed.
>
> results like:
> -rw-r--r--   2 zhouhh supergroup    7309058 2013-02-07 16:14
> /user/flume/1302/07/MAgent-44-game.log-20130207-161423054+0800.885170506522053.00035553
> -rw-r--r--   2 zhouhh supergroup   17922102 2013-02-07 16:14
> /user/flume/1302/07/MAgent-44-user.log-20130207-161453158+0800.885200610609053.00035551
> -rw-r--r--   2 zhouhh supergroup   17854942 2013-02-07 16:15
> /user/flume/1302/07/MAgent-45-game.log-20130207-161523249+0800.885230701184053.00035551
> -rw-r--r--   2 zhouhh supergroup   17827087 2013-02-07 16:15
> /user/flume/1302/07/MAgent-45-user.log-20130207-161553269+0800.885260721933053.00035551
> -rw-r--r--   2 zhouhh supergroup   17820650 2013-02-07 16:16
> /user/flume/1302/07/MAgent-44-game.log-20130207-161623290+0800.885290742023053.00035551
>
> how can I do this?
> can I using different ports of collector to map different source files?
>
> Thanks.
> Andy
>