Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> HDFS custom escape sequence


+
Camp, Roy 2013-06-10, 21:51
Copy link to this message
-
RE: HDFS custom escape sequence
Yes, any header works using that same notation. I use it to allow us to bucket many event 'types' with a single channel.

for example, a path with custom headers and Hive ready partition folders:
/flumelogs/%{someHeader}/%{someOtherHeader}/datekey=%Y%m%d/hour=%H

Hope that helps,
Paul Chavez

________________________________
From: Camp, Roy [mailto:[EMAIL PROTECTED]]
Sent: Monday, June 10, 2013 2:51 PM
To: [EMAIL PROTECTED]
Subject: HDFS custom escape sequence

Is it possible to use other headers besides the host header, %{host} escape sequence, to name the HDFS files?  We are interested in attaching a category header and want the events stored separately.

If not possible, is there a best practice for achieving this?

Thanks,

Roy
+
Hari Shreedharan 2013-06-10, 22:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB