Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Not able to set date %d%m%Y in Sink Directory name


+
Deepak Subhramanian 2013-10-23, 15:25
+
David Sinclair 2013-10-23, 17:40
Copy link to this message
-
Re: Not able to set date %d%m%Y in Sink Directory name
I am using a SimpleHTTPEventHandler which implements HTTPSourceHandler
which reads a log message from HTTP Request and create a Flume event. May
be I have to add the header to the event with the timestamp.

String message =  (String)request.getParameter("log");

     //LOG.info(message );

 LOG.info("Msg Read" );
  Event event = EventBuilder.withBody(message.getBytes());
eventList.add(event);
On Wed, Oct 23, 2013 at 6:40 PM, David Sinclair <
[EMAIL PROTECTED]> wrote:

> The sink is dependent on a header with the key "timestamp" being present
> in the event for this to work. What source are you using?
>
>
> On Wed, Oct 23, 2013 at 11:25 AM, Deepak Subhramanian <
> [EMAIL PROTECTED]> wrote:
>
>> Hi ,
>>
>> I am trying to store my logs in folders named with date for my file_roll
>> and hdfssink. For some reason when I pass %d%m%Y in the sink directory it
>> is not working . Any thoughts .
>>
>> My Flume source is a simple HTTP Handler extended from HTTPSourceHandler
>>
>> tier1.sinks.filesink1.type = file_roll
>> tier1.sinks.filesink1.channel = c2
>> tier1.sinks.filesink1.batchSize = 100000
>> tier1.sinks.filesink1.rollInterval = 0
>> #tier1.sinks.filesink1.rollInterval = 86400
>> tier1.sinks.filesink1.sink.directory = /projects/test/logs/%d%m%Y
>>
>>
>> 13/10/23 16:20:36 ERROR flume.SinkRunner: Unable to deliver event.
>> Exception follows.
>> org.apache.flume.EventDeliveryException: Failed to open file
>> /projects/test/logs/%d%m%Y/1382541251307-1 while delivering event
>>  at
>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>  at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Unknown Source)
>> Caused by: java.io.FileNotFoundException:
>> /projects/ETHAN_FLUME/logs/%d%m%Y/1382541251307-1 (No such file or
>> directory)
>> at java.io.FileOutputStream.open(Native Method)
>>  at java.io.FileOutputStream.<init>(Unknown Source)
>> at java.io.FileOutputStream.<init>(Unknown Source)
>>  at
>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
>>
>>
>> Thanks,
>> Deepak Subhramanian
>>
>
>
--
Deepak Subhramanian
+
David Sinclair 2013-10-24, 13:47
+
Deepak Subhramanian 2013-10-24, 14:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB