Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Not able to set date %d%m%Y in Sink Directory name


Copy link to this message
-
Re: Not able to set date %d%m%Y in Sink Directory name
David Sinclair 2013-10-24, 13:47
You could do that or add the TimestampInterceptor to your source as well.
On Thu, Oct 24, 2013 at 5:16 AM, Deepak Subhramanian <
[EMAIL PROTECTED]> wrote:

> I am using a SimpleHTTPEventHandler which implements HTTPSourceHandler
> which reads a log message from HTTP Request and create a Flume event. May
> be I have to add the header to the event with the timestamp.
>
> String message =  (String)request.getParameter("log");
>
>      //LOG.info(message );
>
>  LOG.info("Msg Read" );
>
>
>   Event event = EventBuilder.withBody(message.getBytes());
>
>
> eventList.add(event);
>
>
> On Wed, Oct 23, 2013 at 6:40 PM, David Sinclair <
> [EMAIL PROTECTED]> wrote:
>
>> The sink is dependent on a header with the key "timestamp" being present
>> in the event for this to work. What source are you using?
>>
>>
>> On Wed, Oct 23, 2013 at 11:25 AM, Deepak Subhramanian <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi ,
>>>
>>> I am trying to store my logs in folders named with date for my file_roll
>>> and hdfssink. For some reason when I pass %d%m%Y in the sink directory it
>>> is not working . Any thoughts .
>>>
>>> My Flume source is a simple HTTP Handler extended from HTTPSourceHandler
>>>
>>> tier1.sinks.filesink1.type = file_roll
>>> tier1.sinks.filesink1.channel = c2
>>> tier1.sinks.filesink1.batchSize = 100000
>>> tier1.sinks.filesink1.rollInterval = 0
>>> #tier1.sinks.filesink1.rollInterval = 86400
>>> tier1.sinks.filesink1.sink.directory = /projects/test/logs/%d%m%Y
>>>
>>>
>>> 13/10/23 16:20:36 ERROR flume.SinkRunner: Unable to deliver event.
>>> Exception follows.
>>> org.apache.flume.EventDeliveryException: Failed to open file
>>> /projects/test/logs/%d%m%Y/1382541251307-1 while delivering event
>>>  at
>>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
>>> at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>  at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>> at java.lang.Thread.run(Unknown Source)
>>> Caused by: java.io.FileNotFoundException:
>>> /projects/ETHAN_FLUME/logs/%d%m%Y/1382541251307-1 (No such file or
>>> directory)
>>> at java.io.FileOutputStream.open(Native Method)
>>>  at java.io.FileOutputStream.<init>(Unknown Source)
>>> at java.io.FileOutputStream.<init>(Unknown Source)
>>>  at
>>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
>>>
>>>
>>> Thanks,
>>> Deepak Subhramanian
>>>
>>
>>
>
>
> --
> Deepak Subhramanian
>