Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - send the whole logs


Copy link to this message
-
Re: send the whole logs
Hari Shreedharan 2013-10-09, 07:29
text does not write the headers, try HEADER_AND_TEXT  
Thanks,
Hari
On Tuesday, October 8, 2013 at 11:26 PM, Martinus m wrote:

> Hi Hari,
>
> I tried to add below serializers in my flume.conf :
>
> agent.sinks.s3Sink.serializer = text
>
> And it's still doesn't have timestamp (date) info from the original log message :
>
> Thanks,
>
> Martinus
>
>
>
> On Wed, Oct 9, 2013 at 1:45 PM, Hari Shreedharan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > The timestamp is in the event header. You would need to use a serializer which also writes out the headers.
> >
> >
> > Thanks,
> > Hari
> >
> >
> > On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:
> >
> > > Hi David,
> > >
> > > I'm using Syslog UDP source and on the syslog messages it have it's timestamp for each message. I also use HDFS sink, but when I saw the result message on HDFS folder, it doesn't have the timestamp (date) info.
> > >
> > > Thanks,
> > >
> > > Martinus
> > >
> > >
> > > On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > > Martinus,
> > > >
> > > > Can you give a little more details? It sounds like you want to use the Spooling Directory Source, http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source, but if you can be clearer about your requirements, I may be able to help you better.
> > > >
> > > > dave
> > > >
> > > >
> > > > On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > > > Hi,
> > > > >
> > > > > What configurations should I put on flume.conf to get the whole logs message to be put into the sink?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Martinus
> > >
> >
>