Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Issues with flume while trying to transfer big size files [1GB every hour]


Copy link to this message
-
Re: Issues with flume while trying to transfer big size files [1GB every hour]
Hi Chandrashekhar,
When you say encrypted channel, can you be more specific about what you are
referring to? i.e. encrypted on-disk or encrypted on the wire?

Regards,
Mike

On Wed, Aug 8, 2012 at 4:05 AM, Chandrashekhar Shaw <[EMAIL PROTECTED]>wrote:

> Or I would rather say "Does flume-ng provide encrypted channel ??"
>
> Thanks
> Chandrashekhar
> On Aug 8, 2012, at 12:55 PM, Chandrashekhar Shaw wrote:
>
> > Hi Harish,
> >
> > Thanks for quick response..
> >
> > No, Actually I did not try flume-ng. Flume-og was working good for me..
> >
> > Just curious to know does flume-ng supports encrypted  channel ?
> >
> > Thanks
> > Chandrashekhar
> >
> > On Aug 7, 2012, at 1:11 PM, Hari Shreedharan wrote:
> >
> >> Chandrashekhar,
> >>
> >> Is there a specific reason you cannot use Flume NG. I believe that has
> already been suggested on this thread. Flume OG is no longer actively
> developed, and it is better to move to Flume NG - which had its second
> release a few weeks back - Flume v 1.2.0.
> >>
> >>
> >> Thanks
> >> Hari
> >>
> >> --
> >> Hari Shreedharan
> >>
> >>
> >> On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:
> >>
> >>> Hi
> >>>
> >>> I was testing flume for log transfer in DFO mode. I was firing the
> single shot command as cron every 10th minute from the agent box. So the
> flume process starts every 10 minutes sends the logs to collector box and
> ends.
> >>>
> >>> It was pushing the logs very well. To test the reliability I stopped
> the collector and brought it back after 1 hour, what I see is
> >>>
> >>> * The flume process (started by the agent cron) kept on running in the
> agent machine and not able to transfer the rest of the file to collector.
> >>> * The process did not end even after 5/6 hours. (I had to manually
> kill it).
> >>>
> >>> As I know flume agent should continue sending the log once the
> collector is re-started. ???
> >>> Is there something I am missing ??
> >>>
> >>> Thanks
> >>> Chandrashekhar Shaw
> >>>
> >>>
> >>>
> >>>
> >>> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
> >>>
> >>> Hello,
> >>>
> >>> Not too sure about Flume OG - I barely used it for a week before
> shifting to NG. Would there be any particular reason you are using OG?
> >>>
> >>> Yes, you can take a look at SyslogTcpSource (in NG),
> https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
> >>>
> >>> This can be modified according to this Netty example (for asynchronous
> large file streaming) :
> http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
> >>>
> >>> This is pretty much what I did when I needed a Flume HTTP source to
> match my requirements.
> >>>
> >>> Regards,
> >>> Harish
> >>>
> >>>
> >>> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> >>> I am using Flume - OG
> https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz.
> >>>
> >>> It there an available version which implements netty??
> >>>
> >>> Thanks
> >>> Chandrashekhar Shaw
> >>>
> >>>
> >>> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
> >>>
> >>> Hi,
> >>>
> >>> Which one of the sources are you using? Maybe a flume source that
> >>> implements something like
> >>> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
> >>>
> >>> Regards,
> >>> Harish
> >>>
> >>>
> >>> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
> >>> <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am using flume for transferring logs from 20 agent boxes to 1
> collector
> >>> box. The file size which I am trying to transfer is of about 1 GB per
> hour.
> >>> So every hour my agents run to transfer the log file generated during
> the
> >>> last hour.
> >>> I am using DFO sink for file transfer.
> >>>
> >>> But then I am facing following problems:
> >>>
> >>> 1) Some of the agents are not able to transfer the whole file even in