Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Issues with flume while trying to transfer big size files [1GB every hour]


+
Chandrashekhar Shaw 2012-07-23, 11:52
+
Harish Mandala 2012-07-23, 14:30
+
Chandrashekhar Shaw 2012-07-24, 07:02
+
Harish Mandala 2012-07-24, 13:20
+
Chandrashekhar Shaw 2012-08-07, 07:16
+
Hari Shreedharan 2012-08-07, 07:41
+
Chandrashekhar Shaw 2012-08-08, 07:25
Copy link to this message
-
Re: Issues with flume while trying to transfer big size files [1GB every hour]
Not at the moment, but there is already effort to support encryption in file channel:

https://issues.apache.org/jira/browse/FLUME-1424

Jarcec

On Wed, Aug 08, 2012 at 12:55:35PM +0530, Chandrashekhar Shaw wrote:
> Hi Harish,
>
> Thanks for quick response..
>
> No, Actually I did not try flume-ng. Flume-og was working good for me..
>
> Just curious to know does flume-ng supports encrypted  channel ?
>
> Thanks
> Chandrashekhar
>
> On Aug 7, 2012, at 1:11 PM, Hari Shreedharan wrote:
>
> > Chandrashekhar,  
> >
> > Is there a specific reason you cannot use Flume NG. I believe that has already been suggested on this thread. Flume OG is no longer actively developed, and it is better to move to Flume NG - which had its second release a few weeks back - Flume v 1.2.0.
> >
> >
> > Thanks
> > Hari
> >
> > --
> > Hari Shreedharan
> >
> >
> > On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:
> >
> >> Hi
> >>
> >> I was testing flume for log transfer in DFO mode. I was firing the single shot command as cron every 10th minute from the agent box. So the flume process starts every 10 minutes sends the logs to collector box and ends.
> >>
> >> It was pushing the logs very well. To test the reliability I stopped the collector and brought it back after 1 hour, what I see is
> >>
> >> * The flume process (started by the agent cron) kept on running in the agent machine and not able to transfer the rest of the file to collector.
> >> * The process did not end even after 5/6 hours. (I had to manually kill it).
> >>
> >> As I know flume agent should continue sending the log once the collector is re-started. ???
> >> Is there something I am missing ??
> >>
> >> Thanks
> >> Chandrashekhar Shaw
> >>
> >>
> >>
> >>
> >> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
> >>
> >> Hello,
> >>
> >> Not too sure about Flume OG - I barely used it for a week before shifting to NG. Would there be any particular reason you are using OG?
> >>
> >> Yes, you can take a look at SyslogTcpSource (in NG), https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
> >>
> >> This can be modified according to this Netty example (for asynchronous large file streaming) : http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
> >>
> >> This is pretty much what I did when I needed a Flume HTTP source to match my requirements.
> >>
> >> Regards,
> >> Harish
> >>
> >>
> >> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> >> I am using Flume - OG https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz .
> >>
> >> It there an available version which implements netty??
> >>
> >> Thanks
> >> Chandrashekhar Shaw
> >>
> >>
> >> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
> >>
> >> Hi,
> >>
> >> Which one of the sources are you using? Maybe a flume source that
> >> implements something like
> >> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
> >>
> >> Regards,
> >> Harish
> >>
> >>
> >> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
> >> <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> >>
> >> Hi,
> >>
> >> I am using flume for transferring logs from 20 agent boxes to 1 collector
> >> box. The file size which I am trying to transfer is of about 1 GB per hour.
> >> So every hour my agents run to transfer the log file generated during the
> >> last hour.
> >> I am using DFO sink for file transfer.
> >>
> >> But then I am facing following problems:
> >>
> >> 1) Some of the agents are not able to transfer the whole file even in one
> >> hour and the processes run continuously for hours.
> >> 2) Some of the agents stop within 5-6 minutes without sending the file
> >> completely.
> >>
> >> I want to know
> >>
> >> 1) if somebody else has faced the same issues?
> >> 2) What could be the problem?
+
Chandrashekhar Shaw 2012-08-08, 11:05
+
Mike Percy 2012-08-08, 18:09
+
Chandrashekhar Shaw 2012-08-09, 11:27
+
Chandrashekhar Shaw 2012-07-24, 07:07