Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Issues with flume while trying to transfer big size files [1GB every hour]


Copy link to this message
-
Re: Issues with flume while trying to transfer big size files [1GB every hour]
Hi Chandrashekhar,
When you say encrypted channel, can you be more specific about what you are
referring to? i.e. encrypted on-disk or encrypted on the wire?

Regards,
Mike

On Wed, Aug 8, 2012 at 4:05 AM, Chandrashekhar Shaw <[EMAIL PROTECTED]>wrote:

> Or I would rather say "Does flume-ng provide encrypted channel ??"
>
> Thanks
> Chandrashekhar
> On Aug 8, 2012, at 12:55 PM, Chandrashekhar Shaw wrote:
>
> > Hi Harish,
> >
> > Thanks for quick response..
> >
> > No, Actually I did not try flume-ng. Flume-og was working good for me..
> >
> > Just curious to know does flume-ng supports encrypted  channel ?
> >
> > Thanks
> > Chandrashekhar
> >
> > On Aug 7, 2012, at 1:11 PM, Hari Shreedharan wrote:
> >
> >> Chandrashekhar,
> >>
> >> Is there a specific reason you cannot use Flume NG. I believe that has
> already been suggested on this thread. Flume OG is no longer actively
> developed, and it is better to move to Flume NG - which had its second
> release a few weeks back - Flume v 1.2.0.
> >>
> >>
> >> Thanks
> >> Hari
> >>
> >> --
> >> Hari Shreedharan
> >>
> >>
> >> On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:
> >>
> >>> Hi
> >>>
> >>> I was testing flume for log transfer in DFO mode. I was firing the
> single shot command as cron every 10th minute from the agent box. So the
> flume process starts every 10 minutes sends the logs to collector box and
> ends.
> >>>
> >>> It was pushing the logs very well. To test the reliability I stopped
> the collector and brought it back after 1 hour, what I see is
> >>>
> >>> * The flume process (started by the agent cron) kept on running in the
> agent machine and not able to transfer the rest of the file to collector.
> >>> * The process did not end even after 5/6 hours. (I had to manually
> kill it).
> >>>
> >>> As I know flume agent should continue sending the log once the
> collector is re-started. ???
> >>> Is there something I am missing ??
> >>>
> >>> Thanks
> >>> Chandrashekhar Shaw
> >>>
> >>>
> >>>
> >>>
> >>> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
> >>>
> >>> Hello,
> >>>
> >>> Not too sure about Flume OG - I barely used it for a week before
> shifting to NG. Would there be any particular reason you are using OG?
> >>>
> >>> Yes, you can take a look at SyslogTcpSource (in NG),
> https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
> >>>
> >>> This can be modified according to this Netty example (for asynchronous
> large file streaming) :
> http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
> >>>
> >>> This is pretty much what I did when I needed a Flume HTTP source to
> match my requirements.
> >>>
> >>> Regards,
> >>> Harish
> >>>
> >>>
> >>> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> >>> I am using Flume - OG
> https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz.
> >>>
> >>> It there an available version which implements netty??
> >>>
> >>> Thanks
> >>> Chandrashekhar Shaw
> >>>
> >>>
> >>> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
> >>>
> >>> Hi,
> >>>
> >>> Which one of the sources are you using? Maybe a flume source that
> >>> implements something like
> >>> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
> >>>
> >>> Regards,
> >>> Harish
> >>>
> >>>
> >>> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
> >>> <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am using flume for transferring logs from 20 agent boxes to 1
> collector
> >>> box. The file size which I am trying to transfer is of about 1 GB per
> hour.
> >>> So every hour my agents run to transfer the log file generated during
> the
> >>> last hour.
> >>> I am using DFO sink for file transfer.
> >>>
> >>> But then I am facing following problems:
> >>>
> >>> 1) Some of the agents are not able to transfer the whole file even in
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB