| clear query|facets|time |
Search criteria: .
Results from 1 to 10 from
13 (0.307s).
|
|
|
Loading phrases to help you
refine your search...
|
|
Re: Flume Ng replaying events when the source is idle - Flume - [mail # user]
|
|
...Hi Guys, So I disabled puppet on one of the boxes and yes there have been no replays. So essentially to summarize - - Puppet was re-writing the flume config file [e...
|
|
|
Author: Sagar Mehta,
2013-03-05, 17:53
|
|
|
Re: Flume Ng replaying events when the source is idle - Flume - [mail # user]
|
|
...I think we found the issue, not sure if this is the root cause but looks highly correlated. So we manage configs using puppet which currently runs in a cron mode with following configu...
|
|
|
Author: Sagar Mehta,
2013-03-04, 23:06
|
|
|
Re: Flume Ng replaying events when the source is idle - Flume - [mail # user]
|
|
...Guys, Yes this issue was also seen in the memory channel. In fact when we moved to File based channel, we initially thought this issue won't occur since it stores check points. &...
|
|
|
Author: Sagar Mehta,
2013-03-04, 22:42
|
|
|
Flume Ng replaying events when the source is idle - Flume - [mail # user]
|
|
...Hi Guys, I'm using Flume-Ng and it is working pretty well except for a weird situation which I observed lately. In essence I'm using an exec source for doing tail -F on a logfile...
|
|
|
Author: Sagar Mehta,
2013-02-27, 19:37
|
|
|
Re: Question about gzip compression when using Flume Ng - Flume - [mail # user]
|
|
...Bhaskar, Your suggestion worked like magic!! I don't believe my eyes!! hadoop@jobtracker301:/home/hadoop/sagar/debug$ hget /ngpipes-raw-logs/2013-01-15/0200/collector102.ngpipes....
|
|
|
Author: Sagar Mehta,
2013-01-15, 02:36
|
|
|
Re: Question about gzip compression when using Flume Ng - Flume - [mail # user]
|
|
...As for S3, in fact that is our current architecture :) [ EMR computations were 2 years back - now we do locally in our clusters] and we want to move away from it since our Hadoop computation...
|
|
|
Author: Sagar Mehta,
2013-01-15, 01:52
|
|
|
Re: Question about gzip compression when using Flume Ng - Flume - [mail # user]
|
|
...Hmm - good point!! Even in the best case say this works, moving to a newer Hadoop version for the entire 2 production clusters that depend on it [400+ nodes] will need some thorough testing ...
|
|
|
Author: Sagar Mehta,
2013-01-15, 01:03
|
|
|
Re: Question about gzip compression when using Flume Ng - Flume - [mail # user]
|
|
...ok so I dropped in the new hadoop-core jar in /opt/flume/lib [I got some errors about the guava dependencies so put in that jar too] smehta@collector102:/opt/flume/lib$ ls -ltrh | grep...
|
|
|
Author: Sagar Mehta,
2013-01-15, 00:43
|
|
|
Re: Question about gzip compression when using Flume Ng - Flume - [mail # user]
|
|
...About the bz2 suggestion, we have a ton of downstream jobs that assume gzip compressed files - so it is better to stick to gzip. The plan B for us is to have a Oozie step to gzip compr...
|
|
|
Author: Sagar Mehta,
2013-01-14, 23:27
|
|
|
Re: Question about gzip compression when using Flume Ng - Flume - [mail # user]
|
|
...hadoop@jobtracker301:/home/hadoop/sagar/debug$ zcat collector102.ngpipes.sac.ngmoco.com.1358204406896.gz | wc -l gzip: collector102.ngpipes.sac.ngmoco.com.1358204406896.gz: decompressi...
|
|
|
Author: Sagar Mehta,
2013-01-14, 23:24
|
|
|
|
