-Re: speeding up demux
Ariel Rabkin 2011-06-01, 20:23
Trunk should be stable at this point.
Performance should be about the same as 0.3 or 0.4 -- most Chukwa
development has been geared to features and bugfixes.
CHANGES.txt in trunk is the changelist since 0.4
Trunk compiles, last I checked.
On Wed, Jun 1, 2011 at 1:01 PM, James Seigel <[EMAIL PROTECTED]> wrote:
> I am seriously considering what you are suggesting in this email, even
> though it goes against what would seem to make sense. I have a couple of
> questions if anyone has the time to answer.
> 1) How stable is trunk right now?
> 2) Any performance improvements/degredations since 0.3
> 3) Is there a pseudo change log between “trunk” and 0.4 that I could take a
> peak at at this point
> 4) does it compile ;)
> Cheers and thanks for your time!
> On 2011-05-27, at 9:58 AM, Eric Yang wrote:
> I would recommend to skip Chukwa 0.4 and go to the trunk. In addition, use
> HBaseWriter to stream data into HBase in parallel, hence, the data can be
> processed in near real time for demux.
> On 5/26/11 8:30 PM, "Bill Graham" <[EMAIL PROTECTED]> wrote:
> This seems possible, but one thing that would need to be changed is the
> directories that demux uses. For example:
> These would need to dynamic directories with the timestamp or something else
> in them to keep two jobs from interfering with each other.
> On Thu, May 26, 2011 at 8:23 PM, Corbin Hoenes <[EMAIL PROTECTED]> wrote:
> Finding demux to be a bit too slow for our needs. It seems like only 1 runs
> at a time; is there some technical reason why we couldn't run a couple in
> parallel? If so any hints on how difficult it would be to run multiple
> demuxers at a time?
Ari Rabkin [EMAIL PROTECTED]
UC Berkeley Computer Science Department