Hi,

I was looking at
http://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-systemand
then at
https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserverand
wondering...

.... would it be possible/interesting/better to "open up" a little bit and
implement something that puts HLog events on a bus like that Databus and
thus allow *anyone* to do *anything* with data being written to (or deleted
from) HBase?  Replication itself, secondary indexing, etc.

Thanks,
Otis
--
HBASE Performance Monitoring
http://sematext.com/spm/hbase-performance-monitoring/

Yes, I know a lot of people would love having that.

J-D

On Sat, Mar 2, 2013 at 7:08 PM, Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I was looking at
> http://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-systemand
> then at
> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserverand
> wondering...
>
> .... would it be possible/interesting/better to "open up" a little bit and
> implement something that puts HLog events on a bus like that Databus and
> thus allow *anyone* to do *anything* with data being written to (or deleted
> from) HBase?  Replication itself, secondary indexing, etc.
>
> Thanks,
> Otis
> --
> HBASE Performance Monitoring
> http://sematext.com/spm/hbase-performance-monitoring/

Hi Otis,

This is a cool idea. Would you see a single event pipe containing all
edits, per RS pipes, something else entirely?

-n

On Sat, Mar 2, 2013 at 7:08 PM, Otis Gospodnetic <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> I was looking at
>
> http://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-systemand
> then at
>
> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserverand
> wondering...
>
> .... would it be possible/interesting/better to "open up" a little bit and
> implement something that puts HLog events on a bus like that Databus and
> thus allow *anyone* to do *anything* with data being written to (or deleted
> from) HBase?  Replication itself, secondary indexing, etc.
>
> Thanks,
> Otis
> --
> HBASE Performance Monitoring
> http://sematext.com/spm/hbase-performance-monitoring/
>

Hi Nick,

I don't know, what would be better/easier?
I'd think per RS may be OK, but you'll know better about where HLogs can be
"caught".

Btw. that Databus project just happened to spark the idea, but I guess the
general version of what I'm thinking is simply: "Can we introduce some HLog
Listener interfaces/hooks so people can plug in there and do whatever they
want with HLog entries".  One such impl. can be for putting HLog entries on
that Databus.

Otis
--
HBASE Performance Monitoring
http://sematext.com/spm/hbase-performance-monitoring/
On Mon, Mar 4, 2013 at 2:39 PM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> Hi Otis,
>
> This is a cool idea. Would you see a single event pipe containing all
> edits, per RS pipes, something else entirely?
>
> -n
>
> On Sat, Mar 2, 2013 at 7:08 PM, Otis Gospodnetic <
> [EMAIL PROTECTED]
> > wrote:
>
> > Hi,
> >
> > I was looking at
> >
> >
> http://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-systemand
> > then at
> >
> >
> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserverand
> > wondering...
> >
> > .... would it be possible/interesting/better to "open up" a little bit
> and
> > implement something that puts HLog events on a bus like that Databus and
> > thus allow *anyone* to do *anything* with data being written to (or
> deleted
> > from) HBase?  Replication itself, secondary indexing, etc.
> >
> > Thanks,
> > Otis
> > --
> > HBASE Performance Monitoring
> > http://sematext.com/spm/hbase-performance-monitoring/
> >
>

Wouldn't the WALObserver coprocessor interface work for this?  Is there
anything missing from what it exposes?
On Tue, Mar 5, 2013 at 2:50 PM, Otis Gospodnetic <[EMAIL PROTECTED]
> wrote:

> Hi Nick,
>
> I don't know, what would be better/easier?
> I'd think per RS may be OK, but you'll know better about where HLogs can be
> "caught".
>
> Btw. that Databus project just happened to spark the idea, but I guess the
> general version of what I'm thinking is simply: "Can we introduce some HLog
> Listener interfaces/hooks so people can plug in there and do whatever they
> want with HLog entries".  One such impl. can be for putting HLog entries on
> that Databus.
>
> Otis
> --
> HBASE Performance Monitoring
> http://sematext.com/spm/hbase-performance-monitoring/
>
>
>
>
> On Mon, Mar 4, 2013 at 2:39 PM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:
>
> > Hi Otis,
> >
> > This is a cool idea. Would you see a single event pipe containing all
> > edits, per RS pipes, something else entirely?
> >
> > -n
> >
> > On Sat, Mar 2, 2013 at 7:08 PM, Otis Gospodnetic <
> > [EMAIL PROTECTED]
> > > wrote:
> >
> > > Hi,
> > >
> > > I was looking at
> > >
> > >
> >
> http://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-systemand
> > > then at
> > >
> > >
> >
> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserverand
> > > wondering...
> > >
> > > .... would it be possible/interesting/better to "open up" a little bit
> > and
> > > implement something that puts HLog events on a bus like that Databus
> and
> > > thus allow *anyone* to do *anything* with data being written to (or
> > deleted
> > > from) HBase?  Replication itself, secondary indexing, etc.
> > >
> > > Thanks,
> > > Otis
> > > --
> > > HBASE Performance Monitoring
> > > http://sematext.com/spm/hbase-performance-monitoring/
> > >
> >
>

It could also be done as a new type of replication sink I'd imagine. Either
way it seems we have the plumbing?

On Wednesday, March 6, 2013, Gary Helmling wrote:

> Wouldn't the WALObserver coprocessor interface work for this?  Is there
> anything missing from what it exposes?
>
>
> On Tue, Mar 5, 2013 at 2:50 PM, Otis Gospodnetic <
> [EMAIL PROTECTED] <javascript:;>
> > wrote:
>
> > Hi Nick,
> >
> > I don't know, what would be better/easier?
> > I'd think per RS may be OK, but you'll know better about where HLogs can
> be
> > "caught".
> >
> > Btw. that Databus project just happened to spark the idea, but I guess
> the
> > general version of what I'm thinking is simply: "Can we introduce some
> HLog
> > Listener interfaces/hooks so people can plug in there and do whatever
> they
> > want with HLog entries".  One such impl. can be for putting HLog entries
> on
> > that Databus.
> >
> > Otis
> > --
> > HBASE Performance Monitoring
> > http://sematext.com/spm/hbase-performance-monitoring/
> >
> >
> >
> >
> > On Mon, Mar 4, 2013 at 2:39 PM, Nick Dimiduk <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> >
> > > Hi Otis,
> > >
> > > This is a cool idea. Would you see a single event pipe containing all
> > > edits, per RS pipes, something else entirely?
> > >
> > > -n
> > >
> > > On Sat, Mar 2, 2013 at 7:08 PM, Otis Gospodnetic <
> > > [EMAIL PROTECTED] <javascript:;>
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I was looking at
> > > >
> > > >
> > >
> >
> http://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-systemand
> > > > then at
> > > >
> > > >
> > >
> >
> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserverand
> > > > wondering...
> > > >
> > > > .... would it be possible/interesting/better to "open up" a little
> bit
> > > and
> > > > implement something that puts HLog events on a bus like that Databus
> > and
> > > > thus allow *anyone* to do *anything* with data being written to (or
> > > deleted
> > > > from) HBase?  Replication itself, secondary indexing, etc.
> > > >
> > > > Thanks,
> > > > Otis
> > > > --
> > > > HBASE Performance Monitoring
> > > > http://sematext.com/spm/hbase-performance-monitoring/
> > > >
> > >
> >
>
--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)