I am implementing some functionality as a HBase coprocessor which relies
on reading WALEdits from the HBase WAL.

I have to requirements:

- get notified whenever a new WALEdit is written to the WAL
- have access to the actual WAL log file such that I can read from it
whenever I want

In a way, this is very similar to how HBase replication works.

The first requirement is easy enough, using the WALObserver coprocessor
interface. It is for the second requirement however, that I am asking
this question.

HBase replication uses the WALActionsListener interface to get notified
when the log files are rolled. There doesn't seem to be a way to do that
in a WALObserver (coprocessor interface) though.

I thought of adding a WALActionsListener myself from inside the
coprocessor code, but I would need to make sure that it is registered
before the preLogRolled method is invoked for the first time.

I also looked at HBASE-4132, but I believe this talks about the
WALActionsListener and not the WALObserver.
So.. any hints on how to do WAL reading (like replication) from inside a
coprocessor?

thanks,
Jan

I filed HBASE-4844 a while ago but never actually provided a patch.
If you can comment on that jira with the exact use case that'd be helpful.

If you wanted to work on a patch that'd be cooler even :)

-- Lars

________________________________
 From: Jan Van Besien <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, October 29, 2012 2:42 AM
Subject: reading WAL log file from inside coprocessor
 
I am implementing some functionality as a HBase coprocessor which relies on reading WALEdits from the HBase WAL.

I have to requirements:

- get notified whenever a new WALEdit is written to the WAL
- have access to the actual WAL log file such that I can read from it whenever I want

In a way, this is very similar to how HBase replication works.

The first requirement is easy enough, using the WALObserver coprocessor interface. It is for the second requirement however, that I am asking this question.

HBase replication uses the WALActionsListener interface to get notified when the log files are rolled. There doesn't seem to be a way to do that in a WALObserver (coprocessor interface) though.

I thought of adding a WALActionsListener myself from inside the coprocessor code, but I would need to make sure that it is registered before the preLogRolled method is invoked for the first time.

I also looked at HBASE-4132, but I believe this talks about the WALActionsListener and not the WALObserver.
So.. any hints on how to do WAL reading (like replication) from inside a coprocessor?

thanks,
Jan

On 10/29/2012 04:34 PM, lars hofhansl wrote:
> I filed HBASE-4844 a while ago but never actually provided a patch.
> If you can comment on that jira with the exact use case that'd be helpful.
>
> If you wanted to work on a patch that'd be cooler even :)

Interesting thanks, I'll look into it.

The ticket seems to confirm that registering a WALActionListener from
inside a coprocessor is not going to work. Something I discovered today
as well.

With a completely different approach, I did manage to get something
working by constructing the path to the HLog files myself (in the
coprocessor) and reading them from there. So I am not notified about the
HLog files being rolled etc, I simply poll the directory on HDFS for
available files. It's a bit an ugly solution, and (apart from additional
delays) maybe there are even issues with it that I did not think of yet?

Anyway, I would like the HBASE-4844 approach a lot better for sure.

thanks,
Jan