Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> HBase Query Support


Copy link to this message
-
Re: HBase Query Support
Hi Aditya/James,

     As per my understanding Phoenix (having great features while
storing[e.g. secondary index] + Schema) can be directly used for data
insertion (Drill does not come in to picture in this flow).

     And for query retrieval;  first level we can use Phoenix end user
API in local Drillbit to query data.

     DrillUsers -> DrillBit+EmbaddedPhoenixJDBCClient (local mode,
single instance) -> {HBase cluster..n}

     In the mean time James can draft a design where DrillBit can use
more lower level API to benefit from Drill's Distributed operators,Joins
and ValueVector memory model, and using Phoenix's Hbase side
coprocessors and custom filters (as operator pushdown)

     DrillUsers -> {DrillBit+EmbaddedPhoenixClient (Distributed mode)
..n} -> -> {HBase cluster..n}

     Also we should be able to use HBase Table Schema generated by Phoenix.

     Your thoughts?

-Dhruv

On 10/29/2013 03:52 AM, James Taylor wrote:
> Sure, I'd be happy to.
>
> Thanks,
> James
>
>
> On Mon, Oct 28, 2013 at 3:16 PM, Aditya <[EMAIL PROTECTED]> wrote:
>
>> James,
>>
>> Jacques and I discussed this recently and we concur that it would be a
>> beneficial to build such an integration between the two projects.
>>
>> Would you have some time to have a meeting/hangout with Drill team to talk
>> this further?
>>
>> Regards,
>> Aditya...
>>
>> Regards,
>> Aditya...
>>
>>
>> On Mon, Oct 28, 2013 at 1:31 PM, James Taylor <[EMAIL PROTECTED]
>>> wrote:
>>> Hi,
>>> The idea for the Phoenix integration was to come up with the right API to
>>> be able to plug Phoenix into Drill as the HBase query engine/store.
>> Jacque
>>> and I got busy on our respective releases, but now that those are both
>> out,
>>> I think we should get back to it. If we did it right, then both products
>>> would be able to continue to evolve independently, but Drill would have
>> top
>>> notch query performance for HBase data and Phoenix would have a new
>>> customer.
>>>
>>> Thoughts/ideas?
>>>
>>> Thanks,
>>> James
>>>
>>>
>>> On Mon, Oct 28, 2013 at 5:34 AM, Dhruv <[EMAIL PROTECTED]>
>> wrote:
>>>> Hi,
>>>>
>>>>   I am interested in contributing towards providing HBase Query support
>> in
>>>> DRILL.
>>>>
>>>>   A) Can anyone summarize existing plan+Priority for it? (Or we can
>>> discuss
>>>> in this thread and I will create WIKI page)
>>>>
>>>>   B) I see already following JIRA issues, Anybody working on this
>> locally?
>>>>     B-1)    https://issues.apache.org/**jira/browse/DRILL-15<
>>> https://issues.apache.org/jira/browse/DRILL-15> last updated 13-Aug-13.
>>>>     B-2)    https://issues.apache.org/**jira/browse/DRILL-88<
>>> https://issues.apache.org/jira/browse/DRILL-88>Unassigned
>>>>     B-3)    https://issues.apache.org/**jira/browse/DRILL-63<
>>> https://issues.apache.org/jira/browse/DRILL-63>James Taylor seems
>>> interested to work
>>>>   C) As per my understanding  implimenting
>>> org.apache.drill.exec.store.**hbase
>>>> package with following for local mode drill bit should suffice for POC?
>>>>       HBASERecordReader.java
>>>>        HBASEGroupScan
>>>>        HBASEScanBatchCreator
>>>>        HbaseSchemaProvider
>>>>        HBASEStorageEngine
>>>>        HBASEStorageEngineConfig
>>>>        HBASESubScan
>>>>
>>>>   D) What part of https://github.com/**forcedotcom/phoenix<
>>> https://github.com/forcedotcom/phoenix>can we reuse? Any phonix
>>> contributors around?
>>>> -Dhruv
>>>>