HBase, mail # user - Re: What's the best approach to search in HBase?

What's the best approach to search in HBase?

Mark Kerzner 2011-06-04, 00:57

Hi,

I need to store, say, 10M-100M documents, with each document having say 100
fields, like author, creation date, access date, etc., and then I want to
ask questions like

give me all documents whose author is like abc**, and creation date any time
in 2010 and access date in 2010-2011, and so on, perhaps 10-20 conditions,
matching a list of some keywords.

What's best, Lucene, Katta, HBase CF with secondary indices, or plain scan
and compare of every record?

Thanks a bunch!

Mark

Re: What's the best approach to search in HBase?

Joey Echeverria 2011-06-04, 02:21

I can't claim its the best, but I'd say solar or katta.

-Joey
On Jun 3, 2011 8:57 PM, "Mark Kerzner" <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I need to store, say, 10M-100M documents, with each document having say
100
> fields, like author, creation date, access date, etc., and then I want to
> ask questions like
>
> give me all documents whose author is like abc**, and creation date any
time
> in 2010 and access date in 2010-2011, and so on, perhaps 10-20 conditions,
> matching a list of some keywords.
>
> What's best, Lucene, Katta, HBase CF with secondary indices, or plain scan
> and compare of every record?
>
> Thanks a bunch!
>
> Mark

Re: What's the best approach to search in HBase?

Jason Rutherglen 2011-06-04, 02:23

Mark,

'Add search to HBase' - HBASE-3529 is in development.

On Fri, Jun 3, 2011 at 5:57 PM, Mark Kerzner <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I need to store, say, 10M-100M documents, with each document having say 100
> fields, like author, creation date, access date, etc., and then I want to
> ask questions like
>
> give me all documents whose author is like abc**, and creation date any time
> in 2010 and access date in 2010-2011, and so on, perhaps 10-20 conditions,
> matching a list of some keywords.
>
> What's best, Lucene, Katta, HBase CF with secondary indices, or plain scan
> and compare of every record?
>
> Thanks a bunch!
>
> Mark
>

Re: What's the best approach to search in HBase?

Matt Davies 2011-06-04, 03:36

I, for one, am interested in learning more about elasticsearch with HBase
after reading the article over at StumbleUpon (
http://www.stumbleupon.com/devblog/searching-for-serendipity/)

Intriguing that it is relatively easy to set up. Anyone else using
elasticsearch?

-Matt

On Fri, Jun 3, 2011 at 8:23 PM, Jason Rutherglen <[EMAIL PROTECTED]
> wrote:

> Mark,
>
> 'Add search to HBase' - HBASE-3529 is in development.
>
> On Fri, Jun 3, 2011 at 5:57 PM, Mark Kerzner <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > I need to store, say, 10M-100M documents, with each document having say
> 100
> > fields, like author, creation date, access date, etc., and then I want to
> > ask questions like
> >
> > give me all documents whose author is like abc**, and creation date any
> time
> > in 2010 and access date in 2010-2011, and so on, perhaps 10-20
> conditions,
> > matching a list of some keywords.
> >
> > What's best, Lucene, Katta, HBase CF with secondary indices, or plain
> scan
> > and compare of every record?
> >
> > Thanks a bunch!
> >
> > Mark
> >
>

Re: What's the best approach to search in HBase?

Jason Rutherglen 2011-06-04, 03:50

I think this's the key line: "ElasticSearch is the search analogue to
HBase that frees us from some restrictions that Solr imposes". It is
quite true however if search is inside of HBase ones gets the same
thing. Solr does have serious limitations in terms of scaling etc. I
think ES has done a great job there, though this could have been done
with Solr just as easily, eg, upgrade Solr with the same functionality
and remove the need for schemas. Solr does allow schema-less, with
eg, dynamic fields.

On Fri, Jun 3, 2011 at 8:36 PM, Matt Davies <[EMAIL PROTECTED]> wrote:
> I, for one, am interested in learning more about elasticsearch with HBase
> after reading the article over at StumbleUpon (
> http://www.stumbleupon.com/devblog/searching-for-serendipity/)
>
> Intriguing that it is relatively easy to set up. Anyone else using
> elasticsearch?
>
> -Matt
>
> On Fri, Jun 3, 2011 at 8:23 PM, Jason Rutherglen <[EMAIL PROTECTED]
>> wrote:
>
>> Mark,
>>
>> 'Add search to HBase' - HBASE-3529 is in development.
>>
>> On Fri, Jun 3, 2011 at 5:57 PM, Mark Kerzner <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > I need to store, say, 10M-100M documents, with each document having say
>> 100
>> > fields, like author, creation date, access date, etc., and then I want to
>> > ask questions like
>> >
>> > give me all documents whose author is like abc**, and creation date any
>> time
>> > in 2010 and access date in 2010-2011, and so on, perhaps 10-20
>> conditions,
>> > matching a list of some keywords.
>> >
>> > What's best, Lucene, Katta, HBase CF with secondary indices, or plain
>> scan
>> > and compare of every record?
>> >
>> > Thanks a bunch!
>> >
>> > Mark
>> >
>>
>

Re: What's the best approach to search in HBase?

Steven Noels 2011-06-04, 18:07

On Sat, Jun 4, 2011 at 2:57 AM, Mark Kerzner <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I need to store, say, 10M-100M documents, with each document having say 100
> fields, like author, creation date, access date, etc., and then I want to
> ask questions like
>
> give me all documents whose author is like abc**, and creation date any
> time
> in 2010 and access date in 2010-2011, and so on, perhaps 10-20 conditions,
> matching a list of some keywords.
>
> What's best, Lucene, Katta, HBase CF with secondary indices, or plain scan
> and compare of every record?
>

I'd say give Lily a spin. Currently, we rely on Solr for search. In the next
few months, we'll take a good look at "HBase-native" secondary indexes as
well.

Lily can be found at www.lilyproject.org.

Thanks,

Steven.
--
Steven Noels
http://outerthought.org/
Scalable Smart Data
Makers of Kauri, Daisy CMS and Lily

Re: What's the best approach to search in HBase?

M. C. Srivas 2011-06-04, 18:15

There's also DrawnToScale http://www.drawntoscale.com/.

Don't know if its released or not.

On Sat, Jun 4, 2011 at 11:07 AM, Steven Noels <[EMAIL PROTECTED]>wrote:

> On Sat, Jun 4, 2011 at 2:57 AM, Mark Kerzner <[EMAIL PROTECTED]>
> wrote:
>
> > Hi,
> >
> > I need to store, say, 10M-100M documents, with each document having say
> 100
> > fields, like author, creation date, access date, etc., and then I want to
> > ask questions like
> >
> > give me all documents whose author is like abc**, and creation date any
> > time
> > in 2010 and access date in 2010-2011, and so on, perhaps 10-20
> conditions,
> > matching a list of some keywords.
> >
> > What's best, Lucene, Katta, HBase CF with secondary indices, or plain
> scan
> > and compare of every record?
> >
>
> I'd say give Lily a spin. Currently, we rely on Solr for search. In the
> next
> few months, we'll take a good look at "HBase-native" secondary indexes as
> well.
>
> Lily can be found at www.lilyproject.org.
>
> Thanks,
>
> Steven.
> --
> Steven Noels
> http://outerthought.org/
> Scalable Smart Data
> Makers of Kauri, Daisy CMS and Lily
>

Re: What's the best approach to search in HBase?

Jason Rutherglen 2011-06-04, 18:40

That doesn't look like it's open source though? Isn't it an SaS?

On Sat, Jun 4, 2011 at 11:15 AM, M. C. Srivas <[EMAIL PROTECTED]> wrote:
> There's also DrawnToScale http://www.drawntoscale.com/.
>
> Don't know if its released or not.
>
> On Sat, Jun 4, 2011 at 11:07 AM, Steven Noels <[EMAIL PROTECTED]>wrote:
>
>> On Sat, Jun 4, 2011 at 2:57 AM, Mark Kerzner <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Hi,
>> >
>> > I need to store, say, 10M-100M documents, with each document having say
>> 100
>> > fields, like author, creation date, access date, etc., and then I want to
>> > ask questions like
>> >
>> > give me all documents whose author is like abc**, and creation date any
>> > time
>> > in 2010 and access date in 2010-2011, and so on, perhaps 10-20
>> conditions,
>> > matching a list of some keywords.
>> >
>> > What's best, Lucene, Katta, HBase CF with secondary indices, or plain
>> scan
>> > and compare of every record?
>> >
>>
>> I'd say give Lily a spin. Currently, we rely on Solr for search. In the
>> next
>> few months, we'll take a good look at "HBase-native" secondary indexes as
>> well.
>>
>> Lily can be found at www.lilyproject.org.
>>
>> Thanks,
>>
>> Steven.
>> --
>> Steven Noels
>> http://outerthought.org/
>> Scalable Smart Data
>> Makers of Kauri, Daisy CMS and Lily
>>
>

RE: What's the best approach to search in HBase?

Buttler, David 2011-06-06, 16:18

I store over 500M documents in HBase, and index using Solr with dynamic fields. This gives you tremendous flexibility to do the type of queries you are looking for -- and to make them simple and intuitive via a faceted interface.

However, there was quite a bit of software that we had to write to get things going, and I can neither release all of it open source, or support other people using it. If I had to start again, I would seriously look at solutions like elastic search and lily.

Dave

-----Original Message-----
From: Mark Kerzner [mailto:[EMAIL PROTECTED]]
Sent: Friday, June 03, 2011 5:57 PM
To: HBase Discussion Group
Subject: What's the best approach to search in HBase?

Hi,

I need to store, say, 10M-100M documents, with each document having say 100
fields, like author, creation date, access date, etc., and then I want to
ask questions like

give me all documents whose author is like abc**, and creation date any time
in 2010 and access date in 2010-2011, and so on, perhaps 10-20 conditions,
matching a list of some keywords.

What's best, Lucene, Katta, HBase CF with secondary indices, or plain scan
and compare of every record?

Thanks a bunch!

Mark

Re: What's the best approach to search in HBase?

Mark Kerzner 2011-06-16, 04:09

Thank you, everybody. I summarized your advice here,
http://shmsoft.blogspot.com/2011/06/search-in-ediscovery.html, because I
need it for my open source eDiscovery, and now just need to try it all :)

Sincerely,
Mark

On Mon, Jun 6, 2011 at 11:18 AM, Buttler, David <[EMAIL PROTECTED]> wrote:

> I store over 500M documents in HBase, and index using Solr with dynamic
> fields. This gives you tremendous flexibility to do the type of queries you
> are looking for -- and to make them simple and intuitive via a faceted
> interface.
>
> However, there was quite a bit of software that we had to write to get
> things going, and I can neither release all of it open source, or support
> other people using it. If I had to start again, I would seriously look at
> solutions like elastic search and lily.
>
> Dave
>
> -----Original Message-----
> From: Mark Kerzner [mailto:[EMAIL PROTECTED]]
> Sent: Friday, June 03, 2011 5:57 PM
> To: HBase Discussion Group
> Subject: What's the best approach to search in HBase?
>
> Hi,
>
> I need to store, say, 10M-100M documents, with each document having say 100
> fields, like author, creation date, access date, etc., and then I want to
> ask questions like
>
> give me all documents whose author is like abc**, and creation date any
> time
> in 2010 and access date in 2010-2011, and so on, perhaps 10-20 conditions,
> matching a list of some keywords.
>
> What's best, Lucene, Katta, HBase CF with secondary indices, or plain scan
> and compare of every record?
>
> Thanks a bunch!
>
> Mark
>

RE: What's the best approach to search in HBase?

Hiller, Dean x66079 2011-06-17, 20:21

What about using Hbasene....is it pretty good....looks just like a distributed Lucene and the same api and everything?

Later,
Dean

-----Original Message-----
From: Mark Kerzner [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 15, 2011 10:10 PM
To: [EMAIL PROTECTED]
Subject: Re: What's the best approach to search in HBase?

Thank you, everybody. I summarized your advice here,
http://shmsoft.blogspot.com/2011/06/search-in-ediscovery.html, because I
need it for my open source eDiscovery, and now just need to try it all :)

Sincerely,
Mark

On Mon, Jun 6, 2011 at 11:18 AM, Buttler, David <[EMAIL PROTECTED]> wrote:

> I store over 500M documents in HBase, and index using Solr with dynamic
> fields. This gives you tremendous flexibility to do the type of queries you
> are looking for -- and to make them simple and intuitive via a faceted
> interface.
>
> However, there was quite a bit of software that we had to write to get
> things going, and I can neither release all of it open source, or support
> other people using it. If I had to start again, I would seriously look at
> solutions like elastic search and lily.
>
> Dave
>
> -----Original Message-----
> From: Mark Kerzner [mailto:[EMAIL PROTECTED]]
> Sent: Friday, June 03, 2011 5:57 PM
> To: HBase Discussion Group
> Subject: What's the best approach to search in HBase?
>
> Hi,
>
> I need to store, say, 10M-100M documents, with each document having say 100
> fields, like author, creation date, access date, etc., and then I want to
> ask questions like
>
> give me all documents whose author is like abc**, and creation date any
> time
> in 2010 and access date in 2010-2011, and so on, perhaps 10-20 conditions,
> matching a list of some keywords.
>
> What's best, Lucene, Katta, HBase CF with secondary indices, or plain scan
> and compare of every record?
>
> Thanks a bunch!
>
> Mark
>
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.

Re: What's the best approach to search in HBase?

Otis Gospodnetic 2011-06-18, 06:27

HBasene is dead. Watch HBASE-3529.
Otis
We're hiring HBase / Hadoop / Hive / Mahout engineers with interest in Big Data Mining and Analytics
http://blog.sematext.com/2011/04/18/hiring-data-mining-analytics-machine-learning-hackers/
From: "Hiller, Dean x66079" <[EMAIL PROTECTED]>

To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Sent: Friday, June 17, 2011 4:21 PM
>Subject: RE: What's the best approach to search in HBase?
>
>What about using Hbasene....is it pretty good....looks just like a distributed Lucene and the same api and everything?
>
>Later,
>Dean
>
>-----Original Message-----
>From: Mark Kerzner [mailto:[EMAIL PROTECTED]]
>Sent: Wednesday, June 15, 2011 10:10 PM
>To: [EMAIL PROTECTED]
>Subject: Re: What's the best approach to search in HBase?
>
>Thank you, everybody. I summarized your advice here,
>http://shmsoft.blogspot.com/2011/06/search-in-ediscovery.html, because I
>need it for my open source eDiscovery, and now just need to try it all :)
>
>Sincerely,
>Mark
>
>On Mon, Jun 6, 2011 at 11:18 AM, Buttler, David <[EMAIL PROTECTED]> wrote:
>
>> I store over 500M documents in HBase, and index using Solr with dynamic
>> fields. This gives you tremendous flexibility to do the type of queries you
>> are looking for -- and to make them simple and intuitive via a faceted
>> interface.
>>
>> However, there was quite a bit of software that we had to write to get
>> things going, and I can neither release all of it open source, or support
>> other people using it. If I had to start again, I would seriously look at
>> solutions like elastic search and lily.
>>
>> Dave
>>
>> -----Original Message-----
>> From: Mark Kerzner [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, June 03, 2011 5:57 PM
>> To: HBase Discussion Group
>> Subject: What's the best approach to search in HBase?
>>
>> Hi,
>>
>> I need to store, say, 10M-100M documents, with each document having say 100
>> fields, like author, creation date, access date, etc., and then I want to
>> ask questions like
>>
>> give me all documents whose author is like abc**, and creation date any
>> time
>> in 2010 and access date in 2010-2011, and so on, perhaps 10-20 conditions,
>> matching a list of some keywords.
>>
>> What's best, Lucene, Katta, HBase CF with secondary indices, or plain scan
>> and compare of every record?
>>
>> Thanks a bunch!
>>
>> Mark
>>
>This message and any attachments are intended only for the use of the addressee and
>may contain information that is privileged and confidential. If the reader of the
>message is not the intended recipient or an authorized representative of the
>intended recipient, you are hereby notified that any dissemination of this
>communication is strictly prohibited. If you have received this communication in
>error, please notify us immediately by e-mail and delete the message and any
>attachments from your system.
>
>
>
>