|
Jonathan Bishop
2012-11-05, 18:41
Jean-Daniel Cryans
2012-11-05, 19:38
anil gupta
2012-11-05, 22:57
lars hofhansl
2012-11-06, 00:01
Dave Latham
2012-11-06, 00:26
Dave Latham
2012-11-06, 00:27
Jonathan Bishop
2012-11-06, 05:07
David Koch
2012-12-01, 15:01
Marcos Ortiz
2012-12-01, 15:27
|
-
Using doubles and longs as ordering row valuesJonathan Bishop 2012-11-05, 18:41
Hi,
In my application my row values are doubles. I would like my scans to traverse the rows in order of increasing values. But if I simply use double d = .... byte[] row = Bytes.toBytes(d); I will get an ordering which is based on the byte values of doubles, not on the value of the doubles themselves. I see also that integer values have the same issue due to the first bit being the sign bit. So negative values will come after positive values. Any suggestions? Thanks, Jon
-
Re: Using doubles and longs as ordering row valuesJean-Daniel Cryans 2012-11-05, 19:38
On Mon, Nov 5, 2012 at 10:41 AM, Jonathan Bishop <[EMAIL PROTECTED]> wrote:
> Hi, > > In my application my row values are doubles. I would like my scans to > traverse the rows in order of increasing values. > > But if I simply use > > double d = .... > byte[] row = Bytes.toBytes(d); > > I will get an ordering which is based on the byte values of doubles, not on > the value of the doubles themselves. It's the same ordering as long as you don't use negative values. > > I see also that integer values have the same issue due to the first bit > being the sign bit. So negative values will come after positive values. > > Any suggestions? Don't use row keys that can be negative? :) Also don't use a single number as a row key, see http://hbase.apache.org/book.html#rowkey.design J-D
-
Re: Using doubles and longs as ordering row valuesanil gupta 2012-11-05, 22:57
Hi Jonathan,
If possible try to avoid using Double for rowkey because Double has precision problems. Here is more details about problems with double: 1. http://stackoverflow.com/questions/12699040/double-vs-long-serialization-in-java 2. http://stackoverflow.com/questions/179427/how-to-resolve-a-java-rounding-double-issue You might get wrong results due problems in Double. You can use two long or int to store the decimal value as RowKey. HTH, Anil Gupta On Mon, Nov 5, 2012 at 11:38 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote: > On Mon, Nov 5, 2012 at 10:41 AM, Jonathan Bishop <[EMAIL PROTECTED]> > wrote: > > Hi, > > > > In my application my row values are doubles. I would like my scans to > > traverse the rows in order of increasing values. > > > > But if I simply use > > > > double d = .... > > byte[] row = Bytes.toBytes(d); > > > > I will get an ordering which is based on the byte values of doubles, not > on > > the value of the doubles themselves. > > It's the same ordering as long as you don't use negative values. > > > > > I see also that integer values have the same issue due to the first bit > > being the sign bit. So negative values will come after positive values. > > > > Any suggestions? > > Don't use row keys that can be negative? :) > > Also don't use a single number as a row key, see > http://hbase.apache.org/book.html#rowkey.design > > J-D > -- Thanks & Regards, Anil Gupta
-
Re: Using doubles and longs as ordering row valueslars hofhansl 2012-11-06, 00:01
Have a look at the lily library. It has code to encode Longs/Doubles into bytes such that resulting bytes sort as expected based on the numbers.
-- Lars ________________________________ From: Jonathan Bishop <[EMAIL PROTECTED]> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> Sent: Monday, November 5, 2012 10:41 AM Subject: Using doubles and longs as ordering row values Hi, In my application my row values are doubles. I would like my scans to traverse the rows in order of increasing values. But if I simply use double d = .... byte[] row = Bytes.toBytes(d); I will get an ordering which is based on the byte values of doubles, not on the value of the doubles themselves. I see also that integer values have the same issue due to the first bit being the sign bit. So negative values will come after positive values. Any suggestions? Thanks, Jon
-
Re: Using doubles and longs as ordering row valuesDave Latham 2012-11-06, 00:26
Here's a project to deal with this issue specifically. I'm not sure of
it's status: https://github.com/conikeec/orderly On Mon, Nov 5, 2012 at 4:01 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: > Have a look at the lily library. It has code to encode Longs/Doubles into > bytes such that resulting bytes sort as expected based on the numbers. > > -- Lars > > > > ________________________________ > From: Jonathan Bishop <[EMAIL PROTECTED]> > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > Sent: Monday, November 5, 2012 10:41 AM > Subject: Using doubles and longs as ordering row values > > Hi, > > In my application my row values are doubles. I would like my scans to > traverse the rows in order of increasing values. > > But if I simply use > > double d = .... > byte[] row = Bytes.toBytes(d); > > I will get an ordering which is based on the byte values of doubles, not on > the value of the doubles themselves. > > I see also that integer values have the same issue due to the first bit > being the sign bit. So negative values will come after positive values. > > Any suggestions? > > Thanks, > > Jon >
-
Re: Using doubles and longs as ordering row valuesDave Latham 2012-11-06, 00:27
This fork looks a bit more up to date:
https://github.com/ndimiduk/orderly On Mon, Nov 5, 2012 at 4:26 PM, Dave Latham <[EMAIL PROTECTED]> wrote: > Here's a project to deal with this issue specifically. I'm not sure of > it's status: > https://github.com/conikeec/orderly > > > On Mon, Nov 5, 2012 at 4:01 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: > >> Have a look at the lily library. It has code to encode Longs/Doubles into >> bytes such that resulting bytes sort as expected based on the numbers. >> >> -- Lars >> >> >> >> ________________________________ >> From: Jonathan Bishop <[EMAIL PROTECTED]> >> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> >> Sent: Monday, November 5, 2012 10:41 AM >> Subject: Using doubles and longs as ordering row values >> >> Hi, >> >> In my application my row values are doubles. I would like my scans to >> traverse the rows in order of increasing values. >> >> But if I simply use >> >> double d = .... >> byte[] row = Bytes.toBytes(d); >> >> I will get an ordering which is based on the byte values of doubles, not >> on >> the value of the doubles themselves. >> >> I see also that integer values have the same issue due to the first bit >> being the sign bit. So negative values will come after positive values. >> >> Any suggestions? >> >> Thanks, >> >> Jon >> > >
-
Re: Using doubles and longs as ordering row valuesJonathan Bishop 2012-11-06, 05:07
Thanks Dave,
That looks like what I need. Jon On Mon, Nov 5, 2012 at 4:27 PM, Dave Latham <[EMAIL PROTECTED]> wrote: > This fork looks a bit more up to date: > https://github.com/ndimiduk/orderly > > On Mon, Nov 5, 2012 at 4:26 PM, Dave Latham <[EMAIL PROTECTED]> wrote: > > > Here's a project to deal with this issue specifically. I'm not sure of > > it's status: > > https://github.com/conikeec/orderly > > > > > > On Mon, Nov 5, 2012 at 4:01 PM, lars hofhansl <[EMAIL PROTECTED]> > wrote: > > > >> Have a look at the lily library. It has code to encode Longs/Doubles > into > >> bytes such that resulting bytes sort as expected based on the numbers. > >> > >> -- Lars > >> > >> > >> > >> ________________________________ > >> From: Jonathan Bishop <[EMAIL PROTECTED]> > >> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > >> Sent: Monday, November 5, 2012 10:41 AM > >> Subject: Using doubles and longs as ordering row values > >> > >> Hi, > >> > >> In my application my row values are doubles. I would like my scans to > >> traverse the rows in order of increasing values. > >> > >> But if I simply use > >> > >> double d = .... > >> byte[] row = Bytes.toBytes(d); > >> > >> I will get an ordering which is based on the byte values of doubles, not > >> on > >> the value of the doubles themselves. > >> > >> I see also that integer values have the same issue due to the first bit > >> being the sign bit. So negative values will come after positive values. > >> > >> Any suggestions? > >> > >> Thanks, > >> > >> Jon > >> > > > > >
-
Re: Using doubles and longs as ordering row valuesDavid Koch 2012-12-01, 15:01
Hello Lars,
Thank you. Where can I find the lily library? I can't find it on github or Google. /David On Fri, Nov 30, 2012 at 3:54 AM, lars hofhansl <[EMAIL PROTECTED]> wrote: > As I said, look at the lily library they have solved exactly that problem, > I've used that before. > > It has encodings for Ints/Longs/Floats/Doubles/BigDecimals, to encode them > such the byte array will sort according to the magnitude of the value which > includes the sign and the floating point exponent. > > It's a very common problem :) > > > -- Lars > > > > ----- Original Message ----- > From: David Koch <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Cc: > Sent: Thursday, November 29, 2012 3:00 PM > Subject: Re: Using doubles and longs as ordering row values > > Hello, > > I am having a similar issue, only I need to preserve the order of > qualifiers which are serialized signed longs - rather than row keys. The > latter is addressed by the orderly library which was mentioned above. Can > this library be re-used for my purpose? I imagine this is not an exotic > requirement so I am also interested in knowing how other people have solved > this problem. > > Thank you, > > /David > > > On Tue, Nov 6, 2012 at 6:07 AM, Jonathan Bishop <[EMAIL PROTECTED] > >wrote: > > > Thanks Dave, > > > > That looks like what I need. > > > > Jon > > > > > > On Mon, Nov 5, 2012 at 4:27 PM, Dave Latham <[EMAIL PROTECTED]> wrote: > > > > > This fork looks a bit more up to date: > > > https://github.com/ndimiduk/orderly > > > > > > On Mon, Nov 5, 2012 at 4:26 PM, Dave Latham <[EMAIL PROTECTED]> > wrote: > > > > > > > Here's a project to deal with this issue specifically. I'm not sure > of > > > > it's status: > > > > https://github.com/conikeec/orderly > > > > > > > > > > > > On Mon, Nov 5, 2012 at 4:01 PM, lars hofhansl <[EMAIL PROTECTED]> > > > wrote: > > > > > > > >> Have a look at the lily library. It has code to encode Longs/Doubles > > > into > > > >> bytes such that resulting bytes sort as expected based on the > numbers. > > > >> > > > >> -- Lars > > > >> > > > >> > > > >> > > > >> ________________________________ > > > >> From: Jonathan Bishop <[EMAIL PROTECTED]> > > > >> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > > > >> Sent: Monday, November 5, 2012 10:41 AM > > > >> Subject: Using doubles and longs as ordering row values > > > >> > > > >> Hi, > > > >> > > > >> In my application my row values are doubles. I would like my scans > to > > > >> traverse the rows in order of increasing values. > > > >> > > > >> But if I simply use > > > >> > > > >> double d = .... > > > >> byte[] row = Bytes.toBytes(d); > > > >> > > > >> I will get an ordering which is based on the byte values of doubles, > > not > > > >> on > > > >> the value of the doubles themselves. > > > >> > > > >> I see also that integer values have the same issue due to the first > > bit > > > >> being the sign bit. So negative values will come after positive > > values. > > > >> > > > >> Any suggestions? > > > >> > > > >> Thanks, > > > >> > > > >> Jon > > > >> > > > > > > > > > > > > > > >
-
Re: Using doubles and longs as ordering row valuesMarcos Ortiz 2012-12-01, 15:27
On 12/01/2012 10:01 AM, David Koch wrote: > Hello Lars, > > Thank you. Where can I find the lily library? I can't find it on github or > Google. http://www.lilyproject.org/lily/index.html > > /David > > > On Fri, Nov 30, 2012 at 3:54 AM, lars hofhansl <[EMAIL PROTECTED]> wrote: > >> As I said, look at the lily library they have solved exactly that problem, >> I've used that before. >> >> It has encodings for Ints/Longs/Floats/Doubles/BigDecimals, to encode them >> such the byte array will sort according to the magnitude of the value which >> includes the sign and the floating point exponent. >> >> It's a very common problem :) >> >> >> -- Lars >> >> >> >> ----- Original Message ----- >> From: David Koch <[EMAIL PROTECTED]> >> To: [EMAIL PROTECTED] >> Cc: >> Sent: Thursday, November 29, 2012 3:00 PM >> Subject: Re: Using doubles and longs as ordering row values >> >> Hello, >> >> I am having a similar issue, only I need to preserve the order of >> qualifiers which are serialized signed longs - rather than row keys. The >> latter is addressed by the orderly library which was mentioned above. Can >> this library be re-used for my purpose? I imagine this is not an exotic >> requirement so I am also interested in knowing how other people have solved >> this problem. >> >> Thank you, >> >> /David >> >> >> On Tue, Nov 6, 2012 at 6:07 AM, Jonathan Bishop <[EMAIL PROTECTED] >>> wrote: >>> Thanks Dave, >>> >>> That looks like what I need. >>> >>> Jon >>> >>> >>> On Mon, Nov 5, 2012 at 4:27 PM, Dave Latham <[EMAIL PROTECTED]> wrote: >>> >>>> This fork looks a bit more up to date: >>>> https://github.com/ndimiduk/orderly >>>> >>>> On Mon, Nov 5, 2012 at 4:26 PM, Dave Latham <[EMAIL PROTECTED]> >> wrote: >>>>> Here's a project to deal with this issue specifically. I'm not sure >> of >>>>> it's status: >>>>> https://github.com/conikeec/orderly >>>>> >>>>> >>>>> On Mon, Nov 5, 2012 at 4:01 PM, lars hofhansl <[EMAIL PROTECTED]> >>>> wrote: >>>>>> Have a look at the lily library. It has code to encode Longs/Doubles >>>> into >>>>>> bytes such that resulting bytes sort as expected based on the >> numbers. >>>>>> -- Lars >>>>>> >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> From: Jonathan Bishop <[EMAIL PROTECTED]> >>>>>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> >>>>>> Sent: Monday, November 5, 2012 10:41 AM >>>>>> Subject: Using doubles and longs as ordering row values >>>>>> >>>>>> Hi, >>>>>> >>>>>> In my application my row values are doubles. I would like my scans >> to >>>>>> traverse the rows in order of increasing values. >>>>>> >>>>>> But if I simply use >>>>>> >>>>>> double d = .... >>>>>> byte[] row = Bytes.toBytes(d); >>>>>> >>>>>> I will get an ordering which is based on the byte values of doubles, >>> not >>>>>> on >>>>>> the value of the doubles themselves. >>>>>> >>>>>> I see also that integer values have the same issue due to the first >>> bit >>>>>> being the sign bit. So negative values will come after positive >>> values. >>>>>> Any suggestions? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jon >>>>>> >>>>> >> > > 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS... > CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION > > http://www.uci.cu > http://www.facebook.com/universidad.uci > http://www.flickr.com/photos/universidad_uci -- Marcos Luis Ort�z Valmaseda about.me/marcosortiz <http://about.me/marcosortiz> @marcosluis2186 <http://twitter.com/marcosluis2186> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS... CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION http://www.uci.cu http://www.facebook.com/universidad.uci http://www.flickr.com/photos/universidad_uci |