Is there anything that needs to be done to upgrade from 1.3 to 1.4?

It should happen automatically when the 1.4 code starts for the first
time.  So shut down 1.3 and start 1.4.

On Mon, Jan 2, 2012 at 9:12 AM, Aaron Cordova <[EMAIL PROTECTED]> wrote:
> Is there anything that needs to be done to upgrade from 1.3 to 1.4?

awesome. is the 1.4 branch stable? if not, is trunk stable?

On Jan 3, 2012, at 11:03 AM, Keith Turner wrote:

> It should happen automatically when the 1.4 code starts for the first
> time.  So shut down 1.3 and start 1.4.
>
> On Mon, Jan 2, 2012 at 9:12 AM, Aaron Cordova <[EMAIL PROTECTED]> wrote:
>> Is there anything that needs to be done to upgrade from 1.3 to 1.4?

Also - if I have the StringSummation aggregator configured on a 1.3.5 table, when I bring up 1.4 will it continue to use that aggregator, given that there's a new combiner framework? Is there anything I need to do to keep it configured?
On Jan 3, 2012, at 11:03 AM, Keith Turner wrote:

> It should happen automatically when the 1.4 code starts for the first
> time.  So shut down 1.3 and start 1.4.
>
> On Mon, Jan 2, 2012 at 9:12 AM, Aaron Cordova <[EMAIL PROTECTED]> wrote:
>> Is there anything that needs to be done to upgrade from 1.3 to 1.4?

The 1.4 branch is pretty stable, although it's not as well tested as the
1.3 branch.

Adam
On Tue, Jan 3, 2012 at 11:04 AM, Aaron Cordova <[EMAIL PROTECTED]> wrote:

> awesome. is the 1.4 branch stable? if not, is trunk stable?
>
> On Jan 3, 2012, at 11:03 AM, Keith Turner wrote:
>
> > It should happen automatically when the 1.4 code starts for the first
> > time.  So shut down 1.3 and start 1.4.
> >
> > On Mon, Jan 2, 2012 at 9:12 AM, Aaron Cordova <[EMAIL PROTECTED]>
> wrote:
> >> Is there anything that needs to be done to upgrade from 1.3 to 1.4?
>
>

On Tuesday, January 3, 2012 11:18:00 AM, "Aaron Cordova" <[EMAIL PROTECTED]> wrote:
> Also - if I have the StringSummation aggregator configured on a 1.3.5
> table, when I bring up 1.4 will it continue to use that aggregator,
> given that there's a new combiner framework? Is there anything I need
> to do to keep it configured?

Yes, that aggregator still exists in 1.4, but it is deprecated.  I would recommend switching to the SummingCombiner at some point (org.apache.accumulo.core.iterators.user) with the "type" parameter set to "STRING".

Billie

Is it ok to disable the stringsummation aggregator and then enable the summingcombiner through the shell? Should I take the table offline first?
On Jan 3, 2012, at 11:36 AM, Billie J Rinaldi wrote:

> On Tuesday, January 3, 2012 11:18:00 AM, "Aaron Cordova" <[EMAIL PROTECTED]> wrote:
>> Also - if I have the StringSummation aggregator configured on a 1.3.5
>> table, when I bring up 1.4 will it continue to use that aggregator,
>> given that there's a new combiner framework? Is there anything I need
>> to do to keep it configured?
>
> Yes, that aggregator still exists in 1.4, but it is deprecated.  I would recommend switching to the SummingCombiner at some point (org.apache.accumulo.core.iterators.user) with the "type" parameter set to "STRING".
>
> Billie

BTW I love the label 'Combiner' instead of aggregator. I've been describing their capabilities as equivalent to MapReduce combiners to new Accumulo users, so this makes it even more clear.

Are they easier to code up too? It'd be sweet if one could basically just define a function such as

public void combine(Key k, Iterator<Value> v)

or some such simple signature
On Jan 3, 2012, at 11:36 AM, Billie J Rinaldi wrote:

> On Tuesday, January 3, 2012 11:18:00 AM, "Aaron Cordova" <[EMAIL PROTECTED]> wrote:
>> Also - if I have the StringSummation aggregator configured on a 1.3.5
>> table, when I bring up 1.4 will it continue to use that aggregator,
>> given that there's a new combiner framework? Is there anything I need
>> to do to keep it configured?
>
> Yes, that aggregator still exists in 1.4, but it is deprecated.  I would recommend switching to the SummingCombiner at some point (org.apache.accumulo.core.iterators.user) with the "type" parameter set to "STRING".
>
> Billie

On Tuesday, January 3, 2012 11:51:19 AM, "Aaron Cordova" <[EMAIL PROTECTED]> wrote:
> Is it ok to disable the stringsummation aggregator and then enable the
> summingcombiner through the shell? Should I take the table offline
> first?

If the table is quiet (i.e. not compacting), then it shouldn't be a problem to do the switch while it's online.  Queries might get unexpected results during the switch.  If you've removed the VersioningIterator from the table, then you could even make the change during compactions.  However, if there is versioning, there would be the possibility of it removing versions instead of aggregating during compactions.

Billie

Why not add the combiner (so that both are enabled) and then remove the
aggregator? The two iterators should be commutative and associative, so
results should be reasonable and compactions should be safe when both are
enabled in the tree.

Adam
On Tue, Jan 3, 2012 at 12:39 PM, Billie J Rinaldi <[EMAIL PROTECTED]
> wrote:

> On Tuesday, January 3, 2012 11:51:19 AM, "Aaron Cordova" <
> [EMAIL PROTECTED]> wrote:
> > Is it ok to disable the stringsummation aggregator and then enable the
> > summingcombiner through the shell? Should I take the table offline
> > first?
>
> If the table is quiet (i.e. not compacting), then it shouldn't be a
> problem to do the switch while it's online.  Queries might get unexpected
> results during the switch.  If you've removed the VersioningIterator from
> the table, then you could even make the change during compactions.
>  However, if there is versioning, there would be the possibility of it
> removing versions instead of aggregating during compactions.
>
> Billie
>

Oh yeah, and they're (mutually) idempotent too, right?

Adam
On Tue, Jan 3, 2012 at 12:43 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:

> Why not add the combiner (so that both are enabled) and then remove the
> aggregator? The two iterators should be commutative and associative, so
> results should be reasonable and compactions should be safe when both are
> enabled in the tree.
>
> Adam
>
>
> On Tue, Jan 3, 2012 at 12:39 PM, Billie J Rinaldi <
> [EMAIL PROTECTED]> wrote:
>
>> On Tuesday, January 3, 2012 11:51:19 AM, "Aaron Cordova" <
>> [EMAIL PROTECTED]> wrote:
>> > Is it ok to disable the stringsummation aggregator and then enable the
>> > summingcombiner through the shell? Should I take the table offline
>> > first?
>>
>> If the table is quiet (i.e. not compacting), then it shouldn't be a
>> problem to do the switch while it's online.  Queries might get unexpected
>> results during the switch.  If you've removed the VersioningIterator from
>> the table, then you could even make the change during compactions.
>>  However, if there is versioning, there would be the possibility of it
>> removing versions instead of aggregating during compactions.
>>
>> Billie
>>
>
>

yeah in this case i suppose there's not much that can go wrong. even if both combiners are turned off at some point, when the summingcombiner is turned on it should do the right thing. As long as the values that get written don't get transformed differently, i.e. as long as the functionality is really identical.

On Jan 3, 2012, at 12:45 PM, Adam Fuchs wrote:

> Oh yeah, and they're (mutually) idempotent too, right?
>
> Adam
>
>
> On Tue, Jan 3, 2012 at 12:43 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:
> Why not add the combiner (so that both are enabled) and then remove the aggregator? The two iterators should be commutative and associative, so results should be reasonable and compactions should be safe when both are enabled in the tree.
>
> Adam
>
>
> On Tue, Jan 3, 2012 at 12:39 PM, Billie J Rinaldi <[EMAIL PROTECTED]> wrote:
> On Tuesday, January 3, 2012 11:51:19 AM, "Aaron Cordova" <[EMAIL PROTECTED]> wrote:
> > Is it ok to disable the stringsummation aggregator and then enable the
> > summingcombiner through the shell? Should I take the table offline
> > first?
>
> If the table is quiet (i.e. not compacting), then it shouldn't be a problem to do the switch while it's online.  Queries might get unexpected results during the switch.  If you've removed the VersioningIterator from the table, then you could even make the change during compactions.  However, if there is versioning, there would be the possibility of it removing versions instead of aggregating during compactions.
>
> Billie
>
>

On Tuesday, January 3, 2012 11:56:50 AM, "Aaron Cordova" <[EMAIL PROTECTED]> wrote:
> BTW I love the label 'Combiner' instead of aggregator. I've been
> describing their capabilities as equivalent to MapReduce combiners to
> new Accumulo users, so this makes it even more clear.
>
> Are they easier to code up too? It'd be sweet if one could basically
> just define a function such as
>
> public void combine(Key k, Iterator<Value> v)
>
> or some such simple signature

That's basically it.  Here's the method you implement:
public Value reduce(Key k, Iterator<Value> v);

We've also done something exciting with the TypedValueCombiner<V>.  It uses an Encoder<V> which has decode and encode methods to translate from a Value's byte array to type V and back.  Then you implement this method to do the combining:
public V typedReduce(Key k, Iterator<V> v);

We provide a LongCombiner which is a TypedValueCombiner<Long> with three built-in encoding types: STRING, VARLEN, and FIXEDLEN.  Separating the encoding from the combining has made writing new combiners really easy.  See the MaxCombiner and MinCombiner in addition to the SummingCombiner.

Billie

Ahh that's awesome. So the function returns exactly one value for each key?

On Jan 3, 2012, at 12:53 PM, Billie J Rinaldi wrote:

> On Tuesday, January 3, 2012 11:56:50 AM, "Aaron Cordova" <[EMAIL PROTECTED]> wrote:
>> BTW I love the label 'Combiner' instead of aggregator. I've been
>> describing their capabilities as equivalent to MapReduce combiners to
>> new Accumulo users, so this makes it even more clear.
>>
>> Are they easier to code up too? It'd be sweet if one could basically
>> just define a function such as
>>
>> public void combine(Key k, Iterator<Value> v)
>>
>> or some such simple signature
>
> That's basically it.  Here's the method you implement:
> public Value reduce(Key k, Iterator<Value> v);
>
> We've also done something exciting with the TypedValueCombiner<V>.  It uses an Encoder<V> which has decode and encode methods to translate from a Value's byte array to type V and back.  Then you implement this method to do the combining:
> public V typedReduce(Key k, Iterator<V> v);
>
> We provide a LongCombiner which is a TypedValueCombiner<Long> with three built-in encoding types: STRING, VARLEN, and FIXEDLEN.  Separating the encoding from the combining has made writing new combiners really easy.  See the MaxCombiner and MinCombiner in addition to the SummingCombiner.
>
> Billie

On Tuesday, January 3, 2012 12:51:20 PM, "Aaron Cordova" <[EMAIL PROTECTED]> wrote:
> yeah in this case i suppose there's not much that can go wrong. even
> if both combiners are turned off at some point, when the
> summingcombiner is turned on it should do the right thing. As long as
> the values that get written don't get transformed differently, i.e. as
> long as the functionality is really identical.

I wrote a test to check the new encodings against the old ones, but if you see anything wrong do let us know.

Billie

On Tuesday, January 3, 2012 12:56:58 PM, "Aaron Cordova" <[EMAIL PROTECTED]> wrote:
> Ahh that's awesome. So the function returns exactly one value for each
> key?

Yes, just like aggregators.

Billie