Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Function To Compute Product of Values in Bag


Copy link to this message
-
Re: Function To Compute Product of Values in Bag
Thanks for the tip about numerical accuracy issues and the elegant solution
exploiting log/exp. It is very much appreciated.

Sergey
On Fri, May 3, 2013 at 11:42 AM, Kai Londenberg <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> Just a hint: It's usually better to work with log probabilites and sum
> over them, than to work with raw probabilities and to use
> multiplication. You might easily run into numerical accuracy issues
> otherwise.
>
> i.e. exploit this fact:
>
> product(x1, ..., xn) = exp(sum(log(x1), ..., log(xn)))
>
> best,
>
> Kai Londenberg
>
> 2013/5/3 Sergey Goder <[EMAIL PROTECTED]>:
> > I'm creating a multinomial naive bayes classifier using pig and need to
> > compute the product of probabilities. There are an arbitrary number of
> > values in the bag so I would like to be able to use a function similar to
> > the builtin SUM to do this. I looked through the source code and found
> that
> > with some really simple changes to SUM.java I can create a PROD.java
> > function. I included it in my piggybank and have been using it
> successfully.
> >
> > I was curious what the community thought about including this function
> as a
> > builtin function in a future release? Or would it make more sense to keep
> > this function as a udf in a piggybank.
> >
> > Thanks,
> > Sergey
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB