Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Function To Compute Product of Values in Bag


+
Sergey Goder 2013-05-03, 18:20
Copy link to this message
-
Re: Function To Compute Product of Values in Bag
Hi,

Just a hint: It's usually better to work with log probabilites and sum
over them, than to work with raw probabilities and to use
multiplication. You might easily run into numerical accuracy issues
otherwise.

i.e. exploit this fact:

product(x1, ..., xn) = exp(sum(log(x1), ..., log(xn)))

best,

Kai Londenberg

2013/5/3 Sergey Goder <[EMAIL PROTECTED]>:
> I'm creating a multinomial naive bayes classifier using pig and need to
> compute the product of probabilities. There are an arbitrary number of
> values in the bag so I would like to be able to use a function similar to
> the builtin SUM to do this. I looked through the source code and found that
> with some really simple changes to SUM.java I can create a PROD.java
> function. I included it in my piggybank and have been using it successfully.
>
> I was curious what the community thought about including this function as a
> builtin function in a future release? Or would it make more sense to keep
> this function as a udf in a piggybank.
>
> Thanks,
> Sergey
+
Sergey Goder 2013-05-03, 20:36
+
Julien Le Dem 2013-05-03, 22:46
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB