We are trying to write some of the UDF to do some machine learning work. We
did a simple experiment to calculate the AUC through a UDF like the
following code in gist
The map-reduce job will only take a couple of few minutes, but will wait
there hours to do the cleanup.
I guess the reason is that the sort inside the foreach will generate lots
of data spill to local fs and takes a long time to do cleanup there.
In a java map-reduce problem, we could made it like a secondary sort. We
make the model + ctr as the key so the same model's ctr will be sorted, and
group by only the model name part, then the sort is done after shuffling.
I am wondering if we could do that kind of optimization in pig as well?