Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Possible to override the context.write() method in ReduceContext?


+
Ross Nordeen 2011-08-12, 21:06
Copy link to this message
-
Re: Possible to override the context.write() method in ReduceContext?
Hi Ross,

The first thing that I should do is to create one main reducer and create
writeToContext method in it. This method should handle your logic with
if(...){...} and context.write(). All other reducers would extend this main
reducer and call  writeToContext instead context.write.

Thanks,
dino
On Fri, Aug 12, 2011 at 11:06 PM, Ross Nordeen <[EMAIL PROTECTED]> wrote:

> Using 0.20.2...
>
> Is it possible to override the context.write() method in ReduceContext? I
> have an entire set of Reducers that I would like to all use a specific
> function just before every context.write() but I don't want them to worry
> about that logic, just to have it handled transparently.
>
> For instance:
>
> Iterator<Text> vit = values.iterator();
>
> if (trans2 != null) {
>    key = (Text) trans2.transform(key);
> }
>
> while (vit.hasNext()) {
>    Text item = vit.next();
>    if (trans1 != null) {
>        item = (Text) trans1.transform(item);
>    }
>    context.write(key, item);
> }
>
> The logic before/after the write is often different, and there may be
> writes that happen in different cases. I want to move the if->transform
> functionality out to the write() function so I can just call
> context.write(a,b) instead of if (trans1 !=....
>
> My end-goal is something like this:
>
> protected void reduce(Text key, Iterable<Text> values, Context context)
>        throws IOException, InterruptedException {
>    Iterator<Text> vit = values.iterator();
>
>    while (vit.hasNext())
>        context.write(key, vit.next());
> }
>
> With the write() method override:
>
> public void write(Text key, Text val) {
>    if (trans1 != null)
>        val = trans1.transform(val);
>    if (trans2 != null)
>        key = trans2.transform(key);
>    super.write(key,val);
> }
>
> or something similar.
>
> Thanks in advance,
>
> -Ross
>
> --
> Ross Nordeen
> Computer Networking And Systems Administration
> Michigan Technological University
> http://www.linkedin.com/in/rjnordee
>
>
+
Ross Nordeen 2011-08-15, 16:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB