Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - setInt & getInt


Copy link to this message
-
Re: setInt & getInt
Joey Echeverria 2011-10-04, 16:31
The Job class copies the Configuraiton that you pass in. You either
need to do your conf.setInt("number", 12345) before you create the Job
object or you need call job.getConfiguration().setInt("number",
12345).

-Joey

On Tue, Oct 4, 2011 at 12:28 PM, Ratner, Alan S (IS)
<[EMAIL PROTECTED]> wrote:
> I have no problem with Hadoop.mapred using JobConf to setInt integers and pass them to my map(s) for getInt as shown in the first program below.  However, when I use Hadoop.mapreduce using Configuration to setInt these values are invisible to my map's getInt's.  Please tell me what I am doing wrong.  Thanks.
>
> Both programs expect to see a file with a line or two of text in a directory named testIn.
>
> Alan Ratner
>
>
> This program uses JobConf and setInt/getInt and works fine.  It outputs:
> number = 12345 (from map)
>
>
> package cbTest;
>
> import java.io.*;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.util.*;
>
> public class ConfTest extends Configured implements Tool {
>
>            @SuppressWarnings("deprecation")
>            public static class MapClass extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> {
>                        public static int number;
>
>                        public void configure(JobConf job) {
>                                    number = job.getInt("number",-999);
>                        }
>
>                        public void map(LongWritable key, Text t, OutputCollector<Text, Text> output,
>                                                Reporter reporter) throws IOException {
>                                    System.out.println("number = " + number);
>                        }
>            }
>
>            @SuppressWarnings("deprecation")
>            public int run(String[] args) throws Exception {
>                        Path InputDirectory = new Path("testIn");
>                        Path OutputDirectory = new Path("testOut");
>                        System.out.println(">>>> Running ConfTest Program");
>                        JobConf conf = new JobConf(getConf(), ConfTest.class);
>                        conf.setInputFormat(TextInputFormat.class);
>                        conf.setOutputKeyClass(Text.class);
>                        conf.setOutputValueClass(Text.class);
>                        conf.setMapperClass(MapClass.class);
>                        conf.setInt("number", 12345);
>                        FileInputFormat.setInputPaths(conf, InputDirectory);
>                        FileOutputFormat.setOutputPath(conf, OutputDirectory);
>                        FileSystem fs = OutputDirectory.getFileSystem(conf);
>                        fs.delete(OutputDirectory, true); //remove output of prior run
>                        JobClient.runJob(conf);
>                        return 0;
>            }
>
>            public static void main(String[] args) throws Exception {
>                        int res = ToolRunner.run(new Configuration(), new ConfTest(), args);
>                        System.exit(res);
>            }
> }
>
>
> This program uses Configuration and setInt/getInt.  Butt getInt in neither map or map:configure works.  It outputs:
>>>>>> Passing integer 12345 in configuration <<<<< (from run)
> map numbers: -999 -1 (from map as intMapConf and intConfConf)
>
>
> package cbTest;
>
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.conf.Configured;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
> import org.apache.hadoop.util.Tool;

Joseph Echeverria
Cloudera, Inc.
443.305.9434