Hi,

 

I am a newbie of hadoop.

Today I was struggling with a hadoop problem for several hours.

 

I initialize a parameter by setting job configuration in main.

E.g. Configuration con = new Configuration();

con.set("test", "1");

Job job = new Job(con);

 

Then in the mapper class, I want to set "test" to "2". I did it by

context.getConfiguration().set("test","2");

 

Finally in the main method, after the job is finished, I check the
"test" again by

job.getConfiguration().get("test");

 

However, the value of "test" is still "1".

 

The reason why I want to change the parameter inside Mapper class is
that I want to determine when to stop an iteration in the main method.
For example, for doing breadth-first search, when there is no new nodes
are added for further expansion, the searching iteration should stop.

 

Your help will be deeply appreciated. Thank you

 

Wei

You can use hadoop counter to pass this information.
This way, you see the counters in job report.

On Thu, Dec 16, 2010 at 10:58 PM, Peng, Wei <[EMAIL PROTECTED]> wrote:

> Hi,
>
>
>
> I am a newbie of hadoop.
>
> Today I was struggling with a hadoop problem for several hours.
>
>
>
> I initialize a parameter by setting job configuration in main.
>
> E.g. Configuration con = new Configuration();
>
> con.set("test", "1");
>
> Job job = new Job(con);
>
>
>
> Then in the mapper class, I want to set "test" to "2". I did it by
>
> context.getConfiguration().set("test","2");
>
>
>
> Finally in the main method, after the job is finished, I check the
> "test" again by
>
> job.getConfiguration().get("test");
>
>
>
> However, the value of "test" is still "1".
>
>
>
> The reason why I want to change the parameter inside Mapper class is
> that I want to determine when to stop an iteration in the main method.
> For example, for doing breadth-first search, when there is no new nodes
> are added for further expansion, the searching iteration should stop.
>
>
>
> Your help will be deeply appreciated. Thank you
>
>
>
> Wei
>
>

If you're wondering where to get a counter from, I'll point you to "context"

Sent from my mobile. Please excuse the typos.

On 2010-12-17, at 7:39 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> You can use hadoop counter to pass this information.
> This way, you see the counters in job report.
>
> On Thu, Dec 16, 2010 at 10:58 PM, Peng, Wei <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>>
>>
>> I am a newbie of hadoop.
>>
>> Today I was struggling with a hadoop problem for several hours.
>>
>>
>>
>> I initialize a parameter by setting job configuration in main.
>>
>> E.g. Configuration con = new Configuration();
>>
>> con.set("test", "1");
>>
>> Job job = new Job(con);
>>
>>
>>
>> Then in the mapper class, I want to set "test" to "2". I did it by
>>
>> context.getConfiguration().set("test","2");
>>
>>
>>
>> Finally in the main method, after the job is finished, I check the
>> "test" again by
>>
>> job.getConfiguration().get("test");
>>
>>
>>
>> However, the value of "test" is still "1".
>>
>>
>>
>> The reason why I want to change the parameter inside Mapper class is
>> that I want to determine when to stop an iteration in the main method.
>> For example, for doing breadth-first search, when there is no new nodes
>> are added for further expansion, the searching iteration should stop.
>>
>>
>>
>> Your help will be deeply appreciated. Thank you
>>
>>
>>
>> Wei
>>
>>

I did something like this using a global static boolean variable  
(flag) while I was implementing breadth first IDA*. In my case, I set  
the flag to something else if a solution was found, which was examined  
in the reducer.

I guess in your case, since you know that if the mappers don't produce  
anything the reducers won't have anything as input, if I am not wrong.

And I had chaining map-reduce jobs ( http://developer.yahoo.com/hadoop/tutorial/module4.html
  ) running until a solution was found.
Kind regards,

Arindam Khaled

On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:

> Hi,
>
>
>
> I am a newbie of hadoop.
>
> Today I was struggling with a hadoop problem for several hours.
>
>
>
> I initialize a parameter by setting job configuration in main.
>
> E.g. Configuration con = new Configuration();
>
> con.set("test", "1");
>
> Job job = new Job(con);
>
>
>
> Then in the mapper class, I want to set "test" to "2". I did it by
>
> context.getConfiguration().set("test","2");
>
>
>
> Finally in the main method, after the job is finished, I check the
> "test" again by
>
> job.getConfiguration().get("test");
>
>
>
> However, the value of "test" is still "1".
>
>
>
> The reason why I want to change the parameter inside Mapper class is
> that I want to determine when to stop an iteration in the main method.
> For example, for doing breadth-first search, when there is no new  
> nodes
> are added for further expansion, the searching iteration should stop.
>
>
>
> Your help will be deeply appreciated. Thank you
>
>
>
> Wei
>

Arindam, how to set this global static Boolean variable?
I have tried to do something similarly yesterday in the following:
Public class BFSearch
{
Private static boolean expansion;
Public static class MapperClass {if no nodes expansion = false;}
Public static class ReducerClass
Public static void main {expansion = true; run job;
print(expansion)}
}
In this case, expansion is still true.
I will look at hadoop counter and report back here later.

Thank you for all your help
Wei

-----Original Message-----
From: Arindam Khaled [mailto:[EMAIL PROTECTED]]
Sent: Friday, December 17, 2010 10:35 AM
To: [EMAIL PROTECTED]
Subject: Re: Please help with hadoop configuration parameter set and get

I did something like this using a global static boolean variable  
(flag) while I was implementing breadth first IDA*. In my case, I set  
the flag to something else if a solution was found, which was examined  
in the reducer.

I guess in your case, since you know that if the mappers don't produce  
anything the reducers won't have anything as input, if I am not wrong.

And I had chaining map-reduce jobs (
http://developer.yahoo.com/hadoop/tutorial/module4.html
  ) running until a solution was found.
Kind regards,

Arindam Khaled

On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:

> Hi,
>
>
>
> I am a newbie of hadoop.
>
> Today I was struggling with a hadoop problem for several hours.
>
>
>
> I initialize a parameter by setting job configuration in main.
>
> E.g. Configuration con = new Configuration();
>
> con.set("test", "1");
>
> Job job = new Job(con);
>
>
>
> Then in the mapper class, I want to set "test" to "2". I did it by
>
> context.getConfiguration().set("test","2");
>
>
>
> Finally in the main method, after the job is finished, I check the
> "test" again by
>
> job.getConfiguration().get("test");
>
>
>
> However, the value of "test" is still "1".
>
>
>
> The reason why I want to change the parameter inside Mapper class is
> that I want to determine when to stop an iteration in the main method.
> For example, for doing breadth-first search, when there is no new  
> nodes
> are added for further expansion, the searching iteration should stop.
>
>
>
> Your help will be deeply appreciated. Thank you
>
>
>
> Wei
>

Statics won't work the way you might think because different mappers and
different reducers are all running in different JVM's.  It might work in
local mode, but don't kid yourself about it working in a distributed mode.
 It won't.

On Fri, Dec 17, 2010 at 8:31 AM, Peng, Wei <[EMAIL PROTECTED]> wrote:

> Arindam, how to set this global static Boolean variable?
> I have tried to do something similarly yesterday in the following:
> Public class BFSearch
> {
>        Private static boolean expansion;
>        Public static class MapperClass {if no nodes expansion = false;}
>        Public static class ReducerClass
>        Public static void main {expansion = true; run job;
> print(expansion)}
> }
> In this case, expansion is still true.
> I will look at hadoop counter and report back here later.
>
> Thank you for all your help
> Wei
>
> -----Original Message-----
> From: Arindam Khaled [mailto:[EMAIL PROTECTED]]
> Sent: Friday, December 17, 2010 10:35 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Please help with hadoop configuration parameter set and get
>
> I did something like this using a global static boolean variable
> (flag) while I was implementing breadth first IDA*. In my case, I set
> the flag to something else if a solution was found, which was examined
> in the reducer.
>
> I guess in your case, since you know that if the mappers don't produce
> anything the reducers won't have anything as input, if I am not wrong.
>
> And I had chaining map-reduce jobs (
> http://developer.yahoo.com/hadoop/tutorial/module4.html
>  ) running until a solution was found.
>
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
>
> > Hi,
> >
> >
> >
> > I am a newbie of hadoop.
> >
> > Today I was struggling with a hadoop problem for several hours.
> >
> >
> >
> > I initialize a parameter by setting job configuration in main.
> >
> > E.g. Configuration con = new Configuration();
> >
> > con.set("test", "1");
> >
> > Job job = new Job(con);
> >
> >
> >
> > Then in the mapper class, I want to set "test" to "2". I did it by
> >
> > context.getConfiguration().set("test","2");
> >
> >
> >
> > Finally in the main method, after the job is finished, I check the
> > "test" again by
> >
> > job.getConfiguration().get("test");
> >
> >
> >
> > However, the value of "test" is still "1".
> >
> >
> >
> > The reason why I want to change the parameter inside Mapper class is
> > that I want to determine when to stop an iteration in the main method.
> > For example, for doing breadth-first search, when there is no new
> > nodes
> > are added for further expansion, the searching iteration should stop.
> >
> >
> >
> > Your help will be deeply appreciated. Thank you
> >
> >
> >
> > Wei
> >
>
>

Exactly.  It is hard to assume anything about order or coordination between maps or reducers.  You should try and design with “sloppy” coordination strategies.

James.

On 2010-12-17, at 10:32 AM, Ted Dunning wrote:

> Statics won't work the way you might think because different mappers and
> different reducers are all running in different JVM's.  It might work in
> local mode, but don't kid yourself about it working in a distributed mode.
> It won't.
>
> On Fri, Dec 17, 2010 at 8:31 AM, Peng, Wei <[EMAIL PROTECTED]> wrote:
>
>> Arindam, how to set this global static Boolean variable?
>> I have tried to do something similarly yesterday in the following:
>> Public class BFSearch
>> {
>>       Private static boolean expansion;
>>       Public static class MapperClass {if no nodes expansion = false;}
>>       Public static class ReducerClass
>>       Public static void main {expansion = true; run job;
>> print(expansion)}
>> }
>> In this case, expansion is still true.
>> I will look at hadoop counter and report back here later.
>>
>> Thank you for all your help
>> Wei
>>
>> -----Original Message-----
>> From: Arindam Khaled [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, December 17, 2010 10:35 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Please help with hadoop configuration parameter set and get
>>
>> I did something like this using a global static boolean variable
>> (flag) while I was implementing breadth first IDA*. In my case, I set
>> the flag to something else if a solution was found, which was examined
>> in the reducer.
>>
>> I guess in your case, since you know that if the mappers don't produce
>> anything the reducers won't have anything as input, if I am not wrong.
>>
>> And I had chaining map-reduce jobs (
>> http://developer.yahoo.com/hadoop/tutorial/module4.html
>> ) running until a solution was found.
>>
>>
>> Kind regards,
>>
>> Arindam Khaled
>>
>>
>>
>>
>>
>> On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I am a newbie of hadoop.
>>>
>>> Today I was struggling with a hadoop problem for several hours.
>>>
>>>
>>>
>>> I initialize a parameter by setting job configuration in main.
>>>
>>> E.g. Configuration con = new Configuration();
>>>
>>> con.set("test", "1");
>>>
>>> Job job = new Job(con);
>>>
>>>
>>>
>>> Then in the mapper class, I want to set "test" to "2". I did it by
>>>
>>> context.getConfiguration().set("test","2");
>>>
>>>
>>>
>>> Finally in the main method, after the job is finished, I check the
>>> "test" again by
>>>
>>> job.getConfiguration().get("test");
>>>
>>>
>>>
>>> However, the value of "test" is still "1".
>>>
>>>
>>>
>>> The reason why I want to change the parameter inside Mapper class is
>>> that I want to determine when to stop an iteration in the main method.
>>> For example, for doing breadth-first search, when there is no new
>>> nodes
>>> are added for further expansion, the searching iteration should stop.
>>>
>>>
>>>
>>> Your help will be deeply appreciated. Thank you
>>>
>>>
>>>
>>> Wei
>>>
>>
>>

Wei,

I implemented one of the algorithms, BFIDA*, included the paper "Out-
of-Core Parallel Frontier Search with MapReduce" by Alexander  
Reinefeld and Thorsten Sch{\"u}tt. FYI, they implemented BFS for 15-
puzzle using map-reduce and MPI, not Hadoop.

For brevity, I am not including the whole source code. Here is my  
pseudo-code:

public class standAloneIDA {

static boolean solved = false;
public static class BFIDAMapClass extends Mapper<Object, Text, Text,  
IntWritable>
{
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException
{
String line = value.toString();
while(there can be more new moves)
{
emit("the new board", "move"); //key, value pair
}
}

}

public static class BFIDAReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
String line = "";

ArrayList<Integer> previousMoves = new ArrayList<Integer> ();

for(IntWritable val: values)
{
String valInt = val.toString();
int[] moves = stringToArrayInt(valInt);

for(int bit: moves)
{
if(!previousMoves.contains(bit))
previousMoves.add(bit);
}
}

for(int val: previousMoves)
{
line = line + Integer.toString(val);
}

if (isSolved(key.toString(), size))
{
solved = true;
}

result.set(Integer.parseInt(line));
context.write(key, result);
}
}

public static void main(String[] args) throws Exception {
// logger.isDebugEnabled();
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf,  
args).getRemainingArgs();
if (otherArgs.length != 2) {
System.err.println("Usage: testUnit2 <in> <out>");
System.exit(2);
}



while(!solved)
{
set up the file system
include the input and output filenames
Job job = new Job(conf, "some name");
job.setJarByClass(standAloneIDA.class);
job.setMapperClass(BFIDAMapClass.class);
job.setReducerClass(BFIDAReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);

job.waitForCompletion(true);
}
}
}

Please excuse me if there are missing braces. There might be more  
efficient ways to setup the jobs and file system. I didn't have much  
time -- so, I ended up with something that worked for me then. Let me  
know if you have more questions.

Kind regards,

Arindam Khaled

On Dec 17, 2010, at 10:31 AM, Peng, Wei wrote:

> Arindam, how to set this global static Boolean variable?
> I have tried to do something similarly yesterday in the following:
> Public class BFSearch
> {
> Private static boolean expansion;
> Public static class MapperClass {if no nodes expansion = false;}
> Public static class ReducerClass
> Public static void main {expansion = true; run job;
> print(expansion)}
> }
> In this case, expansion is still true.
> I will look at hadoop counter and report back here later.
>
> Thank you for all your help
> Wei
>
> -----Original Message-----
> From: Arindam Khaled [mailto:[EMAIL PROTECTED]]
> Sent: Friday, December 17, 2010 10:35 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Please help with hadoop configuration parameter set and  
> get
>
> I did something like this using a global static boolean variable
> (flag) while I was implementing breadth first IDA*. In my case, I set
> the flag to something else if a solution was found, which was examined
> in the reducer.
>
> I guess in your case, since you know that if the mappers don't produce
> anything the reducers won't have anything as input, if I am not wrong.
>
> And I had chaining map-reduce jobs (
> http://developer.yahoo.com/hadoop/tutorial/module4.html
>  ) running until a solution was found.
>
>
> Kind regards,
>
> Arindam Khaled

There's no guarantee that it might work for a distributed environment  
as a couple of developers suggested. I didn't have the time to play  
with it in the distributed mode, rather, executed it in a standalone  
environment.

Arindam

On Dec 17, 2010, at 12:07 PM, Arindam Khaled wrote:

> Wei,
>
> I implemented one of the algorithms, BFIDA*, included the paper "Out-
> of-Core Parallel Frontier Search with MapReduce" by Alexander  
> Reinefeld and Thorsten Sch{\"u}tt. FYI, they implemented BFS for 15-
> puzzle using map-reduce and MPI, not Hadoop.
>
> For brevity, I am not including the whole source code. Here is my  
> pseudo-code:
>
> public class standAloneIDA {
>
> static boolean solved = false;
> public static class BFIDAMapClass extends Mapper<Object, Text,  
> Text, IntWritable>
> {
> public void map(Object key, Text value, Context context)
> throws IOException, InterruptedException
> {
> String line = value.toString();
> while(there can be more new moves)
> {
> emit("the new board", "move"); //key, value pair
> }
> }
>
> }
>
> public static class BFIDAReducer
> extends Reducer<Text,IntWritable,Text,IntWritable> {
> private IntWritable result = new IntWritable();
>
> public void reduce(Text key, Iterable<IntWritable> values,
> Context context
> ) throws IOException, InterruptedException {
> String line = "";
>
> ArrayList<Integer> previousMoves = new ArrayList<Integer> ();
>
> for(IntWritable val: values)
> {
> String valInt = val.toString();
> int[] moves = stringToArrayInt(valInt);
>
> for(int bit: moves)
> {
> if(!previousMoves.contains(bit))
> previousMoves.add(bit);
> }
> }
>
> for(int val: previousMoves)
> {
> line = line + Integer.toString(val);
> }
>
> if (isSolved(key.toString(), size))
> {
> solved = true;
> }
>
> result.set(Integer.parseInt(line));
> context.write(key, result);
> }
> }
>
> public static void main(String[] args) throws Exception {
> // logger.isDebugEnabled();
> Configuration conf = new Configuration();
> String[] otherArgs = new GenericOptionsParser(conf,  
> args).getRemainingArgs();
> if (otherArgs.length != 2) {
> System.err.println("Usage: testUnit2 <in> <out>");
> System.exit(2);
> }
>
>
>
> while(!solved)
> {
> set up the file system
> include the input and output filenames
> Job job = new Job(conf, "some name");
> job.setJarByClass(standAloneIDA.class);
> job.setMapperClass(BFIDAMapClass.class);
> job.setReducerClass(BFIDAReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(IntWritable.class);
>
> job.waitForCompletion(true);
> }
> }
> }
>
> Please excuse me if there are missing braces. There might be more  
> efficient ways to setup the jobs and file system. I didn't have much  
> time -- so, I ended up with something that worked for me then. Let  
> me know if you have more questions.
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 10:31 AM, Peng, Wei wrote:
>
>> Arindam, how to set this global static Boolean variable?
>> I have tried to do something similarly yesterday in the following:
>> Public class BFSearch
>> {
>> Private static boolean expansion;
>> Public static class MapperClass {if no nodes expansion = false;}
>> Public static class ReducerClass
>> Public static void main {expansion = true; run job;
>> print(expansion)}
>> }
>> In this case, expansion is still true.
>> I will look at hadoop counter and report back here later.
>>
>> Thank you for all your help
>> Wei
>>
>> -----Original Message-----
>> From: Arindam Khaled [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, December 17, 2010 10:35 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Please help with hadoop configuration parameter set  
>> and get
>>
>> I did something like this using a global static boolean variable

Thank you for your kind reply.
I am running hadoop in the distributed mode.

It might be out of the topic.
For a lot of algorithms (especially data mining/machine learning
algorithms), we do need many iterations. However, hadoop needs to save
all the intermediate step output as the input of the next iteration. I
doubt it would be efficient.
Is hadoop the right choice for data mining/machine learning algorithm?
Can hadoop go with gpu computing?

Wei

-----Original Message-----
From: Arindam Khaled [mailto:[EMAIL PROTECTED]]
Sent: Friday, December 17, 2010 1:14 PM
To: [EMAIL PROTECTED]
Subject: Re: Please help with hadoop configuration parameter set and get

There's no guarantee that it might work for a distributed environment  
as a couple of developers suggested. I didn't have the time to play  
with it in the distributed mode, rather, executed it in a standalone  
environment.

Arindam

On Dec 17, 2010, at 12:07 PM, Arindam Khaled wrote:

> Wei,
>
> I implemented one of the algorithms, BFIDA*, included the paper "Out-
> of-Core Parallel Frontier Search with MapReduce" by Alexander  
> Reinefeld and Thorsten Sch{\"u}tt. FYI, they implemented BFS for 15-
> puzzle using map-reduce and MPI, not Hadoop.
>
> For brevity, I am not including the whole source code. Here is my  
> pseudo-code:
>
> public class standAloneIDA {
>
> static boolean solved = false;
> public static class BFIDAMapClass extends Mapper<Object, Text,  
> Text, IntWritable>
> {
> public void map(Object key, Text value, Context context)
> throws IOException, InterruptedException
> {
> String line = value.toString();
> while(there can be more new moves)
> {
> emit("the new board", "move"); //key,
value pair
> }
> }
>
> }
>
> public static class BFIDAReducer
> extends Reducer<Text,IntWritable,Text,IntWritable> {
> private IntWritable result = new IntWritable();
>
> public void reduce(Text key, Iterable<IntWritable>
values,
> Context context
> ) throws IOException, InterruptedException {
> String line = "";
>
> ArrayList<Integer> previousMoves = new
ArrayList<Integer> ();
>
> for(IntWritable val: values)
> {
> String valInt = val.toString();
> int[] moves = stringToArrayInt(valInt);
>
> for(int bit: moves)
> {
> if(!previousMoves.contains(bit))
> previousMoves.add(bit);
> }
> }
>
> for(int val: previousMoves)
> {
> line = line + Integer.toString(val);
> }
>
> if (isSolved(key.toString(), size))
> {
> solved = true;
> }
>
> result.set(Integer.parseInt(line));
> context.write(key, result);
> }
> }
>
> public static void main(String[] args) throws Exception {
> // logger.isDebugEnabled();
> Configuration conf = new Configuration();
> String[] otherArgs = new GenericOptionsParser(conf,  
> args).getRemainingArgs();
> if (otherArgs.length != 2) {
> System.err.println("Usage: testUnit2 <in>
<out>");
> System.exit(2);
> }
>
>
>
> while(!solved)
> {
> set up the file system
> include the input and output filenames
> Job job = new Job(conf, "some name");
> job.setJarByClass(standAloneIDA.class);
> job.setMapperClass(BFIDAMapClass.class);
> job.setReducerClass(BFIDAReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(IntWritable.class);
>
> job.waitForCompletion(true);
> }
> }
> }
>
> Please excuse me if there are missing braces. There might be more  
> efficient ways to setup the jobs and file system. I didn't have much  
> time -- so, I ended up with something that worked for me then. Let  
> me know if you have more questions.
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 10:31 AM, Peng, Wei wrote:
>
>> Arindam, how to set this global static Boolean variable?
>> I have tried to do something similarly yesterday in the following:

I just tried hadoop counters. It works great now!!

I have one more question.
Once there is no new node is found, I would like to exit mapper or reducer process.
For example, if no new nodes are found in the mapper class, the setup of reducer will stop reducer work (or the cleanup of mapper will stop the reducer work).

Wei

-----Original Message-----
From: Arindam Khaled [mailto:[EMAIL PROTECTED]]
Sent: Fri 12/17/2010 10:13 AM
To: [EMAIL PROTECTED]
Subject: Re: Please help with hadoop configuration parameter set and get
 
There's no guarantee that it might work for a distributed environment  
as a couple of developers suggested. I didn't have the time to play  
with it in the distributed mode, rather, executed it in a standalone  
environment.

Arindam

On Dec 17, 2010, at 12:07 PM, Arindam Khaled wrote:

> Wei,
>
> I implemented one of the algorithms, BFIDA*, included the paper "Out-
> of-Core Parallel Frontier Search with MapReduce" by Alexander  
> Reinefeld and Thorsten Sch{\"u}tt. FYI, they implemented BFS for 15-
> puzzle using map-reduce and MPI, not Hadoop.
>
> For brevity, I am not including the whole source code. Here is my  
> pseudo-code:
>
> public class standAloneIDA {
>
> static boolean solved = false;
> public static class BFIDAMapClass extends Mapper<Object, Text,  
> Text, IntWritable>
> {
> public void map(Object key, Text value, Context context)
> throws IOException, InterruptedException
> {
> String line = value.toString();
> while(there can be more new moves)
> {
> emit("the new board", "move"); //key, value pair
> }
> }
>
> }
>
> public static class BFIDAReducer
> extends Reducer<Text,IntWritable,Text,IntWritable> {
> private IntWritable result = new IntWritable();
>
> public void reduce(Text key, Iterable<IntWritable> values,
> Context context
> ) throws IOException, InterruptedException {
> String line = "";
>
> ArrayList<Integer> previousMoves = new ArrayList<Integer> ();
>
> for(IntWritable val: values)
> {
> String valInt = val.toString();
> int[] moves = stringToArrayInt(valInt);
>
> for(int bit: moves)
> {
> if(!previousMoves.contains(bit))
> previousMoves.add(bit);
> }
> }
>
> for(int val: previousMoves)
> {
> line = line + Integer.toString(val);
> }
>
> if (isSolved(key.toString(), size))
> {
> solved = true;
> }
>
> result.set(Integer.parseInt(line));
> context.write(key, result);
> }
> }
>
> public static void main(String[] args) throws Exception {
> // logger.isDebugEnabled();
> Configuration conf = new Configuration();
> String[] otherArgs = new GenericOptionsParser(conf,  
> args).getRemainingArgs();
> if (otherArgs.length != 2) {
> System.err.println("Usage: testUnit2 <in> <out>");
> System.exit(2);
> }
>
>
>
> while(!solved)
> {
> set up the file system
> include the input and output filenames
> Job job = new Job(conf, "some name");
> job.setJarByClass(standAloneIDA.class);
> job.setMapperClass(BFIDAMapClass.class);
> job.setReducerClass(BFIDAReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(IntWritable.class);
>
> job.waitForCompletion(true);
> }
> }
> }
>
> Please excuse me if there are missing braces. There might be more  
> efficient ways to setup the jobs and file system. I didn't have much  
> time -- so, I ended up with something that worked for me then. Let  
> me know if you have more questions.
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 10:31 AM, Peng, Wei wrote:
>
>> Arindam, how to set this global static Boolean variable?
>> I have tried to do something similarly yesterday in the following:
>> Public class BFSearch
>> {
>> Private static boolean expansion;
>> Public static class MapperClass {if no nodes expansion = false;}

Pei
You cannot change configuration in Mapper method and see it in main method.
There are 2 ways to do this
1. Have your mapper store data in HDFS
2. Use counters

I recommend using the 2nd approach. What you can do is set a counter in the
Mapper, something like
context.getCounter("MyCounter").increment(1);
and then extract this counter in the main method. I had this class that
emits out all counters which I then load into a DB. You can tweak this class
to get the value of the counter you want.

 public class JobMetadata {
    Job job = null;

    public JobMetadata(Job job) {
        this.job = job;
    }
    public String getJobMetadata() throws IOException {
        /**
         * Get Job Name, Id and URL
         */
        String jobName = job.getJobName();
        String trackingUrl = job.getTrackingURL();
        /**
         * Get Job Counter
         */
        Counters jobCounters = job.getCounters();
        int totalCounters = jobCounters.countCounters();
        CounterGroup counterGroup = null;
        Counter counter = null;

        String counterGroupName = null;
        String counterName = null;
        long counterValue = 0;

        Iterator itGroups = jobCounters.iterator();
        StringBuffer sb = new StringBuffer();
        while (itGroups.hasNext()) {
            counterGroup = (CounterGroup) itGroups.next();
            counterGroupName = counterGroup.getDisplayName();

            Iterator itCounter = counterGroup.iterator();
            while (itCounter.hasNext()) {
                counter = (Counter) itCounter.next();
                counterName = counter.getDisplayName();
                counterValue = counter.getValue();
                sb.append(jobName + "\t" + trackingUrl + "\t" +
counterGroupName
                        + "\t" + counterName + "\t" + counterValue + "\n");
            }
        }
        return sb.toString();
    }
}
Peng, Wei wrote:
>
> Hi,
>
>  
>
> I am a newbie of hadoop.
>
> Today I was struggling with a hadoop problem for several hours.
>
>  
>
> I initialize a parameter by setting job configuration in main.
>
> E.g. Configuration con = new Configuration();
>
> con.set("test", "1");
>
> Job job = new Job(con);
>
>  
>
> Then in the mapper class, I want to set "test" to "2". I did it by
>
> context.getConfiguration().set("test","2");
>
>  
>
> Finally in the main method, after the job is finished, I check the
> "test" again by
>
> job.getConfiguration().get("test");
>
>  
>
> However, the value of "test" is still "1".
>
>  
>
> The reason why I want to change the parameter inside Mapper class is
> that I want to determine when to stop an iteration in the main method.
> For example, for doing breadth-first search, when there is no new nodes
> are added for further expansion, the searching iteration should stop.
>
>  
>
> Your help will be deeply appreciated. Thank you
>
>  
>
> Wei
>
>
>

--
View this message in context: http://lucene.472066.n3.nabble.com/Please-help-with-hadoop-configuration-parameter-set-and-get-tp2103303p2107316.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Set the counter in the mapper and in the reducer code, check for this
counter. If counter is set then run the reducer if not then exit without
doing anything.
Once there is no new node is found, I would like to exit mapper or reducer
process.
For example, if no new nodes are found in the mapper class, the setup of
reducer will stop reducer work (or the cleanup of mapper will stop the
reducer work).

--
View this message in context: http://lucene.472066.n3.nabble.com/Please-help-with-hadoop-configuration-parameter-set-and-get-tp2103303p2107388.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

My question is how to exit hadoop reducer.
Can I just write "return;" in reducer?

Wei

-----Original Message-----
From: Aman [mailto:[EMAIL PROTECTED]]
Sent: Fri 12/17/2010 1:24 PM
To: [EMAIL PROTECTED]
Subject: RE: Please help with hadoop configuration parameter set and get
 

Set the counter in the mapper and in the reducer code, check for this
counter. If counter is set then run the reducer if not then exit without
doing anything.
Once there is no new node is found, I would like to exit mapper or reducer
process.
For example, if no new nodes are found in the mapper class, the setup of
reducer will stop reducer work (or the cleanup of mapper will stop the
reducer work).

--
View this message in context: http://lucene.472066.n3.nabble.com/Please-help-with-hadoop-configuration-parameter-set-and-get-tp2103303p2107388.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

If I wrote "return" in reducer class like
public class reducerClass{
   public void reduce{
      if(counter <= 0)
          return;
   }
}
The program has to go through all the mapper keys in order to return.
I even want to stop all mapreduce process before the mapper class does the sorting and shuffling.
Is there anything that I can call to exit mapreduce and return back to the main method?

Thanks
Wei

-----Original Message-----
From: Peng, Wei [mailto:[EMAIL PROTECTED]]
Sent: Fri 12/17/2010 2:14 PM
To: [EMAIL PROTECTED]
Subject: RE: Please help with hadoop configuration parameter set and get
 
My question is how to exit hadoop reducer.
Can I just write "return;" in reducer?

Wei

-----Original Message-----
From: Aman [mailto:[EMAIL PROTECTED]]
Sent: Fri 12/17/2010 1:24 PM
To: [EMAIL PROTECTED]
Subject: RE: Please help with hadoop configuration parameter set and get
 

Set the counter in the mapper and in the reducer code, check for this
counter. If counter is set then run the reducer if not then exit without
doing anything.
Once there is no new node is found, I would like to exit mapper or reducer
process.
For example, if no new nodes are found in the mapper class, the setup of
reducer will stop reducer work (or the cleanup of mapper will stop the
reducer work).

--
View this message in context: http://lucene.472066.n3.nabble.com/Please-help-with-hadoop-configuration-parameter-set-and-get-tp2103303p2107388.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.