Now that I think about it I wonder about oracle's description.

[...] if more than 98% of the total time is spent in garbage collection [...]

I took that to mean 98% of CPU time for the application, but it can
also mean that the GC is active for 98% percent of wall clock time.

If it is the former it implies the application isn't doing useful
stuff anymore (e.g., keeping a connection open). If it is the latter I
don't know if it possible to say anything about how much the program
is still doing useful stuff.

On Wed, Sep 5, 2012 at 2:48 PM, Björn-Elmar Macek
<[EMAIL PROTECTED]> wrote:
> Hi Vasco,
>
> thank you for your help!
>
> I can try to add the limit again (i currently have it turned off for all
> Java processes spawned by Hadoop). Also i do not have any persistent
> (member-) variables, that could store things that use alot of data: at the
> very moment i only got 2 local variables in the reduce-method, that could
> get a little larger - but those should be gc'ed easily, shouldnt they?
>
> Thank you for your thoughts again. I will pay with the GC a little. If i
> know the reason, i'll let you know!
> Best regards,
> Elmar
>
>
> Am 05.09.2012 14:38, schrieb Vasco Visser:
>
>> Just a guess, but could it simply be memory issue on reducer?
>>
>> In your adjusted program, maybe try running without
>> -UseGCOverheadLimit and see if you still got OOM errors.
>>
>>  From the sun website:
>> The parallel collector will throw an OutOfMemoryError if too much time
>> is being spent in garbage collection: if more than 98% of the total
>> time is spent in garbage collection and less than 2% of the heap is
>> recovered, an OutOfMemoryError will be thrown. This feature is
>> designed to prevent applications from running for an extended period
>> of time while making little or no progress because the heap is too
>> small. If necessary, this feature can be disabled by adding the option
>> -XX:-UseGCOverheadLimit to the command line.
>>
>> So basically if you get a GC overhead limit exceeded OOM error it
>> means that your app is doing nothing but garbage collection, the vm is
>> fighting against running out of memory. If that happens it might
>> result in a timeout of the connection fetching the map data (i don't
>> know if it can result in timeout, but I can imagine it could).
>>
>> Also, note that 63 GB on disk is probably going to be inflated in
>> memory. So in general you cant say that 60 GB on disk will need 60GB
>> of mem. Actually, some people us a rule of thumb to do x4 to get
>> approx mem requirement.
>>
>> Just some ideas, not really a solution but maybe it helps you further.
>>
>> On Wed, Sep 5, 2012 at 2:02 PM, Björn-Elmar Macek
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> Excuse me: in my last code section was some old code included. Here is it
>>> again stripped of deprecated code:
>>>
>>>
>>> package uni.kassel.macek.rtprep;
>>>
>>>
>>>
>>> import gnu.trove.iterator.TIntIterator;
>>> import gnu.trove.map.hash.TIntObjectHashMap;
>>> import gnu.trove.set.TIntSet;
>>>
>>> import java.io.IOException;
>>> import java.util.ArrayList;
>>> import java.util.Calendar;
>>> import java.util.Iterator;
>>> import java.util.List;
>>> import java.util.StringTokenizer;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IntWritable;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapred.MapReduceBase;
>>> import org.apache.hadoop.mapred.OutputCollector;
>>> import org.apache.hadoop.mapred.Reporter;
>>> import org.apache.hadoop.mapreduce.Job;
>>> import org.apache.hadoop.mapreduce.Mapper;
>>> import org.apache.hadoop.mapreduce.Reducer;
>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>
>>> public class RetweetApplication {
>>>
>>>      public static class RetweetMapper1 extends Mapper<Object, Text,

Hi Vasco,

i think both heuristics dont imply anything about the sense or nonsense
of the program:
The first version of my program used the "values" as "keys", so i could
accomplish some sorting. But this also meant, that since each value is
unique my iterator in the reduce method was also always of size 1. So i
had to store possible joinpartners in a reduce-function-external
variable and i decided to put it as a membervariable, which i regularly
cleaned of outdated data. This data structure grew with every call of
the reduce function and with this the size of non-garbage-collectable
data, while the code in my class didnt do too much: since i used a
hashmap here, there was only very few time-consuming traversing needed
and the execution time of my code was most probably smaller than the
execution time of the GC since a certain treshold. In fact: it is easy
to create a program that does meaningfull stuff and still trigger the
98%-2% rule and let the GC throw an exception.

My 2 cents,
Elmar
Am 05.09.2012 15:32, schrieb Vasco Visser:
> Now that I think about it I wonder about oracle's description.
>
> [...] if more than 98% of the total time is spent in garbage collection [...]
>
> I took that to mean 98% of CPU time for the application, but it can
> also mean that the GC is active for 98% percent of wall clock time.
>
> If it is the former it implies the application isn't doing useful
> stuff anymore (e.g., keeping a connection open). If it is the latter I
> don't know if it possible to say anything about how much the program
> is still doing useful stuff.
>
>
>
> On Wed, Sep 5, 2012 at 2:48 PM, Bj�rn-Elmar Macek
> <[EMAIL PROTECTED]> wrote:
>> Hi Vasco,
>>
>> thank you for your help!
>>
>> I can try to add the limit again (i currently have it turned off for all
>> Java processes spawned by Hadoop). Also i do not have any persistent
>> (member-) variables, that could store things that use alot of data: at the
>> very moment i only got 2 local variables in the reduce-method, that could
>> get a little larger - but those should be gc'ed easily, shouldnt they?
>>
>> Thank you for your thoughts again. I will pay with the GC a little. If i
>> know the reason, i'll let you know!
>> Best regards,
>> Elmar
>>
>>
>> Am 05.09.2012 14:38, schrieb Vasco Visser:
>>
>>> Just a guess, but could it simply be memory issue on reducer?
>>>
>>> In your adjusted program, maybe try running without
>>> -UseGCOverheadLimit and see if you still got OOM errors.
>>>
>>>   From the sun website:
>>> The parallel collector will throw an OutOfMemoryError if too much time
>>> is being spent in garbage collection: if more than 98% of the total
>>> time is spent in garbage collection and less than 2% of the heap is
>>> recovered, an OutOfMemoryError will be thrown. This feature is
>>> designed to prevent applications from running for an extended period
>>> of time while making little or no progress because the heap is too
>>> small. If necessary, this feature can be disabled by adding the option
>>> -XX:-UseGCOverheadLimit to the command line.
>>>
>>> So basically if you get a GC overhead limit exceeded OOM error it
>>> means that your app is doing nothing but garbage collection, the vm is
>>> fighting against running out of memory. If that happens it might
>>> result in a timeout of the connection fetching the map data (i don't
>>> know if it can result in timeout, but I can imagine it could).
>>>
>>> Also, note that 63 GB on disk is probably going to be inflated in
>>> memory. So in general you cant say that 60 GB on disk will need 60GB
>>> of mem. Actually, some people us a rule of thumb to do x4 to get
>>> approx mem requirement.
>>>
>>> Just some ideas, not really a solution but maybe it helps you further.
>>>
>>> On Wed, Sep 5, 2012 at 2:02 PM, Bj�rn-Elmar Macek
>>> <[EMAIL PROTECTED]> wrote:
>>>> Excuse me: in my last code section was some old code included. Here is it
>>>> again stripped of deprecated code:
>>>>
>>>>
>>>> package uni.kassel.macek.rtprep;

Hi Christoph,

oh its so nice to hear from you! :) Nice greetings back from the gang! :)

Well i implemented it as i u said: it is a merge join:
Join attributes are used to define the partition the values are going to
and the Comparators sort my data, so i can do a merge join and need only
one set of data to look possible merge partners. I just use Hasmaps to
get to the possible join partners faster.

Thank you for the tips nevertheless.

Best regards,
Elmar
Am 29.08.2012 16:23, schrieb Christoph Schmitz:
> Hi Elmar,
>
> I don't know about the technicalities of your problem, but why don't you use a reduce-side join in the first place? (I.e., a sort-merge join instead of a hash join.)
>
> <short-explanation-skip-if-you-already-know>
> Assuming you want to self-join a table of the form (FROM, TO) in order to compute all paths of length two, you'd emit each (FROM, TO) tuple twice, once with FROM as key and once with TO as key, plus a tag indicating which is which:
>
> (1, 47), (1, 12) and (47, 15) become, in the Reducer:
>
> ((1, F), (1, 12))
> ((1, F), (1, 47))
> ((12, T), (1, 12))
> ((15, T), (47, 15))
> ((47, F), (47, 15))
> ((47, T), (1, 47))
>
> F, T indicate if the key was a FROM or a TO value.
>
> You'll make Hadoop group by the first component of the key, which will allow you to see both of the "47" tuples that would be joined in one reducer. From those, you build the resulting (1, 47, 15) tuple.
>
> (Detailed description in the O'Reilly/Tom White book.)
> </short-explanation-skip-if-you-already-know>
>
> Hope this helps (and kind regards to the Kassel gang ;-),
>
> Christoph
>
> -----Urspr�ngliche Nachricht-----
> Von: Bj�rn-Elmar Macek [mailto:[EMAIL PROTECTED]]
> Gesendet: Mittwoch, 29. August 2012 15:54
> An: [EMAIL PROTECTED]
> Betreff: mortbay, huge files and the ulimit
>
> Hi there,
>
> i am currently running a job where i selfjoin a 63 gigabyte big csv file
> on 20 physically distinct nodes with 15GB each:
>
> While the mapping works just fine and is low cost, the reducer does the
> main work: holding a hashmap with elements to join with and finding join
> tuples for evry incoming key-value-pair.
>
> The jobs works perfectly on small files with 2 gigabytes, but starts to
> get "unstable" as the file size goes up: this becomes evident with a
> look into the tasktracker's logs saying:
>
> ERROR org.mortbay.log: /mapOutput
> java.lang.IllegalStateException: Committed
>       at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>       at org.mortbay.jetty.Response.sendError(Response.java:240)
>       at
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>       at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>       at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>       at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>       at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>       at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>       at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>       at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>       at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>       at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>       at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>       at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>       at org.mortbay.jetty.Server.handle(Server.java:326)
>       at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>       at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)

Hi there,

i am currently running a job where i selfjoin a 63 gigabyte big csv file
on 20 physically distinct nodes with 15GB each:

While the mapping works just fine and is low cost, the reducer does the
main work: holding a hashmap with elements to join with and finding join
tuples for evry incoming key-value-pair.

The jobs works perfectly on small files with 2 gigabytes, but starts to
get "unstable" as the file size goes up: this becomes evident with a
look into the tasktracker's logs saying:

ERROR org.mortbay.log: /mapOutput
java.lang.IllegalStateException: Committed
     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
     at org.mortbay.jetty.Response.sendError(Response.java:240)
     at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
And while it is no problem at the beginning of the reduce process, where
this happens only on a few nodes and rarely, it becomes crucial as the
progress rises. The reason for this (afaik from reading articles), is
that there are memory or file handle problems. I addressed the memory
problem by conitiously purging the map of outdated elements evry 5
million processed key-value-pairs. And i set  mapred.child.ulimit to
100000000 (ulimit in the shell tells me it is 400000000).

Anyway i am still running into those mortbay errors and i start to
wonder, if hadoop can manage the job with this algorithmn anyways. By
pure naive math it should be:
i explicily assigned 10GB memory to each JVM on each node and set
mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops
-XX:-UseGCOverheadLimit" (its a 64 bit environment and large
datastructures cause the GC to throw exceptions). This would naively
make 18 slave machines with 10GB each resulting in an overall memory of
180GB - three times as much as needed... i would think. So if the
Partitioner distributes them just about equally to all nodes i should
not run into any errors, do i?

Can anybody help me with this issue?

Best regards,
Elmar

Hi,

is there anybody, who knows more about this issue: it has already been
recently marked here:
https://issues.apache.org/jira/browse/MAPREDUCE-5

I really want to do something about it, if i knew how... i tried so many
different setup parameters and JVM options and nothing did the trick...

It was opened 2009 and gets still updated (2012): I mean - this does NOT
look OK, does it? The whole TT seems like a big time waster:

2012-08-30 12:05:22,582 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
141.51.205.130:35497, dest: 141.51.205.126:39586, bytes: 146086410, op:
MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000189_0, duration:
5456122852
2012-08-30 12:05:56,275 INFO org.mortbay.log:
org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
injecting delay2 times
2012-08-30 12:05:56,276 INFO org.mortbay.log:
org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
recreating selector 2 times, canceled keys 70 times
2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
org.mortbay.jetty.EofException
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
     at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
     at
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
     at
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
     at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
     at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
     at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.io.IOException: Broken pipe
     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
     at
org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
     ... 27 more

2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410
getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
org.mortbay.jetty.EofException
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
     at
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
     at
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
     at
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
     at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
     at
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
     at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at
org.mortbay.jett

Hi all,

i got some more information:

It is always only one TT, where mortbay errors do not occur: all other
slaves fail relatively fast after the reduce process started: maybe it
is a comparator issue (since the operation is MAPRED_SHUFFLE)
And it has nothing to do with the local configuration of the server, since
1) if i rerun the job on the same set of servers, the ONE working
differs from one job to another.
2) the same happens even when i am using a completely different set of
servers.

Who is capable to solve this unsolved mystery? :D

Am 30.08.2012 12:27, schrieb Bj�rn-Elmar Macek:
> Hi,
>
> is there anybody, who knows more about this issue: it has already been
> recently marked here:
> https://issues.apache.org/jira/browse/MAPREDUCE-5
>
> I really want to do something about it, if i knew how... i tried so
> many different setup parameters and JVM options and nothing did the
> trick...
>
> It was opened 2009 and gets still updated (2012): I mean - this does
> NOT look OK, does it? The whole TT seems like a big time waster:
>
> 2012-08-30 12:05:22,582 INFO
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src:
> 141.51.205.130:35497, dest: 141.51.205.126:39586, bytes: 146086410,
> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000189_0,
> duration: 5456122852
> 2012-08-30 12:05:56,275 INFO org.mortbay.log:
> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
> injecting delay2 times
> 2012-08-30 12:05:56,276 INFO org.mortbay.log:
> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) -
> recreating selector 2 times, canceled keys 70 times
> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker:
> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
> org.mortbay.jetty.EofException
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>     at
> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>     at
> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>     at
> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>     at
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>     at
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>     at
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

Hello again,

i just wanted to keep you updated, in case anybody reads this and is
interested in the mortbay-issue:
I applied the wordcount example to a big input file and everything
worked fine. I watched the TT logs all the time and no single
mortbay-eof exception occurred. The only possible explanation for me
seemed to be, that my code somehow produces the exceptions. So i started
to keep my code as close to the wordcount example as possible:

* first getting rid of any members within the reducer class, that may
take lots of memory.
* using the Context class instead of OutputCollector (in Mapper and Reducer)
* removing any Comparator (and restructuring the code so it just
calculates the same as with the sorting)
* using standart HashPartitioner

Especially the first 2 points made my job succeed in the end, ALTHOUGH
the problem of "n-1 mortbay-exception-throwing TTs" still remains: one
node seems to complete all the work while the others are busy with their
exceptions (at least from what i can see in the log).

I REALLY dont know what i can do, to make this job run through like the
wordcount example. I dont see any big difference between my program and
the example anymore.

I added the code here:

*Please* help me here.

Best regards,
Elmar
I added the code here:
########################################
package uni.kassel.macek.rtprep;

import java.util.Date;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class RetweetApplication extends Configured implements Tool{

     public static int DEBUG = -1;
     private Configuration jobConf;

     //static String INPUT = "/home/fb16/bmacek/HADOOP-INPUT";
     //static String OUTPUT = "/home/fb16/bmacek/HADOOP-OUTPUT "+ (new
Date()).toString();

     public static void main(String[] args) {
          int res=-1;
         try {
             res = ToolRunner.run(new Configuration(), new
RetweetApplication(), args);
         } catch (Exception e) {
             // TODO Auto-generated catch block
             e.printStackTrace();
         }
         System.exit(res);

     }

     public void setConf(Configuration conf) {
         jobConf = conf;

     }

     public Configuration getConf() {
         return jobConf;
     }

     public int run(String[] args) throws Exception {
         JobClient client = new JobClient();
         JobConf conf = new JobConf(RetweetApplication.class);
         if(DEBUG > 0){
             conf.set("mapred.job.tracker", "local");
             conf.set("fs.default.name", "file:///");
             conf.set("dfs.replication", "1");
         }

         //conf.set("mapred.child.java.opts","-Xmx3000m
-XX:+UseCompressedOops -XX:-UseGCOverheadLimit");
         conf.set("mapred.child.ulimit","100000000");

         FileInputFormat.setInputPaths(conf, new Path(args[0]));
         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
         //conf.setOutputKeyClass(Text.class);
         //conf.setOutputValueClass(Text.class);
         conf.setMapOutputKeyClass(Text.class);
         conf.setMapOutputValueClass(Text.class);

         conf.setMapperClass(RetweetMapper.class);
         //conf.setPartitionerClass(TweetPartitioner.class);
//conf.setOutputKeyComparatorClass(TwitterValueGroupingComparator.class);
//conf.setOutputValueGroupingComparator(TwitterKeyGroupingComparator.class);
         conf.setReducerClass(RetweetReducer.class);

         conf.setOutputFormat(TextOutputFormat.class);

         client.setConf(conf);
         try {
             JobClient.runJob(conf);
         } catch (Exception e) {
             e.printStackTrace();
         }
         return 0;
     }
}
*/

package uni.kassel.macek.rtprep;

import gnu.trove.iterator.TIntIterator;
import gnu.trove.map.hash.TIntObjectHashMap;
import gnu.trove.set.TIntSet;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Iterator;
import java.util.List;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class RetweetApplication {

     public static class RetweetMapper1 extends Mapper<Object, Text,
Text, Text> {

         public void map(Object key, Text value,
             Context context)
             throws IOException {

             String line = value.toString();

             Tweet aTweet = new Tweet(line);
             if(aTweet.getMention()!=null &&
aTweet.getMention().length()>0){
                 /*aTweet.setRT(0);
                 output.collect(new Text(aTweet.toString()), new
Text(aTweet.toString()));*/
                 aTweet.setRT(1);
                 try {
                     context.write(new Text(aTweet.getMention()), new
Text(aTweet.toString()));
                 } catch (InterruptedException e) {
                     // TODO Auto-generated catch block
                     e.printStackTrace();
                 }
             }
             aTweet.setRT(0);
             try {
                 context.write(new Text(aTweet.getAuthor())

Excuse me: in my last code section was some old code included. Here is
it again stripped of deprecated code:

package uni.kassel.macek.rtprep;

import gnu.trove.iterator.TIntIterator;
import gnu.trove.map.hash.TIntObjectHashMap;
import gnu.trove.set.TIntSet;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Iterator;
import java.util.List;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class RetweetApplication {

     public static class RetweetMapper1 extends Mapper<Object, Text,
Text, Text> {

         public void map(Object key, Text value,
             Context context)
             throws IOException {

             String line = value.toString();

             Tweet aTweet = new Tweet(line);
             if(aTweet.getMention()!=null &&
aTweet.getMention().length()>0){
                 /*aTweet.setRT(0);
                 output.collect(new Text(aTweet.toString()), new
Text(aTweet.toString()));*/
                 aTweet.setRT(1);
                 try {
                     context.write(new Text(aTweet.getMention()), new
Text(aTweet.toString()));
                 } catch (InterruptedException e) {
                     // TODO Auto-generated catch block
                     e.printStackTrace();
                 }
             }
             aTweet.setRT(0);
             try {
                 context.write(new Text(aTweet.getAuthor()), new
Text(aTweet.toString()));
             } catch (InterruptedException e) {
                 // TODO Auto-generated catch block
                 e.printStackTrace();
             }

         }
     }

public static class RetweetReducer1 extends Reducer<Text, Text, Text,
Text> {

     public void reduce(Text atweet, Iterator<Text> tweets,
             Context context)
             throws IOException {
         List<Tweet> originals = new ArrayList<Tweet>();
         List<Tweet> retweets = new ArrayList<Tweet>();
         while(tweets.hasNext()){
             Tweet aTweet = new Tweet(tweets.toString());
             originals.add(aTweet);
             if(aTweet.getRT()>0) retweets.add(aTweet);
         }

         for(Tweet originalTweet : originals){
             for(Tweet aRetweet : retweets){
                 if(aRetweet.isRetweeted(originalTweet))
                 {
aRetweet.setReactionTime(originalTweet.getDate());
                     try {
                         context.write(
                                 atweet,
                                 new Text(aRetweet.toString()));
                     } catch (InterruptedException e) {
                         // TODO Auto-generated catch block
                         e.printStackTrace();
                     }
                 }
             }
         }
     }
}

   public static void main(String[] args) throws Exception {
     Configuration conf = new Configuration();
     String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();
     if (otherArgs.length != 2) {
       System.err.println("Usage: wordcount <in> <out>");
       System.exit(2);
     }
     Job job = new Job(conf, "word count");
     job.setJarByClass(RetweetApplication.class);
     job.setMapperClass(RetweetMapper1.class);
     job.setReducerClass(RetweetReducer1.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(Text.class);
     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
     System.exit(job.waitForCompletion(true) ? 0 : 1);
   }
}
Am 05.09.2012 13:56, schrieb Bj�rn-Elmar Macek: