-Re: M/R, Strange behavior with multiple Gzip files
x6i4uybz labs 2012-12-06, 09:57
I wrote a job M/R to process several gz files (about 2000). I've a 80 map
JT instantiates one map per gz file (not splittable, it's OK).
The first 80 maps spawn. But after "initializing" state, it seems there is
one map running. And when this map is finished, another one started (not 80
maps in parallel) and another is affected to the empty slot.
I've also noticed, the first maps last more than one hour and the last maps
Each gz file is between 10mb and 100mb.
I don't understand the behavior.
I will launch again the job to see if I've the same issue.
On Wed, Dec 5, 2012 at 6:33 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Your problem isn't clear in your description - can you please
> rephrase/redefine in terms of what you are expecting vs. what you are
> Also note that Gzip files are not splittable by nature of their codec
> algorithm, and hence a TextInputFormat over plain/regular Gzip files
> would end up spawning and/or processing one whole Gzip file via one
> mapper, instead of multiple mappers per file.
> On Wed, Dec 5, 2012 at 9:32 PM, x6i4uybz labs <[EMAIL PROTECTED]>
> > Hi everybody,
> > I have a M/R job which does a bulk import to hbase.
> > I have to process many gzip files (2800 x ~ 100mb)
> > I don't understand why my job instanciates 80 maps but runs each map
> > sequentialy like if there is only one big gz file.
> > Is there a problem in my driver ? Or maybe I miss something.
> > I use "FileInputFormat.addInputPath(job, new Path(args))" where
> > is a directory.
> > Can you help me, please ?
> > Thanks, Guillaume
> Harsh J