Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Re: Passing values from InputFormat via the Configuration object


Copy link to this message
-
Re: Passing values from InputFormat via the Configuration object
Public Network Services 2013-05-18, 00:53
A more standard approach could be converting the JobContext parameter of
the getSplits() method, into a Job object, which then allows retrieving the
counters, e.g.:

public List<InputSplit> getSplits(JobContext job) throws IOException
{
...
Job work = new Job(job.getConfiguration());
Counters counters = work.getCounters();
}

Would that be correct?
On Fri, May 17, 2013 at 5:33 PM, Public Network Services <
[EMAIL PROTECTED]> wrote:

> Hi...
>
> I need to communicate some proprietary number (long) values from the
> getSplits() method of a custom InputFormat class to the Hadoop driver class
> (used to launch the job), but the JobContext object passed to the
> getSplits() method has no access to a Counters object.
>
> From the source code, it seems that the Configuration object of the
> launched job is passed around, so the JobContext object of getSplits() has
> direct access to it via getConfiguration().
>
> So, what about using a loop like
>
>         Job job = ... // The launched job
>         Configuration conf = job.getConfiguration();
>         while (!job.isComplete()) {
>         // Read the values from the configuration
>         }
>
> from the driver class, which presumably runs in the same framework that
> creates the splits?
>
> The getSplits() method of the custom InputFormat would set each of the
> values once.
>
> All this does seem like a hack, so I would like some expert advice before
> starting implementation. That is,
>
>    1. Will it work?
>    2. Is there a better method?
>
> Thanks!
>
>