I have large files where the userid is the first line of each file.  I want to use that value as the output of the map phase for each subsequent line of the file.  If each map task gets a chunk of this file only one map task will read the key value from the first line.  Is there anyway I can force the other map tasks to wait until this key is read and then somehow pass this value to other map tasks?  Or is my reasoning incorrect?  Thanks.

I will do that like this: at each map task, I get the input file to
this mapper in the configure(), and manually read the first line of
that file to get the user ID. Then start running the map function.
-Gang
----- 原始邮件 ----
发件人： Raymond Jennings III <[EMAIL PROTECTED]>
收件人： [EMAIL PROTECTED]
发送日期： 2010/1/8 (周�
� 4:23:15 下午
主   题： Is it possible to share a key across maps?

I have large files where the userid is the first line of each file.  I want to use that value as the output of the map phase for each subsequent line of the file.  If each map task gets a chunk of this file only one map task will read the key value from the first line.  Is there anyway I can force the other map tasks to wait until this key is read and then somehow pass this value to other map tasks?  Or is my reasoning incorrect?  Thanks.
      ___________________________________________________________
  好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/

Hi, you do this in the map method (open the file and read the first line?)  Could you explain a little more how you do it with configure(), thank you.

--- On Fri, 1/8/10, Gang Luo <[EMAIL PROTECTED]> wrote:

> From: Gang Luo <[EMAIL PROTECTED]>
> Subject: Re: Is it possible to share a key across maps?
> To: [EMAIL PROTECTED]
> Date: Friday, January 8, 2010, 4:46 PM
> I will do that like this: at each map
> task, I get the input file to
> this mapper in the configure(), and manually read the first
> line of
> that file to get the user ID. Then start running the map
> function.
>
>
> -Gang
>
>
> ----- ���始邮件 ----
> 发件人��� Raymond Jennings III <[EMAIL PROTECTED]>
> 收件人： [EMAIL PROTECTED]
> 发送���期： 2010/1/8 (周五) 4:23:15 下���
> 主   题： Is it possible to share a key
> across maps?
>
> I have large files where the userid is the first line of
> each file.  I want to use that value as the output of
> the map phase for each subsequent line of the file.  If
> each map task gets a chunk of this file only one map task
> will read the key value from the first line.  Is there
> anyway I can force the other map tasks to wait until this
> key is read and then somehow pass this value to other map
> tasks?  Or is my reasoning incorrect?  Thanks.
>
>
>      
> ___________________________________________________________
>
>   好玩��卡等你发，邮箱贺卡全新上线！
>
> http://card.mail.cn.yahoo.com/
>

I don't do that in map method, but in configure( JobConf ) method which runs ahead of any map method call in that map task. JobConf.get("map.input.file") can tell you which file this map task is processing. Use this path to read first line of corresponding file. All these are done in configure method, that means, before any map method is called.
-Gang

----- 原始邮件 ----
发件人： Raymond Jennings III <[EMAIL PROTECTED]>
收件人： [EMAIL PROTECTED]
发送日期： 2010/1/8 (周五) 7:54:30 下午
主   题： Re: Is it possible to share a key across maps?

Hi, you do this in the map method (open the file and read the first line?)  Could you explain a little more how you do it with configure(), thank you.

--- On Fri, 1/8/10, Gang Luo <[EMAIL PROTECTED]> wrote:

> From: Gang Luo <[EMAIL PROTECTED]>
> Subject: Re: Is it possible to share a key across maps?
> To: [EMAIL PROTECTED]
> Date: Friday, January 8, 2010, 4:46 PM
> I will do that like this: at each map
> task, I get the input file to
> this mapper in the configure(), and manually read the first
> line of
> that file to get the user ID. Then start running the map
> function.
>
>
> -Gang
>
>
> ----- 原始邮件 ----
> 发件人： Raymond Jennings III <[EMAIL PROTECTED]>
> 收件人： [EMAIL PROTECTED]
> 发送日期： 2010/1/8 (�
芪� 4:23:15 下午
> 主   题： Is it possible to share a key
> across maps?
>
> I have large files where the userid is the first line of
> each file.  I want to use that value as the output of
> the map phase for each subsequent line of the file.  If
> each map task gets a chunk of this file only one map task
> will read the key value from the first line.  Is there
> anyway I can force the other map tasks to wait until this
> key is read and then somehow pass this value to other map
> tasks?  Or is my reasoning incorrect?  Thanks.
>
>
>      
> ___________________________________________________________
>
>   好玩贺卡等你发，邮箱贺卡全新上线！
>
> http://card.mail.cn.yahoo.com/
>
      ___________________________________________________________
  好玩贺卡等你发，邮箱�
乜ㄈ�律舷撸�
http://card.mail.cn.yahoo.com/

Actually you can treat the mapper task as a template design pattern, here's
the persuade code:

Mapper.configure(JobConf)
for each record in InputSplit:
      do Mapper.map(key,value,outputkey,outputvalue)
Mapper.close()

Any sub class of mapper can override the three method: configure(),
map(),close() to do customization.

2010/1/8 Gang Luo <[EMAIL PROTECTED]>

> I don't do that in map method, but in configure( JobConf ) method which
> runs ahead of any map method call in that map task.
> JobConf.get("map.input.file") can tell you which file this map task is
> processing. Use this path to read first line of corresponding file. All
> these are done in configure method, that means, before any map method is
> called.
>
>
> -Gang
>
>
>
> ----- 原始邮件 ----
> 发件人： Raymond Jennings III <[EMAIL PROTECTED]>
> 收件人： [EMAIL PROTECTED]
> 发送日期： 2010/1/8 (周��) 7:54:30 下午
> 主   题： Re: Is it possible to share a key across maps?
>
> Hi, you do this in the map method (open the file and read the first line?)
>  Could you explain a little more how you do it with configure(), thank you.
>
> --- On Fri, 1/8/10, Gang Luo <[EMAIL PROTECTED]> wrote:
>
> > From: Gang Luo <[EMAIL PROTECTED]>
> > Subject: Re: Is it possible to share a key across maps?
> > To: [EMAIL PROTECTED]
> > Date: Friday, January 8, 2010, 4:46 PM
> > I will do that like this: at each map
> > task, I get the input file to
> > this mapper in the configure(), and manually read the first
> > line of
> > that file to get the user ID. Then start running the map
> > function.
> >
> >
> > -Gang
> >
> >
> > ----- 原始邮件 ----
> > 发件人： Raymond Jennings III <[EMAIL PROTECTED]>
> > 收件人： [EMAIL PROTECTED]
> > 发送日期： 2010/1/8 (周��) 4:23:15 下午
> > 主   题： Is it possible to share a key
> > across maps?
> >
> > I have large files where the userid is the first line of
> > each file.  I want to use that value as the output of
> > the map phase for each subsequent line of the file.  If
> > each map task gets a chunk of this file only one map task
> > will read the key value from the first line.  Is there
> > anyway I can force the other map tasks to wait until this
> > key is read and then somehow pass this value to other map
> > tasks?  Or is my reasoning incorrect?  Thanks.
> >
> >
> >
> > ___________________________________________________________
> >
> >   好玩贺卡等你发��邮箱贺卡全新上线！
> >
> > http://card.mail.cn.yahoo.com/
> >
>
>
>       ___________________________________________________________
>   好玩贺卡等你发��邮箱贺卡全新上线！
> http://card.mail.cn.yahoo.com/
>

--
Best Regards

Jeff Zhang

It looks like what you are referring to is the deprecated class - which has made for some confusing conversations in the past.  It seems like many users still use the older API and most of the examples still use it.  I would like to stay with the more recent api which looks the call is actually "setup()" instead of configure().  Not sure if it's a one to one mapping though.

--- On Fri, 1/8/10, Jeff Zhang <[EMAIL PROTECTED]> wrote:

> From: Jeff Zhang <[EMAIL PROTECTED]>
> Subject: Re: Is it possible to share a key across maps?
> To: [EMAIL PROTECTED]
> Date: Friday, January 8, 2010, 11:15 PM
> Actually you can treat the mapper
> task as a template design pattern, here's
> the persuade code:
>
> Mapper.configure(JobConf)
> for each record in InputSplit:
>       do
> Mapper.map(key,value,outputkey,outputvalue)
> Mapper.close()
>
> Any sub class of mapper can override the three method:
> configure(),
> map(),close() to do customization.
>
>
>
> 2010/1/8 Gang Luo <[EMAIL PROTECTED]>
>
> > I don't do that in map method, but in configure(
> JobConf ) method which
> > runs ahead of any map method call in that map task.
> > JobConf.get("map.input.file") can tell you which file
> this map task is
> > processing. Use this path to read first line of
> corresponding file. All
> > these are done in configure method, that means, before
> any map method is
> > called.
> >
> >
> > -Gang
> >
> >
> >
> > ----- 原始邮件 ----
> > 发件人： Raymond Jennings III <[EMAIL PROTECTED]>
> > 收件人： [EMAIL PROTECTED]
> > 发送日期： 2010/1/8 (周五) 7:54:30 下午
> > 主   题： Re: Is it possible to
> share a key across maps?
> >
> > Hi, you do this in the map method (open the file and
> read the first line?)
> >  Could you explain a little more how you do it
> with configure(), thank you.
> >
> > --- On Fri, 1/8/10, Gang Luo <[EMAIL PROTECTED]>
> wrote:
> >
> > > From: Gang Luo <[EMAIL PROTECTED]>
> > > Subject: Re: Is it possible to share a key across
> maps?
> > > To: [EMAIL PROTECTED]
> > > Date: Friday, January 8, 2010, 4:46 PM
> > > I will do that like this: at each map
> > > task, I get the input file to
> > > this mapper in the configure(), and manually read
> the first
> > > line of
> > > that file to get the user ID. Then start running
> the map
> > > function.
> > >
> > >
> > > -Gang
> > >
> > >
> > > ----- 原始邮件 ----
> > > 发件人： Raymond Jennings III <[EMAIL PROTECTED]>
> > > 收件人： [EMAIL PROTECTED]
> > > 发送日期： 2010/1/8 (周五) 4:23:15 下午
> > > 主   题： Is it possible to
> share a key
> > > across maps?
> > >
> > > I have large files where the userid is the first
> line of
> > > each file.  I want to use that value as the
> output of
> > > the map phase for each subsequent line of the
> file.  If
> > > each map task gets a chunk of this file only one
> map task
> > > will read the key value from the first
> line.  Is there
> > > anyway I can force the other map tasks to wait
> until this
> > > key is read and then somehow pass this value to
> other map
> > > tasks?  Or is my reasoning incorrect? 
> Thanks.
> > >
> > >
> > >
> > >
> ___________________________________________________________
> > >
> >
> >   好玩贺卡等你发，邮箱贺���全新上线！
> > >
> > > http://card.mail.cn.yahoo.com/
> > >
> >
> >
> >   
>    ___________________________________________________________
> >   好玩贺卡等你发，邮箱贺卡全新���线！
> > http://card.mail.cn.yahoo.com/
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Hi Gang,
I was able to use this on an older version that uses the JobClient class to run the job but not on the newer api with the Job class.  The Job class appears to use a setup() method instead of a configure() method but the "map.input.file" attribute does not appear to be available via the conf class the setup() method.  Have you tried to do what you described using the newer api?  Thank you.

--- On Fri, 1/8/10, Gang Luo <[EMAIL PROTECTED]> wrote:

> From: Gang Luo <[EMAIL PROTECTED]>
> Subject: Re: Is it possible to share a key across maps?
> To: [EMAIL PROTECTED]
> Date: Friday, January 8, 2010, 10:03 PM
> I don't do that in map method, but in
> configure( JobConf ) method which runs ahead of any map
> method call in that map task. JobConf.get("map.input.file")
> can tell you which file this map task is processing. Use
> this path to read first line of corresponding file. All
> these are done in configure method, that means, before any
> map method is called.
>
>
> -Gang
>
>
>
> ----- 原始邮件 ----
> 发件人： Raymond Jennings III <[EMAIL PROTECTED]>
> 收件人： [EMAIL PROTECTED]
> 发送日期： 2010/1/8 (周���) 7:54:30 下午
> 主   题： Re: Is it possible to share a
> key across maps?
>
> Hi, you do this in the map method (open the file and read
> the first line?)  Could you explain a little more how
> you do it with configure(), thank you.
>
> --- On Fri, 1/8/10, Gang Luo <[EMAIL PROTECTED]>
> wrote:
>
> > From: Gang Luo <[EMAIL PROTECTED]>
> > Subject: Re: Is it possible to share a key across
> maps?
> > To: [EMAIL PROTECTED]
> > Date: Friday, January 8, 2010, 4:46 PM
> > I will do that like this: at each map
> > task, I get the input file to
> > this mapper in the configure(), and manually read the
> first
> > line of
> > that file to get the user ID. Then start running the
> map
> > function.
> >
> >
> > -Gang
> >
> >
> > ----- 原���邮件 ----
> > 发件人： Raymond Jennings III <[EMAIL PROTECTED]>
> > 收件��： [EMAIL PROTECTED]
> > 发送��期： 2010/1/8 (周五) 4:23:15 下��
> > 主   题： Is it possible to share a
> key
> > across maps?
> >
> > I have large files where the userid is the first line
> of
> > each file.  I want to use that value as the
> output of
> > the map phase for each subsequent line of the
> file.  If
> > each map task gets a chunk of this file only one map
> task
> > will read the key value from the first line.  Is
> there
> > anyway I can force the other map tasks to wait until
> this
> > key is read and then somehow pass this value to other
> map
> > tasks?  Or is my reasoning incorrect? 
> Thanks.
> >
> >
> >     
> >
> ___________________________________________________________
> >
> >   好玩贺卡等你发，邮箱���卡全新上线！
> >
> > http://card.mail.cn.yahoo.com/
> >
>
>
>      
> ___________________________________________________________
>
> �� 好玩贺卡等你发��邮箱贺卡全新上线！
>
> http://card.mail.cn.yahoo.com/
>

(Sorry for the spam if any, mails are bouncing back for me)

Hi,
In setup() use this,
FileSplit split = (FileSplit)context.getInputSplit();
 split.getPath() will return you the Path.
Hope this helps.

Amogh
On 1/13/10 1:25 AM, "Raymond Jennings III" <[EMAIL PROTECTED]> wrote:

Hi Gang,
I was able to use this on an older version that uses the JobClient class to run the job but not on the newer api with the Job class.  The Job class appears to use a setup() method instead of a configure() method but the "map.input.file" attribute does not appear to be available via the conf class the setup() method.  Have you tried to do what you described using the newer api?  Thank you.

--- On Fri, 1/8/10, Gang Luo <[EMAIL PROTECTED]> wrote:

+1 for the documentation change in mapred-tutorial. Can we do that and publish using a normal apache account?

Thanks,
Amogh
On 1/13/10 2:29 AM, "Raymond Jennings III" <[EMAIL PROTECTED]> wrote:

Amogh,
You bet it helps!  Thanks!  Sometimes it's very difficult to map between the old and the new APIs.  I was digging for that answer for awhile.  Thanks.

--- On Tue, 1/12/10, Amogh Vasekar <[EMAIL PROTECTED]> wrote:

> From: Amogh Vasekar <[EMAIL PROTECTED]>
> Subject: Re: Is it possible to share a key across maps?
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Tuesday, January 12, 2010, 3:32 PM
>
>
> Re: Is it possible to share a key across
> maps?
>
>
> (Sorry for the spam if any, mails
> are bouncing back for me)
>
>
>
> Hi,
>
> In setup() use this,
>
> FileSplit split = (FileSplit)context.getInputSplit();
>
>  split.getPath() will return you the Path.
>
> Hope this helps.
>
>
>
> Amogh
>
>
>
>
>
> On 1/13/10 1:25 AM, "Raymond Jennings III" <[EMAIL PROTECTED]> wrote:
>
>
>
> Hi Gang,
>
> I was able to use this on an older version that uses the
> JobClient class to run the job but not on the newer api with
> the Job class.  The Job class appears to use a setup()
> method instead of a configure() method but the
> "map.input.file" attribute does not appear to be
> available via the conf class the setup() method.  Have
> you tried to do what you described using the newer api?
>  Thank you.
>
>
>
> --- On Fri, 1/8/10, Gang Luo <[EMAIL PROTECTED]> wrote:
>
>
>
>
>
>
>
>

Please submit a patch for the documentation change - perhaps at
https://issues.apache.org/jira/browse/HADOOP-5973.

Cheers,
Tom

On Wed, Jan 13, 2010 at 12:09 AM, Amogh Vasekar <[EMAIL PROTECTED]> wrote:
> +1 for the documentation change in mapred-tutorial. Can we do that and publish using a normal apache account?
>
> Thanks,
> Amogh
>
>
> On 1/13/10 2:29 AM, "Raymond Jennings III" <[EMAIL PROTECTED]> wrote:
>
> Amogh,
> You bet it helps!  Thanks!  Sometimes it's very difficult to map between the old and the new APIs.  I was digging for that answer for awhile.  Thanks.
>
> --- On Tue, 1/12/10, Amogh Vasekar <[EMAIL PROTECTED]> wrote:
>
>> From: Amogh Vasekar <[EMAIL PROTECTED]>
>> Subject: Re: Is it possible to share a key across maps?
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Date: Tuesday, January 12, 2010, 3:32 PM
>>
>>
>> Re: Is it possible to share a key across
>> maps?
>>
>>
>> (Sorry for the spam if any, mails
>> are bouncing back for me)
>>
>>
>>
>> Hi,
>>
>> In setup() use this,
>>
>> FileSplit split = (FileSplit)context.getInputSplit();
>>
>>  split.getPath() will return you the Path.
>>
>> Hope this helps.
>>
>>
>>
>> Amogh
>>
>>
>>
>>
>>
>> On 1/13/10 1:25 AM, "Raymond Jennings III" <[EMAIL PROTECTED]> wrote:
>>
>>
>>
>> Hi Gang,
>>
>> I was able to use this on an older version that uses the
>> JobClient class to run the job but not on the newer api with
>> the Job class.  The Job class appears to use a setup()
>> method instead of a configure() method but the
>> "map.input.file" attribute does not appear to be
>> available via the conf class the setup() method.  Have
>> you tried to do what you described using the newer api?
>>  Thank you.
>>
>>
>>
>> --- On Fri, 1/8/10, Gang Luo <[EMAIL PROTECTED]> wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
>
>