Hi,
I am trying to override mapred-site.xml (more specifically
mapred.compress.map.output
and mapred.output.compression.
codec) from the command line when I
execute the jar.
I have been using hadoop jar <jarname> <class> -
Dmapred.compress.map.output=true and
-Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec

The above doesnt work as the job.xml for the jar still uses the default
properties and not the one i specify here. Is there a different approach to
override these properties. I am submitting jobs from a client machine that
has the same version of configuration files as my cluster.

Thanks

Sid

Check your mapred site xml if these parameters have <final>true</final>

making final to false should solve your problem.

On Wed, Jun 6, 2012 at 3:41 PM, Sid Kumar <[EMAIL PROTECTED]> wrote:

> Hi,
> I am trying to override mapred-site.xml (more specifically
> mapred.compress.map.output
> and mapred.output.compression.
> codec) from the command line when I
> execute the jar.
> I have been using hadoop jar <jarname> <class> -
> Dmapred.compress.map.output=true and
> -Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec
>
> The above doesnt work as the job.xml for the jar still uses the default
> properties and not the one i specify here. Is there a different approach to
> override these properties. I am submitting jobs from a client machine that
> has the same version of configuration files as my cluster.
>
> Thanks
>
> Sid
>

Mayank,
I dont have a final tag for that property set. I looked at the
mapred-default.xml in the src/mapred folder and that doesn't have a final
tag too. Should I set it explicitly to false?

Sid

On Wed, Jun 6, 2012 at 3:50 PM, Mayank Bansal <[EMAIL PROTECTED]> wrote:

> Check your mapred site xml if these parameters have <final>true</final>
>
> making final to false should solve your problem.
>
>
> On Wed, Jun 6, 2012 at 3:41 PM, Sid Kumar <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>> I am trying to override mapred-site.xml (more specifically
>> mapred.compress.map.output
>> and mapred.output.compression.
>> codec) from the command line when I
>> execute the jar.
>> I have been using hadoop jar <jarname> <class> -
>> Dmapred.compress.map.output=true and
>> -Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec
>>
>> The above doesnt work as the job.xml for the jar still uses the default
>> properties and not the one i specify here. Is there a different approach to
>> override these properties. I am submitting jobs from a client machine that
>> has the same version of configuration files as my cluster.
>>
>> Thanks
>>
>> Sid
>>
>
>

I am able to set it via the API.
Configuration.setBoolean(mapred.output.compress,true). This works!

But the -D from the command line still doesn't work. Any idea what I may be
missing here?

Some additional info - Also when I try running the -D on command line on a
local cluster (pseudo distributed mode) it works, but when I try it on a
fully distributed cluster running jobs from a client machine it doesn't
work. Is there a different way for setting it in this case - in hadoop-env
perhaps?

Thanks
Sid

On Wed, Jun 6, 2012 at 4:06 PM, Sid Kumar <[EMAIL PROTECTED]> wrote:

> Mayank,
> I dont have a final tag for that property set. I looked at the
> mapred-default.xml in the src/mapred folder and that doesn't have a final
> tag too. Should I set it explicitly to false?
>
> Sid
>
>
> On Wed, Jun 6, 2012 at 3:50 PM, Mayank Bansal <[EMAIL PROTECTED]> wrote:
>
>> Check your mapred site xml if these parameters have <final>true</final>
>>
>> making final to false should solve your problem.
>>
>>
>> On Wed, Jun 6, 2012 at 3:41 PM, Sid Kumar <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>> I am trying to override mapred-site.xml (more specifically
>>> mapred.compress.map.output
>>> and mapred.output.compression.
>>> codec) from the command line when I
>>> execute the jar.
>>> I have been using hadoop jar <jarname> <class> -
>>> Dmapred.compress.map.output=true and
>>> -Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec
>>>
>>> The above doesnt work as the job.xml for the jar still uses the default
>>> properties and not the one i specify here. Is there a different approach to
>>> override these properties. I am submitting jobs from a client machine that
>>> has the same version of configuration files as my cluster.
>>>
>>> Thanks
>>>
>>> Sid
>>>
>>
>>
>

On 06/06/2012 07:44 PM, Sid Kumar wrote:
> I am able to set it via the API.
> Configuration.setBoolean(mapred.output.compress,true). This works!
>
> But the -D from the command line still doesn't work. Any idea what I
> may be missing here?
>
> Some additional info - Also when I try running the -D on command line
> on a local cluster (pseudo distributed mode) it works, but when I try
> it on a fully distributed cluster running jobs from a client machine
> it doesn't work. Is there a different way for setting it in this case
> - in hadoop-env perhaps?
>
> Thanks
> Sid
>
> On Wed, Jun 6, 2012 at 4:06 PM, Sid Kumar <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Mayank,
>     I dont have a final tag for that property set. I looked at the
>     mapred-default.xml in the src/mapred folder and that doesn't have
>     a final tag too. Should I set it explicitly to false?
>
You should do it explicitly.
You should read the excellent blog post from Lars Francke where he did a
great job explaining parameter by parameter and why is recommendable to
set them to final.
http://gbif.blogspot.com/2011/01/setting-up-hadoop-cluster-part-1-manual.html

Regards
>
>
>     Sid
>
>
>     On Wed, Jun 6, 2012 at 3:50 PM, Mayank Bansal <[EMAIL PROTECTED]
>     <mailto:[EMAIL PROTECTED]>> wrote:
>
>         Check your mapred site xml if these parameters have
>         <final>true</final>
>
>         making final to false should solve your problem.
>
>
>         On Wed, Jun 6, 2012 at 3:41 PM, Sid Kumar <[EMAIL PROTECTED]
>         <mailto:[EMAIL PROTECTED]>> wrote:
>
>             Hi,
>             I am trying to override mapred-site.xml (more specifically
>             mapred.compress.map.output
>             and mapred.output.compression.
>             codec) from the command line when I
>             execute the jar.
>             I have been using hadoop jar <jarname> <class> -
>             Dmapred.compress.map.output=true and
>             -Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec
>
>
>             The above doesnt work as the job.xml for the jar still
>             uses the default properties and not the one i specify
>             here. Is there a different approach to override these
>             properties. I am submitting jobs from a client machine
>             that has the same version of configuration files as my
>             cluster.
>
>             Thanks
>
>             Sid
>
>
>
>

--
Marcos Luis Ort�z Valmaseda
  Data Engineer&&  Sr. System Administrator at UCI
  http://marcosluis2186.posterous.com
  http://www.linkedin.com/in/marcosluis2186
  Twitter: @marcosluis2186

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

You can also define a new config file ("mymapred.xml"), and put all the
properties you want to change there. Then you can do "hadoop --config
mymapred.xml jar {yourjar} {classname}".  Also, I add the extra properties
with a space between -D and property name, "hadoop --config mymapred.xml
jar {yourjar} {classname} -D mapred.create.symlink=yes ....     {your
program arguments}".  Hadoop program is somewhat picky when interperate
the arguments.  In your runner, try to print out all the arguments, if the
"-D..." goes there, you know that hadoop does not pick up those property
override.

Zhu, Guojun
Modeling Sr Graduate
571-3824370
[EMAIL PROTECTED]
Financial Engineering
Freddie Mac

   Marcos Ortiz <[EMAIL PROTECTED]>
   06/07/2012 09:16 AM
   Please respond to
[EMAIL PROTECTED]
To
[EMAIL PROTECTED]
cc
Sid Kumar <[EMAIL PROTECTED]>
Subject
Re: override mapred-site.xml from command line
On 06/06/2012 07:44 PM, Sid Kumar wrote:
I am able to set it via the API.
Configuration.setBoolean(mapred.output.compress,true). This works!

But the -D from the command line still doesn't work. Any idea what I may
be missing here?

Some additional info - Also when I try running the -D on command line on a
local cluster (pseudo distributed mode) it works, but when I try it on a
fully distributed cluster running jobs from a client machine it doesn't
work. Is there a different way for setting it in this case - in hadoop-env
perhaps?

Thanks
Sid

On Wed, Jun 6, 2012 at 4:06 PM, Sid Kumar <[EMAIL PROTECTED]> wrote:
Mayank,
I dont have a final tag for that property set. I looked at the
mapred-default.xml in the src/mapred folder and that doesn't have a final
tag too. Should I set it explicitly to false?
You should do it explicitly.
You should read the excellent blog post from Lars Francke where he did a
great job explaining parameter by parameter and why is recommendable to
set them to final.
http://gbif.blogspot.com/2011/01/setting-up-hadoop-cluster-part-1-manual.html
Regards

Sid
On Wed, Jun 6, 2012 at 3:50 PM, Mayank Bansal <[EMAIL PROTECTED]> wrote:
Check your mapred site xml if these parameters have <final>true</final>

making final to false should solve your problem.
On Wed, Jun 6, 2012 at 3:41 PM, Sid Kumar <[EMAIL PROTECTED]> wrote:
Hi,
I am trying to override mapred-site.xml (more specifically
mapred.compress.map.output
and mapred.output.compression.
codec) from the command line when I
execute the jar.
I have been using hadoop jar <jarname> <class> -
Dmapred.compress.map.output=true and
-Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec
The above doesnt work as the job.xml for the jar still uses the default
properties and not the one i specify here. Is there a different approach
to override these properties. I am submitting jobs from a client machine
that has the same version of configuration files as my cluster.
Thanks
Sid
--
Marcos Luis Ortíz Valmaseda
 Data Engineer && Sr. System Administrator at UCI
 http://marcosluis2186.posterous.com
 http://www.linkedin.com/in/marcosluis2186
 Twitter: @marcosluis2186

Thanks Marcus and Zhu. I set them explicitly in my program for now. The
config approach as suggested by Zhu makes sense and wil try that too.

Sid

On Thu, Jun 7, 2012 at 7:07 AM, GUOJUN Zhu <[EMAIL PROTECTED]>wrote:

>
> You can also define a new config file ("mymapred.xml"), and put all the
> properties you want to change there. Then you can do "hadoop --config
> mymapred.xml jar {yourjar} {classname}".  Also, I add the extra properties
> with a space between -D and property name, "hadoop --config mymapred.xml
> jar {yourjar} {classname} -D mapred.create.symlink=yes ....     {your
> program arguments}".  Hadoop program is somewhat picky when interperate the
> arguments.  In your runner, try to print out all the arguments, if the
> "-D..." goes there, you know that hadoop does not pick up those property
> override.
>
> Zhu, Guojun
> Modeling Sr Graduate
> 571-3824370
> [EMAIL PROTECTED]
> Financial Engineering
> Freddie Mac
>
>
>     *Marcos Ortiz <[EMAIL PROTECTED]>*
>
>    06/07/2012 09:16 AM
>     Please respond to
> [EMAIL PROTECTED]
>
>   To
> [EMAIL PROTECTED]
> cc
> Sid Kumar <[EMAIL PROTECTED]>
> Subject
> Re: override mapred-site.xml from command line
>
>
>
>
>
>
> On 06/06/2012 07:44 PM, Sid Kumar wrote:
> I am able to set it via the API.
> Configuration.setBoolean(mapred.output.compress,true). This works!
>
> But the -D from the command line still doesn't work. Any idea what I may
> be missing here?
>
> Some additional info - Also when I try running the -D on command line on a
> local cluster (pseudo distributed mode) it works, but when I try it on a
> fully distributed cluster running jobs from a client machine it doesn't
> work. Is there a different way for setting it in this case - in hadoop-env
> perhaps?
>
> Thanks
> Sid
>
> On Wed, Jun 6, 2012 at 4:06 PM, Sid Kumar <*[EMAIL PROTECTED]*<[EMAIL PROTECTED]>>
> wrote:
> Mayank,
> I dont have a final tag for that property set. I looked at the
> mapred-default.xml in the src/mapred folder and that doesn't have a final
> tag too. Should I set it explicitly to false?
> You should do it explicitly.
> You should read the excellent blog post from Lars Francke where he did a
> great job explaining parameter by parameter and why is recommendable to set
> them to final.*
> **
> http://gbif.blogspot.com/2011/01/setting-up-hadoop-cluster-part-1-manual.html
> *<http://gbif.blogspot.com/2011/01/setting-up-hadoop-cluster-part-1-manual.html>
>
> Regards
>
> Sid
>
>
> On Wed, Jun 6, 2012 at 3:50 PM, Mayank Bansal <*[EMAIL PROTECTED]*<[EMAIL PROTECTED]>>
> wrote:
> Check your mapred site xml if these parameters have <final>true</final>
>
> making final to false should solve your problem.
>
>
> On Wed, Jun 6, 2012 at 3:41 PM, Sid Kumar <*[EMAIL PROTECTED]*<[EMAIL PROTECTED]>>
> wrote:
> Hi,
> I am trying to override mapred-site.xml (more specifically
> mapred.compress.map.output
> and mapred.output.compression.
> codec) from the command line when I
> execute the jar.
> I have been using hadoop jar <jarname> <class> -
> Dmapred.compress.map.output=true and
> -Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec
>
> The above doesnt work as the job.xml for the jar still uses the default
> properties and not the one i specify here. Is there a different approach to
> override these properties. I am submitting jobs from a client machine that
> has the same version of configuration files as my cluster.
>
> Thanks
>
> Sid
>
>
>
>
>
> --
> Marcos Luis Ortíz Valmaseda
> Data Engineer && Sr. System Administrator at UCI
> *http://marcosluis2186.posterous.com*<http://marcosluis2186.posterous.com/>
> *http://www.linkedin.com/in/marcosluis2186*<http://www.linkedin.com/in/marcosluis2186>
> Twitter: @marcosluis2186
>
> <http://www.uci.cu/>
>
>