Hello,
The release of Hadoop-1.0.4 has been voted, accepted, and posted.
It is available in SVN and Maven, as well as at
    http://www.us.apache.org/dist/hadoop/common/hadoop-1.0.4/

It is still propagating to mirrors, and should be available on all mirrors
by this time Saturday.
The documentation update is still being worked on and will be available by
Monday.

This release is noteworthy for including a Security bug fix, related to
CVE-2012-4449,
discovered by Daryn Sharp and fixed by Owen O'Malley.  The CVE announcement
is below.

Best regards,
--Matt Foley
Release Manager

*CVE-2012-4449: Apache Hadoop security token vulnerabilities
*
Severity: Critical

Vendor: The Apache Software Foundation

Versions Affected:
0.20.X: All versions
0.23: All versions before 0.23.4
1.0: All versions before 1.0.4
2.0: All versions before 2.0.2

Users affected:
Users who have enabled Hadoop's Kerberos security features.

Impact:
Malicious users may crack the secret keys used to sign security
tokens, thus granting them the ability to fabricate tokens for
privilege escalation.  Malicious users may also launch unauthorized
tasks as an arbitrary user for privilege escalation.

Description:
When Hadoop's security features are enabled, clients initially present
Kerberos credentials to authenticate to a service such as the
NameNode.  A client may then request a security token for subsequent
authentication within the Hadoop cluster.  The client receives a
security token and a corresponding signature for the token, generated
using the HMAC algorithm and a SHA1 hash.

Token passwords are generated using a trivial secret key length (20
bits).  A key of this size can be brute forced in at most a few
seconds.  Once the secret is cracked, one can generate arbitrary
tokens to impersonate other users.  These fraudulent tokens may be
used to gain unauthorized access to data or disrupt services within
the cluster.  With default secret key rolling values, a cracked secret
may often be exploited for a couple days before another secret has to
be cracked.

Some token-based services, such as the NameNode's delegation tokens
for the namespace, are immune from a compromised secret key because
they record the generated tokens.  A fraudulent token with a valid
password will rejected since the service will know it did not generate
the token.  Services that generate a token on behalf of another
service and rely on a shared secret for the other service to validate
the token's password are especially vulnerable.

HDFS (all versions):
Malicious clients cannot gain unauthorized access to the namespace.
Malicious clients may however gain full access (read, write, and
delete) to any block based on knowledge of the block id.

MapReduce (1.x):
Malicious clients may intercept task data, task logs, alter task
status, and disrupt tasks from executing or completing.  A malicious
client may also inject data into a Pipes-based job.

Yarn (2.x only):
Malicious clients may perform the same attacks as MapReduce.  An
unauthorized yarn task may be launched unbeknownst to the
ResourceManager.  Additionally, the security tokens for launching
tasks do not contain the job submitter.  The user for task execution
is specified in an untrusted container launch context, thus allowing a
task to be launched as an arbitrary user.  When combined, an
unauthorized task may be launched as an arbitrary user.

Other Hadoop projects:
Hadoop projects using the token management framework may be
susceptible if their services do not store the tokens issued, or if a
service generates tokens for other services.  This includes Apache
HBase version 0.92.0 or higher when the Kerberos-based security
features are enabled.

Mitigation:
Users should immediately upgrade to the latest applicable release
(0.23.4, 1.0.4 or later, or 2.0.2), or should immediately apply the
patch provided below to their systems.

Credit: This issue was discovered by Daryn Sharp of Yahoo! Inc.

Hi Matt,

Could you please explain what is the difference between Hadoop 1.0.4
just accepted and Hadoop 1.1.0 being
voted at the same time? Also why is it important to keep and release
both of these branches?
I am lost here. I assume other people might have that question in mind as well.

Thanks,
--Konstantin

On Fri, Oct 12, 2012 at 2:01 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> Hello,
> The release of Hadoop-1.0.4 has been voted, accepted, and posted.
> It is available in SVN and Maven, as well as at
>     http://www.us.apache.org/dist/hadoop/common/hadoop-1.0.4/
>
> It is still propagating to mirrors, and should be available on all mirrors
> by this time Saturday.
> The documentation update is still being worked on and will be available by
> Monday.
>
> This release is noteworthy for including a Security bug fix, related to
> CVE-2012-4449,
> discovered by Daryn Sharp and fixed by Owen O'Malley.  The CVE announcement
> is below.
>
> Best regards,
> --Matt Foley
> Release Manager
>
> *CVE-2012-4449: Apache Hadoop security token vulnerabilities
> *
> Severity: Critical
>
> Vendor: The Apache Software Foundation
>
> Versions Affected:
> 0.20.X: All versions
> 0.23: All versions before 0.23.4
> 1.0: All versions before 1.0.4
> 2.0: All versions before 2.0.2
>
> Users affected:
> Users who have enabled Hadoop's Kerberos security features.
>
> Impact:
> Malicious users may crack the secret keys used to sign security
> tokens, thus granting them the ability to fabricate tokens for
> privilege escalation.  Malicious users may also launch unauthorized
> tasks as an arbitrary user for privilege escalation.
>
> Description:
> When Hadoop's security features are enabled, clients initially present
> Kerberos credentials to authenticate to a service such as the
> NameNode.  A client may then request a security token for subsequent
> authentication within the Hadoop cluster.  The client receives a
> security token and a corresponding signature for the token, generated
> using the HMAC algorithm and a SHA1 hash.
>
> Token passwords are generated using a trivial secret key length (20
> bits).  A key of this size can be brute forced in at most a few
> seconds.  Once the secret is cracked, one can generate arbitrary
> tokens to impersonate other users.  These fraudulent tokens may be
> used to gain unauthorized access to data or disrupt services within
> the cluster.  With default secret key rolling values, a cracked secret
> may often be exploited for a couple days before another secret has to
> be cracked.
>
> Some token-based services, such as the NameNode's delegation tokens
> for the namespace, are immune from a compromised secret key because
> they record the generated tokens.  A fraudulent token with a valid
> password will rejected since the service will know it did not generate
> the token.  Services that generate a token on behalf of another
> service and rely on a shared secret for the other service to validate
> the token's password are especially vulnerable.
>
> HDFS (all versions):
> Malicious clients cannot gain unauthorized access to the namespace.
> Malicious clients may however gain full access (read, write, and
> delete) to any block based on knowledge of the block id.
>
> MapReduce (1.x):
> Malicious clients may intercept task data, task logs, alter task
> status, and disrupt tasks from executing or completing.  A malicious
> client may also inject data into a Pipes-based job.
>
> Yarn (2.x only):
> Malicious clients may perform the same attacks as MapReduce.  An
> unauthorized yarn task may be launched unbeknownst to the
> ResourceManager.  Additionally, the security tokens for launching
> tasks do not contain the job submitter.  The user for task execution
> is specified in an untrusted container launch context, thus allowing a
> task to be launched as an arbitrary user.  When combined, an
> unauthorized task may be launched as an arbitrary user.
>
> Other Hadoop projects:
> Hadoop projects using the token management framework may be

Yup, I was wondering about the same thing. BigTop is working on 0.3.1 release
based on Hadoop 1.1.0, so having and update for - essentially - 1.0.3 is a bit
confusing.

Thanks,
  Cos

On Sun, Oct 14, 2012 at 12:16AM, Konstantin Shvachko wrote:
> Hi Matt,
>
> Could you please explain what is the difference between Hadoop 1.0.4
> just accepted and Hadoop 1.1.0 being
> voted at the same time? Also why is it important to keep and release
> both of these branches?
> I am lost here. I assume other people might have that question in mind as well.
>
> Thanks,
> --Konstantin
>
> On Fri, Oct 12, 2012 at 2:01 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> > Hello,
> > The release of Hadoop-1.0.4 has been voted, accepted, and posted.
> > It is available in SVN and Maven, as well as at
> >     http://www.us.apache.org/dist/hadoop/common/hadoop-1.0.4/
> >
> > It is still propagating to mirrors, and should be available on all mirrors
> > by this time Saturday.
> > The documentation update is still being worked on and will be available by
> > Monday.
> >
> > This release is noteworthy for including a Security bug fix, related to
> > CVE-2012-4449,
> > discovered by Daryn Sharp and fixed by Owen O'Malley.  The CVE announcement
> > is below.
> >
> > Best regards,
> > --Matt Foley
> > Release Manager
> >
> > *CVE-2012-4449: Apache Hadoop security token vulnerabilities
> > *
> > Severity: Critical
> >
> > Vendor: The Apache Software Foundation
> >
> > Versions Affected:
> > 0.20.X: All versions
> > 0.23: All versions before 0.23.4
> > 1.0: All versions before 1.0.4
> > 2.0: All versions before 2.0.2
> >
> > Users affected:
> > Users who have enabled Hadoop's Kerberos security features.
> >
> > Impact:
> > Malicious users may crack the secret keys used to sign security
> > tokens, thus granting them the ability to fabricate tokens for
> > privilege escalation.  Malicious users may also launch unauthorized
> > tasks as an arbitrary user for privilege escalation.
> >
> > Description:
> > When Hadoop's security features are enabled, clients initially present
> > Kerberos credentials to authenticate to a service such as the
> > NameNode.  A client may then request a security token for subsequent
> > authentication within the Hadoop cluster.  The client receives a
> > security token and a corresponding signature for the token, generated
> > using the HMAC algorithm and a SHA1 hash.
> >
> > Token passwords are generated using a trivial secret key length (20
> > bits).  A key of this size can be brute forced in at most a few
> > seconds.  Once the secret is cracked, one can generate arbitrary
> > tokens to impersonate other users.  These fraudulent tokens may be
> > used to gain unauthorized access to data or disrupt services within
> > the cluster.  With default secret key rolling values, a cracked secret
> > may often be exploited for a couple days before another secret has to
> > be cracked.
> >
> > Some token-based services, such as the NameNode's delegation tokens
> > for the namespace, are immune from a compromised secret key because
> > they record the generated tokens.  A fraudulent token with a valid
> > password will rejected since the service will know it did not generate
> > the token.  Services that generate a token on behalf of another
> > service and rely on a shared secret for the other service to validate
> > the token's password are especially vulnerable.
> >
> > HDFS (all versions):
> > Malicious clients cannot gain unauthorized access to the namespace.
> > Malicious clients may however gain full access (read, write, and
> > delete) to any block based on knowledge of the block id.
> >
> > MapReduce (1.x):
> > Malicious clients may intercept task data, task logs, alter task
> > status, and disrupt tasks from executing or completing.  A malicious
> > client may also inject data into a Pipes-based job.
> >
> > Yarn (2.x only):
> > Malicious clients may perform the same attacks as MapReduce.  An
> > unauthorized yarn task may be launched unbeknownst to the

Hi Konstantin, Cos, & all,
The sole purpose of the 1.0.4 release was to provide an immediate fix for
Security issue CVE-2012-4449.  It was specifically requested by the Hadoop
security subcommittee.  Of course, the way security fixes are handled in
Hadoop, that reason couldn't be revealed until the release was actually
done.

Other than that fix, 1.0.4 has only 3 straightforward fixes for rather
severe issues, that were previously committed to branch-1.0:
    HADOOP-7154 <https://issues.apache.org/jira/browse/HADOOP-7154> - set
MALLOC_ARENA_MAX in hadoop-config.sh to resolve problems with glibc in
RHEL-6
    HDFS-3652 <https://issues.apache.org/jira/browse/HDFS-3652> - FSEditLog
failure removes the wrong edit stream when storage dirs have same name
    MAPREDUCE-4399 <https://issues.apache.org/jira/browse/MAPREDUCE-4399> - Fix
(up to 3x) performance regression in shuffle

This upgrade should be easily adopted by users who just want a simple
update to 1.0.x for the security issue.

Release 1.1.0, on the other hand, has approximately 135 enhancements and
bug fixes compared to Hadoop-1.0.4, including:

   - many performance improvements in HDFS, backported from trunk
   - improvements in Security to use SPNEGO instead of Kerberized SSL for
   HTTP transactions
   - lower default minimum heartbeat for task trackers from 3 sec to
   300msec to increase job throughput on small clusters
   - port Gridmix v3
   - set MALLOC_ARENA_MAX in hadoop-config.sh to resolve problems with
   glibc in RHEL-6
   - splittable bzip2 files

This is a significant release with a lot of great improvements.  Of course
it also has the security fix.  We can expect that 1.0.x users will want to
upgrade to 1.1.0 to get the many improvements, but it may take longer than
an update to 1.0.4.  In order to get the fix for CVE-2012-4449 into
circulation as soon as possible, it made sense to release 1.0.4 as well.

Thanks,
--Matt

On Sun, Oct 14, 2012 at 7:41 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> Yup, I was wondering about the same thing. BigTop is working on 0.3.1
> release
> based on Hadoop 1.1.0, so having and update for - essentially - 1.0.3 is a
> bit
> confusing.
>
> Thanks,
>   Cos
>
> On Sun, Oct 14, 2012 at 12:16AM, Konstantin Shvachko wrote:
> > Hi Matt,
> >
> > Could you please explain what is the difference between Hadoop 1.0.4
> > just accepted and Hadoop 1.1.0 being
> > voted at the same time? Also why is it important to keep and release
> > both of these branches?
> > I am lost here. I assume other people might have that question in mind
> as well.
> >
> > Thanks,
> > --Konstantin
> >
> > On Fri, Oct 12, 2012 at 2:01 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> > > Hello,
> > > The release of Hadoop-1.0.4 has been voted, accepted, and posted.
> > > It is available in SVN and Maven, as well as at
> > >     http://www.us.apache.org/dist/hadoop/common/hadoop-1.0.4/
> > >
> > > It is still propagating to mirrors, and should be available on all
> mirrors
> > > by this time Saturday.
> > > The documentation update is still being worked on and will be
> available by
> > > Monday.
> > >
> > > This release is noteworthy for including a Security bug fix, related to
> > > CVE-2012-4449,
> > > discovered by Daryn Sharp and fixed by Owen O'Malley.  The CVE
> announcement
> > > is below.
> > >
> > > Best regards,
> > > --Matt Foley
> > > Release Manager
> > >
> > > *CVE-2012-4449: Apache Hadoop security token vulnerabilities
> > > *
> > > Severity: Critical
> > >
> > > Vendor: The Apache Software Foundation
> > >
> > > Versions Affected:
> > > 0.20.X: All versions
> > > 0.23: All versions before 0.23.4
> > > 1.0: All versions before 1.0.4
> > > 2.0: All versions before 2.0.2
> > >
> > > Users affected:
> > > Users who have enabled Hadoop's Kerberos security features.
> > >
> > > Impact:
> > > Malicious users may crack the secret keys used to sign security
> > > tokens, thus granting them the ability to fabricate tokens for
> > > privilege escalation.  Malicious users may also launch unauthorized

Thanks Matt - makes perfect sense!

Cos

On Sun, Oct 14, 2012 at 09:24PM, Matt Foley wrote:
> Hi Konstantin, Cos, & all,
> The sole purpose of the 1.0.4 release was to provide an immediate fix for
> Security issue CVE-2012-4449.  It was specifically requested by the Hadoop
> security subcommittee.  Of course, the way security fixes are handled in
> Hadoop, that reason couldn't be revealed until the release was actually
> done.
>
> Other than that fix, 1.0.4 has only 3 straightforward fixes for rather
> severe issues, that were previously committed to branch-1.0:
>     HADOOP-7154 <https://issues.apache.org/jira/browse/HADOOP-7154> - set
> MALLOC_ARENA_MAX in hadoop-config.sh to resolve problems with glibc in
> RHEL-6
>     HDFS-3652 <https://issues.apache.org/jira/browse/HDFS-3652> - FSEditLog
> failure removes the wrong edit stream when storage dirs have same name
>     MAPREDUCE-4399 <https://issues.apache.org/jira/browse/MAPREDUCE-4399> - Fix
> (up to 3x) performance regression in shuffle
>
> This upgrade should be easily adopted by users who just want a simple
> update to 1.0.x for the security issue.
>
> Release 1.1.0, on the other hand, has approximately 135 enhancements and
> bug fixes compared to Hadoop-1.0.4, including:
>
>    - many performance improvements in HDFS, backported from trunk
>    - improvements in Security to use SPNEGO instead of Kerberized SSL for
>    HTTP transactions
>    - lower default minimum heartbeat for task trackers from 3 sec to
>    300msec to increase job throughput on small clusters
>    - port Gridmix v3
>    - set MALLOC_ARENA_MAX in hadoop-config.sh to resolve problems with
>    glibc in RHEL-6
>    - splittable bzip2 files
>
> This is a significant release with a lot of great improvements.  Of course
> it also has the security fix.  We can expect that 1.0.x users will want to
> upgrade to 1.1.0 to get the many improvements, but it may take longer than
> an update to 1.0.4.  In order to get the fix for CVE-2012-4449 into
> circulation as soon as possible, it made sense to release 1.0.4 as well.
>
> Thanks,
> --Matt
>
> On Sun, Oct 14, 2012 at 7:41 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
> > Yup, I was wondering about the same thing. BigTop is working on 0.3.1
> > release
> > based on Hadoop 1.1.0, so having and update for - essentially - 1.0.3 is a
> > bit
> > confusing.
> >
> > Thanks,
> >   Cos
> >
> > On Sun, Oct 14, 2012 at 12:16AM, Konstantin Shvachko wrote:
> > > Hi Matt,
> > >
> > > Could you please explain what is the difference between Hadoop 1.0.4
> > > just accepted and Hadoop 1.1.0 being
> > > voted at the same time? Also why is it important to keep and release
> > > both of these branches?
> > > I am lost here. I assume other people might have that question in mind
> > as well.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > > On Fri, Oct 12, 2012 at 2:01 PM, Matt Foley <[EMAIL PROTECTED]> wrote:
> > > > Hello,
> > > > The release of Hadoop-1.0.4 has been voted, accepted, and posted.
> > > > It is available in SVN and Maven, as well as at
> > > >     http://www.us.apache.org/dist/hadoop/common/hadoop-1.0.4/
> > > >
> > > > It is still propagating to mirrors, and should be available on all
> > mirrors
> > > > by this time Saturday.
> > > > The documentation update is still being worked on and will be
> > available by
> > > > Monday.
> > > >
> > > > This release is noteworthy for including a Security bug fix, related to
> > > > CVE-2012-4449,
> > > > discovered by Daryn Sharp and fixed by Owen O'Malley.  The CVE
> > announcement
> > > > is below.
> > > >
> > > > Best regards,
> > > > --Matt Foley
> > > > Release Manager
> > > >
> > > > *CVE-2012-4449: Apache Hadoop security token vulnerabilities
> > > > *
> > > > Severity: Critical
> > > >
> > > > Vendor: The Apache Software Foundation
> > > >
> > > > Versions Affected:
> > > > 0.20.X: All versions
> > > > 0.23: All versions before 0.23.4
> > > > 1.0: All versions before 1.0.4
> > > > 2.0: All versions before 2.0.2