After the "project unsplit" this weekend, we're now back to a place where we
have a single SVN/git tree that encompasses all of the subprojects. This
opens up the next question: should we merge the JIRAs and allow a single
issue to have a patch which spans projects?

My thoughts are:
- the biggest pain point with the project split is dealing with
cross-project patches
- one of the biggest reasons we did the project split was that the combined
traffic from the HADOOP JIRA was hard to follow for people who really care
about certain subprojects.
- the jira split is a coarse-grained way of allowing people to watch just
the sub-areas they care about.

So, I was thinking the following... what if there were a way to watch JIRAs
based on subtrees? I'm imagining a web page where any community user could
have an account and manage a "watch list" of subtrees. If you want to watch
all MR jiras, you could simply watch mapreduce/*. If you care only about
libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would watch
all patches attached to JIRA, and any time a patch is uploaded that touches
something on your watch list, it automatically adds you as a watcher on the
ticket and sends you a notification via email. It would also be easy to set
up a watch based on patch size, for example.

I think even if we don't recombine the JIRAs, this might be a handy way to
cut down on mailing list traffic for contributors who have a more narrow
focus on certain areas of the code.

Does this sound useful? I don't know if/when I'd have time to build such a
thing, but if the community thinks it would be really helpful, I might
become inspired.

-Todd
--
Todd Lipcon
Software Engineer, Cloudera

I tend to agree: JIRA separation was the benefit of the split.

I'd rather keep the current JIRA split in effect (e.g. separate JIRA projects
for separate Hadoop components; don't recombine them) and file patches in the
same way (for common, hdfs, mapreduce). If a cross component patch is needed
then HADOOP project JIRA can be used for tracking, patches, etc.

Tree-based watch-list seems like a great idea, but won't it narrow the scope
somehow? Are you saying that if I am interested in say hdfs/src/c++/libhdfs,
but a JIRA is open which affects libhdfs and something else (e.g. NameNode) I
will still get the notification?

Cos

On Mon, Jun 13, 2011 at 11:28AM, Todd Lipcon wrote:
> After the "project unsplit" this weekend, we're now back to a place where we
> have a single SVN/git tree that encompasses all of the subprojects. This
> opens up the next question: should we merge the JIRAs and allow a single
> issue to have a patch which spans projects?
>
> My thoughts are:
> - the biggest pain point with the project split is dealing with
> cross-project patches
> - one of the biggest reasons we did the project split was that the combined
> traffic from the HADOOP JIRA was hard to follow for people who really care
> about certain subprojects.
> - the jira split is a coarse-grained way of allowing people to watch just
> the sub-areas they care about.
>
> So, I was thinking the following... what if there were a way to watch JIRAs
> based on subtrees? I'm imagining a web page where any community user could
> have an account and manage a "watch list" of subtrees. If you want to watch
> all MR jiras, you could simply watch mapreduce/*. If you care only about
> libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would watch
> all patches attached to JIRA, and any time a patch is uploaded that touches
> something on your watch list, it automatically adds you as a watcher on the
> ticket and sends you a notification via email. It would also be easy to set
> up a watch based on patch size, for example.
>
> I think even if we don't recombine the JIRAs, this might be a handy way to
> cut down on mailing list traffic for contributors who have a more narrow
> focus on certain areas of the code.
>
> Does this sound useful? I don't know if/when I'd have time to build such a
> thing, but if the community thinks it would be really helpful, I might
> become inspired.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera

On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> I tend to agree: JIRA separation was the benefit of the split.
>
> I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> projects
> for separate Hadoop components; don't recombine them) and file patches in
> the
> same way (for common, hdfs, mapreduce). If a cross component patch is
> needed
> then HADOOP project JIRA can be used for tracking, patches, etc.
>

Yea, perhaps we just need the QA bot to be smart enough that it could handle
a cross-project patch attached to HADOOP? Maybe we do something crazy and
make a new HADOOPCROSS jira for patches that affect multiple projects? (just
brainstorming here...)
> Tree-based watch-list seems like a great idea, but won't it narrow the
> scope
> somehow? Are you saying that if I am interested in say
> hdfs/src/c++/libhdfs,
> but a JIRA is open which affects libhdfs and something else (e.g. NameNode)
> I
> will still get the notification?
>

Right, that's the idea. You'd be added as a watcher (and get notified) for
any patch that touches the area you care about, regardless of whether it
also touches some other areas.

-Todd

On Mon, Jun 13, 2011 at 11:28AM, Todd Lipcon wrote:
> > After the "project unsplit" this weekend, we're now back to a place where
> we
> > have a single SVN/git tree that encompasses all of the subprojects. This
> > opens up the next question: should we merge the JIRAs and allow a single
> > issue to have a patch which spans projects?
> >
> > My thoughts are:
> > - the biggest pain point with the project split is dealing with
> > cross-project patches
> > - one of the biggest reasons we did the project split was that the
> combined
> > traffic from the HADOOP JIRA was hard to follow for people who really
> care
> > about certain subprojects.
> > - the jira split is a coarse-grained way of allowing people to watch just
> > the sub-areas they care about.
> >
> > So, I was thinking the following... what if there were a way to watch
> JIRAs
> > based on subtrees? I'm imagining a web page where any community user
> could
> > have an account and manage a "watch list" of subtrees. If you want to
> watch
> > all MR jiras, you could simply watch mapreduce/*. If you care only about
> > libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would
> watch
> > all patches attached to JIRA, and any time a patch is uploaded that
> touches
> > something on your watch list, it automatically adds you as a watcher on
> the
> > ticket and sends you a notification via email. It would also be easy to
> set
> > up a watch based on patch size, for example.
> >
> > I think even if we don't recombine the JIRAs, this might be a handy way
> to
> > cut down on mailing list traffic for contributors who have a more narrow
> > focus on certain areas of the code.
> >
> > Does this sound useful? I don't know if/when I'd have time to build such
> a
> > thing, but if the community thinks it would be really helpful, I might
> > become inspired.
> >
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>

--
Todd Lipcon
Software Engineer, Cloudera

On Mon, Jun 13, 2011 at 1:37 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Maybe we do something crazy and  make a new HADOOPCROSS jira for patches that affect multiple projects? (just
> brainstorming here...)

Sounds like a good idea. Having separate JIRA components for common,
hdfs and mapreduce is a way to assert code locality. Having a HADOOPX
jira component make the cross-componentness explicit (and making CI
easier), which is a good thing, IMO.

__Luke

On Mon, Jun 13, 2011 at 01:37PM, Todd Lipcon wrote:
> On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
> > I tend to agree: JIRA separation was the benefit of the split.
> >
> > I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> > projects
> > for separate Hadoop components; don't recombine them) and file patches in
> > the
> > same way (for common, hdfs, mapreduce). If a cross component patch is
> > needed
> > then HADOOP project JIRA can be used for tracking, patches, etc.
> >
>
> Yea, perhaps we just need the QA bot to be smart enough that it could handle
> a cross-project patch attached to HADOOP? Maybe we do something crazy and
> make a new HADOOPCROSS jira for patches that affect multiple projects? (just
> brainstorming here...)

Correct me if I'm wrong but in the new structure cross-component patch differs
from a component one by a patch level (i.e. p0 vs p1 if looked from
common/trunk), right? I guess the bot can be hacked to use this distinction
thus saving us an extra JIRA project which will merely serve the purpose of
meta-project.

cos

> > Tree-based watch-list seems like a great idea, but won't it narrow the
> > scope
> > somehow? Are you saying that if I am interested in say
> > hdfs/src/c++/libhdfs,
> > but a JIRA is open which affects libhdfs and something else (e.g. NameNode)
> > I
> > will still get the notification?
> >
>
> Right, that's the idea. You'd be added as a watcher (and get notified) for
> any patch that touches the area you care about, regardless of whether it
> also touches some other areas.
>
> -Todd
>
> On Mon, Jun 13, 2011 at 11:28AM, Todd Lipcon wrote:
> > > After the "project unsplit" this weekend, we're now back to a place where
> > we
> > > have a single SVN/git tree that encompasses all of the subprojects. This
> > > opens up the next question: should we merge the JIRAs and allow a single
> > > issue to have a patch which spans projects?
> > >
> > > My thoughts are:
> > > - the biggest pain point with the project split is dealing with
> > > cross-project patches
> > > - one of the biggest reasons we did the project split was that the
> > combined
> > > traffic from the HADOOP JIRA was hard to follow for people who really
> > care
> > > about certain subprojects.
> > > - the jira split is a coarse-grained way of allowing people to watch just
> > > the sub-areas they care about.
> > >
> > > So, I was thinking the following... what if there were a way to watch
> > JIRAs
> > > based on subtrees? I'm imagining a web page where any community user
> > could
> > > have an account and manage a "watch list" of subtrees. If you want to
> > watch
> > > all MR jiras, you could simply watch mapreduce/*. If you care only about
> > > libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would
> > watch
> > > all patches attached to JIRA, and any time a patch is uploaded that
> > touches
> > > something on your watch list, it automatically adds you as a watcher on
> > the
> > > ticket and sends you a notification via email. It would also be easy to
> > set
> > > up a watch based on patch size, for example.
> > >
> > > I think even if we don't recombine the JIRAs, this might be a handy way
> > to
> > > cut down on mailing list traffic for contributors who have a more narrow
> > > focus on certain areas of the code.
> > >
> > > Does this sound useful? I don't know if/when I'd have time to build such
> > a
> > > thing, but if the community thinks it would be really helpful, I might
> > > become inspired.
> > >
> > > -Todd
> > > --
> > > Todd Lipcon
> > > Software Engineer, Cloudera
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

On Mon, Jun 13, 2011 at 4:54 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> On Mon, Jun 13, 2011 at 01:37PM, Todd Lipcon wrote:
> > On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <[EMAIL PROTECTED]>
> wrote:
> >
> > > I tend to agree: JIRA separation was the benefit of the split.
> > >
> > > I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> > > projects
> > > for separate Hadoop components; don't recombine them) and file patches
> in
> > > the
> > > same way (for common, hdfs, mapreduce). If a cross component patch is
> > > needed
> > > then HADOOP project JIRA can be used for tracking, patches, etc.
> > >
> >
> > Yea, perhaps we just need the QA bot to be smart enough that it could
> handle
> > a cross-project patch attached to HADOOP? Maybe we do something crazy and
> > make a new HADOOPCROSS jira for patches that affect multiple projects?
> (just
> > brainstorming here...)
>
> Correct me if I'm wrong but in the new structure cross-component patch
> differs
> from a component one by a patch level (i.e. p0 vs p1 if looked from
> common/trunk), right? I guess the bot can be hacked to use this distinction
> thus saving us an extra JIRA project which will merely serve the purpose of
> meta-project.
>
>
Yes, I am about to commit HADOOP-7384 which can at least deal with patches
relative to either trunk/ or trunk/<project>. But, it will also detect a
cross-project patch and barf.

It could certainly be extended to apply and test a cross-project patch,
though it would be substantially more work.

The advantage of a separate HADOOPX jira would be to allow people to notice
cross-project patches. For example, a dev who primarily works on HDFS may
not subscribe to mapreduce-dev or mapreduce-issues, but if an MR issue is
going to modify something in the HDFS codebase, he or she will certainly
want to be aware of it.

-Todd
> > > Tree-based watch-list seems like a great idea, but won't it narrow the
> > > scope
> > > somehow? Are you saying that if I am interested in say
> > > hdfs/src/c++/libhdfs,
> > > but a JIRA is open which affects libhdfs and something else (e.g.
> NameNode)
> > > I
> > > will still get the notification?
> > >
> >
> > Right, that's the idea. You'd be added as a watcher (and get notified)
> for
> > any patch that touches the area you care about, regardless of whether it
> > also touches some other areas.
> >
> > -Todd
> >
> > On Mon, Jun 13, 2011 at 11:28AM, Todd Lipcon wrote:
> > > > After the "project unsplit" this weekend, we're now back to a place
> where
> > > we
> > > > have a single SVN/git tree that encompasses all of the subprojects.
> This
> > > > opens up the next question: should we merge the JIRAs and allow a
> single
> > > > issue to have a patch which spans projects?
> > > >
> > > > My thoughts are:
> > > > - the biggest pain point with the project split is dealing with
> > > > cross-project patches
> > > > - one of the biggest reasons we did the project split was that the
> > > combined
> > > > traffic from the HADOOP JIRA was hard to follow for people who really
> > > care
> > > > about certain subprojects.
> > > > - the jira split is a coarse-grained way of allowing people to watch
> just
> > > > the sub-areas they care about.
> > > >
> > > > So, I was thinking the following... what if there were a way to watch
> > > JIRAs
> > > > based on subtrees? I'm imagining a web page where any community user
> > > could
> > > > have an account and manage a "watch list" of subtrees. If you want to
> > > watch
> > > > all MR jiras, you could simply watch mapreduce/*. If you care only
> about
> > > > libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would
> > > watch
> > > > all patches attached to JIRA, and any time a patch is uploaded that
> > > touches
> > > > something on your watch list, it automatically adds you as a watcher
> on
> > > the
> > > > ticket and sends you a notification via email. It would also be easy
> to
> > > set
> > > > up a watch based on patch size, for example.

Todd Lipcon
Software Engineer, Cloudera

Project un-split definitely simplifies things.

Todd, if people add a watch based on patches, would they not miss notifictions for those entries in an earlier phase of their lifecycle?
For example when issues are just reported, discussed and assigned, but no patch has been attached yet?

A separate HADOOPX Jira project would eliminate such issues.

It does raise another question though: What happens if an issue starts out in one area, and then turns out to require changes in other areas?
Would one then first create a HADOOP-x, a HDFS-y, or MAPREDUCE-z and then when it turns out other components are involved a new HADOOPX- referring to such earlier Jira?

Cheers,

Joep

________________________________________
From: Todd Lipcon [[EMAIL PROTECTED]]
Sent: Monday, June 13, 2011 1:37 PM
To: [EMAIL PROTECTED]
Subject: Re: JIRAs post-"unsplit"

On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> I tend to agree: JIRA separation was the benefit of the split.
>
> I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> projects
> for separate Hadoop components; don't recombine them) and file patches in
> the
> same way (for common, hdfs, mapreduce). If a cross component patch is
> needed
> then HADOOP project JIRA can be used for tracking, patches, etc.
>

Yea, perhaps we just need the QA bot to be smart enough that it could handle
a cross-project patch attached to HADOOP? Maybe we do something crazy and
make a new HADOOPCROSS jira for patches that affect multiple projects? (just
brainstorming here...)
> Tree-based watch-list seems like a great idea, but won't it narrow the
> scope
> somehow? Are you saying that if I am interested in say
> hdfs/src/c++/libhdfs,
> but a JIRA is open which affects libhdfs and something else (e.g. NameNode)
> I
> will still get the notification?
>

Right, that's the idea. You'd be added as a watcher (and get notified) for
any patch that touches the area you care about, regardless of whether it
also touches some other areas.

-Todd

On Tue, Jun 14, 2011 at 9:35 AM, Rottinghuis, Joep <[EMAIL PROTECTED]>wrote:

> Project un-split definitely simplifies things.
>
> Todd, if people add a watch based on patches, would they not miss
> notifictions for those entries in an earlier phase of their lifecycle?
> For example when issues are just reported, discussed and assigned, but no
> patch has been attached yet?
>
>
Another thought that Alejandro just suggested offline is to use JIRA
components rather than just the file paths. So, assuming there is a bot that
watches the JIRA, it would be easy enough to allow you to permawatch a
component (JIRA itself doesn't give this option).

Then, assuming the patch is assigned the right components, it will be seen
by people who care early on. If it's not given the right components, then it
will be seen once you upload a patch.

> A separate HADOOPX Jira project would eliminate such issues.
>
> It does raise another question though: What happens if an issue starts out
> in one area, and then turns out to require changes in other areas?
> Would one then first create a HADOOP-x, a HDFS-y, or MAPREDUCE-z and then
> when it turns out other components are involved a new HADOOPX- referring to
> such earlier Jira?
>
> Cheers,
>
> Joep
>
> ________________________________________
> From: Todd Lipcon [[EMAIL PROTECTED]]
> Sent: Monday, June 13, 2011 1:37 PM
> To: [EMAIL PROTECTED]
> Subject: Re: JIRAs post-"unsplit"
>
> On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <[EMAIL PROTECTED]>
> wrote:
>
> > I tend to agree: JIRA separation was the benefit of the split.
> >
> > I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> > projects
> > for separate Hadoop components; don't recombine them) and file patches in
> > the
> > same way (for common, hdfs, mapreduce). If a cross component patch is
> > needed
> > then HADOOP project JIRA can be used for tracking, patches, etc.
> >
>
> Yea, perhaps we just need the QA bot to be smart enough that it could
> handle
> a cross-project patch attached to HADOOP? Maybe we do something crazy and
> make a new HADOOPCROSS jira for patches that affect multiple projects?
> (just
> brainstorming here...)
>
>
> > Tree-based watch-list seems like a great idea, but won't it narrow the
> > scope
> > somehow? Are you saying that if I am interested in say
> > hdfs/src/c++/libhdfs,
> > but a JIRA is open which affects libhdfs and something else (e.g.
> NameNode)
> > I
> > will still get the notification?
> >
>
> Right, that's the idea. You'd be added as a watcher (and get notified) for
> any patch that touches the area you care about, regardless of whether it
> also touches some other areas.
>
> -Todd
>

--
Todd Lipcon
Software Engineer, Cloudera