Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - profiling hdfs write path


+
Radim Kolar 2012-11-25, 21:41
+
Todd Lipcon 2012-11-26, 01:35
+
Radim Kolar 2012-11-29, 15:17
+
Todd Lipcon 2012-11-29, 18:25
+
Radim Kolar 2012-12-04, 17:07
+
Todd Lipcon 2012-12-04, 17:27
+
Suresh Srinivas 2012-12-04, 17:49
+
Radim Kolar 2012-12-04, 17:39
+
Eli Collins 2012-12-04, 17:44
+
Radim Kolar 2012-12-05, 02:00
+
Andy Isaacson 2012-12-05, 22:21
+
Radim Kolar 2012-12-06, 02:02
Copy link to this message
-
Re: profiling hdfs write path
Andy Isaacson 2012-12-06, 23:06
On Wed, Dec 5, 2012 at 6:02 PM, Radim Kolar <[EMAIL PROTECTED]> wrote:
> YARN-223 <https://issues.apache.org/jira/browse/YARN-223>
> YARN-211 <https://issues.apache.org/jira/browse/YARN-211>
> YARN-210 <https://issues.apache.org/jira/browse/YARN-210>
> MAPREDUCE-4839 <https://issues.apache.org/jira/browse/MAPREDUCE-4839>
> MAPREDUCE-4827 <https://issues.apache.org/jira/browse/MAPREDUCE-4827>
> MAPREDUCE-4594 <https://issues.apache.org/jira/browse/MAPREDUCE-4594>
> MAPREDUCE-3968 <https://issues.apache.org/jira/browse/MAPREDUCE-3968>

I don't really know the YARN or MAPREDUCE code bases so I'm going to
pass on those ones...

> HADOOP-9088 <https://issues.apache.org/jira/browse/HADOOP-9088>

Todd asked a pretty reasonable question that I don't see an answer to
-- where will murmur3 actually be used? We generally don't add code,
even if it's good code that we're sure to need someday, until there's
an actual user for it.

> HADOOP-9041 <https://issues.apache.org/jira/browse/HADOOP-9041>

There needs to be a complete, up-to-date patch uploaded. This one
seems to have two patches that need to be applied to get a working
commit -- HADOOP-9041.patch and fsinit-unit.txt. Also the latter has a
misspelled classname, Initialization is spelled with a "t" rather than
a "c".

It would be really good to develop a JUnit test that fails reliably
both under mvn and Eclipse that shows the problem to avoid regressions
in the future... even if the unit test has to do moderately unclean
things to force the failure. (But that's not a hard requirement, if
it's really impossible to do the current situation is OK.)

> HADOOP-8698 <https://issues.apache.org/jira/browse/HADOOP-8698>

I don't understand this patch at all.  Since it makes the constructor
vacuous, why not just delete the constructor entirely? If avoiding the
possible "could be null" makes other code simpler, go ahead and
include the simplification in this patch. (see below for more on
including stuff in a single jira.)

Generally if Jenkins posts a -1 on a patch, you should follow up with
a comment explaining why it's OK for this patch to fail the given
test. For example I had a change recently that fixed an intermittent
test failure, so I didn't need to add a test. Jenkins said "-1 no
tests included" and I commented "fixes TestFoo intermittent failures".

One of the ways the community has compensated for the heavyweight JIRA
process is to allow a single JIRA to include more change than I would
normally put into a git commit. I do my development locally in a
per-jira branch "hdfs1337" with normal small git-style commits, and
then when I'm ready to post a patch I "git diff
upstream/trunk..hdfs1337 > hdfs1337.txt" to squash all the sane git
commits into a single large diff to upload.

Thanks,
-andy
+
Radim Kolar 2012-12-08, 04:39
+
Steve Loughran 2012-12-08, 12:38
+
Steve Loughran 2012-12-05, 08:57
+
Radim Kolar 2012-11-26, 03:07