Home | About | Sematext search-lucene.com search-hadoop.com
clear query|facets|time Search criteria: .   Results from 1 to 10 from 13 (0.136s).
Loading phrases to help you
refine your search...
[expand - 1 more] - Re: Computing mean and standard deviation by key - Spark - [mail # user]
...I meant not sure how to do variance in one shot :-)With mean in hand, you can obvious broadcast the variable, and do anothermap/reduce to calculate variance per key.On Fri, Aug 1, 2014 at 4:...
   Author: Xu, 2014-08-01, 20:49
[expand - 3 more] - Re: access hdfs file name in map() - Spark - [mail # user]
...Hi Roberto,Ultimately, the info you need is set here:https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala#L69Being a spark newbie, I extended ...
   Author: Xu, 2014-08-01, 17:43
Task progress in ipython? - Spark - [mail # user]
...I am pretty happy with using pyspark with ipython notebook. The only issueis that I need to look at the console output or spark ui to track taskprogress. I wonder if anyone thought of or bet...
   Author: Xu, 2014-06-26, 23:09
performance difference between spark-shell and spark-submit - Spark - [mail # user]
...Hi all,I implemented a transformation on hdfs files with spark. First tested inspark-shell (with yarn), I implemented essentially the same logic with aspark program (scala), built a jar file...
   Author: Xu, 2014-06-10, 03:19
[expand - 1 more] - Re: cache spark sql parquet file in memory? - Spark - [mail # user]
...Is there a way to start tachyon on top of a yarn cluster? On Jun 7, 2014 2:11 PM, "Marek Wiewiorka" wrote: ...
   Author: Xu, 2014-06-07, 21:24
[expand - 1 more] - Re: spark worker and yarn memory - Spark - [mail # user]
...Nice explanation... Thanks!On Thu, Jun 5, 2014 at 5:50 PM, Sandy Ryza  wrote: ...
   Author: Xu, 2014-06-06, 03:35
[expand - 1 more] - Re: compress in-memory cache? - Spark - [mail # user]
...Thanks.. it works now.-SimonOn Thu, Jun 5, 2014 at 10:47 AM, Nick Pentreath wrote: ...
   Author: Xu, 2014-06-05, 16:00
Re: Join : Giving incorrect result - Spark - [mail # user]
...Maybe your two workers have different assembly jar files?I just ran into a similar problem that my spark-shell is using a differentjar file than my workers - got really confusing results.On ...
   Author: Xu, 2014-06-04, 19:58
[expand - 5 more] - Re: pyspark problems on yarn (job not parallelized, and Py4JJavaError) - Spark - [mail # user]
...Nope... didn't try java 6. The standard installation guide didn't sayanything about java 7 and suggested to do "-DskipTests" for the build..http://spark.apache.org/docs/latest/building-with-...
   Author: Xu, 2014-06-02, 20:15
[expand - 4 more] - Re: spark 1.0.0 on yarn - Spark - [mail # user]
...I built my new package like this:"mvn -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0-cdh5.0.1 -DskipTests cleanpackage"Spark-shell is working now, but pyspark is still broken. I reported theprob...
   Author: Xu, 2014-06-02, 18:52
Sort:
project
Spark (10)
Flume (3)
type
mail # user (13)
date
last 7 days (0)
last 30 days (0)
last 90 days (3)
last 6 months (10)
last 9 months (13)
author
Ted Yu (1642)
Harsh J (1293)
Jun Rao (1033)
Todd Lipcon (1002)
Stack (973)
Jonathan Ellis (842)
Andrew Purtell (797)
Jean-Daniel Cryans (754)
jacques@... (738)
stack (716)
Yusaku Sako (708)
Jarek Jarcec Cecho (698)
Eric Newton (697)
Jonathan Hsieh (675)
Roman Shaposhnik (659)
Brock Noland (656)
Namit Jain (649)
Neha Narkhede (647)
Hitesh Shah (626)
Owen O'Malley (625)
Steve Loughran (616)
Siddharth Seth (614)
Josh Elser (565)
Eli Collins (545)
Arun C Murthy (543)
Xu