Home | About | Sematext search-lucene.com search-hadoop.com
clear query|facets|time Search criteria: .   Results from 1 to 10 from 140 (0.191s).
Loading phrases to help you
refine your search...
[expand - 1 more] - Re: com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994 - Spark - [mail # user]
...I wrote a custom class loader to find all classes that were loaded thatimplement Serializabke. I ran it locally to load all classes and registeredALL of these - I still get these issuesOn Tu...
   Author: Steve Lewis, 2014-10-29, 03:36
Re: Need some help with RecordReader - Hadoop - [mail # user]
...This InputFormat reads a Fasta file (See below)Format is a line starting >plus N lines of DataThe projects inhttps://code.google.com/p/distributed-tools/Have other samples of more complex...
   Author: Steve Lewis, 2014-10-28, 21:36
[expand - 2 more] - Re: How do you write a JavaRDD into a single file - Spark - [mail # user]
...Collect will store the entire output in a List in memory. This solution isacceptable for "Little Data" problems although if the entire problem fitsin the memory of a single machine there is ...
   Author: Steve Lewis, 2014-10-21, 16:27
How to I get at a SparkContext or better a JavaSparkContext from the middle of a function - Spark - [mail # user]
...I am running a couple of functions on an RDD which require access to dataon the file system known to the context. If I create a class with a contexta a member variable I get a serialization ...
   Author: Steve Lewis, 2014-10-14, 23:48
[expand - 1 more] - Re: Broadcast Torrent fail - then the job dies - Spark - [mail # user]
...That converts the error to the following14/10/08 13:27:40 INFO executor.Executor: Running task 3.0 in stage 0.0(TID 3)14/10/08 13:27:40 INFO broadcast.HttpBroadcast: Started reading broadcas...
   Author: Steve Lewis, 2014-10-08, 22:01
anyone else seeing something like https://issues.apache.org/jira/browse/SPARK-3637 - Spark - [mail # user]
...java.lang.NullPointerExceptionat java.nio.ByteBuffer.wrap(ByteBuffer.java:392)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)at org.apache.spark.scheduler.Task.run(Task...
   Author: Steve Lewis, 2014-10-07, 21:46
Stupid Spark question - Spark - [mail # user]
...I am porting a Hadoop job to Spark - One issue is that the workers need toread files from hdfs reading a different file based on the key or in somecases reading an object that is expensive t...
   Author: Steve Lewis, 2014-10-07, 18:01
Re: Spark and Python using generator of data bigger than RAM as input to sc.parallelize() - Spark - [mail # user]
...Try a Hadoop Custom InputFormat - I can give you some samples -While I have not tried this an input split has only a length (could beignores if the format treats as non splittable) and a Str...
   Author: Steve Lewis, 2014-10-06, 20:39
What can be done if a FlatMapFunctions generated more data that can be held in memory - Spark - [mail # user]
... I number of the problems I want to work with generate datasets which aretoo large to hold in memory. This becomes an issue when building aFlatMapFunction and also when the data used in...
   Author: Steve Lewis, 2014-10-02, 01:02
A sample for generating big data - and some design questions - Spark - [mail # user]
...This sample below is essentially word count modified to be big data byturning lines into groups ofupper case letters and then generating all case variants - it is modeledafter some real prob...
   Author: Steve Lewis, 2014-10-01, 00:17
MapReduce (74)
Hadoop (38)
Spark (23)
HDFS (5)
mail # user (140)
last 7 days (2)
last 30 days (10)
last 90 days (24)
last 6 months (24)
last 9 months (140)
Ted Yu (1708)
Harsh J (1299)
Jun Rao (1057)
Todd Lipcon (995)
Stack (978)
Jonathan Ellis (844)
Andrew Purtell (822)
Jean-Daniel Cryans (753)
Yusaku Sako (735)
stack (723)
Jarek Jarcec Cecho (703)
Eric Newton (697)
Jonathan Hsieh (674)
Neha Narkhede (673)
Roman Shaposhnik (666)
Namit Jain (649)
Hitesh Shah (627)
Steve Loughran (627)
Owen O'Malley (625)
Siddharth Seth (615)
Josh Elser (599)
Brock Noland (563)
Eli Collins (545)
Arun C Murthy (543)
Doug Cutting (536)
Steve Lewis