Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Bigtop >> mail # dev >> Review Request 17151: BIGTOP-1181. Add pyspark to spark package

Sean Mackrory 2014-01-21, 19:20
Copy link to this message
Re: Review Request 17151: BIGTOP-1181. Add pyspark to spark package

This is an automatically generated e-mail. To reply, visit:

(Updated Jan. 21, 2014, 7:22 p.m.)
Review request for bigtop.
Bugs: BIGTOP-1181
Repository: bigtop
Description (updated)

Adding a separate package "spark-python" to add pyspark. It depends on Python, and there's a mechanism to override which python is used (which I tested on RHEL 5), but the macros to activate that mechanism are not included in this patch, pending the OS support decisions we make for Bigtop 0.8.0.

  bigtop-packages/src/common/spark/install_spark.sh 75dd337
  bigtop-packages/src/deb/spark/control fc61489
  bigtop-packages/src/deb/spark/rules bd0ad6b
  bigtop-packages/src/deb/spark/spark-core.install PRE-CREATION
  bigtop-packages/src/deb/spark/spark-python.install PRE-CREATION
  bigtop-packages/src/rpm/spark/SPECS/spark.spec a8db290
  bigtop-tests/test-artifacts/package/src/main/resources/package_data.xml 166323c

Diff: https://reviews.apache.org/r/17151/diff/

I've made minor modifications based on feedback and other recent patches, but I've built these packages and run pyspark on multiple platforms. A good way to test the ability to run computation or access data is "sc.parallelize([1,2,3]).sum()" or "sc.textFile("/a-file-in-hdfs.txt").

Sean Mackrory

Mark Grover 2014-01-21, 20:03