Sign Up
Log In
Log In
or
Sign Up
Places
All Projects
Status Monitor
Collapse sidebar
home:aagate
spark-kit
README.building
Overview
Repositories
Revisions
Requests
Users
Attributes
Meta
File README.building of Package spark-kit
This package provides build dependencies for building Apache Spark using Tetra (ruby2.2-rubygem-tetra). Using Tetra is neccessary due the Maven-based build process downloading dependencies at build time. Tetra will keep track of these downloaded dependencies during a so-called "dry-run" build on the maintainer's machine and generate a tarball (spark-kit.tar.xz) containing all of them. This tarball is then used to allow the same build to run in an offline manner in OBS (where there is no Internet connectivity on purpose in order to allow for reproducible builds). To generate spark-kit.tar.xz for a new version of Spark, proceed as follows (you will need to have Tetra installed for this): Note: these steps need to be performed on the machine where you keep your spark{,-kit} working copy. 0) If not installed, install tetra gem. sudo gem install tetra If having path trouble, use the full path to the tetra script in the following commands. /usr/lib64/ruby/gems/2.5.0/gems/tetra-2.0.5/bin/tetra 1) Download an updated Spark tarball. We'll assume the new tarball's version is 2.2.3 throughout the rest of this file, which is the version it was at the time of this writing. Note this is the same .tgz file that is checked in to the base spark project and should be updated there to match versions. Reference the spark/spark.spec file for the Source0 url. 2) Initialize a Tetra build directory for your new spark tarball by running these commands from your working copy. (Note that the colons in OBS checkout directory names can cause issues building packages, so you are advised to run this process from within /tmp or another location containing your source tarball.) tetra init spark spark-2.2.3.tgz 6) Perform a dry-run build with Tetra: cd spark/src/spark-2.2.3 tetra dry-run dev/change-scala-version.sh 2.10 mvn clean install -DskipTests -Phive -Phive-thriftserver -Dscala-2.10 -Dkafka-0-8 mvn dependency:go-offline exit 7) Generate the new kit tarball: tetra generate-all You will now find an updated kit tarball in spark/packages/spark-kit.tar.xz Substitute this tarball for the existing one in the kit package and bump its version. Also, ensure that any jar paths and names are updated both in the spark project as well as any services which depend on spark (such as openstack-monasca-transform).
Locations
Projects
Search
Status Monitor
Help
OpenBuildService.org
Documentation
API Documentation
Code of Conduct
Contact
Support
@OBShq
Terms
openSUSE Build Service is sponsored by
The Open Build Service is an
openSUSE project
.
Sign Up
Log In
Places
Places
All Projects
Status Monitor