File README.building of Package spark-kit

Overview Repositories Revisions Requests Users Attributes Meta

File README.building of Package spark-kit (Revision 963948462d2098cb27c65ad3f27799a9)

Currently displaying revision 963948462d2098cb27c65ad3f27799a9 , Show latest

This package provides build dependencies for building Apache Spark
using Tetra (ruby2.2-rubygem-tetra). Using Tetra is neccessary due
the Maven-based build process downloading dependencies at build time.
Tetra will keep track of these downloaded dependencies during a so-called
"dry-run" build on the maintainer's machine and generate a tarball
(spark-kit.tar.xz) containing all of them. This tarball is then used to
allow the same build to run in an offline manner in OBS (where there is no
Internet connectivity on purpose in order to allow for reproducible
builds). To generate spark-kit.tar.xz for a new version of Spark, proceed
as follows (you will need to have Tetra installed for this):

Note: these steps need to be performed on the machine where you keep your
spark{,-kit} working copy.

0) If not installed, install tetra gem.

sudo gem install tetra

If having path trouble, use the full path to the tetra script in the
   following commands.

/usr/lib64/ruby/gems/2.5.0/gems/tetra-2.0.5/bin/tetra

1) Download an updated Spark tarball. We'll assume the new tarball's version
   is 2.2.3 throughout the rest of this file, which is the version it was at
   the time of this writing.

Note this is the same .tgz file that is checked in to the base spark
   project and should be updated there to match versions. Reference the
   spark/spark.spec file for the Source0 url.

2) Initialize a Tetra build directory for your new spark tarball by running
   these commands from your working copy. (Note that the colons in OBS
   checkout directory names can cause issues building packages, so you are
   advised to run this process from within /tmp or another location
   containing your source tarball.)

tetra init spark spark-2.2.3.tgz

6) Perform a dry-run build with Tetra:

cd spark/src/spark-2.2.3
     tetra dry-run

dev/change-scala-version.sh 2.10

mvn clean install -DskipTests -Phive -Phive-thriftserver -Dscala-2.10 -Dkafka-0-8
     mvn dependency:go-offline

exit

7) Generate the new kit tarball:

tetra generate-all

You will now find an updated kit tarball in
spark/packages/spark-kit.tar.xz

Substitute this tarball for the existing one in the kit package and bump its
version.

Also, ensure that any jar paths and names are updated both in the spark project as
well as any services which depend on spark (such as openstack-monasca-transform).

Places

File README.building of Package spark-kit (Revision 963948462d2098cb27c65ad3f27799a9)

Places