File ardana-spark-8.0+git.1539709555.5b31c25.obscpio of Package ardana-spark
07070100000000000081A40000000000000000000000015BC61A7300000075000000000000000000000000000000000000003300000000ardana-spark-8.0+git.1539709555.5b31c25/.gitignore.vagrant
*.sw?
.tox
*.egg
*.egg-info
.testrepository
*.pyc
.project
.pydevproject
.settings/*
.settings
*.iml
.idea
07070100000001000081A40000000000000000000000015BC61A7300000081000000000000000000000000000000000000003300000000ardana-spark-8.0+git.1539709555.5b31c25/.gitreview[gerrit]
host=gerrit.suse.provo.cloud
port=29418
project=ardana/spark-ansible.git
defaultremote=ardana
defaultbranch=stable/pike
07070100000002000081A40000000000000000000000015BC61A730000000C000000000000000000000000000000000000003600000000ardana-spark-8.0+git.1539709555.5b31c25/.rsync-filter- ardana-ci
07070100000003000081A40000000000000000000000015BC61A730000279F000000000000000000000000000000000000003000000000ardana-spark-8.0+git.1539709555.5b31c25/LICENSE
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
07070100000004000081A40000000000000000000000015BC61A73000003C4000000000000000000000000000000000000003200000000ardana-spark-8.0+git.1539709555.5b31c25/README.md<!--
(c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP
(c) Copyright 2017 SUSE LLC
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
-->
README
======
This repo contains the following roles
- FND-SPA: Spark service
The verbs:
- install - install the service/role
- configure - configure the service/role
- start - start the service/role
- stop - stop the service/role
The operations:
- deploy - deploy the service (install, configure and start)
07070100000005000081A40000000000000000000000015BC61A73000002DF000000000000000000000000000000000000004400000000ardana-spark-8.0+git.1539709555.5b31c25/_spark-schedule-restart.yml#
# (c) Copyright 2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
# Schedule a restart of all spark services using ardana_notify_... variables
- hosts: all
tasks:
- include: roles/FND-SPA/tasks/_schedule_restart.yml07070100000006000041ED0000000000000000000000045BC61A7300000000000000000000000000000000000000000000003200000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci07070100000007000041ED0000000000000000000000035BC61A7300000000000000000000000000000000000000000000003A00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/project07070100000008000041ED0000000000000000000000035BC61A7300000000000000000000000000000000000000000000004600000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/project/input-model07070100000009000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000004B00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/project/input-model/data0707010000000A000081A40000000000000000000000015BC61A73000009BE000000000000000000000000000000000000005D00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/project/input-model/data/control_plane.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
product:
version: 2
control-planes:
- name: ccp
control-plane-prefix: ccp
region-name: region1
failure-zones:
- AZ1
- AZ2
- AZ3
common-service-components:
- lifecycle-manager-target
- monasca-agent
clusters:
- name: cluster0
cluster-prefix: c0
server-role:
- SERVER1-ROLE
member-count: 1
allocation-policy: strict
service-components:
- lifecycle-manager
- tempest
- ntp-client
# Required for testing in (run-test.sh)
- openstack-client
- name: cluster1
cluster-prefix: c1
server-role:
- SERVER2-ROLE
member-count: 1
allocation-policy: strict
service-components:
- ntp-client
# Monasca server
- kafka
- zookeeper
- vertica
- monasca-api
- monasca-persister
- spark
- name: cluster2
cluster-prefix: c2
server-role:
- SERVER3-ROLE
member-count: 1
allocation-policy: strict
service-components:
- ntp-server
- mysql
- rabbitmq
- keystone-api
- name: cluster3
cluster-prefix: c3
server-role:
- SERVER4-ROLE
- SERVER5-ROLE
- SERVER6-ROLE
member-count: 1
allocation-policy: strict
service-components:
- ntp-client
- ip-cluster
resources:
- name: resource
resource-prefix: res
server-role:
- SERVER7-ROLE
allocation-policy: any
service-components:
- ntp-client
0707010000000B000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003800000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests0707010000000C000081ED0000000000000000000000015BC61A730000034A000000000000000000000000000000000000004C00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/exercise_spark.bash#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#!/usr/bin/env bash
ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
exit 00707010000000D000081A40000000000000000000000015BC61A730000049C000000000000000000000000000000000000004700000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/test-plan.yaml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: Exercise spark
logfile: exercise-spark.log
exec:
- exercise_spark.bash
- name: Validate spark-up
logfile: testsuite-test_up.log
exec:
- validate-spark-up.bash
- name: Validate stop-start spark
logfile: testsuite-test_stop-start.log
exec:
- validate-spark-stop-start.bash
- name: Validate spark-upgrade
logfile: testsuite-test_upgrade.log
exec:
- validate-spark-upgrade.bash
- name: Validate spark-reconfigure
logfile: testsuite-test_reconfigure.log
exec:
- validate-spark-reconfigure.bash
0707010000000E000081ED0000000000000000000000015BC61A7300000402000000000000000000000000000000000000004900000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/update-conf.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
set -vxe
echo "Changing $1 in $2"
pushd ~/openstack/my_cloud/config
sed -e "$1" -i "$2" --follow-symlinks
git add -A
git commit --allow-empty -m "My Config Change"
pushd ~/openstack/ardana/ansible/
ansible-playbook -i hosts/localhost config-processor-run.yml -e encrypt="" \
-e rekey=""
ansible-playbook -i hosts/localhost ready-deployment.yml
popd
popd
0707010000000F000081ED0000000000000000000000015BC61A73000004A8000000000000000000000000000000000000005800000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/validate-spark-reconfigure.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
ansible-playbook -i hosts/verb_hosts spark-reconfigure.yml
SPARK_STATUS=$?
echo "Spark reconfigure"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
ansible-playbook -i hosts/verb_hosts spark-status.yml
SPARK_STATUS=$?
echo "Spark Service Status is $SPARK_STATUS"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi07070100000010000081ED0000000000000000000000015BC61A730000060D000000000000000000000000000000000000005700000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/validate-spark-stop-start.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
ansible-playbook -i hosts/verb_hosts spark-stop.yml
SPARK_STATUS=$?
echo "Spark Service Stop"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
ansible-playbook -i hosts/verb_hosts spark-status.yml
SPARK_STATUS=$?
echo "Spark Service Status is $SPARK_STATUS"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Fail"
exit 1
else
echo "Ok"
fi
ansible-playbook -i hosts/verb_hosts spark-start.yml
SPARK_STATUS=$?
echo "Spark Service Start"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
ansible-playbook -i hosts/verb_hosts spark-status.yml
SPARK_STATUS=$?
echo "Spark Service Status is $SPARK_STATUS"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi07070100000011000081ED0000000000000000000000015BC61A7300000405000000000000000000000000000000000000004F00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/validate-spark-up.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
ansible-playbook -i hosts/verb_hosts spark-status.yml
spark_test_status=$?
echo "Spark Test Status is $spark_test_status"
if [ $spark_test_status -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
07070100000012000081ED0000000000000000000000015BC61A73000004A0000000000000000000000000000000000000005400000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/validate-spark-upgrade.bash#!/bin/bash
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
ansible-playbook -i hosts/verb_hosts spark-upgrade.yml
SPARK_STATUS=$?
echo "Spark upgrade"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
ansible-playbook -i hosts/verb_hosts spark-status.yml
SPARK_STATUS=$?
echo "Spark Service Status is $SPARK_STATUS"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi
ansible-playbook -i hosts/verb_hosts spark-submit.yml
SPARK_STATUS=$?
echo "Spark submit quick sum"
if [ $SPARK_STATUS -eq 0 ]
then
echo "Ok"
else
echo "Fail"
exit 1
fi07070100000013000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000002F00000000ardana-spark-8.0+git.1539709555.5b31c25/config07070100000014000081A40000000000000000000000015BC61A73000003CF000000000000000000000000000000000000004200000000ardana-spark-8.0+git.1539709555.5b31c25/config/spark-symlinks.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# The following relative symlinks are created under the
# my_cloud/config directory.
---
symlinks:
"spark/spark-defaults.conf.j2": "roles/FND-SPA/templates/spark-defaults.conf.j2"
"spark/spark-env.sh.j2": "roles/FND-SPA/templates/spark-env.sh.j2"
"spark/spark-worker-env.sh.j2": "roles/FND-SPA/templates/spark-worker-env.sh.j2"
07070100000015000041ED0000000000000000000000035BC61A7300000000000000000000000000000000000000000000002E00000000ardana-spark-8.0+git.1539709555.5b31c25/roles07070100000016000041ED0000000000000000000000075BC61A7300000000000000000000000000000000000000000000003600000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA07070100000017000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003F00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/defaults07070100000018000081A40000000000000000000000015BC61A7300001289000000000000000000000000000000000000004800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/defaults/main.yml#
# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
spark_component_name: spark
spark_hosts: "{{ FND_SPA.members.master }}"
spark_group_name: "{{ verb_hosts.FND_SPA }}"
spark_local_hostname: >
{{ (spark_hosts |
selectattr('ardana_ansible_host','equalto', inventory_hostname) |
first).host }}
spark_ver: 1.6.3
spark_hadoop_ver: "{{ '2.6' if deployer_media_legacy_layout|bool else '2.4' }}"
spark_tarball_name: spark-{{ spark_ver }}-bin-hadoop{{ spark_hadoop_ver }}.tgz
spark_streaming_kafka_version_jar: spark-streaming-kafka_2.10-{{ spark_ver }}.jar
spark_streaming_kafka_jar: spark-streaming-kafka.jar
spark_root_dir: "{{ spark_component_name | bin_dir() }}/.."
spark_venv_lib_dir: "{%- if deployer_media_legacy_layout|bool -%}
{{ spark_component_name | jar_dir() }}
{%- else -%}
/usr/share/spark/lib/
{%- endif -%}"
spark_jdbc_jar: "{%- if deployer_media_legacy_layout|bool -%}
drizzle-jdbc-1.3.jar
{%- else -%}
mysql-connector-java.jar
{%- endif -%}"
spark_libs:
- ["kafka_2.10-0.8.1.1.jar", "kafka_2.10-0.8.1.1/libs"]
- ["scala-library-2.10.1.jar", "kafka_2.10-0.8.1.1/libs"]
- ["metrics-core-2.2.0.jar", "kafka_2.10-0.8.1.1/libs"]
- ["{{ spark_streaming_kafka_version_jar }}", ""]
- ["{{ spark_jdbc_jar }}", ""]
spark_current_dir: "{{ spark_root_dir }}/current"
spark_versioned_dir: "{{ spark_root_dir }}/spark-{{ spark_ver }}-bin-hadoop{{ spark_hadoop_ver }}"
spark_lib_jars_location: "{{ spark_versioned_dir }}/lib"
spark_run_dir: /var/run/spark
spark_worker_directory: "{{ spark_run_dir }}/work"
spark_conf_dir: /etc/spark/conf
spark_init_dir: /etc/spark/init
spark_log_dir: /var/log/spark
spark_tools_dir: "{{ spark_versioned_dir }}/tools"
spark_master_log_file: "{{ spark_log_dir }}/{{ spark_component_name }}-master.log"
spark_worker_log_file: "{{ spark_log_dir }}/{{ spark_component_name }}-worker.log"
spark_event_log_directory: "{{ spark_log_dir }}/events"
spark_group: spark
spark_user: spark
# spark configuration properties
spark_executor_cores: 1
spark_executor_memory: 512m
spark_cores_max: 3
spark_driver_memory: 512m
spark_python_worker_memory: 16m
spark_worker_memory: 512m
spark_worker_cores: 1
spark_daemon_x_ms: 16m
spark_daemon_x_mx: 16m
spark_daemon_max_perm_size: 32m
spark_sql_shuffle_partitions: 32
spark_worker_cleanup_enabled: true
spark_cleaner_ttl: 900
spark_sql_ui_retainedExecutions: 10
spark_streaming_ui_retainedBatches: 10
spark_worker_ui_retainedExecutors: 10
spark_worker_ui_retainedDrivers: 10
spark_ui_retainedJobs: 10
spark_ui_retainedStages: 10
spark_executor_logs_rolling_maxRetainedFiles: 24
spark_executor_logs_rolling_strategy: time
spark_executor_logs_rolling_time_interval: hourly
spark_master_port: "{{ host.bind.FND_SPA.master.port }}"
spark_worker_port: "{{ host.bind.FND_SPA.worker.port }}"
spark_master_webui_port: "{{ host.bind.FND_SPA.master_web_ui.port }}"
spark_worker_webui_port: "{{ host.bind.FND_SPA.worker_web_ui.port }}"
spark_blockmanager_port: "{{ host.bind.FND_SPA.block_manager.port }}"
spark_broadcast_port: "{{ host.bind.FND_SPA.broadcast.port }}"
spark_driver_port: "{{ host.bind.FND_SPA.driver.port }}"
spark_executor_port: "{{ host.bind.FND_SPA.executor.port }}"
spark_fileserver_port: "{{ host.bind.FND_SPA.fileserver.port }}"
spark_masters: "{% for node in FND_SPA.members.master %}{{ node.host }}:{{ node.port }}{% if not loop.last %},{% endif %}{% endfor %}"
# zookeeper
spark_zookeeper_nodes: "{{ FND_SPA.consumes_FND_ZOO.members.private }}"
spark_zookeeper_hosts: "{% for node in spark_zookeeper_nodes %}{{ node.host }}:{{ node.port }}{% if not loop.last %},{% endif %}{% endfor %}"
spark_init: systemd
# upstart
spark_master_upstart_conf: /etc/init/spark-master.conf
spark_worker_upstart_conf: /etc/init/spark-worker.conf
# systemd
spark_master_systemd_service: /etc/systemd/system/spark-master.service
spark_worker_systemd_service: /etc/systemd/system/spark-worker.service
# monitoring
spark_agent_detection_plugin_dir: "{{ monasca_agent_detection_plugin_dir }}"
07070100000019000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003C00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/files0707010000001A000081A40000000000000000000000015BC61A7300000547000000000000000000000000000000000000004B00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/files/SparkDetect.py#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
import logging
import monasca_setup.detection
log = logging.getLogger(__name__)
class SparkDetect(monasca_setup.detection.ServicePlugin):
"""Detect Spark daemons and setup configuration to monitor them."""
def __init__(self, template_dir, overwrite=True, args=None):
log.info("\tWatching the spark processes.")
service_params = {
'args': {},
'template_dir': template_dir,
'overwrite': overwrite,
'service_name': 'spark',
'process_names': ['org.apache.spark.deploy.master.Master',
'org.apache.spark.deploy.worker.Worker']
}
super(SparkDetect, self).__init__(service_params)
0707010000001B000081A40000000000000000000000015BC61A7300000422000000000000000000000000000000000000004900000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/files/quick_sum.py#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
from __future__ import print_function
from pyspark import SparkConf
from pyspark import SparkContext
if __name__ == "__main__":
my_spark_conf = SparkConf().setAppName("Quicksum")
spark_context = SparkContext(conf=my_spark_conf)
data = [1, 2, 3, 4, 5]
distData = spark_context.parallelize(data)
total = distData.reduce(lambda a, b: a + b)
print("Total is %s" % total)
spark_context.stop()
0707010000001C000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003C00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks0707010000001D000081A40000000000000000000000015BC61A730000030F000000000000000000000000000000000000005200000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/_schedule_restart.yml#
# (c) Copyright 2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | _schedule_restart | Schedule a restart for all services
debug:
msg: "Trigger a change notification in spark"
changed_when: true
register: ardana_notify_spark_restart_required
0707010000001E000081A40000000000000000000000015BC61A7300000604000000000000000000000000000000000000004A00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/configure.yml#
# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | configure | Link current version
become: yes
file:
path: "{{ spark_current_dir }}"
state: link
src: "{{ spark_versioned_dir }}"
when: deployer_media_legacy_layout|bool
- name: FND-SPA | configure | Set up spark-env (rpm)
become: yes
template:
src: spark-env.j2
dest: /etc/spark/spark-env
owner: "{{ spark_user }}"
group: "{{ spark_group }}"
mode: 0644
when: deployer_media_legacy_layout|bool == False
- name: FND-SPA | configure | Set up spark-defaults (rpm)
become: yes
template:
src: spark-defaults.conf.j2
dest: /etc/spark/spark-defaults.conf
owner: "{{ spark_user }}"
group: "{{ spark_group }}"
mode: 0644
when: deployer_media_legacy_layout|bool == False
- name: FND-SPA | configure | include configure tasks (legacy)
include: configure_legacy.yml
when: deployer_media_legacy_layout|bool
0707010000001F000081A40000000000000000000000015BC61A7300000E88000000000000000000000000000000000000005100000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/configure_legacy.yml#
# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | configure_legacy | Link spark commands into /usr/bin
become: yes
file:
path: /usr/bin/{{ item }}
state: link
src: "{{ spark_current_dir }}/bin/{{ item }}"
with_items:
- spark-submit
- spark-class
- spark-shell
- spark-sql
- name: FND-SPA | configure_legacy | Link spark libs
become: yes
file:
path: "{{ spark_lib_jars_location }}/{{ spark_streaming_kafka_jar }}"
state: link
src: "{{ spark_lib_jars_location }}/{{ spark_streaming_kafka_version_jar }}"
- name: FND-SPA | configure_legacy | Set up spark-env
become: yes
template:
src: spark-env.sh.j2
dest: /etc/spark/conf/spark-env.sh
owner: "{{ spark_user }}"
group: "{{ spark_group }}"
mode: 0644
- name: FND-SPA | configure_legacy | Set up spark-worker-env
become: yes
template:
src: spark-worker-env.sh.j2
dest: /etc/spark/conf/spark-worker-env.sh
owner: "{{ spark_user }}"
group: "{{ spark_group }}"
mode: 0644
- name: FND-SPA | configure_legacy | Set up spark-defaults (legacy)
become: yes
template:
src: spark-defaults.conf.j2
dest: /etc/spark/conf/spark-defaults.conf
owner: "{{ spark_user }}"
group: "{{ spark_group }}"
mode: 0644
- name: FND-SPA | configure_legacy | Create spark-master start script
become: yes
template:
dest: "{{ spark_init_dir }}/start-spark-master.sh"
owner: root
group: root
mode: 0755
src: start-spark-master.sh.j2
- name: FND-SPA | configure_legacy | Create spark-worker start script
become: yes
template:
dest: "{{ spark_init_dir }}/start-spark-worker.sh"
owner: root
group: root
mode: 0755
src: start-spark-worker.sh.j2
- name: FND-SPA | configure_legacy | Create spark-master systemd config
become: yes
template:
src: spark-master.service.j2
dest: "{{ spark_master_systemd_service }}"
owner: root
group: root
mode: 0644
- name: FND-SPA | configure_legacy | Create spark-worker systemd config
become: yes
template:
src: spark-worker.service.j2
dest: "{{ spark_worker_systemd_service }}"
owner: root
group: root
mode: 0644
# See: http://www.freedesktop.org/software/systemd/man/tmpfiles.d.html
- name: FND-SPA | configure_legacy | Create spark-master.conf
become: yes
template:
src: spark-master.conf.j2
dest: "/usr/lib/tmpfiles.d/spark-master.conf"
owner: root
group: root
mode: 0644
- name: FND-SPA | configure_legacy | Create spark-worker.conf
become: yes
template:
src: spark-worker.conf.j2
dest: "/usr/lib/tmpfiles.d/spark-worker.conf"
owner: root
group: root
mode: 0644
- name: FND-SPA | configure_legacy | Reload systemctl
become: yes
ignore_errors: True
command: /bin/systemctl daemon-reload
- name: FND-SPA | configure_legacy | Link config version
become: yes
file:
path: "{{ spark_current_dir }}/conf/{{ item }}"
state: link
src: "{{ spark_conf_dir }}/{{ item }}"
with_items:
- "spark-env.sh"
- "spark-worker-env.sh"
- "spark-defaults.conf"
07070100000020000081A40000000000000000000000015BC61A73000008C1000000000000000000000000000000000000004800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/install.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | install | include OS specific variables
include_vars: "{{ ansible_os_family | lower }}.yml"
- name: FND-SPA | install | Install the spark package from repo
become: yes
package:
name: "{{ item }}"
state: present
when: deployer_media_legacy_layout|bool == False
with_items:
- "spark"
- "kafka"
- "mysql-connector-java"
- name: FND-SPA | install | create symlink to mysql connector jdbc jar
become: yes
file:
src: "/usr/share/java/{{ spark_jdbc_jar }}"
dest: "{{ spark_venv_lib_dir }}/{{ spark_jdbc_jar }}"
state: link
when: deployer_media_legacy_layout|bool == False
- name: FND-SPA | install | include legacy layout tasks
include: install_legacy.yml
when: deployer_media_legacy_layout|bool
- name: FND-SPA | install | Create Spark local directories
become: yes
file:
path: "{{ item }}"
state: directory
owner: "{{ spark_user }}"
group: "{{ spark_group }}"
mode: 0775
with_items:
- "{{ spark_run_dir }}"
- "{{ spark_worker_directory }}"
- "{{ spark_tools_dir }}"
- name: FND-SPA | install | Create Spark log directory
become: yes
file:
path: "{{ item }}"
state: directory
owner: "{{ spark_user }}"
group: adm
mode: 0770
with_items:
- "{{ spark_log_dir }}"
when: deployer_media_legacy_layout|bool
- name: FND-SPA | install | Touch logfiles
become: true
file:
path: "{{ item }}"
state: touch
owner: "{{ spark_user }}"
group: adm
mode: 0640
with_items:
- "{{ spark_master_log_file }}"
- "{{ spark_worker_log_file }}"
07070100000021000081A40000000000000000000000015BC61A7300000A1F000000000000000000000000000000000000004F00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/install_legacy.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | install_legacy | Setup group
become_user: root
become: True
group:
name: "{{ spark_group }}"
system: yes
- name: FND-SPA | install_legacy | Setup user
become: yes
user:
name: "{{ spark_user }}"
system: yes
group: "{{ spark_group }}"
createhome: no
home: /nonexistent
shell: /usr/sbin/nologin
comment: "Apache Spark system user"
- name: FND-SPA | install_legacy | Install OS specific required packages
become: yes
package:
name: "{{ item }}"
state: present
with_items: fnd_spark_required_packages
- name: FND-SPA | install_legacy | Update installed packager cache
become: yes
install_package:
cache: update
- name: FND-SPA | install_legacy | Install the spark venv
become: yes
install_package:
name: spark
service: spark
state: present
- name: FND-SPA | install_legacy | Uncompress the spark tar
become: yes
unarchive:
copy: no
creates: >
"{{ spark_root_dir }}/spark-{{ spark_ver }}-
bin-hadoop{{ spark_hadoop_ver }}"
dest: "{{ spark_root_dir }}"
src: "{{ spark_venv_lib_dir }}/{{ spark_tarball_name }}"
- name: FND-SPA | install_legacy | Untar kafka tarball
become: yes
unarchive:
copy: no
creates: "{{ spark_venv_lib_dir }}/kafka_2.10-0.8.1.1"
dest: "{{ spark_venv_lib_dir }}"
src: "{{ spark_venv_lib_dir }}/kafka_2.10-0.8.1.1.tgz"
- name: FND-SPA | install_legacy | Link lib jars to spark lib directory
become: yes
file:
state: link
dest: "{{ spark_lib_jars_location }}/{{ item[0] }}"
src: "{{ spark_venv_lib_dir }}/{{ item[1] }}/{{ item[0] }}"
with_items:
- "{{ spark_libs }}"
- name: FND-SPA | install_legacy | Create Spark local directories
become: yes
file:
path: "{{ item }}"
state: directory
owner: "{{ spark_user }}"
group: "{{ spark_group }}"
mode: 0775
with_items:
- "{{ spark_conf_dir }}"
- "{{ spark_init_dir }}"07070100000022000081A40000000000000000000000015BC61A7300000417000000000000000000000000000000000000005200000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/monasca_configure.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | monasca_configure |
Copy Spark Custom plugin for Ardana only
become: yes
copy:
src: SparkDetect.py
dest: "{{ spark_agent_detection_plugin_dir }}/SparkDetect.py"
owner: root
group: root
mode: 0440
- name: FND-SPA | monasca_configure |
Run Monasca agent Ardana specific Spark detection plugin
become: yes
monasca_agent_plugin:
name: "SparkDetect"
07070100000023000081A40000000000000000000000015BC61A73000002A2000000000000000000000000000000000000005000000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/password-change.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
# nothing to do I don't think
07070100000024000081A40000000000000000000000015BC61A73000005F5000000000000000000000000000000000000004D00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/spark_submit.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | spark_submit | Push quick_sum.py script for validation
become: yes
copy:
src: quick_sum.py
dest: "{{ spark_tools_dir }}/quick_sum.py"
owner: "{{ spark_user }}"
group: "{{ spark_group }}"
mode: 0775
- name: FND-SPA | spark_submit | Submit quick_sum.py to spark
become_user: "{{ spark_user }}"
become: yes
shell: >
export SPARK_HOME={{ spark_current_dir }} &&
spark-submit --supervise --master spark://{{ spark_masters }} quick_sum.py
register: spark_quick_sum_result
args:
chdir: "{{ spark_tools_dir }}"
run_once: True
- name: FND-SPA | spark_submit | Announce quick sum result if failed
fail:
msg: |
Quick sum failed with output -
{{ spark_quick_sum_result.stdout }}
{{ spark_quick_sum_result.stderr }}
when: not ("{{ spark_quick_sum_result.stdout }}" == "Total is 15")
run_once: true
07070100000025000081A40000000000000000000000015BC61A7300000592000000000000000000000000000000000000004600000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/start.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | start | restart spark master
# start spark master
service:
name: spark-master
state: restarted
become: yes
when: ardana_notify_spark_restart_required is defined and
ardana_notify_spark_restart_required.changed
- name: FND-SPA | start | start spark master
# start spark master
service:
name: spark-master
state: started
become: yes
- name: FND-SPA | start | restart spark worker
# start spark worker
service:
name: spark-worker
state: restarted
become: yes
when: ardana_notify_spark_restart_required is defined and
ardana_notify_spark_restart_required.changed
- name: FND-SPA | start | start spark worker
# start spark worker
service:
name: spark-worker
state: started
become: yes
07070100000026000081A40000000000000000000000015BC61A730000071B000000000000000000000000000000000000004700000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/status.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | status | Print service being checked
debug:
msg: "Running service check for spark"
run_once: true
- name: FND-SPA | status | Check systemd service running spark-master
command: systemctl status spark-master
ignore_errors: yes
changed_when: false
register: spark_systemctl_status_master_result
- name: FND-SPA | status | Report status of spark-master
fail:
msg: |
spark-master is not running.
systemctl status spark-master output:
{{ spark_systemctl_status_master_result.stdout }}
{{ spark_systemctl_status_master_result.stderr }}
when: spark_systemctl_status_master_result | failed
- name: FND-SPA | status | Check systemd service running spark-worker
command: systemctl status spark-worker
ignore_errors: yes
changed_when: false
register: spark_systemctl_status_worker_result
- name: FND-SPA | status | Report status of spark-worker
fail:
msg: |
spark-worker is not running.
systemctl status spark-worker output:
{{ spark_systemctl_status_worker_result.stdout }}
{{ spark_systemctl_status_worker_result.stderr }}
when: spark_systemctl_status_worker_result | failed
07070100000027000081A40000000000000000000000015BC61A7300000360000000000000000000000000000000000000004500000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/stop.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | stop | stop spark worker
become: yes
service:
name: spark-worker
state: stopped
- name: FND-SPA | stop | stop spark master
become: yes
service:
name: spark-master
state: stopped
07070100000028000081A40000000000000000000000015BC61A7300000434000000000000000000000000000000000000004700000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/unlink.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- name: FND-SPA | unlink | Unlink current version
become: yes
file:
path: "{{ spark_current_dir }}"
state: absent
when: deployer_media_legacy_layout|bool
- name: FND-SPA | unlink | Unlink spark commands in /usr/bin
become: yes
file:
path: /usr/bin/{{ item }}
state: absent
with_items:
- spark-submit
- spark-class
- spark-shell
- spark-sql
when: deployer_media_legacy_layout|bool
07070100000029000081A40000000000000000000000015BC61A730000035B000000000000000000000000000000000000004800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/upgrade.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
## upgrade is not yet implemented as this is the initial release
- include: stop.yml
- include: unlink.yml
- include: install.yml
- include: configure.yml
- include: start.yml
- include: monasca_configure.yml0707010000002A000081A40000000000000000000000015BC61A730000040B000000000000000000000000000000000000004D00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/upstart_init.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: create spark-master upstart config
become: yes
template:
dest: "{{ spark_master_upstart_conf }}"
owner: root
group: root
mode: 0644
src: upstart/spark-master.conf.j2
- name: create spark-worker upstart config
become: yes
template:
dest: "{{ spark_worker_upstart_conf }}"
owner: root
group: root
mode: 0644
src: upstart/spark-worker.conf.j20707010000002B000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000004000000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates0707010000002C000081ED0000000000000000000000015BC61A730000033F000000000000000000000000000000000000005500000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/exercise_spark.sh.j2#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#!/usr/bin/env bash
export SPARK_HOME=/opt/stack/service/spark/venv/bin/../current
CALC_VALUE=`bash $SPARK_HOME/bin/spark-submit quick_sum.py`
echo The calculated value is $CALC_VALUE
exit 00707010000002D000081A40000000000000000000000015BC61A73000008A9000000000000000000000000000000000000005700000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-defaults.conf.j2{#
#
# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017-2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
spark.driver.extraClassPath {{ spark_venv_lib_dir }}/{{ spark_jdbc_jar }}
spark.executor.extraClassPath {{ spark_venv_lib_dir }}/{{ spark_jdbc_jar }}
spark.blockManager.port {{ spark_blockmanager_port }}
spark.broadcast.port {{ spark_broadcast_port }}
spark.cores.max {{ spark_cores_max }}
spark.driver.memory {{ spark_driver_memory }}
spark.driver.port {{ spark_driver_port }}
spark.eventLog.dir {{ spark_event_log_directory }}
spark.executor.cores {{ spark_executor_cores }}
spark.executor.memory {{ spark_executor_memory }}
spark.executor.port {{ spark_executor_port }}
spark.fileserver.port {{ spark_fileserver_port }}
spark.python.worker.memory {{ spark_python_worker_memory }}
spark.speculation true
spark.speculation.interval 200
spark.sql.shuffle.partitions {{ spark_sql_shuffle_partitions }}
spark.worker.cleanup.enabled {{ spark_worker_cleanup_enabled }}
spark.cleaner.ttl {{ spark_cleaner_ttl }}
spark.sql.ui.retainedExecutions {{ spark_sql_ui_retainedExecutions }}
spark.streaming.ui.retainedBatches {{ spark_streaming_ui_retainedBatches }}
spark.worker.ui.retainedExecutors {{ spark_worker_ui_retainedExecutors }}
spark.worker.ui.retainedDrivers {{ spark_worker_ui_retainedDrivers }}
spark.ui.retainedJobs {{ spark_ui_retainedJobs }}
spark.ui.retainedStages {{ spark_ui_retainedStages }}
spark.executor.logs.rolling.maxRetainedFiles {{spark_executor_logs_rolling_maxRetainedFiles}}
spark.executor.logs.rolling.strategy {{spark_executor_logs_rolling_strategy}}
spark.executor.logs.rolling.time.interval {{spark_executor_logs_rolling_time_interval}}
0707010000002E000081A40000000000000000000000015BC61A7300000696000000000000000000000000000000000000004D00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-env.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
SPARK_HOME=/usr/share/spark
SPARK_LOCAL_IP={{ spark_local_hostname }}
SPARK_MASTER_IP={{ spark_local_hostname }}
SPARK_MASTER_PORT={{ spark_master_port }}
SPARK_MASTER_WEBUI_PORT={{ spark_master_webui_port }}
SPARK_WORKER_PORT={{ spark_worker_port }}
SPARK_WORKER_WEBUI_PORT={{ spark_worker_webui_port }}
SPARK_WORKER_DIR={{ spark_worker_directory }}
SPARK_WORKER_MEMORY={{ spark_worker_memory }}
SPARK_WORKER_CORES={{ spark_worker_cores }}
SPARK_DAEMON_JAVA_OPTS="-Xms{{ spark_daemon_x_ms }} -Xmx{{ spark_daemon_x_mx }} -XX:MaxPermSize={{ spark_daemon_max_perm_size }} -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url={{ spark_zookeeper_hosts }} -Dspark.deploy.zookeeper.dir={{ spark_run_dir }} -Dspark.executor.logs.rolling.maxRetainedFiles=10 -Dspark.executor.logs.rolling.strategy=size -Dspark.executor.logs.rolling.maxSize=20971520 -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=1800 -Dspark.worker.cleanup.appDataTtl=86400 -Dspark.master.rest.enabled=false"
SPARK_MASTERS={{ spark_masters }}
0707010000002F000081A40000000000000000000000015BC61A73000007DA000000000000000000000000000000000000005000000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-env.sh.j2#!/usr/bin/env bash
{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017, 2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
export SPARK_LOCAL_IP={{ spark_local_hostname }}
export SPARK_MASTER_IP={{ spark_local_hostname }}
export SPARK_MASTER_PORT={{ spark_master_port }}
export SPARK_MASTERS={{ spark_masters }}
export SPARK_MASTER_WEBUI_PORT={{ spark_master_webui_port }}
export SPARK_WORKER_PORT={{ spark_worker_port }}
export SPARK_WORKER_WEBUI_PORT={{ spark_worker_webui_port }}
export SPARK_WORKER_DIR={{ spark_worker_directory }}
export SPARK_WORKER_MEMORY={{ spark_worker_memory }}
export SPARK_WORKER_CORES={{ spark_worker_cores }}
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=file://{{ spark_event_log_directory }} -Dspark.history.ui.port=18082"
export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Xms{{ spark_daemon_x_ms }} -Xmx{{ spark_daemon_x_mx }} -XX:MaxPermSize={{ spark_daemon_max_perm_size }} -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url={{ spark_zookeeper_hosts }} -Dspark.deploy.zookeeper.dir={{ spark_run_dir }}"
export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Dspark.executor.logs.rolling.maxRetainedFiles=10 -Dspark.executor.logs.rolling.strategy=size -Dspark.executor.logs.rolling.maxSize=20971520"
export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=1800 -Dspark.worker.cleanup.appDataTtl=86400"07070100000030000081A40000000000000000000000015BC61A73000002CC000000000000000000000000000000000000005500000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-master.conf.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
d {{ spark_worker_directory }} 0755 {{ spark_user }} {{ spark_group }}
07070100000031000081A40000000000000000000000015BC61A7300000338000000000000000000000000000000000000005800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-master.service.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
[Unit]
Description=Spark Master Daemon
After=zookeeper.service
[Service]
Type=simple
ExecStart={{ spark_init_dir }}/start-spark-master.sh &
[Install]
WantedBy=multi-user.target
07070100000032000081A40000000000000000000000015BC61A73000006A8000000000000000000000000000000000000005700000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-worker-env.sh.j2#!/usr/bin/env bash
{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
export SPARK_LOCAL_IP={{ spark_local_hostname }}
export SPARK_MASTER_IP={{ spark_local_hostname }}
export SPARK_MASTER_PORT={{ spark_master_port }}
export SPARK_MASTERS={{ spark_masters }}
export SPARK_MASTER_WEBUI_PORT={{ spark_master_webui_port }}
export SPARK_WORKER_PORT={{ spark_worker_port }}
export SPARK_WORKER_WEBUI_PORT={{ spark_worker_webui_port }}
export SPARK_WORKER_DIR={{ spark_worker_directory }}
export SPARK_WORKER_MEMORY={{ spark_worker_memory }}
export SPARK_WORKER_CORES={{ spark_worker_cores }}
# history server
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=file://{{ spark_event_log_directory }} -Dspark.history.ui.port=18082"
# set up for multiple masters using zookeeper
export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Xms{{ spark_daemon_x_ms }} -Xmx{{ spark_daemon_x_mx }} -XX:MaxPermSize={{ spark_daemon_max_perm_size }} -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url={{ spark_zookeeper_hosts }} -Dspark.deploy.zookeeper.dir={{ spark_run_dir }}"
07070100000033000081A40000000000000000000000015BC61A73000002CC000000000000000000000000000000000000005500000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-worker.conf.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
d {{ spark_worker_directory }} 0755 {{ spark_user }} {{ spark_group }}
07070100000034000081A40000000000000000000000015BC61A7300000337000000000000000000000000000000000000005800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-worker.service.j2{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
[Unit]
Description=Spark Worker Daemon
After=zookeeper.service
[Service]
Type=simple
ExecStart={{ spark_init_dir }}/start-spark-worker.sh &
[Install]
WantedBy=multi-user.target
07070100000035000081A40000000000000000000000015BC61A73000007E3000000000000000000000000000000000000005900000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/start-spark-master.sh.j2#!/usr/bin/env bash
{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
. {{ spark_current_dir }}/conf/spark-env.sh
export EXEC_CLASS=org.apache.spark.deploy.master.Master
export INSTANCE_ID=1
export SPARK_CLASSPATH={{ spark_current_dir }}/conf/:{{ spark_current_dir }}/lib/spark-assembly-{{ spark_ver }}-hadoop{{ spark_hadoop_ver }}.0.jar:{{ spark_current_dir }}/lib/datanucleus-core-3.2.10.jar:{{ spark_current_dir }}/lib/datanucleus-rdbms-3.2.9.jar:{{ spark_current_dir }}/lib/datanucleus-api-jdo-3.2.6.jar
export SPARK_HOME={{ spark_current_dir }}
{% if ansible_os_family | lower == 'debian' %}
start-stop-daemon -c {{ spark_user }}:{{ spark_group }} --pidfile {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid --name spark-master --start --exec /usr/bin/java -- -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --ip "$SPARK_MASTER_IP" --port "$SPARK_MASTER_PORT" --webui-port "$SPARK_MASTER_WEBUI_PORT" >> "{{ spark_master_log_file }}" 2>&1 < /dev/null
{% elif ansible_os_family | lower == 'suse' %}
start_daemon -u {{ spark_user }} -g {{ spark_group }} -p {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid /usr/bin/java -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --ip "$SPARK_MASTER_IP" --port "$SPARK_MASTER_PORT" --webui-port "$SPARK_MASTER_WEBUI_PORT" >> "{{ spark_master_log_file }}" 2>&1 < /dev/null
{% endif %}07070100000036000081A40000000000000000000000015BC61A73000007CE000000000000000000000000000000000000005900000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/start-spark-worker.sh.j2#!/usr/bin/env bash
{#
#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#}
. {{ spark_current_dir }}/conf/spark-env.sh
export EXEC_CLASS=org.apache.spark.deploy.worker.Worker
export INSTANCE_ID=1
export SPARK_CLASSPATH={{ spark_current_dir }}/conf/:{{ spark_current_dir }}/lib/spark-assembly-{{ spark_ver }}-hadoop{{ spark_hadoop_ver }}.0.jar:{{ spark_current_dir }}/lib/datanucleus-core-3.2.10.jar:{{ spark_current_dir }}/lib/datanucleus-rdbms-3.2.9.jar:{{ spark_current_dir }}/lib/datanucleus-api-jdo-3.2.6.jar
export SPARK_HOME={{ spark_current_dir }}
{% if ansible_os_family | lower == 'debian' %}
start-stop-daemon -c {{ spark_user }}:{{ spark_group }} --pidfile {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid --name spark-worker --start --exec /usr/bin/java -- -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --webui-port "$SPARK_WORKER_WEBUI_PORT" --port $SPARK_WORKER_PORT $SPARK_MASTERS >> "{{ spark_worker_log_file }}" 2>&1 < /dev/null
{% elif ansible_os_family | lower == 'suse' %}
start_daemon -u {{ spark_user }} -g {{ spark_group }} -p {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid /usr/bin/java -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --webui-port "$SPARK_WORKER_WEBUI_PORT" --port $SPARK_WORKER_PORT $SPARK_MASTERS >> "{{ spark_worker_log_file }}" 2>&1 < /dev/null
{% endif %}
07070100000037000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003B00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/vars07070100000038000081A40000000000000000000000015BC61A73000002AF000000000000000000000000000000000000004600000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/vars/debian.yml#
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
fnd_spark_required_packages:
- default-jre-headless
fnd_spark_restart_packages:
- default-jre-headless07070100000039000081A40000000000000000000000015BC61A73000002BD000000000000000000000000000000000000004400000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/vars/suse.yml#
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
fnd_spark_required_packages:
- java-1_8_0-openjdk-headless
fnd_spark_restart_packages:
- java-1_8_0-openjdk-headless0707010000003A000081A40000000000000000000000015BC61A7300000347000000000000000000000000000000000000004A00000000ardana-spark-8.0+git.1539709555.5b31c25/spark-deploy-monitoring-agent.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: deploy spark-monasca-agent-plugin
hosts: FND-SPA
roles:
roles:
- {role: monasca-agent, run_mode: Use}
- FND-SPA
tasks:
- include: roles/FND-SPA/tasks/monasca_configure.yml
0707010000003B000081A40000000000000000000000015BC61A7300000373000000000000000000000000000000000000004100000000ardana-spark-8.0+git.1539709555.5b31c25/spark-deploy-service.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development Company LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: deploy spark-master and spark-worker
hosts: FND-SPA
roles:
- FND-SPA
tasks:
- include: roles/FND-SPA/tasks/install.yml
- include: roles/FND-SPA/tasks/configure.yml
- include: roles/FND-SPA/tasks/start.yml
0707010000003C000081A40000000000000000000000015BC61A730000031D000000000000000000000000000000000000003900000000ardana-spark-8.0+git.1539709555.5b31c25/spark-deploy.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# deploy the service itself
- include: spark-deploy-service.yml
# create monitoring checks for spark Services
- include: spark-deploy-monitoring-agent.yml
0707010000003D000081A40000000000000000000000015BC61A7300000392000000000000000000000000000000000000004200000000ardana-spark-8.0+git.1539709555.5b31c25/spark-password-change.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: reconfigure spark service
hosts: FND-SPA
roles:
- FND-SPA
tasks:
- include: roles/FND-SPA/tasks/stop.yml
- include: roles/FND-SPA/tasks/password-change.yml
- include: roles/FND-SPA/tasks/configure.yml
- include: roles/FND-SPA/tasks/start.yml0707010000003E000081A40000000000000000000000015BC61A7300000367000000000000000000000000000000000000003E00000000ardana-spark-8.0+git.1539709555.5b31c25/spark-reconfigure.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: reconfigure spark service
hosts: FND-SPA
roles:
- FND-SPA
serial: 1
tasks:
- include: roles/FND-SPA/tasks/stop.yml
- include: roles/FND-SPA/tasks/configure.yml
- include: roles/FND-SPA/tasks/start.yml0707010000003F000081A40000000000000000000000015BC61A7300000287000000000000000000000000000000000000003A00000000ardana-spark-8.0+git.1539709555.5b31c25/spark-restart.yml#
# (c) Copyright 2018 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- include: _spark-schedule-restart.yml
- include: spark-start.yml
07070100000040000081A40000000000000000000000015BC61A7300000308000000000000000000000000000000000000003800000000ardana-spark-8.0+git.1539709555.5b31c25/spark-start.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: start spark-master and spark-worker
hosts: FND-SPA
roles:
- FND-SPA
tasks:
- include: roles/FND-SPA/tasks/start.yml07070100000041000081A40000000000000000000000015BC61A730000030A000000000000000000000000000000000000003900000000ardana-spark-8.0+git.1539709555.5b31c25/spark-status.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: status spark-master and spark-worker
hosts: FND-SPA
roles:
- FND-SPA
tasks:
- include: roles/FND-SPA/tasks/status.yml07070100000042000081A40000000000000000000000015BC61A7300000306000000000000000000000000000000000000003700000000ardana-spark-8.0+git.1539709555.5b31c25/spark-stop.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: stop spark-master and spark-worker
hosts: FND-SPA
roles:
- FND-SPA
tasks:
- include: roles/FND-SPA/tasks/stop.yml07070100000043000081A40000000000000000000000015BC61A7300000306000000000000000000000000000000000000003900000000ardana-spark-8.0+git.1539709555.5b31c25/spark-submit.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
- name: submit sample job to spark
hosts: FND-SPA
roles:
- FND-SPA
tasks:
- include: roles/FND-SPA/tasks/spark_submit.yml07070100000044000081A40000000000000000000000015BC61A730000031A000000000000000000000000000000000000003A00000000ardana-spark-8.0+git.1539709555.5b31c25/spark-upgrade.yml#
# (c) Copyright 2016 Hewlett Packard Enterprise Development LP
# (c) Copyright 2017 SUSE LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
---
- hosts: FND-SPA
roles:
- {role: monasca-agent, run_mode: Use}
- FND-SPA
serial: 1
tasks:
- include: roles/FND-SPA/tasks/upgrade.yml
07070100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000B00000000TRAILER!!!177 blocks
