Sign Up
Log In
Log In
or
Sign Up
Places
All Projects
Status Monitor
Collapse sidebar
systemsmanagement:Ardana:8:CentOS
ardana-spark
ardana-spark-8.0+git.1539709555.5b31c25.obscpio
Overview
Repositories
Revisions
Requests
Users
Attributes
Meta
File ardana-spark-8.0+git.1539709555.5b31c25.obscpio of Package ardana-spark
07070100000000000081A40000000000000000000000015BC61A7300000075000000000000000000000000000000000000003300000000ardana-spark-8.0+git.1539709555.5b31c25/.gitignore.vagrant *.sw? .tox *.egg *.egg-info .testrepository *.pyc .project .pydevproject .settings/* .settings *.iml .idea 07070100000001000081A40000000000000000000000015BC61A7300000081000000000000000000000000000000000000003300000000ardana-spark-8.0+git.1539709555.5b31c25/.gitreview[gerrit] host=gerrit.suse.provo.cloud port=29418 project=ardana/spark-ansible.git defaultremote=ardana defaultbranch=stable/pike 07070100000002000081A40000000000000000000000015BC61A730000000C000000000000000000000000000000000000003600000000ardana-spark-8.0+git.1539709555.5b31c25/.rsync-filter- ardana-ci 07070100000003000081A40000000000000000000000015BC61A730000279F000000000000000000000000000000000000003000000000ardana-spark-8.0+git.1539709555.5b31c25/LICENSE Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. 07070100000004000081A40000000000000000000000015BC61A73000003C4000000000000000000000000000000000000003200000000ardana-spark-8.0+git.1539709555.5b31c25/README.md<!-- (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> README ====== This repo contains the following roles - FND-SPA: Spark service The verbs: - install - install the service/role - configure - configure the service/role - start - start the service/role - stop - stop the service/role The operations: - deploy - deploy the service (install, configure and start) 07070100000005000081A40000000000000000000000015BC61A73000002DF000000000000000000000000000000000000004400000000ardana-spark-8.0+git.1539709555.5b31c25/_spark-schedule-restart.yml# # (c) Copyright 2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- # Schedule a restart of all spark services using ardana_notify_... variables - hosts: all tasks: - include: roles/FND-SPA/tasks/_schedule_restart.yml07070100000006000041ED0000000000000000000000045BC61A7300000000000000000000000000000000000000000000003200000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci07070100000007000041ED0000000000000000000000035BC61A7300000000000000000000000000000000000000000000003A00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/project07070100000008000041ED0000000000000000000000035BC61A7300000000000000000000000000000000000000000000004600000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/project/input-model07070100000009000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000004B00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/project/input-model/data0707010000000A000081A40000000000000000000000015BC61A73000009BE000000000000000000000000000000000000005D00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/project/input-model/data/control_plane.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 control-planes: - name: ccp control-plane-prefix: ccp region-name: region1 failure-zones: - AZ1 - AZ2 - AZ3 common-service-components: - lifecycle-manager-target - monasca-agent clusters: - name: cluster0 cluster-prefix: c0 server-role: - SERVER1-ROLE member-count: 1 allocation-policy: strict service-components: - lifecycle-manager - tempest - ntp-client # Required for testing in (run-test.sh) - openstack-client - name: cluster1 cluster-prefix: c1 server-role: - SERVER2-ROLE member-count: 1 allocation-policy: strict service-components: - ntp-client # Monasca server - kafka - zookeeper - vertica - monasca-api - monasca-persister - spark - name: cluster2 cluster-prefix: c2 server-role: - SERVER3-ROLE member-count: 1 allocation-policy: strict service-components: - ntp-server - mysql - rabbitmq - keystone-api - name: cluster3 cluster-prefix: c3 server-role: - SERVER4-ROLE - SERVER5-ROLE - SERVER6-ROLE member-count: 1 allocation-policy: strict service-components: - ntp-client - ip-cluster resources: - name: resource resource-prefix: res server-role: - SERVER7-ROLE allocation-policy: any service-components: - ntp-client 0707010000000B000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003800000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests0707010000000C000081ED0000000000000000000000015BC61A730000034A000000000000000000000000000000000000004C00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/exercise_spark.bash# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #!/usr/bin/env bash ansible-playbook -i hosts/verb_hosts spark-submit.yml SPARK_STATUS=$? echo "Spark submit quick sum" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi exit 00707010000000D000081A40000000000000000000000015BC61A730000049C000000000000000000000000000000000000004700000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/test-plan.yaml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: Exercise spark logfile: exercise-spark.log exec: - exercise_spark.bash - name: Validate spark-up logfile: testsuite-test_up.log exec: - validate-spark-up.bash - name: Validate stop-start spark logfile: testsuite-test_stop-start.log exec: - validate-spark-stop-start.bash - name: Validate spark-upgrade logfile: testsuite-test_upgrade.log exec: - validate-spark-upgrade.bash - name: Validate spark-reconfigure logfile: testsuite-test_reconfigure.log exec: - validate-spark-reconfigure.bash 0707010000000E000081ED0000000000000000000000015BC61A7300000402000000000000000000000000000000000000004900000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/update-conf.bash#!/bin/bash # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # set -vxe echo "Changing $1 in $2" pushd ~/openstack/my_cloud/config sed -e "$1" -i "$2" --follow-symlinks git add -A git commit --allow-empty -m "My Config Change" pushd ~/openstack/ardana/ansible/ ansible-playbook -i hosts/localhost config-processor-run.yml -e encrypt="" \ -e rekey="" ansible-playbook -i hosts/localhost ready-deployment.yml popd popd 0707010000000F000081ED0000000000000000000000015BC61A73000004A8000000000000000000000000000000000000005800000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/validate-spark-reconfigure.bash#!/bin/bash # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # ansible-playbook -i hosts/verb_hosts spark-reconfigure.yml SPARK_STATUS=$? echo "Spark reconfigure" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi ansible-playbook -i hosts/verb_hosts spark-status.yml SPARK_STATUS=$? echo "Spark Service Status is $SPARK_STATUS" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi ansible-playbook -i hosts/verb_hosts spark-submit.yml SPARK_STATUS=$? echo "Spark submit quick sum" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi07070100000010000081ED0000000000000000000000015BC61A730000060D000000000000000000000000000000000000005700000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/validate-spark-stop-start.bash#!/bin/bash # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # ansible-playbook -i hosts/verb_hosts spark-stop.yml SPARK_STATUS=$? echo "Spark Service Stop" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi ansible-playbook -i hosts/verb_hosts spark-status.yml SPARK_STATUS=$? echo "Spark Service Status is $SPARK_STATUS" if [ $SPARK_STATUS -eq 0 ] then echo "Fail" exit 1 else echo "Ok" fi ansible-playbook -i hosts/verb_hosts spark-start.yml SPARK_STATUS=$? echo "Spark Service Start" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi ansible-playbook -i hosts/verb_hosts spark-status.yml SPARK_STATUS=$? echo "Spark Service Status is $SPARK_STATUS" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi ansible-playbook -i hosts/verb_hosts spark-submit.yml SPARK_STATUS=$? echo "Spark submit quick sum" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi07070100000011000081ED0000000000000000000000015BC61A7300000405000000000000000000000000000000000000004F00000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/validate-spark-up.bash#!/bin/bash # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # ansible-playbook -i hosts/verb_hosts spark-status.yml spark_test_status=$? echo "Spark Test Status is $spark_test_status" if [ $spark_test_status -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi ansible-playbook -i hosts/verb_hosts spark-submit.yml SPARK_STATUS=$? echo "Spark submit quick sum" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi 07070100000012000081ED0000000000000000000000015BC61A73000004A0000000000000000000000000000000000000005400000000ardana-spark-8.0+git.1539709555.5b31c25/ardana-ci/tests/validate-spark-upgrade.bash#!/bin/bash # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # ansible-playbook -i hosts/verb_hosts spark-upgrade.yml SPARK_STATUS=$? echo "Spark upgrade" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi ansible-playbook -i hosts/verb_hosts spark-status.yml SPARK_STATUS=$? echo "Spark Service Status is $SPARK_STATUS" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi ansible-playbook -i hosts/verb_hosts spark-submit.yml SPARK_STATUS=$? echo "Spark submit quick sum" if [ $SPARK_STATUS -eq 0 ] then echo "Ok" else echo "Fail" exit 1 fi07070100000013000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000002F00000000ardana-spark-8.0+git.1539709555.5b31c25/config07070100000014000081A40000000000000000000000015BC61A73000003CF000000000000000000000000000000000000004200000000ardana-spark-8.0+git.1539709555.5b31c25/config/spark-symlinks.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # # The following relative symlinks are created under the # my_cloud/config directory. --- symlinks: "spark/spark-defaults.conf.j2": "roles/FND-SPA/templates/spark-defaults.conf.j2" "spark/spark-env.sh.j2": "roles/FND-SPA/templates/spark-env.sh.j2" "spark/spark-worker-env.sh.j2": "roles/FND-SPA/templates/spark-worker-env.sh.j2" 07070100000015000041ED0000000000000000000000035BC61A7300000000000000000000000000000000000000000000002E00000000ardana-spark-8.0+git.1539709555.5b31c25/roles07070100000016000041ED0000000000000000000000075BC61A7300000000000000000000000000000000000000000000003600000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA07070100000017000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003F00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/defaults07070100000018000081A40000000000000000000000015BC61A7300001289000000000000000000000000000000000000004800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/defaults/main.yml# # (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- spark_component_name: spark spark_hosts: "{{ FND_SPA.members.master }}" spark_group_name: "{{ verb_hosts.FND_SPA }}" spark_local_hostname: > {{ (spark_hosts | selectattr('ardana_ansible_host','equalto', inventory_hostname) | first).host }} spark_ver: 1.6.3 spark_hadoop_ver: "{{ '2.6' if deployer_media_legacy_layout|bool else '2.4' }}" spark_tarball_name: spark-{{ spark_ver }}-bin-hadoop{{ spark_hadoop_ver }}.tgz spark_streaming_kafka_version_jar: spark-streaming-kafka_2.10-{{ spark_ver }}.jar spark_streaming_kafka_jar: spark-streaming-kafka.jar spark_root_dir: "{{ spark_component_name | bin_dir() }}/.." spark_venv_lib_dir: "{%- if deployer_media_legacy_layout|bool -%} {{ spark_component_name | jar_dir() }} {%- else -%} /usr/share/spark/lib/ {%- endif -%}" spark_jdbc_jar: "{%- if deployer_media_legacy_layout|bool -%} drizzle-jdbc-1.3.jar {%- else -%} mysql-connector-java.jar {%- endif -%}" spark_libs: - ["kafka_2.10-0.8.1.1.jar", "kafka_2.10-0.8.1.1/libs"] - ["scala-library-2.10.1.jar", "kafka_2.10-0.8.1.1/libs"] - ["metrics-core-2.2.0.jar", "kafka_2.10-0.8.1.1/libs"] - ["{{ spark_streaming_kafka_version_jar }}", ""] - ["{{ spark_jdbc_jar }}", ""] spark_current_dir: "{{ spark_root_dir }}/current" spark_versioned_dir: "{{ spark_root_dir }}/spark-{{ spark_ver }}-bin-hadoop{{ spark_hadoop_ver }}" spark_lib_jars_location: "{{ spark_versioned_dir }}/lib" spark_run_dir: /var/run/spark spark_worker_directory: "{{ spark_run_dir }}/work" spark_conf_dir: /etc/spark/conf spark_init_dir: /etc/spark/init spark_log_dir: /var/log/spark spark_tools_dir: "{{ spark_versioned_dir }}/tools" spark_master_log_file: "{{ spark_log_dir }}/{{ spark_component_name }}-master.log" spark_worker_log_file: "{{ spark_log_dir }}/{{ spark_component_name }}-worker.log" spark_event_log_directory: "{{ spark_log_dir }}/events" spark_group: spark spark_user: spark # spark configuration properties spark_executor_cores: 1 spark_executor_memory: 512m spark_cores_max: 3 spark_driver_memory: 512m spark_python_worker_memory: 16m spark_worker_memory: 512m spark_worker_cores: 1 spark_daemon_x_ms: 16m spark_daemon_x_mx: 16m spark_daemon_max_perm_size: 32m spark_sql_shuffle_partitions: 32 spark_worker_cleanup_enabled: true spark_cleaner_ttl: 900 spark_sql_ui_retainedExecutions: 10 spark_streaming_ui_retainedBatches: 10 spark_worker_ui_retainedExecutors: 10 spark_worker_ui_retainedDrivers: 10 spark_ui_retainedJobs: 10 spark_ui_retainedStages: 10 spark_executor_logs_rolling_maxRetainedFiles: 24 spark_executor_logs_rolling_strategy: time spark_executor_logs_rolling_time_interval: hourly spark_master_port: "{{ host.bind.FND_SPA.master.port }}" spark_worker_port: "{{ host.bind.FND_SPA.worker.port }}" spark_master_webui_port: "{{ host.bind.FND_SPA.master_web_ui.port }}" spark_worker_webui_port: "{{ host.bind.FND_SPA.worker_web_ui.port }}" spark_blockmanager_port: "{{ host.bind.FND_SPA.block_manager.port }}" spark_broadcast_port: "{{ host.bind.FND_SPA.broadcast.port }}" spark_driver_port: "{{ host.bind.FND_SPA.driver.port }}" spark_executor_port: "{{ host.bind.FND_SPA.executor.port }}" spark_fileserver_port: "{{ host.bind.FND_SPA.fileserver.port }}" spark_masters: "{% for node in FND_SPA.members.master %}{{ node.host }}:{{ node.port }}{% if not loop.last %},{% endif %}{% endfor %}" # zookeeper spark_zookeeper_nodes: "{{ FND_SPA.consumes_FND_ZOO.members.private }}" spark_zookeeper_hosts: "{% for node in spark_zookeeper_nodes %}{{ node.host }}:{{ node.port }}{% if not loop.last %},{% endif %}{% endfor %}" spark_init: systemd # upstart spark_master_upstart_conf: /etc/init/spark-master.conf spark_worker_upstart_conf: /etc/init/spark-worker.conf # systemd spark_master_systemd_service: /etc/systemd/system/spark-master.service spark_worker_systemd_service: /etc/systemd/system/spark-worker.service # monitoring spark_agent_detection_plugin_dir: "{{ monasca_agent_detection_plugin_dir }}" 07070100000019000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003C00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/files0707010000001A000081A40000000000000000000000015BC61A7300000547000000000000000000000000000000000000004B00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/files/SparkDetect.py# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # import logging import monasca_setup.detection log = logging.getLogger(__name__) class SparkDetect(monasca_setup.detection.ServicePlugin): """Detect Spark daemons and setup configuration to monitor them.""" def __init__(self, template_dir, overwrite=True, args=None): log.info("\tWatching the spark processes.") service_params = { 'args': {}, 'template_dir': template_dir, 'overwrite': overwrite, 'service_name': 'spark', 'process_names': ['org.apache.spark.deploy.master.Master', 'org.apache.spark.deploy.worker.Worker'] } super(SparkDetect, self).__init__(service_params) 0707010000001B000081A40000000000000000000000015BC61A7300000422000000000000000000000000000000000000004900000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/files/quick_sum.py# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # from __future__ import print_function from pyspark import SparkConf from pyspark import SparkContext if __name__ == "__main__": my_spark_conf = SparkConf().setAppName("Quicksum") spark_context = SparkContext(conf=my_spark_conf) data = [1, 2, 3, 4, 5] distData = spark_context.parallelize(data) total = distData.reduce(lambda a, b: a + b) print("Total is %s" % total) spark_context.stop() 0707010000001C000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003C00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks0707010000001D000081A40000000000000000000000015BC61A730000030F000000000000000000000000000000000000005200000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/_schedule_restart.yml# # (c) Copyright 2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | _schedule_restart | Schedule a restart for all services debug: msg: "Trigger a change notification in spark" changed_when: true register: ardana_notify_spark_restart_required 0707010000001E000081A40000000000000000000000015BC61A7300000604000000000000000000000000000000000000004A00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/configure.yml# # (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | configure | Link current version become: yes file: path: "{{ spark_current_dir }}" state: link src: "{{ spark_versioned_dir }}" when: deployer_media_legacy_layout|bool - name: FND-SPA | configure | Set up spark-env (rpm) become: yes template: src: spark-env.j2 dest: /etc/spark/spark-env owner: "{{ spark_user }}" group: "{{ spark_group }}" mode: 0644 when: deployer_media_legacy_layout|bool == False - name: FND-SPA | configure | Set up spark-defaults (rpm) become: yes template: src: spark-defaults.conf.j2 dest: /etc/spark/spark-defaults.conf owner: "{{ spark_user }}" group: "{{ spark_group }}" mode: 0644 when: deployer_media_legacy_layout|bool == False - name: FND-SPA | configure | include configure tasks (legacy) include: configure_legacy.yml when: deployer_media_legacy_layout|bool 0707010000001F000081A40000000000000000000000015BC61A7300000E88000000000000000000000000000000000000005100000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/configure_legacy.yml# # (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | configure_legacy | Link spark commands into /usr/bin become: yes file: path: /usr/bin/{{ item }} state: link src: "{{ spark_current_dir }}/bin/{{ item }}" with_items: - spark-submit - spark-class - spark-shell - spark-sql - name: FND-SPA | configure_legacy | Link spark libs become: yes file: path: "{{ spark_lib_jars_location }}/{{ spark_streaming_kafka_jar }}" state: link src: "{{ spark_lib_jars_location }}/{{ spark_streaming_kafka_version_jar }}" - name: FND-SPA | configure_legacy | Set up spark-env become: yes template: src: spark-env.sh.j2 dest: /etc/spark/conf/spark-env.sh owner: "{{ spark_user }}" group: "{{ spark_group }}" mode: 0644 - name: FND-SPA | configure_legacy | Set up spark-worker-env become: yes template: src: spark-worker-env.sh.j2 dest: /etc/spark/conf/spark-worker-env.sh owner: "{{ spark_user }}" group: "{{ spark_group }}" mode: 0644 - name: FND-SPA | configure_legacy | Set up spark-defaults (legacy) become: yes template: src: spark-defaults.conf.j2 dest: /etc/spark/conf/spark-defaults.conf owner: "{{ spark_user }}" group: "{{ spark_group }}" mode: 0644 - name: FND-SPA | configure_legacy | Create spark-master start script become: yes template: dest: "{{ spark_init_dir }}/start-spark-master.sh" owner: root group: root mode: 0755 src: start-spark-master.sh.j2 - name: FND-SPA | configure_legacy | Create spark-worker start script become: yes template: dest: "{{ spark_init_dir }}/start-spark-worker.sh" owner: root group: root mode: 0755 src: start-spark-worker.sh.j2 - name: FND-SPA | configure_legacy | Create spark-master systemd config become: yes template: src: spark-master.service.j2 dest: "{{ spark_master_systemd_service }}" owner: root group: root mode: 0644 - name: FND-SPA | configure_legacy | Create spark-worker systemd config become: yes template: src: spark-worker.service.j2 dest: "{{ spark_worker_systemd_service }}" owner: root group: root mode: 0644 # See: http://www.freedesktop.org/software/systemd/man/tmpfiles.d.html - name: FND-SPA | configure_legacy | Create spark-master.conf become: yes template: src: spark-master.conf.j2 dest: "/usr/lib/tmpfiles.d/spark-master.conf" owner: root group: root mode: 0644 - name: FND-SPA | configure_legacy | Create spark-worker.conf become: yes template: src: spark-worker.conf.j2 dest: "/usr/lib/tmpfiles.d/spark-worker.conf" owner: root group: root mode: 0644 - name: FND-SPA | configure_legacy | Reload systemctl become: yes ignore_errors: True command: /bin/systemctl daemon-reload - name: FND-SPA | configure_legacy | Link config version become: yes file: path: "{{ spark_current_dir }}/conf/{{ item }}" state: link src: "{{ spark_conf_dir }}/{{ item }}" with_items: - "spark-env.sh" - "spark-worker-env.sh" - "spark-defaults.conf" 07070100000020000081A40000000000000000000000015BC61A73000008C1000000000000000000000000000000000000004800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/install.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | install | include OS specific variables include_vars: "{{ ansible_os_family | lower }}.yml" - name: FND-SPA | install | Install the spark package from repo become: yes package: name: "{{ item }}" state: present when: deployer_media_legacy_layout|bool == False with_items: - "spark" - "kafka" - "mysql-connector-java" - name: FND-SPA | install | create symlink to mysql connector jdbc jar become: yes file: src: "/usr/share/java/{{ spark_jdbc_jar }}" dest: "{{ spark_venv_lib_dir }}/{{ spark_jdbc_jar }}" state: link when: deployer_media_legacy_layout|bool == False - name: FND-SPA | install | include legacy layout tasks include: install_legacy.yml when: deployer_media_legacy_layout|bool - name: FND-SPA | install | Create Spark local directories become: yes file: path: "{{ item }}" state: directory owner: "{{ spark_user }}" group: "{{ spark_group }}" mode: 0775 with_items: - "{{ spark_run_dir }}" - "{{ spark_worker_directory }}" - "{{ spark_tools_dir }}" - name: FND-SPA | install | Create Spark log directory become: yes file: path: "{{ item }}" state: directory owner: "{{ spark_user }}" group: adm mode: 0770 with_items: - "{{ spark_log_dir }}" when: deployer_media_legacy_layout|bool - name: FND-SPA | install | Touch logfiles become: true file: path: "{{ item }}" state: touch owner: "{{ spark_user }}" group: adm mode: 0640 with_items: - "{{ spark_master_log_file }}" - "{{ spark_worker_log_file }}" 07070100000021000081A40000000000000000000000015BC61A7300000A1F000000000000000000000000000000000000004F00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/install_legacy.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | install_legacy | Setup group become_user: root become: True group: name: "{{ spark_group }}" system: yes - name: FND-SPA | install_legacy | Setup user become: yes user: name: "{{ spark_user }}" system: yes group: "{{ spark_group }}" createhome: no home: /nonexistent shell: /usr/sbin/nologin comment: "Apache Spark system user" - name: FND-SPA | install_legacy | Install OS specific required packages become: yes package: name: "{{ item }}" state: present with_items: fnd_spark_required_packages - name: FND-SPA | install_legacy | Update installed packager cache become: yes install_package: cache: update - name: FND-SPA | install_legacy | Install the spark venv become: yes install_package: name: spark service: spark state: present - name: FND-SPA | install_legacy | Uncompress the spark tar become: yes unarchive: copy: no creates: > "{{ spark_root_dir }}/spark-{{ spark_ver }}- bin-hadoop{{ spark_hadoop_ver }}" dest: "{{ spark_root_dir }}" src: "{{ spark_venv_lib_dir }}/{{ spark_tarball_name }}" - name: FND-SPA | install_legacy | Untar kafka tarball become: yes unarchive: copy: no creates: "{{ spark_venv_lib_dir }}/kafka_2.10-0.8.1.1" dest: "{{ spark_venv_lib_dir }}" src: "{{ spark_venv_lib_dir }}/kafka_2.10-0.8.1.1.tgz" - name: FND-SPA | install_legacy | Link lib jars to spark lib directory become: yes file: state: link dest: "{{ spark_lib_jars_location }}/{{ item[0] }}" src: "{{ spark_venv_lib_dir }}/{{ item[1] }}/{{ item[0] }}" with_items: - "{{ spark_libs }}" - name: FND-SPA | install_legacy | Create Spark local directories become: yes file: path: "{{ item }}" state: directory owner: "{{ spark_user }}" group: "{{ spark_group }}" mode: 0775 with_items: - "{{ spark_conf_dir }}" - "{{ spark_init_dir }}"07070100000022000081A40000000000000000000000015BC61A7300000417000000000000000000000000000000000000005200000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/monasca_configure.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | monasca_configure | Copy Spark Custom plugin for Ardana only become: yes copy: src: SparkDetect.py dest: "{{ spark_agent_detection_plugin_dir }}/SparkDetect.py" owner: root group: root mode: 0440 - name: FND-SPA | monasca_configure | Run Monasca agent Ardana specific Spark detection plugin become: yes monasca_agent_plugin: name: "SparkDetect" 07070100000023000081A40000000000000000000000015BC61A73000002A2000000000000000000000000000000000000005000000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/password-change.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- # nothing to do I don't think 07070100000024000081A40000000000000000000000015BC61A73000005F5000000000000000000000000000000000000004D00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/spark_submit.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | spark_submit | Push quick_sum.py script for validation become: yes copy: src: quick_sum.py dest: "{{ spark_tools_dir }}/quick_sum.py" owner: "{{ spark_user }}" group: "{{ spark_group }}" mode: 0775 - name: FND-SPA | spark_submit | Submit quick_sum.py to spark become_user: "{{ spark_user }}" become: yes shell: > export SPARK_HOME={{ spark_current_dir }} && spark-submit --supervise --master spark://{{ spark_masters }} quick_sum.py register: spark_quick_sum_result args: chdir: "{{ spark_tools_dir }}" run_once: True - name: FND-SPA | spark_submit | Announce quick sum result if failed fail: msg: | Quick sum failed with output - {{ spark_quick_sum_result.stdout }} {{ spark_quick_sum_result.stderr }} when: not ("{{ spark_quick_sum_result.stdout }}" == "Total is 15") run_once: true 07070100000025000081A40000000000000000000000015BC61A7300000592000000000000000000000000000000000000004600000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/start.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | start | restart spark master # start spark master service: name: spark-master state: restarted become: yes when: ardana_notify_spark_restart_required is defined and ardana_notify_spark_restart_required.changed - name: FND-SPA | start | start spark master # start spark master service: name: spark-master state: started become: yes - name: FND-SPA | start | restart spark worker # start spark worker service: name: spark-worker state: restarted become: yes when: ardana_notify_spark_restart_required is defined and ardana_notify_spark_restart_required.changed - name: FND-SPA | start | start spark worker # start spark worker service: name: spark-worker state: started become: yes 07070100000026000081A40000000000000000000000015BC61A730000071B000000000000000000000000000000000000004700000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/status.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | status | Print service being checked debug: msg: "Running service check for spark" run_once: true - name: FND-SPA | status | Check systemd service running spark-master command: systemctl status spark-master ignore_errors: yes changed_when: false register: spark_systemctl_status_master_result - name: FND-SPA | status | Report status of spark-master fail: msg: | spark-master is not running. systemctl status spark-master output: {{ spark_systemctl_status_master_result.stdout }} {{ spark_systemctl_status_master_result.stderr }} when: spark_systemctl_status_master_result | failed - name: FND-SPA | status | Check systemd service running spark-worker command: systemctl status spark-worker ignore_errors: yes changed_when: false register: spark_systemctl_status_worker_result - name: FND-SPA | status | Report status of spark-worker fail: msg: | spark-worker is not running. systemctl status spark-worker output: {{ spark_systemctl_status_worker_result.stdout }} {{ spark_systemctl_status_worker_result.stderr }} when: spark_systemctl_status_worker_result | failed 07070100000027000081A40000000000000000000000015BC61A7300000360000000000000000000000000000000000000004500000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/stop.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | stop | stop spark worker become: yes service: name: spark-worker state: stopped - name: FND-SPA | stop | stop spark master become: yes service: name: spark-master state: stopped 07070100000028000081A40000000000000000000000015BC61A7300000434000000000000000000000000000000000000004700000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/unlink.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - name: FND-SPA | unlink | Unlink current version become: yes file: path: "{{ spark_current_dir }}" state: absent when: deployer_media_legacy_layout|bool - name: FND-SPA | unlink | Unlink spark commands in /usr/bin become: yes file: path: /usr/bin/{{ item }} state: absent with_items: - spark-submit - spark-class - spark-shell - spark-sql when: deployer_media_legacy_layout|bool 07070100000029000081A40000000000000000000000015BC61A730000035B000000000000000000000000000000000000004800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/upgrade.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- ## upgrade is not yet implemented as this is the initial release - include: stop.yml - include: unlink.yml - include: install.yml - include: configure.yml - include: start.yml - include: monasca_configure.yml0707010000002A000081A40000000000000000000000015BC61A730000040B000000000000000000000000000000000000004D00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/tasks/upstart_init.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: create spark-master upstart config become: yes template: dest: "{{ spark_master_upstart_conf }}" owner: root group: root mode: 0644 src: upstart/spark-master.conf.j2 - name: create spark-worker upstart config become: yes template: dest: "{{ spark_worker_upstart_conf }}" owner: root group: root mode: 0644 src: upstart/spark-worker.conf.j20707010000002B000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000004000000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates0707010000002C000081ED0000000000000000000000015BC61A730000033F000000000000000000000000000000000000005500000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/exercise_spark.sh.j2# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #!/usr/bin/env bash export SPARK_HOME=/opt/stack/service/spark/venv/bin/../current CALC_VALUE=`bash $SPARK_HOME/bin/spark-submit quick_sum.py` echo The calculated value is $CALC_VALUE exit 00707010000002D000081A40000000000000000000000015BC61A73000008A9000000000000000000000000000000000000005700000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-defaults.conf.j2{# # # (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} spark.driver.extraClassPath {{ spark_venv_lib_dir }}/{{ spark_jdbc_jar }} spark.executor.extraClassPath {{ spark_venv_lib_dir }}/{{ spark_jdbc_jar }} spark.blockManager.port {{ spark_blockmanager_port }} spark.broadcast.port {{ spark_broadcast_port }} spark.cores.max {{ spark_cores_max }} spark.driver.memory {{ spark_driver_memory }} spark.driver.port {{ spark_driver_port }} spark.eventLog.dir {{ spark_event_log_directory }} spark.executor.cores {{ spark_executor_cores }} spark.executor.memory {{ spark_executor_memory }} spark.executor.port {{ spark_executor_port }} spark.fileserver.port {{ spark_fileserver_port }} spark.python.worker.memory {{ spark_python_worker_memory }} spark.speculation true spark.speculation.interval 200 spark.sql.shuffle.partitions {{ spark_sql_shuffle_partitions }} spark.worker.cleanup.enabled {{ spark_worker_cleanup_enabled }} spark.cleaner.ttl {{ spark_cleaner_ttl }} spark.sql.ui.retainedExecutions {{ spark_sql_ui_retainedExecutions }} spark.streaming.ui.retainedBatches {{ spark_streaming_ui_retainedBatches }} spark.worker.ui.retainedExecutors {{ spark_worker_ui_retainedExecutors }} spark.worker.ui.retainedDrivers {{ spark_worker_ui_retainedDrivers }} spark.ui.retainedJobs {{ spark_ui_retainedJobs }} spark.ui.retainedStages {{ spark_ui_retainedStages }} spark.executor.logs.rolling.maxRetainedFiles {{spark_executor_logs_rolling_maxRetainedFiles}} spark.executor.logs.rolling.strategy {{spark_executor_logs_rolling_strategy}} spark.executor.logs.rolling.time.interval {{spark_executor_logs_rolling_time_interval}} 0707010000002E000081A40000000000000000000000015BC61A7300000696000000000000000000000000000000000000004D00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-env.j2{# # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} SPARK_HOME=/usr/share/spark SPARK_LOCAL_IP={{ spark_local_hostname }} SPARK_MASTER_IP={{ spark_local_hostname }} SPARK_MASTER_PORT={{ spark_master_port }} SPARK_MASTER_WEBUI_PORT={{ spark_master_webui_port }} SPARK_WORKER_PORT={{ spark_worker_port }} SPARK_WORKER_WEBUI_PORT={{ spark_worker_webui_port }} SPARK_WORKER_DIR={{ spark_worker_directory }} SPARK_WORKER_MEMORY={{ spark_worker_memory }} SPARK_WORKER_CORES={{ spark_worker_cores }} SPARK_DAEMON_JAVA_OPTS="-Xms{{ spark_daemon_x_ms }} -Xmx{{ spark_daemon_x_mx }} -XX:MaxPermSize={{ spark_daemon_max_perm_size }} -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url={{ spark_zookeeper_hosts }} -Dspark.deploy.zookeeper.dir={{ spark_run_dir }} -Dspark.executor.logs.rolling.maxRetainedFiles=10 -Dspark.executor.logs.rolling.strategy=size -Dspark.executor.logs.rolling.maxSize=20971520 -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=1800 -Dspark.worker.cleanup.appDataTtl=86400 -Dspark.master.rest.enabled=false" SPARK_MASTERS={{ spark_masters }} 0707010000002F000081A40000000000000000000000015BC61A73000007DA000000000000000000000000000000000000005000000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-env.sh.j2#!/usr/bin/env bash {# # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017, 2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} export SPARK_LOCAL_IP={{ spark_local_hostname }} export SPARK_MASTER_IP={{ spark_local_hostname }} export SPARK_MASTER_PORT={{ spark_master_port }} export SPARK_MASTERS={{ spark_masters }} export SPARK_MASTER_WEBUI_PORT={{ spark_master_webui_port }} export SPARK_WORKER_PORT={{ spark_worker_port }} export SPARK_WORKER_WEBUI_PORT={{ spark_worker_webui_port }} export SPARK_WORKER_DIR={{ spark_worker_directory }} export SPARK_WORKER_MEMORY={{ spark_worker_memory }} export SPARK_WORKER_CORES={{ spark_worker_cores }} export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=file://{{ spark_event_log_directory }} -Dspark.history.ui.port=18082" export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Xms{{ spark_daemon_x_ms }} -Xmx{{ spark_daemon_x_mx }} -XX:MaxPermSize={{ spark_daemon_max_perm_size }} -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url={{ spark_zookeeper_hosts }} -Dspark.deploy.zookeeper.dir={{ spark_run_dir }}" export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Dspark.executor.logs.rolling.maxRetainedFiles=10 -Dspark.executor.logs.rolling.strategy=size -Dspark.executor.logs.rolling.maxSize=20971520" export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=1800 -Dspark.worker.cleanup.appDataTtl=86400"07070100000030000081A40000000000000000000000015BC61A73000002CC000000000000000000000000000000000000005500000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-master.conf.j2{# # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} d {{ spark_worker_directory }} 0755 {{ spark_user }} {{ spark_group }} 07070100000031000081A40000000000000000000000015BC61A7300000338000000000000000000000000000000000000005800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-master.service.j2{# # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} [Unit] Description=Spark Master Daemon After=zookeeper.service [Service] Type=simple ExecStart={{ spark_init_dir }}/start-spark-master.sh & [Install] WantedBy=multi-user.target 07070100000032000081A40000000000000000000000015BC61A73000006A8000000000000000000000000000000000000005700000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-worker-env.sh.j2#!/usr/bin/env bash {# # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} export SPARK_LOCAL_IP={{ spark_local_hostname }} export SPARK_MASTER_IP={{ spark_local_hostname }} export SPARK_MASTER_PORT={{ spark_master_port }} export SPARK_MASTERS={{ spark_masters }} export SPARK_MASTER_WEBUI_PORT={{ spark_master_webui_port }} export SPARK_WORKER_PORT={{ spark_worker_port }} export SPARK_WORKER_WEBUI_PORT={{ spark_worker_webui_port }} export SPARK_WORKER_DIR={{ spark_worker_directory }} export SPARK_WORKER_MEMORY={{ spark_worker_memory }} export SPARK_WORKER_CORES={{ spark_worker_cores }} # history server export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=file://{{ spark_event_log_directory }} -Dspark.history.ui.port=18082" # set up for multiple masters using zookeeper export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Xms{{ spark_daemon_x_ms }} -Xmx{{ spark_daemon_x_mx }} -XX:MaxPermSize={{ spark_daemon_max_perm_size }} -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url={{ spark_zookeeper_hosts }} -Dspark.deploy.zookeeper.dir={{ spark_run_dir }}" 07070100000033000081A40000000000000000000000015BC61A73000002CC000000000000000000000000000000000000005500000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-worker.conf.j2{# # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} d {{ spark_worker_directory }} 0755 {{ spark_user }} {{ spark_group }} 07070100000034000081A40000000000000000000000015BC61A7300000337000000000000000000000000000000000000005800000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/spark-worker.service.j2{# # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} [Unit] Description=Spark Worker Daemon After=zookeeper.service [Service] Type=simple ExecStart={{ spark_init_dir }}/start-spark-worker.sh & [Install] WantedBy=multi-user.target 07070100000035000081A40000000000000000000000015BC61A73000007E3000000000000000000000000000000000000005900000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/start-spark-master.sh.j2#!/usr/bin/env bash {# # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} . {{ spark_current_dir }}/conf/spark-env.sh export EXEC_CLASS=org.apache.spark.deploy.master.Master export INSTANCE_ID=1 export SPARK_CLASSPATH={{ spark_current_dir }}/conf/:{{ spark_current_dir }}/lib/spark-assembly-{{ spark_ver }}-hadoop{{ spark_hadoop_ver }}.0.jar:{{ spark_current_dir }}/lib/datanucleus-core-3.2.10.jar:{{ spark_current_dir }}/lib/datanucleus-rdbms-3.2.9.jar:{{ spark_current_dir }}/lib/datanucleus-api-jdo-3.2.6.jar export SPARK_HOME={{ spark_current_dir }} {% if ansible_os_family | lower == 'debian' %} start-stop-daemon -c {{ spark_user }}:{{ spark_group }} --pidfile {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid --name spark-master --start --exec /usr/bin/java -- -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --ip "$SPARK_MASTER_IP" --port "$SPARK_MASTER_PORT" --webui-port "$SPARK_MASTER_WEBUI_PORT" >> "{{ spark_master_log_file }}" 2>&1 < /dev/null {% elif ansible_os_family | lower == 'suse' %} start_daemon -u {{ spark_user }} -g {{ spark_group }} -p {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid /usr/bin/java -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --ip "$SPARK_MASTER_IP" --port "$SPARK_MASTER_PORT" --webui-port "$SPARK_MASTER_WEBUI_PORT" >> "{{ spark_master_log_file }}" 2>&1 < /dev/null {% endif %}07070100000036000081A40000000000000000000000015BC61A73000007CE000000000000000000000000000000000000005900000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/templates/start-spark-worker.sh.j2#!/usr/bin/env bash {# # # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # #} . {{ spark_current_dir }}/conf/spark-env.sh export EXEC_CLASS=org.apache.spark.deploy.worker.Worker export INSTANCE_ID=1 export SPARK_CLASSPATH={{ spark_current_dir }}/conf/:{{ spark_current_dir }}/lib/spark-assembly-{{ spark_ver }}-hadoop{{ spark_hadoop_ver }}.0.jar:{{ spark_current_dir }}/lib/datanucleus-core-3.2.10.jar:{{ spark_current_dir }}/lib/datanucleus-rdbms-3.2.9.jar:{{ spark_current_dir }}/lib/datanucleus-api-jdo-3.2.6.jar export SPARK_HOME={{ spark_current_dir }} {% if ansible_os_family | lower == 'debian' %} start-stop-daemon -c {{ spark_user }}:{{ spark_group }} --pidfile {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid --name spark-worker --start --exec /usr/bin/java -- -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --webui-port "$SPARK_WORKER_WEBUI_PORT" --port $SPARK_WORKER_PORT $SPARK_MASTERS >> "{{ spark_worker_log_file }}" 2>&1 < /dev/null {% elif ansible_os_family | lower == 'suse' %} start_daemon -u {{ spark_user }} -g {{ spark_group }} -p {{ spark_run_dir }}/{{ spark_user }}-{{ spark_group }}-"$EXEC_CLASS"-"$INSTANCE_ID".pid /usr/bin/java -cp $SPARK_CLASSPATH $SPARK_DAEMON_JAVA_OPTS "$EXEC_CLASS" --webui-port "$SPARK_WORKER_WEBUI_PORT" --port $SPARK_WORKER_PORT $SPARK_MASTERS >> "{{ spark_worker_log_file }}" 2>&1 < /dev/null {% endif %} 07070100000037000041ED0000000000000000000000025BC61A7300000000000000000000000000000000000000000000003B00000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/vars07070100000038000081A40000000000000000000000015BC61A73000002AF000000000000000000000000000000000000004600000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/vars/debian.yml# # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- fnd_spark_required_packages: - default-jre-headless fnd_spark_restart_packages: - default-jre-headless07070100000039000081A40000000000000000000000015BC61A73000002BD000000000000000000000000000000000000004400000000ardana-spark-8.0+git.1539709555.5b31c25/roles/FND-SPA/vars/suse.yml# # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- fnd_spark_required_packages: - java-1_8_0-openjdk-headless fnd_spark_restart_packages: - java-1_8_0-openjdk-headless0707010000003A000081A40000000000000000000000015BC61A7300000347000000000000000000000000000000000000004A00000000ardana-spark-8.0+git.1539709555.5b31c25/spark-deploy-monitoring-agent.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: deploy spark-monasca-agent-plugin hosts: FND-SPA roles: roles: - {role: monasca-agent, run_mode: Use} - FND-SPA tasks: - include: roles/FND-SPA/tasks/monasca_configure.yml 0707010000003B000081A40000000000000000000000015BC61A7300000373000000000000000000000000000000000000004100000000ardana-spark-8.0+git.1539709555.5b31c25/spark-deploy-service.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development Company LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: deploy spark-master and spark-worker hosts: FND-SPA roles: - FND-SPA tasks: - include: roles/FND-SPA/tasks/install.yml - include: roles/FND-SPA/tasks/configure.yml - include: roles/FND-SPA/tasks/start.yml 0707010000003C000081A40000000000000000000000015BC61A730000031D000000000000000000000000000000000000003900000000ardana-spark-8.0+git.1539709555.5b31c25/spark-deploy.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # # deploy the service itself - include: spark-deploy-service.yml # create monitoring checks for spark Services - include: spark-deploy-monitoring-agent.yml 0707010000003D000081A40000000000000000000000015BC61A7300000392000000000000000000000000000000000000004200000000ardana-spark-8.0+git.1539709555.5b31c25/spark-password-change.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: reconfigure spark service hosts: FND-SPA roles: - FND-SPA tasks: - include: roles/FND-SPA/tasks/stop.yml - include: roles/FND-SPA/tasks/password-change.yml - include: roles/FND-SPA/tasks/configure.yml - include: roles/FND-SPA/tasks/start.yml0707010000003E000081A40000000000000000000000015BC61A7300000367000000000000000000000000000000000000003E00000000ardana-spark-8.0+git.1539709555.5b31c25/spark-reconfigure.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: reconfigure spark service hosts: FND-SPA roles: - FND-SPA serial: 1 tasks: - include: roles/FND-SPA/tasks/stop.yml - include: roles/FND-SPA/tasks/configure.yml - include: roles/FND-SPA/tasks/start.yml0707010000003F000081A40000000000000000000000015BC61A7300000287000000000000000000000000000000000000003A00000000ardana-spark-8.0+git.1539709555.5b31c25/spark-restart.yml# # (c) Copyright 2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - include: _spark-schedule-restart.yml - include: spark-start.yml 07070100000040000081A40000000000000000000000015BC61A7300000308000000000000000000000000000000000000003800000000ardana-spark-8.0+git.1539709555.5b31c25/spark-start.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: start spark-master and spark-worker hosts: FND-SPA roles: - FND-SPA tasks: - include: roles/FND-SPA/tasks/start.yml07070100000041000081A40000000000000000000000015BC61A730000030A000000000000000000000000000000000000003900000000ardana-spark-8.0+git.1539709555.5b31c25/spark-status.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: status spark-master and spark-worker hosts: FND-SPA roles: - FND-SPA tasks: - include: roles/FND-SPA/tasks/status.yml07070100000042000081A40000000000000000000000015BC61A7300000306000000000000000000000000000000000000003700000000ardana-spark-8.0+git.1539709555.5b31c25/spark-stop.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: stop spark-master and spark-worker hosts: FND-SPA roles: - FND-SPA tasks: - include: roles/FND-SPA/tasks/stop.yml07070100000043000081A40000000000000000000000015BC61A7300000306000000000000000000000000000000000000003900000000ardana-spark-8.0+git.1539709555.5b31c25/spark-submit.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # - name: submit sample job to spark hosts: FND-SPA roles: - FND-SPA tasks: - include: roles/FND-SPA/tasks/spark_submit.yml07070100000044000081A40000000000000000000000015BC61A730000031A000000000000000000000000000000000000003A00000000ardana-spark-8.0+git.1539709555.5b31c25/spark-upgrade.yml# # (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- - hosts: FND-SPA roles: - {role: monasca-agent, run_mode: Use} - FND-SPA serial: 1 tasks: - include: roles/FND-SPA/tasks/upgrade.yml 07070100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000B00000000TRAILER!!!177 blocks
Locations
Projects
Search
Status Monitor
Help
OpenBuildService.org
Documentation
API Documentation
Code of Conduct
Contact
Support
@OBShq
Terms
openSUSE Build Service is sponsored by
The Open Build Service is an
openSUSE project
.
Sign Up
Log In
Places
Places
All Projects
Status Monitor
