Sign Up
Log In
Log In
or
Sign Up
Places
All Projects
Status Monitor
Collapse sidebar
home:dirkmueller:acdc:sp5-rebuild
salt.33748
early-feature-support-config.patch
Overview
Repositories
Revisions
Requests
Users
Attributes
Meta
File early-feature-support-config.patch of Package salt.33748
From 7b47e6f19b38d773a6ec744209753f3d29b094ea Mon Sep 17 00:00:00 2001 From: Alexander Graul <agraul@suse.com> Date: Tue, 18 Jan 2022 16:40:45 +0100 Subject: [PATCH] early feature: support-config Add support script function Add salt-support starter Initial support wrapper Add data collector skeleton Add default scenario of the support configuration Add main flow for the collector. Move support library to its own package Add default support collection scenario Add logging Handle CLI error. Update format of the default support scenario Default archive name Finalise local data collection Write archive from memory objects. Add colored console outputter for salt-support. Use colored outputter Add message output class Remove try/except capture from the scripts and move to the runner directly Implement output highlighter methods for CLI output Move scenarios to profiles Get return section from the output. Tolerate raw data. Implement internal data collector Add network stack examination to the default profile Add an internal filetree function Add a method to discard current session Add a method to link a static file to the resulting archive Implement internal function caller Add internal functions Add default root for the one-file support data Set output device Separate dynamic data and static files on the fs Update color theme Add ident to the error message Report rejected files with the ident Reuse system error exceptions and reduce stat on the file check Use socket name of the host machine Add options for profile and archive settings Use archive name from options. Get profile by config/options Cleanup broken archive on crash/exception Use profile from the options/configuration Add more colored messages :-) Initial implementation of get static profiles Update docstring Move PostgreSQL profile to its own Handle profile listing, do not yield sys.exit on specific module Add network profile Add Salt's profile Uncomment package profile Allow several profiles to be specified Remove comments, add parameter to get more profiles Implement existing configuration finder Add options to handle unit configurations Pre-parse options prior run() to choose proper configuration target Handle arg parse generic errors, unit mis-choose Let cleanup be aware of pre-config state Fix imports Handle exit codes properly Allow to overwrite existing archive Use py2/3 exceptions equally Include exit exception on debugging Render profiles as Jinja2, add basic recursive caller to the template of the profile Add "users" profile Implement basic caller for the profile template Add table output renderer Fix typo Remove table outputter Allow default outputters and specify outputters inside the profile Remove group.getent from the loop per each user Add table outputter to network profile Add text outputter to hostname/fqdn data Remove network part from the default profile. Add text/table outputters. Fix Py3 compat Collect status (initial) Avoid irrelevant to profile files Add job profiles Add profile template trace Add inspection through the runners Allow parameters in callers and runners Handle non-dict iterables Highlight template content in the trace log Add return extractor from the local call returns Move local runner to its own namespace Lintfix: PEP8 Remove duplicate code Fix caller return Add description tag to the scenario Add generic colored message Add wrapping function. NOTE: it should be refactored with the other similar functions Print description while processing the scenario Turn off default profile and print help instead Move command-line check before collector Do not verify archive if help needs to be printed Add console output unit test for indent output Fix docstring Rename test class Refactor test to add setup/teardown Add unit test to verify indent Use direct constants instead of encoded strings Add unit test for color indent rotation check Add a test case for Collector class Add unit test for closing the archive Add unit test for add/write sections on the collector object Add test for linking an external file Cleanup tests on tear-down method Add call count check Add unit test for support collection section discard Add unittest for SaltSupport's function config preparation Fix docstring Add unit test for local caller Add unit test for local runner Add unit test for internal function call Add unit test for getting an action description from the action meta Add unit test for internal function call Add unit test for return extration Add unit test for determine action type from the action meta Add unit test for cleanup routine Fix typo of method name Add unit test for check existing archive Add test suite for profile testing Add unit test for default profile is YAML-parseable Add unit test for user template profile rendering Update unit test for all non-template profiles parse check Add function to render a Jinja2 template by name Use template rendering function Add unit test on jobs-trace template for runner Move function above the tests Add current logfile, if defined in configuration Bugfix: ignore logfile, if path was not found or not defined or is None Lintfix: iteration over .keys() Remove template "salt" from non-template checks Lintfix: use salt.utils.files.fopen for resource leak prevention Lintfix: PEP8 E302: expected 2 blank lines, found 0 Lintfix: use salt.utils.files.fopen instead of open Lintfix: PEP8 E303: too many blank lines (3) Lintfix: Uses of an external blacklisted import 'six': Please use 'import salt.ext.six as six' Lintfix: use salt.utils.files.fopen instead of open Fix unit tests Fix six import Mute pylint: file handler explicitly needed Lintfix: explicitly close filehandle Lintfix: mute fopen warning Remove development stub. Ughh... Removed blacklist of pkg_resources Make profiles a package. Add UTF-8 encoding Add a docstring Support-config non-root permission issues fixes (U#50095) Do not crash if there is no configuration available at all Handle CLI and log errors Catch overwriting exiting archive error by other users Suppress excessive tracebacks on error log level Add multi-file support and globbing to the filetree (U#50018) Add more possible logs Support multiple files grabbing Collect system logs and boot logs Support globbing in filetree Add supportconfig module for remote calls and SaltSSH Add log collector for remote purposes Implement default archive name Fix imports Implement runner function Remove targets data collector function as it is now called by a module instead Add external method decorator marker Add utility class for detecting exportable methods Mark run method as an external function Implement function setter Fix imports Setup config from __opts__ Use utility class Remove utils class Allow specify profile from the API parameter directly Rename module by virtual name Bypass parent subclass Implement profiles listing (local only for now) Specify profile from the state/call Set default or personalised archive name Add archives lister Add personalised name element to the archive name Use proper args/kwargs to the exported function Add archives deletion function Change log level when debugging rendered profiles Add ability to directly pass profile source when taking local data Add pillar profile support Remove extra-line Fix header Change output format for deleting archives Refactor logger output format Add time/milliseconds to each log notification Fix imports Switch output destination by context Add last archive function Lintfix Return consistent type Change output format for deleted archives report Implement report archive syncing to the reporting node Send multiple files at once via rsync, instead of send one after another Add sync stats formatter Change signature: cleanup -> move. Update docstring. Flush empty data from the output format Report archfiles activity Refactor imports Do not remove retcode if it is EX_OK Do not raise rsync error for undefined archives. Update header Add salt-support state module Move all functions into a callable class object Support __call__ function in state and command modules as default entrance that does not need to be specified in SLS state syntax Access from the outside only allowed class methods Pre-create destination of the archive, preventing single archive copied as a group name Handle functions exceptions Add unit test scaffold Add LogCollector UT for testing regular message Add LogCollector UT for testing INFO message Add LogCollector UT for testing WARNING message Replace hardcoded variables with defined constants Add LogCollector UT for testing ERROR message Test title attribute in msg method of LogCollector Add UT for LogCollector on highlighter method Add UT for LogCollector on put method Fix docstrings Add UT for archive name generator Add UT for custom archive name Fix docstring for the UT Add UT for checking profiles list format Add Unit Test for existing archives listing Add UT for the last archive function Create instance of the support class Add UT for successfully deleting all archives Add UT for deleting archives with failures Add UI for formatting sync stats and order preservation Add UT for testing sync failure when no archives has been specified Add UT for last picked archive has not found Add UT for last specified archive was not found Bugfix: do not create an array with None element in it Fix UT for found bugfix Add UT for syncing no archives failure Add UT for sync function Add UT for run support function Fix docstring for function "run" lintfix: use 'salt.support.mock' and 'patch()' Rewrite subdirectory creation and do not rely on Python3-only code Lintfix: remove unused imports Lintfix: regexp strings Break-down oneliner if/else clause Use ordered dictionary to preserve order of the state. This has transparent effect to the current process: OrderedDict is the same as just Python dict, except it is preserving order of the state chunks. Refactor state processing class. Add __call__ function to process single-id syntax Add backward-compatibility with default SLS syntax (id-per-call) Lintfix: E1120 no value in argument 'name' for class constructor Remove unused import Check last function by full name --- doc/ref/modules/all/index.rst | 1 + doc/ref/states/all/index.rst | 1 + salt/cli/support/__init__.py | 76 +++ salt/cli/support/collector.py | 563 ++++++++++++++++++++++ salt/cli/support/console.py | 184 +++++++ salt/cli/support/intfunc.py | 51 ++ salt/cli/support/localrunner.py | 33 ++ salt/cli/support/profiles/__init__.py | 4 + salt/cli/support/profiles/default.yml | 78 +++ salt/cli/support/profiles/jobs-active.yml | 3 + salt/cli/support/profiles/jobs-last.yml | 3 + salt/cli/support/profiles/jobs-trace.yml | 7 + salt/cli/support/profiles/network.yml | 27 ++ salt/cli/support/profiles/postgres.yml | 11 + salt/cli/support/profiles/salt.yml | 9 + salt/cli/support/profiles/users.yml | 22 + salt/loader/lazy.py | 6 +- salt/modules/saltsupport.py | 405 ++++++++++++++++ salt/scripts.py | 15 + salt/state.py | 38 +- salt/states/saltsupport.py | 225 +++++++++ salt/utils/args.py | 3 +- salt/utils/decorators/__init__.py | 24 + salt/utils/parsers.py | 114 +++++ scripts/salt-support | 11 + setup.py | 2 + tests/pytests/unit/cli/test_support.py | 553 +++++++++++++++++++++ tests/unit/modules/test_saltsupport.py | 496 +++++++++++++++++++ 28 files changed, 2958 insertions(+), 7 deletions(-) create mode 100644 salt/cli/support/__init__.py create mode 100644 salt/cli/support/collector.py create mode 100644 salt/cli/support/console.py create mode 100644 salt/cli/support/intfunc.py create mode 100644 salt/cli/support/localrunner.py create mode 100644 salt/cli/support/profiles/__init__.py create mode 100644 salt/cli/support/profiles/default.yml create mode 100644 salt/cli/support/profiles/jobs-active.yml create mode 100644 salt/cli/support/profiles/jobs-last.yml create mode 100644 salt/cli/support/profiles/jobs-trace.yml create mode 100644 salt/cli/support/profiles/network.yml create mode 100644 salt/cli/support/profiles/postgres.yml create mode 100644 salt/cli/support/profiles/salt.yml create mode 100644 salt/cli/support/profiles/users.yml create mode 100644 salt/modules/saltsupport.py create mode 100644 salt/states/saltsupport.py create mode 100755 scripts/salt-support create mode 100644 tests/pytests/unit/cli/test_support.py create mode 100644 tests/unit/modules/test_saltsupport.py diff --git a/doc/ref/modules/all/index.rst b/doc/ref/modules/all/index.rst index cbd8b0cdc5..abd40e0bc7 100644 --- a/doc/ref/modules/all/index.rst +++ b/doc/ref/modules/all/index.rst @@ -416,6 +416,7 @@ execution modules salt_version saltcheck saltcloudmod + saltsupport saltutil schedule scp_mod diff --git a/doc/ref/states/all/index.rst b/doc/ref/states/all/index.rst index 13ff645b59..7a062c227b 100644 --- a/doc/ref/states/all/index.rst +++ b/doc/ref/states/all/index.rst @@ -283,6 +283,7 @@ state modules rvm salt_proxy saltmod + saltsupport saltutil schedule selinux diff --git a/salt/cli/support/__init__.py b/salt/cli/support/__init__.py new file mode 100644 index 0000000000..59c2609e07 --- /dev/null +++ b/salt/cli/support/__init__.py @@ -0,0 +1,76 @@ +""" +Get default scenario of the support. +""" +import logging +import os + +import jinja2 +import salt.exceptions +import yaml + +log = logging.getLogger(__name__) + + +def _render_profile(path, caller, runner): + """ + Render profile as Jinja2. + :param path: + :return: + """ + env = jinja2.Environment( + loader=jinja2.FileSystemLoader(os.path.dirname(path)), trim_blocks=False + ) + return ( + env.get_template(os.path.basename(path)) + .render(salt=caller, runners=runner) + .strip() + ) + + +def get_profile(profile, caller, runner): + """ + Get profile. + + :param profile: + :return: + """ + profiles = profile.split(",") + data = {} + for profile in profiles: + if os.path.basename(profile) == profile: + profile = profile.split(".")[0] # Trim extension if someone added it + profile_path = os.path.join( + os.path.dirname(__file__), "profiles", profile + ".yml" + ) + else: + profile_path = profile + if os.path.exists(profile_path): + try: + rendered_template = _render_profile(profile_path, caller, runner) + log.debug("\n{d}\n{t}\n{d}\n".format(d="-" * 80, t=rendered_template)) + data.update(yaml.load(rendered_template)) + except Exception as ex: + log.debug(ex, exc_info=True) + raise salt.exceptions.SaltException( + "Rendering profile failed: {}".format(ex) + ) + else: + raise salt.exceptions.SaltException( + 'Profile "{}" is not found.'.format(profile) + ) + + return data + + +def get_profiles(config): + """ + Get available profiles. + + :return: + """ + profiles = [] + for profile_name in os.listdir(os.path.join(os.path.dirname(__file__), "profiles")): + if profile_name.endswith(".yml"): + profiles.append(profile_name.split(".")[0]) + + return sorted(profiles) diff --git a/salt/cli/support/collector.py b/salt/cli/support/collector.py new file mode 100644 index 0000000000..1879cc5220 --- /dev/null +++ b/salt/cli/support/collector.py @@ -0,0 +1,563 @@ +import builtins as exceptions +import copy +import json +import logging +import os +import sys +import tarfile +import time +from io import BytesIO +from io import IOBase as file + +import salt.cli.caller +import salt.cli.support +import salt.cli.support.console +import salt.cli.support.intfunc +import salt.cli.support.localrunner +import salt.defaults.exitcodes +import salt.exceptions +import salt.ext.six as six +import salt.output.table_out +import salt.runner +import salt.utils.files +import salt.utils.parsers +import salt.utils.platform +import salt.utils.process +import salt.utils.stringutils +import salt.utils.verify +import yaml + +salt.output.table_out.__opts__ = {} +log = logging.getLogger(__name__) + + +class SupportDataCollector: + """ + Data collector. It behaves just like another outputter, + except it grabs the data to the archive files. + """ + + def __init__(self, name, output): + """ + constructor of the data collector + :param name: + :param path: + :param format: + """ + self.archive_path = name + self.__default_outputter = output + self.__format = format + self.__arch = None + self.__current_section = None + self.__current_section_name = None + self.__default_root = time.strftime("%Y.%m.%d-%H.%M.%S-snapshot") + self.out = salt.cli.support.console.MessagesOutput() + + def open(self): + """ + Opens archive. + :return: + """ + if self.__arch is not None: + raise salt.exceptions.SaltException("Archive already opened.") + self.__arch = tarfile.TarFile.bz2open(self.archive_path, "w") + + def close(self): + """ + Closes the archive. + :return: + """ + if self.__arch is None: + raise salt.exceptions.SaltException("Archive already closed") + self._flush_content() + self.__arch.close() + self.__arch = None + + def _flush_content(self): + """ + Flush content to the archive + :return: + """ + if self.__current_section is not None: + buff = BytesIO() + buff._dirty = False + for action_return in self.__current_section: + for title, ret_data in action_return.items(): + if isinstance(ret_data, file): + self.out.put(ret_data.name, indent=4) + self.__arch.add(ret_data.name, arcname=ret_data.name) + else: + buff.write(salt.utils.stringutils.to_bytes(title + "\n")) + buff.write( + salt.utils.stringutils.to_bytes(("-" * len(title)) + "\n\n") + ) + buff.write(salt.utils.stringutils.to_bytes(ret_data)) + buff.write(salt.utils.stringutils.to_bytes("\n\n\n")) + buff._dirty = True + if buff._dirty: + buff.seek(0) + tar_info = tarfile.TarInfo( + name="{}/{}".format( + self.__default_root, self.__current_section_name + ) + ) + if not hasattr(buff, "getbuffer"): # Py2's BytesIO is older + buff.getbuffer = buff.getvalue + tar_info.size = len(buff.getbuffer()) + self.__arch.addfile(tarinfo=tar_info, fileobj=buff) + + def add(self, name): + """ + Start a new section. + :param name: + :return: + """ + if self.__current_section: + self._flush_content() + self.discard_current(name) + + def discard_current(self, name=None): + """ + Discard current section + :return: + """ + self.__current_section = [] + self.__current_section_name = name + + def _printout(self, data, output): + """ + Use salt outputter to printout content. + + :return: + """ + opts = {"extension_modules": "", "color": False} + try: + printout = salt.output.get_printout(output, opts)(data) + if printout is not None: + return printout.rstrip() + except (KeyError, AttributeError, TypeError) as err: + log.debug(err, exc_info=True) + try: + printout = salt.output.get_printout("nested", opts)(data) + if printout is not None: + return printout.rstrip() + except (KeyError, AttributeError, TypeError) as err: + log.debug(err, exc_info=True) + printout = salt.output.get_printout("raw", opts)(data) + if printout is not None: + return printout.rstrip() + + return salt.output.try_printout(data, output, opts) + + def write(self, title, data, output=None): + """ + Add a data to the current opened section. + :return: + """ + if not isinstance(data, (dict, list, tuple)): + data = {"raw-content": str(data)} + output = output or self.__default_outputter + + if output != "null": + try: + if isinstance(data, dict) and "return" in data: + data = data["return"] + content = self._printout(data, output) + except Exception: # Fall-back to just raw YAML + content = None + else: + content = None + + if content is None: + data = json.loads(json.dumps(data)) + if isinstance(data, dict) and data.get("return"): + data = data.get("return") + content = yaml.safe_dump(data, default_flow_style=False, indent=4) + + self.__current_section.append({title: content}) + + def link(self, title, path): + """ + Add a static file on the file system. + + :param title: + :param path: + :return: + """ + # The filehandler needs to be explicitly passed here, so PyLint needs to accept that. + # pylint: disable=W8470 + if not isinstance(path, file): + path = salt.utils.files.fopen(path) + self.__current_section.append({title: path}) + # pylint: enable=W8470 + + +class SaltSupport(salt.utils.parsers.SaltSupportOptionParser): + """ + Class to run Salt Support subsystem. + """ + + RUNNER_TYPE = "run" + CALL_TYPE = "call" + + def _setup_fun_config(self, fun_conf): + """ + Setup function configuration. + + :param conf: + :return: + """ + conf = copy.deepcopy(self.config) + conf["file_client"] = "local" + conf["fun"] = "" + conf["arg"] = [] + conf["kwarg"] = {} + conf["cache_jobs"] = False + conf["print_metadata"] = False + conf.update(fun_conf) + conf["fun"] = conf["fun"].split(":")[-1] # Discard typing prefix + + return conf + + def _get_runner(self, conf): + """ + Get & setup runner. + + :param conf: + :return: + """ + conf = self._setup_fun_config(copy.deepcopy(conf)) + if not getattr(self, "_runner", None): + self._runner = salt.cli.support.localrunner.LocalRunner(conf) + else: + self._runner.opts = conf + return self._runner + + def _get_caller(self, conf): + """ + Get & setup caller from the factory. + + :param conf: + :return: + """ + conf = self._setup_fun_config(copy.deepcopy(conf)) + if not getattr(self, "_caller", None): + self._caller = salt.cli.caller.Caller.factory(conf) + else: + self._caller.opts = conf + return self._caller + + def _local_call(self, call_conf): + """ + Execute local call + """ + try: + ret = self._get_caller(call_conf).call() + except SystemExit: + ret = "Data is not available at this moment" + self.out.error(ret) + except Exception as ex: + ret = "Unhandled exception occurred: {}".format(ex) + log.debug(ex, exc_info=True) + self.out.error(ret) + + return ret + + def _local_run(self, run_conf): + """ + Execute local runner + + :param run_conf: + :return: + """ + try: + ret = self._get_runner(run_conf).run() + except SystemExit: + ret = "Runner is not available at this moment" + self.out.error(ret) + except Exception as ex: + ret = "Unhandled exception occurred: {}".format(ex) + log.debug(ex, exc_info=True) + + return ret + + def _internal_function_call(self, call_conf): + """ + Call internal function. + + :param call_conf: + :return: + """ + + def stub(*args, **kwargs): + message = "Function {} is not available".format(call_conf["fun"]) + self.out.error(message) + log.debug( + 'Attempt to run "{fun}" with {arg} arguments and {kwargs} parameters.'.format( + **call_conf + ) + ) + return message + + return getattr(salt.cli.support.intfunc, call_conf["fun"], stub)( + self.collector, *call_conf["arg"], **call_conf["kwargs"] + ) + + def _get_action(self, action_meta): + """ + Parse action and turn into a calling point. + :param action_meta: + :return: + """ + conf = { + "fun": list(action_meta.keys())[0], + "arg": [], + "kwargs": {}, + } + if not len(conf["fun"].split(".")) - 1: + conf["salt.int.intfunc"] = True + + action_meta = action_meta[conf["fun"]] + info = action_meta.get("info", "Action for {}".format(conf["fun"])) + for arg in action_meta.get("args") or []: + if not isinstance(arg, dict): + conf["arg"].append(arg) + else: + conf["kwargs"].update(arg) + + return info, action_meta.get("output"), conf + + def collect_internal_data(self): + """ + Dumps current running pillars, configuration etc. + :return: + """ + section = "configuration" + self.out.put(section) + self.collector.add(section) + self.out.put("Saving config", indent=2) + self.collector.write("General Configuration", self.config) + self.out.put("Saving pillars", indent=2) + self.collector.write( + "Active Pillars", self._local_call({"fun": "pillar.items"}) + ) + + section = "highstate" + self.out.put(section) + self.collector.add(section) + self.out.put("Saving highstate", indent=2) + self.collector.write( + "Rendered highstate", self._local_call({"fun": "state.show_highstate"}) + ) + + def _extract_return(self, data): + """ + Extracts return data from the results. + + :param data: + :return: + """ + if isinstance(data, dict): + data = data.get("return", data) + + return data + + def collect_local_data(self, profile=None, profile_source=None): + """ + Collects master system data. + :return: + """ + + def call(func, *args, **kwargs): + """ + Call wrapper for templates + :param func: + :return: + """ + return self._extract_return( + self._local_call({"fun": func, "arg": args, "kwarg": kwargs}) + ) + + def run(func, *args, **kwargs): + """ + Runner wrapper for templates + :param func: + :return: + """ + return self._extract_return( + self._local_run({"fun": func, "arg": args, "kwarg": kwargs}) + ) + + scenario = profile_source or salt.cli.support.get_profile( + profile or self.config["support_profile"], call, run + ) + for category_name in scenario: + self.out.put(category_name) + self.collector.add(category_name) + for action in scenario[category_name]: + if not action: + continue + action_name = next(iter(action)) + if not isinstance(action[action_name], str): + info, output, conf = self._get_action(action) + action_type = self._get_action_type( + action + ) # run:<something> for runners + if action_type == self.RUNNER_TYPE: + self.out.put("Running {}".format(info.lower()), indent=2) + self.collector.write(info, self._local_run(conf), output=output) + elif action_type == self.CALL_TYPE: + if not conf.get("salt.int.intfunc"): + self.out.put("Collecting {}".format(info.lower()), indent=2) + self.collector.write( + info, self._local_call(conf), output=output + ) + else: + self.collector.discard_current() + self._internal_function_call(conf) + else: + self.out.error( + 'Unknown action type "{}" for action: {}'.format( + action_type, action + ) + ) + else: + # TODO: This needs to be moved then to the utils. + # But the code is not yet there (other PRs) + self.out.msg( + "\n".join(salt.cli.support.console.wrap(action[action_name])), + ident=2, + ) + + def _get_action_type(self, action): + """ + Get action type. + :param action: + :return: + """ + action_name = next(iter(action or {"": None})) + if ":" not in action_name: + action_name = "{}:{}".format(self.CALL_TYPE, action_name) + + return action_name.split(":")[0] or None + + def _cleanup(self): + """ + Cleanup if crash/exception + :return: + """ + if ( + hasattr(self, "config") + and self.config.get("support_archive") + and os.path.exists(self.config["support_archive"]) + ): + self.out.warning("Terminated earlier, cleaning up") + try: + os.unlink(self.config["support_archive"]) + except Exception as err: + log.debug(err) + self.out.error("{} while cleaning up.".format(err)) + + def _check_existing_archive(self): + """ + Check if archive exists or not. If exists and --force was not specified, + bail out. Otherwise remove it and move on. + + :return: + """ + if os.path.exists(self.config["support_archive"]): + if self.config["support_archive_force_overwrite"]: + self.out.warning( + "Overwriting existing archive: {}".format( + self.config["support_archive"] + ) + ) + try: + os.unlink(self.config["support_archive"]) + except Exception as err: + log.debug(err) + self.out.error( + "{} while trying to overwrite existing archive.".format(err) + ) + ret = True + else: + self.out.warning( + "File {} already exists.".format(self.config["support_archive"]) + ) + ret = False + else: + ret = True + + return ret + + def run(self): + exit_code = salt.defaults.exitcodes.EX_OK + self.out = salt.cli.support.console.MessagesOutput() + try: + self.parse_args() + except (Exception, SystemExit) as ex: + if not isinstance(ex, exceptions.SystemExit): + exit_code = salt.defaults.exitcodes.EX_GENERIC + self.out.error(ex) + elif isinstance(ex, exceptions.SystemExit): + exit_code = ex.code + else: + exit_code = salt.defaults.exitcodes.EX_GENERIC + self.out.error(ex) + else: + if self.config["log_level"] not in ("quiet",): + self.setup_logfile_logger() + salt.utils.verify.verify_log(self.config) + salt.cli.support.log = log # Pass update logger so trace is available + + if self.config["support_profile_list"]: + self.out.put("List of available profiles:") + for idx, profile in enumerate( + salt.cli.support.get_profiles(self.config) + ): + msg_template = " {}. ".format(idx + 1) + "{}" + self.out.highlight(msg_template, profile) + exit_code = salt.defaults.exitcodes.EX_OK + elif self.config["support_show_units"]: + self.out.put("List of available units:") + for idx, unit in enumerate(self.find_existing_configs(None)): + msg_template = " {}. ".format(idx + 1) + "{}" + self.out.highlight(msg_template, unit) + exit_code = salt.defaults.exitcodes.EX_OK + else: + if not self.config["support_profile"]: + self.print_help() + raise SystemExit() + + if self._check_existing_archive(): + try: + self.collector = SupportDataCollector( + self.config["support_archive"], + output=self.config["support_output_format"], + ) + except Exception as ex: + self.out.error(ex) + exit_code = salt.defaults.exitcodes.EX_GENERIC + log.debug(ex, exc_info=True) + else: + try: + self.collector.open() + self.collect_local_data() + self.collect_internal_data() + self.collector.close() + + archive_path = self.collector.archive_path + self.out.highlight( + '\nSupport data has been written to "{}" file.\n', + archive_path, + _main="YELLOW", + ) + except Exception as ex: + self.out.error(ex) + log.debug(ex, exc_info=True) + exit_code = salt.defaults.exitcodes.EX_SOFTWARE + + if exit_code: + self._cleanup() + + sys.exit(exit_code) diff --git a/salt/cli/support/console.py b/salt/cli/support/console.py new file mode 100644 index 0000000000..266b645479 --- /dev/null +++ b/salt/cli/support/console.py @@ -0,0 +1,184 @@ +""" +Collection of tools to report messages to console. + +NOTE: This is subject to incorporate other formatting bits + from all around everywhere and then to be moved to utils. +""" + + +import os +import sys +import textwrap + +import salt.utils.color + + +class IndentOutput: + """ + Paint different indends in different output. + """ + + def __init__(self, conf=None, device=sys.stdout): + if conf is None: + conf = {0: "CYAN", 2: "GREEN", 4: "LIGHT_BLUE", 6: "BLUE"} + self._colors_conf = conf + self._device = device + self._colors = salt.utils.color.get_colors() + self._default_color = "GREEN" + self._default_hl_color = "LIGHT_GREEN" + + def put(self, message, indent=0): + """ + Print message with an indent. + + :param message: + :param indent: + :return: + """ + color = self._colors_conf.get( + indent + indent % 2, self._colors_conf.get(0, self._default_color) + ) + + for chunk in [" " * indent, self._colors[color], message, self._colors["ENDC"]]: + self._device.write(str(chunk)) + self._device.write(os.linesep) + self._device.flush() + + +class MessagesOutput(IndentOutput): + """ + Messages output to the CLI. + """ + + def msg(self, message, title=None, title_color=None, color="BLUE", ident=0): + """ + Hint message. + + :param message: + :param title: + :param title_color: + :param color: + :param ident: + :return: + """ + if title and not title_color: + title_color = color + if title_color and not title: + title_color = None + + self.__colored_output(title, message, title_color, color, ident=ident) + + def info(self, message, ident=0): + """ + Write an info message to the CLI. + + :param message: + :param ident: + :return: + """ + self.__colored_output("Info", message, "GREEN", "LIGHT_GREEN", ident=ident) + + def warning(self, message, ident=0): + """ + Write a warning message to the CLI. + + :param message: + :param ident: + :return: + """ + self.__colored_output("Warning", message, "YELLOW", "LIGHT_YELLOW", ident=ident) + + def error(self, message, ident=0): + """ + Write an error message to the CLI. + + :param message: + :param ident + :return: + """ + self.__colored_output("Error", message, "RED", "LIGHT_RED", ident=ident) + + def __colored_output(self, title, message, title_color, message_color, ident=0): + if title and not title.endswith(":"): + _linesep = title.endswith(os.linesep) + title = "{}:{}".format(title.strip(), _linesep and os.linesep or " ") + + for chunk in [ + title_color and self._colors[title_color] or None, + " " * ident, + title, + self._colors[message_color], + message, + self._colors["ENDC"], + ]: + if chunk: + self._device.write(str(chunk)) + self._device.write(os.linesep) + self._device.flush() + + def highlight(self, message, *values, **colors): + """ + Highlighter works the way that message parameter is a template, + the "values" is a list of arguments going one after another as values there. + And so the "colors" should designate either highlight color or alternate for each. + + Example: + + highlight('Hello {}, there! It is {}.', 'user', 'daytime', _main='GREEN', _highlight='RED') + highlight('Hello {}, there! It is {}.', 'user', 'daytime', _main='GREEN', _highlight='RED', 'daytime'='YELLOW') + + First example will highlight all the values in the template with the red color. + Second example will highlight the second value with the yellow color. + + Usage: + + colors: + _main: Sets the main color (or default is used) + _highlight: Sets the alternative color for everything + 'any phrase' that is the same in the "values" can override color. + + :param message: + :param formatted: + :param colors: + :return: + """ + + m_color = colors.get("_main", self._default_color) + h_color = colors.get("_highlight", self._default_hl_color) + + _values = [] + for value in values: + _values.append( + "{p}{c}{r}".format( + p=self._colors[colors.get(value, h_color)], + c=value, + r=self._colors[m_color], + ) + ) + self._device.write( + "{s}{m}{e}".format( + s=self._colors[m_color], + m=message.format(*_values), + e=self._colors["ENDC"], + ) + ) + self._device.write(os.linesep) + self._device.flush() + + +def wrap(txt, width=80, ident=0): + """ + Wrap text to the required dimensions and clean it up, prepare for display. + + :param txt: + :param width: + :return: + """ + ident = " " * ident + txt = (txt or "").replace(os.linesep, " ").strip() + + wrapper = textwrap.TextWrapper() + wrapper.fix_sentence_endings = False + wrapper.initial_indent = wrapper.subsequent_indent = ident + + return wrapper.wrap(txt) diff --git a/salt/cli/support/intfunc.py b/salt/cli/support/intfunc.py new file mode 100644 index 0000000000..a9f76a6003 --- /dev/null +++ b/salt/cli/support/intfunc.py @@ -0,0 +1,51 @@ +""" +Internal functions. +""" +# Maybe this needs to be a modules in a future? + +import glob +import os + +import salt.utils.files +from salt.cli.support.console import MessagesOutput + +out = MessagesOutput() + + +def filetree(collector, *paths): + """ + Add all files in the tree. If the "path" is a file, + only that file will be added. + + :param path: File or directory + :return: + """ + _paths = [] + # Unglob + for path in paths: + _paths += glob.glob(path) + for path in set(_paths): + if not path: + out.error("Path not defined", ident=2) + elif not os.path.exists(path): + out.warning("Path {} does not exists".format(path)) + else: + # The filehandler needs to be explicitly passed here, so PyLint needs to accept that. + # pylint: disable=W8470 + if os.path.isfile(path): + filename = os.path.basename(path) + try: + file_ref = salt.utils.files.fopen(path) # pylint: disable=W + out.put("Add {}".format(filename), indent=2) + collector.add(filename) + collector.link(title=path, path=file_ref) + except Exception as err: + out.error(err, ident=4) + # pylint: enable=W8470 + else: + try: + for fname in os.listdir(path): + fname = os.path.join(path, fname) + filetree(collector, [fname]) + except Exception as err: + out.error(err, ident=4) diff --git a/salt/cli/support/localrunner.py b/salt/cli/support/localrunner.py new file mode 100644 index 0000000000..ad10eda0b0 --- /dev/null +++ b/salt/cli/support/localrunner.py @@ -0,0 +1,33 @@ +""" +Local Runner +""" + +import logging + +import salt.runner +import salt.utils.platform +import salt.utils.process + +log = logging.getLogger(__name__) + + +class LocalRunner(salt.runner.Runner): + """ + Runner class that changes its default behaviour. + """ + + def _proc_function(self, fun, low, user, tag, jid, daemonize=True): + """ + Same as original _proc_function in AsyncClientMixin, + except it calls "low" without firing a print event. + """ + if daemonize and not salt.utils.platform.is_windows(): + salt.log.setup.shutdown_multiprocessing_logging() + salt.utils.process.daemonize() + salt.log.setup.setup_multiprocessing_logging() + + low["__jid__"] = jid + low["__user__"] = user + low["__tag__"] = tag + + return self.low(fun, low, print_event=False, full_return=False) diff --git a/salt/cli/support/profiles/__init__.py b/salt/cli/support/profiles/__init__.py new file mode 100644 index 0000000000..b86aef30b8 --- /dev/null +++ b/salt/cli/support/profiles/__init__.py @@ -0,0 +1,4 @@ +# coding=utf-8 +''' +Profiles for salt-support. +''' diff --git a/salt/cli/support/profiles/default.yml b/salt/cli/support/profiles/default.yml new file mode 100644 index 0000000000..3defb5eef3 --- /dev/null +++ b/salt/cli/support/profiles/default.yml @@ -0,0 +1,78 @@ +sysinfo: + - description: | + Get the Salt grains of the current system. + - grains.items: + info: System grains + +packages: + - description: | + Fetch list of all the installed packages. + - pkg.list_pkgs: + info: Installed packages + +repositories: + - pkg.list_repos: + info: Available repositories + +upgrades: + - pkg.list_upgrades: + info: Possible upgrades + +## TODO: Some data here belongs elsewhere and also is duplicated +status: + - status.version: + info: Status version + - status.cpuinfo: + info: CPU information + - status.cpustats: + info: CPU stats + - status.diskstats: + info: Disk stats + - status.loadavg: + info: Average load of the current system + - status.uptime: + info: Uptime of the machine + - status.meminfo: + info: Information about memory + - status.vmstats: + info: Virtual memory stats + - status.netdev: + info: Network device stats + - status.nproc: + info: Number of processing units available on this system + - status.procs: + info: Process data + +general-health: + - ps.boot_time: + info: System Boot Time + - ps.swap_memory: + info: Swap Memory + output: txt + - ps.cpu_times: + info: CPU times + - ps.disk_io_counters: + info: Disk IO counters + - ps.disk_partition_usage: + info: Disk partition usage + output: table + - ps.disk_partitions: + info: Disk partitions + output: table + - ps.top: + info: Top CPU consuming processes + +boot_log: + - filetree: + info: Collect boot logs + args: + - /var/log/boot.* + +system.log: + # This works on any file system object. + - filetree: + info: Add system log + args: + - /var/log/syslog + - /var/log/messages + diff --git a/salt/cli/support/profiles/jobs-active.yml b/salt/cli/support/profiles/jobs-active.yml new file mode 100644 index 0000000000..508c54ece7 --- /dev/null +++ b/salt/cli/support/profiles/jobs-active.yml @@ -0,0 +1,3 @@ +jobs-active: + - run:jobs.active: + info: List of all actively running jobs diff --git a/salt/cli/support/profiles/jobs-last.yml b/salt/cli/support/profiles/jobs-last.yml new file mode 100644 index 0000000000..e3b719f552 --- /dev/null +++ b/salt/cli/support/profiles/jobs-last.yml @@ -0,0 +1,3 @@ +jobs-last: + - run:jobs.last_run: + info: List all detectable jobs and associated functions diff --git a/salt/cli/support/profiles/jobs-trace.yml b/salt/cli/support/profiles/jobs-trace.yml new file mode 100644 index 0000000000..00b28e0502 --- /dev/null +++ b/salt/cli/support/profiles/jobs-trace.yml @@ -0,0 +1,7 @@ +jobs-details: + {% for job in runners('jobs.list_jobs') %} + - run:jobs.list_job: + info: Details on JID {{job}} + args: + - {{job}} + {% endfor %} diff --git a/salt/cli/support/profiles/network.yml b/salt/cli/support/profiles/network.yml new file mode 100644 index 0000000000..268f02e61f --- /dev/null +++ b/salt/cli/support/profiles/network.yml @@ -0,0 +1,27 @@ +network: + - network.get_hostname: + info: Hostname + output: txt + - network.get_fqdn: + info: FQDN + output: txt + - network.default_route: + info: Default route + output: table + - network.interfaces: + info: All the available interfaces + output: table + - network.subnets: + info: List of IPv4 subnets + - network.subnets6: + info: List of IPv6 subnets + - network.routes: + info: Network configured routes from routing tables + output: table + - network.netstat: + info: Information on open ports and states + output: table + - network.active_tcp: + info: All running TCP connections + - network.arp: + info: ARP table diff --git a/salt/cli/support/profiles/postgres.yml b/salt/cli/support/profiles/postgres.yml new file mode 100644 index 0000000000..2238752c7a --- /dev/null +++ b/salt/cli/support/profiles/postgres.yml @@ -0,0 +1,11 @@ +system.log: + - filetree: + info: Add system log + args: + - /var/log/syslog + +etc/postgres: + - filetree: + info: Pick entire /etc/postgresql + args: + - /etc/postgresql diff --git a/salt/cli/support/profiles/salt.yml b/salt/cli/support/profiles/salt.yml new file mode 100644 index 0000000000..4b18d98870 --- /dev/null +++ b/salt/cli/support/profiles/salt.yml @@ -0,0 +1,9 @@ +sysinfo: + - grains.items: + info: System grains + +logfile: + - filetree: + info: Add current logfile + args: + - {{salt('config.get', 'log_file')}} diff --git a/salt/cli/support/profiles/users.yml b/salt/cli/support/profiles/users.yml new file mode 100644 index 0000000000..391acdb606 --- /dev/null +++ b/salt/cli/support/profiles/users.yml @@ -0,0 +1,22 @@ +all-users: + {%for uname in salt('user.list_users') %} + - user.info: + info: Information about "{{uname}}" + args: + - {{uname}} + - user.list_groups: + info: List groups for user "{{uname}}" + args: + - {{uname}} + - shadow.info: + info: Shadow information about user "{{uname}}" + args: + - {{uname}} + - cron.raw_cron: + info: Cron for user "{{uname}}" + args: + - {{uname}} + {%endfor%} + - group.getent: + info: List of all available groups + output: table diff --git a/salt/loader/lazy.py b/salt/loader/lazy.py index d319fe54b4..5de995d446 100644 --- a/salt/loader/lazy.py +++ b/salt/loader/lazy.py @@ -972,8 +972,10 @@ class LazyLoader(salt.utils.lazy.LazyDict): mod_names = [module_name] + list(virtual_aliases) for attr in funcs_to_load: - if attr.startswith("_"): - # private functions are skipped + if attr.startswith("_") and attr != "__call__": + # private functions are skipped, + # except __call__ which is default entrance + # for multi-function batch-like state syntax continue func = getattr(mod, attr) if not inspect.isfunction(func) and not isinstance(func, functools.partial): diff --git a/salt/modules/saltsupport.py b/salt/modules/saltsupport.py new file mode 100644 index 0000000000..e800e3bf1f --- /dev/null +++ b/salt/modules/saltsupport.py @@ -0,0 +1,405 @@ +# +# Author: Bo Maryniuk <bo@suse.de> +# +# Copyright 2018 SUSE LLC +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +:codeauthor: :email:`Bo Maryniuk <bo@suse.de>` + +Module to run salt-support within Salt. +""" +# pylint: disable=W0231,W0221 + + +import datetime +import logging +import os +import re +import sys +import tempfile +import time + +import salt.cli.support +import salt.cli.support.intfunc +import salt.defaults.exitcodes +import salt.exceptions +import salt.utils.decorators +import salt.utils.dictupdate +import salt.utils.odict +import salt.utils.path +import salt.utils.stringutils +from salt.cli.support.collector import SaltSupport, SupportDataCollector + +__virtualname__ = "support" +log = logging.getLogger(__name__) + + +class LogCollector: + """ + Output collector. + """ + + INFO = "info" + WARNING = "warning" + ERROR = "error" + + class MessagesList(list): + def append(self, obj): + list.append( + self, + "{} - {}".format( + datetime.datetime.utcnow().strftime("%T.%f")[:-3], obj + ), + ) + + __call__ = append + + def __init__(self): + self.messages = { + self.INFO: self.MessagesList(), + self.WARNING: self.MessagesList(), + self.ERROR: self.MessagesList(), + } + + def msg(self, message, *args, **kwargs): + title = kwargs.get("title") + if title: + message = "{}: {}".format(title, message) + self.messages[self.INFO](message) + + def info(self, message, *args, **kwargs): + self.msg(message) + + def warning(self, message, *args, **kwargs): + self.messages[self.WARNING](message) + + def error(self, message, *args, **kwargs): + self.messages[self.ERROR](message) + + def put(self, message, *args, **kwargs): + self.messages[self.INFO](message) + + def highlight(self, message, *values, **kwargs): + self.msg(message.format(*values)) + + +class SaltSupportModule(SaltSupport): + """ + Salt Support module class. + """ + + def __init__(self): + """ + Constructor + """ + self.config = self.setup_config() + + def setup_config(self): + """ + Return current configuration + :return: + """ + return __opts__ + + def _get_archive_name(self, archname=None): + """ + Create default archive name. + + :return: + """ + archname = re.sub("[^a-z0-9]", "", (archname or "").lower()) or "support" + for grain in ["fqdn", "host", "localhost", "nodename"]: + host = __grains__.get(grain) + if host: + break + if not host: + host = "localhost" + + return os.path.join( + tempfile.gettempdir(), + "{hostname}-{archname}-{date}-{time}.bz2".format( + archname=archname, + hostname=host, + date=time.strftime("%Y%m%d"), + time=time.strftime("%H%M%S"), + ), + ) + + @salt.utils.decorators.external + def profiles(self): + """ + Get list of profiles. + + :return: + """ + return { + "standard": salt.cli.support.get_profiles(self.config), + "custom": [], + } + + @salt.utils.decorators.external + def archives(self): + """ + Get list of existing archives. + :return: + """ + arc_files = [] + tmpdir = tempfile.gettempdir() + for filename in os.listdir(tmpdir): + mtc = re.match(r"\w+-\w+-\d+-\d+\.bz2", filename) + if mtc and len(filename) == mtc.span()[-1]: + arc_files.append(os.path.join(tmpdir, filename)) + + return arc_files + + @salt.utils.decorators.external + def last_archive(self): + """ + Get the last available archive + :return: + """ + archives = {} + for archive in self.archives(): + archives[int(archive.split(".")[0].split("-")[-1])] = archive + + return archives and archives[max(archives)] or None + + @salt.utils.decorators.external + def delete_archives(self, *archives): + """ + Delete archives + :return: + """ + # Remove paths + _archives = [] + for archive in archives: + _archives.append(os.path.basename(archive)) + archives = _archives[:] + + ret = {"files": {}, "errors": {}} + for archive in self.archives(): + arc_dir = os.path.dirname(archive) + archive = os.path.basename(archive) + if archives and archive in archives or not archives: + archive = os.path.join(arc_dir, archive) + try: + os.unlink(archive) + ret["files"][archive] = "removed" + except Exception as err: + ret["errors"][archive] = str(err) + ret["files"][archive] = "left" + + return ret + + def format_sync_stats(self, cnt): + """ + Format stats of the sync output. + + :param cnt: + :return: + """ + stats = salt.utils.odict.OrderedDict() + if cnt.get("retcode") == salt.defaults.exitcodes.EX_OK: + for line in cnt.get("stdout", "").split(os.linesep): + line = line.split(": ") + if len(line) == 2: + stats[line[0].lower().replace(" ", "_")] = line[1] + cnt["transfer"] = stats + del cnt["stdout"] + + # Remove empty + empty_sections = [] + for section in cnt: + if not cnt[section] and section != "retcode": + empty_sections.append(section) + for section in empty_sections: + del cnt[section] + + return cnt + + @salt.utils.decorators.depends("rsync") + @salt.utils.decorators.external + def sync(self, group, name=None, host=None, location=None, move=False, all=False): + """ + Sync the latest archive to the host on given location. + + CLI Example: + + .. code-block:: bash + + salt '*' support.sync group=test + salt '*' support.sync group=test name=/tmp/myspecial-12345-67890.bz2 + salt '*' support.sync group=test name=/tmp/myspecial-12345-67890.bz2 host=allmystuff.lan + salt '*' support.sync group=test name=/tmp/myspecial-12345-67890.bz2 host=allmystuff.lan location=/opt/ + + :param group: name of the local directory to which sync is going to put the result files + :param name: name of the archive. Latest, if not specified. + :param host: name of the destination host for rsync. Default is master, if not specified. + :param location: local destination directory, default temporary if not specified + :param move: move archive file[s]. Default is False. + :param all: work with all available archives. Default is False (i.e. latest available) + + :return: + """ + tfh, tfn = tempfile.mkstemp() + processed_archives = [] + src_uri = uri = None + + last_arc = self.last_archive() + if name: + archives = [name] + elif all: + archives = self.archives() + elif last_arc: + archives = [last_arc] + else: + archives = [] + + for name in archives: + err = None + if not name: + err = "No support archive has been defined." + elif not os.path.exists(name): + err = 'Support archive "{}" was not found'.format(name) + if err is not None: + log.error(err) + raise salt.exceptions.SaltInvocationError(err) + + if not uri: + src_uri = os.path.dirname(name) + uri = "{host}:{loc}".format( + host=host or __opts__["master"], + loc=os.path.join(location or tempfile.gettempdir(), group), + ) + + os.write(tfh, salt.utils.stringutils.to_bytes(os.path.basename(name))) + os.write(tfh, salt.utils.stringutils.to_bytes(os.linesep)) + processed_archives.append(name) + log.debug("Syncing {filename} to {uri}".format(filename=name, uri=uri)) + os.close(tfh) + + if not processed_archives: + raise salt.exceptions.SaltInvocationError("No archives found to transfer.") + + ret = __salt__["rsync.rsync"]( + src=src_uri, + dst=uri, + additional_opts=["--stats", "--files-from={}".format(tfn)], + ) + ret["files"] = {} + for name in processed_archives: + if move: + salt.utils.dictupdate.update(ret, self.delete_archives(name)) + log.debug("Deleting {filename}".format(filename=name)) + ret["files"][name] = "moved" + else: + ret["files"][name] = "copied" + + try: + os.unlink(tfn) + except OSError as err: + log.error( + "Cannot remove temporary rsync file {fn}: {err}".format(fn=tfn, err=err) + ) + + return self.format_sync_stats(ret) + + @salt.utils.decorators.external + def run(self, profile="default", pillar=None, archive=None, output="nested"): + """ + Run Salt Support on the minion. + + profile + Set available profile name. Default is "default". + + pillar + Set available profile from the pillars. + + archive + Override archive name. Default is "support". This results to "hostname-support-YYYYMMDD-hhmmss.bz2". + + output + Change the default outputter. Default is "nested". + + CLI Example: + + .. code-block:: bash + + salt '*' support.run + salt '*' support.run profile=network + salt '*' support.run pillar=something_special + """ + + class outputswitch: + """ + Output switcher on context + """ + + def __init__(self, output_device): + self._tmp_out = output_device + self._orig_out = None + + def __enter__(self): + self._orig_out = salt.cli.support.intfunc.out + salt.cli.support.intfunc.out = self._tmp_out + + def __exit__(self, *args): + salt.cli.support.intfunc.out = self._orig_out + + self.out = LogCollector() + with outputswitch(self.out): + self.collector = SupportDataCollector( + archive or self._get_archive_name(archname=archive), output + ) + self.collector.out = self.out + self.collector.open() + self.collect_local_data( + profile=profile, profile_source=__pillar__.get(pillar) + ) + self.collect_internal_data() + self.collector.close() + + return {"archive": self.collector.archive_path, "messages": self.out.messages} + + +def __virtual__(): + """ + Set method references as module functions aliases + :return: + """ + support = SaltSupportModule() + + def _set_function(obj): + """ + Create a Salt function for the SaltSupport class. + """ + + def _cmd(*args, **kwargs): + """ + Call support method as a function from the Salt. + """ + _kwargs = {} + for kw in kwargs: + if not kw.startswith("__"): + _kwargs[kw] = kwargs[kw] + return obj(*args, **_kwargs) + + _cmd.__doc__ = obj.__doc__ + return _cmd + + for m_name in dir(support): + obj = getattr(support, m_name) + if getattr(obj, "external", False): + setattr(sys.modules[__name__], m_name, _set_function(obj)) + + return __virtualname__ diff --git a/salt/scripts.py b/salt/scripts.py index 07393373c9..16b032af2e 100644 --- a/salt/scripts.py +++ b/salt/scripts.py @@ -622,3 +622,18 @@ def salt_pip(): ] + _pip_args(sys.argv[1:], extras) ret = subprocess.run(command, shell=False, check=False, env=env) sys.exit(ret.returncode) + + +def salt_support(): + """ + Run Salt Support that collects system data, logs etc for debug and support purposes. + :return: + """ + + import salt.cli.support.collector + + if "" in sys.path: + sys.path.remove("") + client = salt.cli.support.collector.SaltSupport() + _install_signal_handlers(client) + client.run() diff --git a/salt/state.py b/salt/state.py index 868be2749e..8352a8defc 100644 --- a/salt/state.py +++ b/salt/state.py @@ -1671,7 +1671,9 @@ class State: names = [] if state.startswith("__"): continue - chunk = {"state": state, "name": name} + chunk = OrderedDict() + chunk["state"] = state + chunk["name"] = name if orchestration_jid is not None: chunk["__orchestration_jid__"] = orchestration_jid if "__sls__" in body: @@ -2382,9 +2384,16 @@ class State: else: self.format_slots(cdata) with salt.utils.files.set_umask(low.get("__umask__")): - ret = self.states[cdata["full"]]( - *cdata["args"], **cdata["kwargs"] - ) + if cdata["full"].split(".")[-1] == "__call__": + # __call__ requires OrderedDict to preserve state order + # kwargs are also invalid overall + ret = self.states[cdata["full"]]( + cdata["args"], module=None, state=cdata["kwargs"] + ) + else: + ret = self.states[cdata["full"]]( + *cdata["args"], **cdata["kwargs"] + ) self.states.inject_globals = {} if "check_cmd" in low: state_check_cmd = "{0[state]}.mod_run_check_cmd".format(low) @@ -3489,10 +3498,31 @@ class State: running.update(errors) return running + def inject_default_call(self, high): + """ + Sets .call function to a state, if not there. + + :param high: + :return: + """ + for chunk in high: + state = high[chunk] + for state_ref in state: + needs_default = True + for argset in state[state_ref]: + if isinstance(argset, str): + needs_default = False + break + if needs_default: + order = state[state_ref].pop(-1) + state[state_ref].append("__call__") + state[state_ref].append(order) + def call_high(self, high, orchestration_jid=None): """ Process a high data call and ensure the defined states. """ + self.inject_default_call(high) errors = [] # If there is extension data reconcile it high, ext_errors = self.reconcile_extend(high) diff --git a/salt/states/saltsupport.py b/salt/states/saltsupport.py new file mode 100644 index 0000000000..fb0c9e0372 --- /dev/null +++ b/salt/states/saltsupport.py @@ -0,0 +1,225 @@ +# +# Author: Bo Maryniuk <bo@suse.de> +# +# Copyright 2018 SUSE LLC +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +r""" +:codeauthor: :email:`Bo Maryniuk <bo@suse.de>` + +Execution of Salt Support from within states +============================================ + +State to collect support data from the systems: + +.. code-block:: yaml + + examine_my_systems: + support.taken: + - profile: default + + support.collected: + - group: somewhere + - move: true + +""" +import logging +import os +import tempfile + +import salt.exceptions + +# Import salt modules +import salt.fileclient +import salt.utils.decorators.path +import salt.utils.odict + +log = logging.getLogger(__name__) +__virtualname__ = "support" + + +class SaltSupportState: + """ + Salt-support. + """ + + EXPORTED = ["collected", "taken"] + + def get_kwargs(self, data): + kwargs = {} + for keyset in data: + kwargs.update(keyset) + + return kwargs + + def __call__(self, state): + """ + Call support. + + :param args: + :param kwargs: + :return: + """ + ret = { + "name": state.pop("name"), + "changes": {}, + "result": True, + "comment": "", + } + + out = {} + functions = ["Functions:"] + try: + for ref_func, ref_kwargs in state.items(): + if ref_func not in self.EXPORTED: + raise salt.exceptions.SaltInvocationError( + "Function {} is not found".format(ref_func) + ) + out[ref_func] = getattr(self, ref_func)(**self.get_kwargs(ref_kwargs)) + functions.append(" - {}".format(ref_func)) + ret["comment"] = "\n".join(functions) + except Exception as ex: + ret["comment"] = str(ex) + ret["result"] = False + ret["changes"] = out + + return ret + + def check_destination(self, location, group): + """ + Check destination for the archives. + :return: + """ + # Pre-create destination, since rsync will + # put one file named as group + try: + destination = os.path.join(location, group) + if os.path.exists(destination) and not os.path.isdir(destination): + raise salt.exceptions.SaltException( + 'Destination "{}" should be directory!'.format(destination) + ) + if not os.path.exists(destination): + os.makedirs(destination) + log.debug("Created destination directory for archives: %s", destination) + else: + log.debug( + "Archives destination directory %s already exists", destination + ) + except OSError as err: + log.error(err) + + def collected( + self, group, filename=None, host=None, location=None, move=True, all=True + ): + """ + Sync archives to a central place. + + :param name: + :param group: + :param filename: + :param host: + :param location: + :param move: + :param all: + :return: + """ + ret = { + "name": "support.collected", + "changes": {}, + "result": True, + "comment": "", + } + location = location or tempfile.gettempdir() + self.check_destination(location, group) + ret["changes"] = __salt__["support.sync"]( + group, name=filename, host=host, location=location, move=move, all=all + ) + + return ret + + def taken(self, profile="default", pillar=None, archive=None, output="nested"): + """ + Takes minion support config data. + + :param profile: + :param pillar: + :param archive: + :param output: + :return: + """ + ret = { + "name": "support.taken", + "changes": {}, + "result": True, + } + + result = __salt__["support.run"]( + profile=profile, pillar=pillar, archive=archive, output=output + ) + if result.get("archive"): + ret[ + "comment" + ] = "Information about this system has been saved to {} file.".format( + result["archive"] + ) + ret["changes"]["archive"] = result["archive"] + ret["changes"]["messages"] = {} + for key in ["info", "error", "warning"]: + if result.get("messages", {}).get(key): + ret["changes"]["messages"][key] = result["messages"][key] + else: + ret["comment"] = "" + + return ret + + +_support_state = SaltSupportState() + + +def __call__(*args, **kwargs): + """ + SLS single-ID syntax processing. + + module: + This module reference, equals to sys.modules[__name__] + + state: + Compiled state in preserved order. The function supposed to look + at first level array of functions. + + :param cdata: + :param kwargs: + :return: + """ + return _support_state(kwargs.get("state", {})) + + +def taken(name, profile="default", pillar=None, archive=None, output="nested"): + return _support_state.taken( + profile=profile, pillar=pillar, archive=archive, output=output + ) + + +def collected( + name, group, filename=None, host=None, location=None, move=True, all=True +): + return _support_state.collected( + group=group, filename=filename, host=host, location=location, move=move, all=all + ) + + +def __virtual__(): + """ + Salt Support state + """ + return __virtualname__ diff --git a/salt/utils/args.py b/salt/utils/args.py index 536aea3816..04a8a14054 100644 --- a/salt/utils/args.py +++ b/salt/utils/args.py @@ -15,6 +15,7 @@ import salt.utils.jid import salt.utils.versions import salt.utils.yaml from salt.exceptions import SaltInvocationError +from salt.utils.odict import OrderedDict log = logging.getLogger(__name__) @@ -399,7 +400,7 @@ def format_call( ret = initial_ret is not None and initial_ret or {} ret["args"] = [] - ret["kwargs"] = {} + ret["kwargs"] = OrderedDict() aspec = get_function_argspec(fun, is_class_method=is_class_method) diff --git a/salt/utils/decorators/__init__.py b/salt/utils/decorators/__init__.py index 1f62d5f3d6..1906cc2ecc 100644 --- a/salt/utils/decorators/__init__.py +++ b/salt/utils/decorators/__init__.py @@ -866,3 +866,27 @@ def ensure_unicode_args(function): return function(*args, **kwargs) return wrapped + + +def external(func): + """ + Mark function as external. + + :param func: + :return: + """ + + def f(*args, **kwargs): + """ + Stub. + + :param args: + :param kwargs: + :return: + """ + return func(*args, **kwargs) + + f.external = True + f.__doc__ = func.__doc__ + + return f diff --git a/salt/utils/parsers.py b/salt/utils/parsers.py index 911b2cbb04..dc125de7d7 100644 --- a/salt/utils/parsers.py +++ b/salt/utils/parsers.py @@ -17,6 +17,7 @@ import optparse import os import signal import sys +import tempfile import traceback import types from functools import partial @@ -31,6 +32,7 @@ import salt.utils.args import salt.utils.data import salt.utils.files import salt.utils.jid +import salt.utils.network import salt.utils.platform import salt.utils.process import salt.utils.stringutils @@ -2026,6 +2028,118 @@ class SyndicOptionParser( return opts +class SaltSupportOptionParser( + OptionParser, + ConfigDirMixIn, + MergeConfigMixIn, + LogLevelMixIn, + TimeoutMixIn, + metaclass=OptionParserMeta, +): + default_timeout = 5 + description = "Salt Support is a program to collect all support data: logs, system configuration etc." + usage = "%prog [options] '<target>' <function> [arguments]" + # ConfigDirMixIn config filename attribute + _config_filename_ = "master" + + # LogLevelMixIn attributes + _default_logging_level_ = config.DEFAULT_MASTER_OPTS["log_level"] + _default_logging_logfile_ = config.DEFAULT_MASTER_OPTS["log_file"] + + def _mixin_setup(self): + self.add_option( + "-P", + "--show-profiles", + default=False, + action="store_true", + dest="support_profile_list", + help="Show available profiles", + ) + self.add_option( + "-p", + "--profile", + default="", + dest="support_profile", + help='Specify support profile or comma-separated profiles, e.g.: "salt,network"', + ) + support_archive = "{t}/{h}-support.tar.bz2".format( + t=tempfile.gettempdir(), h=salt.utils.network.get_fqhostname() + ) + self.add_option( + "-a", + "--archive", + default=support_archive, + dest="support_archive", + help=( + "Specify name of the resulting support archive. " + 'Default is "{f}".'.format(f=support_archive) + ), + ) + self.add_option( + "-u", + "--unit", + default="", + dest="support_unit", + help='Specify examined unit (default "master").', + ) + self.add_option( + "-U", + "--show-units", + default=False, + action="store_true", + dest="support_show_units", + help="Show available units", + ) + self.add_option( + "-f", + "--force", + default=False, + action="store_true", + dest="support_archive_force_overwrite", + help="Force overwrite existing archive, if exists", + ) + self.add_option( + "-o", + "--out", + default="null", + dest="support_output_format", + help=( + "Set the default output using the specified outputter, " + 'unless profile does not overrides this. Default: "yaml".' + ), + ) + + def find_existing_configs(self, default): + """ + Find configuration files on the system. + :return: + """ + configs = [] + for cfg in [default, self._config_filename_, "minion", "proxy", "cloud", "spm"]: + if not cfg: + continue + config_path = self.get_config_file_path(cfg) + if os.path.exists(config_path): + configs.append(cfg) + + if default and default not in configs: + raise SystemExit("Unknown configuration unit: {}".format(default)) + + return configs + + def setup_config(self, cfg=None): + """ + Open suitable config file. + :return: + """ + _opts, _args = optparse.OptionParser.parse_args(self) + configs = self.find_existing_configs(_opts.support_unit) + if configs and cfg not in configs: + cfg = configs[0] + + return config.master_config(self.get_config_file_path(cfg)) + + class SaltCMDOptionParser( OptionParser, ConfigDirMixIn, diff --git a/scripts/salt-support b/scripts/salt-support new file mode 100755 index 0000000000..4e0e79f3ea --- /dev/null +++ b/scripts/salt-support @@ -0,0 +1,11 @@ +#!/usr/bin/env python +""" +Salt support is to collect logs, +debug data and system information +for support purposes. +""" + +from salt.scripts import salt_support + +if __name__ == "__main__": + salt_support() diff --git a/setup.py b/setup.py index 931ed40a51..e60f1b7085 100755 --- a/setup.py +++ b/setup.py @@ -1061,6 +1061,7 @@ class SaltDistribution(distutils.dist.Distribution): "scripts/salt-minion", "scripts/salt-proxy", "scripts/salt-run", + "scripts/salt-support", "scripts/salt-ssh", "scripts/salt-syndic", "scripts/spm", @@ -1109,6 +1110,7 @@ class SaltDistribution(distutils.dist.Distribution): "salt-master = salt.scripts:salt_master", "salt-minion = salt.scripts:salt_minion", "salt-run = salt.scripts:salt_run", + "salt-support = salt.scripts:salt_support", "salt-ssh = salt.scripts:salt_ssh", "salt-syndic = salt.scripts:salt_syndic", "spm = salt.scripts:salt_spm", diff --git a/tests/pytests/unit/cli/test_support.py b/tests/pytests/unit/cli/test_support.py new file mode 100644 index 0000000000..dc0e99bb3d --- /dev/null +++ b/tests/pytests/unit/cli/test_support.py @@ -0,0 +1,553 @@ +""" + :codeauthor: Bo Maryniuk <bo@suse.de> +""" + + +import os + +import jinja2 +import salt.cli.support.collector +import salt.exceptions +import salt.utils.files +import yaml +from salt.cli.support.collector import SaltSupport, SupportDataCollector +from salt.cli.support.console import IndentOutput +from salt.utils.color import get_colors +from salt.utils.stringutils import to_bytes +from tests.support.mock import NO_MOCK, NO_MOCK_REASON, MagicMock, patch +from tests.support.unit import TestCase, skipIf + +try: + import pytest +except ImportError: + pytest = None + + +@skipIf(not bool(pytest), "Pytest needs to be installed") +@skipIf(NO_MOCK, NO_MOCK_REASON) +class SaltSupportIndentOutputTestCase(TestCase): + """ + Unit Tests for the salt-support indent output. + """ + + def setUp(self): + """ + Setup test + :return: + """ + + self.message = "Stubborn processes on dumb terminal" + self.device = MagicMock() + self.iout = IndentOutput(device=self.device) + self.colors = get_colors() + + def tearDown(self): + """ + Remove instances after test run + :return: + """ + del self.message + del self.device + del self.iout + del self.colors + + def test_standard_output(self): + """ + Test console standard output. + """ + self.iout.put(self.message) + assert self.device.write.called + assert self.device.write.call_count == 5 + for idx, data in enumerate( + ["", str(self.colors["CYAN"]), self.message, str(self.colors["ENDC"]), "\n"] + ): + assert self.device.write.call_args_list[idx][0][0] == data + + def test_indent_output(self): + """ + Test indent distance. + :return: + """ + self.iout.put(self.message, indent=10) + for idx, data in enumerate( + [ + " " * 10, + str(self.colors["CYAN"]), + self.message, + str(self.colors["ENDC"]), + "\n", + ] + ): + assert self.device.write.call_args_list[idx][0][0] == data + + def test_color_config(self): + """ + Test color config changes on each ident. + :return: + """ + + conf = {0: "MAGENTA", 2: "RED", 4: "WHITE", 6: "YELLOW"} + self.iout = IndentOutput(conf=conf, device=self.device) + for indent in sorted(list(conf)): + self.iout.put(self.message, indent=indent) + + step = 1 + for ident_key in sorted(list(conf)): + assert str(self.device.write.call_args_list[step][0][0]) == str( + self.colors[conf[ident_key]] + ) + step += 5 + + +@skipIf(not bool(pytest), "Pytest needs to be installed") +@skipIf(NO_MOCK, NO_MOCK_REASON) +class SaltSupportCollectorTestCase(TestCase): + """ + Collector tests. + """ + + def setUp(self): + """ + Setup the test case + :return: + """ + self.archive_path = "/highway/to/hell" + self.output_device = MagicMock() + self.collector = SupportDataCollector(self.archive_path, self.output_device) + + def tearDown(self): + """ + Tear down the test case elements + :return: + """ + del self.collector + del self.archive_path + del self.output_device + + @patch("salt.cli.support.collector.tarfile.TarFile", MagicMock()) + def test_archive_open(self): + """ + Test archive is opened. + + :return: + """ + self.collector.open() + assert self.collector.archive_path == self.archive_path + with pytest.raises(salt.exceptions.SaltException) as err: + self.collector.open() + assert "Archive already opened" in str(err) + + @patch("salt.cli.support.collector.tarfile.TarFile", MagicMock()) + def test_archive_close(self): + """ + Test archive is opened. + + :return: + """ + self.collector.open() + self.collector._flush_content = lambda: None + self.collector.close() + assert self.collector.archive_path == self.archive_path + with pytest.raises(salt.exceptions.SaltException) as err: + self.collector.close() + assert "Archive already closed" in str(err) + + def test_archive_addwrite(self): + """ + Test add to the archive a section and write to it. + + :return: + """ + archive = MagicMock() + with patch("salt.cli.support.collector.tarfile.TarFile", archive): + self.collector.open() + self.collector.add("foo") + self.collector.write(title="title", data="data", output="null") + self.collector._flush_content() + + assert archive.bz2open().addfile.call_args[1]["fileobj"].read() == to_bytes( + "title\n-----\n\nraw-content: data\n\n\n\n" + ) + + @patch("salt.utils.files.fopen", MagicMock(return_value="path=/dev/null")) + def test_archive_addlink(self): + """ + Test add to the archive a section and link an external file or directory to it. + + :return: + """ + archive = MagicMock() + with patch("salt.cli.support.collector.tarfile.TarFile", archive): + self.collector.open() + self.collector.add("foo") + self.collector.link(title="Backup Path", path="/path/to/backup.config") + self.collector._flush_content() + + assert archive.bz2open().addfile.call_count == 1 + assert archive.bz2open().addfile.call_args[1]["fileobj"].read() == to_bytes( + "Backup Path\n-----------\n\npath=/dev/null\n\n\n" + ) + + @patch("salt.utils.files.fopen", MagicMock(return_value="path=/dev/null")) + def test_archive_discard_section(self): + """ + Test discard a section from the archive. + + :return: + """ + archive = MagicMock() + with patch("salt.cli.support.collector.tarfile.TarFile", archive): + self.collector.open() + self.collector.add("solar-interference") + self.collector.link( + title="Thermal anomaly", path="/path/to/another/great.config" + ) + self.collector.add("foo") + self.collector.link(title="Backup Path", path="/path/to/backup.config") + self.collector._flush_content() + assert archive.bz2open().addfile.call_count == 2 + assert archive.bz2open().addfile.mock_calls[0][2][ + "fileobj" + ].read() == to_bytes( + "Thermal anomaly\n---------------\n\npath=/dev/null\n\n\n" + ) + self.collector.close() + + archive = MagicMock() + with patch("salt.cli.support.collector.tarfile.TarFile", archive): + self.collector.open() + self.collector.add("solar-interference") + self.collector.link( + title="Thermal anomaly", path="/path/to/another/great.config" + ) + self.collector.discard_current() + self.collector.add("foo") + self.collector.link(title="Backup Path", path="/path/to/backup.config") + self.collector._flush_content() + assert archive.bz2open().addfile.call_count == 2 + assert archive.bz2open().addfile.mock_calls[0][2][ + "fileobj" + ].read() == to_bytes("Backup Path\n-----------\n\npath=/dev/null\n\n\n") + self.collector.close() + + +@skipIf(not bool(pytest), "Pytest needs to be installed") +@skipIf(NO_MOCK, NO_MOCK_REASON) +class SaltSupportRunnerTestCase(TestCase): + """ + Test runner class. + """ + + def setUp(self): + """ + Set up test suite. + :return: + """ + self.archive_path = "/dev/null" + self.output_device = MagicMock() + self.runner = SaltSupport() + self.runner.collector = SupportDataCollector( + self.archive_path, self.output_device + ) + + def tearDown(self): + """ + Tear down. + + :return: + """ + del self.archive_path + del self.output_device + del self.runner + + def test_function_config(self): + """ + Test function config formation. + + :return: + """ + self.runner.config = {} + msg = "Electromagnetic energy loss" + assert self.runner._setup_fun_config({"description": msg}) == { + "print_metadata": False, + "file_client": "local", + "fun": "", + "kwarg": {}, + "description": msg, + "cache_jobs": False, + "arg": [], + } + + def test_local_caller(self): + """ + Test local caller. + + :return: + """ + msg = "Because of network lag due to too many people playing deathmatch" + caller = MagicMock() + caller().call = MagicMock(return_value=msg) + + self.runner._get_caller = caller + self.runner.out = MagicMock() + assert self.runner._local_call({}) == msg + + caller().call = MagicMock(side_effect=SystemExit) + assert self.runner._local_call({}) == "Data is not available at this moment" + + err_msg = "The UPS doesn't have a battery backup." + caller().call = MagicMock(side_effect=Exception(err_msg)) + assert ( + self.runner._local_call({}) + == "Unhandled exception occurred: The UPS doesn't have a battery backup." + ) + + def test_local_runner(self): + """ + Test local runner. + + :return: + """ + msg = "Big to little endian conversion error" + runner = MagicMock() + runner().run = MagicMock(return_value=msg) + + self.runner._get_runner = runner + self.runner.out = MagicMock() + assert self.runner._local_run({}) == msg + + runner().run = MagicMock(side_effect=SystemExit) + assert self.runner._local_run({}) == "Runner is not available at this moment" + + err_msg = "Trojan horse ran out of hay" + runner().run = MagicMock(side_effect=Exception(err_msg)) + assert ( + self.runner._local_run({}) + == "Unhandled exception occurred: Trojan horse ran out of hay" + ) + + @patch("salt.cli.support.intfunc", MagicMock(spec=[])) + def test_internal_function_call_stub(self): + """ + Test missing internal function call is handled accordingly. + + :return: + """ + self.runner.out = MagicMock() + out = self.runner._internal_function_call( + {"fun": "everythingisawesome", "arg": [], "kwargs": {}} + ) + assert out == "Function everythingisawesome is not available" + + def test_internal_function_call(self): + """ + Test missing internal function call is handled accordingly. + + :return: + """ + msg = "Internet outage" + intfunc = MagicMock() + intfunc.everythingisawesome = MagicMock(return_value=msg) + self.runner.out = MagicMock() + with patch("salt.cli.support.intfunc", intfunc): + out = self.runner._internal_function_call( + {"fun": "everythingisawesome", "arg": [], "kwargs": {}} + ) + assert out == msg + + def test_get_action(self): + """ + Test action meta gets parsed. + + :return: + """ + action_meta = { + "run:jobs.list_jobs_filter": {"info": "List jobs filter", "args": [1]} + } + assert self.runner._get_action(action_meta) == ( + "List jobs filter", + None, + {"fun": "run:jobs.list_jobs_filter", "kwargs": {}, "arg": [1]}, + ) + action_meta = { + "user.info": {"info": 'Information about "usbmux"', "args": ["usbmux"]} + } + assert self.runner._get_action(action_meta) == ( + 'Information about "usbmux"', + None, + {"fun": "user.info", "kwargs": {}, "arg": ["usbmux"]}, + ) + + def test_extract_return(self): + """ + Test extract return from the output. + + :return: + """ + out = {"key": "value"} + assert self.runner._extract_return(out) == out + assert self.runner._extract_return({"return": out}) == out + + def test_get_action_type(self): + """ + Test action meta determines action type. + + :return: + """ + action_meta = { + "run:jobs.list_jobs_filter": {"info": "List jobs filter", "args": [1]} + } + assert self.runner._get_action_type(action_meta) == "run" + + action_meta = { + "user.info": {"info": 'Information about "usbmux"', "args": ["usbmux"]} + } + assert self.runner._get_action_type(action_meta) == "call" + + @patch("os.path.exists", MagicMock(return_value=True)) + def test_cleanup(self): + """ + Test cleanup routine. + + :return: + """ + arch = "/tmp/killme.zip" + unlink = MagicMock() + with patch("os.unlink", unlink): + self.runner.config = {"support_archive": arch} + self.runner.out = MagicMock() + self.runner._cleanup() + + assert ( + self.runner.out.warning.call_args[0][0] + == "Terminated earlier, cleaning up" + ) + unlink.assert_called_once_with(arch) + + @patch("os.path.exists", MagicMock(return_value=True)) + def test_check_existing_archive(self): + """ + Test check existing archive. + + :return: + """ + arch = "/tmp/endothermal-recalibration.zip" + unlink = MagicMock() + with patch("os.unlink", unlink), patch( + "os.path.exists", MagicMock(return_value=False) + ): + self.runner.config = { + "support_archive": "", + "support_archive_force_overwrite": True, + } + self.runner.out = MagicMock() + assert self.runner._check_existing_archive() + assert self.runner.out.warning.call_count == 0 + + with patch("os.unlink", unlink): + self.runner.config = { + "support_archive": arch, + "support_archive_force_overwrite": False, + } + self.runner.out = MagicMock() + assert not self.runner._check_existing_archive() + assert self.runner.out.warning.call_args[0][ + 0 + ] == "File {} already exists.".format(arch) + + with patch("os.unlink", unlink): + self.runner.config = { + "support_archive": arch, + "support_archive_force_overwrite": True, + } + self.runner.out = MagicMock() + assert self.runner._check_existing_archive() + assert self.runner.out.warning.call_args[0][ + 0 + ] == "Overwriting existing archive: {}".format(arch) + + +@skipIf(not bool(pytest), "Pytest needs to be installed") +@skipIf(NO_MOCK, NO_MOCK_REASON) +class ProfileIntegrityTestCase(TestCase): + """ + Default profile integrity + """ + + def setUp(self): + """ + Set up test suite. + + :return: + """ + self.profiles = {} + profiles = os.path.join( + os.path.dirname(salt.cli.support.collector.__file__), "profiles" + ) + for profile in os.listdir(profiles): + self.profiles[profile.split(".")[0]] = os.path.join(profiles, profile) + + def tearDown(self): + """ + Tear down test suite. + + :return: + """ + del self.profiles + + def _render_template_to_yaml(self, name, *args, **kwargs): + """ + Get template referene for rendering. + :return: + """ + with salt.utils.files.fopen(self.profiles[name]) as t_fh: + template = t_fh.read() + return yaml.load( + jinja2.Environment().from_string(template).render(*args, **kwargs) + ) + + def test_non_template_profiles_parseable(self): + """ + Test shipped default profile is YAML parse-able. + + :return: + """ + for t_name in ["default", "jobs-active", "jobs-last", "network", "postgres"]: + with salt.utils.files.fopen(self.profiles[t_name]) as ref: + try: + yaml.load(ref) + parsed = True + except Exception: + parsed = False + assert parsed + + def test_users_template_profile(self): + """ + Test users template profile. + + :return: + """ + users_data = self._render_template_to_yaml( + "users", salt=MagicMock(return_value=["pokemon"]) + ) + assert len(users_data["all-users"]) == 5 + for user_data in users_data["all-users"]: + for tgt in ["user.list_groups", "shadow.info", "cron.raw_cron"]: + if tgt in user_data: + assert user_data[tgt]["args"] == ["pokemon"] + + def test_jobs_trace_template_profile(self): + """ + Test jobs-trace template profile. + + :return: + """ + jobs_trace = self._render_template_to_yaml( + "jobs-trace", runners=MagicMock(return_value=["0000"]) + ) + assert len(jobs_trace["jobs-details"]) == 1 + assert ( + jobs_trace["jobs-details"][0]["run:jobs.list_job"]["info"] + == "Details on JID 0000" + ) + assert jobs_trace["jobs-details"][0]["run:jobs.list_job"]["args"] == [0] diff --git a/tests/unit/modules/test_saltsupport.py b/tests/unit/modules/test_saltsupport.py new file mode 100644 index 0000000000..f9ce7be29a --- /dev/null +++ b/tests/unit/modules/test_saltsupport.py @@ -0,0 +1,496 @@ +""" + :codeauthor: Bo Maryniuk <bo@suse.de> +""" + + +import datetime + +import salt.exceptions +from salt.modules import saltsupport +from tests.support.mixins import LoaderModuleMockMixin +from tests.support.mock import NO_MOCK, NO_MOCK_REASON, MagicMock, patch +from tests.support.unit import TestCase, skipIf + +try: + import pytest +except ImportError: + pytest = None + + +@skipIf(not bool(pytest), "Pytest required") +@skipIf(NO_MOCK, NO_MOCK_REASON) +class SaltSupportModuleTestCase(TestCase, LoaderModuleMockMixin): + """ + Test cases for salt.modules.support::SaltSupportModule + """ + + def setup_loader_modules(self): + return {saltsupport: {}} + + @patch("tempfile.gettempdir", MagicMock(return_value="/mnt/storage")) + @patch("salt.modules.saltsupport.__grains__", {"fqdn": "c-3po"}) + @patch("time.strftime", MagicMock(return_value="000")) + def test_get_archive_name(self): + """ + Test archive name construction. + + :return: + """ + support = saltsupport.SaltSupportModule() + assert support._get_archive_name() == "/mnt/storage/c-3po-support-000-000.bz2" + + @patch("tempfile.gettempdir", MagicMock(return_value="/mnt/storage")) + @patch("salt.modules.saltsupport.__grains__", {"fqdn": "c-3po"}) + @patch("time.strftime", MagicMock(return_value="000")) + def test_get_custom_archive_name(self): + """ + Test get custom archive name. + + :return: + """ + support = saltsupport.SaltSupportModule() + temp_name = support._get_archive_name(archname="Darth Wader") + assert temp_name == "/mnt/storage/c-3po-darthwader-000-000.bz2" + temp_name = support._get_archive_name(archname="Яйця з сіллю") + assert temp_name == "/mnt/storage/c-3po-support-000-000.bz2" + temp_name = support._get_archive_name(archname="!@#$%^&*()Fillip J. Fry") + assert temp_name == "/mnt/storage/c-3po-fillipjfry-000-000.bz2" + + @patch( + "salt.cli.support.get_profiles", + MagicMock(return_value={"message": "Feature was not beta tested"}), + ) + def test_profiles_format(self): + """ + Test profiles format. + + :return: + """ + support = saltsupport.SaltSupportModule() + profiles = support.profiles() + assert "custom" in profiles + assert "standard" in profiles + assert "message" in profiles["standard"] + assert profiles["custom"] == [] + assert profiles["standard"]["message"] == "Feature was not beta tested" + + @patch("tempfile.gettempdir", MagicMock(return_value="/mnt/storage")) + @patch( + "os.listdir", + MagicMock( + return_value=[ + "one-support-000-000.bz2", + "two-support-111-111.bz2", + "trash.bz2", + "hostname-000-000.bz2", + "three-support-wrong222-222.bz2", + "000-support-000-000.bz2", + ] + ), + ) + def test_get_existing_archives(self): + """ + Get list of existing archives. + + :return: + """ + support = saltsupport.SaltSupportModule() + out = support.archives() + assert len(out) == 3 + for name in [ + "/mnt/storage/one-support-000-000.bz2", + "/mnt/storage/two-support-111-111.bz2", + "/mnt/storage/000-support-000-000.bz2", + ]: + assert name in out + + def test_last_archive(self): + """ + Get last archive name + :return: + """ + support = saltsupport.SaltSupportModule() + support.archives = MagicMock( + return_value=[ + "/mnt/storage/one-support-000-000.bz2", + "/mnt/storage/two-support-111-111.bz2", + "/mnt/storage/three-support-222-222.bz2", + ] + ) + assert support.last_archive() == "/mnt/storage/three-support-222-222.bz2" + + @patch("os.unlink", MagicMock(return_value=True)) + def test_delete_all_archives_success(self): + """ + Test delete archives + :return: + """ + support = saltsupport.SaltSupportModule() + support.archives = MagicMock( + return_value=[ + "/mnt/storage/one-support-000-000.bz2", + "/mnt/storage/two-support-111-111.bz2", + "/mnt/storage/three-support-222-222.bz2", + ] + ) + ret = support.delete_archives() + assert "files" in ret + assert "errors" in ret + assert not bool(ret["errors"]) + assert bool(ret["files"]) + assert isinstance(ret["errors"], dict) + assert isinstance(ret["files"], dict) + + for arc in support.archives(): + assert ret["files"][arc] == "removed" + + @patch( + "os.unlink", + MagicMock( + return_value=False, + side_effect=[ + OSError("Decreasing electron flux"), + OSError("Solar flares interference"), + None, + ], + ), + ) + def test_delete_all_archives_failure(self): + """ + Test delete archives failure + :return: + """ + support = saltsupport.SaltSupportModule() + support.archives = MagicMock( + return_value=[ + "/mnt/storage/one-support-000-000.bz2", + "/mnt/storage/two-support-111-111.bz2", + "/mnt/storage/three-support-222-222.bz2", + ] + ) + ret = support.delete_archives() + assert "files" in ret + assert "errors" in ret + assert bool(ret["errors"]) + assert bool(ret["files"]) + assert isinstance(ret["errors"], dict) + assert isinstance(ret["files"], dict) + + assert ret["files"]["/mnt/storage/three-support-222-222.bz2"] == "removed" + assert ret["files"]["/mnt/storage/one-support-000-000.bz2"] == "left" + assert ret["files"]["/mnt/storage/two-support-111-111.bz2"] == "left" + + assert len(ret["errors"]) == 2 + assert ( + ret["errors"]["/mnt/storage/one-support-000-000.bz2"] + == "Decreasing electron flux" + ) + assert ( + ret["errors"]["/mnt/storage/two-support-111-111.bz2"] + == "Solar flares interference" + ) + + def test_format_sync_stats(self): + """ + Test format rsync stats for preserving ordering of the keys + + :return: + """ + support = saltsupport.SaltSupportModule() + stats = """ +robot: Bender +cute: Leela +weird: Zoidberg +professor: Farnsworth + """ + f_stats = support.format_sync_stats({"retcode": 0, "stdout": stats}) + assert list(f_stats["transfer"].keys()) == [ + "robot", + "cute", + "weird", + "professor", + ] + assert list(f_stats["transfer"].values()) == [ + "Bender", + "Leela", + "Zoidberg", + "Farnsworth", + ] + + @patch("tempfile.mkstemp", MagicMock(return_value=(0, "dummy"))) + @patch("os.close", MagicMock()) + def test_sync_no_archives_failure(self): + """ + Test sync failed when no archives specified. + + :return: + """ + support = saltsupport.SaltSupportModule() + support.archives = MagicMock(return_value=[]) + + with pytest.raises(salt.exceptions.SaltInvocationError) as err: + support.sync("group-name") + assert "No archives found to transfer" in str(err) + + @patch("tempfile.mkstemp", MagicMock(return_value=(0, "dummy"))) + @patch("os.path.exists", MagicMock(return_value=False)) + def test_sync_last_picked_archive_not_found_failure(self): + """ + Test sync failed when archive was not found (last picked) + + :return: + """ + support = saltsupport.SaltSupportModule() + support.archives = MagicMock( + return_value=[ + "/mnt/storage/one-support-000-000.bz2", + "/mnt/storage/two-support-111-111.bz2", + "/mnt/storage/three-support-222-222.bz2", + ] + ) + + with pytest.raises(salt.exceptions.SaltInvocationError) as err: + support.sync("group-name") + assert ( + ' Support archive "/mnt/storage/three-support-222-222.bz2" was not found' + in str(err) + ) + + @patch("tempfile.mkstemp", MagicMock(return_value=(0, "dummy"))) + @patch("os.path.exists", MagicMock(return_value=False)) + def test_sync_specified_archive_not_found_failure(self): + """ + Test sync failed when archive was not found (last picked) + + :return: + """ + support = saltsupport.SaltSupportModule() + support.archives = MagicMock( + return_value=[ + "/mnt/storage/one-support-000-000.bz2", + "/mnt/storage/two-support-111-111.bz2", + "/mnt/storage/three-support-222-222.bz2", + ] + ) + + with pytest.raises(salt.exceptions.SaltInvocationError) as err: + support.sync("group-name", name="lost.bz2") + assert ' Support archive "lost.bz2" was not found' in str(err) + + @patch("tempfile.mkstemp", MagicMock(return_value=(0, "dummy"))) + @patch("os.path.exists", MagicMock(return_value=False)) + @patch("os.close", MagicMock()) + def test_sync_no_archive_to_transfer_failure(self): + """ + Test sync failed when no archive was found to transfer + + :return: + """ + support = saltsupport.SaltSupportModule() + support.archives = MagicMock(return_value=[]) + with pytest.raises(salt.exceptions.SaltInvocationError) as err: + support.sync("group-name", all=True) + assert "No archives found to transfer" in str(err) + + @patch("tempfile.mkstemp", MagicMock(return_value=(0, "dummy"))) + @patch("os.path.exists", MagicMock(return_value=True)) + @patch("os.close", MagicMock()) + @patch("os.write", MagicMock()) + @patch("os.unlink", MagicMock()) + @patch( + "salt.modules.saltsupport.__salt__", {"rsync.rsync": MagicMock(return_value={})} + ) + def test_sync_archives(self): + """ + Test sync archives + :return: + """ + support = saltsupport.SaltSupportModule() + support.archives = MagicMock( + return_value=[ + "/mnt/storage/one-support-000-000.bz2", + "/mnt/storage/two-support-111-111.bz2", + "/mnt/storage/three-support-222-222.bz2", + ] + ) + out = support.sync("group-name", host="buzz", all=True, move=False) + assert "files" in out + for arc_name in out["files"]: + assert out["files"][arc_name] == "copied" + assert saltsupport.os.unlink.call_count == 1 + assert saltsupport.os.unlink.call_args_list[0][0][0] == "dummy" + calls = [] + for call in saltsupport.os.write.call_args_list: + assert len(call) == 2 + calls.append(call[0]) + assert calls == [ + (0, b"one-support-000-000.bz2"), + (0, b"\n"), + (0, b"two-support-111-111.bz2"), + (0, b"\n"), + (0, b"three-support-222-222.bz2"), + (0, b"\n"), + ] + + @patch("salt.modules.saltsupport.__pillar__", {}) + @patch("salt.modules.saltsupport.SupportDataCollector", MagicMock()) + def test_run_support(self): + """ + Test run support + :return: + """ + saltsupport.SupportDataCollector(None, None).archive_path = "dummy" + support = saltsupport.SaltSupportModule() + support.collect_internal_data = MagicMock() + support.collect_local_data = MagicMock() + out = support.run() + + for section in ["messages", "archive"]: + assert section in out + assert out["archive"] == "dummy" + for section in ["warning", "error", "info"]: + assert section in out["messages"] + ld_call = support.collect_local_data.call_args_list[0][1] + assert "profile" in ld_call + assert ld_call["profile"] == "default" + assert "profile_source" in ld_call + assert ld_call["profile_source"] is None + assert support.collector.open.call_count == 1 + assert support.collector.close.call_count == 1 + assert support.collect_internal_data.call_count == 1 + + +@skipIf(not bool(pytest), "Pytest required") +@skipIf(NO_MOCK, NO_MOCK_REASON) +class LogCollectorTestCase(TestCase, LoaderModuleMockMixin): + """ + Test cases for salt.modules.support::LogCollector + """ + + def setup_loader_modules(self): + return {saltsupport: {}} + + def test_msg(self): + """ + Test message to the log collector. + + :return: + """ + utcmock = MagicMock() + utcmock.utcnow = MagicMock(return_value=datetime.datetime.utcfromtimestamp(0)) + with patch("datetime.datetime", utcmock): + msg = "Upgrading /dev/null device" + out = saltsupport.LogCollector() + out.msg(msg, title="Here") + assert saltsupport.LogCollector.INFO in out.messages + assert ( + type(out.messages[saltsupport.LogCollector.INFO]) + == saltsupport.LogCollector.MessagesList + ) + assert out.messages[saltsupport.LogCollector.INFO] == [ + "00:00:00.000 - {}: {}".format("Here", msg) + ] + + def test_info_message(self): + """ + Test info message to the log collector. + + :return: + """ + utcmock = MagicMock() + utcmock.utcnow = MagicMock(return_value=datetime.datetime.utcfromtimestamp(0)) + with patch("datetime.datetime", utcmock): + msg = "SIMM crosstalk during tectonic stress" + out = saltsupport.LogCollector() + out.info(msg) + assert saltsupport.LogCollector.INFO in out.messages + assert ( + type(out.messages[saltsupport.LogCollector.INFO]) + == saltsupport.LogCollector.MessagesList + ) + assert out.messages[saltsupport.LogCollector.INFO] == [ + "00:00:00.000 - {}".format(msg) + ] + + def test_put_message(self): + """ + Test put message to the log collector. + + :return: + """ + utcmock = MagicMock() + utcmock.utcnow = MagicMock(return_value=datetime.datetime.utcfromtimestamp(0)) + with patch("datetime.datetime", utcmock): + msg = "Webmaster kidnapped by evil cult" + out = saltsupport.LogCollector() + out.put(msg) + assert saltsupport.LogCollector.INFO in out.messages + assert ( + type(out.messages[saltsupport.LogCollector.INFO]) + == saltsupport.LogCollector.MessagesList + ) + assert out.messages[saltsupport.LogCollector.INFO] == [ + "00:00:00.000 - {}".format(msg) + ] + + def test_warning_message(self): + """ + Test warning message to the log collector. + + :return: + """ + utcmock = MagicMock() + utcmock.utcnow = MagicMock(return_value=datetime.datetime.utcfromtimestamp(0)) + with patch("datetime.datetime", utcmock): + msg = "Your e-mail is now being delivered by USPS" + out = saltsupport.LogCollector() + out.warning(msg) + assert saltsupport.LogCollector.WARNING in out.messages + assert ( + type(out.messages[saltsupport.LogCollector.WARNING]) + == saltsupport.LogCollector.MessagesList + ) + assert out.messages[saltsupport.LogCollector.WARNING] == [ + "00:00:00.000 - {}".format(msg) + ] + + def test_error_message(self): + """ + Test error message to the log collector. + + :return: + """ + utcmock = MagicMock() + utcmock.utcnow = MagicMock(return_value=datetime.datetime.utcfromtimestamp(0)) + with patch("datetime.datetime", utcmock): + msg = "Learning curve appears to be fractal" + out = saltsupport.LogCollector() + out.error(msg) + assert saltsupport.LogCollector.ERROR in out.messages + assert ( + type(out.messages[saltsupport.LogCollector.ERROR]) + == saltsupport.LogCollector.MessagesList + ) + assert out.messages[saltsupport.LogCollector.ERROR] == [ + "00:00:00.000 - {}".format(msg) + ] + + def test_hl_message(self): + """ + Test highlighter message to the log collector. + + :return: + """ + utcmock = MagicMock() + utcmock.utcnow = MagicMock(return_value=datetime.datetime.utcfromtimestamp(0)) + with patch("datetime.datetime", utcmock): + out = saltsupport.LogCollector() + out.highlight("The {} TTYs became {} TTYs and vice versa", "real", "pseudo") + assert saltsupport.LogCollector.INFO in out.messages + assert ( + type(out.messages[saltsupport.LogCollector.INFO]) + == saltsupport.LogCollector.MessagesList + ) + assert out.messages[saltsupport.LogCollector.INFO] == [ + "00:00:00.000 - The real TTYs became " "pseudo TTYs and vice versa" + ] -- 2.39.2
Locations
Projects
Search
Status Monitor
Help
OpenBuildService.org
Documentation
API Documentation
Code of Conduct
Contact
Support
@OBShq
Terms
openSUSE Build Service is sponsored by
The Open Build Service is an
openSUSE project
.
Sign Up
Log In
Places
Places
All Projects
Status Monitor
