File backport-virt-patches-from-3001-256.patch of Package salt
From e932261250fbdd4deff7341c1c2826ed562b102c Mon Sep 17 00:00:00 2001
From: Cedric Bosdonnat <cbosdonnat@suse.com>
Date: Mon, 7 Sep 2020 15:00:40 +0200
Subject: [PATCH] Backport virt patches from 3001+ (#256)
* Fix various spelling mistakes in master branch (#55954)
* Fix typo of additional
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of against
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of amount
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of argument
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of attempt
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of bandwidth
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of caught
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of compatibility
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of consistency
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of conversions
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of corresponding
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of dependent
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of dictionary
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of disabled
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of adapters
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of disassociates
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of changes
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of command
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of communicate
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of community
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of configuration
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of default
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of absence
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of attribute
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of container
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of described
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of existence
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of explicit
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of formatted
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of guarantees
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of hexadecimal
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of hierarchy
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of initialize
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of label
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of management
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of mismatch
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of don't
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of manually
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of getting
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of information
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of meant
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of nonexistent
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of occur
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of omitted
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of normally
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of overridden
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of repository
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of separate
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of separator
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of specific
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of successful
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of succeeded
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of support
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of version
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of that's
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of "will be removed"
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of release
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of synchronize
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of python
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of usually
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of override
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of running
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of whether
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of package
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of persist
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of preferred
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of present
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix typo of run
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix spelling mistake of "allows someone to..."
"Allows to" is not correct English. It must either be "allows someone
to" or "allows doing".
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix spelling mistake of "number of times"
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix spelling mistake of msgpack
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix spelling mistake of daemonized
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix spelling mistake of daemons
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix spelling mistake of extemporaneous
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix spelling mistake of instead
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix spelling mistake of returning
Signed-off-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
* Fix literal comparissons
* virt: Convert cpu_baseline ElementTree to string
In commit 0f5184c (Remove minidom use in virt module) the value
of `cpu` become `xml.etree.ElementTree.Element` and no longer
has a method `toxml()`. This results in the following error:
$ salt '*' virt.cpu_baseline
host2:
The minion function caused an exception: Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/salt/minion.py", line 1675, in _thread_return
return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
File "/usr/lib/python3.7/site-packages/salt/executors/direct_call.py", line 12, in execute
return func(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/salt/modules/virt.py", line 4410, in cpu_baseline
return cpu.toxml()
AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'toxml'
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
* PR#57374 backport
virt: pool secret should be undefined in pool_undefine not pool_delete
virt: handle build differently depending on the pool type
virt: don't fail if the pool secret has been removed
* PR #57396 backport
add firmware auto select feature
* virt: Update dependencies
Closes: #57641
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
* use null in sls file to map None object
add sls file example
reword doc
* Update virt module and states and their tests to python3
* PR #57545 backport
Move virt.init boot_dev parameter away from the kwargs
virt: handle boot device in virt.update()
virt: add boot_dev parameter to virt.running state
* PR #57431 backport
virt: Handle no available hypervisors
virt: Remove unused imports
* Blacken salt
* Add method to remove circular references in data objects and add test (#54930)
* Add method to remove circular references in data objects and add test
* remove trailing whitespace
* Blacken changed files
Co-authored-by: xeacott <jeacott@saltstack.com>
Co-authored-by: Frode Gundersen <fgundersen@saltstack.com>
Co-authored-by: Daniel A. Wozniak <dwozniak@saltstack.com>
* PR #58332 backport
virt: add debug log with VM XML definition
Add xmlutil.get_xml_node() helper function
Add salt.utils.data.get_value function
Add change_xml() function to xmlutil
virt.update: refactor the XML diffing code
virt.test_update: move some code to make test more readable
Co-authored-by: Benjamin Drung <benjamin.drung@cloud.ionos.com>
Co-authored-by: Pedro Algarvio <pedro@algarvio.me>
Co-authored-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Co-authored-by: Firefly <leevn2011@hotmail.com>
Co-authored-by: Blacken Salt <jenkins@saltstack.com>
Co-authored-by: Joe Eacott <31625359+xeacott@users.noreply.github.com>
Co-authored-by: xeacott <jeacott@saltstack.com>
Co-authored-by: Frode Gundersen <fgundersen@saltstack.com>
Co-authored-by: Daniel A. Wozniak <dwozniak@saltstack.com>
---
changelog/56454.fixed | 1 +
changelog/57544.added | 1 +
changelog/58331.fixed | 1 +
salt/modules/virt.py | 442 ++++---
salt/states/virt.py | 171 ++-
salt/templates/virt/libvirt_domain.jinja | 2 +-
salt/utils/data.py | 976 +++++++++------
salt/utils/xmlutil.py | 251 +++-
tests/pytests/unit/utils/test_data.py | 57 +
tests/pytests/unit/utils/test_xmlutil.py | 169 +++
tests/unit/modules/test_virt.py | 218 ++--
tests/unit/states/test_virt.py | 98 +-
tests/unit/utils/test_data.py | 1399 ++++++++++++----------
tests/unit/utils/test_xmlutil.py | 164 +--
14 files changed, 2588 insertions(+), 1362 deletions(-)
create mode 100644 changelog/56454.fixed
create mode 100644 changelog/57544.added
create mode 100644 changelog/58331.fixed
create mode 100644 tests/pytests/unit/utils/test_data.py
create mode 100644 tests/pytests/unit/utils/test_xmlutil.py
diff --git a/changelog/56454.fixed b/changelog/56454.fixed
new file mode 100644
index 0000000000..978b4b6e03
--- /dev/null
+++ b/changelog/56454.fixed
@@ -0,0 +1 @@
+Better handle virt.pool_rebuild in virt.pool_running and virt.pool_defined states
diff --git a/changelog/57544.added b/changelog/57544.added
new file mode 100644
index 0000000000..52071cf2c7
--- /dev/null
+++ b/changelog/57544.added
@@ -0,0 +1 @@
+Allow setting VM boot devices order in virt.running and virt.defined states
diff --git a/changelog/58331.fixed b/changelog/58331.fixed
new file mode 100644
index 0000000000..4b8f78dd53
--- /dev/null
+++ b/changelog/58331.fixed
@@ -0,0 +1 @@
+Leave boot parameters untouched if boot parameter is set to None in virt.update
diff --git a/salt/modules/virt.py b/salt/modules/virt.py
index a78c21e323..cd80fbe608 100644
--- a/salt/modules/virt.py
+++ b/salt/modules/virt.py
@@ -1,8 +1,11 @@
-# -*- coding: utf-8 -*-
"""
Work with virtual machines managed by libvirt
-:depends: libvirt Python module
+:depends:
+ * libvirt Python module
+ * libvirt client
+ * qemu-img
+ * grep
Connection
==========
@@ -73,7 +76,7 @@ The calls not using the libvirt connection setup are:
# of his in the virt func module have been used
# Import python libs
-from __future__ import absolute_import, print_function, unicode_literals
+
import base64
import copy
import datetime
@@ -89,23 +92,19 @@ from xml.etree import ElementTree
from xml.sax import saxutils
# Import third party libs
-import jinja2
import jinja2.exceptions
# Import salt libs
+import salt.utils.data
import salt.utils.files
import salt.utils.json
-import salt.utils.network
import salt.utils.path
import salt.utils.stringutils
import salt.utils.templates
-import salt.utils.validate.net
-import salt.utils.versions
import salt.utils.xmlutil as xmlutil
import salt.utils.yaml
from salt._compat import ipaddress
from salt.exceptions import CommandExecutionError, SaltInvocationError
-from salt.ext import six
from salt.ext.six.moves import range # pylint: disable=import-error,redefined-builtin
from salt.ext.six.moves.urllib.parse import urlparse, urlunparse
from salt.utils.virt import check_remote, download_remote
@@ -227,8 +226,8 @@ def __get_conn(**kwargs):
)
except Exception: # pylint: disable=broad-except
raise CommandExecutionError(
- "Sorry, {0} failed to open a connection to the hypervisor "
- "software at {1}".format(__grains__["fqdn"], conn_str)
+ "Sorry, {} failed to open a connection to the hypervisor "
+ "software at {}".format(__grains__["fqdn"], conn_str)
)
return conn
@@ -405,7 +404,7 @@ def _get_nics(dom):
# driver, source, and match can all have optional attributes
if re.match("(driver|source|address)", v_node.tag):
temp = {}
- for key, value in six.iteritems(v_node.attrib):
+ for key, value in v_node.attrib.items():
temp[key] = value
nic[v_node.tag] = temp
# virtualport needs to be handled separately, to pick up the
@@ -413,7 +412,7 @@ def _get_nics(dom):
if v_node.tag == "virtualport":
temp = {}
temp["type"] = v_node.get("type")
- for key, value in six.iteritems(v_node.attrib):
+ for key, value in v_node.attrib.items():
temp[key] = value
nic["virtualport"] = temp
if "mac" not in nic:
@@ -435,7 +434,7 @@ def _get_graphics(dom):
}
doc = ElementTree.fromstring(dom.XMLDesc(0))
for g_node in doc.findall("devices/graphics"):
- for key, value in six.iteritems(g_node.attrib):
+ for key, value in g_node.attrib.items():
out[key] = value
return out
@@ -448,7 +447,7 @@ def _get_loader(dom):
doc = ElementTree.fromstring(dom.XMLDesc(0))
for g_node in doc.findall("os/loader"):
out["path"] = g_node.text
- for key, value in six.iteritems(g_node.attrib):
+ for key, value in g_node.attrib.items():
out[key] = value
return out
@@ -503,7 +502,7 @@ def _get_disks(conn, dom):
qemu_target = source.get("protocol")
source_name = source.get("name")
if source_name:
- qemu_target = "{0}:{1}".format(qemu_target, source_name)
+ qemu_target = "{}:{}".format(qemu_target, source_name)
# Reverse the magic for the rbd and gluster pools
if source.get("protocol") in ["rbd", "gluster"]:
@@ -633,7 +632,7 @@ def _get_target(target, ssh):
proto = "qemu"
if ssh:
proto += "+ssh"
- return " {0}://{1}/{2}".format(proto, target, "system")
+ return " {}://{}/{}".format(proto, target, "system")
def _gen_xml(
@@ -648,6 +647,7 @@ def _gen_xml(
arch,
graphics=None,
boot=None,
+ boot_dev=None,
**kwargs
):
"""
@@ -657,8 +657,8 @@ def _gen_xml(
context = {
"hypervisor": hypervisor,
"name": name,
- "cpu": six.text_type(cpu),
- "mem": six.text_type(mem),
+ "cpu": str(cpu),
+ "mem": str(mem),
}
if hypervisor in ["qemu", "kvm"]:
context["controller_model"] = False
@@ -681,15 +681,17 @@ def _gen_xml(
graphics = None
context["graphics"] = graphics
- if "boot_dev" in kwargs:
- context["boot_dev"] = []
- for dev in kwargs["boot_dev"].split():
- context["boot_dev"].append(dev)
- else:
- context["boot_dev"] = ["hd"]
+ context["boot_dev"] = boot_dev.split() if boot_dev is not None else ["hd"]
context["boot"] = boot if boot else {}
+ # if efi parameter is specified, prepare os_attrib
+ efi_value = context["boot"].get("efi", None) if boot else None
+ if efi_value is True:
+ context["boot"]["os_attrib"] = "firmware='efi'"
+ elif efi_value is not None and type(efi_value) != bool:
+ raise SaltInvocationError("Invalid efi value")
+
if os_type == "xen":
# Compute the Xen PV boot method
if __grains__["os_family"] == "Suse":
@@ -720,7 +722,7 @@ def _gen_xml(
"target_dev": _get_disk_target(targets, len(diskp), prefix),
"disk_bus": disk["model"],
"format": disk.get("format", "raw"),
- "index": six.text_type(i),
+ "index": str(i),
}
targets.append(disk_context["target_dev"])
if disk.get("source_file"):
@@ -825,8 +827,8 @@ def _gen_vol_xml(
"name": name,
"target": {"permissions": permissions, "nocow": nocow},
"format": format,
- "size": six.text_type(size),
- "allocation": six.text_type(int(allocation) * 1024),
+ "size": str(size),
+ "allocation": str(int(allocation) * 1024),
"backingStore": backing_store,
}
fn_ = "libvirt_volume.jinja"
@@ -978,31 +980,29 @@ def _zfs_image_create(
"""
if not disk_image_name and not disk_size:
raise CommandExecutionError(
- "Unable to create new disk {0}, please specify"
+ "Unable to create new disk {}, please specify"
" the disk image name or disk size argument".format(disk_name)
)
if not pool:
raise CommandExecutionError(
- "Unable to create new disk {0}, please specify"
+ "Unable to create new disk {}, please specify"
" the disk pool name".format(disk_name)
)
- destination_fs = os.path.join(pool, "{0}.{1}".format(vm_name, disk_name))
+ destination_fs = os.path.join(pool, "{}.{}".format(vm_name, disk_name))
log.debug("Image destination will be %s", destination_fs)
existing_disk = __salt__["zfs.list"](name=pool)
if "error" in existing_disk:
raise CommandExecutionError(
- "Unable to create new disk {0}. {1}".format(
+ "Unable to create new disk {}. {}".format(
destination_fs, existing_disk["error"]
)
)
elif destination_fs in existing_disk:
log.info(
- "ZFS filesystem {0} already exists. Skipping creation".format(
- destination_fs
- )
+ "ZFS filesystem {} already exists. Skipping creation".format(destination_fs)
)
blockdevice_path = os.path.join("/dev/zvol", pool, vm_name)
return blockdevice_path
@@ -1025,7 +1025,7 @@ def _zfs_image_create(
)
blockdevice_path = os.path.join(
- "/dev/zvol", pool, "{0}.{1}".format(vm_name, disk_name)
+ "/dev/zvol", pool, "{}.{}".format(vm_name, disk_name)
)
log.debug("Image path will be %s", blockdevice_path)
return blockdevice_path
@@ -1042,7 +1042,7 @@ def _qemu_image_create(disk, create_overlay=False, saltenv="base"):
if not disk_size and not disk_image:
raise CommandExecutionError(
- "Unable to create new disk {0}, please specify"
+ "Unable to create new disk {}, please specify"
" disk size and/or disk image argument".format(disk["filename"])
)
@@ -1066,7 +1066,7 @@ def _qemu_image_create(disk, create_overlay=False, saltenv="base"):
if create_overlay and qcow2:
log.info("Cloning qcow2 image %s using copy on write", sfn)
__salt__["cmd.run"](
- 'qemu-img create -f qcow2 -o backing_file="{0}" "{1}"'.format(
+ 'qemu-img create -f qcow2 -o backing_file="{}" "{}"'.format(
sfn, img_dest
).split()
)
@@ -1079,16 +1079,16 @@ def _qemu_image_create(disk, create_overlay=False, saltenv="base"):
if disk_size and qcow2:
log.debug("Resize qcow2 image to %sM", disk_size)
__salt__["cmd.run"](
- 'qemu-img resize "{0}" {1}M'.format(img_dest, disk_size)
+ 'qemu-img resize "{}" {}M'.format(img_dest, disk_size)
)
log.debug("Apply umask and remove exec bit")
mode = (0o0777 ^ mask) & 0o0666
os.chmod(img_dest, mode)
- except (IOError, OSError) as err:
+ except OSError as err:
raise CommandExecutionError(
- "Problem while copying image. {0} - {1}".format(disk_image, err)
+ "Problem while copying image. {} - {}".format(disk_image, err)
)
else:
@@ -1099,13 +1099,13 @@ def _qemu_image_create(disk, create_overlay=False, saltenv="base"):
if disk_size:
log.debug("Create empty image with size %sM", disk_size)
__salt__["cmd.run"](
- 'qemu-img create -f {0} "{1}" {2}M'.format(
+ 'qemu-img create -f {} "{}" {}M'.format(
disk.get("format", "qcow2"), img_dest, disk_size
)
)
else:
raise CommandExecutionError(
- "Unable to create new disk {0},"
+ "Unable to create new disk {},"
" please specify <size> argument".format(img_dest)
)
@@ -1113,9 +1113,9 @@ def _qemu_image_create(disk, create_overlay=False, saltenv="base"):
mode = (0o0777 ^ mask) & 0o0666
os.chmod(img_dest, mode)
- except (IOError, OSError) as err:
+ except OSError as err:
raise CommandExecutionError(
- "Problem while creating volume {0} - {1}".format(img_dest, err)
+ "Problem while creating volume {} - {}".format(img_dest, err)
)
return img_dest
@@ -1252,7 +1252,7 @@ def _disk_profile(conn, profile, hypervisor, disks, vm_name):
__salt__["config.get"]("virt:disk", {}).get(profile, default)
)
- # Transform the list to remove one level of dictionnary and add the name as a property
+ # Transform the list to remove one level of dictionary and add the name as a property
disklist = [dict(d, name=name) for disk in disklist for name, d in disk.items()]
# Merge with the user-provided disks definitions
@@ -1274,7 +1274,7 @@ def _disk_profile(conn, profile, hypervisor, disks, vm_name):
disk["model"] = "ide"
# Add the missing properties that have defaults
- for key, val in six.iteritems(overlay):
+ for key, val in overlay.items():
if key not in disk:
disk[key] = val
@@ -1296,7 +1296,7 @@ def _fill_disk_filename(conn, vm_name, disk, hypervisor, pool_caps):
Compute the disk file name and update it in the disk value.
"""
# Compute the filename without extension since it may not make sense for some pool types
- disk["filename"] = "{0}_{1}".format(vm_name, disk["name"])
+ disk["filename"] = "{}_{}".format(vm_name, disk["name"])
# Compute the source file path
base_dir = disk.get("pool", None)
@@ -1311,7 +1311,7 @@ def _fill_disk_filename(conn, vm_name, disk, hypervisor, pool_caps):
# For path-based disks, keep the qcow2 default format
if not disk.get("format"):
disk["format"] = "qcow2"
- disk["filename"] = "{0}.{1}".format(disk["filename"], disk["format"])
+ disk["filename"] = "{}.{}".format(disk["filename"], disk["format"])
disk["source_file"] = os.path.join(base_dir, disk["filename"])
else:
if "pool" not in disk:
@@ -1365,7 +1365,7 @@ def _fill_disk_filename(conn, vm_name, disk, hypervisor, pool_caps):
disk["format"] = volume_options.get("default_format", None)
elif hypervisor == "bhyve" and vm_name:
- disk["filename"] = "{0}.{1}".format(vm_name, disk["name"])
+ disk["filename"] = "{}.{}".format(vm_name, disk["name"])
disk["source_file"] = os.path.join(
"/dev/zvol", base_dir or "", disk["filename"]
)
@@ -1373,8 +1373,8 @@ def _fill_disk_filename(conn, vm_name, disk, hypervisor, pool_caps):
elif hypervisor in ["esxi", "vmware"]:
if not base_dir:
base_dir = __salt__["config.get"]("virt:storagepool", "[0] ")
- disk["filename"] = "{0}.{1}".format(disk["filename"], disk["format"])
- disk["source_file"] = "{0}{1}".format(base_dir, disk["filename"])
+ disk["filename"] = "{}.{}".format(disk["filename"], disk["format"])
+ disk["source_file"] = "{}{}".format(base_dir, disk["filename"])
def _complete_nics(interfaces, hypervisor):
@@ -1422,7 +1422,7 @@ def _complete_nics(interfaces, hypervisor):
"""
Apply the default overlay to attributes
"""
- for key, value in six.iteritems(overlays[hypervisor]):
+ for key, value in overlays[hypervisor].items():
if key not in attributes or not attributes[key]:
attributes[key] = value
@@ -1449,7 +1449,7 @@ def _nic_profile(profile_name, hypervisor):
"""
Append dictionary profile data to interfaces list
"""
- for interface_name, attributes in six.iteritems(profile_dict):
+ for interface_name, attributes in profile_dict.items():
attributes["name"] = interface_name
interfaces.append(attributes)
@@ -1522,17 +1522,24 @@ def _handle_remote_boot_params(orig_boot):
new_boot = orig_boot.copy()
keys = orig_boot.keys()
cases = [
+ {"efi"},
+ {"kernel", "initrd", "efi"},
+ {"kernel", "initrd", "cmdline", "efi"},
{"loader", "nvram"},
{"kernel", "initrd"},
{"kernel", "initrd", "cmdline"},
- {"loader", "nvram", "kernel", "initrd"},
- {"loader", "nvram", "kernel", "initrd", "cmdline"},
+ {"kernel", "initrd", "loader", "nvram"},
+ {"kernel", "initrd", "cmdline", "loader", "nvram"},
]
try:
if keys in cases:
for key in keys:
- if orig_boot.get(key) is not None and check_remote(orig_boot.get(key)):
+ if key == "efi" and type(orig_boot.get(key)) == bool:
+ new_boot[key] = orig_boot.get(key)
+ elif orig_boot.get(key) is not None and check_remote(
+ orig_boot.get(key)
+ ):
if saltinst_dir is None:
os.makedirs(CACHE_DIR)
saltinst_dir = CACHE_DIR
@@ -1540,12 +1547,41 @@ def _handle_remote_boot_params(orig_boot):
return new_boot
else:
raise SaltInvocationError(
- "Invalid boot parameters, (kernel, initrd) or/and (loader, nvram) must be both present"
+ "Invalid boot parameters,It has to follow this combination: [(kernel, initrd) or/and cmdline] or/and [(loader, nvram) or efi]"
)
except Exception as err: # pylint: disable=broad-except
raise err
+def _handle_efi_param(boot, desc):
+ """
+ Checks if boot parameter contains efi boolean value, if so, handles the firmware attribute.
+ :param boot: The boot parameters passed to the init or update functions.
+ :param desc: The XML description of that domain.
+ :return: A boolean value.
+ """
+ efi_value = boot.get("efi", None) if boot else None
+ parent_tag = desc.find("os")
+ os_attrib = parent_tag.attrib
+
+ # newly defined vm without running, loader tag might not be filled yet
+ if efi_value is False and os_attrib != {}:
+ parent_tag.attrib.pop("firmware", None)
+ return True
+
+ # check the case that loader tag might be present. This happens after the vm ran
+ elif type(efi_value) == bool and os_attrib == {}:
+ if efi_value is True and parent_tag.find("loader") is None:
+ parent_tag.set("firmware", "efi")
+ if efi_value is False and parent_tag.find("loader") is not None:
+ parent_tag.remove(parent_tag.find("loader"))
+ parent_tag.remove(parent_tag.find("nvram"))
+ return True
+ elif type(efi_value) != bool:
+ raise SaltInvocationError("Invalid efi value")
+ return False
+
+
def init(
name,
cpu,
@@ -1566,6 +1602,7 @@ def init(
os_type=None,
arch=None,
boot=None,
+ boot_dev=None,
**kwargs
):
"""
@@ -1635,7 +1672,8 @@ def init(
This is an optional parameter, all of the keys are optional within the dictionary. The structure of
the dictionary is documented in :ref:`init-boot-def`. If a remote path is provided to kernel or initrd,
salt will handle the downloading of the specified remote file and modify the XML accordingly.
- To boot VM with UEFI, specify loader and nvram path.
+ To boot VM with UEFI, specify loader and nvram path or specify 'efi': ``True`` if your libvirtd version
+ is >= 5.2.0 and QEMU >= 3.0.0.
.. versionadded:: 3000
@@ -1649,6 +1687,12 @@ def init(
'nvram': '/usr/share/OVMF/OVMF_VARS.ms.fd'
}
+ :param boot_dev:
+ Space separated list of devices to boot from sorted by decreasing priority.
+ Values can be ``hd``, ``fd``, ``cdrom`` or ``network``.
+
+ By default, the value will ``"hd"``.
+
.. _init-boot-def:
.. rubric:: Boot parameters definition
@@ -1674,6 +1718,11 @@ def init(
.. versionadded:: sodium
+ efi
+ A boolean value.
+
+ .. versionadded:: sodium
+
.. _init-nic-def:
.. rubric:: Network Interfaces Definitions
@@ -1797,7 +1846,7 @@ def init(
.. rubric:: Graphics Definition
- The graphics dictionnary can have the following properties:
+ The graphics dictionary can have the following properties:
type
Graphics type. The possible values are ``none``, ``'spice'``, ``'vnc'`` and other values
@@ -1858,6 +1907,8 @@ def init(
for x in y
}
)
+ if len(hypervisors) == 0:
+ raise SaltInvocationError("No supported hypervisors were found")
virt_hypervisor = "kvm" if "kvm" in hypervisors else hypervisors[0]
# esxi used to be a possible value for the hypervisor: map it to vmware since it's the same
@@ -1890,8 +1941,8 @@ def init(
else:
# assume libvirt manages disks for us
log.debug("Generating libvirt XML for %s", _disk)
- volume_name = "{0}/{1}".format(name, _disk["name"])
- filename = "{0}.{1}".format(volume_name, _disk["format"])
+ volume_name = "{}/{}".format(name, _disk["name"])
+ filename = "{}.{}".format(volume_name, _disk["format"])
vol_xml = _gen_vol_xml(
filename, _disk["size"], format=_disk["format"]
)
@@ -1939,7 +1990,7 @@ def init(
else:
# Unknown hypervisor
raise SaltInvocationError(
- "Unsupported hypervisor when handling disk image: {0}".format(
+ "Unsupported hypervisor when handling disk image: {}".format(
virt_hypervisor
)
)
@@ -1965,8 +2016,10 @@ def init(
arch,
graphics,
boot,
+ boot_dev,
**kwargs
)
+ log.debug("New virtual machine definition: %s", vm_xml)
conn.defineXML(vm_xml)
except libvirt.libvirtError as err:
conn.close()
@@ -2192,6 +2245,7 @@ def update(
live=True,
boot=None,
test=False,
+ boot_dev=None,
**kwargs
):
"""
@@ -2234,11 +2288,28 @@ def update(
Refer to :ref:`init-boot-def` for the complete boot parameter description.
- To update any boot parameters, specify the new path for each. To remove any boot parameters,
- pass a None object, for instance: 'kernel': ``None``.
+ To update any boot parameters, specify the new path for each. To remove any boot parameters, pass ``None`` object,
+ for instance: 'kernel': ``None``. To switch back to BIOS boot, specify ('loader': ``None`` and 'nvram': ``None``)
+ or 'efi': ``False``. Please note that ``None`` is mapped to ``null`` in sls file, pass ``null`` in sls file instead.
+
+ SLS file Example:
+
+ .. code-block:: yaml
+
+ - boot:
+ loader: null
+ nvram: null
.. versionadded:: 3000
+ :param boot_dev:
+ Space separated list of devices to boot from sorted by decreasing priority.
+ Values can be ``hd``, ``fd``, ``cdrom`` or ``network``.
+
+ By default, the value will ``"hd"``.
+
+ .. versionadded:: Magnesium
+
:param test: run in dry-run mode if set to True
.. versionadded:: sodium
@@ -2286,6 +2357,8 @@ def update(
if boot is not None:
boot = _handle_remote_boot_params(boot)
+ if boot.get("efi", None) is not None:
+ need_update = _handle_efi_param(boot, desc)
new_desc = ElementTree.fromstring(
_gen_xml(
@@ -2307,76 +2380,58 @@ def update(
# Update the cpu
cpu_node = desc.find("vcpu")
if cpu and int(cpu_node.text) != cpu:
- cpu_node.text = six.text_type(cpu)
- cpu_node.set("current", six.text_type(cpu))
+ cpu_node.text = str(cpu)
+ cpu_node.set("current", str(cpu))
need_update = True
- # Update the kernel boot parameters
- boot_tags = ["kernel", "initrd", "cmdline", "loader", "nvram"]
- parent_tag = desc.find("os")
-
- # We need to search for each possible subelement, and update it.
- for tag in boot_tags:
- # The Existing Tag...
- found_tag = parent_tag.find(tag)
-
- # The new value
- boot_tag_value = boot.get(tag, None) if boot else None
-
- # Existing tag is found and values don't match
- if found_tag is not None and found_tag.text != boot_tag_value:
-
- # If the existing tag is found, but the new value is None
- # remove it. If the existing tag is found, and the new value
- # doesn't match update it. In either case, mark for update.
- if boot_tag_value is None and boot is not None and parent_tag is not None:
- parent_tag.remove(found_tag)
- else:
- found_tag.text = boot_tag_value
+ def _set_loader(node, value):
+ salt.utils.xmlutil.set_node_text(node, value)
+ if value is not None:
+ node.set("readonly", "yes")
+ node.set("type", "pflash")
- # If the existing tag is loader or nvram, we need to update the corresponding attribute
- if found_tag.tag == "loader" and boot_tag_value is not None:
- found_tag.set("readonly", "yes")
- found_tag.set("type", "pflash")
+ def _set_nvram(node, value):
+ node.set("template", value)
- if found_tag.tag == "nvram" and boot_tag_value is not None:
- found_tag.set("template", found_tag.text)
- found_tag.text = None
+ def _set_with_mib_unit(node, value):
+ node.text = str(value)
+ node.set("unit", "MiB")
- need_update = True
-
- # Existing tag is not found, but value is not None
- elif found_tag is None and boot_tag_value is not None:
-
- # Need to check for parent tag, and add it if it does not exist.
- # Add a subelement and set the value to the new value, and then
- # mark for update.
- if parent_tag is not None:
- child_tag = ElementTree.SubElement(parent_tag, tag)
- else:
- new_parent_tag = ElementTree.Element("os")
- child_tag = ElementTree.SubElement(new_parent_tag, tag)
-
- child_tag.text = boot_tag_value
-
- # If the newly created tag is loader or nvram, we need to update the corresponding attribute
- if child_tag.tag == "loader":
- child_tag.set("readonly", "yes")
- child_tag.set("type", "pflash")
-
- if child_tag.tag == "nvram":
- child_tag.set("template", child_tag.text)
- child_tag.text = None
-
- need_update = True
+ # Update the kernel boot parameters
+ params_mapping = [
+ {"path": "boot:kernel", "xpath": "os/kernel"},
+ {"path": "boot:initrd", "xpath": "os/initrd"},
+ {"path": "boot:cmdline", "xpath": "os/cmdline"},
+ {"path": "boot:loader", "xpath": "os/loader", "set": _set_loader},
+ {"path": "boot:nvram", "xpath": "os/nvram", "set": _set_nvram},
+ # Update the memory, note that libvirt outputs all memory sizes in KiB
+ {
+ "path": "mem",
+ "xpath": "memory",
+ "get": lambda n: int(n.text) / 1024,
+ "set": _set_with_mib_unit,
+ },
+ {
+ "path": "mem",
+ "xpath": "currentMemory",
+ "get": lambda n: int(n.text) / 1024,
+ "set": _set_with_mib_unit,
+ },
+ {
+ "path": "boot_dev:{dev}",
+ "xpath": "os/boot[$dev]",
+ "get": lambda n: n.get("dev"),
+ "set": lambda n, v: n.set("dev", v),
+ "del": salt.utils.xmlutil.del_attribute("dev"),
+ },
+ ]
- # Update the memory, note that libvirt outputs all memory sizes in KiB
- for mem_node_name in ["memory", "currentMemory"]:
- mem_node = desc.find(mem_node_name)
- if mem and int(mem_node.text) != mem * 1024:
- mem_node.text = six.text_type(mem)
- mem_node.set("unit", "MiB")
- need_update = True
+ data = {k: v for k, v in locals().items() if bool(v)}
+ if boot_dev:
+ data["boot_dev"] = {i + 1: dev for i, dev in enumerate(boot_dev.split())}
+ need_update = need_update or salt.utils.xmlutil.change_xml(
+ desc, data, params_mapping
+ )
# Update the XML definition with the new disks and diff changes
devices_node = desc.find("devices")
@@ -2395,8 +2450,8 @@ def update(
if func_locals.get(param, None) is not None
]:
old = devices_node.findall(dev_type)
- new = new_desc.findall("devices/{0}".format(dev_type))
- changes[dev_type] = globals()["_diff_{0}_lists".format(dev_type)](old, new)
+ new = new_desc.findall("devices/{}".format(dev_type))
+ changes[dev_type] = globals()["_diff_{}_lists".format(dev_type)](old, new)
if changes[dev_type]["deleted"] or changes[dev_type]["new"]:
for item in old:
devices_node.remove(item)
@@ -2423,9 +2478,9 @@ def update(
_disk_volume_create(conn, all_disks[idx])
if not test:
- conn.defineXML(
- salt.utils.stringutils.to_str(ElementTree.tostring(desc))
- )
+ xml_desc = ElementTree.tostring(desc)
+ log.debug("Update virtual machine definition: %s", xml_desc)
+ conn.defineXML(salt.utils.stringutils.to_str(xml_desc))
status["definition"] = True
except libvirt.libvirtError as err:
conn.close()
@@ -2554,7 +2609,7 @@ def update(
except libvirt.libvirtError as err:
if "errors" not in status:
status["errors"] = []
- status["errors"].append(six.text_type(err))
+ status["errors"].append(str(err))
conn.close()
return status
@@ -2768,7 +2823,7 @@ def _node_info(conn):
info = {
"cpucores": raw[6],
"cpumhz": raw[3],
- "cpumodel": six.text_type(raw[0]),
+ "cpumodel": str(raw[0]),
"cpus": raw[2],
"cputhreads": raw[7],
"numanodes": raw[4],
@@ -3207,24 +3262,21 @@ def get_profiles(hypervisor=None, **kwargs):
for x in y
}
)
- default_hypervisor = "kvm" if "kvm" in hypervisors else hypervisors[0]
+ if len(hypervisors) == 0:
+ raise SaltInvocationError("No supported hypervisors were found")
if not hypervisor:
- hypervisor = default_hypervisor
+ hypervisor = "kvm" if "kvm" in hypervisors else hypervisors[0]
virtconf = __salt__["config.get"]("virt", {})
for typ in ["disk", "nic"]:
- _func = getattr(sys.modules[__name__], "_{0}_profile".format(typ))
+ _func = getattr(sys.modules[__name__], "_{}_profile".format(typ))
ret[typ] = {
- "default": _func(
- "default", hypervisor if hypervisor else default_hypervisor
- )
+ "default": _func("default", hypervisor)
}
if typ in virtconf:
ret.setdefault(typ, {})
for prf in virtconf[typ]:
- ret[typ][prf] = _func(
- prf, hypervisor if hypervisor else default_hypervisor
- )
+ ret[typ][prf] = _func(prf, hypervisor)
return ret
@@ -3506,7 +3558,7 @@ def create_xml_path(path, **kwargs):
return create_xml_str(
salt.utils.stringutils.to_unicode(fp_.read()), **kwargs
)
- except (OSError, IOError):
+ except OSError:
return False
@@ -3564,7 +3616,7 @@ def define_xml_path(path, **kwargs):
return define_xml_str(
salt.utils.stringutils.to_unicode(fp_.read()), **kwargs
)
- except (OSError, IOError):
+ except OSError:
return False
@@ -3576,7 +3628,7 @@ def _define_vol_xml_str(conn, xml, pool=None): # pylint: disable=redefined-oute
poolname = (
pool if pool else __salt__["config.get"]("virt:storagepool", default_pool)
)
- pool = conn.storagePoolLookupByName(six.text_type(poolname))
+ pool = conn.storagePoolLookupByName(str(poolname))
ret = pool.createXML(xml, 0) is not None
return ret
@@ -3660,7 +3712,7 @@ def define_vol_xml_path(path, pool=None, **kwargs):
return define_vol_xml_str(
salt.utils.stringutils.to_unicode(fp_.read()), pool=pool, **kwargs
)
- except (OSError, IOError):
+ except OSError:
return False
@@ -3777,7 +3829,7 @@ def seed_non_shared_migrate(disks, force=False):
salt '*' virt.seed_non_shared_migrate <disks>
"""
- for _, data in six.iteritems(disks):
+ for _, data in disks.items():
fn_ = data["file"]
form = data["file format"]
size = data["virtual size"].split()[1][1:]
@@ -3921,14 +3973,14 @@ def purge(vm_, dirs=False, removables=False, **kwargs):
# TODO create solution for 'dataset is busy'
time.sleep(3)
fs_name = disks[disk]["file"][len("/dev/zvol/") :]
- log.info("Destroying VM ZFS volume {0}".format(fs_name))
+ log.info("Destroying VM ZFS volume {}".format(fs_name))
__salt__["zfs.destroy"](name=fs_name, force=True)
elif os.path.exists(disks[disk]["file"]):
os.remove(disks[disk]["file"])
directories.add(os.path.dirname(disks[disk]["file"]))
else:
# We may have a volume to delete here
- matcher = re.match("^(?P<pool>[^/]+)/(?P<volume>.*)$", disks[disk]["file"],)
+ matcher = re.match("^(?P<pool>[^/]+)/(?P<volume>.*)$", disks[disk]["file"])
if matcher:
pool_name = matcher.group("pool")
pool = None
@@ -3975,7 +4027,7 @@ def _is_kvm_hyper():
with salt.utils.files.fopen("/proc/modules") as fp_:
if "kvm_" not in salt.utils.stringutils.to_unicode(fp_.read()):
return False
- except IOError:
+ except OSError:
# No /proc/modules? Are we on Windows? Or Solaris?
return False
return "libvirtd" in __salt__["cmd.run"](__grains__["ps"])
@@ -3995,7 +4047,7 @@ def _is_xen_hyper():
with salt.utils.files.fopen("/proc/modules") as fp_:
if "xen_" not in salt.utils.stringutils.to_unicode(fp_.read()):
return False
- except (OSError, IOError):
+ except OSError:
# No /proc/modules? Are we on Windows? Or Solaris?
return False
return "libvirtd" in __salt__["cmd.run"](__grains__["ps"])
@@ -4110,7 +4162,7 @@ def vm_cputime(vm_=None, **kwargs):
cputime_percent = (1.0e-7 * cputime / host_cpus) / vcpus
return {
"cputime": int(raw[4]),
- "cputime_percent": int("{0:.0f}".format(cputime_percent)),
+ "cputime_percent": int("{:.0f}".format(cputime_percent)),
}
info = {}
@@ -4180,7 +4232,7 @@ def vm_netstats(vm_=None, **kwargs):
"tx_errs": 0,
"tx_drop": 0,
}
- for attrs in six.itervalues(nics):
+ for attrs in nics.values():
if "target" in attrs:
dev = attrs["target"]
stats = dom.interfaceStats(dev)
@@ -4508,7 +4560,7 @@ def revert_snapshot(name, vm_snapshot=None, cleanup=False, **kwargs):
conn.close()
raise CommandExecutionError(
snapshot
- and 'Snapshot "{0}" not found'.format(vm_snapshot)
+ and 'Snapshot "{}" not found'.format(vm_snapshot)
or "No more previous snapshots available"
)
elif snap.isCurrent():
@@ -5102,10 +5154,10 @@ def cpu_baseline(full=False, migratable=False, out="libvirt", **kwargs):
]
if not cpu_specs:
- raise ValueError("Model {0} not found in CPU map".format(cpu_model))
+ raise ValueError("Model {} not found in CPU map".format(cpu_model))
elif len(cpu_specs) > 1:
raise ValueError(
- "Multiple models {0} found in CPU map".format(cpu_model)
+ "Multiple models {} found in CPU map".format(cpu_model)
)
cpu_specs = cpu_specs[0]
@@ -5126,7 +5178,7 @@ def cpu_baseline(full=False, migratable=False, out="libvirt", **kwargs):
"vendor": cpu.find("vendor").text,
"features": [feature.get("name") for feature in cpu.findall("feature")],
}
- return cpu.toxml()
+ return ElementTree.tostring(cpu)
def network_define(name, bridge, forward, ipv4_config=None, ipv6_config=None, **kwargs):
@@ -5250,7 +5302,7 @@ def list_networks(**kwargs):
def network_info(name=None, **kwargs):
"""
- Return informations on a virtual network provided its name.
+ Return information on a virtual network provided its name.
:param name: virtual network name
:param connection: libvirt connection URI, overriding defaults
@@ -5446,20 +5498,20 @@ def _parse_pools_caps(doc):
for option_kind in ["pool", "vol"]:
options = {}
default_format_node = pool.find(
- "{0}Options/defaultFormat".format(option_kind)
+ "{}Options/defaultFormat".format(option_kind)
)
if default_format_node is not None:
options["default_format"] = default_format_node.get("type")
options_enums = {
enum.get("name"): [value.text for value in enum.findall("value")]
- for enum in pool.findall("{0}Options/enum".format(option_kind))
+ for enum in pool.findall("{}Options/enum".format(option_kind))
}
if options_enums:
options.update(options_enums)
if options:
if "options" not in pool_caps:
pool_caps["options"] = {}
- kind = option_kind if option_kind is not "vol" else "volume"
+ kind = option_kind if option_kind != "vol" else "volume"
pool_caps["options"][kind] = options
return pool_caps
@@ -5695,7 +5747,7 @@ def pool_define(
keys. The path is the qualified name for iSCSI devices.
Report to `this libvirt page <https://libvirt.org/formatstorage.html#StoragePool>`_
- for more informations on the use of ``part_separator``
+ for more information on the use of ``part_separator``
:param source_dir:
Path to the source directory for pools of type ``dir``, ``netfs`` or ``gluster``.
(Default: ``None``)
@@ -5847,15 +5899,19 @@ def _pool_set_secret(
if secret_type:
# Get the previously defined secret if any
secret = None
- if usage:
- usage_type = (
- libvirt.VIR_SECRET_USAGE_TYPE_CEPH
- if secret_type == "ceph"
- else libvirt.VIR_SECRET_USAGE_TYPE_ISCSI
- )
- secret = conn.secretLookupByUsage(usage_type, usage)
- elif uuid:
- secret = conn.secretLookupByUUIDString(uuid)
+ try:
+ if usage:
+ usage_type = (
+ libvirt.VIR_SECRET_USAGE_TYPE_CEPH
+ if secret_type == "ceph"
+ else libvirt.VIR_SECRET_USAGE_TYPE_ISCSI
+ )
+ secret = conn.secretLookupByUsage(usage_type, usage)
+ elif uuid:
+ secret = conn.secretLookupByUUIDString(uuid)
+ except libvirt.libvirtError as err:
+ # For some reason the secret has been removed. Don't fail since we'll recreate it
+ log.info("Secret not found: %s", err.get_error_message())
# Create secret if needed
if not secret:
@@ -5918,7 +5974,7 @@ def pool_update(
keys. The path is the qualified name for iSCSI devices.
Report to `this libvirt page <https://libvirt.org/formatstorage.html#StoragePool>`_
- for more informations on the use of ``part_separator``
+ for more information on the use of ``part_separator``
:param source_dir:
Path to the source directory for pools of type ``dir``, ``netfs`` or ``gluster``.
(Default: ``None``)
@@ -6107,7 +6163,7 @@ def list_pools(**kwargs):
def pool_info(name=None, **kwargs):
"""
- Return informations on a storage pool provided its name.
+ Return information on a storage pool provided its name.
:param name: libvirt storage pool name
:param connection: libvirt connection URI, overriding defaults
@@ -6283,6 +6339,22 @@ def pool_undefine(name, **kwargs):
conn = __get_conn(**kwargs)
try:
pool = conn.storagePoolLookupByName(name)
+ desc = ElementTree.fromstring(pool.XMLDesc())
+
+ # Is there a secret that we generated and would need to be removed?
+ # Don't remove the other secrets
+ auth_node = desc.find("source/auth")
+ if auth_node is not None:
+ auth_types = {
+ "ceph": libvirt.VIR_SECRET_USAGE_TYPE_CEPH,
+ "iscsi": libvirt.VIR_SECRET_USAGE_TYPE_ISCSI,
+ }
+ secret_type = auth_types[auth_node.get("type")]
+ secret_usage = auth_node.find("secret").get("usage")
+ if secret_type and "pool_{}".format(name) == secret_usage:
+ secret = conn.secretLookupByUsage(secret_type, secret_usage)
+ secret.undefine()
+
return not bool(pool.undefine())
finally:
conn.close()
@@ -6308,22 +6380,6 @@ def pool_delete(name, **kwargs):
conn = __get_conn(**kwargs)
try:
pool = conn.storagePoolLookupByName(name)
- desc = ElementTree.fromstring(pool.XMLDesc())
-
- # Is there a secret that we generated and would need to be removed?
- # Don't remove the other secrets
- auth_node = desc.find("source/auth")
- if auth_node is not None:
- auth_types = {
- "ceph": libvirt.VIR_SECRET_USAGE_TYPE_CEPH,
- "iscsi": libvirt.VIR_SECRET_USAGE_TYPE_ISCSI,
- }
- secret_type = auth_types[auth_node.get("type")]
- secret_usage = auth_node.find("secret").get("usage")
- if secret_type and "pool_{}".format(name) == secret_usage:
- secret = conn.secretLookupByUsage(secret_type, secret_usage)
- secret.undefine()
-
return not bool(pool.delete(libvirt.VIR_STORAGE_POOL_DELETE_NORMAL))
finally:
conn.close()
@@ -6768,7 +6824,7 @@ def _volume_upload(conn, pool, volume, file, offset=0, length=0, sparse=False):
stream.abort()
if ret:
raise CommandExecutionError(
- "Failed to close file: {0}".format(err.strerror)
+ "Failed to close file: {}".format(err.strerror)
)
if stream:
try:
@@ -6776,7 +6832,7 @@ def _volume_upload(conn, pool, volume, file, offset=0, length=0, sparse=False):
except libvirt.libvirtError as err:
if ret:
raise CommandExecutionError(
- "Failed to finish stream: {0}".format(err.get_error_message())
+ "Failed to finish stream: {}".format(err.get_error_message())
)
return ret
diff --git a/salt/states/virt.py b/salt/states/virt.py
index fdef002293..3d99fd53c8 100644
--- a/salt/states/virt.py
+++ b/salt/states/virt.py
@@ -1,4 +1,3 @@
-# -*- coding: utf-8 -*-
"""
Manage virt
===========
@@ -13,9 +12,9 @@ for the generation and signing of certificates for systems running libvirt:
"""
# Import Python libs
-from __future__ import absolute_import, print_function, unicode_literals
import fnmatch
+import logging
import os
# Import Salt libs
@@ -25,9 +24,6 @@ import salt.utils.stringutils
import salt.utils.versions
from salt.exceptions import CommandExecutionError, SaltInvocationError
-# Import 3rd-party libs
-from salt.ext import six
-
try:
import libvirt # pylint: disable=import-error
@@ -38,6 +34,8 @@ except ImportError:
__virtualname__ = "virt"
+log = logging.getLogger(__name__)
+
def __virtual__():
"""
@@ -99,8 +97,8 @@ def keys(name, basepath="/etc/pki", **kwargs):
# rename them to something hopefully unique to avoid
# overriding anything existing
pillar_kwargs = {}
- for key, value in six.iteritems(kwargs):
- pillar_kwargs["ext_pillar_virt.{0}".format(key)] = value
+ for key, value in kwargs.items():
+ pillar_kwargs["ext_pillar_virt.{}".format(key)] = value
pillar = __salt__["pillar.ext"]({"libvirt": "_"}, pillar_kwargs)
paths = {
@@ -112,7 +110,7 @@ def keys(name, basepath="/etc/pki", **kwargs):
}
for key in paths:
- p_key = "libvirt.{0}.pem".format(key)
+ p_key = "libvirt.{}.pem".format(key)
if p_key not in pillar:
continue
if not os.path.exists(os.path.dirname(paths[key])):
@@ -134,7 +132,7 @@ def keys(name, basepath="/etc/pki", **kwargs):
for key in ret["changes"]:
with salt.utils.files.fopen(paths[key], "w+") as fp_:
fp_.write(
- salt.utils.stringutils.to_str(pillar["libvirt.{0}.pem".format(key)])
+ salt.utils.stringutils.to_str(pillar["libvirt.{}.pem".format(key)])
)
ret["comment"] = "Updated libvirt certs and keys"
@@ -176,7 +174,7 @@ def _virt_call(
domain_state = __salt__["virt.vm_state"](targeted_domain)
action_needed = domain_state.get(targeted_domain) != state
if action_needed:
- response = __salt__["virt.{0}".format(function)](
+ response = __salt__["virt.{}".format(function)](
targeted_domain,
connection=connection,
username=username,
@@ -189,9 +187,7 @@ def _virt_call(
else:
noaction_domains.append(targeted_domain)
except libvirt.libvirtError as err:
- ignored_domains.append(
- {"domain": targeted_domain, "issue": six.text_type(err)}
- )
+ ignored_domains.append({"domain": targeted_domain, "issue": str(err)})
if not changed_domains:
ret["result"] = not ignored_domains and bool(targeted_domains)
ret["comment"] = "No changes had happened"
@@ -292,6 +288,7 @@ def defined(
arch=None,
boot=None,
update=True,
+ boot_dev=None,
):
"""
Starts an existing guest, or defines and starts a new VM with specified arguments.
@@ -352,6 +349,14 @@ def defined(
.. deprecated:: sodium
+ :param boot_dev:
+ Space separated list of devices to boot from sorted by decreasing priority.
+ Values can be ``hd``, ``fd``, ``cdrom`` or ``network``.
+
+ By default, the value will ``"hd"``.
+
+ .. versionadded:: Magnesium
+
.. rubric:: Example States
Make sure a virtual machine called ``domain_name`` is defined:
@@ -362,6 +367,7 @@ def defined(
virt.defined:
- cpu: 2
- mem: 2048
+ - boot_dev: network hd
- disk_profile: prod
- disks:
- name: system
@@ -414,17 +420,18 @@ def defined(
password=password,
boot=boot,
test=__opts__["test"],
+ boot_dev=boot_dev,
)
ret["changes"][name] = status
if not status.get("definition"):
- ret["comment"] = "Domain {0} unchanged".format(name)
+ ret["comment"] = "Domain {} unchanged".format(name)
ret["result"] = True
elif status.get("errors"):
ret[
"comment"
- ] = "Domain {0} updated with live update(s) failures".format(name)
+ ] = "Domain {} updated with live update(s) failures".format(name)
else:
- ret["comment"] = "Domain {0} updated".format(name)
+ ret["comment"] = "Domain {} updated".format(name)
else:
if not __opts__["test"]:
__salt__["virt.init"](
@@ -448,12 +455,13 @@ def defined(
password=password,
boot=boot,
start=False,
+ boot_dev=boot_dev,
)
ret["changes"][name] = {"definition": True}
- ret["comment"] = "Domain {0} defined".format(name)
+ ret["comment"] = "Domain {} defined".format(name)
except libvirt.libvirtError as err:
# Something bad happened when defining / updating the VM, report it
- ret["comment"] = six.text_type(err)
+ ret["comment"] = str(err)
ret["result"] = False
return ret
@@ -480,6 +488,7 @@ def running(
os_type=None,
arch=None,
boot=None,
+ boot_dev=None,
):
"""
Starts an existing guest, or defines and starts a new VM with specified arguments.
@@ -591,6 +600,14 @@ def running(
.. versionadded:: 3000
+ :param boot_dev:
+ Space separated list of devices to boot from sorted by decreasing priority.
+ Values can be ``hd``, ``fd``, ``cdrom`` or ``network``.
+
+ By default, the value will ``"hd"``.
+
+ .. versionadded:: Magnesium
+
.. rubric:: Example States
Make sure an already-defined virtual machine called ``domain_name`` is running:
@@ -609,6 +626,7 @@ def running(
- cpu: 2
- mem: 2048
- disk_profile: prod
+ - boot_dev: network hd
- disks:
- name: system
size: 8192
@@ -657,6 +675,7 @@ def running(
arch=arch,
boot=boot,
update=update,
+ boot_dev=boot_dev,
connection=connection,
username=username,
password=password,
@@ -681,11 +700,11 @@ def running(
ret["comment"] = comment
ret["changes"][name]["started"] = True
elif not changed:
- ret["comment"] = "Domain {0} exists and is running".format(name)
+ ret["comment"] = "Domain {} exists and is running".format(name)
except libvirt.libvirtError as err:
# Something bad happened when starting / updating the VM, report it
- ret["comment"] = six.text_type(err)
+ ret["comment"] = str(err)
ret["result"] = False
return ret
@@ -830,7 +849,7 @@ def reverted(
try:
domains = fnmatch.filter(__salt__["virt.list_domains"](), name)
if not domains:
- ret["comment"] = 'No domains found for criteria "{0}"'.format(name)
+ ret["comment"] = 'No domains found for criteria "{}"'.format(name)
else:
ignored_domains = list()
if len(domains) > 1:
@@ -848,9 +867,7 @@ def reverted(
}
except CommandExecutionError as err:
if len(domains) > 1:
- ignored_domains.append(
- {"domain": domain, "issue": six.text_type(err)}
- )
+ ignored_domains.append({"domain": domain, "issue": str(err)})
if len(domains) > 1:
if result:
ret["changes"]["reverted"].append(result)
@@ -860,7 +877,7 @@ def reverted(
ret["result"] = len(domains) != len(ignored_domains)
if ret["result"]:
- ret["comment"] = "Domain{0} has been reverted".format(
+ ret["comment"] = "Domain{} has been reverted".format(
len(domains) > 1 and "s" or ""
)
if ignored_domains:
@@ -868,9 +885,9 @@ def reverted(
if not ret["changes"]["reverted"]:
ret["changes"].pop("reverted")
except libvirt.libvirtError as err:
- ret["comment"] = six.text_type(err)
+ ret["comment"] = str(err)
except CommandExecutionError as err:
- ret["comment"] = six.text_type(err)
+ ret["comment"] = str(err)
return ret
@@ -955,7 +972,7 @@ def network_defined(
name, connection=connection, username=username, password=password
)
if info and info[name]:
- ret["comment"] = "Network {0} exists".format(name)
+ ret["comment"] = "Network {} exists".format(name)
ret["result"] = True
else:
if not __opts__["test"]:
@@ -974,7 +991,7 @@ def network_defined(
password=password,
)
ret["changes"][name] = "Network defined"
- ret["comment"] = "Network {0} defined".format(name)
+ ret["comment"] = "Network {} defined".format(name)
except libvirt.libvirtError as err:
ret["result"] = False
ret["comment"] = err.get_error_message()
@@ -1108,6 +1125,10 @@ def network_running(
return ret
+# Some of the libvirt storage drivers do not support the build action
+BUILDABLE_POOL_TYPES = {"disk", "fs", "netfs", "dir", "logical", "vstorage", "zfs"}
+
+
def pool_defined(
name,
ptype=None,
@@ -1222,25 +1243,35 @@ def pool_defined(
action = ""
if info[name]["state"] != "running":
- if not __opts__["test"]:
- __salt__["virt.pool_build"](
- name,
- connection=connection,
- username=username,
- password=password,
- )
- action = ", built"
+ if ptype in BUILDABLE_POOL_TYPES:
+ if not __opts__["test"]:
+ # Storage pools build like disk or logical will fail if the disk or LV group
+ # was already existing. Since we can't easily figure that out, just log the
+ # possible libvirt error.
+ try:
+ __salt__["virt.pool_build"](
+ name,
+ connection=connection,
+ username=username,
+ password=password,
+ )
+ except libvirt.libvirtError as err:
+ log.warning(
+ "Failed to build libvirt storage pool: %s",
+ err.get_error_message(),
+ )
+ action = ", built"
action = (
"{}, autostart flag changed".format(action)
if needs_autostart
else action
)
- ret["changes"][name] = "Pool updated{0}".format(action)
- ret["comment"] = "Pool {0} updated{1}".format(name, action)
+ ret["changes"][name] = "Pool updated{}".format(action)
+ ret["comment"] = "Pool {} updated{}".format(name, action)
else:
- ret["comment"] = "Pool {0} unchanged".format(name)
+ ret["comment"] = "Pool {} unchanged".format(name)
ret["result"] = True
else:
needs_autostart = autostart
@@ -1265,15 +1296,28 @@ def pool_defined(
password=password,
)
- __salt__["virt.pool_build"](
- name, connection=connection, username=username, password=password
- )
+ if ptype in BUILDABLE_POOL_TYPES:
+ # Storage pools build like disk or logical will fail if the disk or LV group
+ # was already existing. Since we can't easily figure that out, just log the
+ # possible libvirt error.
+ try:
+ __salt__["virt.pool_build"](
+ name,
+ connection=connection,
+ username=username,
+ password=password,
+ )
+ except libvirt.libvirtError as err:
+ log.warning(
+ "Failed to build libvirt storage pool: %s",
+ err.get_error_message(),
+ )
if needs_autostart:
ret["changes"][name] = "Pool defined, marked for autostart"
- ret["comment"] = "Pool {0} defined, marked for autostart".format(name)
+ ret["comment"] = "Pool {} defined, marked for autostart".format(name)
else:
ret["changes"][name] = "Pool defined"
- ret["comment"] = "Pool {0} defined".format(name)
+ ret["comment"] = "Pool {} defined".format(name)
if needs_autostart:
if not __opts__["test"]:
@@ -1374,7 +1418,7 @@ def pool_running(
is_running = info.get(name, {}).get("state", "stopped") == "running"
if is_running:
if updated:
- action = "built, restarted"
+ action = "restarted"
if not __opts__["test"]:
__salt__["virt.pool_stop"](
name,
@@ -1382,13 +1426,16 @@ def pool_running(
username=username,
password=password,
)
- if not __opts__["test"]:
- __salt__["virt.pool_build"](
- name,
- connection=connection,
- username=username,
- password=password,
- )
+ # if the disk or LV group is already existing build will fail (issue #56454)
+ if ptype in BUILDABLE_POOL_TYPES - {"disk", "logical"}:
+ if not __opts__["test"]:
+ __salt__["virt.pool_build"](
+ name,
+ connection=connection,
+ username=username,
+ password=password,
+ )
+ action = "built, {}".format(action)
else:
action = "already running"
result = True
@@ -1402,16 +1449,16 @@ def pool_running(
password=password,
)
- comment = "Pool {0}".format(name)
+ comment = "Pool {}".format(name)
change = "Pool"
if name in ret["changes"]:
- comment = "{0},".format(ret["comment"])
- change = "{0},".format(ret["changes"][name])
+ comment = "{},".format(ret["comment"])
+ change = "{},".format(ret["changes"][name])
if action != "already running":
- ret["changes"][name] = "{0} {1}".format(change, action)
+ ret["changes"][name] = "{} {}".format(change, action)
- ret["comment"] = "{0} {1}".format(comment, action)
+ ret["comment"] = "{} {}".format(comment, action)
ret["result"] = result
except libvirt.libvirtError as err:
@@ -1539,15 +1586,13 @@ def pool_deleted(name, purge=False, connection=None, username=None, password=Non
ret["result"] = None
if unsupported:
- ret[
- "comment"
- ] = 'Unsupported actions for pool of type "{0}": {1}'.format(
+ ret["comment"] = 'Unsupported actions for pool of type "{}": {}'.format(
info[name]["type"], ", ".join(unsupported)
)
else:
- ret["comment"] = "Storage pool could not be found: {0}".format(name)
+ ret["comment"] = "Storage pool could not be found: {}".format(name)
except libvirt.libvirtError as err:
- ret["comment"] = "Failed deleting pool: {0}".format(err.get_error_message())
+ ret["comment"] = "Failed deleting pool: {}".format(err.get_error_message())
ret["result"] = False
return ret
diff --git a/salt/templates/virt/libvirt_domain.jinja b/salt/templates/virt/libvirt_domain.jinja
index aac6283eb0..04a61ffa78 100644
--- a/salt/templates/virt/libvirt_domain.jinja
+++ b/salt/templates/virt/libvirt_domain.jinja
@@ -3,7 +3,7 @@
<vcpu>{{ cpu }}</vcpu>
<memory unit='KiB'>{{ mem }}</memory>
<currentMemory unit='KiB'>{{ mem }}</currentMemory>
- <os>
+ <os {{boot.os_attrib}}>
<type arch='{{ arch }}'>{{ os_type }}</type>
{% if boot %}
{% if 'kernel' in boot %}
diff --git a/salt/utils/data.py b/salt/utils/data.py
index 8f84c2ea42..1c4c22efb3 100644
--- a/salt/utils/data.py
+++ b/salt/utils/data.py
@@ -1,22 +1,16 @@
# -*- coding: utf-8 -*-
-'''
+"""
Functions for manipulating, inspecting, or otherwise working with data types
and data structures.
-'''
+"""
-from __future__ import absolute_import, print_function, unicode_literals
# Import Python libs
import copy
import fnmatch
+import functools
import logging
import re
-import functools
-
-try:
- from collections.abc import Mapping, MutableMapping, Sequence
-except ImportError:
- from collections import Mapping, MutableMapping, Sequence
# Import Salt libs
import salt.utils.dictupdate
@@ -24,13 +18,22 @@ import salt.utils.stringutils
import salt.utils.yaml
from salt.defaults import DEFAULT_TARGET_DELIM
from salt.exceptions import SaltException
-from salt.utils.decorators.jinja import jinja_filter
-from salt.utils.odict import OrderedDict
+from salt.ext import six
# Import 3rd-party libs
-from salt.ext.six.moves import zip # pylint: disable=redefined-builtin
-from salt.ext import six
from salt.ext.six.moves import range # pylint: disable=redefined-builtin
+from salt.ext.six.moves import zip # pylint: disable=redefined-builtin
+from salt.utils.decorators.jinja import jinja_filter
+from salt.utils.odict import OrderedDict
+
+try:
+ from collections.abc import Mapping, MutableMapping, Sequence
+except ImportError:
+ # pylint: disable=no-name-in-module
+ from collections import Mapping, MutableMapping, Sequence
+
+ # pylint: enable=no-name-in-module
+
try:
import jmespath
@@ -41,15 +44,16 @@ log = logging.getLogger(__name__)
class CaseInsensitiveDict(MutableMapping):
- '''
+ """
Inspired by requests' case-insensitive dict implementation, but works with
non-string keys as well.
- '''
+ """
+
def __init__(self, init=None, **kwargs):
- '''
+ """
Force internal dict to be ordered to ensure a consistent iteration
order, irrespective of case.
- '''
+ """
self._data = OrderedDict()
self.update(init or {}, **kwargs)
@@ -67,7 +71,7 @@ class CaseInsensitiveDict(MutableMapping):
return self._data[to_lowercase(key)][1]
def __iter__(self):
- return (item[0] for item in six.itervalues(self._data))
+ return (item[0] for item in self._data.values())
def __eq__(self, rval):
if not isinstance(rval, Mapping):
@@ -76,28 +80,28 @@ class CaseInsensitiveDict(MutableMapping):
return dict(self.items_lower()) == dict(CaseInsensitiveDict(rval).items_lower())
def __repr__(self):
- return repr(dict(six.iteritems(self)))
+ return repr(dict(self.items()))
def items_lower(self):
- '''
+ """
Returns a generator iterating over keys and values, with the keys all
being lowercase.
- '''
- return ((key, val[1]) for key, val in six.iteritems(self._data))
+ """
+ return ((key, val[1]) for key, val in self._data.items())
def copy(self):
- '''
+ """
Returns a copy of the object
- '''
- return CaseInsensitiveDict(six.iteritems(self._data))
+ """
+ return CaseInsensitiveDict(self._data.items())
def __change_case(data, attr, preserve_dict_class=False):
- '''
+ """
Calls data.attr() if data has an attribute/method called attr.
Processes data recursively if data is a Mapping or Sequence.
For Mapping, processes both keys and values.
- '''
+ """
try:
return getattr(data, attr)()
except AttributeError:
@@ -107,73 +111,120 @@ def __change_case(data, attr, preserve_dict_class=False):
if isinstance(data, Mapping):
return (data_type if preserve_dict_class else dict)(
- (__change_case(key, attr, preserve_dict_class),
- __change_case(val, attr, preserve_dict_class))
- for key, val in six.iteritems(data)
+ (
+ __change_case(key, attr, preserve_dict_class),
+ __change_case(val, attr, preserve_dict_class),
+ )
+ for key, val in data.items()
)
if isinstance(data, Sequence):
return data_type(
- __change_case(item, attr, preserve_dict_class) for item in data)
+ __change_case(item, attr, preserve_dict_class) for item in data
+ )
return data
def to_lowercase(data, preserve_dict_class=False):
- '''
+ """
Recursively changes everything in data to lowercase.
- '''
- return __change_case(data, 'lower', preserve_dict_class)
+ """
+ return __change_case(data, "lower", preserve_dict_class)
def to_uppercase(data, preserve_dict_class=False):
- '''
+ """
Recursively changes everything in data to uppercase.
- '''
- return __change_case(data, 'upper', preserve_dict_class)
+ """
+ return __change_case(data, "upper", preserve_dict_class)
-@jinja_filter('compare_dicts')
+@jinja_filter("compare_dicts")
def compare_dicts(old=None, new=None):
- '''
+ """
Compare before and after results from various salt functions, returning a
dict describing the changes that were made.
- '''
+ """
ret = {}
- for key in set((new or {})).union((old or {})):
+ for key in set(new or {}).union(old or {}):
if key not in old:
# New key
- ret[key] = {'old': '',
- 'new': new[key]}
+ ret[key] = {"old": "", "new": new[key]}
elif key not in new:
# Key removed
- ret[key] = {'new': '',
- 'old': old[key]}
+ ret[key] = {"new": "", "old": old[key]}
elif new[key] != old[key]:
# Key modified
- ret[key] = {'old': old[key],
- 'new': new[key]}
+ ret[key] = {"old": old[key], "new": new[key]}
return ret
-@jinja_filter('compare_lists')
+@jinja_filter("compare_lists")
def compare_lists(old=None, new=None):
- '''
+ """
Compare before and after results from various salt functions, returning a
dict describing the changes that were made
- '''
+ """
ret = {}
for item in new:
if item not in old:
- ret.setdefault('new', []).append(item)
+ ret.setdefault("new", []).append(item)
for item in old:
if item not in new:
- ret.setdefault('old', []).append(item)
+ ret.setdefault("old", []).append(item)
return ret
-def decode(data, encoding=None, errors='strict', keep=False,
- normalize=False, preserve_dict_class=False, preserve_tuples=False,
- to_str=False):
- '''
+def _remove_circular_refs(ob, _seen=None):
+ """
+ Generic method to remove circular references from objects.
+ This has been taken from author Martijn Pieters
+ https://stackoverflow.com/questions/44777369/
+ remove-circular-references-in-dicts-lists-tuples/44777477#44777477
+ :param ob: dict, list, typle, set, and frozenset
+ Standard python object
+ :param object _seen:
+ Object that has circular reference
+ :returns:
+ Cleaned Python object
+ :rtype:
+ type(ob)
+ """
+ if _seen is None:
+ _seen = set()
+ if id(ob) in _seen:
+ # Here we caught a circular reference.
+ # Alert user and cleanup to continue.
+ log.exception(
+ "Caught a circular reference in data structure below."
+ "Cleaning and continuing execution.\n%r\n",
+ ob,
+ )
+ return None
+ _seen.add(id(ob))
+ res = ob
+ if isinstance(ob, dict):
+ res = {
+ _remove_circular_refs(k, _seen): _remove_circular_refs(v, _seen)
+ for k, v in ob.items()
+ }
+ elif isinstance(ob, (list, tuple, set, frozenset)):
+ res = type(ob)(_remove_circular_refs(v, _seen) for v in ob)
+ # remove id again; only *nested* references count
+ _seen.remove(id(ob))
+ return res
+
+
+def decode(
+ data,
+ encoding=None,
+ errors="strict",
+ keep=False,
+ normalize=False,
+ preserve_dict_class=False,
+ preserve_tuples=False,
+ to_str=False,
+):
+ """
Generic function which will decode whichever type is passed, if necessary.
Optionally use to_str=True to ensure strings are str types and not unicode
on Python 2.
@@ -199,22 +250,55 @@ def decode(data, encoding=None, errors='strict', keep=False,
two strings above, in which "й" is represented as two code points (i.e. one
for the base character, and one for the breve mark). Normalizing allows for
a more reliable test case.
- '''
- _decode_func = salt.utils.stringutils.to_unicode \
- if not to_str \
+
+ """
+ # Clean data object before decoding to avoid circular references
+ data = _remove_circular_refs(data)
+
+ _decode_func = (
+ salt.utils.stringutils.to_unicode
+ if not to_str
else salt.utils.stringutils.to_str
+ )
if isinstance(data, Mapping):
- return decode_dict(data, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ return decode_dict(
+ data,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
if isinstance(data, list):
- return decode_list(data, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ return decode_list(
+ data,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
if isinstance(data, tuple):
- return decode_tuple(data, encoding, errors, keep, normalize,
- preserve_dict_class, to_str) \
- if preserve_tuples \
- else decode_list(data, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ return (
+ decode_tuple(
+ data, encoding, errors, keep, normalize, preserve_dict_class, to_str
+ )
+ if preserve_tuples
+ else decode_list(
+ data,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
+ )
try:
data = _decode_func(data, encoding, errors, normalize)
except TypeError:
@@ -228,25 +312,48 @@ def decode(data, encoding=None, errors='strict', keep=False,
return data
-def decode_dict(data, encoding=None, errors='strict', keep=False,
- normalize=False, preserve_dict_class=False,
- preserve_tuples=False, to_str=False):
- '''
+def decode_dict(
+ data,
+ encoding=None,
+ errors="strict",
+ keep=False,
+ normalize=False,
+ preserve_dict_class=False,
+ preserve_tuples=False,
+ to_str=False,
+):
+ """
Decode all string values to Unicode. Optionally use to_str=True to ensure
strings are str types and not unicode on Python 2.
- '''
- _decode_func = salt.utils.stringutils.to_unicode \
- if not to_str \
+ """
+ # Clean data object before decoding to avoid circular references
+ data = _remove_circular_refs(data)
+
+ _decode_func = (
+ salt.utils.stringutils.to_unicode
+ if not to_str
else salt.utils.stringutils.to_str
+ )
# Make sure we preserve OrderedDicts
ret = data.__class__() if preserve_dict_class else {}
- for key, value in six.iteritems(data):
+ for key, value in data.items():
if isinstance(key, tuple):
- key = decode_tuple(key, encoding, errors, keep, normalize,
- preserve_dict_class, to_str) \
- if preserve_tuples \
- else decode_list(key, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ key = (
+ decode_tuple(
+ key, encoding, errors, keep, normalize, preserve_dict_class, to_str
+ )
+ if preserve_tuples
+ else decode_list(
+ key,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
+ )
else:
try:
key = _decode_func(key, encoding, errors, normalize)
@@ -260,17 +367,50 @@ def decode_dict(data, encoding=None, errors='strict', keep=False,
raise
if isinstance(value, list):
- value = decode_list(value, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ value = decode_list(
+ value,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
elif isinstance(value, tuple):
- value = decode_tuple(value, encoding, errors, keep, normalize,
- preserve_dict_class, to_str) \
- if preserve_tuples \
- else decode_list(value, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ value = (
+ decode_tuple(
+ value,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ to_str,
+ )
+ if preserve_tuples
+ else decode_list(
+ value,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
+ )
elif isinstance(value, Mapping):
- value = decode_dict(value, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ value = decode_dict(
+ value,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
else:
try:
value = _decode_func(value, encoding, errors, normalize)
@@ -287,30 +427,69 @@ def decode_dict(data, encoding=None, errors='strict', keep=False,
return ret
-def decode_list(data, encoding=None, errors='strict', keep=False,
- normalize=False, preserve_dict_class=False,
- preserve_tuples=False, to_str=False):
- '''
+def decode_list(
+ data,
+ encoding=None,
+ errors="strict",
+ keep=False,
+ normalize=False,
+ preserve_dict_class=False,
+ preserve_tuples=False,
+ to_str=False,
+):
+ """
Decode all string values to Unicode. Optionally use to_str=True to ensure
strings are str types and not unicode on Python 2.
- '''
- _decode_func = salt.utils.stringutils.to_unicode \
- if not to_str \
+ """
+ # Clean data object before decoding to avoid circular references
+ data = _remove_circular_refs(data)
+
+ _decode_func = (
+ salt.utils.stringutils.to_unicode
+ if not to_str
else salt.utils.stringutils.to_str
+ )
ret = []
for item in data:
if isinstance(item, list):
- item = decode_list(item, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ item = decode_list(
+ item,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
elif isinstance(item, tuple):
- item = decode_tuple(item, encoding, errors, keep, normalize,
- preserve_dict_class, to_str) \
- if preserve_tuples \
- else decode_list(item, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ item = (
+ decode_tuple(
+ item, encoding, errors, keep, normalize, preserve_dict_class, to_str
+ )
+ if preserve_tuples
+ else decode_list(
+ item,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
+ )
elif isinstance(item, Mapping):
- item = decode_dict(item, encoding, errors, keep, normalize,
- preserve_dict_class, preserve_tuples, to_str)
+ item = decode_dict(
+ item,
+ encoding,
+ errors,
+ keep,
+ normalize,
+ preserve_dict_class,
+ preserve_tuples,
+ to_str,
+ )
else:
try:
item = _decode_func(item, encoding, errors, normalize)
@@ -327,21 +506,35 @@ def decode_list(data, encoding=None, errors='strict', keep=False,
return ret
-def decode_tuple(data, encoding=None, errors='strict', keep=False,
- normalize=False, preserve_dict_class=False, to_str=False):
- '''
+def decode_tuple(
+ data,
+ encoding=None,
+ errors="strict",
+ keep=False,
+ normalize=False,
+ preserve_dict_class=False,
+ to_str=False,
+):
+ """
Decode all string values to Unicode. Optionally use to_str=True to ensure
strings are str types and not unicode on Python 2.
- '''
+ """
return tuple(
- decode_list(data, encoding, errors, keep, normalize,
- preserve_dict_class, True, to_str)
+ decode_list(
+ data, encoding, errors, keep, normalize, preserve_dict_class, True, to_str
+ )
)
-def encode(data, encoding=None, errors='strict', keep=False,
- preserve_dict_class=False, preserve_tuples=False):
- '''
+def encode(
+ data,
+ encoding=None,
+ errors="strict",
+ keep=False,
+ preserve_dict_class=False,
+ preserve_tuples=False,
+):
+ """
Generic function which will encode whichever type is passed, if necessary
If `strict` is True, and `keep` is False, and we fail to encode, a
@@ -349,18 +542,27 @@ def encode(data, encoding=None, errors='strict', keep=False,
original value to silently be returned in cases where encoding fails. This
can be useful for cases where the data passed to this function is likely to
contain binary blobs.
- '''
+
+ """
+ # Clean data object before encoding to avoid circular references
+ data = _remove_circular_refs(data)
+
if isinstance(data, Mapping):
- return encode_dict(data, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ return encode_dict(
+ data, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
if isinstance(data, list):
- return encode_list(data, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ return encode_list(
+ data, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
if isinstance(data, tuple):
- return encode_tuple(data, encoding, errors, keep, preserve_dict_class) \
- if preserve_tuples \
- else encode_list(data, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ return (
+ encode_tuple(data, encoding, errors, keep, preserve_dict_class)
+ if preserve_tuples
+ else encode_list(
+ data, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
+ )
try:
return salt.utils.stringutils.to_bytes(data, encoding, errors)
except TypeError:
@@ -374,20 +576,31 @@ def encode(data, encoding=None, errors='strict', keep=False,
return data
-@jinja_filter('json_decode_dict') # Remove this for Aluminium
-@jinja_filter('json_encode_dict')
-def encode_dict(data, encoding=None, errors='strict', keep=False,
- preserve_dict_class=False, preserve_tuples=False):
- '''
+@jinja_filter("json_decode_dict") # Remove this for Aluminium
+@jinja_filter("json_encode_dict")
+def encode_dict(
+ data,
+ encoding=None,
+ errors="strict",
+ keep=False,
+ preserve_dict_class=False,
+ preserve_tuples=False,
+):
+ """
Encode all string values to bytes
- '''
+ """
+ # Clean data object before encoding to avoid circular references
+ data = _remove_circular_refs(data)
ret = data.__class__() if preserve_dict_class else {}
- for key, value in six.iteritems(data):
+ for key, value in data.items():
if isinstance(key, tuple):
- key = encode_tuple(key, encoding, errors, keep, preserve_dict_class) \
- if preserve_tuples \
- else encode_list(key, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ key = (
+ encode_tuple(key, encoding, errors, keep, preserve_dict_class)
+ if preserve_tuples
+ else encode_list(
+ key, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
+ )
else:
try:
key = salt.utils.stringutils.to_bytes(key, encoding, errors)
@@ -401,16 +614,21 @@ def encode_dict(data, encoding=None, errors='strict', keep=False,
raise
if isinstance(value, list):
- value = encode_list(value, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ value = encode_list(
+ value, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
elif isinstance(value, tuple):
- value = encode_tuple(value, encoding, errors, keep, preserve_dict_class) \
- if preserve_tuples \
- else encode_list(value, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ value = (
+ encode_tuple(value, encoding, errors, keep, preserve_dict_class)
+ if preserve_tuples
+ else encode_list(
+ value, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
+ )
elif isinstance(value, Mapping):
- value = encode_dict(value, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ value = encode_dict(
+ value, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
else:
try:
value = salt.utils.stringutils.to_bytes(value, encoding, errors)
@@ -427,26 +645,40 @@ def encode_dict(data, encoding=None, errors='strict', keep=False,
return ret
-@jinja_filter('json_decode_list') # Remove this for Aluminium
-@jinja_filter('json_encode_list')
-def encode_list(data, encoding=None, errors='strict', keep=False,
- preserve_dict_class=False, preserve_tuples=False):
- '''
+@jinja_filter("json_decode_list") # Remove this for Aluminium
+@jinja_filter("json_encode_list")
+def encode_list(
+ data,
+ encoding=None,
+ errors="strict",
+ keep=False,
+ preserve_dict_class=False,
+ preserve_tuples=False,
+):
+ """
Encode all string values to bytes
- '''
+ """
+ # Clean data object before encoding to avoid circular references
+ data = _remove_circular_refs(data)
+
ret = []
for item in data:
if isinstance(item, list):
- item = encode_list(item, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ item = encode_list(
+ item, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
elif isinstance(item, tuple):
- item = encode_tuple(item, encoding, errors, keep, preserve_dict_class) \
- if preserve_tuples \
- else encode_list(item, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ item = (
+ encode_tuple(item, encoding, errors, keep, preserve_dict_class)
+ if preserve_tuples
+ else encode_list(
+ item, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
+ )
elif isinstance(item, Mapping):
- item = encode_dict(item, encoding, errors, keep,
- preserve_dict_class, preserve_tuples)
+ item = encode_dict(
+ item, encoding, errors, keep, preserve_dict_class, preserve_tuples
+ )
else:
try:
item = salt.utils.stringutils.to_bytes(item, encoding, errors)
@@ -463,42 +695,37 @@ def encode_list(data, encoding=None, errors='strict', keep=False,
return ret
-def encode_tuple(data, encoding=None, errors='strict', keep=False,
- preserve_dict_class=False):
- '''
+def encode_tuple(
+ data, encoding=None, errors="strict", keep=False, preserve_dict_class=False
+):
+ """
Encode all string values to Unicode
- '''
- return tuple(
- encode_list(data, encoding, errors, keep, preserve_dict_class, True))
+ """
+ return tuple(encode_list(data, encoding, errors, keep, preserve_dict_class, True))
-@jinja_filter('exactly_n_true')
+@jinja_filter("exactly_n_true")
def exactly_n(iterable, amount=1):
- '''
+ """
Tests that exactly N items in an iterable are "truthy" (neither None,
False, nor 0).
- '''
+ """
i = iter(iterable)
return all(any(i) for j in range(amount)) and not any(i)
-@jinja_filter('exactly_one_true')
+@jinja_filter("exactly_one_true")
def exactly_one(iterable):
- '''
+ """
Check if only one item is not None, False, or 0 in an iterable.
- '''
+ """
return exactly_n(iterable)
-def filter_by(lookup_dict,
- lookup,
- traverse,
- merge=None,
- default='default',
- base=None):
- '''
+def filter_by(lookup_dict, lookup, traverse, merge=None, default="default", base=None):
+ """
Common code to filter data structures like grains and pillar
- '''
+ """
ret = None
# Default value would be an empty list if lookup not found
val = traverse_dict_and_list(traverse, lookup, [])
@@ -507,10 +734,8 @@ def filter_by(lookup_dict,
# lookup_dict keys
for each in val if isinstance(val, list) else [val]:
for key in lookup_dict:
- test_key = key if isinstance(key, six.string_types) \
- else six.text_type(key)
- test_each = each if isinstance(each, six.string_types) \
- else six.text_type(each)
+ test_key = key if isinstance(key, str) else str(key)
+ test_each = each if isinstance(each, str) else str(each)
if fnmatch.fnmatchcase(test_each, test_key):
ret = lookup_dict[key]
break
@@ -528,14 +753,13 @@ def filter_by(lookup_dict,
elif isinstance(base_values, Mapping):
if not isinstance(ret, Mapping):
raise SaltException(
- 'filter_by default and look-up values must both be '
- 'dictionaries.')
+ "filter_by default and look-up values must both be " "dictionaries."
+ )
ret = salt.utils.dictupdate.update(copy.deepcopy(base_values), ret)
if merge:
if not isinstance(merge, Mapping):
- raise SaltException(
- 'filter_by merge argument must be a dictionary.')
+ raise SaltException("filter_by merge argument must be a dictionary.")
if ret is None:
ret = merge
@@ -546,12 +770,12 @@ def filter_by(lookup_dict,
def traverse_dict(data, key, default=None, delimiter=DEFAULT_TARGET_DELIM):
- '''
+ """
Traverse a dict using a colon-delimited (or otherwise delimited, using the
'delimiter' param) target string. The target 'foo:bar:baz' will return
data['foo']['bar']['baz'] if this value exists, and will otherwise return
the dict in the default argument.
- '''
+ """
ptr = data
try:
for each in key.split(delimiter):
@@ -562,9 +786,9 @@ def traverse_dict(data, key, default=None, delimiter=DEFAULT_TARGET_DELIM):
return ptr
-@jinja_filter('traverse')
+@jinja_filter("traverse")
def traverse_dict_and_list(data, key, default=None, delimiter=DEFAULT_TARGET_DELIM):
- '''
+ """
Traverse a dict or list using a colon-delimited (or otherwise delimited,
using the 'delimiter' param) target string. The target 'foo:bar:0' will
return data['foo']['bar'][0] if this value exists, and will otherwise
@@ -573,7 +797,7 @@ def traverse_dict_and_list(data, key, default=None, delimiter=DEFAULT_TARGET_DEL
The target 'foo:bar:0' will return data['foo']['bar'][0] if data like
{'foo':{'bar':['baz']}} , if data like {'foo':{'bar':{'0':'baz'}}}
then return data['foo']['bar']['0']
- '''
+ """
ptr = data
for each in key.split(delimiter):
if isinstance(ptr, list):
@@ -605,18 +829,17 @@ def traverse_dict_and_list(data, key, default=None, delimiter=DEFAULT_TARGET_DEL
return ptr
-def subdict_match(data,
- expr,
- delimiter=DEFAULT_TARGET_DELIM,
- regex_match=False,
- exact_match=False):
- '''
+def subdict_match(
+ data, expr, delimiter=DEFAULT_TARGET_DELIM, regex_match=False, exact_match=False
+):
+ """
Check for a match in a dictionary using a delimiter character to denote
levels of subdicts, and also allowing the delimiter character to be
matched. Thus, 'foo:bar:baz' will match data['foo'] == 'bar:baz' and
data['foo']['bar'] == 'baz'. The latter would take priority over the
former, as more deeply-nested matches are tried first.
- '''
+ """
+
def _match(target, pattern, regex_match=False, exact_match=False):
# The reason for using six.text_type first and _then_ using
# to_unicode as a fallback is because we want to eventually have
@@ -628,11 +851,11 @@ def subdict_match(data,
# begin with is that (by design) to_unicode will raise a TypeError if a
# non-string/bytestring/bytearray value is passed.
try:
- target = six.text_type(target).lower()
+ target = str(target).lower()
except UnicodeDecodeError:
target = salt.utils.stringutils.to_unicode(target).lower()
try:
- pattern = six.text_type(pattern).lower()
+ pattern = str(pattern).lower()
except UnicodeDecodeError:
pattern = salt.utils.stringutils.to_unicode(pattern).lower()
@@ -640,48 +863,54 @@ def subdict_match(data,
try:
return re.match(pattern, target)
except Exception: # pylint: disable=broad-except
- log.error('Invalid regex \'%s\' in match', pattern)
+ log.error("Invalid regex '%s' in match", pattern)
return False
else:
- return target == pattern if exact_match \
- else fnmatch.fnmatch(target, pattern)
+ return (
+ target == pattern if exact_match else fnmatch.fnmatch(target, pattern)
+ )
def _dict_match(target, pattern, regex_match=False, exact_match=False):
ret = False
- wildcard = pattern.startswith('*:')
+ wildcard = pattern.startswith("*:")
if wildcard:
pattern = pattern[2:]
- if pattern == '*':
+ if pattern == "*":
# We are just checking that the key exists
ret = True
if not ret and pattern in target:
# We might want to search for a key
ret = True
- if not ret and subdict_match(target,
- pattern,
- regex_match=regex_match,
- exact_match=exact_match):
+ if not ret and subdict_match(
+ target, pattern, regex_match=regex_match, exact_match=exact_match
+ ):
ret = True
if not ret and wildcard:
for key in target:
if isinstance(target[key], dict):
- if _dict_match(target[key],
- pattern,
- regex_match=regex_match,
- exact_match=exact_match):
+ if _dict_match(
+ target[key],
+ pattern,
+ regex_match=regex_match,
+ exact_match=exact_match,
+ ):
return True
elif isinstance(target[key], list):
for item in target[key]:
- if _match(item,
- pattern,
- regex_match=regex_match,
- exact_match=exact_match):
- return True
- elif _match(target[key],
+ if _match(
+ item,
pattern,
regex_match=regex_match,
- exact_match=exact_match):
+ exact_match=exact_match,
+ ):
+ return True
+ elif _match(
+ target[key],
+ pattern,
+ regex_match=regex_match,
+ exact_match=exact_match,
+ ):
return True
return ret
@@ -695,7 +924,7 @@ def subdict_match(data,
# want to use are 3, 2, and 1, in that order.
for idx in range(num_splits - 1, 0, -1):
key = delimiter.join(splits[:idx])
- if key == '*':
+ if key == "*":
# We are matching on everything under the top level, so we need to
# treat the match as the entire data being passed in
matchstr = expr
@@ -703,54 +932,55 @@ def subdict_match(data,
else:
matchstr = delimiter.join(splits[idx:])
match = traverse_dict_and_list(data, key, {}, delimiter=delimiter)
- log.debug("Attempting to match '%s' in '%s' using delimiter '%s'",
- matchstr, key, delimiter)
+ log.debug(
+ "Attempting to match '%s' in '%s' using delimiter '%s'",
+ matchstr,
+ key,
+ delimiter,
+ )
if match == {}:
continue
if isinstance(match, dict):
- if _dict_match(match,
- matchstr,
- regex_match=regex_match,
- exact_match=exact_match):
+ if _dict_match(
+ match, matchstr, regex_match=regex_match, exact_match=exact_match
+ ):
return True
continue
if isinstance(match, (list, tuple)):
# We are matching a single component to a single list member
for member in match:
if isinstance(member, dict):
- if _dict_match(member,
- matchstr,
- regex_match=regex_match,
- exact_match=exact_match):
+ if _dict_match(
+ member,
+ matchstr,
+ regex_match=regex_match,
+ exact_match=exact_match,
+ ):
return True
- if _match(member,
- matchstr,
- regex_match=regex_match,
- exact_match=exact_match):
+ if _match(
+ member, matchstr, regex_match=regex_match, exact_match=exact_match
+ ):
return True
continue
- if _match(match,
- matchstr,
- regex_match=regex_match,
- exact_match=exact_match):
+ if _match(match, matchstr, regex_match=regex_match, exact_match=exact_match):
return True
return False
-@jinja_filter('substring_in_list')
+@jinja_filter("substring_in_list")
def substr_in_list(string_to_search_for, list_to_search):
- '''
+ """
Return a boolean value that indicates whether or not a given
string is present in any of the strings which comprise a list
- '''
+ """
return any(string_to_search_for in s for s in list_to_search)
def is_dictlist(data):
- '''
+ """
Returns True if data is a list of one-element dicts (as found in many SLS
schemas), otherwise returns False
- '''
+ """
if isinstance(data, list):
for element in data:
if isinstance(element, dict):
@@ -762,16 +992,12 @@ def is_dictlist(data):
return False
-def repack_dictlist(data,
- strict=False,
- recurse=False,
- key_cb=None,
- val_cb=None):
- '''
+def repack_dictlist(data, strict=False, recurse=False, key_cb=None, val_cb=None):
+ """
Takes a list of one-element dicts (as found in many SLS schemas) and
repacks into a single dictionary.
- '''
- if isinstance(data, six.string_types):
+ """
+ if isinstance(data, str):
try:
data = salt.utils.yaml.safe_load(data)
except salt.utils.yaml.parser.ParserError as err:
@@ -783,7 +1009,7 @@ def repack_dictlist(data,
if val_cb is None:
val_cb = lambda x, y: y
- valid_non_dict = (six.string_types, six.integer_types, float)
+ valid_non_dict = ((str,), (int,), float)
if isinstance(data, list):
for element in data:
if isinstance(element, valid_non_dict):
@@ -791,21 +1017,21 @@ def repack_dictlist(data,
if isinstance(element, dict):
if len(element) != 1:
log.error(
- 'Invalid input for repack_dictlist: key/value pairs '
- 'must contain only one element (data passed: %s).',
- element
+ "Invalid input for repack_dictlist: key/value pairs "
+ "must contain only one element (data passed: %s).",
+ element,
)
return {}
else:
log.error(
- 'Invalid input for repack_dictlist: element %s is '
- 'not a string/dict/numeric value', element
+ "Invalid input for repack_dictlist: element %s is "
+ "not a string/dict/numeric value",
+ element,
)
return {}
else:
log.error(
- 'Invalid input for repack_dictlist, data passed is not a list '
- '(%s)', data
+ "Invalid input for repack_dictlist, data passed is not a list " "(%s)", data
)
return {}
@@ -821,8 +1047,8 @@ def repack_dictlist(data,
ret[key_cb(key)] = repack_dictlist(val, recurse=recurse)
elif strict:
log.error(
- 'Invalid input for repack_dictlist: nested dictlist '
- 'found, but recurse is set to False'
+ "Invalid input for repack_dictlist: nested dictlist "
+ "found, but recurse is set to False"
)
return {}
else:
@@ -832,17 +1058,17 @@ def repack_dictlist(data,
return ret
-@jinja_filter('is_list')
+@jinja_filter("is_list")
def is_list(value):
- '''
+ """
Check if a variable is a list.
- '''
+ """
return isinstance(value, list)
-@jinja_filter('is_iter')
-def is_iter(thing, ignore=six.string_types):
- '''
+@jinja_filter("is_iter")
+def is_iter(thing, ignore=(str,)):
+ """
Test if an object is iterable, but not a string type.
Test if an object is an iterator or is iterable itself. By default this
@@ -853,7 +1079,7 @@ def is_iter(thing, ignore=six.string_types):
dictionaries or named tuples.
Based on https://bitbucket.org/petershinners/yter
- '''
+ """
if ignore and isinstance(thing, ignore):
return False
try:
@@ -863,9 +1089,9 @@ def is_iter(thing, ignore=six.string_types):
return False
-@jinja_filter('sorted_ignorecase')
+@jinja_filter("sorted_ignorecase")
def sorted_ignorecase(to_sort):
- '''
+ """
Sort a list of strings ignoring case.
>>> L = ['foo', 'Foo', 'bar', 'Bar']
@@ -874,19 +1100,19 @@ def sorted_ignorecase(to_sort):
>>> sorted(L, key=lambda x: x.lower())
['bar', 'Bar', 'foo', 'Foo']
>>>
- '''
+ """
return sorted(to_sort, key=lambda x: x.lower())
def is_true(value=None):
- '''
+ """
Returns a boolean value representing the "truth" of the value passed. The
rules for what is a "True" value are:
1. Integer/float values greater than 0
2. The string values "True" and "true"
3. Any object for which bool(obj) returns True
- '''
+ """
# First, try int/float conversion
try:
value = int(value)
@@ -898,26 +1124,26 @@ def is_true(value=None):
pass
# Now check for truthiness
- if isinstance(value, (six.integer_types, float)):
+ if isinstance(value, ((int,), float)):
return value > 0
- if isinstance(value, six.string_types):
- return six.text_type(value).lower() == 'true'
+ if isinstance(value, str):
+ return str(value).lower() == "true"
return bool(value)
-@jinja_filter('mysql_to_dict')
+@jinja_filter("mysql_to_dict")
def mysql_to_dict(data, key):
- '''
+ """
Convert MySQL-style output to a python dictionary
- '''
+ """
ret = {}
- headers = ['']
+ headers = [""]
for line in data:
if not line:
continue
- if line.startswith('+'):
+ if line.startswith("+"):
continue
- comps = line.split('|')
+ comps = line.split("|")
for comp in range(len(comps)):
comps[comp] = comps[comp].strip()
if len(headers) > 1:
@@ -934,14 +1160,14 @@ def mysql_to_dict(data, key):
def simple_types_filter(data):
- '''
+ """
Convert the data list, dictionary into simple types, i.e., int, float, string,
bool, etc.
- '''
+ """
if data is None:
return data
- simpletypes_keys = (six.string_types, six.text_type, six.integer_types, float, bool)
+ simpletypes_keys = ((str,), str, (int,), float, bool)
simpletypes_values = tuple(list(simpletypes_keys) + [list, tuple])
if isinstance(data, (list, tuple)):
@@ -957,7 +1183,7 @@ def simple_types_filter(data):
if isinstance(data, dict):
simpledict = {}
- for key, value in six.iteritems(data):
+ for key, value in data.items():
if key is not None and not isinstance(key, simpletypes_keys):
key = repr(key)
if value is not None and isinstance(value, (dict, list, tuple)):
@@ -971,23 +1197,23 @@ def simple_types_filter(data):
def stringify(data):
- '''
+ """
Given an iterable, returns its items as a list, with any non-string items
converted to unicode strings.
- '''
+ """
ret = []
for item in data:
if six.PY2 and isinstance(item, str):
item = salt.utils.stringutils.to_unicode(item)
- elif not isinstance(item, six.string_types):
- item = six.text_type(item)
+ elif not isinstance(item, str):
+ item = str(item)
ret.append(item)
return ret
-@jinja_filter('json_query')
+@jinja_filter("json_query")
def json_query(data, expr):
- '''
+ """
Query data using JMESPath language (http://jmespath.org).
Requires the https://github.com/jmespath/jmespath.py library.
@@ -1009,16 +1235,16 @@ def json_query(data, expr):
.. code-block:: text
[80, 25, 22]
- '''
+ """
if jmespath is None:
- err = 'json_query requires jmespath module installed'
+ err = "json_query requires jmespath module installed"
log.error(err)
raise RuntimeError(err)
return jmespath.search(expr, data)
def _is_not_considered_falsey(value, ignore_types=()):
- '''
+ """
Helper function for filter_falsey to determine if something is not to be
considered falsey.
@@ -1026,12 +1252,12 @@ def _is_not_considered_falsey(value, ignore_types=()):
:param list ignore_types: The types to ignore when considering the value.
:return bool
- '''
+ """
return isinstance(value, bool) or type(value) in ignore_types or value
def filter_falsey(data, recurse_depth=None, ignore_types=()):
- '''
+ """
Helper function to remove items from an iterable with falsey value.
Removes ``None``, ``{}`` and ``[]``, 0, '' (but does not remove ``False``).
Recurses into sub-iterables if ``recurse`` is set to ``True``.
@@ -1045,37 +1271,42 @@ def filter_falsey(data, recurse_depth=None, ignore_types=()):
:return type(data)
.. versionadded:: 3000
- '''
+ """
filter_element = (
- functools.partial(filter_falsey,
- recurse_depth=recurse_depth-1,
- ignore_types=ignore_types)
- if recurse_depth else lambda x: x
+ functools.partial(
+ filter_falsey, recurse_depth=recurse_depth - 1, ignore_types=ignore_types
+ )
+ if recurse_depth
+ else lambda x: x
)
if isinstance(data, dict):
- processed_elements = [(key, filter_element(value)) for key, value in six.iteritems(data)]
- return type(data)([
- (key, value)
- for key, value in processed_elements
- if _is_not_considered_falsey(value, ignore_types=ignore_types)
- ])
+ processed_elements = [
+ (key, filter_element(value)) for key, value in data.items()
+ ]
+ return type(data)(
+ [
+ (key, value)
+ for key, value in processed_elements
+ if _is_not_considered_falsey(value, ignore_types=ignore_types)
+ ]
+ )
if is_iter(data):
processed_elements = (filter_element(value) for value in data)
- return type(data)([
- value for value in processed_elements
- if _is_not_considered_falsey(value, ignore_types=ignore_types)
- ])
+ return type(data)(
+ [
+ value
+ for value in processed_elements
+ if _is_not_considered_falsey(value, ignore_types=ignore_types)
+ ]
+ )
return data
def recursive_diff(
- old,
- new,
- ignore_keys=None,
- ignore_order=False,
- ignore_missing_keys=False):
- '''
+ old, new, ignore_keys=None, ignore_order=False, ignore_missing_keys=False
+):
+ """
Performs a recursive diff on mappings and/or iterables and returns the result
in a {'old': values, 'new': values}-style.
Compares dicts and sets unordered (obviously), OrderedDicts and Lists ordered
@@ -1090,12 +1321,16 @@ def recursive_diff(
but missing in ``new``. Only works for regular dicts.
:return dict: Returns dict with keys 'old' and 'new' containing the differences.
- '''
+ """
ignore_keys = ignore_keys or []
res = {}
ret_old = copy.deepcopy(old)
ret_new = copy.deepcopy(new)
- if isinstance(old, OrderedDict) and isinstance(new, OrderedDict) and not ignore_order:
+ if (
+ isinstance(old, OrderedDict)
+ and isinstance(new, OrderedDict)
+ and not ignore_order
+ ):
append_old, append_new = [], []
if len(old) != len(new):
min_length = min(len(old), len(new))
@@ -1114,13 +1349,14 @@ def recursive_diff(
new[key_new],
ignore_keys=ignore_keys,
ignore_order=ignore_order,
- ignore_missing_keys=ignore_missing_keys)
+ ignore_missing_keys=ignore_missing_keys,
+ )
if not res: # Equal
del ret_old[key_old]
del ret_new[key_new]
else:
- ret_old[key_old] = res['old']
- ret_new[key_new] = res['new']
+ ret_old[key_old] = res["old"]
+ ret_new[key_new] = res["new"]
else:
if key_old in ignore_keys:
del ret_old[key_old]
@@ -1131,7 +1367,7 @@ def recursive_diff(
ret_old[item] = old[item]
for item in append_new:
ret_new[item] = new[item]
- ret = {'old': ret_old, 'new': ret_new} if ret_old or ret_new else {}
+ ret = {"old": ret_old, "new": ret_new} if ret_old or ret_new else {}
elif isinstance(old, Mapping) and isinstance(new, Mapping):
# Compare unordered
for key in set(list(old) + list(new)):
@@ -1146,16 +1382,17 @@ def recursive_diff(
new[key],
ignore_keys=ignore_keys,
ignore_order=ignore_order,
- ignore_missing_keys=ignore_missing_keys)
+ ignore_missing_keys=ignore_missing_keys,
+ )
if not res: # Equal
del ret_old[key]
del ret_new[key]
else:
- ret_old[key] = res['old']
- ret_new[key] = res['new']
- ret = {'old': ret_old, 'new': ret_new} if ret_old or ret_new else {}
+ ret_old[key] = res["old"]
+ ret_new[key] = res["new"]
+ ret = {"old": ret_old, "new": ret_new} if ret_old or ret_new else {}
elif isinstance(old, set) and isinstance(new, set):
- ret = {'old': old - new, 'new': new - old} if old - new or new - old else {}
+ ret = {"old": old - new, "new": new - old} if old - new or new - old else {}
elif is_iter(old) and is_iter(new):
# Create a list so we can edit on an index-basis.
list_old = list(ret_old)
@@ -1168,7 +1405,8 @@ def recursive_diff(
item_new,
ignore_keys=ignore_keys,
ignore_order=ignore_order,
- ignore_missing_keys=ignore_missing_keys)
+ ignore_missing_keys=ignore_missing_keys,
+ )
if not res:
list_old.remove(item_old)
list_new.remove(item_new)
@@ -1181,19 +1419,87 @@ def recursive_diff(
iter_new,
ignore_keys=ignore_keys,
ignore_order=ignore_order,
- ignore_missing_keys=ignore_missing_keys)
+ ignore_missing_keys=ignore_missing_keys,
+ )
if not res: # Equal
remove_indices.append(index)
else:
- list_old[index] = res['old']
- list_new[index] = res['new']
+ list_old[index] = res["old"]
+ list_new[index] = res["new"]
for index in reversed(remove_indices):
list_old.pop(index)
list_new.pop(index)
# Instantiate a new whatever-it-was using the list as iterable source.
# This may not be the most optimized in way of speed and memory usage,
# but it will work for all iterable types.
- ret = {'old': type(old)(list_old), 'new': type(new)(list_new)} if list_old or list_new else {}
+ ret = (
+ {"old": type(old)(list_old), "new": type(new)(list_new)}
+ if list_old or list_new
+ else {}
+ )
else:
- ret = {} if old == new else {'old': ret_old, 'new': ret_new}
+ ret = {} if old == new else {"old": ret_old, "new": ret_new}
return ret
+
+
+def get_value(obj, path, default=None):
+ """
+ Get the values for a given path.
+
+ :param path:
+ keys of the properties in the tree separated by colons.
+ One segment in the path can be replaced by an id surrounded by curly braces.
+ This will match all items in a list of dictionary.
+
+ :param default:
+ default value to return when no value is found
+
+ :return:
+ a list of dictionaries, with at least the "value" key providing the actual value.
+ If a placeholder was used, the placeholder id will be a key providing the replacement for it.
+ Note that a value that wasn't found in the tree will be an empty list.
+ This ensures we can make the difference with a None value set by the user.
+ """
+ res = [{"value": obj}]
+ if path:
+ key = path[: path.find(":")] if ":" in path else path
+ next_path = path[path.find(":") + 1 :] if ":" in path else None
+
+ if key.startswith("{") and key.endswith("}"):
+ placeholder_name = key[1:-1]
+ # There will be multiple values to get here
+ items = []
+ if obj is None:
+ return res
+ if isinstance(obj, dict):
+ items = obj.items()
+ elif isinstance(obj, list):
+ items = enumerate(obj)
+
+ def _append_placeholder(value_dict, key):
+ value_dict[placeholder_name] = key
+ return value_dict
+
+ values = [
+ [
+ _append_placeholder(item, key)
+ for item in get_value(val, next_path, default)
+ ]
+ for key, val in items
+ ]
+
+ # flatten the list
+ values = [y for x in values for y in x]
+ return values
+ elif isinstance(obj, dict):
+ if key not in obj.keys():
+ return [{"value": default}]
+
+ value = obj.get(key)
+ if res is not None:
+ res = get_value(value, next_path, default)
+ else:
+ res = [{"value": value}]
+ else:
+ return [{"value": default if obj is not None else obj}]
+ return res
diff --git a/salt/utils/xmlutil.py b/salt/utils/xmlutil.py
index 6d8d74fd3f..2b9c7bf43f 100644
--- a/salt/utils/xmlutil.py
+++ b/salt/utils/xmlutil.py
@@ -1,30 +1,34 @@
-# -*- coding: utf-8 -*-
-'''
+"""
Various XML utilities
-'''
+"""
# Import Python libs
-from __future__ import absolute_import, print_function, unicode_literals
+import re
+import string # pylint: disable=deprecated-module
+from xml.etree import ElementTree
+
+# Import salt libs
+import salt.utils.data
def _conv_name(x):
- '''
+ """
If this XML tree has an xmlns attribute, then etree will add it
to the beginning of the tag, like: "{http://path}tag".
- '''
- if '}' in x:
- comps = x.split('}')
+ """
+ if "}" in x:
+ comps = x.split("}")
name = comps[1]
return name
return x
def _to_dict(xmltree):
- '''
+ """
Converts an XML ElementTree to a dictionary that only contains items.
This is the default behavior in version 2017.7. This will default to prevent
unexpected parsing issues on modules dependant on this.
- '''
+ """
# If this object has no children, the for..loop below will return nothing
# for it, so just return a single dict representing it.
if len(xmltree.getchildren()) < 1:
@@ -51,9 +55,9 @@ def _to_dict(xmltree):
def _to_full_dict(xmltree):
- '''
+ """
Returns the full XML dictionary including attributes.
- '''
+ """
xmldict = {}
for attrName, attrValue in xmltree.attrib.items():
@@ -87,15 +91,234 @@ def _to_full_dict(xmltree):
def to_dict(xmltree, attr=False):
- '''
+ """
Convert an XML tree into a dict. The tree that is passed in must be an
ElementTree object.
Args:
xmltree: An ElementTree object.
attr: If true, attributes will be parsed. If false, they will be ignored.
- '''
+ """
if attr:
return _to_full_dict(xmltree)
else:
return _to_dict(xmltree)
+
+
+def get_xml_node(node, xpath):
+ """
+ Get an XML node using a path (super simple xpath showing complete node ancestry).
+ This also creates the missing nodes.
+
+ The supported XPath can contain elements filtering using [@attr='value'].
+
+ Args:
+ node: an Element object
+ xpath: simple XPath to look for.
+ """
+ if not xpath.startswith("./"):
+ xpath = "./{}".format(xpath)
+ res = node.find(xpath)
+ if res is None:
+ parent_xpath = xpath[: xpath.rfind("/")]
+ parent = node.find(parent_xpath)
+ if parent is None:
+ parent = get_xml_node(node, parent_xpath)
+ segment = xpath[xpath.rfind("/") + 1 :]
+ # We may have [] filter in the segment
+ matcher = re.match(
+ r"""(?P<tag>[^[]+)(?:\[@(?P<attr>\w+)=["'](?P<value>[^"']+)["']])?""",
+ segment,
+ )
+ attrib = (
+ {matcher.group("attr"): matcher.group("value")}
+ if matcher.group("attr") and matcher.group("value")
+ else {}
+ )
+ res = ElementTree.SubElement(parent, matcher.group("tag"), attrib)
+ return res
+
+
+def set_node_text(node, value):
+ """
+ Function to use in the ``set`` value in the :py:func:`change_xml` mapping items to set the text.
+ This is the default.
+
+ :param node: the node to set the text to
+ :param value: the value to set
+ """
+ node.text = str(value)
+
+
+def clean_node(parent_map, node, ignored=None):
+ """
+ Remove the node from its parent if it has no attribute but the ignored ones, no text and no child.
+ Recursively called up to the document root to ensure no empty node is left.
+
+ :param parent_map: dictionary mapping each node to its parent
+ :param node: the node to clean
+ :param ignored: a list of ignored attributes.
+ """
+ has_text = node.text is not None and node.text.strip()
+ parent = parent_map.get(node)
+ if (
+ len(node.attrib.keys() - (ignored or [])) == 0
+ and not list(node)
+ and not has_text
+ ):
+ parent.remove(node)
+ # Clean parent nodes if needed
+ if parent is not None:
+ clean_node(parent_map, parent, ignored)
+
+
+def del_text(parent_map, node):
+ """
+ Function to use as ``del`` value in the :py:func:`change_xml` mapping items to remove the text.
+ This is the default function.
+ Calls :py:func:`clean_node` before returning.
+ """
+ parent = parent_map[node]
+ parent.remove(node)
+ clean_node(parent, node)
+
+
+def del_attribute(attribute, ignored=None):
+ """
+ Helper returning a function to use as ``del`` value in the :py:func:`change_xml` mapping items to
+ remove an attribute.
+
+ The generated function calls :py:func:`clean_node` before returning.
+
+ :param attribute: the name of the attribute to remove
+ :param ignored: the list of attributes to ignore during the cleanup
+
+ :return: the function called by :py:func:`change_xml`.
+ """
+
+ def _do_delete(parent_map, node):
+ if attribute not in node.keys():
+ return
+ node.attrib.pop(attribute)
+ clean_node(parent_map, node, ignored)
+
+ return _do_delete
+
+
+def change_xml(doc, data, mapping):
+ """
+ Change an XML ElementTree document according.
+
+ :param doc: the ElementTree parsed XML document to modify
+ :param data: the dictionary of values used to modify the XML.
+ :param mapping: a list of items describing how to modify the XML document.
+ Each item is a dictionary containing the following keys:
+
+ .. glossary::
+ path
+ the path to the value to set or remove in the ``data`` parameter.
+ See :py:func:`salt.utils.data.get_value <salt.utils.data.get_value>` for the format
+ of the value.
+
+ xpath
+ Simplified XPath expression used to locate the change in the XML tree.
+ See :py:func:`get_xml_node` documentation for details on the supported XPath syntax
+
+ get
+ function gettin the value from the XML.
+ Takes a single parameter for the XML node found by the XPath expression.
+ Default returns the node text value.
+ This may be used to return an attribute or to perform value transformation.
+
+ set
+ function setting the value in the XML.
+ Takes two parameters for the XML node and the value to set.
+ Default is to set the text value.
+
+ del
+ function deleting the value in the XML.
+ Takes two parameters for the parent node and the node matched by the XPath.
+ Default is to remove the text value.
+ More cleanup may be performed, see the :py:func:`clean_node` function for details.
+
+ convert
+ function modifying the user-provided value right before comparing it with the one from the XML.
+ Takes the value as single parameter.
+ Default is to apply no conversion.
+
+ :return: ``True`` if the XML has been modified, ``False`` otherwise.
+ """
+ need_update = False
+ for param in mapping:
+ # Get the value from the function parameter using the path-like description
+ # Using an empty list as a default value will cause values not provided by the user
+ # to be left untouched, as opposed to explicit None unsetting the value
+ values = salt.utils.data.get_value(data, param["path"], [])
+ xpath = param["xpath"]
+ # Prepend the xpath with ./ to handle the root more easily
+ if not xpath.startswith("./"):
+ xpath = "./{}".format(xpath)
+
+ placeholders = [
+ s[1:-1]
+ for s in param["path"].split(":")
+ if s.startswith("{") and s.endswith("}")
+ ]
+
+ ctx = {placeholder: "$$$" for placeholder in placeholders}
+ all_nodes_xpath = string.Template(xpath).substitute(ctx)
+ all_nodes_xpath = re.sub(
+ r"""(?:=['"]\$\$\$["'])|(?:\[\$\$\$\])""", "", all_nodes_xpath
+ )
+
+ # Store the nodes that are not removed for later cleanup
+ kept_nodes = set()
+
+ for value_item in values:
+ new_value = value_item["value"]
+
+ # Only handle simple type values. Use multiple entries or a custom get for dict or lists
+ if isinstance(new_value, list) or isinstance(new_value, dict):
+ continue
+
+ if new_value is not None:
+ ctx = {
+ placeholder: value_item.get(placeholder, "")
+ for placeholder in placeholders
+ }
+ node_xpath = string.Template(xpath).substitute(ctx)
+ node = get_xml_node(doc, node_xpath)
+
+ kept_nodes.add(node)
+
+ get_fn = param.get("get", lambda n: n.text)
+ set_fn = param.get("set", set_node_text)
+ current_value = get_fn(node)
+
+ # Do we need to apply some conversion to the user-provided value?
+ convert_fn = param.get("convert")
+ if convert_fn:
+ new_value = convert_fn(new_value)
+
+ if current_value != new_value:
+ set_fn(node, new_value)
+ need_update = True
+ else:
+ nodes = doc.findall(all_nodes_xpath)
+ del_fn = param.get("del", del_text)
+ parent_map = {c: p for p in doc.iter() for c in p}
+ for node in nodes:
+ del_fn(parent_map, node)
+ need_update = True
+
+ # Clean the left over XML elements if there were placeholders
+ if placeholders and values[0].get("value") != []:
+ all_nodes = set(doc.findall(all_nodes_xpath))
+ to_remove = all_nodes - kept_nodes
+ del_fn = param.get("del", del_text)
+ parent_map = {c: p for p in doc.iter() for c in p}
+ for node in to_remove:
+ del_fn(parent_map, node)
+ need_update = True
+
+ return need_update
diff --git a/tests/pytests/unit/utils/test_data.py b/tests/pytests/unit/utils/test_data.py
new file mode 100644
index 0000000000..b3f0ba04ae
--- /dev/null
+++ b/tests/pytests/unit/utils/test_data.py
@@ -0,0 +1,57 @@
+import salt.utils.data
+
+
+def test_get_value_simple_path():
+ data = {"a": {"b": {"c": "foo"}}}
+ assert [{"value": "foo"}] == salt.utils.data.get_value(data, "a:b:c")
+
+
+def test_get_value_placeholder_dict():
+ data = {"a": {"b": {"name": "foo"}, "c": {"name": "bar"}}}
+ assert [
+ {"value": "foo", "id": "b"},
+ {"value": "bar", "id": "c"},
+ ] == salt.utils.data.get_value(data, "a:{id}:name")
+
+
+def test_get_value_placeholder_list():
+ data = {"a": [{"name": "foo"}, {"name": "bar"}]}
+ assert [
+ {"value": "foo", "id": 0},
+ {"value": "bar", "id": 1},
+ ] == salt.utils.data.get_value(data, "a:{id}:name")
+
+
+def test_get_value_nested_placeholder():
+ data = {
+ "a": {
+ "b": {"b1": {"name": "foo1"}, "b2": {"name": "foo2"}},
+ "c": {"c1": {"name": "bar"}},
+ }
+ }
+ assert [
+ {"value": "foo1", "id": "b", "sub": "b1"},
+ {"value": "foo2", "id": "b", "sub": "b2"},
+ {"value": "bar", "id": "c", "sub": "c1"},
+ ] == salt.utils.data.get_value(data, "a:{id}:{sub}:name")
+
+
+def test_get_value_nested_notfound():
+ data = {"a": {"b": {"c": "foo"}}}
+ assert [{"value": []}] == salt.utils.data.get_value(data, "a:b:d", [])
+
+
+def test_get_value_not_found():
+ assert [{"value": []}] == salt.utils.data.get_value({}, "a", [])
+
+
+def test_get_value_none():
+ assert [{"value": None}] == salt.utils.data.get_value({"a": None}, "a")
+
+
+def test_get_value_simple_type_path():
+ assert [{"value": []}] == salt.utils.data.get_value({"a": 1024}, "a:b", [])
+
+
+def test_get_value_None_path():
+ assert [{"value": None}] == salt.utils.data.get_value({"a": None}, "a:b", [])
diff --git a/tests/pytests/unit/utils/test_xmlutil.py b/tests/pytests/unit/utils/test_xmlutil.py
new file mode 100644
index 0000000000..081cc64193
--- /dev/null
+++ b/tests/pytests/unit/utils/test_xmlutil.py
@@ -0,0 +1,169 @@
+import pytest
+import salt.utils.xmlutil as xml
+from salt._compat import ElementTree as ET
+
+
+@pytest.fixture
+def xml_doc():
+ return ET.fromstring(
+ """
+ <domain>
+ <name>test01</name>
+ <memory unit="MiB">1024</memory>
+ <cpu>
+ <topology sockets="1"/>
+ </cpu>
+ <vcpus>
+ <vcpu enabled="yes" id="1"/>
+ </vcpus>
+ </domain>
+ """
+ )
+
+
+def test_change_xml_text(xml_doc):
+ ret = xml.change_xml(
+ xml_doc, {"name": "test02"}, [{"path": "name", "xpath": "name"}]
+ )
+ assert ret
+ assert "test02" == xml_doc.find("name").text
+
+
+def test_change_xml_text_nochange(xml_doc):
+ ret = xml.change_xml(
+ xml_doc, {"name": "test01"}, [{"path": "name", "xpath": "name"}]
+ )
+ assert not ret
+
+
+def test_change_xml_text_notdefined(xml_doc):
+ ret = xml.change_xml(xml_doc, {}, [{"path": "name", "xpath": "name"}])
+ assert not ret
+
+
+def test_change_xml_text_removed(xml_doc):
+ ret = xml.change_xml(xml_doc, {"name": None}, [{"path": "name", "xpath": "name"}])
+ assert ret
+ assert xml_doc.find("name") is None
+
+
+def test_change_xml_text_add(xml_doc):
+ ret = xml.change_xml(
+ xml_doc,
+ {"cpu": {"vendor": "ACME"}},
+ [{"path": "cpu:vendor", "xpath": "cpu/vendor"}],
+ )
+ assert ret
+ assert "ACME" == xml_doc.find("cpu/vendor").text
+
+
+def test_change_xml_convert(xml_doc):
+ ret = xml.change_xml(
+ xml_doc,
+ {"mem": 2},
+ [{"path": "mem", "xpath": "memory", "convert": lambda v: v * 1024}],
+ )
+ assert ret
+ assert "2048" == xml_doc.find("memory").text
+
+
+def test_change_xml_attr(xml_doc):
+ ret = xml.change_xml(
+ xml_doc,
+ {"cpu": {"topology": {"cores": 4}}},
+ [
+ {
+ "path": "cpu:topology:cores",
+ "xpath": "cpu/topology",
+ "get": lambda n: int(n.get("cores")) if n.get("cores") else None,
+ "set": lambda n, v: n.set("cores", str(v)),
+ "del": xml.del_attribute("cores"),
+ }
+ ],
+ )
+ assert ret
+ assert "4" == xml_doc.find("cpu/topology").get("cores")
+
+
+def test_change_xml_attr_unchanged(xml_doc):
+ ret = xml.change_xml(
+ xml_doc,
+ {"cpu": {"topology": {"sockets": 1}}},
+ [
+ {
+ "path": "cpu:topology:sockets",
+ "xpath": "cpu/topology",
+ "get": lambda n: int(n.get("sockets")) if n.get("sockets") else None,
+ "set": lambda n, v: n.set("sockets", str(v)),
+ "del": xml.del_attribute("sockets"),
+ }
+ ],
+ )
+ assert not ret
+
+
+def test_change_xml_attr_remove(xml_doc):
+ ret = xml.change_xml(
+ xml_doc,
+ {"cpu": {"topology": {"sockets": None}}},
+ [
+ {
+ "path": "cpu:topology:sockets",
+ "xpath": "./cpu/topology",
+ "get": lambda n: int(n.get("sockets")) if n.get("sockets") else None,
+ "set": lambda n, v: n.set("sockets", str(v)),
+ "del": xml.del_attribute("sockets"),
+ }
+ ],
+ )
+ assert ret
+ assert xml_doc.find("cpu") is None
+
+
+def test_change_xml_not_simple_value(xml_doc):
+ ret = xml.change_xml(
+ xml_doc,
+ {"cpu": {"topology": {"sockets": None}}},
+ [{"path": "cpu", "xpath": "vcpu", "get": lambda n: int(n.text)}],
+ )
+ assert not ret
+
+
+def test_change_xml_template(xml_doc):
+ ret = xml.change_xml(
+ xml_doc,
+ {"cpu": {"vcpus": {2: {"enabled": True}, 4: {"enabled": False}}}},
+ [
+ {
+ "path": "cpu:vcpus:{id}:enabled",
+ "xpath": "vcpus/vcpu[@id='$id']",
+ "convert": lambda v: "yes" if v else "no",
+ "get": lambda n: n.get("enabled"),
+ "set": lambda n, v: n.set("enabled", v),
+ "del": xml.del_attribute("enabled", ["id"]),
+ },
+ ],
+ )
+ assert ret
+ assert xml_doc.find("vcpus/vcpu[@id='1']") is None
+ assert "yes" == xml_doc.find("vcpus/vcpu[@id='2']").get("enabled")
+ assert "no" == xml_doc.find("vcpus/vcpu[@id='4']").get("enabled")
+
+
+def test_change_xml_template_remove(xml_doc):
+ ret = xml.change_xml(
+ xml_doc,
+ {"cpu": {"vcpus": None}},
+ [
+ {
+ "path": "cpu:vcpus:{id}:enabled",
+ "xpath": "vcpus/vcpu[@id='$id']",
+ "convert": lambda v: "yes" if v else "no",
+ "get": lambda n: n.get("enabled"),
+ "set": lambda n, v: n.set("enabled", v),
+ "del": xml.del_attribute("enabled", ["id"]),
+ },
+ ],
+ )
+ assert ret
+ assert xml_doc.find("vcpus") is None
diff --git a/tests/unit/modules/test_virt.py b/tests/unit/modules/test_virt.py
index d3988464f6..5ec8de77e7 100644
--- a/tests/unit/modules/test_virt.py
+++ b/tests/unit/modules/test_virt.py
@@ -1,4 +1,3 @@
-# -*- coding: utf-8 -*-
"""
virt execution module unit tests
"""
@@ -6,7 +5,6 @@ virt execution module unit tests
# pylint: disable=3rd-party-module-not-gated
# Import python libs
-from __future__ import absolute_import, print_function, unicode_literals
import datetime
import os
@@ -23,9 +21,6 @@ import salt.utils.yaml
from salt._compat import ElementTree as ET
from salt.exceptions import CommandExecutionError, SaltInvocationError
-# Import third party libs
-from salt.ext import six
-
# pylint: disable=import-error
from salt.ext.six.moves import range # pylint: disable=redefined-builtin
@@ -136,7 +131,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
"model": "virtio",
"filename": "myvm_system.qcow2",
"image": "/path/to/image",
- "source_file": "{0}{1}myvm_system.qcow2".format(root_dir, os.sep),
+ "source_file": "{}{}myvm_system.qcow2".format(root_dir, os.sep),
},
{
"name": "data",
@@ -145,7 +140,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
"format": "raw",
"model": "virtio",
"filename": "myvm_data.raw",
- "source_file": "{0}{1}myvm_data.raw".format(root_dir, os.sep),
+ "source_file": "{}{}myvm_data.raw".format(root_dir, os.sep),
},
],
disks,
@@ -582,8 +577,8 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
self.assertIsNone(root.get("type"))
self.assertEqual(root.find("name").text, "vmname/system.vmdk")
self.assertEqual(root.find("capacity").attrib["unit"], "KiB")
- self.assertEqual(root.find("capacity").text, six.text_type(8192 * 1024))
- self.assertEqual(root.find("allocation").text, six.text_type(0))
+ self.assertEqual(root.find("capacity").text, str(8192 * 1024))
+ self.assertEqual(root.find("allocation").text, str(0))
self.assertEqual(root.find("target/format").get("type"), "vmdk")
self.assertIsNone(root.find("target/permissions"))
self.assertIsNone(root.find("target/nocow"))
@@ -615,9 +610,9 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
self.assertIsNone(root.find("target/path"))
self.assertEqual(root.find("target/format").get("type"), "qcow2")
self.assertEqual(root.find("capacity").attrib["unit"], "KiB")
- self.assertEqual(root.find("capacity").text, six.text_type(8192 * 1024))
+ self.assertEqual(root.find("capacity").text, str(8192 * 1024))
self.assertEqual(root.find("capacity").attrib["unit"], "KiB")
- self.assertEqual(root.find("allocation").text, six.text_type(4096 * 1024))
+ self.assertEqual(root.find("allocation").text, str(4096 * 1024))
self.assertEqual(root.find("target/permissions/mode").text, "0775")
self.assertEqual(root.find("target/permissions/owner").text, "123")
self.assertEqual(root.find("target/permissions/group").text, "456")
@@ -638,7 +633,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
root = ET.fromstring(xml_data)
self.assertEqual(root.attrib["type"], "kvm")
self.assertEqual(root.find("vcpu").text, "1")
- self.assertEqual(root.find("memory").text, six.text_type(512 * 1024))
+ self.assertEqual(root.find("memory").text, str(512 * 1024))
self.assertEqual(root.find("memory").attrib["unit"], "KiB")
disks = root.findall(".//disk")
@@ -671,7 +666,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
root = ET.fromstring(xml_data)
self.assertEqual(root.attrib["type"], "vmware")
self.assertEqual(root.find("vcpu").text, "1")
- self.assertEqual(root.find("memory").text, six.text_type(512 * 1024))
+ self.assertEqual(root.find("memory").text, str(512 * 1024))
self.assertEqual(root.find("memory").attrib["unit"], "KiB")
disks = root.findall(".//disk")
@@ -714,7 +709,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
root = ET.fromstring(xml_data)
self.assertEqual(root.attrib["type"], "xen")
self.assertEqual(root.find("vcpu").text, "1")
- self.assertEqual(root.find("memory").text, six.text_type(512 * 1024))
+ self.assertEqual(root.find("memory").text, str(512 * 1024))
self.assertEqual(root.find("memory").attrib["unit"], "KiB")
self.assertEqual(
root.find(".//kernel").text, "/usr/lib/grub2/x86_64-xen/grub.xen"
@@ -768,7 +763,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
root = ET.fromstring(xml_data)
self.assertEqual(root.attrib["type"], "vmware")
self.assertEqual(root.find("vcpu").text, "1")
- self.assertEqual(root.find("memory").text, six.text_type(512 * 1024))
+ self.assertEqual(root.find("memory").text, str(512 * 1024))
self.assertEqual(root.find("memory").attrib["unit"], "KiB")
self.assertTrue(len(root.findall(".//disk")) == 2)
self.assertTrue(len(root.findall(".//interface")) == 2)
@@ -801,7 +796,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
root = ET.fromstring(xml_data)
self.assertEqual(root.attrib["type"], "kvm")
self.assertEqual(root.find("vcpu").text, "1")
- self.assertEqual(root.find("memory").text, six.text_type(512 * 1024))
+ self.assertEqual(root.find("memory").text, str(512 * 1024))
self.assertEqual(root.find("memory").attrib["unit"], "KiB")
disks = root.findall(".//disk")
self.assertTrue(len(disks) == 2)
@@ -1635,7 +1630,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
self.assertIsNone(definition.find("./devices/disk[2]/source"))
self.assertEqual(
mock_run.call_args[0][0],
- 'qemu-img create -f qcow2 "{0}" 10240M'.format(expected_disk_path),
+ 'qemu-img create -f qcow2 "{}" 10240M'.format(expected_disk_path),
)
self.assertEqual(mock_chmod.call_args[0][0], expected_disk_path)
@@ -1729,11 +1724,12 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
<vcpu placement='auto'>1</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type>
+ <boot dev="hd"/>
</os>
<devices>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
- <source file='{0}{1}my_vm_system.qcow2'/>
+ <source file='{}{}my_vm_system.qcow2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
@@ -1850,17 +1846,36 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
"cmdline": "console=ttyS0 ks=http://example.com/f8-i386/os/",
}
- boot_uefi = {
- "loader": "/usr/share/OVMF/OVMF_CODE.fd",
- "nvram": "/usr/share/OVMF/OVMF_VARS.ms.fd",
- }
+ # Update boot devices case
+ define_mock.reset_mock()
+ self.assertEqual(
+ {
+ "definition": True,
+ "disk": {"attached": [], "detached": [], "updated": []},
+ "interface": {"attached": [], "detached": []},
+ },
+ virt.update("my_vm", boot_dev="cdrom network hd"),
+ )
+ setxml = ET.fromstring(define_mock.call_args[0][0])
+ self.assertEqual(
+ ["cdrom", "network", "hd"],
+ [node.get("dev") for node in setxml.findall("os/boot")],
+ )
- invalid_boot = {
- "loader": "/usr/share/OVMF/OVMF_CODE.fd",
- "initrd": "/root/f8-i386-initrd",
- }
+ # Update unchanged boot devices case
+ define_mock.reset_mock()
+ self.assertEqual(
+ {
+ "definition": False,
+ "disk": {"attached": [], "detached": [], "updated": []},
+ "interface": {"attached": [], "detached": []},
+ },
+ virt.update("my_vm", boot_dev="hd"),
+ )
+ define_mock.assert_not_called()
# Update with boot parameter case
+ define_mock.reset_mock()
self.assertEqual(
{
"definition": True,
@@ -1884,6 +1899,11 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
"console=ttyS0 ks=http://example.com/f8-i386/os/",
)
+ boot_uefi = {
+ "loader": "/usr/share/OVMF/OVMF_CODE.fd",
+ "nvram": "/usr/share/OVMF/OVMF_VARS.ms.fd",
+ }
+
self.assertEqual(
{
"definition": True,
@@ -1903,9 +1923,28 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
"/usr/share/OVMF/OVMF_VARS.ms.fd",
)
+ self.assertEqual(
+ {
+ "definition": True,
+ "disk": {"attached": [], "detached": [], "updated": []},
+ "interface": {"attached": [], "detached": []},
+ },
+ virt.update("my_vm", boot={"efi": True}),
+ )
+ setxml = ET.fromstring(define_mock.call_args[0][0])
+ self.assertEqual(setxml.find("os").attrib.get("firmware"), "efi")
+
+ invalid_boot = {
+ "loader": "/usr/share/OVMF/OVMF_CODE.fd",
+ "initrd": "/root/f8-i386-initrd",
+ }
+
with self.assertRaises(SaltInvocationError):
virt.update("my_vm", boot=invalid_boot)
+ with self.assertRaises(SaltInvocationError):
+ virt.update("my_vm", boot={"efi": "Not a boolean value"})
+
# Update memory case
setmem_mock = MagicMock(return_value=0)
domain_mock.setMemoryFlags = setmem_mock
@@ -1955,7 +1994,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
) # pylint: disable=no-member
self.assertEqual(
mock_run.call_args[0][0],
- 'qemu-img create -f qcow2 "{0}" 2048M'.format(added_disk_path),
+ 'qemu-img create -f qcow2 "{}" 2048M'.format(added_disk_path),
)
self.assertEqual(mock_chmod.call_args[0][0], added_disk_path)
self.assertListEqual(
@@ -2397,6 +2436,43 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
],
)
+ def test_update_xen_boot_params(self):
+ """
+ Test virt.update() a Xen definition no boot parameter.
+ """
+ root_dir = os.path.join(salt.syspaths.ROOT_DIR, "srv", "salt-images")
+ xml_boot = """
+ <domain type='xen' id='8'>
+ <name>vm</name>
+ <memory unit='KiB'>1048576</memory>
+ <currentMemory unit='KiB'>1048576</currentMemory>
+ <vcpu placement='auto'>1</vcpu>
+ <os>
+ <type arch='x86_64' machine='xenfv'>hvm</type>
+ <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader>
+ </os>
+ </domain>
+ """
+ domain_mock_boot = self.set_mock_vm("vm", xml_boot)
+ domain_mock_boot.OSType = MagicMock(return_value="hvm")
+ define_mock_boot = MagicMock(return_value=True)
+ define_mock_boot.setVcpusFlags = MagicMock(return_value=0)
+ self.mock_conn.defineXML = define_mock_boot
+ self.assertEqual(
+ {
+ "cpu": False,
+ "definition": True,
+ "disk": {"attached": [], "detached": [], "updated": []},
+ "interface": {"attached": [], "detached": []},
+ },
+ virt.update("vm", cpu=2),
+ )
+ setxml = ET.fromstring(define_mock_boot.call_args[0][0])
+ self.assertEqual(setxml.find("os").find("loader").attrib.get("type"), "rom")
+ self.assertEqual(
+ setxml.find("os").find("loader").text, "/usr/lib/xen/boot/hvmloader"
+ )
+
def test_update_existing_boot_params(self):
"""
Test virt.update() with existing boot parameters.
@@ -2537,6 +2613,18 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
self.assertEqual(setxml.find("os").find("initrd"), None)
self.assertEqual(setxml.find("os").find("cmdline"), None)
+ self.assertEqual(
+ {
+ "definition": True,
+ "disk": {"attached": [], "detached": [], "updated": []},
+ "interface": {"attached": [], "detached": []},
+ },
+ virt.update("vm_with_boot_param", boot={"efi": False}),
+ )
+ setxml = ET.fromstring(define_mock_boot.call_args[0][0])
+ self.assertEqual(setxml.find("os").find("nvram"), None)
+ self.assertEqual(setxml.find("os").find("loader"), None)
+
self.assertEqual(
{
"definition": True,
@@ -2582,7 +2670,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
salt.modules.config.__opts__, mock_config # pylint: disable=no-member
):
- for name in six.iterkeys(mock_config["virt"]["nic"]):
+ for name in mock_config["virt"]["nic"].keys():
profile = salt.modules.virt._nic_profile(name, "kvm")
self.assertEqual(len(profile), 2)
@@ -3592,8 +3680,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
"44454c4c-3400-105a-8033-b3c04f4b344a", caps["host"]["host"]["uuid"]
)
self.assertEqual(
- set(["qemu", "kvm"]),
- set([domainCaps["domain"] for domainCaps in caps["domains"]]),
+ {"qemu", "kvm"}, {domainCaps["domain"] for domainCaps in caps["domains"]},
)
def test_network_tag(self):
@@ -3694,9 +3781,9 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
for i in range(2):
net_mock = MagicMock()
- net_mock.name.return_value = "net{0}".format(i)
+ net_mock.name.return_value = "net{}".format(i)
net_mock.UUIDString.return_value = "some-uuid"
- net_mock.bridgeName.return_value = "br{0}".format(i)
+ net_mock.bridgeName.return_value = "br{}".format(i)
net_mock.autostart.return_value = True
net_mock.isActive.return_value = False
net_mock.isPersistent.return_value = True
@@ -4156,8 +4243,8 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
pool_mocks = []
for i in range(2):
pool_mock = MagicMock()
- pool_mock.name.return_value = "pool{0}".format(i)
- pool_mock.UUIDString.return_value = "some-uuid-{0}".format(i)
+ pool_mock.name.return_value = "pool{}".format(i)
+ pool_mock.UUIDString.return_value = "some-uuid-{}".format(i)
pool_mock.info.return_value = [0, 1234, 5678, 123]
pool_mock.autostart.return_value = True
pool_mock.isPersistent.return_value = True
@@ -4257,7 +4344,6 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
"""
mock_pool = MagicMock()
mock_pool.delete = MagicMock(return_value=0)
- mock_pool.XMLDesc.return_value = "<pool type='dir'/>"
self.mock_conn.storagePoolLookupByName = MagicMock(return_value=mock_pool)
res = virt.pool_delete("test-pool")
@@ -4271,12 +4357,12 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
self.mock_libvirt.VIR_STORAGE_POOL_DELETE_NORMAL
)
- def test_pool_delete_secret(self):
+ def test_pool_undefine_secret(self):
"""
- Test virt.pool_delete function where the pool has a secret
+ Test virt.pool_undefine function where the pool has a secret
"""
mock_pool = MagicMock()
- mock_pool.delete = MagicMock(return_value=0)
+ mock_pool.undefine = MagicMock(return_value=0)
mock_pool.XMLDesc.return_value = """
<pool type='rbd'>
<name>test-ses</name>
@@ -4293,16 +4379,11 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
mock_undefine = MagicMock(return_value=0)
self.mock_conn.secretLookupByUsage.return_value.undefine = mock_undefine
- res = virt.pool_delete("test-ses")
+ res = virt.pool_undefine("test-ses")
self.assertTrue(res)
self.mock_conn.storagePoolLookupByName.assert_called_once_with("test-ses")
-
- # Shouldn't be called with another parameter so far since those are not implemented
- # and thus throwing exceptions.
- mock_pool.delete.assert_called_once_with(
- self.mock_libvirt.VIR_STORAGE_POOL_DELETE_NORMAL
- )
+ mock_pool.undefine.assert_called_once_with()
self.mock_conn.secretLookupByUsage.assert_called_once_with(
self.mock_libvirt.VIR_SECRET_USAGE_TYPE_CEPH, "pool_test-ses"
@@ -4571,24 +4652,6 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
</source>
</pool>"""
- expected_xml = (
- '<pool type="rbd">'
- "<name>default</name>"
- "<uuid>20fbe05c-ab40-418a-9afa-136d512f0ede</uuid>"
- '<capacity unit="bytes">1999421108224</capacity>'
- '<allocation unit="bytes">713207042048</allocation>'
- '<available unit="bytes">1286214066176</available>'
- "<source>"
- '<host name="ses4.tf.local" />'
- '<host name="ses5.tf.local" />'
- '<auth type="ceph" username="libvirt">'
- '<secret uuid="14e9a0f1-8fbf-4097-b816-5b094c182212" />'
- "</auth>"
- "<name>iscsi-images</name>"
- "</source>"
- "</pool>"
- )
-
mock_secret = MagicMock()
self.mock_conn.secretLookupByUUIDString = MagicMock(return_value=mock_secret)
@@ -4609,6 +4672,23 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
self.mock_conn.storagePoolDefineXML.assert_not_called()
mock_secret.setValue.assert_called_once_with(b"secret")
+ # Case where the secret can't be found
+ self.mock_conn.secretLookupByUUIDString = MagicMock(
+ side_effect=self.mock_libvirt.libvirtError("secret not found")
+ )
+ self.assertFalse(
+ virt.pool_update(
+ "default",
+ "rbd",
+ source_name="iscsi-images",
+ source_hosts=["ses4.tf.local", "ses5.tf.local"],
+ source_auth={"username": "libvirt", "password": "c2VjcmV0"},
+ )
+ )
+ self.mock_conn.storagePoolDefineXML.assert_not_called()
+ self.mock_conn.secretDefineXML.assert_called_once()
+ mock_secret.setValue.assert_called_once_with(b"secret")
+
def test_pool_update_password_create(self):
"""
Test the pool_update function, where the password only is changed
@@ -4695,11 +4775,11 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
for idx, disk in enumerate(vms_disks):
vm = MagicMock()
# pylint: disable=no-member
- vm.name.return_value = "vm{0}".format(idx)
+ vm.name.return_value = "vm{}".format(idx)
vm.XMLDesc.return_value = """
<domain type='kvm' id='1'>
- <name>vm{0}</name>
- <devices>{1}</devices>
+ <name>vm{}</name>
+ <devices>{}</devices>
</domain>
""".format(
idx, disk
@@ -4760,7 +4840,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
# pylint: disable=no-member
mock_volume.name.return_value = vol_data["name"]
mock_volume.key.return_value = vol_data["key"]
- mock_volume.path.return_value = "/path/to/{0}.qcow2".format(
+ mock_volume.path.return_value = "/path/to/{}.qcow2".format(
vol_data["name"]
)
if vol_data["info"]:
@@ -4769,7 +4849,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
"""
<backingStore>
<format type="qcow2"/>
- <path>{0}</path>
+ <path>{}</path>
</backingStore>
""".format(
vol_data["backingStore"]
@@ -5234,7 +5314,7 @@ class VirtTestCase(TestCase, LoaderModuleMockMixin):
def create_mock_vm(idx):
mock_vm = MagicMock()
- mock_vm.name.return_value = "vm{0}".format(idx)
+ mock_vm.name.return_value = "vm{}".format(idx)
return mock_vm
mock_vms = [create_mock_vm(idx) for idx in range(3)]
diff --git a/tests/unit/states/test_virt.py b/tests/unit/states/test_virt.py
index c76f8a5fc0..f03159334b 100644
--- a/tests/unit/states/test_virt.py
+++ b/tests/unit/states/test_virt.py
@@ -1,9 +1,7 @@
-# -*- coding: utf-8 -*-
"""
:codeauthor: Jayesh Kariya <jayeshk@saltstack.com>
"""
# Import Python libs
-from __future__ import absolute_import, print_function, unicode_literals
import shutil
import tempfile
@@ -14,7 +12,6 @@ import salt.utils.files
from salt.exceptions import CommandExecutionError, SaltInvocationError
# Import 3rd-party libs
-from salt.ext import six
from tests.support.mixins import LoaderModuleMockMixin
from tests.support.mock import MagicMock, mock_open, patch
@@ -37,7 +34,7 @@ class LibvirtMock(MagicMock): # pylint: disable=too-many-ancestors
"""
Fake function return error message
"""
- return six.text_type(self)
+ return str(self)
class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
@@ -341,6 +338,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
"myvm",
cpu=2,
mem=2048,
+ boot_dev="cdrom hd",
os_type="linux",
arch="i686",
vm_type="qemu",
@@ -363,6 +361,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
"myvm",
cpu=2,
mem=2048,
+ boot_dev="cdrom hd",
os_type="linux",
arch="i686",
disk="prod",
@@ -471,10 +470,13 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
"comment": "Domain myvm updated with live update(s) failures",
}
)
- self.assertDictEqual(virt.defined("myvm", cpu=2), ret)
+ self.assertDictEqual(
+ virt.defined("myvm", cpu=2, boot_dev="cdrom hd"), ret
+ )
update_mock.assert_called_with(
"myvm",
cpu=2,
+ boot_dev="cdrom hd",
mem=None,
disk_profile=None,
disks=None,
@@ -598,6 +600,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
password=None,
boot=None,
test=True,
+ boot_dev=None,
)
# No changes case
@@ -632,6 +635,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
password=None,
boot=None,
test=True,
+ boot_dev=None,
)
def test_running(self):
@@ -708,6 +712,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
install=True,
pub_key=None,
priv_key=None,
+ boot_dev=None,
connection=None,
username=None,
password=None,
@@ -769,6 +774,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
install=False,
pub_key="/path/to/key.pub",
priv_key="/path/to/key",
+ boot_dev="network hd",
connection="someconnection",
username="libvirtuser",
password="supersecret",
@@ -793,6 +799,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
start=False,
pub_key="/path/to/key.pub",
priv_key="/path/to/key",
+ boot_dev="network hd",
connection="someconnection",
username="libvirtuser",
password="supersecret",
@@ -937,6 +944,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
password=None,
boot=None,
test=False,
+ boot_dev=None,
)
# Failed definition update case
@@ -1055,6 +1063,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
password=None,
boot=None,
test=True,
+ boot_dev=None,
)
start_mock.assert_not_called()
@@ -1091,6 +1100,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
password=None,
boot=None,
test=True,
+ boot_dev=None,
)
def test_stopped(self):
@@ -1978,6 +1988,72 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
password="secret",
)
+ # Define a pool that doesn't handle build
+ for mock in mocks:
+ mocks[mock].reset_mock()
+ with patch.dict(
+ virt.__salt__,
+ { # pylint: disable=no-member
+ "virt.pool_info": MagicMock(
+ side_effect=[
+ {},
+ {"mypool": {"state": "stopped", "autostart": True}},
+ ]
+ ),
+ "virt.pool_define": mocks["define"],
+ "virt.pool_build": mocks["build"],
+ "virt.pool_set_autostart": mocks["autostart"],
+ },
+ ):
+ ret.update(
+ {
+ "changes": {"mypool": "Pool defined, marked for autostart"},
+ "comment": "Pool mypool defined, marked for autostart",
+ }
+ )
+ self.assertDictEqual(
+ virt.pool_defined(
+ "mypool",
+ ptype="rbd",
+ source={
+ "name": "libvirt-pool",
+ "hosts": ["ses2.tf.local", "ses3.tf.local"],
+ "auth": {
+ "username": "libvirt",
+ "password": "AQAz+PRdtquBBRAASMv7nlMZYfxIyLw3St65Xw==",
+ },
+ },
+ autostart=True,
+ ),
+ ret,
+ )
+ mocks["define"].assert_called_with(
+ "mypool",
+ ptype="rbd",
+ target=None,
+ permissions=None,
+ source_devices=None,
+ source_dir=None,
+ source_adapter=None,
+ source_hosts=["ses2.tf.local", "ses3.tf.local"],
+ source_auth={
+ "username": "libvirt",
+ "password": "AQAz+PRdtquBBRAASMv7nlMZYfxIyLw3St65Xw==",
+ },
+ source_name="libvirt-pool",
+ source_format=None,
+ source_initiator=None,
+ start=False,
+ transient=False,
+ connection=None,
+ username=None,
+ password=None,
+ )
+ mocks["autostart"].assert_called_with(
+ "mypool", state="on", connection=None, username=None, password=None,
+ )
+ mocks["build"].assert_not_called()
+
mocks["update"] = MagicMock(return_value=False)
for mock in mocks:
mocks[mock].reset_mock()
@@ -2027,6 +2103,9 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
for mock in mocks:
mocks[mock].reset_mock()
mocks["update"] = MagicMock(return_value=True)
+ mocks["build"] = MagicMock(
+ side_effect=self.mock_libvirt.libvirtError("Existing VG")
+ )
with patch.dict(
virt.__salt__,
{ # pylint: disable=no-member
@@ -2130,6 +2209,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
),
ret,
)
+ mocks["build"].assert_not_called()
mocks["update"].assert_called_with(
"mypool",
ptype="logical",
@@ -2477,8 +2557,8 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
):
ret.update(
{
- "changes": {"mypool": "Pool updated, built, restarted"},
- "comment": "Pool mypool updated, built, restarted",
+ "changes": {"mypool": "Pool updated, restarted"},
+ "comment": "Pool mypool updated, restarted",
"result": True,
}
)
@@ -2504,9 +2584,7 @@ class LibvirtTestCase(TestCase, LoaderModuleMockMixin):
mocks["start"].assert_called_with(
"mypool", connection=None, username=None, password=None
)
- mocks["build"].assert_called_with(
- "mypool", connection=None, username=None, password=None
- )
+ mocks["build"].assert_not_called()
mocks["update"].assert_called_with(
"mypool",
ptype="logical",
diff --git a/tests/unit/utils/test_data.py b/tests/unit/utils/test_data.py
index 8fa352321c..8a6956d442 100644
--- a/tests/unit/utils/test_data.py
+++ b/tests/unit/utils/test_data.py
@@ -1,38 +1,38 @@
-# -*- coding: utf-8 -*-
-'''
+"""
Tests for salt.utils.data
-'''
+"""
# Import Python libs
-from __future__ import absolute_import, print_function, unicode_literals
+
import logging
# Import Salt libs
import salt.utils.data
import salt.utils.stringutils
-from salt.utils.odict import OrderedDict
-from tests.support.unit import TestCase, LOREM_IPSUM
-from tests.support.mock import patch
# Import 3rd party libs
-from salt.ext.six.moves import builtins # pylint: disable=import-error,redefined-builtin
-from salt.ext import six
+from salt.ext.six.moves import ( # pylint: disable=import-error,redefined-builtin
+ builtins,
+)
+from salt.utils.odict import OrderedDict
+from tests.support.mock import patch
+from tests.support.unit import LOREM_IPSUM, TestCase
log = logging.getLogger(__name__)
-_b = lambda x: x.encode('utf-8')
+_b = lambda x: x.encode("utf-8")
_s = lambda x: salt.utils.stringutils.to_str(x, normalize=True)
# Some randomized data that will not decode
-BYTES = b'1\x814\x10'
+BYTES = b"1\x814\x10"
# This is an example of a unicode string with й constructed using two separate
# code points. Do not modify it.
-EGGS = '\u044f\u0438\u0306\u0446\u0430'
+EGGS = "\u044f\u0438\u0306\u0446\u0430"
class DataTestCase(TestCase):
test_data = [
- 'unicode_str',
- _b('питон'),
+ "unicode_str",
+ _b("питон"),
123,
456.789,
True,
@@ -40,71 +40,79 @@ class DataTestCase(TestCase):
None,
EGGS,
BYTES,
- [123, 456.789, _b('спам'), True, False, None, EGGS, BYTES],
- (987, 654.321, _b('яйца'), EGGS, None, (True, EGGS, BYTES)),
- {_b('str_key'): _b('str_val'),
- None: True,
- 123: 456.789,
- EGGS: BYTES,
- _b('subdict'): {'unicode_key': EGGS,
- _b('tuple'): (123, 'hello', _b('world'), True, EGGS, BYTES),
- _b('list'): [456, _b('спам'), False, EGGS, BYTES]}},
- OrderedDict([(_b('foo'), 'bar'), (123, 456), (EGGS, BYTES)])
+ [123, 456.789, _b("спам"), True, False, None, EGGS, BYTES],
+ (987, 654.321, _b("яйца"), EGGS, None, (True, EGGS, BYTES)),
+ {
+ _b("str_key"): _b("str_val"),
+ None: True,
+ 123: 456.789,
+ EGGS: BYTES,
+ _b("subdict"): {
+ "unicode_key": EGGS,
+ _b("tuple"): (123, "hello", _b("world"), True, EGGS, BYTES),
+ _b("list"): [456, _b("спам"), False, EGGS, BYTES],
+ },
+ },
+ OrderedDict([(_b("foo"), "bar"), (123, 456), (EGGS, BYTES)]),
]
def test_sorted_ignorecase(self):
- test_list = ['foo', 'Foo', 'bar', 'Bar']
- expected_list = ['bar', 'Bar', 'foo', 'Foo']
- self.assertEqual(
- salt.utils.data.sorted_ignorecase(test_list), expected_list)
+ test_list = ["foo", "Foo", "bar", "Bar"]
+ expected_list = ["bar", "Bar", "foo", "Foo"]
+ self.assertEqual(salt.utils.data.sorted_ignorecase(test_list), expected_list)
def test_mysql_to_dict(self):
- test_mysql_output = ['+----+------+-----------+------+---------+------+-------+------------------+',
- '| Id | User | Host | db | Command | Time | State | Info |',
- '+----+------+-----------+------+---------+------+-------+------------------+',
- '| 7 | root | localhost | NULL | Query | 0 | init | show processlist |',
- '+----+------+-----------+------+---------+------+-------+------------------+']
+ test_mysql_output = [
+ "+----+------+-----------+------+---------+------+-------+------------------+",
+ "| Id | User | Host | db | Command | Time | State | Info |",
+ "+----+------+-----------+------+---------+------+-------+------------------+",
+ "| 7 | root | localhost | NULL | Query | 0 | init | show processlist |",
+ "+----+------+-----------+------+---------+------+-------+------------------+",
+ ]
- ret = salt.utils.data.mysql_to_dict(test_mysql_output, 'Info')
+ ret = salt.utils.data.mysql_to_dict(test_mysql_output, "Info")
expected_dict = {
- 'show processlist': {'Info': 'show processlist', 'db': 'NULL', 'State': 'init', 'Host': 'localhost',
- 'Command': 'Query', 'User': 'root', 'Time': 0, 'Id': 7}}
+ "show processlist": {
+ "Info": "show processlist",
+ "db": "NULL",
+ "State": "init",
+ "Host": "localhost",
+ "Command": "Query",
+ "User": "root",
+ "Time": 0,
+ "Id": 7,
+ }
+ }
self.assertDictEqual(ret, expected_dict)
def test_subdict_match(self):
- test_two_level_dict = {'foo': {'bar': 'baz'}}
- test_two_level_comb_dict = {'foo': {'bar': 'baz:woz'}}
+ test_two_level_dict = {"foo": {"bar": "baz"}}
+ test_two_level_comb_dict = {"foo": {"bar": "baz:woz"}}
test_two_level_dict_and_list = {
- 'abc': ['def', 'ghi', {'lorem': {'ipsum': [{'dolor': 'sit'}]}}],
+ "abc": ["def", "ghi", {"lorem": {"ipsum": [{"dolor": "sit"}]}}],
}
- test_three_level_dict = {'a': {'b': {'c': 'v'}}}
+ test_three_level_dict = {"a": {"b": {"c": "v"}}}
self.assertTrue(
- salt.utils.data.subdict_match(
- test_two_level_dict, 'foo:bar:baz'
- )
+ salt.utils.data.subdict_match(test_two_level_dict, "foo:bar:baz")
)
# In test_two_level_comb_dict, 'foo:bar' corresponds to 'baz:woz', not
# 'baz'. This match should return False.
self.assertFalse(
- salt.utils.data.subdict_match(
- test_two_level_comb_dict, 'foo:bar:baz'
- )
+ salt.utils.data.subdict_match(test_two_level_comb_dict, "foo:bar:baz")
)
# This tests matching with the delimiter in the value part (in other
# words, that the path 'foo:bar' corresponds to the string 'baz:woz').
self.assertTrue(
- salt.utils.data.subdict_match(
- test_two_level_comb_dict, 'foo:bar:baz:woz'
- )
+ salt.utils.data.subdict_match(test_two_level_comb_dict, "foo:bar:baz:woz")
)
# This would match if test_two_level_comb_dict['foo']['bar'] was equal
# to 'baz:woz:wiz', or if there was more deep nesting. But it does not,
# so this should return False.
self.assertFalse(
salt.utils.data.subdict_match(
- test_two_level_comb_dict, 'foo:bar:baz:woz:wiz'
+ test_two_level_comb_dict, "foo:bar:baz:woz:wiz"
)
)
# This tests for cases when a key path corresponds to a list. The
@@ -115,189 +123,171 @@ class DataTestCase(TestCase):
# salt.utils.traverse_list_and_dict() so this particular assertion is a
# sanity check.
self.assertTrue(
- salt.utils.data.subdict_match(
- test_two_level_dict_and_list, 'abc:ghi'
- )
+ salt.utils.data.subdict_match(test_two_level_dict_and_list, "abc:ghi")
)
# This tests the use case of a dict embedded in a list, embedded in a
# list, embedded in a dict. This is a rather absurd case, but it
# confirms that match recursion works properly.
self.assertTrue(
salt.utils.data.subdict_match(
- test_two_level_dict_and_list, 'abc:lorem:ipsum:dolor:sit'
+ test_two_level_dict_and_list, "abc:lorem:ipsum:dolor:sit"
)
)
# Test four level dict match for reference
- self.assertTrue(
- salt.utils.data.subdict_match(
- test_three_level_dict, 'a:b:c:v'
- )
- )
+ self.assertTrue(salt.utils.data.subdict_match(test_three_level_dict, "a:b:c:v"))
# Test regression in 2015.8 where 'a:c:v' would match 'a:b:c:v'
- self.assertFalse(
- salt.utils.data.subdict_match(
- test_three_level_dict, 'a:c:v'
- )
- )
+ self.assertFalse(salt.utils.data.subdict_match(test_three_level_dict, "a:c:v"))
# Test wildcard match
- self.assertTrue(
- salt.utils.data.subdict_match(
- test_three_level_dict, 'a:*:c:v'
- )
- )
+ self.assertTrue(salt.utils.data.subdict_match(test_three_level_dict, "a:*:c:v"))
def test_subdict_match_with_wildcards(self):
- '''
+ """
Tests subdict matching when wildcards are used in the expression
- '''
- data = {
- 'a': {
- 'b': {
- 'ç': 'd',
- 'é': ['eff', 'gee', '8ch'],
- 'ĩ': {'j': 'k'}
- }
- }
- }
- assert salt.utils.data.subdict_match(data, '*:*:*:*')
- assert salt.utils.data.subdict_match(data, 'a:*:*:*')
- assert salt.utils.data.subdict_match(data, 'a:b:*:*')
- assert salt.utils.data.subdict_match(data, 'a:b:ç:*')
- assert salt.utils.data.subdict_match(data, 'a:b:*:d')
- assert salt.utils.data.subdict_match(data, 'a:*:ç:d')
- assert salt.utils.data.subdict_match(data, '*:b:ç:d')
- assert salt.utils.data.subdict_match(data, '*:*:ç:d')
- assert salt.utils.data.subdict_match(data, '*:*:*:d')
- assert salt.utils.data.subdict_match(data, 'a:*:*:d')
- assert salt.utils.data.subdict_match(data, 'a:b:*:ef*')
- assert salt.utils.data.subdict_match(data, 'a:b:*:g*')
- assert salt.utils.data.subdict_match(data, 'a:b:*:j:*')
- assert salt.utils.data.subdict_match(data, 'a:b:*:j:k')
- assert salt.utils.data.subdict_match(data, 'a:b:*:*:k')
- assert salt.utils.data.subdict_match(data, 'a:b:*:*:*')
+ """
+ data = {"a": {"b": {"ç": "d", "é": ["eff", "gee", "8ch"], "ĩ": {"j": "k"}}}}
+ assert salt.utils.data.subdict_match(data, "*:*:*:*")
+ assert salt.utils.data.subdict_match(data, "a:*:*:*")
+ assert salt.utils.data.subdict_match(data, "a:b:*:*")
+ assert salt.utils.data.subdict_match(data, "a:b:ç:*")
+ assert salt.utils.data.subdict_match(data, "a:b:*:d")
+ assert salt.utils.data.subdict_match(data, "a:*:ç:d")
+ assert salt.utils.data.subdict_match(data, "*:b:ç:d")
+ assert salt.utils.data.subdict_match(data, "*:*:ç:d")
+ assert salt.utils.data.subdict_match(data, "*:*:*:d")
+ assert salt.utils.data.subdict_match(data, "a:*:*:d")
+ assert salt.utils.data.subdict_match(data, "a:b:*:ef*")
+ assert salt.utils.data.subdict_match(data, "a:b:*:g*")
+ assert salt.utils.data.subdict_match(data, "a:b:*:j:*")
+ assert salt.utils.data.subdict_match(data, "a:b:*:j:k")
+ assert salt.utils.data.subdict_match(data, "a:b:*:*:k")
+ assert salt.utils.data.subdict_match(data, "a:b:*:*:*")
def test_traverse_dict(self):
- test_two_level_dict = {'foo': {'bar': 'baz'}}
+ test_two_level_dict = {"foo": {"bar": "baz"}}
self.assertDictEqual(
- {'not_found': 'nope'},
+ {"not_found": "nope"},
salt.utils.data.traverse_dict(
- test_two_level_dict, 'foo:bar:baz', {'not_found': 'nope'}
- )
+ test_two_level_dict, "foo:bar:baz", {"not_found": "nope"}
+ ),
)
self.assertEqual(
- 'baz',
+ "baz",
salt.utils.data.traverse_dict(
- test_two_level_dict, 'foo:bar', {'not_found': 'not_found'}
- )
+ test_two_level_dict, "foo:bar", {"not_found": "not_found"}
+ ),
)
def test_traverse_dict_and_list(self):
- test_two_level_dict = {'foo': {'bar': 'baz'}}
+ test_two_level_dict = {"foo": {"bar": "baz"}}
test_two_level_dict_and_list = {
- 'foo': ['bar', 'baz', {'lorem': {'ipsum': [{'dolor': 'sit'}]}}]
+ "foo": ["bar", "baz", {"lorem": {"ipsum": [{"dolor": "sit"}]}}]
}
# Check traversing too far: salt.utils.data.traverse_dict_and_list() returns
# the value corresponding to a given key path, and baz is a value
# corresponding to the key path foo:bar.
self.assertDictEqual(
- {'not_found': 'nope'},
+ {"not_found": "nope"},
salt.utils.data.traverse_dict_and_list(
- test_two_level_dict, 'foo:bar:baz', {'not_found': 'nope'}
- )
+ test_two_level_dict, "foo:bar:baz", {"not_found": "nope"}
+ ),
)
# Now check to ensure that foo:bar corresponds to baz
self.assertEqual(
- 'baz',
+ "baz",
salt.utils.data.traverse_dict_and_list(
- test_two_level_dict, 'foo:bar', {'not_found': 'not_found'}
- )
+ test_two_level_dict, "foo:bar", {"not_found": "not_found"}
+ ),
)
# Check traversing too far
self.assertDictEqual(
- {'not_found': 'nope'},
+ {"not_found": "nope"},
salt.utils.data.traverse_dict_and_list(
- test_two_level_dict_and_list, 'foo:bar', {'not_found': 'nope'}
- )
+ test_two_level_dict_and_list, "foo:bar", {"not_found": "nope"}
+ ),
)
# Check index 1 (2nd element) of list corresponding to path 'foo'
self.assertEqual(
- 'baz',
+ "baz",
salt.utils.data.traverse_dict_and_list(
- test_two_level_dict_and_list, 'foo:1', {'not_found': 'not_found'}
- )
+ test_two_level_dict_and_list, "foo:1", {"not_found": "not_found"}
+ ),
)
# Traverse a couple times into dicts embedded in lists
self.assertEqual(
- 'sit',
+ "sit",
salt.utils.data.traverse_dict_and_list(
test_two_level_dict_and_list,
- 'foo:lorem:ipsum:dolor',
- {'not_found': 'not_found'}
- )
+ "foo:lorem:ipsum:dolor",
+ {"not_found": "not_found"},
+ ),
)
def test_compare_dicts(self):
- ret = salt.utils.data.compare_dicts(old={'foo': 'bar'}, new={'foo': 'bar'})
+ ret = salt.utils.data.compare_dicts(old={"foo": "bar"}, new={"foo": "bar"})
self.assertEqual(ret, {})
- ret = salt.utils.data.compare_dicts(old={'foo': 'bar'}, new={'foo': 'woz'})
- expected_ret = {'foo': {'new': 'woz', 'old': 'bar'}}
+ ret = salt.utils.data.compare_dicts(old={"foo": "bar"}, new={"foo": "woz"})
+ expected_ret = {"foo": {"new": "woz", "old": "bar"}}
self.assertDictEqual(ret, expected_ret)
def test_compare_lists_no_change(self):
- ret = salt.utils.data.compare_lists(old=[1, 2, 3, 'a', 'b', 'c'],
- new=[1, 2, 3, 'a', 'b', 'c'])
+ ret = salt.utils.data.compare_lists(
+ old=[1, 2, 3, "a", "b", "c"], new=[1, 2, 3, "a", "b", "c"]
+ )
expected = {}
self.assertDictEqual(ret, expected)
def test_compare_lists_changes(self):
- ret = salt.utils.data.compare_lists(old=[1, 2, 3, 'a', 'b', 'c'],
- new=[1, 2, 4, 'x', 'y', 'z'])
- expected = {'new': [4, 'x', 'y', 'z'], 'old': [3, 'a', 'b', 'c']}
+ ret = salt.utils.data.compare_lists(
+ old=[1, 2, 3, "a", "b", "c"], new=[1, 2, 4, "x", "y", "z"]
+ )
+ expected = {"new": [4, "x", "y", "z"], "old": [3, "a", "b", "c"]}
self.assertDictEqual(ret, expected)
def test_compare_lists_changes_new(self):
- ret = salt.utils.data.compare_lists(old=[1, 2, 3],
- new=[1, 2, 3, 'x', 'y', 'z'])
- expected = {'new': ['x', 'y', 'z']}
+ ret = salt.utils.data.compare_lists(old=[1, 2, 3], new=[1, 2, 3, "x", "y", "z"])
+ expected = {"new": ["x", "y", "z"]}
self.assertDictEqual(ret, expected)
def test_compare_lists_changes_old(self):
- ret = salt.utils.data.compare_lists(old=[1, 2, 3, 'a', 'b', 'c'],
- new=[1, 2, 3])
- expected = {'old': ['a', 'b', 'c']}
+ ret = salt.utils.data.compare_lists(old=[1, 2, 3, "a", "b", "c"], new=[1, 2, 3])
+ expected = {"old": ["a", "b", "c"]}
self.assertDictEqual(ret, expected)
def test_decode(self):
- '''
+ """
Companion to test_decode_to_str, they should both be kept up-to-date
with one another.
NOTE: This uses the lambda "_b" defined above in the global scope,
which encodes a string to a bytestring, assuming utf-8.
- '''
+ """
expected = [
- 'unicode_str',
- 'питон',
+ "unicode_str",
+ "питон",
123,
456.789,
True,
False,
None,
- 'яйца',
+ "яйца",
BYTES,
- [123, 456.789, 'спам', True, False, None, 'яйца', BYTES],
- (987, 654.321, 'яйца', 'яйца', None, (True, 'яйца', BYTES)),
- {'str_key': 'str_val',
- None: True,
- 123: 456.789,
- 'яйца': BYTES,
- 'subdict': {'unicode_key': 'яйца',
- 'tuple': (123, 'hello', 'world', True, 'яйца', BYTES),
- 'list': [456, 'спам', False, 'яйца', BYTES]}},
- OrderedDict([('foo', 'bar'), (123, 456), ('яйца', BYTES)])
+ [123, 456.789, "спам", True, False, None, "яйца", BYTES],
+ (987, 654.321, "яйца", "яйца", None, (True, "яйца", BYTES)),
+ {
+ "str_key": "str_val",
+ None: True,
+ 123: 456.789,
+ "яйца": BYTES,
+ "subdict": {
+ "unicode_key": "яйца",
+ "tuple": (123, "hello", "world", True, "яйца", BYTES),
+ "list": [456, "спам", False, "яйца", BYTES],
+ },
+ },
+ OrderedDict([("foo", "bar"), (123, 456), ("яйца", BYTES)]),
]
ret = salt.utils.data.decode(
@@ -305,7 +295,8 @@ class DataTestCase(TestCase):
keep=True,
normalize=True,
preserve_dict_class=True,
- preserve_tuples=True)
+ preserve_tuples=True,
+ )
self.assertEqual(ret, expected)
# The binary data in the data structure should fail to decode, even
@@ -317,74 +308,100 @@ class DataTestCase(TestCase):
keep=False,
normalize=True,
preserve_dict_class=True,
- preserve_tuples=True)
+ preserve_tuples=True,
+ )
# Now munge the expected data so that we get what we would expect if we
# disable preservation of dict class and tuples
- expected[10] = [987, 654.321, 'яйца', 'яйца', None, [True, 'яйца', BYTES]]
- expected[11]['subdict']['tuple'] = [123, 'hello', 'world', True, 'яйца', BYTES]
- expected[12] = {'foo': 'bar', 123: 456, 'яйца': BYTES}
+ expected[10] = [987, 654.321, "яйца", "яйца", None, [True, "яйца", BYTES]]
+ expected[11]["subdict"]["tuple"] = [123, "hello", "world", True, "яйца", BYTES]
+ expected[12] = {"foo": "bar", 123: 456, "яйца": BYTES}
ret = salt.utils.data.decode(
self.test_data,
keep=True,
normalize=True,
preserve_dict_class=False,
- preserve_tuples=False)
+ preserve_tuples=False,
+ )
self.assertEqual(ret, expected)
# Now test single non-string, non-data-structure items, these should
# return the same value when passed to this function
for item in (123, 4.56, True, False, None):
- log.debug('Testing decode of %s', item)
+ log.debug("Testing decode of %s", item)
self.assertEqual(salt.utils.data.decode(item), item)
# Test single strings (not in a data structure)
- self.assertEqual(salt.utils.data.decode('foo'), 'foo')
- self.assertEqual(salt.utils.data.decode(_b('bar')), 'bar')
- self.assertEqual(salt.utils.data.decode(EGGS, normalize=True), 'яйца')
+ self.assertEqual(salt.utils.data.decode("foo"), "foo")
+ self.assertEqual(salt.utils.data.decode(_b("bar")), "bar")
+ self.assertEqual(salt.utils.data.decode(EGGS, normalize=True), "яйца")
self.assertEqual(salt.utils.data.decode(EGGS, normalize=False), EGGS)
# Test binary blob
self.assertEqual(salt.utils.data.decode(BYTES, keep=True), BYTES)
- self.assertRaises(
- UnicodeDecodeError,
- salt.utils.data.decode,
- BYTES,
- keep=False)
+ self.assertRaises(UnicodeDecodeError, salt.utils.data.decode, BYTES, keep=False)
+
+ def test_circular_refs_dicts(self):
+ test_dict = {"key": "value", "type": "test1"}
+ test_dict["self"] = test_dict
+ ret = salt.utils.data._remove_circular_refs(ob=test_dict)
+ self.assertDictEqual(ret, {"key": "value", "type": "test1", "self": None})
+
+ def test_circular_refs_lists(self):
+ test_list = {
+ "foo": [],
+ }
+ test_list["foo"].append((test_list,))
+ ret = salt.utils.data._remove_circular_refs(ob=test_list)
+ self.assertDictEqual(ret, {"foo": [(None,)]})
+
+ def test_circular_refs_tuple(self):
+ test_dup = {"foo": "string 1", "bar": "string 1", "ham": 1, "spam": 1}
+ ret = salt.utils.data._remove_circular_refs(ob=test_dup)
+ self.assertDictEqual(
+ ret, {"foo": "string 1", "bar": "string 1", "ham": 1, "spam": 1}
+ )
def test_decode_to_str(self):
- '''
+ """
Companion to test_decode, they should both be kept up-to-date with one
another.
NOTE: This uses the lambda "_s" defined above in the global scope,
which converts the string/bytestring to a str type.
- '''
+ """
expected = [
- _s('unicode_str'),
- _s('питон'),
+ _s("unicode_str"),
+ _s("питон"),
123,
456.789,
True,
False,
None,
- _s('яйца'),
+ _s("яйца"),
BYTES,
- [123, 456.789, _s('спам'), True, False, None, _s('яйца'), BYTES],
- (987, 654.321, _s('яйца'), _s('яйца'), None, (True, _s('яйца'), BYTES)),
+ [123, 456.789, _s("спам"), True, False, None, _s("яйца"), BYTES],
+ (987, 654.321, _s("яйца"), _s("яйца"), None, (True, _s("яйца"), BYTES)),
{
- _s('str_key'): _s('str_val'),
+ _s("str_key"): _s("str_val"),
None: True,
123: 456.789,
- _s('яйца'): BYTES,
- _s('subdict'): {
- _s('unicode_key'): _s('яйца'),
- _s('tuple'): (123, _s('hello'), _s('world'), True, _s('яйца'), BYTES),
- _s('list'): [456, _s('спам'), False, _s('яйца'), BYTES]
- }
+ _s("яйца"): BYTES,
+ _s("subdict"): {
+ _s("unicode_key"): _s("яйца"),
+ _s("tuple"): (
+ 123,
+ _s("hello"),
+ _s("world"),
+ True,
+ _s("яйца"),
+ BYTES,
+ ),
+ _s("list"): [456, _s("спам"), False, _s("яйца"), BYTES],
+ },
},
- OrderedDict([(_s('foo'), _s('bar')), (123, 456), (_s('яйца'), BYTES)])
+ OrderedDict([(_s("foo"), _s("bar")), (123, 456), (_s("яйца"), BYTES)]),
]
ret = salt.utils.data.decode(
@@ -393,27 +410,42 @@ class DataTestCase(TestCase):
normalize=True,
preserve_dict_class=True,
preserve_tuples=True,
- to_str=True)
+ to_str=True,
+ )
self.assertEqual(ret, expected)
- if six.PY3:
- # The binary data in the data structure should fail to decode, even
- # using the fallback, and raise an exception.
- self.assertRaises(
- UnicodeDecodeError,
- salt.utils.data.decode,
- self.test_data,
- keep=False,
- normalize=True,
- preserve_dict_class=True,
- preserve_tuples=True,
- to_str=True)
+ # The binary data in the data structure should fail to decode, even
+ # using the fallback, and raise an exception.
+ self.assertRaises(
+ UnicodeDecodeError,
+ salt.utils.data.decode,
+ self.test_data,
+ keep=False,
+ normalize=True,
+ preserve_dict_class=True,
+ preserve_tuples=True,
+ to_str=True,
+ )
# Now munge the expected data so that we get what we would expect if we
# disable preservation of dict class and tuples
- expected[10] = [987, 654.321, _s('яйца'), _s('яйца'), None, [True, _s('яйца'), BYTES]]
- expected[11][_s('subdict')][_s('tuple')] = [123, _s('hello'), _s('world'), True, _s('яйца'), BYTES]
- expected[12] = {_s('foo'): _s('bar'), 123: 456, _s('яйца'): BYTES}
+ expected[10] = [
+ 987,
+ 654.321,
+ _s("яйца"),
+ _s("яйца"),
+ None,
+ [True, _s("яйца"), BYTES],
+ ]
+ expected[11][_s("subdict")][_s("tuple")] = [
+ 123,
+ _s("hello"),
+ _s("world"),
+ True,
+ _s("яйца"),
+ BYTES,
+ ]
+ expected[12] = {_s("foo"): _s("bar"), 123: 456, _s("яйца"): BYTES}
ret = salt.utils.data.decode(
self.test_data,
@@ -421,47 +453,41 @@ class DataTestCase(TestCase):
normalize=True,
preserve_dict_class=False,
preserve_tuples=False,
- to_str=True)
+ to_str=True,
+ )
self.assertEqual(ret, expected)
# Now test single non-string, non-data-structure items, these should
# return the same value when passed to this function
for item in (123, 4.56, True, False, None):
- log.debug('Testing decode of %s', item)
+ log.debug("Testing decode of %s", item)
self.assertEqual(salt.utils.data.decode(item, to_str=True), item)
# Test single strings (not in a data structure)
- self.assertEqual(salt.utils.data.decode('foo', to_str=True), _s('foo'))
- self.assertEqual(salt.utils.data.decode(_b('bar'), to_str=True), _s('bar'))
+ self.assertEqual(salt.utils.data.decode("foo", to_str=True), _s("foo"))
+ self.assertEqual(salt.utils.data.decode(_b("bar"), to_str=True), _s("bar"))
# Test binary blob
- self.assertEqual(
- salt.utils.data.decode(BYTES, keep=True, to_str=True),
- BYTES
+ self.assertEqual(salt.utils.data.decode(BYTES, keep=True, to_str=True), BYTES)
+ self.assertRaises(
+ UnicodeDecodeError, salt.utils.data.decode, BYTES, keep=False, to_str=True,
)
- if six.PY3:
- self.assertRaises(
- UnicodeDecodeError,
- salt.utils.data.decode,
- BYTES,
- keep=False,
- to_str=True)
def test_decode_fallback(self):
- '''
+ """
Test fallback to utf-8
- '''
- with patch.object(builtins, '__salt_system_encoding__', 'ascii'):
- self.assertEqual(salt.utils.data.decode(_b('яйца')), 'яйца')
+ """
+ with patch.object(builtins, "__salt_system_encoding__", "ascii"):
+ self.assertEqual(salt.utils.data.decode(_b("яйца")), "яйца")
def test_encode(self):
- '''
+ """
NOTE: This uses the lambda "_b" defined above in the global scope,
which encodes a string to a bytestring, assuming utf-8.
- '''
+ """
expected = [
- _b('unicode_str'),
- _b('питон'),
+ _b("unicode_str"),
+ _b("питон"),
123,
456.789,
True,
@@ -469,67 +495,71 @@ class DataTestCase(TestCase):
None,
_b(EGGS),
BYTES,
- [123, 456.789, _b('спам'), True, False, None, _b(EGGS), BYTES],
- (987, 654.321, _b('яйца'), _b(EGGS), None, (True, _b(EGGS), BYTES)),
+ [123, 456.789, _b("спам"), True, False, None, _b(EGGS), BYTES],
+ (987, 654.321, _b("яйца"), _b(EGGS), None, (True, _b(EGGS), BYTES)),
{
- _b('str_key'): _b('str_val'),
+ _b("str_key"): _b("str_val"),
None: True,
123: 456.789,
_b(EGGS): BYTES,
- _b('subdict'): {
- _b('unicode_key'): _b(EGGS),
- _b('tuple'): (123, _b('hello'), _b('world'), True, _b(EGGS), BYTES),
- _b('list'): [456, _b('спам'), False, _b(EGGS), BYTES]
- }
+ _b("subdict"): {
+ _b("unicode_key"): _b(EGGS),
+ _b("tuple"): (123, _b("hello"), _b("world"), True, _b(EGGS), BYTES),
+ _b("list"): [456, _b("спам"), False, _b(EGGS), BYTES],
+ },
},
- OrderedDict([(_b('foo'), _b('bar')), (123, 456), (_b(EGGS), BYTES)])
+ OrderedDict([(_b("foo"), _b("bar")), (123, 456), (_b(EGGS), BYTES)]),
]
# Both keep=True and keep=False should work because the BYTES data is
# already bytes.
ret = salt.utils.data.encode(
- self.test_data,
- keep=True,
- preserve_dict_class=True,
- preserve_tuples=True)
+ self.test_data, keep=True, preserve_dict_class=True, preserve_tuples=True
+ )
self.assertEqual(ret, expected)
ret = salt.utils.data.encode(
- self.test_data,
- keep=False,
- preserve_dict_class=True,
- preserve_tuples=True)
+ self.test_data, keep=False, preserve_dict_class=True, preserve_tuples=True
+ )
self.assertEqual(ret, expected)
# Now munge the expected data so that we get what we would expect if we
# disable preservation of dict class and tuples
- expected[10] = [987, 654.321, _b('яйца'), _b(EGGS), None, [True, _b(EGGS), BYTES]]
- expected[11][_b('subdict')][_b('tuple')] = [
- 123, _b('hello'), _b('world'), True, _b(EGGS), BYTES
+ expected[10] = [
+ 987,
+ 654.321,
+ _b("яйца"),
+ _b(EGGS),
+ None,
+ [True, _b(EGGS), BYTES],
]
- expected[12] = {_b('foo'): _b('bar'), 123: 456, _b(EGGS): BYTES}
+ expected[11][_b("subdict")][_b("tuple")] = [
+ 123,
+ _b("hello"),
+ _b("world"),
+ True,
+ _b(EGGS),
+ BYTES,
+ ]
+ expected[12] = {_b("foo"): _b("bar"), 123: 456, _b(EGGS): BYTES}
ret = salt.utils.data.encode(
- self.test_data,
- keep=True,
- preserve_dict_class=False,
- preserve_tuples=False)
+ self.test_data, keep=True, preserve_dict_class=False, preserve_tuples=False
+ )
self.assertEqual(ret, expected)
ret = salt.utils.data.encode(
- self.test_data,
- keep=False,
- preserve_dict_class=False,
- preserve_tuples=False)
+ self.test_data, keep=False, preserve_dict_class=False, preserve_tuples=False
+ )
self.assertEqual(ret, expected)
# Now test single non-string, non-data-structure items, these should
# return the same value when passed to this function
for item in (123, 4.56, True, False, None):
- log.debug('Testing encode of %s', item)
+ log.debug("Testing encode of %s", item)
self.assertEqual(salt.utils.data.encode(item), item)
# Test single strings (not in a data structure)
- self.assertEqual(salt.utils.data.encode('foo'), _b('foo'))
- self.assertEqual(salt.utils.data.encode(_b('bar')), _b('bar'))
+ self.assertEqual(salt.utils.data.encode("foo"), _b("foo"))
+ self.assertEqual(salt.utils.data.encode(_b("bar")), _b("bar"))
# Test binary blob, nothing should happen even when keep=False since
# the data is already bytes
@@ -537,41 +567,43 @@ class DataTestCase(TestCase):
self.assertEqual(salt.utils.data.encode(BYTES, keep=False), BYTES)
def test_encode_keep(self):
- '''
+ """
Whereas we tested the keep argument in test_decode, it is much easier
to do a more comprehensive test of keep in its own function where we
can force the encoding.
- '''
- unicode_str = 'питон'
- encoding = 'ascii'
+ """
+ unicode_str = "питон"
+ encoding = "ascii"
# Test single string
self.assertEqual(
- salt.utils.data.encode(unicode_str, encoding, keep=True),
- unicode_str)
+ salt.utils.data.encode(unicode_str, encoding, keep=True), unicode_str
+ )
self.assertRaises(
UnicodeEncodeError,
salt.utils.data.encode,
unicode_str,
encoding,
- keep=False)
+ keep=False,
+ )
data = [
unicode_str,
- [b'foo', [unicode_str], {b'key': unicode_str}, (unicode_str,)],
- {b'list': [b'foo', unicode_str],
- b'dict': {b'key': unicode_str},
- b'tuple': (b'foo', unicode_str)},
- ([b'foo', unicode_str], {b'key': unicode_str}, (unicode_str,))
+ [b"foo", [unicode_str], {b"key": unicode_str}, (unicode_str,)],
+ {
+ b"list": [b"foo", unicode_str],
+ b"dict": {b"key": unicode_str},
+ b"tuple": (b"foo", unicode_str),
+ },
+ ([b"foo", unicode_str], {b"key": unicode_str}, (unicode_str,)),
]
# Since everything was a bytestring aside from the bogus data, the
# return data should be identical. We don't need to test recursive
# decoding, that has already been tested in test_encode.
self.assertEqual(
- salt.utils.data.encode(data, encoding,
- keep=True, preserve_tuples=True),
- data
+ salt.utils.data.encode(data, encoding, keep=True, preserve_tuples=True),
+ data,
)
self.assertRaises(
UnicodeEncodeError,
@@ -579,13 +611,15 @@ class DataTestCase(TestCase):
data,
encoding,
keep=False,
- preserve_tuples=True)
+ preserve_tuples=True,
+ )
for index, _ in enumerate(data):
self.assertEqual(
- salt.utils.data.encode(data[index], encoding,
- keep=True, preserve_tuples=True),
- data[index]
+ salt.utils.data.encode(
+ data[index], encoding, keep=True, preserve_tuples=True
+ ),
+ data[index],
)
self.assertRaises(
UnicodeEncodeError,
@@ -593,31 +627,36 @@ class DataTestCase(TestCase):
data[index],
encoding,
keep=False,
- preserve_tuples=True)
+ preserve_tuples=True,
+ )
def test_encode_fallback(self):
- '''
+ """
Test fallback to utf-8
- '''
- with patch.object(builtins, '__salt_system_encoding__', 'ascii'):
- self.assertEqual(salt.utils.data.encode('яйца'), _b('яйца'))
- with patch.object(builtins, '__salt_system_encoding__', 'CP1252'):
- self.assertEqual(salt.utils.data.encode('Ψ'), _b('Ψ'))
+ """
+ with patch.object(builtins, "__salt_system_encoding__", "ascii"):
+ self.assertEqual(salt.utils.data.encode("яйца"), _b("яйца"))
+ with patch.object(builtins, "__salt_system_encoding__", "CP1252"):
+ self.assertEqual(salt.utils.data.encode("Ψ"), _b("Ψ"))
def test_repack_dict(self):
- list_of_one_element_dicts = [{'dict_key_1': 'dict_val_1'},
- {'dict_key_2': 'dict_val_2'},
- {'dict_key_3': 'dict_val_3'}]
- expected_ret = {'dict_key_1': 'dict_val_1',
- 'dict_key_2': 'dict_val_2',
- 'dict_key_3': 'dict_val_3'}
+ list_of_one_element_dicts = [
+ {"dict_key_1": "dict_val_1"},
+ {"dict_key_2": "dict_val_2"},
+ {"dict_key_3": "dict_val_3"},
+ ]
+ expected_ret = {
+ "dict_key_1": "dict_val_1",
+ "dict_key_2": "dict_val_2",
+ "dict_key_3": "dict_val_3",
+ }
ret = salt.utils.data.repack_dictlist(list_of_one_element_dicts)
self.assertDictEqual(ret, expected_ret)
# Try with yaml
- yaml_key_val_pair = '- key1: val1'
+ yaml_key_val_pair = "- key1: val1"
ret = salt.utils.data.repack_dictlist(yaml_key_val_pair)
- self.assertDictEqual(ret, {'key1': 'val1'})
+ self.assertDictEqual(ret, {"key1": "val1"})
# Make sure we handle non-yaml junk data
ret = salt.utils.data.repack_dictlist(LOREM_IPSUM)
@@ -626,43 +665,47 @@ class DataTestCase(TestCase):
def test_stringify(self):
self.assertRaises(TypeError, salt.utils.data.stringify, 9)
self.assertEqual(
- salt.utils.data.stringify(['one', 'two', str('three'), 4, 5]), # future lint: disable=blacklisted-function
- ['one', 'two', 'three', '4', '5']
+ salt.utils.data.stringify(
+ ["one", "two", "three", 4, 5]
+ ), # future lint: disable=blacklisted-function
+ ["one", "two", "three", "4", "5"],
)
def test_json_query(self):
# Raises exception if jmespath module is not found
- with patch('salt.utils.data.jmespath', None):
+ with patch("salt.utils.data.jmespath", None):
self.assertRaisesRegex(
- RuntimeError, 'requires jmespath',
- salt.utils.data.json_query, {}, '@'
+ RuntimeError, "requires jmespath", salt.utils.data.json_query, {}, "@"
)
# Test search
user_groups = {
- 'user1': {'groups': ['group1', 'group2', 'group3']},
- 'user2': {'groups': ['group1', 'group2']},
- 'user3': {'groups': ['group3']},
+ "user1": {"groups": ["group1", "group2", "group3"]},
+ "user2": {"groups": ["group1", "group2"]},
+ "user3": {"groups": ["group3"]},
}
- expression = '*.groups[0]'
- primary_groups = ['group1', 'group1', 'group3']
+ expression = "*.groups[0]"
+ primary_groups = ["group1", "group1", "group3"]
self.assertEqual(
- sorted(salt.utils.data.json_query(user_groups, expression)),
- primary_groups
+ sorted(salt.utils.data.json_query(user_groups, expression)), primary_groups
)
class FilterFalseyTestCase(TestCase):
- '''
+ """
Test suite for salt.utils.data.filter_falsey
- '''
+ """
def test_nop(self):
- '''
+ """
Test cases where nothing will be done.
- '''
+ """
# Test with dictionary without recursion
- old_dict = {'foo': 'bar', 'bar': {'baz': {'qux': 'quux'}}, 'baz': ['qux', {'foo': 'bar'}]}
+ old_dict = {
+ "foo": "bar",
+ "bar": {"baz": {"qux": "quux"}},
+ "baz": ["qux", {"foo": "bar"}],
+ }
new_dict = salt.utils.data.filter_falsey(old_dict)
self.assertEqual(old_dict, new_dict)
# Check returned type equality
@@ -671,23 +714,25 @@ class FilterFalseyTestCase(TestCase):
new_dict = salt.utils.data.filter_falsey(old_dict, recurse_depth=3)
self.assertEqual(old_dict, new_dict)
# Test with list
- old_list = ['foo', 'bar']
+ old_list = ["foo", "bar"]
new_list = salt.utils.data.filter_falsey(old_list)
self.assertEqual(old_list, new_list)
# Check returned type equality
self.assertIs(type(old_list), type(new_list))
# Test with set
- old_set = set(['foo', 'bar'])
+ old_set = {"foo", "bar"}
new_set = salt.utils.data.filter_falsey(old_set)
self.assertEqual(old_set, new_set)
# Check returned type equality
self.assertIs(type(old_set), type(new_set))
# Test with OrderedDict
- old_dict = OrderedDict([
- ('foo', 'bar'),
- ('bar', OrderedDict([('qux', 'quux')])),
- ('baz', ['qux', OrderedDict([('foo', 'bar')])])
- ])
+ old_dict = OrderedDict(
+ [
+ ("foo", "bar"),
+ ("bar", OrderedDict([("qux", "quux")])),
+ ("baz", ["qux", OrderedDict([("foo", "bar")])]),
+ ]
+ )
new_dict = salt.utils.data.filter_falsey(old_dict)
self.assertEqual(old_dict, new_dict)
self.assertIs(type(old_dict), type(new_dict))
@@ -696,8 +741,8 @@ class FilterFalseyTestCase(TestCase):
new_list = salt.utils.data.filter_falsey(old_list, ignore_types=[type(0)])
self.assertEqual(old_list, new_list)
# Test excluding str (or unicode) (or both)
- old_list = ['']
- new_list = salt.utils.data.filter_falsey(old_list, ignore_types=[type('')])
+ old_list = [""]
+ new_list = salt.utils.data.filter_falsey(old_list, ignore_types=[type("")])
self.assertEqual(old_list, new_list)
# Test excluding list
old_list = [[]]
@@ -709,185 +754,264 @@ class FilterFalseyTestCase(TestCase):
self.assertEqual(old_list, new_list)
def test_filter_dict_no_recurse(self):
- '''
+ """
Test filtering a dictionary without recursing.
This will only filter out key-values where the values are falsey.
- '''
- old_dict = {'foo': None,
- 'bar': {'baz': {'qux': None, 'quux': '', 'foo': []}},
- 'baz': ['qux'],
- 'qux': {},
- 'quux': []}
+ """
+ old_dict = {
+ "foo": None,
+ "bar": {"baz": {"qux": None, "quux": "", "foo": []}},
+ "baz": ["qux"],
+ "qux": {},
+ "quux": [],
+ }
new_dict = salt.utils.data.filter_falsey(old_dict)
- expect_dict = {'bar': {'baz': {'qux': None, 'quux': '', 'foo': []}}, 'baz': ['qux']}
+ expect_dict = {
+ "bar": {"baz": {"qux": None, "quux": "", "foo": []}},
+ "baz": ["qux"],
+ }
self.assertEqual(expect_dict, new_dict)
self.assertIs(type(expect_dict), type(new_dict))
def test_filter_dict_recurse(self):
- '''
+ """
Test filtering a dictionary with recursing.
This will filter out any key-values where the values are falsey or when
the values *become* falsey after filtering their contents (in case they
are lists or dicts).
- '''
- old_dict = {'foo': None,
- 'bar': {'baz': {'qux': None, 'quux': '', 'foo': []}},
- 'baz': ['qux'],
- 'qux': {},
- 'quux': []}
+ """
+ old_dict = {
+ "foo": None,
+ "bar": {"baz": {"qux": None, "quux": "", "foo": []}},
+ "baz": ["qux"],
+ "qux": {},
+ "quux": [],
+ }
new_dict = salt.utils.data.filter_falsey(old_dict, recurse_depth=3)
- expect_dict = {'baz': ['qux']}
+ expect_dict = {"baz": ["qux"]}
self.assertEqual(expect_dict, new_dict)
self.assertIs(type(expect_dict), type(new_dict))
def test_filter_list_no_recurse(self):
- '''
+ """
Test filtering a list without recursing.
This will only filter out items which are falsey.
- '''
- old_list = ['foo', None, [], {}, 0, '']
+ """
+ old_list = ["foo", None, [], {}, 0, ""]
new_list = salt.utils.data.filter_falsey(old_list)
- expect_list = ['foo']
+ expect_list = ["foo"]
self.assertEqual(expect_list, new_list)
self.assertIs(type(expect_list), type(new_list))
# Ensure nested values are *not* filtered out.
old_list = [
- 'foo',
- ['foo'],
- ['foo', None],
- {'foo': 0},
- {'foo': 'bar', 'baz': []},
- [{'foo': ''}],
+ "foo",
+ ["foo"],
+ ["foo", None],
+ {"foo": 0},
+ {"foo": "bar", "baz": []},
+ [{"foo": ""}],
]
new_list = salt.utils.data.filter_falsey(old_list)
self.assertEqual(old_list, new_list)
self.assertIs(type(old_list), type(new_list))
def test_filter_list_recurse(self):
- '''
+ """
Test filtering a list with recursing.
This will filter out any items which are falsey, or which become falsey
after filtering their contents (in case they are lists or dicts).
- '''
+ """
old_list = [
- 'foo',
- ['foo'],
- ['foo', None],
- {'foo': 0},
- {'foo': 'bar', 'baz': []},
- [{'foo': ''}]
+ "foo",
+ ["foo"],
+ ["foo", None],
+ {"foo": 0},
+ {"foo": "bar", "baz": []},
+ [{"foo": ""}],
]
new_list = salt.utils.data.filter_falsey(old_list, recurse_depth=3)
- expect_list = ['foo', ['foo'], ['foo'], {'foo': 'bar'}]
+ expect_list = ["foo", ["foo"], ["foo"], {"foo": "bar"}]
self.assertEqual(expect_list, new_list)
self.assertIs(type(expect_list), type(new_list))
def test_filter_set_no_recurse(self):
- '''
+ """
Test filtering a set without recursing.
Note that a set cannot contain unhashable types, so recursion is not possible.
- '''
- old_set = set([
- 'foo',
- None,
- 0,
- '',
- ])
+ """
+ old_set = {"foo", None, 0, ""}
new_set = salt.utils.data.filter_falsey(old_set)
- expect_set = set(['foo'])
+ expect_set = {"foo"}
self.assertEqual(expect_set, new_set)
self.assertIs(type(expect_set), type(new_set))
def test_filter_ordereddict_no_recurse(self):
- '''
+ """
Test filtering an OrderedDict without recursing.
- '''
- old_dict = OrderedDict([
- ('foo', None),
- ('bar', OrderedDict([('baz', OrderedDict([('qux', None), ('quux', ''), ('foo', [])]))])),
- ('baz', ['qux']),
- ('qux', {}),
- ('quux', [])
- ])
+ """
+ old_dict = OrderedDict(
+ [
+ ("foo", None),
+ (
+ "bar",
+ OrderedDict(
+ [
+ (
+ "baz",
+ OrderedDict([("qux", None), ("quux", ""), ("foo", [])]),
+ )
+ ]
+ ),
+ ),
+ ("baz", ["qux"]),
+ ("qux", {}),
+ ("quux", []),
+ ]
+ )
new_dict = salt.utils.data.filter_falsey(old_dict)
- expect_dict = OrderedDict([
- ('bar', OrderedDict([('baz', OrderedDict([('qux', None), ('quux', ''), ('foo', [])]))])),
- ('baz', ['qux']),
- ])
+ expect_dict = OrderedDict(
+ [
+ (
+ "bar",
+ OrderedDict(
+ [
+ (
+ "baz",
+ OrderedDict([("qux", None), ("quux", ""), ("foo", [])]),
+ )
+ ]
+ ),
+ ),
+ ("baz", ["qux"]),
+ ]
+ )
self.assertEqual(expect_dict, new_dict)
self.assertIs(type(expect_dict), type(new_dict))
def test_filter_ordereddict_recurse(self):
- '''
+ """
Test filtering an OrderedDict with recursing.
- '''
- old_dict = OrderedDict([
- ('foo', None),
- ('bar', OrderedDict([('baz', OrderedDict([('qux', None), ('quux', ''), ('foo', [])]))])),
- ('baz', ['qux']),
- ('qux', {}),
- ('quux', [])
- ])
+ """
+ old_dict = OrderedDict(
+ [
+ ("foo", None),
+ (
+ "bar",
+ OrderedDict(
+ [
+ (
+ "baz",
+ OrderedDict([("qux", None), ("quux", ""), ("foo", [])]),
+ )
+ ]
+ ),
+ ),
+ ("baz", ["qux"]),
+ ("qux", {}),
+ ("quux", []),
+ ]
+ )
new_dict = salt.utils.data.filter_falsey(old_dict, recurse_depth=3)
- expect_dict = OrderedDict([
- ('baz', ['qux']),
- ])
+ expect_dict = OrderedDict([("baz", ["qux"])])
self.assertEqual(expect_dict, new_dict)
self.assertIs(type(expect_dict), type(new_dict))
def test_filter_list_recurse_limit(self):
- '''
+ """
Test filtering a list with recursing, but with a limited depth.
Note that the top-level is always processed, so a recursion depth of 2
means that two *additional* levels are processed.
- '''
+ """
old_list = [None, [None, [None, [None]]]]
new_list = salt.utils.data.filter_falsey(old_list, recurse_depth=2)
self.assertEqual([[[[None]]]], new_list)
def test_filter_dict_recurse_limit(self):
- '''
+ """
Test filtering a dict with recursing, but with a limited depth.
Note that the top-level is always processed, so a recursion depth of 2
means that two *additional* levels are processed.
- '''
- old_dict = {'one': None,
- 'foo': {'two': None, 'bar': {'three': None, 'baz': {'four': None}}}}
+ """
+ old_dict = {
+ "one": None,
+ "foo": {"two": None, "bar": {"three": None, "baz": {"four": None}}},
+ }
new_dict = salt.utils.data.filter_falsey(old_dict, recurse_depth=2)
- self.assertEqual({'foo': {'bar': {'baz': {'four': None}}}}, new_dict)
+ self.assertEqual({"foo": {"bar": {"baz": {"four": None}}}}, new_dict)
def test_filter_exclude_types(self):
- '''
+ """
Test filtering a list recursively, but also ignoring (i.e. not filtering)
out certain types that can be falsey.
- '''
+ """
# Ignore int, unicode
- old_list = ['foo', ['foo'], ['foo', None], {'foo': 0}, {'foo': 'bar', 'baz': []}, [{'foo': ''}]]
- new_list = salt.utils.data.filter_falsey(old_list, recurse_depth=3, ignore_types=[type(0), type('')])
- self.assertEqual(['foo', ['foo'], ['foo'], {'foo': 0}, {'foo': 'bar'}, [{'foo': ''}]], new_list)
+ old_list = [
+ "foo",
+ ["foo"],
+ ["foo", None],
+ {"foo": 0},
+ {"foo": "bar", "baz": []},
+ [{"foo": ""}],
+ ]
+ new_list = salt.utils.data.filter_falsey(
+ old_list, recurse_depth=3, ignore_types=[type(0), type("")]
+ )
+ self.assertEqual(
+ ["foo", ["foo"], ["foo"], {"foo": 0}, {"foo": "bar"}, [{"foo": ""}]],
+ new_list,
+ )
# Ignore list
- old_list = ['foo', ['foo'], ['foo', None], {'foo': 0}, {'foo': 'bar', 'baz': []}, [{'foo': ''}]]
- new_list = salt.utils.data.filter_falsey(old_list, recurse_depth=3, ignore_types=[type([])])
- self.assertEqual(['foo', ['foo'], ['foo'], {'foo': 'bar', 'baz': []}, []], new_list)
+ old_list = [
+ "foo",
+ ["foo"],
+ ["foo", None],
+ {"foo": 0},
+ {"foo": "bar", "baz": []},
+ [{"foo": ""}],
+ ]
+ new_list = salt.utils.data.filter_falsey(
+ old_list, recurse_depth=3, ignore_types=[type([])]
+ )
+ self.assertEqual(
+ ["foo", ["foo"], ["foo"], {"foo": "bar", "baz": []}, []], new_list
+ )
# Ignore dict
- old_list = ['foo', ['foo'], ['foo', None], {'foo': 0}, {'foo': 'bar', 'baz': []}, [{'foo': ''}]]
- new_list = salt.utils.data.filter_falsey(old_list, recurse_depth=3, ignore_types=[type({})])
- self.assertEqual(['foo', ['foo'], ['foo'], {}, {'foo': 'bar'}, [{}]], new_list)
+ old_list = [
+ "foo",
+ ["foo"],
+ ["foo", None],
+ {"foo": 0},
+ {"foo": "bar", "baz": []},
+ [{"foo": ""}],
+ ]
+ new_list = salt.utils.data.filter_falsey(
+ old_list, recurse_depth=3, ignore_types=[type({})]
+ )
+ self.assertEqual(["foo", ["foo"], ["foo"], {}, {"foo": "bar"}, [{}]], new_list)
# Ignore NoneType
- old_list = ['foo', ['foo'], ['foo', None], {'foo': 0}, {'foo': 'bar', 'baz': []}, [{'foo': ''}]]
- new_list = salt.utils.data.filter_falsey(old_list, recurse_depth=3, ignore_types=[type(None)])
- self.assertEqual(['foo', ['foo'], ['foo', None], {'foo': 'bar'}], new_list)
+ old_list = [
+ "foo",
+ ["foo"],
+ ["foo", None],
+ {"foo": 0},
+ {"foo": "bar", "baz": []},
+ [{"foo": ""}],
+ ]
+ new_list = salt.utils.data.filter_falsey(
+ old_list, recurse_depth=3, ignore_types=[type(None)]
+ )
+ self.assertEqual(["foo", ["foo"], ["foo", None], {"foo": "bar"}], new_list)
class FilterRecursiveDiff(TestCase):
- '''
+ """
Test suite for salt.utils.data.recursive_diff
- '''
+ """
def test_list_equality(self):
- '''
+ """
Test cases where equal lists are compared.
- '''
+ """
test_list = [0, 1, 2]
self.assertEqual({}, salt.utils.data.recursive_diff(test_list, test_list))
@@ -895,392 +1019,455 @@ class FilterRecursiveDiff(TestCase):
self.assertEqual({}, salt.utils.data.recursive_diff(test_list, test_list))
def test_dict_equality(self):
- '''
+ """
Test cases where equal dicts are compared.
- '''
- test_dict = {'foo': 'bar', 'bar': {'baz': {'qux': 'quux'}}, 'frop': 0}
+ """
+ test_dict = {"foo": "bar", "bar": {"baz": {"qux": "quux"}}, "frop": 0}
self.assertEqual({}, salt.utils.data.recursive_diff(test_dict, test_dict))
def test_ordereddict_equality(self):
- '''
+ """
Test cases where equal OrderedDicts are compared.
- '''
- test_dict = OrderedDict([
- ('foo', 'bar'),
- ('bar', OrderedDict([('baz', OrderedDict([('qux', 'quux')]))])),
- ('frop', 0)])
+ """
+ test_dict = OrderedDict(
+ [
+ ("foo", "bar"),
+ ("bar", OrderedDict([("baz", OrderedDict([("qux", "quux")]))])),
+ ("frop", 0),
+ ]
+ )
self.assertEqual({}, salt.utils.data.recursive_diff(test_dict, test_dict))
def test_mixed_equality(self):
- '''
+ """
Test cases where mixed nested lists and dicts are compared.
- '''
+ """
test_data = {
- 'foo': 'bar',
- 'baz': [0, 1, 2],
- 'bar': {'baz': [{'qux': 'quux'}, {'froop', 0}]}
+ "foo": "bar",
+ "baz": [0, 1, 2],
+ "bar": {"baz": [{"qux": "quux"}, {"froop", 0}]},
}
self.assertEqual({}, salt.utils.data.recursive_diff(test_data, test_data))
def test_set_equality(self):
- '''
+ """
Test cases where equal sets are compared.
- '''
- test_set = set([0, 1, 2, 3, 'foo'])
+ """
+ test_set = {0, 1, 2, 3, "foo"}
self.assertEqual({}, salt.utils.data.recursive_diff(test_set, test_set))
# This is a bit of an oddity, as python seems to sort the sets in memory
# so both sets end up with the same ordering (0..3).
- set_one = set([0, 1, 2, 3])
- set_two = set([3, 2, 1, 0])
+ set_one = {0, 1, 2, 3}
+ set_two = {3, 2, 1, 0}
self.assertEqual({}, salt.utils.data.recursive_diff(set_one, set_two))
def test_tuple_equality(self):
- '''
+ """
Test cases where equal tuples are compared.
- '''
- test_tuple = (0, 1, 2, 3, 'foo')
+ """
+ test_tuple = (0, 1, 2, 3, "foo")
self.assertEqual({}, salt.utils.data.recursive_diff(test_tuple, test_tuple))
def test_list_inequality(self):
- '''
+ """
Test cases where two inequal lists are compared.
- '''
+ """
list_one = [0, 1, 2]
- list_two = ['foo', 'bar', 'baz']
- expected_result = {'old': list_one, 'new': list_two}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(list_one, list_two))
- expected_result = {'new': list_one, 'old': list_two}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(list_two, list_one))
-
- list_one = [0, 'foo', 1, 'bar']
- list_two = [1, 'foo', 1, 'qux']
- expected_result = {'old': [0, 'bar'], 'new': [1, 'qux']}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(list_one, list_two))
- expected_result = {'new': [0, 'bar'], 'old': [1, 'qux']}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(list_two, list_one))
+ list_two = ["foo", "bar", "baz"]
+ expected_result = {"old": list_one, "new": list_two}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(list_one, list_two)
+ )
+ expected_result = {"new": list_one, "old": list_two}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(list_two, list_one)
+ )
+
+ list_one = [0, "foo", 1, "bar"]
+ list_two = [1, "foo", 1, "qux"]
+ expected_result = {"old": [0, "bar"], "new": [1, "qux"]}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(list_one, list_two)
+ )
+ expected_result = {"new": [0, "bar"], "old": [1, "qux"]}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(list_two, list_one)
+ )
list_one = [0, 1, [2, 3]]
- list_two = [0, 1, ['foo', 'bar']]
- expected_result = {'old': [[2, 3]], 'new': [['foo', 'bar']]}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(list_one, list_two))
- expected_result = {'new': [[2, 3]], 'old': [['foo', 'bar']]}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(list_two, list_one))
+ list_two = [0, 1, ["foo", "bar"]]
+ expected_result = {"old": [[2, 3]], "new": [["foo", "bar"]]}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(list_one, list_two)
+ )
+ expected_result = {"new": [[2, 3]], "old": [["foo", "bar"]]}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(list_two, list_one)
+ )
def test_dict_inequality(self):
- '''
+ """
Test cases where two inequal dicts are compared.
- '''
- dict_one = {'foo': 1, 'bar': 2, 'baz': 3}
- dict_two = {'foo': 2, 1: 'bar', 'baz': 3}
- expected_result = {'old': {'foo': 1, 'bar': 2}, 'new': {'foo': 2, 1: 'bar'}}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(dict_one, dict_two))
- expected_result = {'new': {'foo': 1, 'bar': 2}, 'old': {'foo': 2, 1: 'bar'}}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(dict_two, dict_one))
-
- dict_one = {'foo': {'bar': {'baz': 1}}}
- dict_two = {'foo': {'qux': {'baz': 1}}}
- expected_result = {'old': dict_one, 'new': dict_two}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(dict_one, dict_two))
- expected_result = {'new': dict_one, 'old': dict_two}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(dict_two, dict_one))
+ """
+ dict_one = {"foo": 1, "bar": 2, "baz": 3}
+ dict_two = {"foo": 2, 1: "bar", "baz": 3}
+ expected_result = {"old": {"foo": 1, "bar": 2}, "new": {"foo": 2, 1: "bar"}}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(dict_one, dict_two)
+ )
+ expected_result = {"new": {"foo": 1, "bar": 2}, "old": {"foo": 2, 1: "bar"}}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(dict_two, dict_one)
+ )
+
+ dict_one = {"foo": {"bar": {"baz": 1}}}
+ dict_two = {"foo": {"qux": {"baz": 1}}}
+ expected_result = {"old": dict_one, "new": dict_two}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(dict_one, dict_two)
+ )
+ expected_result = {"new": dict_one, "old": dict_two}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(dict_two, dict_one)
+ )
def test_ordereddict_inequality(self):
- '''
+ """
Test cases where two inequal OrderedDicts are compared.
- '''
- odict_one = OrderedDict([('foo', 'bar'), ('bar', 'baz')])
- odict_two = OrderedDict([('bar', 'baz'), ('foo', 'bar')])
- expected_result = {'old': odict_one, 'new': odict_two}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(odict_one, odict_two))
+ """
+ odict_one = OrderedDict([("foo", "bar"), ("bar", "baz")])
+ odict_two = OrderedDict([("bar", "baz"), ("foo", "bar")])
+ expected_result = {"old": odict_one, "new": odict_two}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(odict_one, odict_two)
+ )
def test_set_inequality(self):
- '''
+ """
Test cases where two inequal sets are compared.
Tricky as the sets are compared zipped, so shuffled sets of equal values
are considered different.
- '''
- set_one = set([0, 1, 2, 4])
- set_two = set([0, 1, 3, 4])
- expected_result = {'old': set([2]), 'new': set([3])}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(set_one, set_two))
- expected_result = {'new': set([2]), 'old': set([3])}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(set_two, set_one))
+ """
+ set_one = {0, 1, 2, 4}
+ set_two = {0, 1, 3, 4}
+ expected_result = {"old": {2}, "new": {3}}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(set_one, set_two)
+ )
+ expected_result = {"new": {2}, "old": {3}}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(set_two, set_one)
+ )
# It is unknown how different python versions will store sets in memory.
# Python 2.7 seems to sort it (i.e. set_one below becomes {0, 1, 'foo', 'bar'}
# However Python 3.6.8 stores it differently each run.
# So just test for "not equal" here.
- set_one = set([0, 'foo', 1, 'bar'])
- set_two = set(['foo', 1, 'bar', 2])
+ set_one = {0, "foo", 1, "bar"}
+ set_two = {"foo", 1, "bar", 2}
expected_result = {}
- self.assertNotEqual(expected_result, salt.utils.data.recursive_diff(set_one, set_two))
+ self.assertNotEqual(
+ expected_result, salt.utils.data.recursive_diff(set_one, set_two)
+ )
def test_mixed_inequality(self):
- '''
+ """
Test cases where two mixed dicts/iterables that are different are compared.
- '''
- dict_one = {'foo': [1, 2, 3]}
- dict_two = {'foo': [3, 2, 1]}
- expected_result = {'old': {'foo': [1, 3]}, 'new': {'foo': [3, 1]}}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(dict_one, dict_two))
- expected_result = {'new': {'foo': [1, 3]}, 'old': {'foo': [3, 1]}}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(dict_two, dict_one))
-
- list_one = [1, 2, {'foo': ['bar', {'foo': 1, 'bar': 2}]}]
- list_two = [3, 4, {'foo': ['qux', {'foo': 1, 'bar': 2}]}]
- expected_result = {'old': [1, 2, {'foo': ['bar']}], 'new': [3, 4, {'foo': ['qux']}]}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(list_one, list_two))
- expected_result = {'new': [1, 2, {'foo': ['bar']}], 'old': [3, 4, {'foo': ['qux']}]}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(list_two, list_one))
-
- mixed_one = {'foo': set([0, 1, 2]), 'bar': [0, 1, 2]}
- mixed_two = {'foo': set([1, 2, 3]), 'bar': [1, 2, 3]}
+ """
+ dict_one = {"foo": [1, 2, 3]}
+ dict_two = {"foo": [3, 2, 1]}
+ expected_result = {"old": {"foo": [1, 3]}, "new": {"foo": [3, 1]}}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(dict_one, dict_two)
+ )
+ expected_result = {"new": {"foo": [1, 3]}, "old": {"foo": [3, 1]}}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(dict_two, dict_one)
+ )
+
+ list_one = [1, 2, {"foo": ["bar", {"foo": 1, "bar": 2}]}]
+ list_two = [3, 4, {"foo": ["qux", {"foo": 1, "bar": 2}]}]
expected_result = {
- 'old': {'foo': set([0]), 'bar': [0, 1, 2]},
- 'new': {'foo': set([3]), 'bar': [1, 2, 3]}
+ "old": [1, 2, {"foo": ["bar"]}],
+ "new": [3, 4, {"foo": ["qux"]}],
}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(mixed_one, mixed_two))
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(list_one, list_two)
+ )
+ expected_result = {
+ "new": [1, 2, {"foo": ["bar"]}],
+ "old": [3, 4, {"foo": ["qux"]}],
+ }
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(list_two, list_one)
+ )
+
+ mixed_one = {"foo": {0, 1, 2}, "bar": [0, 1, 2]}
+ mixed_two = {"foo": {1, 2, 3}, "bar": [1, 2, 3]}
expected_result = {
- 'new': {'foo': set([0]), 'bar': [0, 1, 2]},
- 'old': {'foo': set([3]), 'bar': [1, 2, 3]}
+ "old": {"foo": {0}, "bar": [0, 1, 2]},
+ "new": {"foo": {3}, "bar": [1, 2, 3]},
}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(mixed_two, mixed_one))
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(mixed_one, mixed_two)
+ )
+ expected_result = {
+ "new": {"foo": {0}, "bar": [0, 1, 2]},
+ "old": {"foo": {3}, "bar": [1, 2, 3]},
+ }
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(mixed_two, mixed_one)
+ )
def test_tuple_inequality(self):
- '''
+ """
Test cases where two tuples that are different are compared.
- '''
+ """
tuple_one = (1, 2, 3)
tuple_two = (3, 2, 1)
- expected_result = {'old': (1, 3), 'new': (3, 1)}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(tuple_one, tuple_two))
+ expected_result = {"old": (1, 3), "new": (3, 1)}
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(tuple_one, tuple_two)
+ )
def test_list_vs_set(self):
- '''
+ """
Test case comparing a list with a set, will be compared unordered.
- '''
+ """
mixed_one = [1, 2, 3]
- mixed_two = set([3, 2, 1])
+ mixed_two = {3, 2, 1}
expected_result = {}
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(mixed_one, mixed_two))
- self.assertEqual(expected_result, salt.utils.data.recursive_diff(mixed_two, mixed_one))
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(mixed_one, mixed_two)
+ )
+ self.assertEqual(
+ expected_result, salt.utils.data.recursive_diff(mixed_two, mixed_one)
+ )
def test_dict_vs_ordereddict(self):
- '''
+ """
Test case comparing a dict with an ordereddict, will be compared unordered.
- '''
- test_dict = {'foo': 'bar', 'bar': 'baz'}
- test_odict = OrderedDict([('foo', 'bar'), ('bar', 'baz')])
+ """
+ test_dict = {"foo": "bar", "bar": "baz"}
+ test_odict = OrderedDict([("foo", "bar"), ("bar", "baz")])
self.assertEqual({}, salt.utils.data.recursive_diff(test_dict, test_odict))
self.assertEqual({}, salt.utils.data.recursive_diff(test_odict, test_dict))
- test_odict2 = OrderedDict([('bar', 'baz'), ('foo', 'bar')])
+ test_odict2 = OrderedDict([("bar", "baz"), ("foo", "bar")])
self.assertEqual({}, salt.utils.data.recursive_diff(test_dict, test_odict2))
self.assertEqual({}, salt.utils.data.recursive_diff(test_odict2, test_dict))
def test_list_ignore_ignored(self):
- '''
+ """
Test case comparing two lists with ignore-list supplied (which is not used
when comparing lists).
- '''
+ """
list_one = [1, 2, 3]
list_two = [3, 2, 1]
- expected_result = {'old': [1, 3], 'new': [3, 1]}
+ expected_result = {"old": [1, 3], "new": [3, 1]}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(list_one, list_two, ignore_keys=[1, 3])
+ salt.utils.data.recursive_diff(list_one, list_two, ignore_keys=[1, 3]),
)
def test_dict_ignore(self):
- '''
+ """
Test case comparing two dicts with ignore-list supplied.
- '''
- dict_one = {'foo': 1, 'bar': 2, 'baz': 3}
- dict_two = {'foo': 3, 'bar': 2, 'baz': 1}
- expected_result = {'old': {'baz': 3}, 'new': {'baz': 1}}
+ """
+ dict_one = {"foo": 1, "bar": 2, "baz": 3}
+ dict_two = {"foo": 3, "bar": 2, "baz": 1}
+ expected_result = {"old": {"baz": 3}, "new": {"baz": 1}}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(dict_one, dict_two, ignore_keys=['foo'])
+ salt.utils.data.recursive_diff(dict_one, dict_two, ignore_keys=["foo"]),
)
def test_ordereddict_ignore(self):
- '''
+ """
Test case comparing two OrderedDicts with ignore-list supplied.
- '''
- odict_one = OrderedDict([('foo', 1), ('bar', 2), ('baz', 3)])
- odict_two = OrderedDict([('baz', 1), ('bar', 2), ('foo', 3)])
+ """
+ odict_one = OrderedDict([("foo", 1), ("bar", 2), ("baz", 3)])
+ odict_two = OrderedDict([("baz", 1), ("bar", 2), ("foo", 3)])
# The key 'foo' will be ignored, which means the key from the other OrderedDict
# will always be considered "different" since OrderedDicts are compared ordered.
- expected_result = {'old': OrderedDict([('baz', 3)]), 'new': OrderedDict([('baz', 1)])}
+ expected_result = {
+ "old": OrderedDict([("baz", 3)]),
+ "new": OrderedDict([("baz", 1)]),
+ }
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(odict_one, odict_two, ignore_keys=['foo'])
+ salt.utils.data.recursive_diff(odict_one, odict_two, ignore_keys=["foo"]),
)
def test_dict_vs_ordereddict_ignore(self):
- '''
+ """
Test case comparing a dict with an OrderedDict with ignore-list supplied.
- '''
- dict_one = {'foo': 1, 'bar': 2, 'baz': 3}
- odict_two = OrderedDict([('foo', 3), ('bar', 2), ('baz', 1)])
- expected_result = {'old': {'baz': 3}, 'new': OrderedDict([('baz', 1)])}
+ """
+ dict_one = {"foo": 1, "bar": 2, "baz": 3}
+ odict_two = OrderedDict([("foo", 3), ("bar", 2), ("baz", 1)])
+ expected_result = {"old": {"baz": 3}, "new": OrderedDict([("baz", 1)])}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(dict_one, odict_two, ignore_keys=['foo'])
+ salt.utils.data.recursive_diff(dict_one, odict_two, ignore_keys=["foo"]),
)
def test_mixed_nested_ignore(self):
- '''
+ """
Test case comparing mixed, nested items with ignore-list supplied.
- '''
- dict_one = {'foo': [1], 'bar': {'foo': 1, 'bar': 2}, 'baz': 3}
- dict_two = {'foo': [2], 'bar': {'foo': 3, 'bar': 2}, 'baz': 1}
- expected_result = {'old': {'baz': 3}, 'new': {'baz': 1}}
+ """
+ dict_one = {"foo": [1], "bar": {"foo": 1, "bar": 2}, "baz": 3}
+ dict_two = {"foo": [2], "bar": {"foo": 3, "bar": 2}, "baz": 1}
+ expected_result = {"old": {"baz": 3}, "new": {"baz": 1}}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(dict_one, dict_two, ignore_keys=['foo'])
+ salt.utils.data.recursive_diff(dict_one, dict_two, ignore_keys=["foo"]),
)
def test_ordered_dict_unequal_length(self):
- '''
+ """
Test case comparing two OrderedDicts of unequal length.
- '''
- odict_one = OrderedDict([('foo', 1), ('bar', 2), ('baz', 3)])
- odict_two = OrderedDict([('foo', 1), ('bar', 2)])
- expected_result = {'old': OrderedDict([('baz', 3)]), 'new': {}}
+ """
+ odict_one = OrderedDict([("foo", 1), ("bar", 2), ("baz", 3)])
+ odict_two = OrderedDict([("foo", 1), ("bar", 2)])
+ expected_result = {"old": OrderedDict([("baz", 3)]), "new": {}}
self.assertEqual(
- expected_result,
- salt.utils.data.recursive_diff(odict_one, odict_two)
+ expected_result, salt.utils.data.recursive_diff(odict_one, odict_two)
)
def test_list_unequal_length(self):
- '''
+ """
Test case comparing two lists of unequal length.
- '''
+ """
list_one = [1, 2, 3]
list_two = [1, 2, 3, 4]
- expected_result = {'old': [], 'new': [4]}
+ expected_result = {"old": [], "new": [4]}
self.assertEqual(
- expected_result,
- salt.utils.data.recursive_diff(list_one, list_two)
+ expected_result, salt.utils.data.recursive_diff(list_one, list_two)
)
def test_set_unequal_length(self):
- '''
+ """
Test case comparing two sets of unequal length.
This does not do anything special, as it is unordered.
- '''
- set_one = set([1, 2, 3])
- set_two = set([4, 3, 2, 1])
- expected_result = {'old': set([]), 'new': set([4])}
+ """
+ set_one = {1, 2, 3}
+ set_two = {4, 3, 2, 1}
+ expected_result = {"old": set(), "new": {4}}
self.assertEqual(
- expected_result,
- salt.utils.data.recursive_diff(set_one, set_two)
+ expected_result, salt.utils.data.recursive_diff(set_one, set_two)
)
def test_tuple_unequal_length(self):
- '''
+ """
Test case comparing two tuples of unequal length.
This should be the same as comparing two ordered lists.
- '''
+ """
tuple_one = (1, 2, 3)
tuple_two = (1, 2, 3, 4)
- expected_result = {'old': (), 'new': (4,)}
+ expected_result = {"old": (), "new": (4,)}
self.assertEqual(
- expected_result,
- salt.utils.data.recursive_diff(tuple_one, tuple_two)
+ expected_result, salt.utils.data.recursive_diff(tuple_one, tuple_two)
)
def test_list_unordered(self):
- '''
+ """
Test case comparing two lists unordered.
- '''
+ """
list_one = [1, 2, 3, 4]
list_two = [4, 3, 2]
- expected_result = {'old': [1], 'new': []}
+ expected_result = {"old": [1], "new": []}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(list_one, list_two, ignore_order=True)
+ salt.utils.data.recursive_diff(list_one, list_two, ignore_order=True),
)
def test_mixed_nested_unordered(self):
- '''
+ """
Test case comparing nested dicts/lists unordered.
- '''
- dict_one = {'foo': {'bar': [1, 2, 3]}, 'bar': [{'foo': 4}, 0]}
- dict_two = {'foo': {'bar': [3, 2, 1]}, 'bar': [0, {'foo': 4}]}
+ """
+ dict_one = {"foo": {"bar": [1, 2, 3]}, "bar": [{"foo": 4}, 0]}
+ dict_two = {"foo": {"bar": [3, 2, 1]}, "bar": [0, {"foo": 4}]}
expected_result = {}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(dict_one, dict_two, ignore_order=True)
+ salt.utils.data.recursive_diff(dict_one, dict_two, ignore_order=True),
)
expected_result = {
- 'old': {'foo': {'bar': [1, 3]}, 'bar': [{'foo': 4}, 0]},
- 'new': {'foo': {'bar': [3, 1]}, 'bar': [0, {'foo': 4}]},
+ "old": {"foo": {"bar": [1, 3]}, "bar": [{"foo": 4}, 0]},
+ "new": {"foo": {"bar": [3, 1]}, "bar": [0, {"foo": 4}]},
}
self.assertEqual(
- expected_result,
- salt.utils.data.recursive_diff(dict_one, dict_two)
+ expected_result, salt.utils.data.recursive_diff(dict_one, dict_two)
)
def test_ordered_dict_unordered(self):
- '''
+ """
Test case comparing OrderedDicts unordered.
- '''
- odict_one = OrderedDict([('foo', 1), ('bar', 2), ('baz', 3)])
- odict_two = OrderedDict([('baz', 3), ('bar', 2), ('foo', 1)])
+ """
+ odict_one = OrderedDict([("foo", 1), ("bar", 2), ("baz", 3)])
+ odict_two = OrderedDict([("baz", 3), ("bar", 2), ("foo", 1)])
expected_result = {}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(odict_one, odict_two, ignore_order=True)
+ salt.utils.data.recursive_diff(odict_one, odict_two, ignore_order=True),
)
def test_ignore_missing_keys_dict(self):
- '''
+ """
Test case ignoring missing keys on a comparison of dicts.
- '''
- dict_one = {'foo': 1, 'bar': 2, 'baz': 3}
- dict_two = {'bar': 3}
- expected_result = {'old': {'bar': 2}, 'new': {'bar': 3}}
+ """
+ dict_one = {"foo": 1, "bar": 2, "baz": 3}
+ dict_two = {"bar": 3}
+ expected_result = {"old": {"bar": 2}, "new": {"bar": 3}}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(dict_one, dict_two, ignore_missing_keys=True)
+ salt.utils.data.recursive_diff(
+ dict_one, dict_two, ignore_missing_keys=True
+ ),
)
def test_ignore_missing_keys_ordered_dict(self):
- '''
+ """
Test case not ignoring missing keys on a comparison of OrderedDicts.
- '''
- odict_one = OrderedDict([('foo', 1), ('bar', 2), ('baz', 3)])
- odict_two = OrderedDict([('bar', 3)])
- expected_result = {'old': odict_one, 'new': odict_two}
+ """
+ odict_one = OrderedDict([("foo", 1), ("bar", 2), ("baz", 3)])
+ odict_two = OrderedDict([("bar", 3)])
+ expected_result = {"old": odict_one, "new": odict_two}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(odict_one, odict_two, ignore_missing_keys=True)
+ salt.utils.data.recursive_diff(
+ odict_one, odict_two, ignore_missing_keys=True
+ ),
)
def test_ignore_missing_keys_recursive(self):
- '''
+ """
Test case ignoring missing keys on a comparison of nested dicts.
- '''
- dict_one = {'foo': {'bar': 2, 'baz': 3}}
- dict_two = {'foo': {'baz': 3}}
+ """
+ dict_one = {"foo": {"bar": 2, "baz": 3}}
+ dict_two = {"foo": {"baz": 3}}
expected_result = {}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(dict_one, dict_two, ignore_missing_keys=True)
+ salt.utils.data.recursive_diff(
+ dict_one, dict_two, ignore_missing_keys=True
+ ),
)
# Compare from dict-in-dict
dict_two = {}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(dict_one, dict_two, ignore_missing_keys=True)
+ salt.utils.data.recursive_diff(
+ dict_one, dict_two, ignore_missing_keys=True
+ ),
)
# Compare from dict-in-list
- dict_one = {'foo': ['bar', {'baz': 3}]}
- dict_two = {'foo': ['bar', {}]}
+ dict_one = {"foo": ["bar", {"baz": 3}]}
+ dict_two = {"foo": ["bar", {}]}
self.assertEqual(
expected_result,
- salt.utils.data.recursive_diff(dict_one, dict_two, ignore_missing_keys=True)
+ salt.utils.data.recursive_diff(
+ dict_one, dict_two, ignore_missing_keys=True
+ ),
)
diff --git a/tests/unit/utils/test_xmlutil.py b/tests/unit/utils/test_xmlutil.py
index c04f39498e..cbf73861e5 100644
--- a/tests/unit/utils/test_xmlutil.py
+++ b/tests/unit/utils/test_xmlutil.py
@@ -1,148 +1,170 @@
-# -*- coding: utf-8 -*-
-'''
+"""
tests.unit.xmlutil_test
~~~~~~~~~~~~~~~~~~~~
-'''
-from __future__ import absolute_import, print_function, unicode_literals
-# Import Salt Testing libs
-from tests.support.unit import TestCase
+"""
+import salt.utils.xmlutil as xml
# Import Salt libs
from salt._compat import ElementTree as ET
-import salt.utils.xmlutil as xml
+
+# Import Salt Testing libs
+from tests.support.unit import TestCase
class XMLUtilTestCase(TestCase):
- '''
+ """
Tests that salt.utils.xmlutil properly parses XML data and returns as a properly formatted
dictionary. The default method of parsing will ignore attributes and return only the child
items. The full method will include parsing attributes.
- '''
+ """
def setUp(self):
# Populate our use cases for specific XML formats.
self.cases = {
- 'a': {
- 'xml': '<parent>data</parent>',
- 'legacy': {'parent': 'data'},
- 'full': 'data'
+ "a": {
+ "xml": "<parent>data</parent>",
+ "legacy": {"parent": "data"},
+ "full": "data",
},
- 'b': {
- 'xml': '<parent value="data">data</parent>',
- 'legacy': {'parent': 'data'},
- 'full': {'parent': 'data', 'value': 'data'}
+ "b": {
+ "xml": '<parent value="data">data</parent>',
+ "legacy": {"parent": "data"},
+ "full": {"parent": "data", "value": "data"},
},
- 'c': {
- 'xml': '<parent><child>data</child><child value="data">data</child>'
- '<child value="data"/><child/></parent>',
- 'legacy': {'child': ['data', {'child': 'data'}, {'child': None}, {'child': None}]},
- 'full': {'child': ['data', {'child': 'data', 'value': 'data'}, {'value': 'data'}, None]}
+ "c": {
+ "xml": '<parent><child>data</child><child value="data">data</child>'
+ '<child value="data"/><child/></parent>',
+ "legacy": {
+ "child": [
+ "data",
+ {"child": "data"},
+ {"child": None},
+ {"child": None},
+ ]
+ },
+ "full": {
+ "child": [
+ "data",
+ {"child": "data", "value": "data"},
+ {"value": "data"},
+ None,
+ ]
+ },
},
- 'd': {
- 'xml': '<parent value="data" another="data"><child>data</child></parent>',
- 'legacy': {'child': 'data'},
- 'full': {'child': 'data', 'another': 'data', 'value': 'data'}
+ "d": {
+ "xml": '<parent value="data" another="data"><child>data</child></parent>',
+ "legacy": {"child": "data"},
+ "full": {"child": "data", "another": "data", "value": "data"},
},
- 'e': {
- 'xml': '<parent value="data" another="data"><child value="data">data</child></parent>',
- 'legacy': {'child': 'data'},
- 'full': {'child': {'child': 'data', 'value': 'data'}, 'another': 'data', 'value': 'data'}
+ "e": {
+ "xml": '<parent value="data" another="data"><child value="data">data</child></parent>',
+ "legacy": {"child": "data"},
+ "full": {
+ "child": {"child": "data", "value": "data"},
+ "another": "data",
+ "value": "data",
+ },
},
- 'f': {
- 'xml': '<parent><child><sub-child value="data">data</sub-child></child>'
- '<child>data</child></parent>',
- 'legacy': {'child': [{'sub-child': 'data'}, {'child': 'data'}]},
- 'full': {'child': [{'sub-child': {'value': 'data', 'sub-child': 'data'}}, 'data']}
+ "f": {
+ "xml": '<parent><child><sub-child value="data">data</sub-child></child>'
+ "<child>data</child></parent>",
+ "legacy": {"child": [{"sub-child": "data"}, {"child": "data"}]},
+ "full": {
+ "child": [
+ {"sub-child": {"value": "data", "sub-child": "data"}},
+ "data",
+ ]
+ },
},
}
def test_xml_case_a(self):
- xmldata = ET.fromstring(self.cases['a']['xml'])
+ xmldata = ET.fromstring(self.cases["a"]["xml"])
defaultdict = xml.to_dict(xmldata)
- self.assertEqual(defaultdict, self.cases['a']['legacy'])
+ self.assertEqual(defaultdict, self.cases["a"]["legacy"])
def test_xml_case_a_legacy(self):
- xmldata = ET.fromstring(self.cases['a']['xml'])
+ xmldata = ET.fromstring(self.cases["a"]["xml"])
defaultdict = xml.to_dict(xmldata, False)
- self.assertEqual(defaultdict, self.cases['a']['legacy'])
+ self.assertEqual(defaultdict, self.cases["a"]["legacy"])
def test_xml_case_a_full(self):
- xmldata = ET.fromstring(self.cases['a']['xml'])
+ xmldata = ET.fromstring(self.cases["a"]["xml"])
defaultdict = xml.to_dict(xmldata, True)
- self.assertEqual(defaultdict, self.cases['a']['full'])
+ self.assertEqual(defaultdict, self.cases["a"]["full"])
def test_xml_case_b(self):
- xmldata = ET.fromstring(self.cases['b']['xml'])
+ xmldata = ET.fromstring(self.cases["b"]["xml"])
defaultdict = xml.to_dict(xmldata)
- self.assertEqual(defaultdict, self.cases['b']['legacy'])
+ self.assertEqual(defaultdict, self.cases["b"]["legacy"])
def test_xml_case_b_legacy(self):
- xmldata = ET.fromstring(self.cases['b']['xml'])
+ xmldata = ET.fromstring(self.cases["b"]["xml"])
defaultdict = xml.to_dict(xmldata, False)
- self.assertEqual(defaultdict, self.cases['b']['legacy'])
+ self.assertEqual(defaultdict, self.cases["b"]["legacy"])
def test_xml_case_b_full(self):
- xmldata = ET.fromstring(self.cases['b']['xml'])
+ xmldata = ET.fromstring(self.cases["b"]["xml"])
defaultdict = xml.to_dict(xmldata, True)
- self.assertEqual(defaultdict, self.cases['b']['full'])
+ self.assertEqual(defaultdict, self.cases["b"]["full"])
def test_xml_case_c(self):
- xmldata = ET.fromstring(self.cases['c']['xml'])
+ xmldata = ET.fromstring(self.cases["c"]["xml"])
defaultdict = xml.to_dict(xmldata)
- self.assertEqual(defaultdict, self.cases['c']['legacy'])
+ self.assertEqual(defaultdict, self.cases["c"]["legacy"])
def test_xml_case_c_legacy(self):
- xmldata = ET.fromstring(self.cases['c']['xml'])
+ xmldata = ET.fromstring(self.cases["c"]["xml"])
defaultdict = xml.to_dict(xmldata, False)
- self.assertEqual(defaultdict, self.cases['c']['legacy'])
+ self.assertEqual(defaultdict, self.cases["c"]["legacy"])
def test_xml_case_c_full(self):
- xmldata = ET.fromstring(self.cases['c']['xml'])
+ xmldata = ET.fromstring(self.cases["c"]["xml"])
defaultdict = xml.to_dict(xmldata, True)
- self.assertEqual(defaultdict, self.cases['c']['full'])
+ self.assertEqual(defaultdict, self.cases["c"]["full"])
def test_xml_case_d(self):
- xmldata = ET.fromstring(self.cases['d']['xml'])
+ xmldata = ET.fromstring(self.cases["d"]["xml"])
defaultdict = xml.to_dict(xmldata)
- self.assertEqual(defaultdict, self.cases['d']['legacy'])
+ self.assertEqual(defaultdict, self.cases["d"]["legacy"])
def test_xml_case_d_legacy(self):
- xmldata = ET.fromstring(self.cases['d']['xml'])
+ xmldata = ET.fromstring(self.cases["d"]["xml"])
defaultdict = xml.to_dict(xmldata, False)
- self.assertEqual(defaultdict, self.cases['d']['legacy'])
+ self.assertEqual(defaultdict, self.cases["d"]["legacy"])
def test_xml_case_d_full(self):
- xmldata = ET.fromstring(self.cases['d']['xml'])
+ xmldata = ET.fromstring(self.cases["d"]["xml"])
defaultdict = xml.to_dict(xmldata, True)
- self.assertEqual(defaultdict, self.cases['d']['full'])
+ self.assertEqual(defaultdict, self.cases["d"]["full"])
def test_xml_case_e(self):
- xmldata = ET.fromstring(self.cases['e']['xml'])
+ xmldata = ET.fromstring(self.cases["e"]["xml"])
defaultdict = xml.to_dict(xmldata)
- self.assertEqual(defaultdict, self.cases['e']['legacy'])
+ self.assertEqual(defaultdict, self.cases["e"]["legacy"])
def test_xml_case_e_legacy(self):
- xmldata = ET.fromstring(self.cases['e']['xml'])
+ xmldata = ET.fromstring(self.cases["e"]["xml"])
defaultdict = xml.to_dict(xmldata, False)
- self.assertEqual(defaultdict, self.cases['e']['legacy'])
+ self.assertEqual(defaultdict, self.cases["e"]["legacy"])
def test_xml_case_e_full(self):
- xmldata = ET.fromstring(self.cases['e']['xml'])
+ xmldata = ET.fromstring(self.cases["e"]["xml"])
defaultdict = xml.to_dict(xmldata, True)
- self.assertEqual(defaultdict, self.cases['e']['full'])
+ self.assertEqual(defaultdict, self.cases["e"]["full"])
def test_xml_case_f(self):
- xmldata = ET.fromstring(self.cases['f']['xml'])
+ xmldata = ET.fromstring(self.cases["f"]["xml"])
defaultdict = xml.to_dict(xmldata)
- self.assertEqual(defaultdict, self.cases['f']['legacy'])
+ self.assertEqual(defaultdict, self.cases["f"]["legacy"])
def test_xml_case_f_legacy(self):
- xmldata = ET.fromstring(self.cases['f']['xml'])
+ xmldata = ET.fromstring(self.cases["f"]["xml"])
defaultdict = xml.to_dict(xmldata, False)
- self.assertEqual(defaultdict, self.cases['f']['legacy'])
+ self.assertEqual(defaultdict, self.cases["f"]["legacy"])
def test_xml_case_f_full(self):
- xmldata = ET.fromstring(self.cases['f']['xml'])
+ xmldata = ET.fromstring(self.cases["f"]["xml"])
defaultdict = xml.to_dict(xmldata, True)
- self.assertEqual(defaultdict, self.cases['f']['full'])
+ self.assertEqual(defaultdict, self.cases["f"]["full"])
--
2.28.0
