| tocdepth: | 2 |
|---|
1 Docker Autotest |version|
.. toctree:: :hidden: :numbered: defaults subtests additional
Docker Autotest is a sub-framework for standalone testing of docker. It does not depend on docker itself. Functionally, testing occurs within any number of subtest modules, which in some cases also include further nested sub-subtests. It is designed to support both extremely simple, linear, and more complex, nested, iterative testing of arbitrary external docker commands.
Test content and configuration is fully modular, supporting both included and external tests. The included content is primarily focused on continuous-integration testing of the Docker CLI client, with a very limited amount of negative and stress testing.
As with other Autotest Client Tests the main entry point is the test control file. Docker Autotest's control file utilizes Autotest's steps-engine. This allows the overall testing state to be stored, recovered, and resumed should a host-kernel panic or if userspace become unresponsive. Consequently, rebooting the host system as a testing step is also supported.
Docker
- Clean environment, only test-related images, containers
and dependent services (besides
docker -d) at the start of every Autotest run. - Docker installation (with docker daemon running)
- Default settings for Docker on your platform/OS, including storage (LVM thin-pool + XFS).
- Sufficient available temporary storage for multiple copies of the test-related images/container content.
- Clean environment, only test-related images, containers
and dependent services (besides
-
- Red Hat Enterprise Linux 7 Server (preferred for development)
- Red Hat Enterprise Linux Atomic Host (preferred for testing)
- Fedora 22 or later including Atomic (not all tests may function)
- Other platforms (such as CentOs, SLES, Ubuntu) un-maintained but possible.
Platform Applications/tools
- Autotest 0.16.0 or later.
- Coreutils or equivalent (i.e.
cat,mkdir,tee, etc.) - Tar and supported compression programs (
bzip2,gzip, etc.) - nfs-utils (nfs-server support daemons)
- Python 2.6 or greater (not 3.0)
- libselinux-python 2.2 or later
- Optional (for building documentation),
make,python-sphinx, anddocutilsor equivalent for your platform. - Optional (for running unittests),
pylint,pep8,python-unittest2, andpython2-mock. - Optional,
python-bugzillafor test exclusion based upon bug status. Seecontrol.inifile comments for details.
Any specific requirements for particular subtest modules. In particular:
- Direct root access for running docker, restarting the daemon, and accessing metadata / storage details. Running through sudo may work to a degree, but is likely to cause problems.
- Access to a remote (to the host) image registry via
docker pull,http,https, andgit. - External testing dependencies must be usable by tests w/in fixed timeout periods. If an external resource or connection is too slow, it should be made more network-local to the test host.
- Most tests with external dependencies will have them flagged as
values to the special
__example__option. See example values for more details.
Double-check you meet all the requirements in docker_autotest_prereq. For the quickstart, either a RHEL 7 or Fedora 25 system is assumed with the Docker daemon started, device-mapper or overlay2 storage configured, and at least 10gig of registry space is available.
As
root, shallow-clone Autotest (non-recursive) into/var/lib/autotest, and set/export theAUTOTEST_PATHenvironment variable to it's location.[root@docker ~]# cd /var/lib [root@docker lib]# git clone --single-branch --depth 1 \ https://github.com/autotest/autotest.git autotest [root@docker lib]# export AUTOTEST_PATH=/var/lib/autotest
Change to the autotest client subdirectory.
Clone autotest-docker repository into the
dockersubdirectory. Based from a formal release or the latest available.[root@docker lib]# cd $AUTOTEST_PATH/client [root@docker client]# git clone --branch $VERSION \ https://github.com/autotest/autotest-docker.git tests/docker
Where
$VERSIONis the docker-autotest release (e.g. "|version|") or to use the current latest release, omit the --branch option:[root@docker client]# git clone \ https://github.com/autotest/autotest-docker.git docker
Make a copy of default configuration, then edit as appropriate. Particularly, verify the CSV list of full-image names and container names,
preserve_fqinsandpreserve_cnamesare set correctly. All other images/containers will be destroyed! See default configuration options for more details.[root@docker client]# cp -abi tests/docker/config_defaults/defaults.ini \ tests/docker/config_custom/ [root@docker client]# $EDITOR tests/docker/config_custom/defaults.ini
The Fedora base-images lack some essential tooling necessary for testing. In addition to the steps above, a custom test-image must be configured for building.
Edit the
defaults.inifile again, and change the registry settings as follows:[root@docker client]# $EDITOR tests/docker/config_custom/defaults.ini ... docker_registry_host = docker_registry_user = docker_repo_name = fedora_test_image docker_repo_tag = latest ...
Make a copy of the
docker_test_images.iniconfiguration file and configure it to build the test image.[root@docker client]# cp -abi tests/docker/config_defaults/docker_test_images.ini \ tests/docker/config_custom/ [root@docker client]# $EDITOR tests/docker/config_custom/docker_test_images.ini ... build_name = fedora_test_image:latest build_dockerfile = https://github.com/autotest/autotest-docker/raw/master/fedora_test_image.tar.gz build_opts_csv = --no-cache,--pull,--force-rm ...
For all platforms, use the standalone autotest client to select and execute subtests.
The default behavior is to run all subtests. However, the example below only
executes the version subtest for demonstration purposes. This will bring in
some additional utility "tests", such as docker_test_images and garbage_check.
Other subtests may be selected via the --args parameter or by customizing
control.ini.
[root@docker /]# cd $AUTOTEST_PATH/client
[root@docker client]# ./autotest-local tests/docker/control --args=docker_cli/version
Writing results to /var/lib/autotest/client/results/default
START ---- ----
Subtest/Sub-subtest requested:
'docker_cli/version'
Subtest/sub-subtest exclude list:
'subexample'
'pretest_example'
'example'
'intratest_example'
'posttest_example'
Executing tests:
'docker/pretests/docker_test_images.1'
'docker/pretests/log_sysconfig.2'
'docker/pretests/log_versions.3'
'docker/subtests/docker_cli/version.4'
'docker/intratests/garbage_check.4'
START docker/pretests/docker_test_images.1
docker_test_images: initialize()
docker_test_images: setup() for subtest version 2055
docker_test_images: Running sub-subtests...
puller: initialize()
puller: run_once()
puller: Pulling registry.access.redhat.com/rhel7/rhel:latest
puller: Pulling docker.io/stackbrew/centos:latest
puller: postprocess()
puller: cleanup()
builder: initialize()
builder: run_once()
builder: postprocess()
builder: cleanup()
docker_test_images: postprocess_iteration() #0 of #1
docker_test_images: full_name:registry.access.redhat.com/rhel7/rhel:latest
docker_test_images: full_name:docker.io/stackbrew/centos:latest
docker_test_images: Updated preserve_fqins: docker.io/stackbrew/centos:latest,registry.access.redhat.com/rhel7/rhel:latest
docker_test_images: Postprocess sub-subtest results...
docker_test_images: cleanup()
GOOD docker/pretests/docker_test_images.1
END GOOD docker/pretests/docker_test_images.1
START docker/pretests/log_sysconfig.2
log_sysconfig: initialize()
log_sysconfig: setup() for subtest version 0
log_sysconfig: run_once()
log_sysconfig: postprocess_iteration() #0 of #1
log_sysconfig: postprocess()
log_sysconfig: cleanup()
GOOD docker/pretests/log_sysconfig.2
END GOOD docker/pretests/log_sysconfig.2
START docker/pretests/log_versions.3
log_versions: initialize()
log_versions: setup() for subtest version 0
log_versions: run_once()
log_versions: Found docker version client: 1.12.6 server 1.12.6
log_versions: postprocess_iteration() #0 of #1
log_versions: postprocess()
log_versions: cleanup()
GOOD docker/pretests/log_versions.3
END GOOD docker/pretests/log_versions.3
START docker/subtests/docker_cli/version.4
version: initialize()
version: setup() for subtest version 0
version: run_once()
version: postprocess_iteration() #0 of #1
version: postprocess()
version: docker version client: 1.12.6 server 1.12.6
version: Docker cli version matches docker client API version
version: cleanup()
GOOD docker/subtests/docker_cli/version.4
END GOOD docker/subtests/docker_cli/version.4
START docker/intratests/garbage_check.4
GOOD docker/intratests/garbage_check.4
END GOOD docker/intratests/garbage_check.4
END GOOD ---- ----
(timestamps and extra inconsequential text removed for clarity)
Examine the test results by changing to the results/default directory.
Note: The name "default" is used when no --tag option is given to the
autotest-local command.
[root@docker client]# cd $AUTOTEST_PATH/client/results/default [root@docker default]# ls -1 control # Copy of the control file used for the run control.ini # Runtime configuration from default control file control.state # Used to support mid-test reboot / test resumption debug # All the client / sydout/stderr recorded by log-level. docker # Directory-tree of subtest results by name job_report.html # Autogenerated report web-page. status # Text-version of test run / results status.json # Same thing, but in JSON format. sysinfo # Directory of important log-files for the run. [root@docker default]# ls -1 docker/subtests/docker_cli/version.4/ debug # Same as above, but ONLY logs for this subtest keyval # Copy of subtest configuration, including defaults profiling # Not used results # Not used status # Same as above, but ONLY for this subtest sysinfo # Logs captured after this subtest ran.
If you wish jUnit format results, execute the included conversion script.
[root@docker client]# cd $AUTOTEST_PATH/client
[root@docker client]# tests/docker/results2junit --name $HOSTNAME results/default
[root@docker client]# cat results/default/results.junit
<testsuites>
<testsuite name="localhost" failures="0" tests="5" skipped="0" errors="0">
<testcase classname="localhost.pretests" name="docker_test_images" time="29"/>
...
All subtest modules reside beneath the subtest directory. A subtest
module must have the same name as the directory it is in (minus the .py
extension). Other files/directories may exist at that level, but they
will not be recognized as subtest modules by the Docker client test. This
ensures each subtest's code is kept separate from all others. Finally,
every subtest is run in its own process and context. It does not have
visibility into any other subtest code, configuration, or runtime.
The structure/layout of the subtest directory tree is relevant for
reference and configuration. The reference and configuration section names
for subtests are formed by its relative location under the subtest
directory. For example, the subtest module subtests/docker_cli/version/version.py
matches with the [docker_cli/version] configuration section and
docker_cli/version subtest name.
Subtests may source their own static content from within their directory and below.
When content needs to be built, or is in some way test-environment specific, the
setup() method should be overridden. Content may be referenced from the subtest's
directory by using its bindir attribute. A version-specific directory to
contain build/setup output is available as the srcdir attribute. The setup()
method will *only* be called once per version number (including revisions).
State may be reset by clearing the autotest client tmp directory.
| Note: | The setup() method runs after the initialize() method
only once. If the configuration version has not changed, the method
will not be called in subsequent Docker Autotest runs. |
|---|
There are provisions for tests that contain, or are composed of multiple child tests or dependent operations. They may share content and code between each other, so long as it lives within the subtest directory or below. Optionally, they may use their own configuration sections, named by appending their class name onto the parent subtest's name. Sub-subtest configuration inherits undefined values from the parent subtest configuration. Additionally, there are multiple methods of executing sub-subtests depending on needs.
It is assumed that any images required for testing are available and built beforehand. A default image for testing purposes is required by most tests. Its fully-qualified name is configurable, though test results will be directly affected if it is or becomes unavailable. Individual subtests may require specific additional images or content. If so, this will be noted in the subtest documentation's Prerequisites section.
The default configuration files are all located under the config_defaults
subdirectory. These are intended to be bundled with the autotest docker test.
To customize any subtest or global default configuration, copies should
be made manually into the config_custom subdirectory. Any content
within config_custom will override all files and sections from
config_defaults.
The subdirectory structure or relative file locations under config_custom
is irrelevant. Multiple sections may appear in the same file, even for
unrelated tests. The only exception is the config_custom/defaults.ini file
and the test control.ini file.
When customizing subtest and sub-subtest configuration options, it is highly
recommended that you add the option config_version with the current
version number, into each section. This way, you will receive warnings when
updating Docker Autotest, if the specific custom configuration doesn't match
the API. Generally this only happens between major/minor version updates.
Configuration files use the familiar ini style format with separate
sections (e.g. [<section name>]) preventing option names from colliding.
All configuration files are loaded into a single name-space, divided by
each section. Section names can be arbitrary, however those which exactly
match a subtest or sub-subtest's name, will be automatically loaded (see Subtests).
The Default, global values for all sections are located within the
special defaults.ini file's DEFAULTS section. These option
names and values are supplied for all sections which do not contain
a identical named option. This file is loaded either from the
config_defaults or config_custom directory.
Long values may be continued onto the next line by prefixing any run of one or more horizontal-whitespace characters with a newline. For example:
- option_name = This option value is super duper long,
- but will be un-folded into a single string with <---- all this whitespace replaced by a single space.
In this case, the runs of multiple whitespace following the newline will be folded into a single space, and combined with the previous line.
In order to help dockertest operate w/o customization, many example or demonstration
value have been configured. While this is fine for development and informal testing
purposes, it adds external dependencies for production testing. Therefore every
option with default example values should be specified in a
comma-separated list to the special __example__ option.
The __example__ option's value is parsed specially. It is not inherited from
defaults, or from subtest to sub-subtest. Instead, it is compounded at each level
then pruned of modified options (as compared to their default/example value). Any
unmodified options remaining at the beginning of a test will cause loud warnings
to be issued.
Within each section, optional inline-value substitution is supported
using the %(<option>)s token format. Where <option> is the
literal name of another option. The referenced option must reside
in the same section or in the special DEFAULTS section. This is
useful when multiple option values need to be different but contain a
shared element.
The config parser will attempt to parse each item in the following order: integers, booleans, floats, then strings.
- Integers are in the form of simple numbers, eg: "123"
- Booleans are in the form 'yes' or 'true', 'no' or 'false' (case insensitive)
- Floats are in the form of numbers with decimals eg: "123.456" or "123.0"
- All other items will be returned as strings.
All configuration options must be documented somewhere at least once.
Documented options in [DEFAULTS] will automatically be documented
in subtests, no need to re-document. Similarly, for subtests with
multiple sub-subtests, options that are overridden by sub-subtests only
need to be documented in the subtest section.
Configuration options are documented in ReStructuredText format
by prefixing each option with one or more comment lines that begin
with the special sequence #:. Regular line comments (beginning
with just # are not treated specially).
Documentation comments always and only apply to the next option = value
sequence encountered and are otherwise ignored. Further, all lines of
documentation are concatenated together and stripped of newlines and multiple
runs of space characters. This is required to fit every item into a bullet
list. Since newlines and indenting is sometimes required
(for example, a bullet list or table), special substitution sequences may
be embedded w/in the comment text:
- Use the [unbroken] sequence
{n}wherever a newline should be inserted. - Use the [unbroken] sequence
{t}wherever a four-space indent should be inserted. - Use the sequence
{{and}}to escape literal{and}characters.
Every test in autotest makes use of a control file as the single
operational bridge between the Autotest server/client and the underlying
testing details. This arrangement allows tests to ignore higher-level
details, like intra-test reboots, reporting, iteration. Instead, tests
can remain focused on exercising their subjects with a loose set of
input variables an provided initial environment. More details about
control files and their standards can be found in the Autotest documentation.
This is the highest level of abstraction exposed to control files.
The job object is implicitly passed into the control file, and used
to produce test objects. Docker autotest makes use of several standard
facilities provided by the job object:
- The
job.resultdirattribute points to the top-level absolute directory where all results and logs will be recorded. This directory contains a copy of the control file used, as well as logs and results for every test object executed. - The
job.controlattribute points to the absolute path of the control file currently in use. Though primarily for reference purposes, any use of this by lower-level tests can only be on an advisory basis. There is no way for tests to predict which control file will be in use nor what its precise behavior will be. - The
job.argsattribute contains a list of space-separated arguments passed to the control file. This can be via the--argsoption to the Autotest client, or it can come from elsewhere. Other than parsing the value into the list, it's entirely up to the control file and/or tests to do what they need with this facility. - The
job.next_step()method is used by the control file to set up and establish the harness-state for each distinct operation. While the state can be arbitrary, Docker Autotest uses this facility to insert test objects for eventual execution. Each step will ultimately be executed in a forked process, so there is no chance they may directly or accidentally influence each other's python-environment.
Autotest test objects are the primary context which contain testing-behavior and implementation details. In Docker autotest, each Subtest is derived from the autotest test.test class. With some help from the control file, this allows otherwise separate tests to be bundled together and executed in sequence, while sharing some common code. The important details/components of the test interface are available in the Subtest Module section.
The example control file provided with Docker autotest is
designed to locate and setup subtests for execution in a particular
way. However, this is only an example, you may customize or provide
your own control file if alternate behavior is desired.
To help with the complex task of locating and queuing subtests
for execution. The default control file examines two possibly
complementary sets of options. One is via the --args autotest client
command-line option. The other is via the control.ini file. This
file is typically copied from config_defaults and modified in
config_custom.
``--args`` Reference:
- The value is treated as a sequence of space-separated then comma-separated list of values. Any subtests or sub-subtest names listed alone (space separated), or with commas and no intervening spaces, will be added to list of candidates.
- If no candidate subtest or sub-subtest list are specified, the list will be generated by searching for applicable modules under
subtests,pretests,intratestsandposttests.- If a space-separated component of
--argsis of the formi=<name>,<name>,.... Then each<name>will be taken from the candidate list, as a test module to include in the run-queue. Unknown names are ignored with a warning.- Similarly, a space-separated component of the form
x=<thing>,<thing>,...is taken as the list of test modules to exclude from the run-queue. Any conflicts with the include list, will result in the item being excluded.- If the string
!!!appears as any of the space-separated items to--args, then no tests will be executed. Instead, the run-queue will simply be displayed and logged.
``control.ini`` Reference:
- A
config_custom/control.iniwill be loaded in preference toconfig_defaults/control.ini.- The
subthingsoption takes a CSV list of candidate module names, similar to the space-separated sequence from--args(above).- The
pretests,intratestsandposttestsitems specify the relative path to directories to search for candidates if they're not specified in thesubthingslist (or in--args).- The
includeandexcludeCSV lists operate just as expected. Either including or excluding items from the candidate list, into the run-queue.- All the other options are fully documented within the
config_custom/control.inifile.
| Note: | The $AUTOTEST_PATH/client/results/default/control.ini
represents the parsed, run-time copy of the file. It is
consumed and used internally by Docker Autotest and should
be considered read-only. |
|---|
Sometimes tests fail temporarily: docker regression, or new test that requires a particular not-yet-in-the-repos docker build, or disparity between RHEL/CentOS/Fedora. We want to acknowledge those and silence them for future runs, but without heavyhandedness such as hardcoded conditionals in the tests themselves.
The mechanism to do this is the file known_failures.txt, maintained within ADEPT and copied into the autotest config_custom subdirectory before a run. This file contains mappings of known failures in particular subtests when run on particular docker builds. If a docker-autotest subtest fails, but the NVR/subtest combination is listed in this file, the test is marked as passing. Test logs are unaffected, so a reader looking at .DEBUG logs will see the original failure.
The format of this file is:
<NVRA> <subtest path> <human-friendly description>
Where:
- NVRA is a full N-V-R.A of a specific docker or docker-latest
- build. As a special case, R.A can be '*' (asterisk) to indicate that this failure is not likely to be fixed in any non-rebase build of this package.
subtest path is the full name of a subtest or subsubtest
- description is a string intended to be read by humans. Bugzilla
- IDs are especially helpful here. This string will be shown not only when a failure is bypassed but also if there's a NON-bypassed failure
Fields are separated by one or more spaces; this allows the columns to be aligned for readability. The description field is optional but strongly recommended; spaces in it are (of course) not delimiters.
Examples:
docker-1.12.3-10.el7.centos docker_cli/iptable/iptable_remove bz1406460: fixed in 1.12.5
docker-1.12.6-49.1.hotfix.git90e29fe.el7.x86_64 docker_cli/run_volumes/oci_umount New oci-umount.conf contains spaces and comments; docker-autotest fix for this is in review.
docker-1.12.5-* docker_cli/negativeusage/iv4 bz1393572: expect fix in docker-1.13?
In order to support external/private subtests and customized configurations, the Docker Autotest API version has been tightly coupled to test content, configuration, and documentation. Version comparison is only significant for the first two numbers (major and minor). The third (revision) number represents insignificant revisions which do not alter the core test or subtest API.
This allows the API to be extended freely, but any changes
which could affect external tests or custom configurations will cause automatic
test failures when encountered. The most likely cause for version problems is custom
and/or outdated configurations. Double-check any customizations within
config_custom match any changes to the current API. The same goes for any private
or local tests.
Documentation versioning is similarly tied to changes in the API version. While non-fatal, it will introduce a delay in subtest execution. This signal is intended to alert developers a documentation-update pass is required to reflect changes in the API.
The following sections detail specific subtests, their configuration and any prerequisites or setup requirements.
The following section details included pretests, intratests, and posttests
modules and any prerequisites or setup requirements.
1.11 Dockertest |release| API Reference
Covers |release| and all revisions.
- 1.11.1 Dockertest Package
- 1.11.2 Subtest Module
- 1.11.3 Images Module
- 1.11.4 Containers Module
- 1.11.5 Environment Module
- 1.11.6 Networking Module
- 1.11.7 Docker_Daemon Module
- 1.11.8 Dockercmd Module
- 1.11.9 Output Module
- 1.11.10 Xceptions Module
- 1.11.11 Sphinx Conf Module
- 1.11.12 Version Module
- 1.11.13 Configuration Module
- 1.11.14 Documentation Module
.. automodule:: dockertest :no-members: :no-undoc-members:
.. automodule:: dockertest.subtest :members: :no-undoc-members:
.. automodule:: dockertest.images :members: :no-undoc-members:
.. automodule:: dockertest.containers :members: :no-undoc-members:
.. automodule:: dockertest.environment :members: :no-undoc-members:
.. automodule:: dockertest.networking :members: :no-undoc-members:
.. automodule:: dockertest.docker_daemon
:members:
:no-undoc-members:
:no-inherited-members:
.. automodule:: dockertest.dockercmd :members: :no-undoc-members:
.. automodule:: dockertest.output :members: :no-undoc-members:
.. automodule:: dockertest.xceptions :members: :undoc-members:
.. automodule:: conf :members: :undoc-members:
.. automodule:: dockertest.version :members: :no-undoc-members:
.. automodule:: dockertest.config :members: :no-undoc-members:
.. automodule:: dockertest.documentation :members: :no-undoc-members:
When the time is right, a large number of PRs have been merged, and you're in
the right mood, it's time for a new minor release. Assuming the last tagged
version was 0.8.6, here are the steps required:
Make sure your master branch exactly matches upstream:
git remote updategit checkout mastergit reset --hard upstream/masterCreate a new branch off master with the next number:
git checkout -tb 0.8.7Bump the version number up in three places:
$EDITOR config_defaults/defaults.ini dockertest/version.py conf.pyCreate a release commit:
git commit -asm "Dockertest Version 0.8.7 (NO API Changes)"Push to your fork, and open a PR targeting the previous milestone,
0.8.7in this case.After PR is merged, switch back to master, update it, and tag the version, and push.
git remote updategit checkout mastergit reset --hard upstream/mastergit tag 0.8.7 HEADgit push --tags upstreamCreate a new release, using the just tagged version and the closed milestone as the title. e.g. "
Dockertest Version 0.8.7 (NO API Changes)". This will cause github to automatically produce a zip and tarball, with URLs for historical reference or use (i.e. somewheregitis not available).For the release notes (big text box under the title), link to the github comparison URL formed by the last two tags with
...in-between (end of the url):[Changes](https://github.com/autotest/autotest-docker/compare/0.8.6...0.8.7)The actual github comparison page (URL above) can also be used as reference for drafting a release-announcement e-mail, providing brownie-points and karma for all the people who helped out.
Next, close the previous milestone (
0.8.7for this example).Create the next milestone (next version PRs will be attached to), named "
Docker Autotest Version 0.8.8 (NO API Changes)", add a description and due-date if desired.Move any currently open PRs the new milestone (right side pane of each PRs page).
- Docker documentation
- Autotest results specification
- Multi-host synchronization and testing
- reStructuredText spec for docstring and
index.rstupdates - Building docs, and for autodoc extension, see the Sphinx documentation