Title: autopkgtest - automatic as-installed package testing
DEP: 8
State: DRAFT
Date: 2011-02-27
Drivers: Ian Jackson <ijackson@chiark.greenend.org.uk>,
 Iustin Pop <iustin@debian.org>,
 Stefano Zacchiroli <zack@debian.org>
URL: http://dep.debian.net/deps/dep8
Source: http://anonscm.debian.org/viewvc/dep/web/deps/dep8.mdwn
License:
 Copying and distribution of this file, with or without modification,
 are permitted in any medium without royalty provided the copyright
 notice and this notice are preserved.
Abstract:
 Establish a standard interface to define and run "as-installed" tests of
 packages, i.e. the testing of packages in a context as close as possible
 to a Debian system where the packages subject to testing are properly
 installed.

Introduction

In the working.

In the meantime you can check the original specification below, which is currently implemented by the autopkgtest package.

---

Appendix A - original specification

Introduction

This document describes how the autopkgtest tester core (the program adt-run) interprets and executes tests found in source packages.

The source package provides a test metadata file debian/tests/control. This is a file containing zero or more RFC822-style stanzas, along these lines:

Tests: fred bill bongo
Restrictions: needs-root breaks-computer

This example defines three tests, called fred, bill and bongo. The tests will be performed by executing debian/tests/fred, debian/tests/bill, etc. Each test program should, on success, exit with status 0 and print nothing to stderr; if a test exits nonzero, or prints to stderr, it is considered to have failed.

The cwd of each test is guaranteed to be the root of the source package which will have been built. HOWEVER note that the tests must test the INSTALLED version of the program. Tests may not modify the source tree (and may not have write access to it).

If the file to be executed has no execute bits set, chmod a+x is applied to it (this means that tests can be added in patches without the need for additional chmod; contrast this with debian/rules).

During execution of the test, the environment variable TMPDIR will point to a directory for the execution of this particular test, which starts empty and will be deleted afterwards (so there is no need for the test to clean up files left there).

Tests must declare all applicable Restrictions - see below.

The fields which may appear in the RFC822-style stanza are:

• Tests: <name-of-test> [<name-of-another-test> ...]

  This field is mandatory. It names the tests which are defined by this stanza. All of the other fields in the same stanza apply to all of the named tests.

  Test names are separated by whitespace and should contain only characters which are legal in package names, plus `/'.

• Restrictions: <restriction-name> [<another-restriction-name> ...]

  Declares some restrictions or problems with the tests defined in this stanza. Depending on the test environment capabilities, user requests, and so on, restrictions can cause tests to be skipped or can cause the test to be run in a different manner. Tests which declare unknown restrictions will be skipped. See below for the defined restrictions.

• Features: <feature-name> [<another-feature-name> ...]

  Declares some additional capabilities or good properties of the tests defined in this stanza. Any unknown features declared will be completely ignored. See below for the defined features.

• Depends: <dpkg dependcy field syntax>

  Declares that the specified packages must be installed for the test to go ahead. @ stands for the package(s) generated by the source package containing the tests; each dependency (strictly, or-clause, which may contain |-s but not commas) containing @ is replicated once for each such binary package, with the binary package name substituted for each @ (but normally @ should occur only once and without a version restriction).

  If no Depends field is present, Depends: @ is assumed. Note that the source tree's Build-Dependencies are not necessarily installed, and if you specify any Depends, no binary packages from the source are installed unless explicitly requested.

• Tests-Directory: <path>

  Replaces the path segment debian/tests in the filenames of the test programs with . Ie, the tests are run by executing /path/to/built/source/tree/<path>/<name-of-test>. <path> must be a relative path and is interpreted starting from the root of the built source tree.

  This allows tests to live outside the debian/ metadata area, so that they can more palatably be shared with non-Debian distributions.

Any unknown fields will cause the whole stanza to be skipped.

Restrictions

The defined Restrictions are:

• rw-build-tree

  The test(s) needs write access to the built source tree (so it may need to be copied first). Even with this restriction, the test is not allowed to make any change to the built source tree which (i) isn't cleaned up by debian/rules clean, (ii) affects the future results of any test, or (iii) affects binary packages produced by the build tree in the future.

• breaks-testbed

  The test, when run, is liable to break the testbed system. This includes causing data loss, causing services that the machine is running to malfunction, or permanently disabling services; it does not include causing services on the machine to temporarily fail.

  When this restriction is present the test will usually be skipped unless the testbed's virtualisation arrangements are sufficiently powerful, or alternatively if the user explicitly requests.

• needs-root

  The test script must be run as root.

Features

The currently defined Features are:

• no-build-needed

  The tests can run in an unbuilt tree.