/openSUSE-release-tools

Tools to aid in staging and release work for openSUSE/SUSE

Primary LanguagePythonGNU General Public License v2.0GPL-2.0

Build Status codecov openSUSE Tumbleweed package

openSUSE-release-tools

This repository contains a set of tools to aid in the process of building, testing and releasing (open)SUSE based distributions and their corresponding maintenance updates. You can find more information in docs/processes.md. The CONTENTS.md file contains a list of the tools that are included in the repository.

Rethink release tooling presentation overview

Everything denoted with a cloud is largely in this repository while the rest is the open-build-service (OBS).

Installation

For non-development usage just install the package.

zypper in openSUSE-release-tools

Many sub-packages are provided which can be found either by searching or looking on the build service.

zypper se openSUSE-release-tools osc-plugin

If CI builds are needed add the appropriate openSUSE:Tools repository.

Usage

All tools provide help documentation accessible via --help.

For osc plugins include the plugin name after osc like the following.

osc staging --help

For other tools execute the tool directly.

osrt-repo-checker --help

See the docs directory or a specific tool directory for specific tool documentation outside of --help. The wiki also contains some additional documentation.

Development

git clone https://github.com/openSUSE/openSUSE-release-tools.git

If working on an osc plugin create symlinks for the plugin and osclib in either ~/.osc-plugins or /usr/lib/osc-plugins. For example to install the staging plugin do the following.

mkdir -p ~/.osc-plugins
ln -sr ./osc-staging.py ./osclib ~/.osc-plugins

It can also be useful to work against a development copy of osc either to utilize new features or to debug/fix functionality. To do so one must place the development copy in the path to be loaded and utilize the wrapper script if working on osc plugins. One method to accomplish this is shown below.

# outside of openSUSE-release-tools checkout
git clone https://github.com/openSUSE/osc.git

# inside openSUSE-release-tools checkout
# note the ending /osc which points to the osc directory within the checkout
ln -s /path/to/osc/osc ./

# to utilize the wrapper for working on osc plugins from osrt checkout
$(realpath ./osc)/../osc-wrapper.py --version

Using Docker Compose, a containerized OBS can be started via one command. The default credentials are Admin and opensuse on 0.0.0.0:3000. You can change the port by setting the environment variable OSRT_EXPOSED_OBS_PORT.

docker-compose -f dist/ci/docker-compose.yml up api

To make things easier, you can add an alias to refer to this instance (do not forget to adjust the port if you are using a different one):

cat <<EOF >> ~/.config/osc/oscrc

[http://0.0.0.0:3000]
user = Admin
pass = opensuse
aliases = local
EOF

Then you can use the new local alias to access this new instance.

osc -A local api /about

Some tests will attempt to run against the local OBS, but not all. It's still recommended to run this through docker-compose (see below)

pytest tests/*.py

Running Continuous Integration

This repository includes all the needed files to set up and run the Continuous Integration test suite. The idea is to use Docker Compose to orchestrate a set of containers, including an OBS instance, and run the tests on top of them. Although they automatically run on GitHub Actions (more on that later), it is easy to run them locally. The following commands must be executed from the root of the repository.

# Mount the current path at the /code directory on the container
sed -i -e "s,../..:,$PWD:," dist/ci/docker-compose.yml

# Run the linter
docker-compose -f dist/ci/docker-compose.yml run flaker

# Run the test full suite (it may take some time)...
docker-compose -f dist/ci/docker-compose.yml run test

# .. or just run a single test (i.e., the 'tests/util_tests.py')
docker-compose -f dist/ci/docker-compose.yml run test run_as_tester pytest tests/util_tests.py

# We are finished. Now you can shut the containers down.
docker-compose -f dist/ci/docker-compose.yml down

The docker-compose.yml mentions two container images that are built in the openSUSE:Tools:Images project:

  • osrt-miniobs-for-ci is the base of OBS-related services (API, caches, SMTP, and so on).
  • osrt-testenv-tumbleweed used to run the tests. The code and the tests are mounted in the /code directory of this container.

As mentioned before, the main repository uses GitHub Actions to automatically run the tests when a pull request is opened or the code is pushed to the master branch. You can find the details in the workflow definition. Note that, in addition to the steps listed before, code coverage data is submitted to Codecov.

Debugging Failures in CI

This section lists a few tricks to debug problems in the CI. You will use your local setup so, as a first step, you need to be able to run the tests as described in the previous section. To see the logs from all the containers, the following command can be executed:

docker-compose -f dist/ci/docker-compose.yml logs -f --tail=10

You can run commands in any container by using the docker-compose exec command. For instance, you can connect to a container through a shell with the following command (in this case, it will connect to the container behind the api service):

docker-compose -f dist/ci/docker-compose.yml exec api sh

Or you could check the API logs by issuing the following command:

docker-compose -f dist/ci/docker-compose.yml exec api sh -c 'tail -f /srv/www/obs/api/log/*.log'

To debug problems in the test suite or in the code, place a breakpoint() call and you will get access to Python's debugger.

You can access your testing OBS instance at http://0.0.0.0:3000 and log in using "Admin" as username and "opensuse" as password. To prevent the data being removed while you are inspecting the OBS instance, you can put a call to the breakpoint() function.

Finally, if you miss anything for debugging, you can use zypper to install it.

Adding tests

Testing the release tools isn't quite trivial as a lot of these tools rely on running openSUSE infrastructure. Some of the workflows we replay (not mock) in above described docker-compose setup. So each test will setup the required projects and e.g. staging workflows in a local containerized OBS installation and then do its assertions. If you want to add coverage, best check existing unit tests in tests/*.py. A generic test case looks similiar to this:

class TestExample(unittest.TestCase):

def test_basic(self):
    # Keep the workflow in local scope so that ending the test case will destroy it.
    # Destroying the workflow will also delete all created projects and packages. The
    # created workflow has a target project, but most of the test assets need to be created
    # as needed
    wf = OBSLocal.FactoryWorkflow()
    staging = wf.create_staging('A', freeze=True)
    wf.create_submit_request('devel:wine', 'wine')

    ret = SelectCommand(wf.api, staging.name).perform(['wine'])
    self.assertEqual(True, ret)

To ease having many such tests, we also have the OBSLocal class, which moves the creation of the workflow into setUp and the destruction in tearDwon functions of pytest. The principle stays the same though.

class TestExampleWithOBS(OBSLocal.TestCase):
    """
    Tests for various api calls to ensure we return expected content
    """

    def setUp(self):
        super(TestExampleWithOBS, self).setUp()
        self.wf = OBSLocal.FactoryWorkflow()
        self.wf.setup_rings()
        self.staging_b = self.wf.create_staging('B')

    def tearDown(self):
        del self.wf
        super(TestExampleWithOBS, self).tearDown()

    def test_list_projects(self):
        """
        List projects and their content
        """
        staging_a = self.wf.create_staging('A')

        # Prepare expected results
        data = [staging_a.name, self.staging_b.name]

        # Compare the results
        self.assertEqual(data, self.wf.api.get_staging_projects())

Note that we have some (older) test cases using httpretty, but those are very special cases and require you a lot of extra mocking as you can't mix httpretty and testing against the minimal OBS. So every extra call that osc libraries or our code do, will require changes in your test case. It can still be a viable option, especially if more than OBS is involved.

The method that you can combine with OBSLocal though is using MagicMock. This class is used to mock individual functions. So splitting the code to use helper functions to retrieve information and then mocking this inside the test case can be a good alternative to mocking the complete HTTP traffic.