joomla-x/joomla-pythagoras

Test Environment

nibra opened this issue · 5 comments

nibra commented

Problem

The Automated Testing Working Group decided to use CodeCeption for testing. In order to provide consistency, the test environment of the CMS should be moved to CodeCeption.

Although CodeCeption supports PHPUnit tests, and actually uses PHPUnit to execute these tests, the test results are not identical for the two environments, as more tests are failing or even throwing errors, when executed with CodeCeption. It has shown to be a huge efford to rewrite the existing tests. The tests are not worth the efford, though, because of at least two reasons:

  • a lot of tests use Reflection to test implementation details instead of API behaviour
  • nearly all tests test things not used/needed in Joomla!4

Solution

The issue can be solved by making the transition stepwise.

Immediate Steps (4.0)

  1. Move current tests directory to tests.phpunit and change phpunit.xml.dist, travisci-phpunit.xml, and other related files accordingly, so the current build procedure works with PHPUnit using the new directory. Done.
  2. Add the CodeCeption test structure, with additional cli and api suites, in the tests directory. Done.
  3. Add CodeCeption test environment to Travis configuration.
  4. Write new tests for CodeCeption as needed. If code changes require changing PHPUnit tests, move these tests to the CodeCeption environment, rewriting them to test API instead of implementation, if needed.

Future Steps (4.x)

  1. Provide PageObjects with a template-aware factory, so the same tests can be executed with any template.
  2. Provide Docker and / or Vagrant support for integration (functional) and system (acceptance) tests.
  3. Move PageObject, Docker, and Vagrant support to an external library, so they can be reused by third party developers.
rdeutz commented

I another project I have also separated API Testing with pure phpunit from the codeception part, I think at the end of the day it comes to personal preference.

For now I would agree of to make two directories codeception and phpunit and move test into these, I don't like the unit dir because it is misleading, not all tests in this dir are unit test

nibra commented

feature/codecept-upstream

Added a new branch, feature/codecept-upstream. The branch is based on upstream, i.e., current Joomla!3. It may be merged into staging, but MUST NOT be deleted.

Purpose

This branch is intended to collect (acceptance) tests for the Joomla!3 series, which have to be fullfilled by Joomla!4, too. The tests will be cherry-picked into staging one by one.

Updates

This branch MUST be kept in sync with upstream, which in turn MUST be kept in sync with joomla/joomla-cms:staging.

javigomez commented

Codeception API tests are very nice, see https://github.com/joomla-projects/webservices/blob/develop/tests/api/administrator.contact.1.0.0.Cest.php. The plan is to have Joomla 3.6 tested with Codeception API tests.

More info http://codeception.com/docs/10-WebServices

Once the API tests will be done for 3.6 in the next monts (we need to wait for the webservices 3.6 team) can be easily ported to 4.0 branch I'm sure.

nibra commented

So you think it makes sense to backport the branch to joomla/joomla-cms?

nibra commented

Moved to joomla/joomla-cms