testing/bdd.ts with flat test grouping style the only option scoped to top level test instead of the file

Question

testing/bdd.ts with flat test grouping style the only option scoped to top level test instead of the file

Opened this issue 3 years ago · 1 comments

Describe the bug

feature: #2067

The issue with this feature is that Deno.test is called by the describe function. If tests are added to the group after the describe function call completes, it can handle that, except for the only option. Because Deno.test was already called without the only option, it cannot retroactively change the test to have the only option set to true if one of the tests or sub groups are focused using describe.only or it.only.

Issue 1: When using flat test grouping, if the top level test grouping is not focused but a child is, other top level tests that are not focused will still run. The test or tests that are focused within the top level describe will just be the only tests that run within the context of that top level test grouping.

Issue 2: When using flat test grouping, If the top level test grouping is not focused but a child is along with another top level test. Only the focused top level tests and test groups will run.

Steps to Reproduce

Issue 1: getAge is focused but the Example A and Example B test cases still run. That's because the "User" test was registered with Deno.test without the only option.

import {
  assert,
  assertEquals,
  assertStrictEquals,
  assertThrows,
} from "../asserts.ts";
import { describe, it } from "../bdd.ts";
import { User } from "./user.ts";

it("Example A", () => assert(true));

const userTests = describe("User");

it(userTests, "users initially empty", () => {
  assertEquals(User.users.size, 0);
});

it(userTests, "constructor", () => {
  try {
    const user = new User("Kyle");
    assertEquals(user.name, "Kyle");
    assertStrictEquals(User.users.get("Kyle"), user);
  } finally {
    User.users.clear();
  }
});

const ageTests = describe({
  name: "age",
  suite: userTests,
  beforeEach(this: { user: User }) {
    this.user = new User("Kyle");
  },
  afterEach() {
    User.users.clear();
  },
});

it.only(ageTests, "getAge", function () {
  const { user } = this;
  assertThrows(() => user.getAge(), Error, "Age unknown");
  user.age = 18;
  assertEquals(user.getAge(), 18);
});

it(ageTests, "setAge", function () {
  const { user } = this;
  user.setAge(18);
  assertEquals(user.getAge(), 18);
});

it("Example B", () => assert(true));

kyle@kyle-XPS-15-9560:~/Projects/deno/deno_std$ deno test testing/bdd_examples/user_flat_only_issue_1_test.ts 
running 3 tests from file:///home/kyle/Projects/deno/deno_std/testing/bdd_examples/user_flat_only_issue_1_test.ts
test Example A ... ok (6ms)
test User ...
  test age ...
    test getAge ... ok (5ms)
  ok (9ms)
ok (13ms)
test Example B ... ok (3ms)

test result: ok. 3 passed (2 steps); 0 failed; 0 ignored; 0 measured; 0 filtered out (42ms)

Issue 2: "getAge" and "Example B" are focused but only the "Example B" test cases runs. That's because the "User" test was registered with Deno.test without the only option and the "Example B" test case was registered with Deno.test with the only option. The following example is the same as the one from issue 1 but with the "Example B" test case focused.

import {
  assert,
  assertEquals,
  assertStrictEquals,
  assertThrows,
} from "../asserts.ts";
import { describe, it } from "../bdd.ts";
import { User } from "./user.ts";

it("Example A", () => assert(true));

const userTests = describe("User");

it(userTests, "users initially empty", () => {
  assertEquals(User.users.size, 0);
});

it(userTests, "constructor", () => {
  try {
    const user = new User("Kyle");
    assertEquals(user.name, "Kyle");
    assertStrictEquals(User.users.get("Kyle"), user);
  } finally {
    User.users.clear();
  }
});

const ageTests = describe({
  name: "age",
  suite: userTests,
  beforeEach(this: { user: User }) {
    this.user = new User("Kyle");
  },
  afterEach() {
    User.users.clear();
  },
});

it.only(ageTests, "getAge", function () {
  const { user } = this;
  assertThrows(() => user.getAge(), Error, "Age unknown");
  user.age = 18;
  assertEquals(user.getAge(), 18);
});

it(ageTests, "setAge", function () {
  const { user } = this;
  user.setAge(18);
  assertEquals(user.getAge(), 18);
});

it.only("Example B", () => assert(true));

kyle@kyle-XPS-15-9560:~/Projects/deno/deno_std$ deno test testing/bdd_examples/user_flat_only_issue_2_test.ts 
running 1 test from file:///home/kyle/Projects/deno/deno_std/testing/bdd_examples/user_flat_only_issue_2_test.ts
test Example B ... ok (5ms)

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out (25ms)

error: Test failed because the "only" option was used

Expected behavior

In the first issue example, the test titled "getAge" should be the only one that runs. The "Example A" and "Example B" test cases should not be running.

In the second issue example, both the test titled "getAge" and the test titled "Example B" should run. The test titled "getAge" is currently not running.

Environment

OS: Ubuntu 20.04
deno version: canary
std version: 0.135.0? The change to add testing/bdd.ts has been merged but a std release has not been made for it yet.

Answer 1 · 2022-04-09T10:05:53.000Z

There is a workaround to both of the above issues. It is to change the describe("User") to describe.only("User") so that it knows to call Deno.test with the only option. The following example is the same as the one from issue 2 but with the "User" test suite focused too.

import {
  assert,
  assertEquals,
  assertStrictEquals,
  assertThrows,
} from "../asserts.ts";
import { describe, it } from "../bdd.ts";
import { User } from "./user.ts";

it("Example A", () => assert(true));

const userTests = describe.only("User");

it(userTests, "users initially empty", () => {
  assertEquals(User.users.size, 0);
});

it(userTests, "constructor", () => {
  try {
    const user = new User("Kyle");
    assertEquals(user.name, "Kyle");
    assertStrictEquals(User.users.get("Kyle"), user);
  } finally {
    User.users.clear();
  }
});

const ageTests = describe({
  name: "age",
  suite: userTests,
  beforeEach(this: { user: User }) {
    this.user = new User("Kyle");
  },
  afterEach() {
    User.users.clear();
  },
});

it.only(ageTests, "getAge", function () {
  const { user } = this;
  assertThrows(() => user.getAge(), Error, "Age unknown");
  user.age = 18;
  assertEquals(user.getAge(), 18);
});

it(ageTests, "setAge", function () {
  const { user } = this;
  user.setAge(18);
  assertEquals(user.getAge(), 18);
});

it.only("Example B", () => assert(true));

kyle@kyle-XPS-15-9560:~/Projects/deno/deno_std$ deno test testing/bdd_examples/user_flat_only_workaround_test.ts 
Check file:///home/kyle/Projects/deno/deno_std/testing/bdd_examples/user_flat_only_workaround_test.ts
running 2 tests from file:///home/kyle/Projects/deno/deno_std/testing/bdd_examples/user_flat_only_workaround_test.ts
test User ...
  test age ...
    test getAge ... ok (4ms)
  ok (7ms)
ok (13ms)
test Example B ... ok (3ms)

test result: ok. 2 passed (2 steps); 0 failed; 0 ignored; 0 measured; 1 filtered out (29ms)

error: Test failed because the "only" option was used