turbomam/examples-first-cookiecutter

existing and proposed test steps

Opened this issue · 6 comments

turbomam commented

make test from ./Makefile currently does the following, at a high level:

  • make test-schema
    • make gen-project
  • make test-python
    • python -m unittest discover
      • applies tests/test_valid_collections.py to selected files in src/data/examples/valid
  • make test-examples
    • runs linkml-run-examples
    • input: src/data/examples
    • output: examples/output

make check-jsonschema-example from ./project.Makefile provides a minimal example of checking one hard-coded data file with check-jsonschema

Change that to iterate over all valid and invalid files. Wrap with ! (not -!) for the invalid files.

Clarify that check-jsonschema exhaustively reports schema violations but is not part of the LinkML ecosystem. Its functionality will probably be replaced by a (potentially even more useful) LinkML tool at some point.

turbomam commented

The dependency project/jsonschema/{project name}.schema.json may not have a corresponding make target. It may assume that gen-project has already been run.

turbomam commented 
JSON_SCHEMA_FILE := project/jsonschema/identifiers_sandbox.schema.json
INVALID_EXAMPLES_DIR := src/data/examples/invalid
INVALID_EXAMPLE_FILES := $(wildcard src/data/examples/invalid/*.yaml)
VALID_EXAMPLES_DIR := src/data/examples/valid
VALID_EXAMPLE_FILES := $(wildcard src/data/examples/valid/*.yaml)

check-all-invalid-examples: $(patsubst $(INVALID_EXAMPLES_DIR)/%.yaml,jsonschema-vs-invalid--%,$(INVALID_EXAMPLE_FILES))

jsonschema-vs-invalid--%: $(JSON_SCHEMA_FILE) $(INVALID_EXAMPLES_DIR)/%.yaml
	! $(RUN) check-jsonschema --schemafile $^

check-all-valid-examples: $(patsubst $(VALID_EXAMPLES_DIR)/%.yaml,jsonschema-vs-valid-%,$(VALID_EXAMPLE_FILES))

jsonschema-vs-valid-%: $(JSON_SCHEMA_FILE) $(VALID_EXAMPLES_DIR)/%.yaml
	$(RUN) check-jsonschema --schemafile $^
turbomam commented

could also use --output-format json

turbomam commented

src/data/examples/valid/NamedThingCollection-dupe-ids.yaml passes make check-valid-and-invalid but fails tests/test_valid_collections.py and make test-examples

turbomam commented

in other words:

  • instantiating a collection containing elements of the same type that share an identifier FAILS AS EXPECTED
    • make test-python with collection file in src/data/examples/valid
  • jsonschema validation of that collection UNEXPECTEDLY PASSES
    • check-valid-and-invalid with collection file in src/data/examples/valid
  • placing that collection in linkml-run-exapmples' valid input directory FAILS AS EXPECTED
    • make test-examples with collection file in src/data/examples/valid
  • placing that collection in linkml-run-exapmples' counter-examples directory UNEXPECTEDLY FAILS
    • make test-examples with collection file in src/data/examples/invalid
turbomam commented

haven't addressed how linkml-validate works with these files