`gx_exceptions.CheckpointError` occurred when validating a manifest

Question

`gx_exceptions.CheckpointError` occurred when validating a manifest

linglp opened this issue 2 years ago · 0 comments

Describe the bug
When I tried to use the model/validate endpoint to validate a manifest, the following error occurred:

Traceback (most recent call last):
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/great_expectations/data_context/store/checkpoint_store.py", line 193, in add_checkpoint
    return self._persist_checkpoint(
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/great_expectations/data_context/store/checkpoint_store.py", line 252, in _persist_checkpoint
    checkpoint_ref = persistence_fn(key=key, value=checkpoint.get_config())
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/great_expectations/data_context/store/store.py", line 191, in add
    return self._store_backend.add(
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/great_expectations/data_context/store/_store_backend.py", line 138, in add
    raise StoreBackendError(f"Store already has the following key: {key}.")
great_expectations.exceptions.exceptions.StoreBackendError: Store already has the following key: ('manifest_checkpoint',).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/connexion/decorators/decorator.py", line 68, in wrapper
    response = function(request)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper
    response = function(request)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/connexion/decorators/validation.py", line 196, in wrapper
    response = function(request)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/connexion/decorators/validation.py", line 399, in wrapper
    return function(request)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/connexion/decorators/parameter.py", line 120, in wrapper
    return function(**kwargs)
  File "/Users/lpeng/Documents/schematic-git2/schematic/api/routes.py", line 352, in validate_manifest_route
    errors, warnings = metadata_model.validateModelManifest(
  File "/Users/lpeng/Documents/schematic-git2/schematic/schematic/models/metadata.py", line 254, in validateModelManifest
    errors, warnings, manifest = validate_all(self, errors, warnings, manifest, manifestPath, self.sg, jsonSchema, restrict_rules, project_scope)
  File "/Users/lpeng/Documents/schematic-git2/schematic/schematic/models/validate_manifest.py", line 253, in validate_all
    manifest, vmr_errors, vmr_warnings = vm.validate_manifest_rules(manifest, sg, restrict_rules, project_scope)
  File "/Users/lpeng/Documents/schematic-git2/schematic/schematic/models/validate_manifest.py", line 138, in validate_manifest_rules
    ge_helpers.build_checkpoint()
  File "/Users/lpeng/Documents/schematic-git2/schematic/schematic/models/GE_Helpers.py", line 363, in build_checkpoint
    self.context.add_checkpoint(**checkpoint_config)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/great_expectations/data_context/data_context/abstract_data_context.py", line 1648, in add_checkpoint
    return self.checkpoint_store.add_checkpoint(checkpoint)
  File "/Users/lpeng/Library/Caches/pypoetry/virtualenvs/schematicpy-9OomxyhV-py3.10/lib/python3.10/site-packages/great_expectations/data_context/store/checkpoint_store.py", line 197, in add_checkpoint
    raise gx_exceptions.CheckpointError(
great_expectations.exceptions.exceptions.CheckpointError: A Checkpoint named manifest_checkpoint already exists.

I used the following setting:

example data model
data_type: Patient
manifests that I used:
synapse-storage-manifest-big.csv
synapse-storage-manifest-big-two.csv

The same error occurred when validating a manifest by using the CLI as well:

schematic model --config config.yml validate --manifest_path /Users/lpeng/Documents/schematic-git2/schematic/tests/data/mock_manifests/Valid_Test_Manifest.csv  --data_type MockComponent

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Priority (select one)

Minor ⬇️
Major 📢
Critical 🆘

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (if applicable, please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.