Oasis: Unresolvable JSON pointer: 'components/schemas/DatasetResponse'

[LarsvdBrandt]

I have an issue with NOMAD Oasis.

When creating a swagger client from bravado through a python script, I get an error because of an unresolved JSON pointer: : 'components/schemas/DatasetResponse'.

I checked the schema, and could not find DatasetResponse in there, but there was a reference to it.

Error:
DEBUG:swagger_spec_validator.ref_validators:Attaching x-scope to {'$ref': '#/components/schemas/DatasetResponse'}
ERROR:main:Error creating SwaggerClient: Unresolvable JSON pointer: 'components/schemas/DatasetResponse'

Code:

import requests
from bravado.client import SwaggerClient
from bravado.requests_client import RequestsClient
from urllib.parse import urlparse
import time
import logging
import sys

# Set up logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

# Import the necessary modules
from bravado.requests_client import Authenticator
from keycloak import KeycloakOpenID

# Define the file path
upload_file = r'***'

# Define NOMAD API URL and user credentials
nomad_url = 'http://localhost/nomad-oasis/api/v1'
user = 'r'***'
password = 'r'***'

# Define a class for authentication using Keycloak
class KeycloakAuthenticator(Authenticator):
    """ A bravado authenticator for NOMAD's keycloak-based user management. """
    def __init__(self, user, password):
        super().__init__(host=urlparse(nomad_url).netloc.split(':')[0])
        self.user = user
        self.password = password
        self.token = None
        self.__oidc = KeycloakOpenID(
            server_url='https://nomad-lab.eu/fairdi/keycloak/auth/',
            realm_name='fairdi_nomad_prod',
            client_id='nomad_public')

    def apply(self, request):
        if self.token is None:
            self.token = self.__oidc.token(username=self.user, password=self.password)
            self.token['time'] = time.time()
            logger.info("Authentication successful!")
        elif self.token['expires_in'] < int(time.time()) - self.token['time'] + 10:
            try:
                self.token = self.__oidc.refresh_token(self.token['refresh_token'])
                self.token['time'] = time.time()
                logger.info("Token refreshed successfully!")
            except Exception as e:
                logger.error(f"Failed to refresh token: {e}")
                self.token = self.__oidc.token(username=self.user, password=self.password)
                self.token['time'] = time.time()
                logger.info("Acquiring new token...")

        request.headers.setdefault('Authorization', 'Bearer %s' % self.token['access_token'])

        return request

# Create a RequestsClient without authenticator
http_client = RequestsClient()

# Set authenticator for the client
http_client.authenticator = KeycloakAuthenticator(user=user, password=password)

# Create a SwaggerClient for NOMAD API with logging
try:
    # Relax schema validation (use with caution in production)
    client = SwaggerClient.from_url(
    '%s/openapi.json' % nomad_url,
    http_client=http_client,
    config={
        'validate_swagger_spec': False,  # Relax schema validation (caution advised)
    }
    )
except Exception as e:
    logger.error(f"Error creating SwaggerClient: {e}")
    sys.exit(1)

# Upload the file
try:
    with open(upload_file, 'rb') as f:
        upload = client.upload.upload(file=f).response().result
except Exception as e:
    logger.error(f"Error uploading file: {e}")
    sys.exit(1)

# Supervise the processing
while upload.tasks_running:
    try:
        upload = client.upload.get_upload(upload_id=upload.upload_id).response().result
        time.sleep(5)
        print('processed: %d, failures: %d' % (upload.processed_calcs, upload.failed_calcs))
    except Exception as e:
        logger.error(f"Error getting upload status: {e}")
        sys.exit(1)

# Check if upload was successful
if upload.tasks_status != 'SUCCESS':
    logger.error("Something went wrong during upload processing.")
    logger.error(f"Errors: {str(upload.errors)}")
    sys.exit(1)

# Publish the upload
try:
    client.upload.exec_upload_operation(upload_id=upload.upload_id, payload={
        'operation': 'publish',
        'metadata': {
            'comment': 'Data uploaded via Python script',
            'authors': ['r'***'],
            'email': 'r'***@me.com'
        }
    })
    logger.info('Upload successful and published.')
except Exception as e:
    logger.error(f"Error publishing upload: {e}")
    sys.exit(1)

[lauri-codes]

Thanks for reporting this @LarsvdBrandt.

I can reproduce this with your example and will try to see what goes wrong.

[lauri-codes]

There was indeed an issue in our models, and we have now fixed it. However, I was still not able to successfully create a bravado client from our specification, but instead got an error about:

Error creating SwaggerClient: Duplicate "Aggregations" model found at "http://localhost:8000/fairdi/nomad/latest/api/v1/openapi.json#/components/schemas/MetadataResponse/properties/aggregations". Original "Aggregations" model at "http://localhost:8000/fairdi/nomad/latest/api/v1/openapi.json#/components/schemas/Metadata/properties/aggregations"

I'm not sure what exactly causes this: these models are definitely different pydantic classes in our codebase. The issue could also be in bravado itself: it has not seen updates in three years, and it seems to only support OpenAPI v2, whereas we are currently using OpenAPI v3. If you can reproduce this issue with an OpenAPI v3 compatible client, then I think we could investigate further.

[lauri-codes]

Please re-open this issue if there are still problems with OpenAPI v3 compatible Python clients, closing for now.