Need a way to accommodate reusable YAML without violating schema or cluttering JSON

Question

Need a way to accommodate reusable YAML without violating schema or cluttering JSON

Opened this issue 10 years ago · 36 comments

A benefit of YAML is that you can define a snippet of YAML once and refer back to it in many places.

However, that snippet is converted into JSON and then triggers a schema violation. Perhaps we should contain all snippets in a parent, such as snippets and then add that as an object in the JSON-schema. Furthermore, we should consider stripping the snippet element from the rendered JSON.

Here is an example:

ExampleQueryParam: &example
  name: example
  description: something something
  required: false

# refer to it later
myArray:
  - name: foo
  - <<: *example

earth2marsh commented 10 years ago

👍

Answer 1 · 2014-08-21T21:59:18.000Z

The problem I see with YAML anchors is several fold:

They take a lot of getting used to
They can cause "magic", where the source of the content can be hard to trace back
They don't really encourage reuse--they encourage DRY and the expense of reuse

I think item 3 is very important. The JSON pointer DOES encourage both DRY and reuse, because the locations and resolution are constrained by the schema. So I could make everyone crazy by using anchors to refer to a parameter description. But json pointers will let the unit of reuse be something more predictable.

Perhaps there's a way to mix the JSON pointer idea into a YAML syntax that makes sense. I don't know if YAML anchors are a good idea, though.

Answer 2 · 2014-08-25T04:13:08.000Z

I don't see the right answer yet. I will try to state the problem more succinctly, howeve…

It sucks to have to write the same thing over and over again, such as limit and offset parameters for paging through collections. Swagger 2.0 needs a way to define these cleanly once and then refer to them over and over again. It doesn't have to be native YAML-snippets, but it is arguably MOST important for YAML, since it is a top-down format where reusability is particularly important.

What do you suggest, @fehguy ?

Answer 3 · 2014-08-26T08:01:01.000Z

OK I've hit a snag. We can allow references as such:

"/pets/{id}": {
  "get": {
    "description": "Returns pets based on ID",
    "summary": "Find pets by ID",
    "operationId": "getPetsById",
    "parameters": [
      { "$ref": "#/definitions/skipParam" },
      { "$ref": "#/definitions/limitParam" }
    ],

however, we cannot define a skipParam in the schema, because the definitions key is restricted to models only.

I see a couple of options.

We support the /definitions section of the schema for model schemas. We can add a /parameters section which allows valid parameter references to be added
We can relax the constraints in the /definitions section to allow for anyOf, which means the /definitions can be either model schemas, parameter schemas, etc.
We can support shared parameters on only external files.

@earth2marsh thoughts?

Answer 4 · 2014-08-26T08:33:48.000Z

Please see the modified schema for parameters:

https://github.com/fehguy/swagger-spec/blob/master/schemas/v2.0/schema.json#L148-L167

and definitions

https://github.com/fehguy/swagger-spec/blob/master/schemas/v2.0/schema.json#L516-L525

Answer 5 · 2014-08-27T15:13:34.000Z

@fehguy @earth2marsh I have been working on a few different models and hit this concern. My preference is to have a top-level 'parameters' section that can be validated against the parameter schema and if needed individual parameter definitions can reference the models in 'definitions'. That way if you want to reuse a model for multiple parameters you can do that. Although this may make Jeremy cringe...

WDYT?

Answer 6 · 2014-08-28T05:53:42.000Z

@jwest-apigee @earth2marsh two thoughts.

we can structure the reusable components in the schema like I suggested earlier.
we can leave the /definitions section "unvalidated" by itself, and require each object in there to be validated based on how it's used. So you can have parameters, models, operations all in there, however, they are validated by their types, which are determined by how they're referenced.

Thinking about it more, I think that's much cleaner, and more extensible.

Answer 7 · 2014-08-29T16:26:42.000Z

I understand the benefit of having parameters in /definitions and it is definitely a valid approach. However, I like having parameter definitions in #/parameters such that it can be clean and explicit. It makes tooling easier since it is explicit where to find the model for which type of 'thing'. It is much easier to validate, semantically and structurally. It may also be easier to understand for newcommers to Swagger.

Also, the model for a request/response is basically free-form and used by the API implementation being described. The model/schema for the structure of a parameter within a Swagger doc is more strict, used within the context of the Swagger doc and we expect certain fields to be there that are not used by the API implementation.

Answer 8 · 2014-08-29T16:47:35.000Z

Here is an example of how I envision it working:

swagger: 2
info:
  title: Petstore Sample API
  description: A sample API that uses a petstore as an example to demonstrate features in the swagger-2.0 specification
  version: 1.0.0
  termsOfService: Do whatever you want with this
  license:
    name: MIT
    url: "http://github.com/gruntjs/grunt/blob/master/LICENSE-MIT"
consumes:
  - application/json
paths:
  /pets/{petId}:
    get:
      description: Specific image for a specific pet
      parameters:
        - $ref: "#/parameters/petId"
          required: true
      responses:
        "200":
          description: to test 200
        default:
          description: to test default
  /pets/{petId}/images:
    get:
      description: "All images for a pet"
      parameters:
        - $ref: "#/parameters/petId
          required: true
      responses:
        "200":
          description: to test 200
        default:
          description: to test default
  /pets/{petId}/images/{imageId}:
    get:
      parameters:
        - $ref: "#/parameters/petId"
          required: true
        - name: imageId
          required: true
      description: Specific image for a specific pet
      responses:
        "200":
          description: to test 200
          schema:
            $ref: petImage
        default:
          description: to test default
parameters:
  petId:
    name: petId
    in: path
    description: "The UUID of a Pet"
definitions:
  petImage:
    required:
      - imageId
    properties:
      imageId:
        type: string
        description: Unique entity ID

Answer 9 · 2014-08-29T16:56:52.000Z

understood. I'm guessing that's the simplest way to go

Answer 10 · 2014-08-29T17:10:27.000Z

updated here:

#114

Answer 11 · 2014-08-29T18:23:39.000Z

OH now we are back to definitions vs models. With above changes, definitions is not JSON-Schema definitions anymore because it's not generally for definitions and it's restricted to models only. Now, shouldn't we rename it back to models?

We need to advise API composers that the shortcut for $ref where you didn't have to put explicti path is only for models.

In @jwest-apigee's example, we can't do the following, right?

...
paths:
  /pets/{petId}:
    get:
      description: Specific image for a specific pet
      parameters:
        - $ref: petId
          required: true
...

Answer 12 · 2014-08-29T18:24:52.000Z

no, we've undone that. there are two things:

EDIT: should be definitions ~~schemas~~: this represents model schemas

parameters: this represents parameter definitions

please see this example:
https://github.com/reverb/swagger-spec/blob/master/fixtures/v2.0/json/resources/reusableParameters.json

Answer 13 · 2014-08-29T18:26:00.000Z

Also, can $ref reference be cascaded the way Jeff's examples is written?

      parameters:
        - $ref: petId
          required: true #this will get ignored with current parser

Answer 14 · 2014-08-29T18:27:37.000Z

not quite, it will look like this:

        parameters: 
          - $ref: "#/parameters/skipParam"
          - $ref: "#/parameters/limitParam"

note: no overriding a referenced parameter

Answer 15 · 2014-08-29T18:41:40.000Z

@fehguy it should also be possible to say:

parameters: 
          - $ref: "skipParam"
          - $ref: "limitParam"

… right?

I love having a top level parameters element. One interesting implication is that we now have a potential improvement to how you represent path parameters. Since path params are always implied by {paramName}, it becomes possible to treat the braces as the implicit parameter key and look there automatically. (Ideally you wouldn't have to specify that it was required, too, based on in: path.)

… continuing the logic, we also have a mechanism for defining parameters in the host and baseUrl elements. :)

Answer 16 · 2014-08-29T18:43:04.000Z

Now that we are at it, let's make one for responses too!

Answer 17 · 2014-08-29T18:49:12.000Z

@earth2marsh @mohsen1 lets open up separate issues for those things and close this one to keep it clean and focused :)

Answer 18 · 2014-08-29T18:49:49.000Z

@mohsen1 I'm ok with co-mingling request and response objects. Both are JSON (potentially other formats), and there may be reuse. For example, I might post a user object, but i'll also expect that same user object when I do a GET.

Answer 19 · 2014-08-29T18:50:48.000Z

note, you can also do this:

parameters: 
   - $ref: http://yourcompany.com/definitions/skipParam
   - $ref: http://yourcompany.com/definitions/limitParam

the reference to #/parameters would need to be implicit in the tooling.

Answer 20 · 2014-09-03T20:33:41.000Z

Do we need the name in a parameter definition?

parameters:
  petId:
    name: petId
    in: path
    description: "The UUID of a Pet"

Also, is it possible to make a parameter required at the definition level and then override it in the method? (I believe, yes)

Answer 21 · 2014-09-03T21:19:10.000Z

@jwest-apigee @fehguy See https://gist.github.com/earth2marsh/264123aefc2e29551d9d#file-uber-yaml-L130 for an example using Uber.

Two things to notice:

the parameter $ref is in its shortest form
name key is omitted from the parameter definition

Answer 22 · 2014-09-03T21:31:55.000Z

#/parameters/{name} refers to the element in the YAML document structure whereas ‘name: petId’ refers to the content that is interpreted by the tooling, no?

I believe ‘required’ only applies to the place where it is used, not necessarily globally.

On Sep 3, 2014, at 2:19 PM, Marsh Gardiner notifications@github.com wrote:

@jwest-apigee @fehguy See https://gist.github.com/earth2marsh/264123aefc2e29551d9d#file-uber-yaml-L130 for an example using Uber.

Two things to notice:

the parameter $ref is in its shortest form

name key is omitted from the parameter definition

—
Reply to this email directly or view it on GitHub.

Answer 23 · 2014-09-03T21:46:32.000Z

@jwest-apigee has a good point. Unfortunately JSON-Schema's $ref is not cascading. So if someone has a parameter which is required is some places they need to make two parameters, one required and one optional.

#....
      parameters:
        - $ref: required_foo
parameters:
  required_foo:
    in: query
    type: number
    required: true
    description: A foo.

I personally don't like it. Any better solutions?

Answer 24 · 2014-09-03T21:50:13.000Z

One solution is to put required values outside of the parameters

#...
      parameters:
        - $ref: "start_latitude"
        - $ref: "start_longitude"
     requiredParams:
        - start_latitude
        - start_longitude
parameters:
  start_latitude:
    in: query
    type: number
    description: Latitude component of start location.
  start_longitude:
    in: query
    type: number
    description: Longitude component of start location.

Answer 25 · 2014-09-03T21:50:24.000Z

see my previous example (snippet below). the point of the $ref is to have one definition of that parameter. Sometimes it may be required, sometimes not - therefore it doesn't make sense to me to include it in the #/parameters/petId section.

  /pets/{petId}/images/{imageId}:
    get:
      parameters:
        - $ref: "#/parameters/petId"
          required: true
  /pets/{petId}/something_else:
    get:
      parameters:
        - $ref: "#/parameters/petId"
          required: false
parameters:
  petId:
    name: petId
    in: path
    description: "The UUID of a Pet"

Answer 26 · 2014-09-03T21:53:37.000Z

We can make this work:

        - $ref: "#/parameters/petId"
          required: true

But then it's not following JSON Schema standard.

that YAML translated to JSON is this:

{
      "$ref": "#/parameters/petId",
      "required": true
}

JSON-Schema says you need to replace the object with it's $ref. So the required key will get lost :(

Answer 27 · 2014-09-03T21:56:10.000Z

I see. We’re in a difficult spot with the stage this change is being made in with regard to tooling. Given that this doesn’t work it looks like we need an alternative that we can agree on quickly.

Thoughts, @fehguy? @earth2marsh?

On Sep 3, 2014, at 2:53 PM, Mohsen Azimi notifications@github.com wrote:

We can make this work:
    - $ref: "#/parameters/petId"
      required: true
But then it's not following JSON Schema standard.

that YAML translated to JSON is this:

{
"$ref": "#/parameters/petId",
"required": true
}
JSON-Schema says you need to replace the object with it's $ref. So the required key will get lost :(

—
Reply to this email directly or view it on GitHub.

Answer 28 · 2014-09-03T22:00:07.000Z

Yes, you cannot merge the required:true with the ref, or the json schema compliance is broken.

Think of the json pointers as just that--simply pointers, not mix-ins.

Answer 29 · 2014-09-03T22:10:07.000Z

Might not work… but what about some riff on:

  /pets/{petId}/images/{imageId}:
    get:
      required: 
        - petId
      parameters:
        - $ref: petId
parameters:
  petId:
    name: petId
    in: path
    description: "The UUID of a Pet"

Answer 30 · 2014-09-03T22:43:47.000Z

That's the same as my proposal. The problem is, it's not following semantic hierarchy the rest of the document is following. required belongs to parameters so it should be under it

Answer 31 · 2014-09-03T22:53:10.000Z

cascading will mean "detailed object clobbers general object", not a merge. In the long run, this will be much better than complex merge strategies.

Answer 32 · 2014-09-21T15:36:28.000Z

@earth2marsh, @mohsen1 - Is this still an open issue?

Answer 33 · 2014-09-21T16:16:46.000Z

@webron, I'm uncertain. @fehguy has argued, this is a question of tooling, which may mean we should handle this in the Swagger Editor as an accommodation for authoring specs. If it has been ruled out of the Swagger spec, then perhaps this issue should be closed.

Answer 34 · 2014-09-21T16:19:27.000Z

It sounds like you're forcing me to read the issue ;)

Answer 35 · 2014-09-21T16:27:04.000Z

@earth2marsh - ok, that was a bit TLTR but I think I managed to get the gist of it. It looks like the original issue of references for parameters (and responses) has been added.
Is the current issue regarding the merging of top level definitions with local ones (such as the required and in and so on)?