[DataLake-job] Model validation fails with errors
veronicagg opened this issue · 8 comments
veronicagg commented
Validating "examples" and "x-ms-examples" in /Users/Veronica/github/openapi-validation-tools/arm-datalake-job/swagger/job.json:
> Operation: Job_Build
> x-ms-examples
> Scenario: SubmitGetListCancelTestPOST341
> InnerErrors:
1. code: MINIMUM
message: Value 0 is less than minimum 1
path: degreeOfParallelism
description: the degree of parallelism used for this job. This must be greater than 0.
Model from swagger:
"degreeOfParallelism": {
"type": "integer",
"minimum": 1,
"default": 1,
"format": "int32",
"description": "the degree of parallelism used for this job. This must be greater than 0."
},Example from .net recording - shows response returns invalid value for property:
{
"parameters": {
"api-version": "2016-11-01",
"expanded": "true",
"parameters": {
"name": "azure sdk data lake analytics job",
"type": "USql",
"degreeOfParallelism": 2,
"properties": {
"type": "USql",
"script": "DROP DATABASE IF EXISTS testdb; CREATE DATABASE testdb;"
}
}
},
"responses": {
"200": {
"body": {
"jobId": "f9d64a74-f687-4227-bfb6-8552cd27d4d3",
"name": "azure sdk data lake analytics job",
"type": "USql",
"submitter": "adlsvc01@benwgoldoutlook.onmicrosoft.com",
"degreeOfParallelism": 0,
"priority": 0,
"submitTime": "0001-01-01T00:00:00-08:00",
"state": "Ended",
"result": "Succeeded",
"properties": {
"resources": [],
"runtimeVersion": "kobo_vnext_signed_5177484",
"script": "DROP DATABASE IF EXISTS testdb; CREATE DATABASE testdb;",
"algebraFilePath": "mafs://accounts/testdatalake12107/fs/system/compilationService/jobs/USql/f9d64a74-f687-4227-bfb6-8552cd27d4d3/2016/11/08/04/16/algebra.xml",
"compileMode": "Semantic",
"errorSource": "Unknown",
"totalCompilationTime": "PT0S",
"totalPausedTime": "PT0S",
"totalQueuedTime": "PT0S",
"totalRunningTime": "PT0S",
"diagnostics": [],
"type": "USql"
}
}
}
}
}> Operation: Job_Build
> Scenario: SubmitGetListCancelTestPOST332
> InnerErrors:
1. code: INVALID_TYPE
message: Expected type array but found type null
path: stateAuditRecords
description: the job state audit records, indicating when various operations have been performed on this job.
2. code: INVALID_TYPE
message: Expected type array but found type null
path: errorMessage
description: the error message details for the job, if the job failed.
3. code: INVALID_TYPE
message: Expected type string but found type null
path: endTime
description: the completion time of the job.
4. code: INVALID_TYPE
message: Expected type string but found type null
path: startTime
description: the start time of the job.
5. code: INVALID_FORMAT
message: Object didn't pass validation for format date-time: Mon, 01 Jan 0001 00:00:00 GMT
path: submitTime
description: the time the job was submitted to the service.
Model from swagger:
"stateAuditRecords": {
"type": "array",
"readOnly": true,
"items": {
"$ref": "#/definitions/JobStateAuditRecord"
},
"description": "the job state audit records, indicating when various operations have been performed on this job."
}, "errorMessage": {
"type": "array",
"readOnly": true,
"items": {
"$ref": "#/definitions/JobErrorDetails"
},
"description": "the error message details for the job, if the job failed."
}, "endTime": {
"type": "string",
"readOnly": true,
"format": "date-time",
"description": "the completion time of the job."
}, "startTime": {
"type": "string",
"readOnly": true,
"format": "date-time",
"description": "the start time of the job."
}, "submitTime": {
"type": "string",
"readOnly": true,
"format": "date-time",
"description": "the time the job was submitted to the service."
},> Operation: Job_Get
> x-ms-examples
> Scenario: SubmitGetListCancelTestGET526
> InnerErrors:
1. code: INVALID_TYPE
message: Expected type string but found type null
path: stateAuditRecords,4,requestedByUser
description: the user who requests the change.
2. code: INVALID_FORMAT
message: Object didn't pass validation for format date-time: Fri, 20 Nov 2015 22:27:00 GMT
path: stateAuditRecords,4,timeStamp
description: the time stamp that the state change took place.
5. code: INVALID_TYPE
message: Expected type string but found type null
path: stateAuditRecords,2,details
description: the details of the audit log.
Model from swagger:
"requestedByUser": {
"type": "string",
"readOnly": true,
"description": "the user who requests the change."
}, "timeStamp": {
"type": "string",
"readOnly": true,
"format": "date-time",
"description": "the time stamp that the state change took place."
}, "details": {
"type": "string",
"readOnly": true,
"description": " the details of the audit log."
}> Operation: Job_Create
> x-ms-examples
> Scenario: CredentialCRUDTestPUT525
> InnerErrors:
1. code: OBJECT_MISSING_REQUIRED_PROPERTY
message: Missing required property: script
path: properties
description: The common Data Lake Analytics job properties.
Model from swagger:
"JobProperties": {
"discriminator": "type",
"required": [
"script",
"type"
],
"properties": {
"runtimeVersion": {
"type": "string",
"description": "the runtime version of the Data Lake Analytics engine to use for the specific type of job being run."
},
"script": {
"type": "string",
"description": "the script to run"
},
"type": {
"type": "string",
"description": "the job type of the current job (i.e. Hive or U-SQL)."
}
},
"description": "The common Data Lake Analytics job properties."
},> Operation: Job_List
> x-ms-examples
> Scenario: SubmitGetListCancelTestGET640
> InnerErrors:
1. code: OBJECT_MISSING_REQUIRED_PROPERTY
message: Missing required property: properties
path: value,1
description: The common Data Lake Analytics job information properties.
> Scenario: SubmitGetListCancelTestGET643
> InnerErrors:
2. code: OBJECT_MISSING_REQUIRED_PROPERTY
message: Missing required property: type
path: value,1
description: The common Data Lake Analytics job information properties.
3. code: OBJECT_MISSING_REQUIRED_PROPERTY
message: Missing required property: name
path: value,1
description: The common Data Lake Analytics job information properties.
Extract from swagger model:
"JobInformation": {
"required": [
"name",
"type",
"properties"
], > Scenario: SubmitGetListCancelTestGET631
> InnerErrors:
1. code: INVALID_TYPE
message: Expected type string but found type null
path: nextLink
description: the link (url) to the next page of results.
Model from swagger:
"JobInfoListResult": {
"properties": {
"value": {
"type": "array",
"readOnly": true,
"items": {
"$ref": "#/definitions/JobInformation"
},
"description": "the list of jobInfo items."
},
"nextLink": {
"type": "string",
"readOnly": true,
"description": "the link (url) to the next page of results."
}
},
"description": "List of jobInfo items."
}begoldsm commented
@veronicagg this is an interesting issue. We do require a minimum degreeOfParallelism for submitting jobs, but for building jobs this field is ignored, and the service simply returns default(int), which happens to be zero. I can remove the restriction in our swagger specs, since our service will fix submissions with degreeOfParallelism < 1 to be = 1. Thoughts?
amarzavery commented
But you still need that constraint while submitting the job. May be documenting this in the description and removing it as a constraint is the best option. I am not sure if using allOf can solve this accurately.
begoldsm commented
That is fair, I can make that update.
begoldsm commented
This should now be resolved with the merge of PR #917. @amarzavery or @veronicagg can you verify and close this issue (I do not have permission to do so).
begoldsm commented
@amarzavery and @veronicagg please verify and close this.
kirthik commented
@veronicagg - Could you please close the issue if this is fixed