Tests failing: google service responses and over query limit
Opened this issue · 1 comments
Tests are failing for multiple reasons:
- variances in google service responses from original
- over query limit due to multiple test runs per day
We need to isolate integration tests from the actual google services. Over time, calling the actual services will (and are) causing false positive test failures due to variances in the responses that the tests wont be able to account for.
A system existed prior that enabled known responses to be tested against, vis-a-vis the json files within the test project.
We need to re-establish the json file handling, and enable a warning to be generated (possibly auto github issue generation) of a variance between the hardcoded json and its cooresponding service response, so a human can evaluate and update the hardcoded responses.
I think that this would add a lot of weight to the code base and result in running tests against stale "cached" json results that will gradually diverge from what the Google API actually produces. Somebody will have to ensure that that the "cached" responses are kept up to date so that we know that they do actually represent what Google is returning, which strikes me as a parallel task to somebody tweaking tests that fail when the live service changes results. It will also increase the complexity for somebody who wants to provide a PR supporting for a new feature of the API, as they will have to understand how to create these cached results and figure out how to capture them etc, etc.
Only two tests have ever failed because Google's results changed - a building number changed and the length of a journey leg changed. I really don't see that as a big deal when compared to the points above.
The query limit can easily be solved by giving appveyor it's own api key.