keboola/sapi-python-client

Test Enhancements

Closed this issue · 6 comments

Ogaday commented

You may have noticed I've looking at the tests ( #43 #44 #45 #46) - Hope these prs are welcome! I want to make a few more enhancements to the tests to make them slightly easier to work with, and then I'll look working on the issues in the milestone for v1.0.0.

  • Tidy up tests a little
    • class name ( #43 )
    • Remove deprecation warning ( #44 )
    • tests.base.TestEndpoint.test_get doesn't actually make a request (c8d7872)
  • Separate functional and mock tests into subpackages ( #45 )
  • Use subtests ( #46 )
  • Add support for skipping functional tests when there is no connection (don't know how this would play with CI). See:
  • Profile tests to hopefully identify and shorten long running tests - Reducing total test time will reduce development friction ( #47 & #49 )
  • Modify functional tests to allow running concurrently - this will also significantly reduce test time if test modules can be run in parallel. I know that green has support for this, and other test runners like pytest may do too. Currently, because each test module makes requests against the same bucket name, they can't run concurrently ( #50 )
  • Increase test coverage - this is a longer term objective
odinuv commented

Hi, this is of course highly appreciated, thank you! I'll look into the PRs in today.

odinuv commented

I'd vote against "skipping functional tests", any conditions in tests are source of potential errors and it does not seem much useful to me - testing an API client without a connection to the API

  • 👍1
odinuv commented

profiled tests

image

  • 👍1
  • ❤️1
odinuv commented

i wouldn't bother too much about the tests speed though, or in other words - i prefer the tests to be rock stable over anything else

Ogaday commented

Ooh, thanks for the profiling - that's great! Is that your IDE? I suppose it makes sense that the slowest tests are for creating and deleting tables. Not sure what can be done about that!

I know that I can reduce the number of times a mock job returns processing - currently it waits for five requests which is why that mock test takes a long time.

I'll do one more thing for the tests right now, which is change the name of the bucket created (in.c-py-test) so it is unique for each test. Then I can run each test module in parallel. To be even safer I could add a timestamp to the name of the bucket so there are no collisions, but I don't think it is necessary.

odinuv commented

I'm using PyCharm Community https://www.jetbrains.com/pycharm/download/#section=windows, works like a charm :)

if adding timestamps to bucket names, you'd have to loop through all buckets in tearDown, it might be a good thing to do anyway if there are more buckets created