saltstack-formulas/mysql-formula

Automated testing feature

amontalban opened this issue · 3 comments

amontalban commented

Hey guys,

I was working on this formula and wanted to have a way to test in multiple OS at once so I come up with a solution that involves TravisCI and KitchenCI.

I tried to draft a global solution that can be easily ported to other formulas because it reads the FORMULA file to get the formula name and based on that knows the name of the state to execute.

At this time for this particular formula it tests on the following OS:

  • Ubuntu 12.04 (Precise)
  • Ubuntu 14.04 (Trusty)
  • Ubuntu 16.04 (Xenial)
  • Debian 7 (Wheezy)
  • Debian 8 (Jessie)

Once #147 is fixed we can add support for CentOS and Fedora, and once test-kitchen/kitchen-docker#156 is merged we will be able to support openSuSE too.

I think this feature will allow contributors to test their commits on multiple scenarios to ensure formula quality.

Let me know if you have any question/improvement!

Thanks,
Andres

javierbertoli commented

Hi @amontalban, @blbradley suggested something similar here for the nginx's formula, and his solution seems to integrate nicely with travis-ci. I personally prefer serverspec over testinfra (the former's tests seem clearer to me when reading and implementing them), but ultimatelly, that's a matter of choice, and both processes can be integrated following a similar approach.

blbradley commented

Hi!

I stuck with testinfra because Saltstack is a Python project. That is a big reason people/companies choose Saltstack. It does come to a matter of personal preference. I'd stick with what is best for possible users. That could depend on the formula, but the sway towards Python is strong since states can be Python also.

Also, this is a better place to look at the overall solution in docker-formula.

Cheers!

amontalban commented

Hey @javierbertoli / @blbradley,

Thank you for your feedback, yeah I'm open to whatever option you think it's best. This initial commit is just to run the formula (Not currently verifying the state is doing what it shoud, just that the state was applied without errors).

I get @blbradley point but also from my POV serverspec / inspec syntax seems cleaner (We use them in our environment).

Would be great to have a defined standard across all formulas so then all PRs can be auto tested with TravisCI.

Cheers!