cben/mathdown

Run test against rhcloud and/or heroku

Closed this issue · 7 comments

cben commented

Currently the (admittedly lame) test runs against tiny local server.
Ideally should run against real app on rhcloud/heroku.

  • Ideally the test runner (e.g. Travis) should deploy new app and use that in test (and then deploy to prod app).
    • Crazy idea: create/fork a new heroku app for each commit!? Since they are free (and would idle all the time)... Poor man's rawgit or abuse of heroku's generosity?
  • A simpler approximation is testing as today, auto-deploying to a staging app, letting me manually(?) test and deploy to prod app.
    • Test should implement option to run automatic test against external URL anyway.
cben commented

An example this would catch that I'm not currently testing: correct NodeJS version
(my ubuntu currently has 0.10.25, so is RHclould, but Heroku runs 0.12.2; these could change any time...)
Forcing a specific version on RHcloud & Heroku is probably smarter than testing, but that's just one example.

cben commented

While not quite a solution to automated tests, Heroku have a cool new thing where they create an app for each pull request:
https://devcenter.heroku.com/articles/github-integration-review-apps

It's also based on app.json and I'm afraid it won't support submodules:

Heroku Button will not work with repos that have Git submodules. Heroku Button relies on the Build API and uses tarballs fetched from GitHub. GitHub does not include submodule contents when repo-content tarballs are generated.

but we'll see shortly :-)

cben commented

Nope, no submodule support :-( Closed #94.
Sending a mail to heroku.

cben commented

Partial improvement:

  • add manual test option to run against existing prod backend.
  • add at least manual way to launch a staging heroku app.
cben commented

Created 2 permanent staging apps:
https://mathdown-staging.herokuapp.com
https://mathdown8staging-cben.rhcloud.com
(8 is a convention I chose because OpenShift allows only letters & numbers there. Makes mnemonic sense in "mathdown 'at' BRANCH" use case.)
https://staging-mathdown.rhcloud.com/

Turns out a simple flow deploying to fixed staging apps, testing them and then deploying for real is very easy! There is still a lot to improve but this is a solid start and big safety win.

[A side effect is that it wouldn't be safe to run deploy.sh from CI context(s).
Do I really want auto-deploy after CI?]

cben commented 
remote: Warning: Gear 55cd18e77628e15f6c000032 is using 97.6% of inodes allowed

will have to increase storage for mathdown8staging as I did for mathdown.
"deploy to openshift" button works for one-off deploys but apparently won't survive repeated deploys?
(or can I trim something low-hanging? submodules?)

cben commented

I consider this Done long ago.

I still want to enable Heroku button, Heroku review apps, Heroku auto-deploy to staging from github (all blocked on submodules) but that's separate.
I don't think I want heroku/rhcloud to be involved in CI.