rocicorp/replicache-old

Run tests continuously

Closed this issue · 5 comments

aboodman commented

We can just use Github for this, but only once the relevant repos are public (e.g., this is dependent upon #19)

phritz commented

@aboodman @arv can you dump anti/requirements here? at the very least:

  • build across repos
  • NO requirement to build against private repos
  • do we want/need to build binary artifacts accessible to the outside world (eg, releases) via CI or is that a manual process? (I claim we should have a strong bias against manual processes that could be automated.)

Questions:

  • do we need to be able to deploy something from CI (eg to zeit or elsewhere)?
  • what kinds of secrets if any does CI need?
  • is testing for flutter and other bindings in or out of scope for this particular issue?
  • do we care if we use a different CI tool for bindings code?
  • what level of GH permissions should be required to trigger CI?
aboodman commented

I don't have a clear idea of requirements. Responding to your questions:

  • build across repos

It seems like have to have this due to the organization of our code

  • NO requirement to build against private repos

It seems likely that we will need this. It seems risky to make a big commitment to something that doesn't support this (the only reason I was supportive of Github as a solution is because it seems like it was just flipping a switch, so not a big commit). If there was something easy that didn't have this restriction I'd be in favor of it.

  • do we want/need to build binary artifacts accessible to the outside world (eg, releases) via CI or is that a manual process? (I claim we should have a strong bias against manual processes that could be automated.)

It would be much better if this was automated.

  • do we need to be able to deploy something from CI (eg to zeit or elsewhere)?

I would say we need to test every commit and we need high assurance that what we run on Zeit is what we tested. I don't think in order to achieve that we need (or even can, easily) ship binaries to Zeit. Go's module system is sufficient IMO to get us close enough to what-we-test is what-we-run.

  • is testing for flutter and other bindings in or out of scope for this particular issue?

It would be really good to test Flutter bindings, but we can't even test them manually on our desktops yet. It would be better if what we choose could eventually support this, but it's not a short-term requirement.

  • do we care if we use a different CI tool for bindings code?

We definitely shouldn't do this.

  • what level of GH permissions should be required to trigger CI?

None. A PR should trigger it.

=====

Other thoughts:

  • It seems like somebody would have already done something to integrate testing into Zeit. Because it seems trivial in some sense. Like the server can just test itself at startup or something. Or we could have another Zeit deploy target that just runs the tests. Then we'd automatically get tests on every PR which would be amazing.

  • The client-side and SDK build is getting pretty gnarly already. I'm beginning to think that we should use a real build system instead of my terrible shell scripts. I have heard that Bazel is the Right Thing but I get scared everytime I look at it.

  • I've had really bad experience with every CI tool I've ever used except internal ones at Google. Travis and Jenkins were ongoing frustrations at Attic Labs. That's why I was drawn to Github because in my experience Github's stuff is usually well-done. Same idea with Zeit, all their stuff works real nicely so I thought layering it on top of that was attractive.

aboodman commented

@arv is it possible to get the flutter tests running continuously somehow?

arv commented

Yes, I believe that if I clean up the record/replay and use that for all test functions it can be run continuously.