Customize asserts in upgrade test

[t0yv0]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

As of the version of 0.0.2 of the provider, Upgrade Tests assert that there are "no changes" when upgrading from a baseline version to the current version. This is not universally reasonable. While unexpected replacements are often undesirable especially for stateful resources like rds.Instance, changes such as in-place updates over resources, or in-place updates over first-class providers are OK and even desired in some scenarios.

Consider extending the assert language so that the user of the test framework can specify more explicitly what behavior is expected:

• distinguish replacements from updates
• distinguish resources by type

Affected area/feature

[t0yv0]

CC @thomas11 and @danielrbradley were running into this I believe and @thomas11 was attempting to work around with custom resource validation.

I'm running into this in the context of PreviewOnly mode in GCP. IT turns out that CI runs and local runs set different provider settings, and in case of using YAML with explicit provider this generates conflicts of the following form:

      ~ project: "pulumi-ci-gcp-provider" => "pulumi-development"
      - region : "us-central1"
        version: "6.67.0"
      - zone   : "us-central1-a"

These conflicts currently trip up the tests. This means snapshots are not reusable across CI and local, which is very unfortunate. Perhaps there's some workarounds like adding this to ignoreChanges.

[t0yv0]

I was able to workaround the GCP use case by using Pulumi Resource option ignoreChanges for region, project, zone.