Feature tracker: Data Replication for Custom Policy

[anlandu]

Currently, several built-in policies make use of data replication, which enables users to sync existing on-cluster resources to the OPA cache and reference them during evaluation of an AdmissionReview request. Data replication policies can be differentiated by the presence of data.inventory in the Rego, as well as the presence of the metadata.gatekeeper.sh/requires-sync-data annotation, which informs the Azure Policy addon what resources need to be cached for policy evaluation to work properly. (Note that this differs from standalone Gatekeeper, where this annotation is descriptive, not prescriptive.)

Data replication is currently blocked for use in custom policy definitions, because replicating resources with high instance counts can dramatically increase the Gatekeeper pods' resource usage if not used carefully. You will see a ConstraintTemplateInstallFailed error when attempting to create a custom policy definition containing a constraint template with this annotation.

Removing the annotation may appear to mitigate the error you see, but then the policy addon will not sync any required resources for that constraint template into the cache. Thus, your policies will be evaluated against an empty data.inventory (assuming that no built-in is assigned that replicates the requisite resources). This will lead to misleading compliance results. Additionally, manually editing the config to cache the required resources is also not permitted.

This issue serves to track custom policy support for data replication.