mykso/myks

myks 3.0 Sync / clean up regression?

Closed this issue · 4 comments

kbudde commented

With 3.0.0/3.0.1 I noticed that my metallb rendered output and one vendored file is being deleted.
During the investigation I've found more "interesting behaviour"

Repo: https://github.com/kbudde/lab

Command myks sync ALL metallb-system is deleting the file envs/zoo/_apps/metallb-system/vendor/ytt/config/manifests/metallb-native.yaml, therefore the rendering will delete the rest.

> myks sync ALL metallb-system
9:13AM DBG [zoo > metallb-system > sync] Running vendir to: vendir sync
vendir sync --chdir=envs/zoo/_apps/metallb-system/vendor --file=../.myks/vendir.yaml --lock-file=../.myks/vendir.lock.yaml --file=-
9:13AM INF [zoo > metallb-system > sync] Synced
9:13AM DBG managed dirs=["ytt//"]
9:13AM DBG [zoo > metallb-system > sync] Checking directory envs/zoo/_apps/metallb-system/vendor
9:13AM DBG [zoo > metallb-system > sync] Checking directory envs/zoo/_apps/metallb-system/vendor/ytt
9:13AM DBG Checking dir dir=ytt// relPath=ytt/
9:13AM DBG ytt// has prefix ytt/
9:13AM DBG [zoo > metallb-system > sync] Checking directory envs/zoo/_apps/metallb-system/vendor/ytt/config
9:13AM DBG Checking dir dir=ytt// relPath=ytt/config/
9:13AM DBG [zoo > metallb-system > sync] Removing directory envs/zoo/_apps/metallb-system/vendor/ytt/config

With myks v2.2.0 the sync was working. But using now myks 2.2.0 to sync the app again is not solving the issue:

> myks220 sync ALL metallb-system
...
[zoo > metallb-system > sync] Resource already synced vendir dir=ytt/config/manifests
# cleanup before sync
> rm -rf envs/zoo/_apps/metallb-system/vendor
> myks220 sync ALL metallb-system
# -> File is recreated
  • myks 3.0.1 is cleaning up to much
  • myks 2.2.0 is skipping the sync if the vendor folder exists; is there any option to force syncing?

Checking why myks 3.0.1 is deleting the file:

> myks sync ALL metallb-system
9:17AM DBG [zoo > metallb-system > sync] Running ytt to: creating vendir config
ytt --file=.myks/tmp/data-schema.ytt.yaml --file=.myks/tmp/myks-data.ytt.yaml --file=prototypes/metallb/vendir -v myks.context.step=sync -v myks.context.app=metallb-system -v myks.context.prototype=prototypes/metallb
9:17AM DBG [zoo > metallb-system > sync] Running vendir to: vendir sync
vendir sync --chdir=envs/zoo/_apps/metallb-system/vendor --file=../.myks/vendir.yaml --lock-file=../.myks/vendir.lock.yaml --file=-
9:17AM INF [zoo > metallb-system > sync] Synced
9:17AM DBG managed dirs=["ytt//"]
9:17AM DBG [zoo > metallb-system > sync] Checking directory envs/zoo/_apps/metallb-system/vendor
9:17AM DBG [zoo > metallb-system > sync] Checking directory envs/zoo/_apps/metallb-system/vendor/ytt
9:17AM DBG Checking dir dir=ytt// relPath=ytt/
9:17AM DBG ytt// has prefix ytt/
9:17AM DBG [zoo > metallb-system > sync] Checking directory envs/zoo/_apps/metallb-system/vendor/ytt/config
9:17AM DBG Checking dir dir=ytt// relPath=ytt/config/
9:17AM DBG [zoo > metallb-system > sync] Removing directory envs/zoo/_apps/metallb-system/vendor/ytt/config

I think this might have been introduced with #151. -> Reverting the commit and syncing is fine:

>myks-151 sync ALL metallb-system 
...
9:25AM INF [zoo > metallb-system > sync] Synced
>git status
clean..

next finding: myks 3.0.1 with and without #151 are not syncing if the vendor folder does not exists:

> rm -rf envs/zoo/_apps/metallb-system/vendor
> myks sync ALL metallb-system
...
9:37AM DBG [zoo > metallb-system > sync] Starting
9:37AM TRC File does not exist path=envs/_apps/metallb-system/vendir
9:37AM TRC File does not exist path=envs/zoo/_apps/metallb-system/vendir
9:37AM DBG [zoo > metallb-system > sync] Running ytt to: creating vendir config
ytt --file=.myks/tmp/data-schema.ytt.yaml --file=.myks/tmp/myks-data.ytt.yaml --file=prototypes/metallb/vendir -v myks.context.step=sync -v myks.context.app=metallb-system -v myks.context.prototype=prototypes/metallb
9:37AM ERR [zoo > metallb-system > sync] Running vendir to: vendir sync
vendir sync --chdir=envs/zoo/_apps/metallb-system/vendor --file=../.myks/vendir.yaml --lock-file=../.myks/vendir.lock.yaml --file=-
9:37AM ERR [zoo > metallb-system > sync] vendir: Error: Running chdir: chdir envs/zoo/_apps/metallb-system/vendor: no such file or directory
9:37AM ERR [zoo > metallb-system > sync] Vendir sync failed error="exit status 1"
9:37AM FTL Unable to sync vendir configs error="exit status 1"

> mkdir envs/zoo/_apps/metallb-system/vendor
> myks sync ALL metallb-system
9:38AM DBG [zoo > metallb-system > sync] Starting
9:38AM TRC File does not exist path=envs/_apps/metallb-system/vendir
9:38AM TRC File does not exist path=envs/zoo/_apps/metallb-system/vendir
9:38AM DBG [zoo > metallb-system > sync] Running ytt to: creating vendir config
ytt --file=.myks/tmp/data-schema.ytt.yaml --file=.myks/tmp/myks-data.ytt.yaml --file=prototypes/metallb/vendir -v myks.context.step=sync -v myks.context.app=metallb-system -v myks.context.prototype=prototypes/metallb
9:38AM DBG [zoo > metallb-system > sync] Running vendir to: vendir sync
vendir sync --chdir=envs/zoo/_apps/metallb-system/vendor --file=../.myks/vendir.yaml --lock-file=../.myks/vendir.lock.yaml --file=-
9:38AM INF [zoo > metallb-system > sync] Synced
9:38AM DBG managed dirs=["ytt//"]
9:38AM DBG [zoo > metallb-system > sync] Checking directory envs/zoo/_apps/metallb-system/vendor
9:38AM DBG [zoo > metallb-system > sync] Checking directory envs/zoo/_apps/metallb-system/vendor/ytt
9:38AM DBG Checking dir dir=ytt// relPath=ytt/
9:38AM DBG ytt// has prefix ytt/
9:38AM DBG [zoo > metallb-system > sync] Checking directory envs/zoo/_apps/metallb-system/vendor/ytt/config
9:38AM DBG Checking dir dir=ytt// relPath=ytt/config/
9:38AM DBG [zoo > metallb-system > sync] Removing directory envs/zoo/_apps/metallb-system/vendor/ytt/config
kbudde commented

Why are we deleting vendored files?

Probably related to this comment:

myks/internal/myks/sync.go

Line 79 in 3e2bc38

// Paths are relative to the vendor directory (BUG: this will brake with multi-level vendor directory, e.g. `vendor/shmendor`)

Removing directory is happening here

myks/internal/myks/sync.go

Line 166 in 3e2bc38

log.Debug().Msg(a.Msg(syncStepName, "Removing directory "+path))

Should we create the vendir folder before sync?

We are setting chdir for vendir. Maybe we should create the folder before running the sync?

myks/internal/myks/sync.go

Lines 82 to 97 in 3e2bc38

vendorDir := a.expandPath(a.e.g.VendorDirName)
// TODO sync retry
if err := a.runVendirSync(vendorDir, vendirConfigFileRelativePath, vendirLockFileRelativePath, vendirSecrets); err != nil {
log.Error().Err(err).Msg(a.Msg(syncStepName, "Vendir sync failed"))
return err
}
vendirConfigFile := a.expandServicePath(a.e.g.VendirConfigFileName)
return a.cleanupVendorDir(vendorDir, vendirConfigFile)
}
func (a *Application) runVendirSync(targetDir string, vendirConfig string, vendirLock string, vendirSecrets string) error {
args := []string{
"sync",
"--chdir=" + targetDir,

Zebradil commented

@kbudde I'm sure it's because of the trailing slash here: https://github.com/kbudde/lab/blob/b243f91f1fc231a079860e0f979307cc035baa73/prototypes/metallb/vendir/base.ytt.yaml#L8

Could you confirm that #156 works for you?

  • 👍1
kbudde commented

Nice, it works :)

Zebradil commented

For the record: covered more cases in #158