release system is down?
Jeffwan opened this issue · 8 comments
Seems CD system didn't pick changes from repos kubeflow/kubeflow for a while.
See centraldashboard latest image tag was built this Aug.
https://console.cloud.google.com/gcr/images/kubeflow-images-public/GLOBAL/centraldashboard?gcrImageListsize=30
Change list
https://github.com/kubeflow/kubeflow/commits/master/components/centraldashboard
I think kubeflow/kubeflow@8d36a38#diff-11c80890c5420bf75d645500ab81851788d9651eb6725194c9b5487e1e5827ee this change should be picked by CD.
I did some check and here's the latest autoPR
kubeflow/manifests#1592
I filed #804 and seems it doesn't build new image tags from new branch
/cc @thesuperzapper @Bobgy @jlewi @PatrickXYS @kubeflow/release-team
I'll take a look. Please take your time blocked on the issue thinking about each WG's own release infra, because 20% contributions from Google will always mean not enough bandwidth to maintain this stuff.
Haven't looked through, here's documentation for the release infra and it mentions postsubmit tests taking a key role in this.
So I wonder if the broken release system related to test infra changes: https://github.com/kubeflow/kubeflow/blob/master/docs_dev/releasing.md
/cc @PatrickXYS
I'm not familiar with release infra which is set up previously, but I can take a look to see if we can root-cause.
We have release-infra story alongside with test-infra, they're on our roadmap, will be discussing with WGs very soon
@Bobgy No problem. just a heads up. @thesuperzapper will raise this in community meeting. We can have some discussion later. Asking WG own release infra makes sense as well, however, we might miss some repos. Let's see. If AWS can sponsor the release infra, that is great as well
As @Bobgy says at this point the WG leads should be responsible for building and releasing their own images.
1.2 release doesn't have dependency on this issue. release infra is a separate story. As we discussed, this should be owned by each WG. For user who like to work on this part for your own release infra, check https://github.com/kubeflow/testing/tree/master/apps-cd