cms-opendata-analyses/AOD2NanoAODOutreachTool

Argo workflow failing

Closed this issue · 0 comments

Argo workflow fails with
##[error]Process completed with exit code 1.

This is due to the parameters passed in the workflow (see down below).
Can be easily fixed removing the parameters and giving nevents and file name directly in the workflow file, will be fixed asap.
Likely affects the CASTOR workflow as well

Not clear why this has changed.
Runs OK in a local minikube.

Comparing to the earlier successful test run:

  • Set up job:
    • runner version
    • Ubuntu 18.04.5 instead of 18.04.4
  • Start minikube: No changes in the minikube setup step compared to the earlier successful test runs.
  • Set up argo:
    • Argo version the same
    • Some changes in the argo setup step through kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/quick-start-postgres.yaml

For the argo setup, the output of the failing job has the following in addition to the old one :

customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io created
serviceaccount/github.com created
role.rbac.authorization.k8s.io/submit-workflow-template created
rolebinding.rbac.authorization.k8s.io/github.com created
secret/argo-workflows-webhook-clients created

Check if it works with older versions of quick-start-postgres.yaml:

https://raw.githubusercontent.com/argoproj/argo/71d6128154587f2e966d1fc2bad4195bc0b4fba8/manifests/quick-start-postgres.yaml

  • does not help
  • diff new old:
258a259,266                                                                                                                                                                          
>   - events                                                                                                                                                                         
>   verbs:                                                                                                                                                                           
>   - watch                                                                                                                                                                          
>   - create                                                                                                                                                                         
>   - patch                                                                                                                                                                          
> - apiGroups:                                                                                                                                                                       
>   - ""                                                                                                                                                                             
>   resources:                                                                                                                                                                       
259a268                                                                                                                                                                              
>   - serviceaccounts                                                                                                                                                                
619c628                                                                                                                                                                              
<         image: argoproj/argocli:latest                                                                                                                                             
---                                                                                                                                                                                  
>         image: argoproj/argocli:v2.11.0                                                                                                                                            690c699                                                                                                                                                                              
<         - argoproj/argoexec:latest                                                                                                                                                 
---                                                                                                                                                                                  
>         - argoproj/argoexec:v2.11.0                                                                                                                                                694c703                                                                                                                                                                              
<         image: argoproj/workflow-controller:latest                                                                                                                                 ---                                                                                                                                                                                  
>         image: argoproj/workflow-controller:v2.11.0    

or the version earlier https://raw.githubusercontent.com/argoproj/argo/5e60decf96e85a4077cd70d1d4e8da299d1d963d/manifests/quick-start-postgres.yaml

  • gives the same output (tbc in detail) as the successful job
  • the workflow still fails

To do: Compare the with the simple workflow which succeeds

  1. argo wf using OD container ok
  2. add entrypoint and git clone ok
  3. add parameters fails 💪