unable to pull a image from my registry signed by trusted roots

[jazzsir]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

I created a secret and applied it to service accounts

kubectl -n admin create secret generic regcred \
    --from-file=.dockerconfigjson=/home/kf/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson

kubectl -n admin patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regcred"}]}'
kubectl -n admin patch serviceaccount default-editor -p '{"imagePullSecrets": [{"name": "regcred"}]}'
kubectl -n admin patch serviceaccount default-viewer -p '{"imagePullSecrets": [{"name": "regcred"}]}'

But I can't run below simple fairing code.

import os
import tensorflow as tf

class MyModel(object):
  def train(self):

    (x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data(path='/home/jovyan/mnist.npz')
    x_train, x_test = x_train / 255.0, x_test / 255.0

    model = tf.keras.models.Sequential([
      tf.keras.layers.Flatten(input_shape=(28, 28)),
      tf.keras.layers.Dense(128, activation='relu'),
      tf.keras.layers.Dropout(0.2),
      tf.keras.layers.Dense(10, activation='softmax')
    ])

    model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])

    model.fit(x_train, y_train, epochs=5)

    model.evaluate(x_test,  y_test, verbose=2)

if __name__ == '__main__':
    if os.getenv('FAIRING_RUNTIME', None) is None:
        from kubeflow import fairing
        from kubeflow.fairing.kubernetes import utils as k8s_utils

        DOCKER_REGISTRY = 'jazzsir.agtion.net/kubeflow-artifact'
        fairing.config.set_builder(
            'append',
            image_name='fairing',
            base_image='jazzsir.agtion.net/kubeflow-image/tensorflow-2-0-0-gpu-lab:1.0.0',
            registry=DOCKER_REGISTRY, 
            push=True)
        # cpu 1, memory 1GiB
        fairing.config.set_deployer('job',
                                    namespace='admin',
                                    pod_spec_mutators=[
                                        k8s_utils.get_resource_mutator(cpu=2,
                                                                       memory=5)]

                                   )
        # python3         
        #fairing.config.set_preprocessor('python', input_files=[__file__])
        fairing.config.run()
    else:
        remote_train = MyModel()
        remote_train.train()

With errors

[I 200623 11:57:12 config:123] Using preprocessor: <kubeflow.fairing.preprocessors.converted_notebook.ConvertNotebookPreprocessor object at 0x7fc60b013358>
[I 200623 11:57:12 config:125] Using builder: <kubeflow.fairing.builders.append.append.AppendBuilder object at 0x7fc60b921e48>
[I 200623 11:57:12 config:127] Using deployer: <kubeflow.fairing.builders.append.append.AppendBuilder object at 0x7fc60b921e48>
[W 200623 11:57:12 append:50] Building image using Append builder...
[I 200623 11:57:12 base:105] Creating docker context: /tmp/fairing_context_burywsy6
[I 200623 11:57:12 converted_notebook:127] Converting fairing2.ipynb to fairing2.py
[I 200623 11:57:12 docker_creds_:234] Loading Docker credentials for repository 'jazzsir.agtion.net/kubeflow-image/tensorflow-2-0-0-gpu-lab:1.0.0'
---------------------------------------------------------------------------
BadStateException                         Traceback (most recent call last)
<ipython-input-2-43201054428f> in <module>
     45         # python3
     46         #fairing.config.set_preprocessor('python', input_files=[__file__])
---> 47         fairing.config.run()
     48     else:
     49         remote_train = MyModel()

/usr/local/lib/python3.6/dist-packages/kubeflow/fairing/config.py in run(self)
    127         logging.info("Using deployer: %s", builder)
    128 
--> 129         builder.build()
    130         pod_spec = builder.generate_pod_spec()
    131         deployer.deploy(pod_spec)

/usr/local/lib/python3.6/dist-packages/kubeflow/fairing/builders/append/append.py in build(self)
     50         logger.warning("Building image using Append builder...")
     51         start = timer()
---> 52         new_img = self._build(transport, src)
     53         end = timer()
     54         logger.warning("Image successfully built in {}s.".format(end-start))

/usr/local/lib/python3.6/dist-packages/kubeflow/fairing/builders/append/append.py in _build(self, transport, src)
     68         self.image_tag = self.full_image_name(self.context_hash)
     69         creds = docker_creds.DefaultKeychain.Resolve(src)
---> 70         with v2_2_image.FromRegistry(src, creds, transport) as src_image:
     71             with open(self.context_file, 'rb') as f:
     72                 new_img = append.Layer(src_image, f.read(), overrides=metadata.Overrides(

/usr/local/lib/python3.6/dist-packages/containerregistry/client/v2_2/docker_image_.py in __enter__(self)
    378     # Create a v2 transport to use for making authenticated requests.
    379     self._transport = docker_http.Transport(
--> 380         self._name, self._creds, self._original_transport, docker_http.PULL)
    381 
    382     return self

/usr/local/lib/python3.6/dist-packages/containerregistry/client/v2_2/docker_http_.py in __init__(self, name, creds, transport, action)
    204     # Ping once to establish realm, and then get a good credential
    205     # for use with this transport.
--> 206     self._Ping()
    207     if self._authentication == _BEARER:
    208       self._Refresh()

/usr/local/lib/python3.6/dist-packages/containerregistry/client/v2_2/docker_http_.py in _Ping(self)
    273     # Make sure these got set.
    274     _CheckState(self._realm, 'Expected a "%s" in "www-authenticate" '
--> 275                 'header: %s' % (_REALM_PFX, challenge))
    276 
    277   def _Scope(self):

/usr/local/lib/python3.6/dist-packages/containerregistry/client/v2_2/docker_http_.py in _CheckState(predicate, message)
    155 def _CheckState(predicate, message = None):
    156   if not predicate:
--> 157     raise BadStateException(message if message else 'Unknown')
    158 
    159 

BadStateException: Expected a "realm=" in "www-authenticate" header: Basic realm=""

I think the problem is that fairing libraries can’t read “~/.docker/config.json” file.

What did you expect to happen:

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Fairing version: (use python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"): 0.7.0.1
• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): kfctl_istio_dex.v1.0.2
• Minikube version: kubeadm 1.15.3
• Kubernetes version: (use kubectl version): 1.15.3
• OS (e.g. from /etc/os-release): CentOS 7.4

NOTE: If you are using fair from master, please provide us the git commit hash.

[issue-label-bot]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[jinchihe]

Would you confirm is that same user (same user home ~/.docker/config.json) ?

[jazzsir]

@jinchihe I logged in my registry with ID: jazzsir on Host(//home/host01) to generate config.json file. And copied the file to Jupyter notebook(/home/jovyan/.docker)

Actually I successfully ran fairing with the config.json several weeks ago, I would like to know what I missed out.

[jinchihe]

@jazzsir If you worked out, and if this is Fairing bug, would please create a PR to fix that? Thanks!

[jazzsir]

I found that the value of WWW-Authenticate for basic access authentication in a response from my registry server was wrong.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/jupyter	0.84

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.