tomaszdudek7/airflow_project

build_images returns docker.errors.BuildError

Opened this issue · 5 comments

pedrovgp commented

I am on master, running in RHEL 7, with docker-py 3.7.2, Docker version 18.03.1-ce, build 9ee9f40.

When I run python build_images.py, I get:

2019-05-08 13:50:27,272 - build_images - INFO - No particular task name specified. Will build every image in /docker/.
2019-05-08 13:50:27,272 - build_images - INFO - Browsing ['task3', 'task1', 'task2']
2019-05-08 13:50:27,272 - build_images - INFO - Handling task3
2019-05-08 13:50:27,272 - build_images - INFO - Copying ./python/libraries/papermill_runner to ./docker/task3/papermill_runner
2019-05-08 13:50:27,272 - build_images - INFO - Copying ./python/libraries/result_saver to ./docker/task3/result_saver
2019-05-08 13:50:27,273 - build_images - INFO - Building image. (run script with -l to see docker logs)
2019-05-08 13:56:56,834 - build_images - INFO - Cleaning up.
2019-05-08 13:56:56,834 - build_images - INFO - Removing ./docker/task3/papermill_runner
2019-05-08 13:56:56,834 - build_images - INFO - Removing ./docker/task3/result_saver
Traceback (most recent call last):
File "build_images.py", line 105, in
ImagesBuilder(parser).build_images()
File "build_images.py", line 33, in build_images
self.build_task(directory)
File "build_images.py", line 40, in build_task
image, build_logs = self.cli.images.build(path=f'./docker/{directory_name}', tag=directory_name, rm=True)
File "/opt/anaconda3/lib/python3.6/site-packages/docker/models/images.py", line 287, in build
raise BuildError(chunk['error'], result_stream)
docker.errors.BuildError: The command '/bin/sh -c pip install virtualenv' returned a non-zero code: 1

mitant commented

The interface for docker-py has changed.
self.cli.images.build only returns an Image now
Update build_task to ignore the output of self.cli.images.build and this will let you build using this script
This means you won't get the build output anymore

If you want the build output looks like you need to get the APIClient from docker-py and call the low level build there. The low level build returns a stream of output but no Image object. The image object isn't used anyways..

mitant commented

here's some updated snippets to help you. Good luck! I'm just trying this repo out right now but it looks promising in terms of what I want to do.

def __init__(self, parser):
    self.args = parser.parse_args()
    self.loud = self.args.loud
    self.log = self.configure_and_get_logger()
    self.cli = docker.from_env()
    self.api = docker.APIClient()
def build_task(self, directory_name):
    self.log.info(f"Handling {directory_name}")
    try:
        self.copy_libraries(directory_name)
        self.log.info("Building image. (run script with -l to see docker logs)")
        build_logs = self.api.build(path=f'./docker/{directory_name}', tag=directory_name, rm=True)

        while True:
            try:
                output = self.parse_output(next(build_logs))
                if self.loud:
                    self.log.info(output)
            except StopIteration:
                self.log.info("Image built.")
                break
    finally:
        self.remove_libraries(directory_name)

def parse_output(self, raw) -> str:
    output = json.loads(raw)
    try:
        return output['stream'].strip('\n')
    except KeyError:
        return raw
  • 🎉1
pedrovgp commented
is2co commented

You can be done using the low-level APIs given in docker-py as follows:
`

def build_task(self, directory_name):
    self.log.info(f'Handling {directory_name}')
    try:
        self.copy_libraries(directory_name)
        self.log.info('Building image. (run script with -l to see docker logs)')

        docker_client = docker.Client(base_url='unix://var/run/docker.sock', timeout=10)
        build_logs = docker_client.build(path=f'./docker/{directory_name}', tag=directory_name, rm=True)
        for output in build_logs:
            json_output = json.loads(output)
            if self.loud and 'stream' in json_output:
                self.log.info(json_output['stream'].strip('\n'))
        self.log.info('Docker image built.')

        # build_logs = self.api.build(path=f'./docker/{directory_name}', tag=directory_name, rm=True)

        # while True:
        #     try:
        #         output = self.parse_output(next(build_logs))
        #         if self.loud:
        #             self.log.info(output)
        #     except StopIteration:
        #         self.log.info('Image built.')
        #         break
    finally:
        self.remove_libraries(directory_name)

`

tomaszdudek7 commented

Is this still an issue? Could any of you make a tiny pull request fixing this? :)