Nvidia APEX in base image

[atselousov]

Is it possible to add Nvidia Apex to the base image? As I know, the version which can be installed with pip does not support new features. It is not clear how to install this module without changing the base image using only tools that are provided by cookiecutter-neuro-project .

[atselousov]

As I understand, cookiecutter does not support any base image (var BASE_ENV_NAME) except neuromation/base:latest image because of changes related to tqdm after the last update to v1.3. So, it makes difficult to use your own environment with the template.

[atselousov]

@mariyadavydova, as I said during today's stand up, I've modified my setup command. It looks as follow:

PROJECT_FILES=requirements.txt apt.txt setup.cfg env.sh

.PHONY: setup
setup: ### Setup remote environment
	$(NEURO) mkdir --parents $(PROJECT_PATH_STORAGE) \
		$(PROJECT_PATH_STORAGE)/$(CODE_DIR) \
		$(DATA_DIR_STORAGE) \
		$(PROJECT_PATH_STORAGE)/$(CONFIG_DIR) \
		$(PROJECT_PATH_STORAGE)/$(NOTEBOOKS_DIR) \
		$(RESULTS_DIR_STORAGE)
	$(NEURO) run \
		--name $(SETUP_JOB) \
		--description "$(PROJECT_ID):setup" \
		--preset cpu-large \
		--detach \
		--env JOB_TIMEOUT=1h \
		--volume $(PROJECT_PATH_STORAGE):$(PROJECT_PATH_ENV):rw \
		$(BASE_ENV_NAME) \
		'sleep infinity'
	for file in $(PROJECT_FILES); do $(NEURO) cp ./$$file $(PROJECT_PATH_STORAGE)/$$file; done
	$(NEURO) exec --no-key-check $(SETUP_JOB) "bash -c 'chmod +x $(PROJECT_PATH_ENV)/env.sh && $(PROJECT_PATH_ENV)/env.sh'"
	$(NEURO) exec --no-key-check $(SETUP_JOB) "bash -c 'export DEBIAN_FRONTEND=noninteractive && $(APT) update && cat $(PROJECT_PATH_ENV)/apt.txt | xargs -I % $(APT) install --no-install-recommends % && $(APT) clean && $(APT) autoremove && rm -rf /var/lib/apt/lists/*'"
	$(NEURO) exec --no-key-check $(SETUP_JOB) "bash -c '$(PIP) -r $(PROJECT_PATH_ENV)/requirements.txt'"
	$(NEURO) --network-timeout 300 job save $(SETUP_JOB) $(CUSTOM_ENV_NAME)
	$(NEURO) kill $(SETUP_JOB)
	@touch .setup_done

The main difference is line $(NEURO) exec --no-key-check $(SETUP_JOB) "bash -c 'chmod +x $(PROJECT_PATH_ENV)/env.sh && $(PROJECT_PATH_ENV)/env.sh'", which is used to install additional packeges. In my case I used script env.sh to install apex and it looks as follow:

mkdir /project-env
cd /project-env

# apex
git clone https://github.com/NVIDIA/apex && cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

[atemate]

We included APEX to base env, thank you.