chanzuckerberg/cellxgene

debug cellxgene annotate on ubuntu

atolopko-czi opened this issue · 3 comments

atolopko-czi commented
debug cellxgene annotate on ubuntu
atolopko-czi commented

Need to have @ambrosejcarr test on ubuntu using --mlflow-env-manager conda -m https://cellxgene-annotation-public.s3.us-west-2.amazonaws.com/cell_type/models/hlca_20220919140324_conda.zip

atolopko-czi commented

Learnings:

  • The conda channel you use will determine whether a python package can be installed as a conda package or must use (nested) pip install; "conda-forge" channel has the ones we use as conda-level pkgs: scvi-tools, pytorch, xgboost, scanpy, but not all of these are available from the "conda" channel. Miniforge defaults to using conda-forge, so MLflow worked for me since I was using miniforge. For those using {ana,mini}conda, the "conda" channel was used and the MLflow conda.yaml file was misconfigured in this case--it specified conda packages for pre-compiled pip pkgs, and these conda pkgs were not available on the "conda" channel. The proposed fix is to explicitly specify "conda-forge" as the channel in the conda.yaml file.
  • miniforge conda channel should be used for M1, since it provides compiled packages for arm64. " miniforge started a few months ago because miniconda doesn't support aarch64" [https://stackoverflow.com/questions/60532678/what-is-the-difference-between-miniconda-and-miniforge]
  • MLflow's support for virtualenv is experimental and introduces some complexities. Thus, we should not recommend using --mlflow-env-manager virtualenv. And as this is the default option value, we should change that conda. However, this will require that users install conda.
    • It requires installing pyenv (if using MLflow conda env, pyenv is not needed)
    • on ubuntu, g++, make must also be installed, and compilation errors can still occur

Testing matrix:
https://docs.google.com/spreadsheets/d/1pPBq3xNzw7k3W_uYgWI7GhEpdrcdZt180DZ_-xiqLd8/edit#gid=0