/HolyCommands

Include in all python projects

Primary LanguageVim script

python_custom_hooks

Include in all python projects
Inspired from https://dmerej.info/blog/post/how-i-lint/#fn:1

  1. pip install pylint, invoke
  2. create tasks.py with the below mentioned content
  3. optional: edit pylint according to https://dmerej.info/blog/post/some-pylint-tips/
  4. pip install pycodestyle
  5. add below lines in setup.cfg (include virtual environment directories)
[pycodestyle]
exclude=<some directory>
  1. vim checkcode.sh
#!/bin/bash -x

pycodestyle .
invoke pylint
  1. chmod 777 checkcode.sh
  2. usage ./checkcode.sh

optional to launch seperately:

  • pylint: invoke pylint
  • pycodestyle: pycodestyle .

tasks.py:

import os
import invoke


def get_pylint_args():
    for file in os.listdir("."):
        if file.endswith(".py") or os.path.exists(file + '/__init__.py'):
            yield file


@invoke.task
def pylint(ctx):
    del ctx
    invoke.run("pylint " + " ".join(get_pylint_args()), echo=True)

Jupyter extension:

  • pip install jupyter_contrib_nbextensions
  • pip install jupyter_nbextensions_configurator
  • (needed?) jupyter nbextensions_configurator enable --user
  • jupyter-contrib-nbextension install --system

Docker commands

Docker ps
Docker images
Docker run -it <> bash # (-p 8888:8888)(-d)
docker exec -it upbeat_panini /bin/bash #  (go into a running docker container)
docker commit -p cocky_goldberg tf_sai_test # (commiting changes)
docker rm $(docker ps -aq) # (remove all containers but not images and not the running ones)
docker run -it -p 0.0.0.0:6006:6006 -p 8888:8888 <TF docker image> # (docker with jupyter and tensorboard enabled)
Docker rmi <image name># (remove an image)
docker rmi $(docker images -f "dangling=true" -q)# (remove none dockers)
docker-compose -f <>.yml build --parallel
docker-compose -f <>.yml up
docker-compose -f <>.yml up -d --no-deps --build <service_name> # update one container defined as servicename fast
docker stats -a --format   'CPU: {{.CPUPerc}}\tMEM: {{.MemPerc}}' `docker ps -q`  > log_mem_AE # log memory
cat log_mem_AE | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g' | cut -d" " -f3 | cut -d "%" -f1 | sort -nr | head -n1 # find peak memory

Docker tensorflow on Bash on windows (you are unfortunate)

AWS guide:

AWS mounting EBS: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

Minikube:

Install as per: https://kubernetes.io/docs/tasks/tools/install-minikube/

Virtual env

apt-get install python-pip git wget curl lsb_release vim
pip install --upgrade pip
pip install virtualenv

sudo pip install virtualenv
virtualenv -p python3.5 venv1
. venv1/bin/activate
pip --no-cache-dir install -r req.txt
Deactivate
pip freeze -r requirements.txt

Jupyter:

pip3 install jupyter
pip install jupyter
jupyter-notebook --ip=0.0.0.0 --allow-root
sudo pip install jupyter_contrib_nbextensions jupyter_nbextensions_configurator # system wide installation
jupyter nbextensions_configurator enable --user
sudo jupyter-contrib-nbextension install --system

# activate virtualenv then execute the below commands
pip install --upgrade jupyterthemes
jt -t onedork -fs 95 -tfs 11 -nfs 115 -cellw 88% -T -N
  • To install the exact python version the virtualenv is using (python3.6 instead of python3)
python -m ipykernel install --user --name=.venv2
  • If you get error with port being empty or '*'
pip install notebook==5.6.0
  • If RuntimeWarning: coroutine 'WebSocketHandler.get' was never awaited
pip install "tornado<6"

SSH connections getting dropped off after 10 min: keep alive

K8s

Force delete k8 Pods

Managing multiple contexts

Memory

watch -n1 "grep ^VmPeak /proc/10122/status"