You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Analytics/Systems/Anaconda
Anaconda is a prepackaged conda distribution for mostly python based analytics and research purposes. WMF maintains a custom debian package of Anaconda that includes some extra packages, but also has scripts for creating 'stacked' conda user environments. These conda user environments allow users to install packages into their own conda environment without modifying the base anaconda environment.
Usage
Anaconda base environment
To use the readonly Anaconda base environment, you can simply run python or other executables directly out of /usr/lib/anaconda-wmf/bin
. If you prefer to activate the anaconda base environment, run source /usr/lib/anaconda-wmf/bin/activate
.
Creating a new conda user environment
Run
conda-create-stacked
and a new conda environment will be created for you in ~/.conda/envs. When used, this environment will automatically append the base conda environment Python load paths to its own. If the same package is installed in both environments, your user conda environment's package will take precedence.
If you prefer, you can name your conda environment
conda-create-stacked my-cool-env
Activating a conda user environment
There are several ways to activate a conda user environment. Just running
source conda-activate-stacked
On its own will attempt to guess at the most recent conda environment to activate. If you only have one conda environment, this will work.
You can also specify the name of the conda env to activate. Run /usr/lib/anaconda-wmf/bin/conda info --envs
to get a list of available conda environments. E.g.
source conda-activate-stacked otto_2020-08-17T20.52.02
Or, you can run the 'activate' script out if your conda environment path:
source ~/.conda/envs/2020-08-17T20.52.02_otto/bin/activate
Installing packages into your user conda environment
After activating your user conda environment, you can set http proxy env vars and install conda and pip packages. E.g.
export http_proxy=http://webproxy.eqiad.wmnet:8080 export https_proxy=http://webproxy.eqiad.wmnet:8080 conda install -c conda-forge <desired_conda_package> pip install <desired_pip_package>
These packages will be installed into the currently activated conda user environment.
Deactivating your user conda environment
source conda-deactivate-stacked
Or, since the user conda env's bin dir has been added to your path, you should also be able to just run
source deactivate
stacked conda environments
Conda supports activating environments 'stacked' on another one. However, all this 'stacking' does by default is leave the base conda environment's bin directory on your PATH. It does not allow for python dependencies from multiple environments.
Our customization fixes this. When conda-create-stacked is run, an anaconda.pth file is created in the new conda environment's site-packages directory. This file tells Python to add the anaconda-wmf base environemnt python search paths to its own. If a package is present in both environments, the stacked conda environment's version will take precedence.