Conda Env List

 
  1. Conda Env List Of States
  2. Conda Env List
  3. Conda Env List Example
  4. Conda Env List Meaning
  5. Conda Env List Command
  1. # list all the conda environment available conda info -envs # Create new environment named as `envname` conda create -name envname # Remove environment and its dependencies conda remove -name.
  2. (The conda-env list command gives the same output.) So, question: how is conda finding environments created by a different user? Moreover, when the entire parent directory of the original instance of conda has been removed and replaced by one in an entirely different location (so no local environments.txt file could be cataloging this.

The command conda list seen previously displays all packages installed in the current environment. You can reduce this list by appending the particular package you want as an option. The package can be specified either as a simple name, or as a regular expression pattern.

Save the environment with conda (and how to let others run your programs)

If you have been developing in Python, you may have tried to distribute your program to friends and colleagues. It can be mildly annoying when they try to run your program and it fails because they don't have obscurePackage42 installed. If you are nearby, then it is easy for you to call pip install a few times and get them started with your program. If you are trying to distribute a program to end users (or even some non-technical executives) then you really want something that is going to work 'out of the box'.

Using an environment has the additional benefit of having us deal with one specific known version of Python. The problem of 'which Python am I using?' is one familiar to many of us.

The old way (and its drawbacks)

One way of doing this was to write a requirements.txt file. The format of this file was pretty simple:

A single command, pip install -r requirements.txt and everything would be written to the main Python repository.

While simple to use, there are a couple of different problems with this approach:

  • Version conflicts: What if one application required version 0.23.4 of Pandas, but a different application required 0.19.0 (because it used a now deprecated feature)? We would have to reinstall from requirements.txt when switching between these applications.
  • Tracking dependencies: It can be difficult to keep track of which packages your application is actually using. You don't want to include all installed packages on your machine, as only a few are relevant to your application.

Environments were designed to address both of these issues.

Environments

An environment is a way of starting with a new Python installation, that doesn't look at your already installed packages. In this way, it simulates having a fresh install of Python. If two applications require different versions of Python, you can simply create different environments for them. If you start from a fresh environment and install as you go, you are able to generate a list of all packages installed in the environment so that others can easily duplicate it.

There are many different environments and dependency managers in the Python ecosystem. The most common ones in use are virtualenv and conda (but there are others such as poetry, pyenv/pipenv, hatch and many more I haven't heard of). This article is about using conda to manage environments, although all of these tools share the same broad goals. Some of the differences between these tools are touched on in the Alternatives section.

There are two steps to using an environment (with a third step needed if you want to use Jupyter notebooks)

  1. Creating the environment, either from scratch (a new project) or from a yaml file (duplicating an environment)
  2. Activating the environment for use.
  3. Register the environment with Jupyter.

To leave an environment, we have to deactivate it. The quickstart below will walk through the typical workflow.

Using an environment (quickstart)

Let's say you wanted to create an environment test_env to do some testing with Python 3.6, and install numpy and Pandas. At the terminal, type the following:

If you want Jupyter notebooks to see your new environment, you need a couple of extra instructions. Jupyter sees the different environments as different kernels. Once we create a new environment, we need to tell Jupyter that it is there:

When loading a jupyter notebook, you can use the menu options Kernel->Change Kernel->test kernel to ensure you are using the test_env environment. (Update 2019-11-13: This process doesn't always work, this article gives you ways of checking and fixing it if it doesn't.)

Now you want to make an environment.yaml file that will allow others to recreate the environment from scratch. To make this file, we use the export command and send the output to environment.yaml:

Once we are done with the environment, we can deactivate and delete the environment:

Making the environment again from the yaml file

If you have the yaml file (created from conda env export), then recreating the environment is a single command:

Note that you don't need to supply the name of the new environment, as the yaml file also contains the name of the environment it saved. Make sure you don't give your environment an embarassing name, as everyone who recreates from the yaml file will see the name you used!

Finding conda environments on your system

Of course, you may choose to deactivate your environment but keep it around for later. If you want to see the environments installed on your system, use

Useful commands

Here is a brief summary of useful commands for environemnts. Anaconda has also published a cheat sheet with a more extensive list of useful commands.

Command
Create a new environment ENV_NAME with Python version 3.Xconda create --name ENV_NAME python=3.X
Create a new environment ENV_NAME with some initial packagesconda create --name ENV_NAME python=3.X pandas ipykernel
Create a new environment from a yaml fileconda env create --file environment.yaml
Activate the environment ENV_NAME (OSX, Linux)source activate ENV_NAME
Activate the environment ENV_NAME (Windows)activate ENV_NAME
Deactivate the current environment (*)source deactivate
Delete the environment ENV_NAMEconda env remove --name ENV_NAME
List all installed environmentsconda env list
Create a YAML file for active environment(*)conda env export > environment.yaml

The commands with (*) require you to have the environment active before using them. The naming is a little odd for creating environments: if creating them yourself the command is conda create ....., but if creating them from a yaml file we use conda env create ....... This is not a typo!

Conda

Tip for maximizing portability

Some packages are system dependent (e.g. the Python Image Library Pillow is used by OSX and Linux, but not Windows). Once you create your environment.yaml file, it is often a good idea to eliminate packages you don't use directly. For example, if you tell conda to install pandas but not numpy it will figure out it needs numpy for pandas to work. You should aim to eliminate all the packages in environment.yaml except the ones you actually import, so that conda can figure out which other packages are needed for the user's system (which may be running a different operating system to yours).

Summary

Creating environments allow us to make sure users we distribute our code to have the right packages (and the right versions of those packages installed) to run our code, without interfering with other programs. We activate and environment to start using it, and deactivate to leave again.

If making a new environment that you want others to use, the workflow is

If you are using an environment someone else has created:

Alternatives

  • The original virtualenv. As the Jake VanderPlas article 'Conda: Myths and Misconceptions' points out, these are mostly interchangle if you are only installing python packages into your environment. In slightly more detail
    • virtualenv/pip installs python packages into any environment, while
    • conda installs any packages into conda environments.

If you are solely installing Python packages, there is not much difference between the two.

  • pyenv/pipenv by Kenneth Reitz. The main goal of this project was to automate/simplify environment creation, but is not as mature as either virtualenv or conda solutions.
  • poetry by Sébastien Eustace, which aims to be a packaging and deployment tool.

With the exception of virtualenv, none of these solutions are as mature as conda. This is a piece of the Python that will hopefully improve and simplify, but for now, Randall Munroe's XKCD comic puts it well.

If your application uses Conda to manage dependencies, you face a dilemma.On the one hand, you want to pin all your dependencies to specific versions, so you get reproducible builds.On the other hand, once you’ve pinned everything, upgrades become difficult: you’ll start encountering the infamous The following specifications were found to be incompatible with each other error.

Conda Env List

Ideally you’d be able to both have a consistent, reproducible build, and still be able to quickly change your dependencies.And you can do this—with a little understanding, and a bit more work.

In this article you’ll learn:

  • Three ways of specifying your dependencies, and how they impede and/or enable reproducibility and upgrades.
  • Why in practice you want to have two different dependency files.
  • How to use a third-party tool, conda-lock, to easily maintain these two different files.

Three kinds of dependency specification

We have two goals, reproducibility and upgradability; we will limit our discussion to just dependencies, keeping in mind that both goals require more work than the limited focus of this article.

Focusing just on dependencies:

  1. Reproducibility: If we reinstall the code, we should get the same libraries.
  2. Upgradability: We should be able to change versions of our dependencies without having to fight the packaging system.
Env

Let’s see how different ways of specifying dependencies can achieve these goals.

Direct dependencies

Let’s say your application depends on Python and Pandas.You create an environment.yml for your dependencies:

You install Python and Pandas with conda env create, write some code, it runs correctly, all is well with the world.

Now, time passes, and you want to rerun the same analysis, with just a minor change to the code.Unfortunately, there’s been a new release of Pandas in the interim—and with a new release, there are differences:

  • It may have dropped some old APIs
  • Some APIs might behave differently on purpose, if the old behavior was a bug.
  • New bugs might have been introduced.

If you have to recreate the environment, it will install the latest version of Pandas: you might get different results, or maybe your code won’t run at all.Similarly, there might be a new version of Python, which can also cause problems.

On the flip side, from the perspective of the packaging infrastructure upgrades are trivial to achieve: they happen automatically every time you recreate your environment.

Versioned direct dependencies

Given how bad our environment.yml is at reproducible installs, we can constrain what it installs a little.We can add a version specifier to Python, Pandas, or both.For example:

This is a versioned direct dependency list; “direct” meaning “this package is something I directly import or run.”

This is an improvement in terms of reproducibility, but it still has issues.If you create a new environment with this file, you’ll see that many other packages are installed, dependencies of dependencies, and those are dependencies whose versions you didn’t specify:

All of those dependencies might change out from under you whenever you recreate the environment.For example, if you installed NumPy 1.19 this time, the next time you install you might get NumPy 1.20, which in theory could have a bug that changes Pandas’ results.

On the other hand, upgrades are still pretty easy: to switch to Python 3.9, just change the version of Python.

Transitively-pinned dependencies, aka locked dependencies

Once you that environment with all the dependencies installed, you can create new environment.yml that has the exact versions of all dependencies, including dependencies of dependencies.This is the “transitively-pinned” or “locked” dependency list, which you can create with conda env export:

In practice there are some technical issues with using conda env export, but we’ll put those aside for now.

For now, we can just notice that every time you create a new environment with this locked dependency file, you will get the exact same packages installed.As far as reproducibility is concerned, this is ideal.

Conda Env List Of States

But there’s a problem: upgrades are going to be hard.Let’s say we want to switch to Python 3.9, so we edit the YAML file to say that.Then we try to install:

So we now have reproducibility, but upgrades are quite difficult, perhaps even impossible without starting from scratch.

Choosing how to specify dependencies

Let’s summarize what we’ve learned about the three kinds of dependency specifications:

Dependency specificationReproducibilityUpgradability
Direct❌ Awful✓ Automatic
Versioned direct😐 OKish✓ Easy
Transitively pinned✓ Great❌ Awful

None of these options are ideal.But we can get both reproducibility and upgradability by having two files.

  1. You use the versioned direct file to generate the locked dependency file.
  2. When creating an environment, you use the locked dependency file.

This gives you the best of both worlds: most of the time you are just creating a new, reproducible environment from the locked dependency file.When you want to upgrade, you regenerate the locked file, and since you’re starting with a versioned direct dependency list, the hope is that the changes of dependencies-of-dependencies won’t be too bad.

And even if something breaks, at least it’ll break at a time of your choosing, rather than every time you recreate the environment.

Some technical difficulties with conda env export

In the example above we used conda env export to generate the locked file from the current environment; the environment in turn was created from the versioned direct environment.yml.This has some issues:

  1. You may have manually installed some files without adding them to environment.yml; the export will grab those too, so now your environment.yml and environment.lock.yml won’t match.
  2. The export file has a path entry at the end, which you probably want to delete before using.
  3. Conda has a bug where channels are exported in random order, instead of the sort order in the original environment.yml.
  4. Different operating systems might install different packages; if you conda env export on macOS, for example, the resulting lock file won’t work in Docker.

Luckily, there’s a tool that solves these issues.

Locking with conda-lock

Rather than creating an environment.yml, conda-lock creates a “lock file”, which is basically a set of URLs to download.This has the benefits of:

  • Speeding up installs, since you don’t have to wait for the Conda package resolver.
  • Allowing for reproducible builds, by transitively pinning the dependencies.

In addition, you can specify which operating system you want to build the lock file for, so you can create a Linux lock file on other operating systems.By default it generates for Linux, macOS, and 64-bit Windows out of the box, which is very convenient.

As a reminder, our environment.yml looks like this:

Here’s how you run conda-lock (you can install pip install conda-lock):

Conda env list packages

As explained in that message, you can now create an environment from the lock files, with a slightly different syntax that a normal conda env crateIn my case, I’ll use the Linux version:

And here’s what the conda-linux-64.lock file looks like:

Conda Env List

As requested, Python 3.8 and Pandas 1.0.

One caveat to keep in mind: pip dependencies won’t be included in the lockfile.To deal with those you can just have a two stage install, where you pip install once the environment has been created.There are a number of tools for managing lock files for pip/PyPI packages.

Conda Env List Example

Recap

Conda Env List Meaning

Since that was a lot, here’s what we’ve learned:

Conda Env List Command

  1. You want both reproducibility and easy updates.
  2. Versioned direct dependency files (“the packages my code imports”) give you easy updates; locked dependency files give you reproducibility.
  3. In practice, you want both.
  4. conda-lock lets you turn a direct dependency environment.yml into a lock file listing specific versions of the transitive dependencies.

Not using a lock file? It’s quick and easy: go and do it right now, and your builds will be reproducible going forward.