Virtual Environment Guide

Managing software dependencies and configurations can be challenging in an HPC environment. Users often need different versions of the same software or libraries, leading to conflicts and complex setups. Environment modules provide a solution by allowing users to dynamically modify their shell environment using simple commands. This simplifies the setup process, ensures that users have the correct software environment for their applications, and reduces conflicts and errors caused by incompatible software versions. Environment modules work on the same principle as virtual environments, i.e. the manipulation of environment variables. If an environment module is not available for a given version you need, you can instead create a virtual environment using the standard version manager tools provided with many common languages. Virtual environments allow for managing different versions of lanugages and dependencies independent of the system version or other virtual environments, so they are often used by developers to isolate dependencies for different projects.

This guide provides different methods for creating virtual environments and managing dependencies accross multiple languages including Python, R, Julia, Rust, C, C++, and others. This allows you to create projects in isolated environments and install dependencies without the use of root or sudo access.

Python

How to create and use a virtual environment in Python

There are several different ways you can create a virtual environment and install packages for Python. The venv module that comes with Python 3 is a lightweight tool that provides a standard way to create virtual environments. It is suitable for simple projects with minimal external dependencies. virtualenv is another tool that serves a similar purpose, but works with both Python 2 and 3. In either case, you would use pip to manage and install the Python packages in the virtual environments created with venv or virtualenv. Another way that is often recommended is using conda, the package manager and environment manager tool provided by Anaconda. Anaconda is better suited for projects that require data science libraries and have more complex dependencies. All these tools allow you to manage virtual environments for Python and packages within that environment. Below are directions for using venv to illustrate the concept. The steps may vary with other tools.

Setup environment

Create a new Python virtual environment using venv.

  1. Ensure Python is installed: 
    python3 --version

    If not installed, use your distribution's package manager to install Python 3.

  2. Create a new directory for your project: 
    mkdir Project
  3. Navigate to the new directory: 
    cd Project
  4. Create a virtual environment: 
    python3 -m venv research1
  5. Check if the environment was created: 
    ls

    It should be listed as a directory in the results of running ls:

    research1
  6. Activate the environment for use: 
    source research1/bin/activate

    Your current line should be prefixed with the environment name:

    (research1) user@super-computer

Deactivate environment

Deactivate environment when finished:

deactivate

Installing packages

Install packages specific to your project.

  1. Upgrade pip to the latest version: 
    pip install --upgrade pip
  2. List installed packages: 
    pip list
  3. Install new package: 
    pip install package_name

    For a specific version: pip install package_name==1.0.0

  4. List installed packages again to ensure the needed packages are installed: 
    pip list

Saving dependencies

Saving your project's dependencies allows you to easily recreate your environment on another machine or share it with others.

  1. Save packages to a file: 
    pip freeze > requirements.txt
  2. Ensure the file was successfully created with package info: 
    cat requirements.txt
  3. Consider separating development and production dependencies: 
    pip freeze > requirements-dev.txt

Using dependencies

Install dependencies from a requirements file, when setting up a project on a new machine or collaborating with others.

  1. Install dependencies from requirements file: 
    pip install -r requirements.txt
  2. For development environments: 
    pip install -r requirements-dev.txt
  3. List installed packages to ensure dependencies are installed: 
    pip list

Running Python scripts

Run a script within the virtual environment:

  1. Ensure the virtual environment is activated
  2. Run the script: 
    python your_script.py

Delete virtual environment

Deleting the virtual environment removes all installed packages and the environment itself.

  1. Deactivate the environment if it's active: 
    deactivate
  2. Delete directory with virtual environment: 
    rm -rf research1

Other tools

Virtualenv

Install virtualenv

python -m pip --user virtualenv

Create an environment

virtualenv <environment_name>

Activate the environment

<environment_name>/bin/activate

Conda

Create an environment with conda

conda create --name <environment_name> python=<version>

Activate the environment

conda activate <environment_name>

R

How to create and use a virtual environment in R

There are several ways to manage project environments and install packages in R. The renv package is a popular tool for creating project-specific libraries. It's suitable for most R projects and works well with version control systems. Another option is packrat, which was a precursor to renv and serves a similar purpose. Regardless, both options allow you to manage project environments and packages within R. Below are directions using renv to create and manage a virtual environment.

Setup environment

Create a new R project and initialize it with renv

  1. Ensure R is installed: 
    R --version
  2. Create directory for project: 
    mkdir Project
  3. Navigate to directory: 
    cd Project
  4. Create an R project: 
    R
> usethis::create_project(".")
  5. Install renv for virtual environments: 
    > install.packages("renv")
  6. Initialize renv: 
    > renv::init()

    This creates a project-specific library and .Rprofile

Activate an environment

Activating an environment in R loads the project-specific library, ensuring that your code uses the correct versions of packages for your project.

> renv::activate()

Deactivate an environment

Deactivating an environment returns you to the global R library, which is useful when switching between projects.

> renv::deactivate()

Installing Packages

Installing packages in an renv environment adds them to your project-specific library.

  1. Install a package: 
    > install.packages("package_name")
  2. Record the new package in the lockfile: 
    > renv::snapshot()

Using dependencies

Save the state of your environment for reuse.

  1. Save environment state to lock file: 
    > renv::snapshot()
  2. Recreate another environment given the lock file: 
    > renv::restore()
  3. Update all packages in the project: 
    > renv::update()

Running R scripts

Run an R script within the project environment:

  1. Ensure you're in the project directory
  2. Run the script: 
    Rscript your_script.R

Sharing environments

To share your project environment:

  1. Include both renv.lock and .Rprofile in your version control system
  2. Others can recreate your exact environment using: 
    > renv::restore()

Delete virtual environment

Remove the renv directory and associated files. This deletes the environment and its packages.

  1. Exit R if you're in an R session

  2. Delete directory with virtual environment: 
    rm -rf renv
  3. Remove .Rprofile and renv.lock files: 
    rm .Rprofile renv.lock

Julia

How to create and use a virtual environment in Julia

Julia's built-in package manager, Pkg, provides functionality similar to virtual environments in other languages. The primary method is using project environments, which are defined by Project.toml and Manifest.toml files. These environments allow you to have project-specific package versions and dependencies. To create and manage these environments, you use Julia's REPL in package mode (accessed by pressing ])

Setup environment

Create a new project directory and activate it as a Julia environment.

  1. Check Julia version: 
    julia -version
  2. Create directory for project: 
    mkdir Project
  3. Navigate to directory: 
    cd Project
  4. Start Julia and enter package manager mode: 
    julia
julia> ]
  5. Create a new environment and activate: 
    (@v1.10) pkg> activate .

    This creates a new environment in the current directory.

  6. Create Project.toml and Manifest.toml files: 
    (@v1.10) pkg> instantiate

Activate environment

Activate an existing environment:

julia> ]
(@v1.10) pkg> activate /path/to/your/project

Deactivate environment

Deactivating an environment in Julia returns you to the default (global) environment.

(@v1.10) pkg> activate

Install packages

Installing packages in a Julia environment adds them to your project-specific manifest, keeping your project dependencies isolated.

  1. Install package: 
    (@v1.10) pkg> add Package

  2. Press backspace to exit package manager

  3. Ensure the new package is installed: 
    julia> using Package

Using dependencies

Install dependencies from your project files, ensuring consistent version usage.

  1. Enter package manager: 
    julia> ]
  2. Install dependencies from .toml files in the project directory: 
    (@v1.10) pkg> instantiate
  3. Update all packages: 
    (@v1.10) pkg> update

Running Julia scripts

Run a Julia script from within the project environment:

  1. Activate the environment in Julia REPL
  2. Run the script: 
    julia> include("your_script.jl")

Or from the command line:

julia --project=. your_script.jl

Managing multiple environments

Julia allows you to have multiple environments for different purposes:

  1. Create a test environment: 
    (@v1.10) pkg> activate --temp
  2. Switch between environments: 
    (@v1.10) pkg> activate /path/to/environment

Delete virtual environment

Removing the project-specific files deletes environment and its package information.

Delete the Project.toml and Manifest.toml files from your project directory.

Sharing environments

To share your project environment:

  1. Include both Project.toml and Manifest.toml in your version control system
  2. Others can recreate your exact environment using: 
    (@v1.10) pkg> instantiate

Node JS

How to create and use a virtual environment in Node JS

Node.js doesn't have traditional virtual environments like Python or R, but it uses npm (Node Package Manager) or Yarn to manage dependencies on a per-project basis. Each project typically has its own package.json file that lists dependencies and scripts. The node_modules folder in each project acts similarly to a virtual environment, containing project-specific packages. The directions below help to initialize a Node.js project using npm.

Setup environment

Create a new directory and initialize it with npm. This creates a package.json file to manage your project's dependencies.

  1. Check node version: 
    node -v
  2. Create directory for project: 
    mkdir Project
  3. Navigate to directory: 
    cd Project
  4. Initialize the project: 
    npm init -y
  5. Create a .nvmrc file that specifies node version: 
    node -v > .nvmrc

Install packages

Installing packages in a Node.js project adds them to your package.json file and the node_modules directory.

  1. Install a package and save it to package.json: 
    npm install package --save
  2. Install a development dependency: 
    npm install package --save-dev

Using dependencies

Use a specific Node.js version for your project and install or update all dependencies listed in your package.json file.

  1. Use .nvmrc for node version: 
    nvm use
  2. Install packages from package.json: 
    npm install
  3. Update all packages: 
    npm update

Managing different Node.js versions

Node Version Manager (nvm) allows you to install and use different versions of Node.js for different projects.

  1. Install a specific Node.js version: 
    nvm install 14.17.0
  2. Switch to a specific Node.js version: 
    nvm use 14.17.0

Running your Node.js application

To run your Node.js application:

node your-app.js

Or, if you've defined a start script in your package.json:

npm start

Creating and using an .npmrc file

An .npmrc file can be used to configure npm behavior for your project:

  1. Create an .npmrc file in your project root: 
    touch .npmrc
  2. Add configuration to the file, for example: 
    save-exact=true

    This will save exact versions of packages instead of using npm's default semantic versioning range operator.

Delete virtual environment

Remove the "virtual environment" by deleting the node_modules directory and optionally the package.json file. This removes all installed packages and dependency information.

  1. Delete the node_modules directory: 
    rm -rf node_modules
  2. Optionally, remove package.json and package-lock.json: 
    rm package.json package-lock.json

You can keep the package.json if you want to reinstall your dependencies later.

C and C++ Development Guide Using Home Directory

C

How to use the home directory for development with C

C doesn't have a built-in package manager or virtual environment system like more modern languages. Instead, C developers typically manage their development environment and dependencies manually. For system-wide installations, package managers like apt, yum, or Homebrew are often used to install libraries and tools. For user-specific or project-specific setups, developers commonly create custom directory structures in their home directory or project folders to house libraries, header files, and binaries. Environment variables like PATH, LD_LIBRARY_PATH, and CPATH are used to tell the compiler and linker where to find these custom installations. Build tools like Make, CMake, or Autotools are used to manage the compilation process and handle dependencies. This offers fine-grained control over the development environment and appeals to C's nature. Below are directions to set up the C environment using various directory and variable configurations.

Setup environment

Create directories for binaries, libraries, and include files, and setting up environment variables to use these directories.

  1. Verify installation: 
    gcc --version
make --version
  2. Make directory for local installations: 
    mkdir -p ~/local/{bin,lib,include}
  3. Add the following to your ~/.bashrc or ~/.bash_profile: 
    export PATH=$HOME/local/bin:$PATH
export LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH
export CPATH=$HOME/local/include:$CPATH
export PKG_CONFIG_PATH=$HOME/local/lib/pkgconfig:$PKG_CONFIG_PATH
  4. Reload shell configuration: 
    source ~/.bashrc  # or source ~/.bash_profile

Install packages

Installing packages (libraries) for C development in your home directory by downloading source code, compiling it, and installing it in your local directories.

Install a library with example libcurl:

wget https://curl.se/download/curl-7.78.0.tar.gz
tar xzf curl-7.78.0.tar.gz
cd curl-7.78.0
./configure --prefix=$HOME/local
make
make install

Create project

Create directory structure, source files, and a Makefile for easy compilation.

  1. Set up project structure: 
    mkdir my_c_project
cd my_c_project
mkdir src
  2. Create file that uses the library: 
    cat << EOF > src/main.c
#include <stdio.h>
#include <curl/curl.h>
   
int main(void) {
    CURL *curl = curl_easy_init();
    if(curl) {
        printf("CURL initialized successfully\n");
        curl_easy_cleanup(curl);
    } else {
        printf("Failed to initialize CURL\n");
    }
    return 0;
}
EOF
  3. Create a makefile for easy compilation: 
    cat << EOF > Makefile
CC = gcc
CFLAGS = -I$(HOME)/local/include
LDFLAGS = -L$(HOME)/local/lib
LIBS = -lcurl
   
main: src/main.c
	$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LIBS) -o $@
   
.PHONY: clean
   
clean:
	rm -f main
EOF
  4. Compile using: 
    make
  5. Clean using: 
    make clean

Using the project

To run your compiled C program:

./main

Updating libraries

To update a library, you can download the new version, compile, and install it similar to the initial installation process. For example, to update libcurl:

  1. Download and extract the new version
  2. Navigate to the extracted directory
  3. Run the installation commands again: 
    ./configure --prefix=$HOME/local
make
make install

Deleting the environment

In C, there's no virtual environment to delete. However, you can remove your local installations and project files:

  1. Remove local installations: 
    rm -rf ~/local
  2. Remove project directory: 
    rm -rf my_c_project

Remember to also remove or comment out the environment variable settings in your ~/.bashrc or ~/.bash_profile if you no longer need them.

C++

How to use the home directory for development with C++

Like C, C++ doesn't have a standardized built-in package manager or virtual environment system like some modern languages. Instead, C++ developers typically manage their development environment and dependencies manually or through third-party tools. For system-wide installations, package managers like apt, yum, or vcpkg are often used to install libraries and tools. For user-specific or project-specific setups, developers commonly create custom directory structures in their home directory or project folders to house libraries, header files, and binaries. Environment variables like PATH, LD_LIBRARY_PATH, and CPATH are used to tell the compiler and linker where to find these custom installations. Build systems like CMake, Make, or Ninja are widely used to manage the compilation process and handle dependencies. This offers fine-grained control over the development environment and appeals to the nature of C++. Below are directions to set up the C++ environment using various directory and variable configurations.

Setup environment

Creating directories for binaries, libraries, and include files, and create environment variables.

  1. Verify installation: 
    g++ --version
make --version
  2. Make directory for local installations: 
    mkdir -p ~/local/{bin,lib,include}
  3. Add the following to your ~/.bashrc or ~/.bash_profile: 
    export PATH=$HOME/local/bin:$PATH
export LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH
export CPATH=$HOME/local/include:$CPATH
export PKG_CONFIG_PATH=$HOME/local/lib/pkgconfig:$PKG_CONFIG_PATH
  4. Reload shell configuration: 
    source ~/.bashrc  # or source ~/.bash_profile

Install packages

Installing packages (libraries) for C++ development in your home directory by downloading source code, compiling it, and installing it in your local directories.

Install a library with example boost:

wget https://boostorg.jfrog.io/artifactory/main/release/1.76.0/source/boost_1_76_0.tar.gz
tar xzf boost_1_76_0.tar.gz
cd boost_1_76_0
./bootstrap.sh --prefix=$HOME/local
./b2 install

Create project

Create a C++ project by creating a directory structure, source files, and a Makefile for easy compilation.

  1. Set up project structure: 
    mkdir my_cpp_project
cd my_cpp_project
mkdir src
  2. Create a C++ file that uses the library: 
    cat << EOF > src/main.cpp
#include <iostream>
#include <boost/version.hpp>
#include <boost/algorithm/string.hpp>
   
int main() {
    std::cout << "Boost version: " 
              << BOOST_VERSION / 100000 << "."
              << BOOST_VERSION / 100 % 1000 << "."
              << BOOST_VERSION % 100 << std::endl;
   
    std::string str = "Hello, World!";
    boost::to_upper(str);
    std::cout << "Uppercase: " << str << std::endl;
   
    return 0;
}
EOF
  3. Create a Makefile for easy compilation: 
    cat << EOF > Makefile
CXX = g++
CXXFLAGS = -I$(HOME)/local/include
LDFLAGS = -L$(HOME)/local/lib
LIBS = -lboost_system
   
main: src/main.cpp
	$(CXX) $(CXXFLAGS) $(LDFLAGS) $^ $(LIBS) -o $@
   
.PHONY: clean
   
clean:
	rm -f main
EOF
  4. Compile using: 
    make
  5. Clean using: 
    make clean

Using the project

To run your compiled C++ program:

./main

Updating libraries

To update a library, you can download the new version, compile, and install it similar to the initial installation process. For example, to update Boost:

  1. Download and extract the new version
  2. Navigate to the extracted directory
  3. Run the installation commands again: 
    ./bootstrap.sh --prefix=$HOME/local
./b2 install

Deleting the environment

In C++, there's no virtual environment to delete. However, you can remove your local installations and project files:

  1. Remove local installations: 
    rm -rf ~/local
  2. Remove project directory: 
    rm -rf my_cpp_project

Remember to also remove or comment out the environment variable settings in your ~/.bashrc or ~/.bash_profile if you no longer need them.

Rust

How to simulate a virtual environment with rust

Rust uses Cargo, its built-in package manager and build system, to manage dependencies and create isolated project environments. Unlike some languages that require separate tools for virtual environments, Rust's approach integrates this functionality directly into its core toolchain. Each Rust project, initialized with cargo new, creates a self-contained environment with its own Cargo.toml file for declaring dependencies and build configurations. Cargo handles downloading, compiling, and linking of dependencies, ensuring that each project has its own isolated set of packages. Below are the directions for creating the development environment in rust.

Setup Environment

Create a new directory and intialize it with Cargo. This creates a new Rust project with its own Cargo.toml file for managing dependencies.

  1. Ensure Rust and Cargo are installed: 
    rustc -V
cargo -V
  2. Create directory for project: 
    mkdir Project
  3. Navigate to directory: 
    cd Project
  4. Initialize a Cargo project: 
    cargo new my_project
cd my_project

Install Packages

Installing packages in a Rust project adds them to your Cargo.toml file and downloads them to a local cache, keeping your project dependencies isolated and easily reproducible.

Install a package:

cargo add package_name

Using Dependencies

Building your Rust project with Cargo automatically downloads and compiles all necessary dependencies specified in your Cargo.toml file.

Install packages using Cargo.toml:

cargo build

Delete virtual environment

In Rust, there isn't a traditional "virtual environment" to delete. Rust's Cargo.toml handles this by default. To "delete" this environment:

  1. Remove the entire project directory: 
    cd ..
rm -rf my_project

This will remove the project files, Cargo.toml, and all compiled artifacts.

Note: If you want to keep the source code but remove all compiled artifacts and dependencies, you can instead run:

cargo clean

This will remove the target directory, which contains all compiled files and downloaded dependencies.