Setting Up a System

This tutorial will walk you through setting up NSP for a simple mock system so you can play around with it.

Important

You will need to have ansible installed. This can be in a python environment (venv or conda). Just run pip install ansible.

Note

This tutorial is not comprehensive in the details of each role. If you want a detailed explanation of how each role works see Ansible Roles. This tutorial is designed to give you working configuration files that you can explore and use the detailed docs to clarify if you have questions.

Setup Configuration Repository

Create a folder and initialize an empty git repository.

$ mkdir nsp_tutorial && cd nsp_tutorial
$ git init --initial-branch=main
Initialized empty Git repository in /home/software/nsp_docs/.git/

Next add the NSP repository as a submodule in the roles directory.

Note

You can create you own ansible.cfg but we provide one with good defaults in the nsp repository which we suggest you link in.

$ git submodule add https://github.com/olcf/nccs-software-provisioning.git roles
Cloning into '/home/software/nsp_tutorial/roles'...
Username for 'https://github.com': REDACTED
Password for 'https://REDACTED@github.com':
remote: Enumerating objects: 115, done.
remote: Counting objects: 100% (115/115), done.
remote: Compressing objects: 100% (77/77), done.
remote: Total 115 (delta 17), reused 110 (delta 12), pack-reused 0 (from 0)
Receiving objects: 100% (115/115), 33.95 KiB | 331.00 KiB/s, done.
Resolving deltas: 100% (17/17), done.
$ ln -s roles/ansible.cfg ansible.cfg
$ git add .gitmodules roles/ ansible.cfg
$ git commit -m "Adding NSP submodule."
[main (root-commit) 5310052] Adding NSP submodule.
2 files changed, 4 insertions(+)
create mode 100644 .gitmodules
create mode 160000 roles

Add a New System

Create a directory with the name of your system. For this tutorial we use the name moria.

$ mkdir moria

Next create a new playbook in the directory you just created for your system.

moria/playbook.yaml

- name: Deploy Moria
  hosts: localhost

  vars:
    NSP_system_name: moria
    NSP_install_root: "{{ ['/tmp', NSP_system_name] | path_join }}"
    NSP_help_email: example@example.com
    NSP_site_name: MySiteName

  roles: [ ]

To validate your setup run.

$ ansible-playbook moria/playbook.yaml
PLAY [Deploy Moria] ************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************
ok: [localhost]

PLAY RECAP *********************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Init Scripts

For most of our systems, we maintain a set of init scripts that are sourced when a user logs in. These scripts allow us to fine tune Lmod, set up convenience variables and more. As a first step for our new system we will add the init role to moria/playbook.yaml

moria/playbook.yaml

- name: Deploy Moria
 hosts: localhost

 vars:
   NSP_system_name: moria
   NSP_install_root: "{{ ['/tmp', NSP_system_name] | path_join }}"
   NSP_help_email: example@example.com
   NSP_site_name: MySiteName

 roles:
   - role: init

Various roles in NSP will add to the init scripts but if you have extra content, that you want added to the init scripts, you can do so by creating a profile.j2 and/or cshrc.j2 file in <system>/init. For this tutorial we will add some content to our init scripts that prints a welcome message.

moria/init/profile.j2

echo "Welcome to {{ NSP_system_name }}!!!"

moria/init/cshrc.j2

echo "Welcome to {{ NSP_system_name }}!!!"

Let’s run our playbook now and see what happens.

$ ansible-playbook moria/playbook.yaml

If you look, you will see that NSP has deployed our init scripts to /tmp/moria/init.

tree /tmp/moria

/tmp/moria/
└── init
    ├── cshrc
    └── profile

/tmp/moria/init/profile

 #!/usr/bin/env bash
 ##
 #| WARNING! This file is managed by Ansible.
 #| Do NOT make manual changes to this file.
 #| Please email example@example.com to request a change.
 #
 #| Info:
 #|   Role: init
 #|   Template: profile.j2
 #|   User: software
 ##

 # BEGIN INIT MANAGED
 echo "Welcome to moria!!!"
 # END INIT MANAGED

/tmp/moria/init/cshrc

 #!/usr/bin/env csh
 ##
 #| WARNING! This file is managed by Ansible.
 #| Do NOT make manual changes to this file.
 #| Please email example@example.com to request a change.
 #
 #| Info:
 #|   Role: init
 #|   Template: cshrc.j2
 #|   User: software
 ##

 # BEGIN INIT MANAGED
 echo "Welcome to moria!!!"
 # END INIT MANAGED

Bootstrap Lmod

The next step in setting up a system is installing Lmod. We will have to add a little more configuration to lmod when we get to setting up the spack role but for now let’s add the lmod role to our playbook.

moria/playbook.yaml

- name: Deploy Moria
  hosts: localhost

  vars:
    NSP_system_name: moria
    NSP_install_root: "{{ ['/tmp', NSP_system_name] | path_join }}"
    NSP_help_email: example@example.com
    NSP_site_name: MySiteName

  roles:
    - role: init
    - role: lmod
      vars:
        NSP_LMOD_version: 8.7.37
        NSP_LMOD_install_type: internal

After we run our playbook again we can source the init script and see our new software stack!

$ ansible-playbook moria/playbook.yaml
...
$ source /tmp/moria/init/profile
$ module avail

------------------------------------------------- [ Base Modules ] -------------------------------------------------
   DefApps (L)

  Where:
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

If you were to look at the init scripts you would see that they now have an additional section that was added by lmod.

/tmp/moria/init/profile

 #!/usr/bin/env bash
 ##
 #| WARNING! This file is managed by Ansible.
 #| Do NOT make manual changes to this file.
 #| Please email example@example.com to request a change.
 #
 #| Info:
 #|   Role: init
 #|   Template: profile.j2
 #|   User: software
 ##

 # BEGIN INIT MANAGED
 echo "Welcome to moria!!!"
 # END INIT MANAGED

 # BEGIN LMOD MANAGED
 type module > /dev/null 2>&1
 if [ "$?" -eq 0 ]; then
     clearLmod -q > /dev/null 2>&1
     unset LMOD_MODULEPATH_INIT
 fi

 export LMOD_SYSTEM_NAME=moria
 export LMOD_SYSTEM_DEFAULT_MODULES=DefApps
 export LMOD_PACKAGE_PATH=/tmp/moria/lmod/etc
 export LMOD_AVAIL_STYLE=nsp-pretty:system
 export LMOD_MODULERCFILE=/tmp/moria/lmod/etc/rc.lua
 export LMOD_ADMIN_FILE=/tmp/moria/lmod/etc/admin.list
 export LMOD_RC=/tmp/moria/lmod/etc/lmodrc.lua

 source /tmp/moria/lmod/lmod/init/profile
 module --initial_load --no_redirect restore
 # END LMOD MANAGED

Adding Software

We can now add a variety of software through different roles. For our purposes we will add one version of miniforge3, gcc and llvm each. Run the playbook and observe where they are installed and where their module files are placed.

Note

The gcc and llvm builds can take some time so be patient.

moria/playbook.yaml

- name: Deploy Moria
  hosts: localhost

  vars:
    NSP_system_name: moria
    NSP_install_root: "{{ ['/tmp', NSP_system_name] | path_join }}"
    NSP_help_email: example@example.com
    NSP_site_name: MySiteName

  roles:
    - role: init
    - role: lmod
      vars:
        NSP_LMOD_install_type: internal
        NSP_LMOD_version: 8.7.37
    - role: miniforge3
      vars:
        NSP_MINIFORGE3_version: 24.11.3
    - role: gcc
      vars:
        NSP_GCC_version: 14.2.0
    - role: llvm
      vars:
        NSP_LLVM_version: 19.1.0
        # if your system architecture is not x86_64 you will need to set `NSP_LLVM_targets`

$ source /tmp/moria/init/profile
$ module avail

------------------------------------------------- [ Base Modules ] -------------------------------------------------
   DefApps (L)    gcc/14.2.0    llvm/19.1.0    miniforge3/24.11.3-0

  Where:
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

$ tree -L 2 /tmp/moria/
/sw/moria/
├── gcc
│ └── 14.2.0
├── init
│ ├── cshrc
│ └── profile
├── llvm
│ └── 19.1.0
├── lmod
│ ├── 8.7.37
│ ├── bootstrap
│ ├── cache
│ ├── etc
│ └── lmod -> 8.7.37
├── miniforge3
│ └── 24.11.3-0
└── modules
    ├── DefApps.lua
    ├── gcc
    ├── llvm
    └── miniforge3

Spack

We are going to set up spack for the system gcc (mine is 13.3.0 but yours may be different) and the gcc and llvm versions that we built above. We will have a Core set of modules built by the system gcc and then a software stack built on gcc 14.2.0 and llvm 19.1.0.

We will name our spack environments according to the following schema core<year>.<month> and sw<year>.<month> (in this tutorial we will use core25.02 and sw25.02). Create the following files:

moria/spack/environments/compilers.yaml.j2

{{ ansible_managed | comment(beginning="##", end="##", decoration="#", prefix_count=0, postfix_count=0) }}

compilers:
  # system GCC
  - compiler:
      spec: gcc@{{ system_gcc.version }}
      paths:
        cc: /usr/bin/gcc
        cxx: /usr/bin/g++
        f77: /usr/bin/gfortran
        fc: /usr/bin/gfortran
      operating_system: {{ os.identifier }}
      modules: [ ]

  # GCC compiler
  - compiler:
      spec: gcc@{{ gcc.version }}
      paths:
        cc: {{ [NSP_install_root, 'gcc', gcc.version, 'bin/gcc'] | path_join }}
        cxx: {{ [NSP_install_root, 'gcc', gcc.version, 'bin/g++'] | path_join }}
        f77: {{ [NSP_install_root, 'gcc', gcc.version, 'bin/gfortran'] | path_join }}
        fc: {{ [NSP_install_root, 'gcc', gcc.version, 'bin/gfortran'] | path_join }}
      operating_system: {{ os.identifier }}
      modules:
        - gcc/{{ gcc.version }}

  # LLVM compiler
{% set gcc_v_list = gcc.version.split(".") %}
  - compiler:
      spec: clang@{{ llvm.version }}-gfortran{{ gcc_v_list[0] }}
      paths:
        cc: {{ [NSP_install_root, 'llvm', llvm.version, 'bin/clang'] | path_join }}
        cxx: {{ [NSP_install_root, 'llvm', llvm.version, 'bin/clang++'] | path_join }}
        f77: {{ [NSP_install_root, 'gcc', gcc.version, 'bin/gfortran'] | path_join }}
        fc: {{ [NSP_install_root, 'gcc', gcc.version, 'bin/gfortran'] | path_join }}
      operating_system: {{ os.identifier }}
      modules:
        - llvm/{{ llvm.version }}

moria/spack/environments/concretizer.yaml.j2

{{ ansible_managed | comment(beginning="##", end="##", decoration="#", prefix_count=0, postfix_count=0) }}

concretizer:
  reuse: false
  targets:
    granularity: microarchitectures
    host_compatible: true
  unify: false
  duplicates:
    strategy: none

moria/spack/environments/config.yaml.j2

{{ ansible_managed | comment(beginning="##", end="##", decoration="#", prefix_count=0, postfix_count=0) }}

config:
  install_tree:
    root: {{ [NSP_install_root, 'spack/envs', _SPACK_environment_name, 'opt'] | path_join }}
    projections:
      all: '{compiler.name}-{compiler.version}/{name}-{version}-{hash}'

  template_dirs:
    - $spack/share/spack/templates

  license_dir: $spack/etc/spack/licenses

  build_stage:
    - {{ [NSP_scratch_directory, "spack/stage", _SPACK_environment_name] | path_join }}

  test_stage: {{ [NSP_scratch_directory, "spack/test", _SPACK_environment_name] | path_join }}
  source_cache: {{ [NSP_install_root, "spack/envs/scache"] | path_join }}
  misc_cache: {{ [NSP_install_root, "spack/envs", _SPACK_environment_name, "mcache"] | path_join }}

  extensions: [ ]

  connect_timeout: 10
  verify_ssl: true
  ssl_certs: $SSL_CERT_FILE
  suppress_gpg_warnings: false
  checksum: true
  deprecated: false
  dirty: false
  build_language: C
  locks: true
  url_fetch_method: urllib
  build_jobs: {{ NSP_max_threads }}
  ccache: false
  concretizer: clingo
  db_lock_timeout: 60
  package_lock_timeout: null

  shared_linking:
    type: rpath
    bind: false

  allow_sgid: true
  install_status: true
  binary_index_ttl: 600

  flags:
    keep_werror: 'none'

  aliases:
    rm: remove
    search: list

moria/spack/environments/modules.yaml.j2

{{ ansible_managed | comment(beginning="##", end="##", decoration="#", prefix_count=0, postfix_count=0) }}

modules:
  prefix_inspections:
    bin:
      - PATH
    man:
      - MANPATH
    share/man:
      - MANPATH
    share/aclocal:
      - ACLOCAL_PATH
    lib:
      - LD_LIBRARY_PATH
    lib64:
      - LD_LIBRARY_PATH
    lib/pkgconfig:
      - PKG_CONFIG_PATH
    lib64/pkgconfig:
      - PKG_CONFIG_PATH
    share/pkgconfig:
      - PKG_CONFIG_PATH
    .:
      - CMAKE_PREFIX_PATH
  default:
    roots:
      lmod: {{ [NSP_install_root, "spack/modules"] | path_join }}
    enable:
      - lmod
    arch_folder: false
    lmod:
      core_compilers:
        - gcc@{{ system_gcc.version }}
      all:
        environment:
          set:
            {{ NSP_site_name.upper() }}_{NAME}_ROOT: "{prefix}"
        autoload: none
        suffixes:
          ^llvm-amdgpu: gpu
          ^cuda: gpu
          ^mpi: mpi
          +openmp: omp
          threads=omp: omp
      exclude_implicits: true
      verbose: true
      exclude: [ ]
      hash_length: 0
      hierarchy:: [ ]
      projections:
        ^llvm-amdgpu ^mpi: "{^mpi.name}-{^mpi.version}/rocm-{^llvm-amdgpu.version}/{compiler.name}-{compiler.version}/{name}/{version}"
        ^cuda ^mpi: "{^mpi.name}-{^mpi.version}/cuda-{^cuda.version}/{compiler.name}-{compiler.version}/{name}/{version}"
        ^llvm-amdgpu: "rocm-{^llvm-amdgpu.version}/{compiler.name}-{compiler.version}/{name}/{version}"
        ^cuda: "cuda-{^cuda.version}/{compiler.name}-{compiler.version}/{name}/{version}"
        ^mpi: "{^mpi.name}-{^mpi.version}/{compiler.name}-{compiler.version}/{name}/{version}"
        all: "{compiler.name}-{compiler.version}/{name}/{version}"
      core_specs: []

moria/spack/environments/packages.yaml.j2

{{ ansible_managed | comment(beginning="##", end="##", decoration="#", prefix_count=0, postfix_count=0) }}

packages:
  all:
    buildable: true
    providers:
      blas: [openblas]
      lapack: [openblas]
      mpi: [openmpi]

moria/spack/environments/core25.02/spack.yaml.j2

{{ ansible_managed | comment(beginning="##", end="##", decoration="#", prefix_count=0, postfix_count=0) }}

# OLCF {{ NSP_system_name }} {{ _SPACK_environment_name }} Spack Environment

spack:
  view: false
  include:
    - concretizer.yaml
    - modules.yaml
    - compilers.yaml
    - config.yaml
    - packages.yaml

  modules:
    default:
      lmod:
        core_compilers: [ gcc@{{ system_gcc.version }} ]
        projections:
          # core
          '%gcc@{{ system_gcc.version }}': '25.02/{name}/{version}'

  # -------------------------------------------------------------------
  # Specs Definitions
  # -------------------------------------------------------------------

  definitions:
    - core_compiler:
      - '%gcc@{{ system_gcc.version }}'

    - core_25.02:
      - matrix:
        - - cmake
          - tmux
          - wget
        - - $core_compiler

  specs:
    - $core_25.02

moria/spack/environments/sw25.02/spack.yaml.j2

{{ ansible_managed | comment(beginning="##", end="##", decoration="#", prefix_count=0, postfix_count=0) }}

# OLCF {{ NSP_system_name }} {{ _SPACK_environment_name }} Spack Environment

spack:
  view: false
  include:
  - concretizer.yaml
  - modules.yaml
  - compilers.yaml
  - config.yaml
  - packages.yaml

  # -------------------------------------------------------------------
  # Specs Definitions
  # -------------------------------------------------------------------

  definitions:
  - gcc_compilers:
    - '%gcc@{{ gcc.version }}'
  - llvm_compilers:
    - '%clang@{{ llvm.version }}'
  - all_compilers:
    - $gcc_compilers
    - $llvm_compilers

  - sw-25.02:
    - boost ~mpi
    - boost +mpi
    - openmpi

  # -------------------------------------------------------------------
  # Final Spec Matrices
  # -------------------------------------------------------------------

  - sw_cpu:
    - matrix:
      - - $sw-25.02
      - - $all_compilers

  specs:
  - $sw_cpu

moria/spack/environments/sw25.02/variables.yaml

# system gcc
system_gcc:
  version: 13.3.0
# GCC
gcc:
  version: 14.2.0
# LLVM
llvm:
  version: 19.1.0
# OS
os:
  identifier: ubuntu24.04

moria/spack/environments/core25.02/variables.yaml

# this should be a symlink to `moria/spack/environments/sw25.02/variables.yaml`

Finally we will need to add the spack role to our playbook.

moria/playbook.yaml

- name: Deploy Moria
  hosts: localhost

  vars:
    NSP_system_name: moria
    NSP_install_root: "{{ ['/sw', NSP_system_name] | path_join }}"
    NSP_help_email: example@example.com
    NSP_site_name: MySiteName

  roles:
    - role: init
    - role: lmod
      vars:
        NSP_LMOD_install_type: internal
        NSP_LMOD_version: 8.7.37
    - role: miniforge3
      vars:
        NSP_MINIFORGE3_version: 24.11.3
    - role: gcc
      vars:
        NSP_GCC_version: 14.2.0
    - role: llvm
      vars:
        NSP_LLVM_version: 19.1.0
        # if your system architecture is not x86_64 you will need to set `NSP_LLVM_targets`
    - role: spack
      vars:
        _shared_templates: &shared_L
          - compilers
          - concretizer
          - config
          - modules
          - packages
        _specific_templates: &specific_L
          - spack
        NSP_SPACK_versions:
          v0.23.1:
            git_reference: 2bfcc69
        NSP_SPACK_environments:
          core25.02: { spack_version: v0.23.1, specific_templates: *specific_L, shared_templates: *shared_L }
          sw25.02: { spack_version: v0.23.1, specific_templates: *specific_L, shared_templates: *shared_L }

After running the playbook explore /tmp/moria/spack/configs. To install our software via spack run the following.

cd /sw/moria/spack/configs
source spacktivate   # choose the core25.02 env
spack concretize
spack install
source spacktivate   # choose the sw25.02 env
spack concretize
spack install

All of our software should now be installed to /tmp/moria/spack/envs; however, none of it shows up for module avail yet, but all of the modules are in /tmp/moria/spack/modules.

Lmod Hook & Core

The final part of our Lmod configuration is setting up the NSP hook. This will make the software that we built with our sw25.02 spack environment visible. We will also create a module file to add the Core (a.k.a core25.02) environment that we installed.

moria/files/Core/25.02.lua

{{ ansible_managed | comment(beginning="--[[", end="]]--", decoration="", prefix_count=0, postfix_count=0) }}

help("Add path for Core 25.02 modules to MODULEPATH")

prepend_path{"MODULEPATH", "{{ [NSP_install_root, 'spack/modules/Core/25.02'] | path_join }}", priority=10}

Modify the playbook to include configuration for our hook and the files role which is where we are keeping the Core module files. Also add some modules to lmod’s NSP_LMOD_default_modules.

erebor.yaml

- name: Deploy Moria
  hosts: localhost

  vars:
    NSP_system_name: moria
    NSP_install_root: "{{ ['/sw', NSP_system_name] | path_join }}"
    NSP_help_email: example@example.com
    NSP_site_name: MySiteName

  roles:
    - role: init
    - role: lmod
      vars:
        NSP_LMOD_install_type: internal
        NSP_LMOD_version: 8.7.37
        NSP_LMOD_default_modules:
          - Core/25.02
          - gcc/14.2.0
        NSP_LMOD_hierarchy:
          compiler:
            members: [ gcc, llvm ]
            paths:
              - {path: '|compiler.name|-|compiler.version|', weight: 20}
              - {path: '|mpi.name|-|mpi.version|/|compiler.name|-|compiler.version|', weight: 30}
            level: 0
          mpi:
            members: [ openmpi ]
            paths:
              - {path: '|mpi.name|-|mpi.version|/|compiler.name|-|compiler.version|', weight: 30}
            level: 1
    - role: miniforge3
      vars:
        NSP_MINIFORGE3_version: 24.11.3
    - role: gcc
      vars:
        NSP_GCC_version: 14.2.0
    - role: llvm
      vars:
        NSP_LLVM_version: 19.1.0
        # if your system architecture is not x86_64 you will need to set `NSP_LLVM_targets`
    - role: spack
      vars:
        NSP_SPACK_versions:
          v0.23.1:
            git_reference: 2bfcc69
        NSP_SPACK_environments:
          core25.02:
            spack_version: v0.23.1
            shared_templates:
              - mirrors
          sw25.02:
            spack_version: v0.23.1
            shared_templates:
              - mirrors
    - role: files
      vars:
        NSP_FILES_inventory:
          - src: Core
            dest: "{{ [NSP_module_root, 'Core'] | path_join }}"

After running the playbook again our example software stack is complete! Run the playbook, source the init scripts and test out the new stack.

$ ansible-playbook moria/playbook.yaml
...
$ source /tmp/moria/init/profile
$ module load openmpi boost

$ module avail

------------------------------------------ [ gcc/14.2.0, openmpi/5.0.5 ] -------------------------------------------
   boost/1.86.0-mpi

-------------------------------------------------- [ gcc/14.2.0 ] --------------------------------------------------
   boost/1.86.0 (D)    openmpi/5.0.5 (L)

-------------------------------------------------- [ Core/25.02 ] --------------------------------------------------
   cmake/3.30.5    tmux/3.4    wget/1.24.5

------------------------------------------------- [ Base Modules ] -------------------------------------------------
   Core/25.02 (L)    DefApps (L)    gcc/14.2.0 (L)    llvm/19.1.0    miniforge3/24.11.3-0

  Where:
   L:  Module is loaded
   D:  Default Module

...

$ ml load llvm

Lmod is automatically replacing "gcc/14.2.0" with "llvm/19.1.0".

Due to MODULEPATH changes, the following have been reloaded:
  1) openmpi/5.0.5

$ module avail

------------------------------------------ [ llvm/19.1.0, openmpi/5.0.5 ] ------------------------------------------
   boost/1.86.0-mpi

------------------------------------------------- [ llvm/19.1.0 ] --------------------------------------------------
   boost/1.86.0 (D)    openmpi/5.0.5 (L)

-------------------------------------------------- [ Core/25.02 ] --------------------------------------------------
   cmake/3.30.5    tmux/3.4    wget/1.24.5

------------------------------------------------- [ Base Modules ] -------------------------------------------------
   Core/25.02 (L)    DefApps (L)    gcc/14.2.0    llvm/19.1.0 (L)    miniforge3/24.11.3-0

...