modules

Custom lmod modules for use on sherlock

First time set up

Execute echo "export MODULEPATH=/home/groups/trc/software/modules:$MODULEPATH" >> ~/.bashrc from a sherlock node. To use our modules in that session, execute source ~/.bashrc to update your MODULEPATH. In future sessions, MODULEPATH will be updated upon login.

Adding a custom module

To add a module you need to do two things:

• add a lua file to this repo describing your package, it's requirements and instructions to lmod on how to load it
• install your package on GROUP_HOME. no version control here, scout's honor. consider setting permissions so that only you have write permission for the packages you 'own'

We have adopted the organizational scheme used by SRCC which you must follow.

The lua file must be listed as [package-name]/[version].lua in this repo, which is deployed to /home/groups/trc/software/modules.

The package files must be installed under /home/groups/trc/software/user/open/[package-name]/[version]/

Installing package files

Python

1. Load module dependencies
2. mkdir -p /home/groups/trc/software/user/open/[package-name]/[version]/
3. PYTHONUSERBASE=/home/groups/trc/software/user/open/[package-name]/[version]/ pip install --user $PACKAGE==$VERSION

For example, if you were installing ray 1.13.0 for Python 3.9.0, you would run:

1. module load python/3.9.0 py-numpy/1.20.3_py39 etc
2. mkdir -p /home/groups/trc/software/user/open/py-ray/1.13.0_py39/
3. PYTHONUSERBASE=/home/groups/trc/software/user/open/py-ray/1.13.0_py39/ pip install --user ray==1.13.0

and the corresponding py-ray/1.13.0_py39.lua file would include

prepend_path("PYTHONPATH", "/home/groups/trc/software/user/open/py-ray/1.13.0_py39/lib/python3.9/site-packages")

Packages distributing binaries would also include

prepend_path("PATH", "/home/groups/trc/software/user/open/py-ray/1.13.0_py39/bin")

Push the lua file and update the clone at /home/groups/trc/software/modules, and you've installed the ray 1.13.0 module.

Note that pip install --target $INSTALLATION_DIRECTORY $PACKAGE==$VERSION does not work as one might expect, as the --target flag silently enables the --ignore-installed flag, which defeats the point of using lmod

Non-python packages

NOTE: for the time being binaries are not version controlled, so it's scouts honor to make sure that the binaries you put in $GROUP_HOME match the lua file and are write protected from everyone but you.

1. compile the package on sherlock
2. copy the resulting binaries to the /bin/[package name] in this repo
3. make sure that you have git-lfs installed and set git-lfs to track the binaries
4. push
5. update the copy of the repo that lives at $GROUP_HOME/software/modules
6. copy the binaries to /home/groups/trc/software/user/open/[package-name]/[version]/bin
7. assuming you have written the lua file to point to the the directory you just copied the bin to, your module is now installed