High-Performance Computing (Argon Supercomputer)

1 Overview

Analysis scripts for High Performance Computing (HPC) at the University of Iowa. Currently uses Argon and IDAS.

1.1 Argon vs. IDAS

Both Argon and IDAS rely on HPC to run intensive, large-scale scripts and processing jobs. See the Argon vs. IDAS comparison page (archived at https://perma.cc/Q7G5-4JA3) for more information about which option to use.

Argon runs on CentOS Linux, which requires users to have a basic understanding of command-line interface (CLI). IDAS is similar to a remote desktop in that users can open and interact with Rstudio, Jupyter Notebook, and Visual Studio Code while benefiting from the higher processing capabilities of the HPC system. This documentation focuses primarily on Argon.

1.3 Data Storage

Significant amounts of data storage are provided on Argon, but data are not backed up in any way unless special arrangements are made. Standard (free) Argon accounts include 1 TB of storage, and standard IDAS accounts include 100 GB of storage. It is the responsibility of the user to back up important information.

1.4 Sensitive Data

User agrees to refrain from storing Restricted data on HPC resources. Data is classified as Restricted when the unauthorized disclosure, alteration or destruction of that data could cause a significant level of risk to the University or its affiliates. Examples of Restricted data include data protected by state or federal privacy regulations and data pertaining to identified human subjects that has not been deidentified.

1.5 Fees

At present, there are no fees for the use of the Argon cluster or IDAS for low-priority usage. For large users, or those who want access to dedicated resources, the option of purchasing or renting supplemental system hardware may be available. If you are interested in dedicated hardware, contact research-computing@uiowa.edu.

2 Accessing Argon via SSH

SSH, or Secure SHell, is a tool that allows users to access and operate remote servers (i.e., other computers). You do not need to download or install anything to use SSH. Users can implement SSH via the terminal or via an emulator software, such as SecureCRT or PuTTY.

Emulators allow users to enter commands into the terminal and often include GUIs that show the file structure of the remote server. This can be helpful for users who have not used Linux, which operates via command line input and does not have an equivalent to Windows File Explorer or Mac Finder.

With SSH access, you can view and edit files as well as run scripts and submit jobs.

2.1 Using the Terminal

  1. Open Command Prompt (Windows) or Terminal (MacOS)
  2. If you are on campus and/or using the VPN, enter ssh hawkid@argon.hpc.uiowa.edu
    • If you are off campus and not using the VPN, enter ssh -p 40 hawkid@argon.hpc.uiowa.edu
  3. Enter your HawkID password when prompted
    • Note: The individual characters of your password will not appear as you type–this is not a bug!
  4. Complete the two-factor authentication

2.2 Using SecureCRT

SecureCRT is the University’s choice of emulator, and it has several customizable settings for more advanced users.

  1. Download SecureCRT from UIowa Informational Technology Services: https://its.uiowa.edu/securecrt
  2. Enter argon.hpc.uiowa.edu as the Hostname and your HawkID as the Username
    • Do not modify any other fields/settings
  3. Click ‘Connect’
  4. Enter your HawkID password when prompted
  5. Complete the two-factor authentication

2.3 Using PuTTY

PuTTY is another popular emulator that is more lightweight but less customizable than SecureCRT. It is not available through ITS, but it is free to download from the creators.

  1. Download PuTTY: https://www.chiark.greenend.org.uk/~sgtatham/putty/
  2. Enter argon.hpc.uiowa.edu as the Host Name
  3. Click ‘Connect’
    • The first time you connect, click Accept on the pop-up alert
  4. Enter your HawkID after the login as prompt
  5. Enter your HawkID password when prompted
  6. Complete the two-factor authentication

2.4 After Log-In

After logging in, the terminal or emulator will take a moment to connect to Argon. Once a connection is established, you will see [hawkid@argon-itf-login-# ~]$ as the final line of text.

3 Map Argon Drive to Local Computer

Mapping your Argon directory to your computer allows you to access your Argon files without CLI. You won’t be able to submit jobs, but it makes editing and moving files much more convenient for those who are new to Linux.

3.1 Windows

  1. Open File Explorer
  2. In the sidebar on the left, right click ‘This PC’, then select ‘Map Network Drive’
  3. Enter \\data.hpc.uiowa.edu\argon_home as the Folder
  4. Ensure that the ‘Reconnect at sign-in’ box is checked
  5. Click ‘Finish’

3.2 Mac

  1. Open Finder
  2. In the menus along the top, select ‘Go’ then ‘Connect to Server…’
  3. Enter smb://IOWA;<hawkid>:*@data.hpc.uiowa.edu/argon_home as the Server Address
  4. Click ‘Connect’
  5. Log in using your HawkID username and password if prompted

4 Argon Navigation

4.1 Quick Reference

Below is a list of common commands with a brief explanation of what each does. More detailed information for each command is located in the corresponding section of the page.

Accessing Argon

  • ssh hawkid@argon.hpc.uiowa.edu logs in to Argon

Nodes

  • qlogin requests a computing node
  • exit closes out of whichever node/environment/connection you are in

File Systems

  • pwd shows the path to the current folder (the working directory)
  • cd insert/file/path brings you to the provided file path (sets the working directory)
    • cd ~ returns to home directory (your Argon folder)
    • cd .. returns to the folder “above” the current folder (i.e., the folder that the current folder is in)
    • cd /Shared/lss_itpetersen/Lab goes to the lab drive
  • ls lists the files and folders in your current folder
  • nano filename.extension opens or creates a file
  • dos2unix filename.extension converts CRLF (Windows) line endings to LF (Linux) line endings
  • rm filename.extension deletes the specified file
  • cp filename.extension newFileOrFolder copies the contents of the specified file into the provided location (folder or file)
  • mv filename.extension new/folder/path moves the specified file to the provided location
    • mv filename.extension newFileName.extension renames the specified file to the new name

Modules

  • module load moduleName loads the specifed module
  • module list lists the modules loaded in the current session/environment
  • module avail optionalSearchTerm lists the modules available to use, narrowing down to modules matching the search term if provided

Containers

  • apptainer build container.sif defFile.def builds a container named ‘container.sif’ based on the definition file named ‘defFile.def’
  • apptainer run container.sif runs the default command in the specified container
    • apptainer run --app appName container.sif runs the specified app’s default command in the specified container
  • apptainer exec container.sif commandName commandArguments runs the specified command (with any provided arguments) in the specified container

Jobs

  • qsub jobFile.job submits the specified job file to the queue
  • qstat -u hawkid lists the active jobs associated with the specified HawkID, including job numbers
  • qdel jobNumber cancels the job associated with the specified job number

Miscellaneous

  • Rscript path/to/script.R runs the specified R script (if an R module is loaded)
    • Rscript path/to/script.R outputFile.txt runs the specified R script and saves the console output to the specified output file
    • See the HPC documentation (archived at https://perma.cc/99JQ-43ZG) for more information

4.2 Nodes

After logging in, you are assigned to a log-in node. There are limited log-in nodes, and they are shared by all Argon users. You can create, edit, and delete files, and you can submit jobs for processing, but you should not do any actual computing from the log-in node. To do more intensive work, you will need to request a computing node by entering qlogin into the terminal or emulator.

You can think of the login nodes as the host stand at a restaurant–you go there when you first arrive (log in) and ask for a table (computing node), and the host (queue manager) seats you when one is available. However, it would be rude and inconvenient to order your food and eat it (run a bunch of scripts) at the host stand.

4.3 File Systems

To interact with files in Argon, you must be able to navigate through the file directory. On Windows and MacOS, File Explorer and Finder allow users to locate the files they need. In Argon (and other Linux-based systems), users navigate through directories (folders) using the command line. There are a few commands that are helpful for moving through folders:

  • pwd shows the full file path to the current folder
    • When you first log in to Argon, you will be in your home directory, /Users/hawkid
    • The home directory can be abbreviated to ~
  • cd insert/file/path allows you to move to different folders
    • If you are moving to a folder that is somewhere inside of the current folder, you can use the relative file path
      • Example: To move from the main Lab folder in the lab drive to the Studies folder, enter cd Studies
      • Example: To move from the main Lab folder in the lab drive to your Members folder, enter cd Members/hawkid
    • If you are moving to a folder that is outside of the current folder, you must use the absolute (full) file path
      • Example: To move from your Argon home directory to the Studies folder on the lab drive, enter cd /Shared/lss_itpetersen/Lab/Studies
    • If the folder name has any spaces in it, put quotation marks around the file path
      • Example: cd "Studies/School Readiness Study"
    • cd ~ will bring you back to your home directory (i.e., your Argon storage folder)
    • cd .. will bring you to the folder “above” the folder that you are currently in
      • Example: If you are in the School Readiness Study folder in the lab drive, cd .. will bring you to the Studies folder
  • ls lists all of the files and folders in your current folder
    • All folders contain . (the current folder) and .. (the parent folder)
    • Below are some common arguments used with ls (arguments can be combined):
      • ls -A lists all files and folders, including hidden ones but excluding . and ..
      • ls -d lists folders only
      • ls -l lists folder contents with permissions, owner, size, and modification date
      • ls -R lists contents of current folder and all subfolders
      • ls /file/path lists the contents of the specified folder
  • mkdir folder/path creates the specified folder in the specified location
    • mkdir -p folder/path creates the specified folder in the specified location, creates any parent folders along the path if needed, and does not throw an error if the folder already exists

In addition to navigating directories, there are a number of commands that are helpful for interacting with files:

  • nano filename.extension opens the specified file if it exists or creates the specified file if it does not
  • dos2unix filename.extension converts CRLF (Windows) line endings to LF (Linux) line endings
    • Windows and Linux use different characters to represent a line break, resulting in \r: command not found errors when running Windows files on Linux
    • The dos2unix command replaces the Windows line break character (\r\n) with the Linux line break character (\n)
    • file filename.extension | grep CRLF checks for the CRLF character and returns a list of instances if found or nothing if not found
    • find ~/folder/path -name '*extension' -exec dos2unix "{}" \; runs the dos2unix command on all files with the provided extension found in the provided directory
    • sudo apt update && sudo apt install -y dos2unix installs the dos2unix package (you may be prompted for your password)
  • rm filename.extension removes the specified file
    • rm -rf folder/path/* removes all files/folders in the specified folder
      • Use with caution–the f argument means it will be deleting without confirming
  • cp fileOrFolder fileOrFolder copies the contents of the first argument to the location of the second argument
    • cp filename.extension newFile.extension copies the contents of the first file to the second file
      • Use with caution–if the second file already exists, the content will be overwritten
    • cp filename.extension folder/path/ copies the specified file into the specified folder
    • cp -r folder/path new/folder/path copies the first folder and all of its contents into the second folder
    • cp filename1.extension filename2.extension new/folder/path copies all listed files into the provided folder
  • mv fileOrFolder new/folder/path moves the specified file or folder to the specified location
    • mv filename.extension newFile.extension renames the specified file to the name specified in the second argument
      • Use with caution–if the second file already exists, the content will be overwritten
    • mv filename.extension new/folder/path moves the specified file to the specified location
    • mv -r folder/path new/folder/path moves the first folder and all of its contents into the second folder
    • mv filename1.extension filename2.extension new/folder/path moves the specified files into the provided folder

4.3.1 Accessing the Lab Drive

After accessing Argon using SSH, enter cd /Shared/lss_itpetersen/Lab in the terminal or emulator.

4.3.2 Project Directories

Like the Lab drive, it is recommended that you store the materials for each project in its own folder:

cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/
cd /Users/itpetersen/Documents/Projects/EXT_pilot/
cd /Users/itpetersen/Documents/Projects/EXT_ParentReport/
cd /Users/itpetersen/Documents/Projects/Multiple_Imputation/
cd /Users/itpetersen/Documents/Projects/SelfRegulation_IRT/
cd /Users/itpetersen/Documents/Projects/Test/

4.4 Bash Scripting

In order to use Argon, it is necessary to know some basic bash scripting. bash is a shell, which is a type of scripting language used to interact with operating systems. The commands described in the preceding sections are all bash commands, but there are a handful of additional commands that are helpful for setting up containers and job scripts.

4.4.1 Keyboard Shortcuts

  • Ctrl + C cancels/interrupts commands
  • Ctrl + Shift + C or right clicking copies highlighted text
  • Ctrl + Shift + V or right clicking pastes text

4.4.2 Syntax & Symbols

  • Folder paths and file names with spaces should be entered in quotation marks (")
  • When creating variables, do not include spaces around the equal sign (e.g., use varName=value rather than varName = value)
  • When referencing variables, use ${varName}
    • If inserting variables into a string, use
  • # at the start of a line indicates comments
  • ; separates commands written on the same line
    • Note: Unless you have very short commands, it is generally preferable to just put the commands on separate lines
  • \ continues a command onto the next line
  • | passes the results of one command into the next command
  • ${varName} indicates a variable

    • "${varName}" (in double quotes) will insert the value of the variable

    • '${varName}' (in single quotes) will keep variable name

    • As an example, run the following commands in Argon:

      test_var="I love Argon!"
echo "${test_var}"
echo '${test_var}'

    • ${varName[n]} selects the nth value from the varName list

  • $(function) inserts the result of the specified function
    • If the function has multiple results (and you want all of them), use ($(function))
  • ./execFile runs the specified executable file/program

4.4.3 Advanced Package Tool Commands

The Advanced Package Tool, or apt, is used to install, update, and otherwise manage packages in Debian-based Linux systems, including Argon. Although you likely will not use these commands in a job script, they are common in definition files. Outside of definition files, these commands should be run with sudo (see Section 4.4.4).

  • apt update retrieves a list of installed packages that need upgrades
  • apt install packageName installs the specified package(s)
    • apt install -y packageName installs the package(s) non-interactively (i.e., assuming ‘yes’ responses to any prompts)
    • apt install --no-install-recommends packageName installs the package(s) with required dependencies only (default installation installs required and recommended packages)
  • apt list --installed lists all installed packages
  • apt-get clean clears temporary files from the package installation folder
    • apt-get clean -y clears files non-interactively
  • apt-get autoremove removes unnecessary packages
    • apt-get autoremove -y removes unnecessary packages non-interactively

4.4.4 Other Useful Commands

  • man insertCommandName opens instructions for the provided command
  • sudo insertCommandName runs the provided command with administrative privileges
    • If you are not logged in as an administrator, you may be prompted for an admin login
  • echo "text and/or variable(s)" prints (or “echoes”) the specified information, either directly into the terminal or (more commonly) through a pipe operator (|) into another command
    • echo "${input_variable}" | program programScript.ext passes the value of input_variable into the specified program to use when running the specified script
  • cat items to combine prints the specified items sequentially
    • cat items to combine > fileName.extension creates a file containing the combined contents of the specified items
      • Use with caution–if the specified file already exists, the contents will be overwritten
    • cat items to add >> fileName.extension adds the specified items to the end of the specified file
    • cat > fileName.extension <<EOF text to add EOF adds the specified text to the end of the specified file
      • Since this command adds anything between the two EOF indicators, it is typically used for adding multiple lines
  • chmod mode fileOrFolder updates permissions for the specified file or folder to the specified mode
    • The mode parameter is very flexible, but for simplicity, only the octal notation is described here
    • Octal permissions are specified by three numbers, representing owner, group, and general permissions, respectively
    • Octal Permission Numbers:
      • 0 - no access
      • 1 - execute only
      • 2 - write only
      • 3 - write and execute
      • 4 - read only
      • 5 - read and execute
      • 6 - read and write
      • 7 - read, write, and execute
    • Enter man chmod for more detailed mode information, or see this AskUbuntu thread for an explanation of how different modes are used
  • wget https://link.to.file downloads the contents of the specified link
    • wget -O fileName.exteions https://link.to.file saves the contents of the link to the provided file name (rather than the linked file’s original name)
    • wget --user="siteUsername" --ask-password https://link.to.file downloads the contents of the specified link from a password-protected source (e.g., GitHub) and prompts for your password when needed

5 Software

To run scripts in Argon, you will need to use either a container or a module. Modules and containers have similar functions: they include software packages and dependencies and allow users to run scripts with those softwares.

As an example of why this might be helpful, consider a script that runs in R. Two different users might have different operating systems (Windows, MacOS, or Linux), different versions of R, different packages installed, or different versions of the required packages installed. One user might not have R installed at all. With a set of modules or a container, the user does not have to figure out which software and software versions are necessary to run a certain script–all they need is the script and the module(s)/container. These tools are also helpful for reproducibility. Because the they is stable (i.e., the software and operating system do not update unless the creator modifies it), the results of a given script will be the same each time it is run with that module/container.

The key difference for practical use is that the Argon modules are created and maintained by ITS, while containers are created and maintained by users. Because modules are “built in”, they are easier to access. However, they have a specific set of software available to use, and they cannot be modified.

5.1 Modules

Argon opens with the default module stack (the 2021.1 stack) already loaded. However, there is a more recent stack (the 2022.2 stack) that has more current versions of the software in it (see the software list, archived at https://perma.cc/WJ4Q-GDUS). There are several commands for interacting with environment modules:

  • module avail lists all of the installed modules in the environment, regardless of whether you have them loaded
    • module avail searchString lists all of the installed modules that include the search string in the name
    • module spider searchString lists all of the available modules (even incompatible ones or ones that can’t be loaded) that include the search string in the name
  • module list lists all of the modules loaded in the current environment
  • module load moduleName loads the specified module (must include version number)
    • module load stack/2022.2 loads the current software stack
    • module load r/4.2.2_gcc-9.5.0 loads R
    • Note that R packages must be loaded in addition to base R
  • module unload moduleName “unloads” the specified module
    • This command is helpful when multiple versions of a given software are loaded into the environment–any extra versions should be unloaded
  • module save environmentName saves the current set of loaded modules under the provided name so you can load them later
  • module restore environmentName loads the set of modules saved under the provided name

5.1.1 R in Argon

If you need R packages that aren’t available in the software stack, or if you need to use the current R version, using a container is recommended. Alternatively, see below for instructions on installing different R versions without a container.

5.1.1.1 Compile Other Versions

If you want to compile a more recent version of R than is available in the software stacks, see here (archived at https://perma.cc/C6EX-EZL4). However, we now compile the most recent version of R (and packages) using a container.

5.1.1.2 Install Packages

https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514803/Adding+R+programs+to+a+personal+library (archived at https://perma.cc/3SRR-2JE7)

  1. In Argon, load the R environment module:

    module load r/4.2.2_gcc-9.5.0

  2. Load compiler modules (not strictly necessary, since not all packages require them, but many do):

    module load cmake/3.25.0_intel-2021.7.1
module load ncurses/6.3_gcc-9.5.0
module load openssl/1.1.1s_gcc-9.5.0

  3. Launch R:

    R

  4. Install the package(s)

    • From CRAN:

      install.packages(
  c("package1", "package2"),
  repos = "https://cloud.r-project.org",
  dependencies = TRUE,
  type = "source",
  Ncpus = 40
)

    • From local source:

      install.packages(
  "path/to/source/file",
  repos = NULL,
  dependencies = TRUE,
  type = "source",
  Ncpus = 40
)

    • If you have not installed other packages before, you will see the following message:

      Warning in install.packages("package_name", repos = "http://cran.r-project.org") :
'lib = "/opt/R/3.0.2/lib64/R/library"' is not writable

Would you like to use a personal library instead?  (yes/No/cancel)
      1. Enter y to use a personal library
      2. Enter y to create a folder for the personal library
      3. Installation will take several minutes, because all dependencies will be installed

  5. After installing packages, you should periodically update them using update.packages(ask = FALSE)

5.1.1.3 Manage Packages with renv

The renv package allows you to create separate environments for each project you are working on. These environments are analogous to containers in that they contain all of the necessary scripts and packages to recreate a given project. However, unlike containers, environments created using renv are specific to R. If you are using a container for your project, you probably do not need to use renv, which is why we have it in the Modules section.

renv Documentation: https://rstudio.github.io/renv/index.html

5.1.1.3.1 Creating an Environment
  1. Create a directory where you want you want to store the project.
  2. cd to the above directory
  3. Start R
  4. Load the renv package: library("renv")
  5. At the R prompt, initialize the renv project on the local repository of R packages with:
    • renv::init(project = "/Users/itpetersen/Documents/Projects/SelfRegulation_IRT/")
    • or renv::init() if you are in the intended working directory
  6. You must restart your R session in the given project directory after running init in order for the changes to take effect!
  7. From this point on you are working in a renv project. Installed packages will go into a library within this project. After initializing the renv project on the local repository of R packages, packages from the local repository can be installed with renv::install():
    • renv::install("package_name")
  8. To restart a renv project, simply start R from the directory created in step (1). The project will initialize automatically.

To update version of renv: renv::upgrade()

To install packages: renv::install("package_name")

To update packages: renv::update()

To save the current state of your library: renv::snapshot()

To restore the state of your library from the lock file: renv::restore()

To disable renv on a project: renv::deactivate()

5.1.1.3.2 To control which packages to install with renv using a DESCRIPTION file

If you want to control which packages are installed in a renv project, you can use a DESCRIPTION file to specify the packages that should be installed:

  1. Create a DESCRIPTION file in the project directory with the following format:

    Type: project
Description: My project.
Depends:
    packageName1,
    packageName2,
    packageName3

  2. Run renv::settings$snapshot.type("explicit") to suppress dependency discover and to enable “explicit” mode: https://rstudio.github.io/renv/reference/dependencies.html#explicit-dependencies

  3. Run renv::init(bare=TRUE) to initialize the project without attempting to discover and install R package dependencies

5.1.1.3.3 To install a new package for a project after renv has been initialized
  1. First, load the R environment module (see Install R packages section above)
  2. cd to the directory of the R project
  3. launch R
    • R
  4. then install the package, (the code below uses a generic package_name):
    • install.packages("package_name", repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)
    • install.packages(c("renv","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags","renv"), repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)
  5. renv::snapshot()

5.2 Containers

Argon does not include any containers by default, so users must create them. After creating your container, there are several commands you can use to interact with it:

  • apptainer build container.sif sourceInfo creates a container called ‘container.sif’ using the provided source
    • The sourceInfo argument can be a definition file (e.g., defFile.def), a link to a docker image (e.g., docker://mathworks/matlab-deps:latest), or one of the other options from the Apptainer documentation
    • apptainer build --sandbox container/ sourceInfo creates a “sandbox” container that you can edit (useful for troubleshooting builds)
      • This creates a directory rather than a .sif file
    • apptainer build container.sif sandboxContainer converts the specified sandboxContainer directory into a .sif file
  • apptainer exec container.sif commandName commandArguments will run the specified command (with arguments, if included) using the specified container file
  • apptainer run container.sif runs the default command(s) in the specified container
    • apptainer run --app appName container.sif runs the default command(s) for the specified app in the specified container
  • apptainer shell container.sif opens a shell within the specified container, allowing you to run commands inside the container like you would within the terminal
    • Since container files are not editable, you will not be able to install or update any software
  • apptainer help commandName displays information about the provided Apptainer sub-command (syntax, options, etc.)

6 Building Containers

Link to ITS documentation: https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513442/Singularity+Containers

Link to Apptainer documentation: https://apptainer.org/documentation/

6.1 Overview

Because the default Argon workspace only has the modules provided by ITS (the “stack”), many users prefer to use a container. A container essentially operates like its own computer, with its own operating system and any files or programs the creator wants to include. However, unlike an actual computer, a container is stored as a file that can be shared (and used) across several different computers.

The University recommends using Apptainer, formerly known as Singularity, to create and manage containers.

To create a container, you must have admin privileges on the computer you are using, which means you cannot create a container in Argon. However, you can create a container on your own computer and use it in Argon. Because none of the lab computers run on Linux, you will need to install a Linux subsystem, which allows users to run Linux from another operating system (such as Windows or MacOS).

6.2 Creating a Container

6.2.1 Installing Apptainer

  1. Install a Linux subsystem with Ubuntu

    • Note: If a Linux subsystem is already installed, open the subsystem as an administrator and proceed to the next step

    Windows

    1. Open PowerShell as an administrator
    2. Enter bash wsl --install
    3. When the installation is complete, restart your computer
    4. Your computer will install Ubuntu (a flavor/type of Linux) during the restart
    5. Open Ubuntu as an administrator

    MacOS (Not Recommended)

    1. Download the latest Ubuntu server

    2. Download UTM

      1. Use the ‘Download from GitHub’ option–the App Store version is identical, but costs $10

    3. Open UTM and click ‘+’ to create a new virtual machine (VM)

    4. Select ‘Virtualize’

    5. Select ‘Linux’

    6. Select the amount of RAM and CPU cores to allocate to the VM (or use defaults)

      • To see how much RAM/CPU cores your computer has:
        1. Click the Apple icon in the top left corner of the screen
        2. Select ‘About this Mac’
        3. The RAM is listed next to ‘Memory’
        4. Click ‘More Info…’
        5. Scroll all the way to the bottom, then click ‘System Report…’
        6. The number of CPU cores is listed next to ‘Total Number of Cores’
      • In general, you should allocate no more than 75% of your total CPU cores and no more than 50% of your total RAM

    7. When prompted for the Boot Image, make sure ‘Boot from ISO image’ is selected

    8. Click ‘Browse’ and select the Ubuntu installation file

    9. Specify the maximum amount of storage space to allocate to the subsystem

      • To see how much storage space your computer has:
        1. Open System Settings
        2. Click ‘Storage’
      • In general, you should overestimate the amount of storage, but you cannot exceed the available storage space on your computer

    10. Do not add a Shared Directory

    11. Click ‘Save’

    12. Click the Play icon to enter the subsystem

    13. Select the ‘Try or Install Ubuntu’ option

    14. Complete the installation prompts, using the up and down arrow keys to move between options and the enter key to select the highlighted option

      Installation Prompts

      • Language: English (or whichever you prefer)
      • Keyboard Layout: English (US)
      • Keyboard Variant: English (US)
      • Installation Type: Ubuntu Server
      • Network Configuration: name varies, but select the option with the ‘eth’ type
      • Proxy Configuration: (leave blank)
      • Mirror Configuration: (leave blank; wait for ‘This mirror location passed tests.’ to appear)
      • Guided Storage Configuration: (use default options)
      • Storage Configuration: (review summary and continue)
      • Confirm Destructive Action: Continue
        • Note: This pop-up is alarming, but (assuming you set an appropriate storage limit) nothing will be destroyed
      • Profile Configuration:
        • Your name: hawkid
        • Your server’s name: hawkid_linux_sub
        • Pick a username: hawkid
        • Choose a password/Confirm your password: (HawkID password recommended)
      • Upgrade to Ubuntu Pro: Skip for now
      • SSH Configuration: (do not enable)
      • Featured server snaps: (select none)

    15. When the installation is complete, select ‘Reboot Now’

    16. After rebooting (on the screen with the ‘Please remove the installation medium…’ prompt), manually quit the VM by clicking the Power icon in the top left corner

    17. Click the Play icon to restart the VM

    18. Select the ‘Boot from next volume’ option

    19. Log in with the username and password created during installation

  2. Update Ubuntu

    1. Enter sudo apt update to check for updates
    2. Enter sudo apt upgrade -y to install the updates identified in the previous step

  3. Install Apptainer

    1. Run the following code to install dependencies:

      sudo apt update
sudo apt install -y \
software-properties-common \
squashfuse \
fuse2fs \
gocryptfs

    2. Run the following code to download and install Apptainer:

      sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y apptainer

      • If Apptainer is not available via apt (E: Unable to locate package apptainer), install it from source.

        Source Installation

        1. Get the most current version number from GitHub (do not need to download–just need version number)

        2. Define a variable to hold version info (insert version number in place of #s):

          export VERSION=#.#.#

        3. Download the source code:

          wget https://github.com/apptainer/apptainer/releases/download/v${VERSION}/apptainer-${VERSION}.tar.gz
tar -xzf apptainer-${VERSION}.tar.gz
cd apptainer-${VERSION}

        4. Build Apptainer: bash ./mconfig make -C builddir sudo make -C builddir install

    3. Verify installation:

      apptainer --version

  4. Go to your home directory (cd ~)

  5. Create a containers folder if one does not exist:

    mkdir containers

6.2.2 Definition Files

Definition files contain the necessary information for installing the container software (e.g., versions, packages). Because the user (i.e., you) specifies the content of the definition file, there is a lot of flexibility when it comes to what to include and how to structure them. However, there are a handful of necessary components for a definition file, which you can read about here.

There are a number of pre-written definition files for R (with or without brms and CmdStan), MPlus, MATLAB, and Python/MNE, which you can view or download here. There are also instructions below for creating a definition file from scratch. To build a container with a program that is not included here, see the Apptainer documentation for building a container and for including multiple apps in one container.

  1. Go to the containers folder (cd containers)

  2. If installing MPlus:

    1. Copy the MPlus installer into your containers folder:

      cd path/to/installer/MplusLinux64.bin ~/containers/

    2. Run the installer to generate a properties file:

      ./MplusLinux64.bin -r ~/containers/mplus_install.properties

    3. Open the properties file:

      nano mplus_install.properties

    4. Under the “Choose Install Folder” and “Install” headings, update the installation path to /opt/mplus

      Example Properties File

       # Tue Feb 24 21:04:04 CST 2026
 # Replay feature output
 # ---------------------
 # This file was built by the Replay feature of InstallAnywhere.
 # It contains variables that were set by Panels, Consoles or Custom Code.
 #Choose Install Set
 #------------------
 CHOSEN_FEATURE_LIST=Application,Examples
 CHOSEN_INSTALL_FEATURE_LIST=Application,Examples
 CHOSEN_INSTALL_SET=Typical
 #Choose Install Folder
 #---------------------
 USER_INSTALL_DIR=/opt/mplus
 #Install
 #-------
 -fileOverwrite_/opt/mplus/uninstall/uninstall_mplus.lax=Yes
 -fileOverwrite_/opt/mplus/mplus=Yes
 -fileOverwrite_/opt/mplus/Documentation/Getting_Started.pdf=Yes
 -fileOverwrite_/opt/mplus/Documentation/Mplus_Users_Guide_v8.pdf=Yes
 -fileOverwrite_/opt/mplus/Documentation/Version_8.1_Language_Addendum.pdf=Yes
 -fileOverwrite_/opt/mplus/Documentation/Version_8.5_Language_Addendum.pdf=Yes

    5. Press Ctrl+O then Enter to save (“write out”) the file

    6. Press Ctrl+X to close the file

  3. If installing MATLAB:

    1. Copy the dependencies file into your containers folder:

      wget -q -O https://raw.githubusercontent.com/mathworks-ref-arch/container-images/refs/heads/main/matlab-deps/r2026a/ubuntu24.04/base-dependencies.txt

  4. Create or download the definition file

    1. To download an existing definition file (see GitLab for options):

      1. Enter the following code into the terminal:

        wget -q -O argon_container.def https://link.to.definition.file

      2. Log in with your HawkID credentials when prompted

    2. To create a definition file from scratch:

      1. Create a new file:

        nano argon_container.def

      2. Add the definition file header:

        Bootstrap: docker
From: ubuntu:24.04

        Other common base images:

        • anaconda/miniconda:latest for a container with miniconda (helpful for Python environments)
        • mathworks/matlab for MATLAB only
        • rocker/r-ver:latest for R, CmdStan, and/or MPlus
          • Note: The definition file below uses r2u, which automatically installs Linux dependencies for R packages. The base R image at rocker/r-ver does not include r2u, so you will have to add it in the definition file or manually install the Linux dependencies.

      3. Add remaining parameters

        The script below includes parameters for R (with brms), CmdStan, MPlus, MATLAB, and Python/MNE. Paste the entire script into the definition file (right click to paste into the terminal). To install only a subset of the software, simply remove the section(s) associated with the unwanted software. Note that MPlus has a subsection in the R section in addition to its standalone section. Also note CmdStan does not have a standalone app section–it is installed as a subsection of the R installation.

        All Software

         %environment

     # MPlus paths
     export PATH=/opt/mplus:$PATH

     # R paths
     export R_LIBS_USER=/usr/local/lib/R/site-library

     # CmdStan paths
     export CMDSTAN=/opt/cmdstan
     export STAN_THREADS=true

 %files

     # MPlus files
     MplusLinux64.bin /opt/MplusLinux64.bin
     mplus_install.properties /opt/mplus_install.properties

     # MATLAB files
     base-dependencies.txt /opt/base-dependencies.txt

 %post

     # install dependencies for all apps (do not delete)
     # includes any dependencies required for 2+ software packages
     apt-get update && apt-get install -y --no-install-recommends \
       bzip2 \
       ca-certificates \
       cmake \
       curl \
       g++ \
       git \
       make \
       nano \
       python3 \
       vim \
       wget

     # clean up package manager
     apt-get clean -y
     apt-get autoremove -y
     rm -rf /var/lib/apt/lists/*

 #############################################
 # Python/MNE
 #############################################

 %apprun python

     exec python "$@"

 %appinstall python

     # install pip
     apt-get update && apt-get install -y --no-install-recommends \
       python3-pip

     # install mne, overriding env warning
     pip install --break-system-packages "mne[full]"

     # install hdf5 (large file handling) support packages, overriding env warning
     pip install --break-system-packages h5io pymatreader

 #############################################
 # R
 #############################################

 %apprun R

     exec R "$@"

 %appinstall R

     # install additional r2u dependencies
     # if installing R without r2u (from rocker base), add Linux deps here
     # gnupg required for r2u, all others only needed for eegUtils package
     apt-get update && apt-get install -y --no-install-recommends \
       gnupg \
       liblapack-dev \
       libblas-dev \
       gfortran

     # add r2u keys to apt folder so they are recognized as safe
     gpg --homedir /tmp --no-default-keyring \
       --keyring /usr/share/keyrings/r2u.gpg \
       --keyserver keyserver.ubuntu.com \
       --recv-keys A1489FE2AB99A21A 67C2D66C4B1D4339 51716619E084DAB9

     # add r2u mirror to apt source list to pull files
     # do not change indentation - I know it's ugly but it will throw an error
     cat > /etc/apt/sources.list.d/r2u.sources <<EOF
 Types: deb
 URIs: https://r2u.stat.illinois.edu/ubuntu
 Suites: noble
 Components: main
 Arch: amd64, arm64
 Signed-By: /usr/share/keyrings/r2u.gpg
 EOF

     # add base r to apt source list just to be safe
     cat > /etc/apt/sources.list.d/cran.sources <<EOF
 Types: deb
 URIs: https://cloud.r-project.org/bin/linux/ubuntu
 Suites: noble-cran40/
 Components:
 Arch: amd64, arm64
 Signed-By: /usr/share/keyrings/r2u.gpg
 EOF

     # install base r
     apt-get update && apt-get install -y --no-install-recommends r-base-core

     # pin r2u to top of package priority list so it gets picked for installs
     # don't change indentation here either
     cat > /etc/apt/preferences.d/99cranapt <<EOF
 Package: *
 Pin: release o=CRAN-Apt Project
 Pin: release l=CRAN-Apt Packages
 Pin-Priority: 700
 EOF

     # install python dependencies for bspm
     apt-get update && apt-get install -y --no-install-recommends \
       python3-dbus \
       python3-gi \
       python3-apt

     # install bspm
     sudo Rscript --vanilla -e "install.packages(
       'bspm',
       repos = 'https://cloud.r-project.org'
     )"

     # update bspm settings
     # don't change indentation
     mkdir -p /etc/R
     cat >> /etc/R/Rprofile.site <<EOF
 suppressMessages(bspm::enable())
 options(bspm.version.check = FALSE)
 EOF

     # install packages (add/remove as needed)
     R -e "install.packages(
       c(
         'brms',
         'data.table',
         'devtools',
         'flowchart',
         'fs',
         'ggh4x',
         'gt',
         'gtsummary',
         'janitor',
         'lavaan',
         'lme4',
         'parallely',
         'performance',
         'psych',
         'R.matlab',
         'remotes',
         'renv',
         'semTools',
         'tidyverse'
       ),
       repos = 'https://cloud.r-project.org'
     )"

     # install packages via remotes
     R -e "remotes::install_github('DevPsyLab/petersenlab')"
     R -e "remotes::install_github('craddm/eegUtils')"

     # clean up package manager
     apt-get clean -y
     apt-get autoremove -y
     rm -rf /var/lib/apt/lists/*

     #######
     # Mplus
     #######

     # install R package
     R -e "install.packages(
       'MplusAutomation',
       repos = 'https://cloud.r-project.org'
     )"

     #############
     # CmdStan
     #############

     # install R package
     R -e "install.packages(
       'cmdstanr',
       repos = c(
         'https://stan-dev.r-universe.dev',
         'https://cloud.r-project.org'
       )
     )"

     # create folder to install into
     mkdir -p /opt/cmdstan

     # install CmdStan using R
     R -e "cmdstanr::install_cmdstan(
       cores = 2,
       dir = '/opt/cmdstan'
     )"

     # create an R environment file
     mkdir -p /root/.R

     # add threading vars to environment and startup
     echo 'STAN_THREADS=true' >> /root/.Renviron
     echo 'STAN_THREADS=true' >> /root/.R/Makevars

     # clean up package manager
     apt-get clean -y
     apt-get autoremove -y
     rm -rf /var/lib/apt/lists/*

 #############################################
 # MPlus
 #############################################

 %apprun mplus

     exec mplus "$@"

 %appinstall mplus

     chmod +x /opt/MplusLinux64.bin
     /opt/MplusLinux64.bin -i silent -f /opt/mplus_install.properties
     chmod -R 755 /opt/mplus

 #############################################
 # MATLAB
 #############################################

 %apprun matlab

     exec matlab -nodesktop -nojvm -nosplash "$@"

 %appinstall matlab

     # define var for noninteractive install
     DEBIAN_FRONTEND=noninteractive \

     # install additional dependencies
     apt-get update && apt-get install -y --no-install-recommends \
       `cat /opt/base-dependencies.txt`

     # clean up package manager
     apt-get clean -y
     apt-get autoremove -y
     rm -rf /var/lib/apt/lists/*

     # define variables needed for setup
     export MATLAB_RELEASE=R2026a
     export MATLAB_PRODUCT_LIST="MATLAB Wavelet_Toolbox Signal_Processing_Toolbox Statistics_and_Machine_Learning_Toolbox Optimization_Toolbox Parallel_Computing_Toolbox"
     export MATLAB_INSTALL_LOCATION="/opt/matlab/${MATLAB_RELEASE}

     # add those variables to apptainer environment
     # do not change the line indentation - it will cause an error
     cat << EOF > $APPTAINER_ENVIRONMENT
 export MATLAB_RELEASE=$MATLAB_RELEASE
 export MATLAB_PRODUCT_LIST=$MATLAB_PRODUCT_LIST
 export MATLAB_INSTALL_LOCATION=$MATLAB_INSTALL_LOCATION
 EOF

     # make sure apptainer has full permissions in tmp
     chmod 777 /tmp

     # get matlab package manager (mpm) and update permissions
     wget -q https://www.mathworks.com/mpm/glnxa64/mpm
     chmod +x mpm

     # run mpm to install matlab
     ./mpm install \
       --release=${MATLAB_RELEASE} \
       --destination=${MATLAB_INSTALL_LOCATION} \
       --products ${MATLAB_PRODUCT_LIST}

     # clean up after installer
     rm -f mpm /tmp/mathworks_root.log
     ln -s ${MATLAB_INSTALL_LOCATION}/bin/matlab /usr/local/bin/matlab

     # add license info
     mkdir -p ${MATLAB_INSTALL_LOCATION}/licenses
     cat << EOF > ${MATLAB_INSTALL_LOCATION}/licenses/network.lic
 SERVER matlab-lm.iowa.uiowa.edu 27000
 USE_SERVER
 EOF

      4. Press Ctrl+O then Enter to save the file

      5. Press ‘Ctrl+X’ to close the file

  5. Build the container:

    sudo apptainer build argon_container.sif argon_container.def

    You may be prompted to enter your (admin login) password. It will take a few minutes to build, and a lot of text will appear in the terminal.

  6. Test the container

    If you used a definition file without an app structure (i.e., no %appinstall sections in the definition file), use apptainer exec argon_container.sif path/to/software/exec_file instead of apptainer run --app app_name argon_container.sif to start the program.

    1. Python/MNE

      1. Start the Python app within the container:

        apptainer run --app python argon_container.sif

        If the app is working, you should see the Python start-up message, followed by >>> in the final line.

      2. Verify the MNE installation:

        import mne; mne.sys_info()

        You should see a printout of the system information, including a list of installed libraries.

      3. Enter exit() to leave the container.

    2. R

      1. Start the R app within the container:

        apptainer run --app R argon_container.sif

        If the app is working, you should see the R start-up message, followed by > in the final line. Containers with r2u may show a warning about D-Bus services–this will not affect functionality. Containers run without sudo or admin access may show a warning about the brms package needing admin permissions–this will not affect functionality.

      2. Check R version:

        version

        You should see a printout of the version information.

      3. Check package installations:

        Ensure that all desired packages have been installed.

      4. Enter q() to leave the container.

    3. MPlus

      1. Start the MPlus app within the container:

        apptainer run --app mplus argon_container.sif
    4. MATLAB

      1. Start the MATLAB app within the container:

        apptainer run --app matlab argon_container.sif

        If the app is working, you should see the MATLAB start-up message, followed by > in the final line. Because MATLAB verifies its license via network connection, you must be connected to UIowa Wi-Fi (or be using the VPN) for it to work.

      2. Check MATLAB version:

        ver

        You should see a printout of the version information, including installed products.

      3. Enter exit to leave the container.

  7. Copy the container to your computer:

    cp ~/containers/r_argon.sif "/mnt/c/Users/hawkid/path/to/folder/"

    The /mnt/c path is equivalent to your computer’s C: drive, so the rest of the file path should be defined from there.

  8. If your admin login is different from your HawkID, restart the terminal (or emulator) to run it as a regular user.

  9. Copy the container to Argon, either by using the terminal or by manually moving it in File Explorer/Finder.

  10. Test the container on Argon, using the same steps described above.

  11. If you installed CmdStan, copy the executable files to a writable location:

    mkdir -p $HOME/cmdstan_cache
apptainer exec /Users/hawkid/path/to/container/argon_container.sif \
  cp -r /opt/cmdstan/. $HOME/cmdstan_cache

  12. To submit a job using the container, use the apptainer exec command:

    # run an R script using a container built with %app sections
/usr/bin/apptainer exec --app R /Users/hawkid/container_path/argon_container.sif my_script.R

# run a MATLAB script using a container built without %app sections
/usr/bin/apptainer exec /Users/hawkid/container_path/argon_container.sif \
  matlab -nodesktop -nojvm -nosplash my_script.mat

    For an example job script using a container, see Section 7.3.

7 Jobs

In order to “run” scripts from a log-in node, you will need to create a job script. A job script contains all of the commands that you would ordinarily run individually in the terminal to run a script, along with some parameters that tell Argon where and how to run the commands. At minimum, a job script should contain a shell specification and at least one program, script, or command to execute. However, there are several additional parameters you can include to control how the script is run.

Basic Job Submission: https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513450/Basic+Job+Submission

Advanced Job Submission: https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513452/Advanced+Job+Submission

7.1 Job Commands

Below are a variety of common commands for submitting and tracking jobs.

  • qsub jobScript.job submits a job script to run
    • qsub -hold_jid JOB_ID jobScript.job submits jobScript.job to run after job JOB_ID is finished
  • qstat shows the status of available queues and jobs
    • qstat -u hawkid shows jobs submitted by the specified HawkID
    • qstat -g c -q QUEUE_NAME shows the status of the specified queue
    • qstat -j JOB_ID shows detailed satus info for the specified job (if still running)
    • Status Abbreviations:
      • d - deleted
      • E - error
      • h - held by user
      • r, t - running or transferring to run status
      • R - restarted
      • s S - job (s) or queue (S) suspended
      • T - queue threshold exceeded (suspended)
      • w - waiting in queue
  • qacct summarizes job and user info that meet the specified parameters
    • qacct -j JOB_ID shows info for the specified job (after running)
    • qacct -o hawkid -j -d n shows info on the specified user’s jobs for the past n days
  • qdel JOB_ID deletes the specified job
    • qdel -u hawkid deletes all jobs submitted by the specified user

7.2 Writing Job Scripts

  1. Log in to Argon

  2. Create the job script:

    nano jobScript.job

  3. Specify the shell:

    #!/bin/sh

    This line indicates which shell (CLI language) to use when running the rest of the job script. In this case, sh sets the shell to the system’s default shell. Regardless of which shell you specify, this line must start with #!.

  4. Add qsub options (optional)

    The next few lines contain settings for the qsub command, which is the command used for submitting job scripts. You can also specify these options in the command line, but that can lead to lengthy inputs. When specifying qsub options in your job script, each one must have #$ at the start of the line. Below are options commonly used in the DevPsy job scripts.

    Option Description
    -cwd execute the job from the current working directory
    -e [path] file name or path
    -j [y|n] whether the error messages should be saved in the same file as outputs (y) or not (n)
    -l mf=[nG|nM] number (n) of gigabytes (G) or megabytes (M) of free memory needed at startup
    -l h_vmem=[nG|nM] number (n) of gigabytes (G) or megabytes (M) of virtual memory needed across entire job
    -M [email] email to send job notifications to
    -m [b|e|a|s|n] when to send job notifications (b = beginning of job; e = end of job; a = job aborted; s = job suspended; n = no notifications); multiple values allowed (e.g., -m eas)
    -N [name] job name
    -o [path] file or folder for saving non-error output messages
    -pe smp [cores] number of cores to use for parallel computing*
    -q [UI|UI-HM|UI-MPI|UI-GPU|UI-DEVELOP|all.q] which job queue to use**
    -t n-m[:s] indicates an array job; includes the index of the first (n) and last (m) task with an optional parameter for step size (s; defaults to 1)
    -tc [n] number of jobs in an array job to run simultaneously
    -wd [path] execute the job from the specified folder

    *We typically use the smp parallel computing environment, but other options are listed on the Advanced Job Submission documentation page and can be substituted for smp.

    **The UI (default) and UI-HM (high-memory) queues are the most used by DevPsy. UI-MPI is for jobs that can use multiple nodes simultaneously, and UI-GPU is for jobs that can use a GPU accelerator. UI-DEVELOP is recommend for test runs and debugging. The all.q queue is the lowest priority queue, so jobs running on it will “give up” resources if needed for jobs in higher-priority queues.

  5. Specify the working directory (optional)

    Argon defaults to running the job script from your home directory (/Users/hawkid). However, if you would like to run the job from a different directory, you can either use the -wd option described in the previous step or manually specify the working directory in the body of the job script:

    # set working directory
cd /Users/hawkid/path/to/working/directory/

  6. Write the job script

    The remainder of the job script is extremely flexible, because it is whatever set of commands you would use to run your script “live” in the terminal. Typically, job scripts start by defining any variables needed for the rest of the script and loading necessary modules. After that, the commands specify which scripts/programs to run and when. See Section 7.3 for examples of a variety of job scripts, and see Section 4.4 for common bash commands used in job scripts.

  7. Press Ctrl+O then Enter to save the file

  8. Press ‘Ctrl+X’ to close the file

7.3 Example Job Scripts

7.3.1 Using A Container

#!/bin/sh

# Set working directory
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/

# Specify qsub options
#$ -pe smp 4
#$ -M isaac-t-petersen@uiowa.edu
#$ -m eas
#$ -l mf=512G
#$ -l h_vmem=512G
#$ -cwd
#$ -q UI-HM
#$ -e /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/
#$ -o /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/

# Load the environment modules
module purge

# Run the R script inside container
apptainer exec \
    --bind $HOME/cmdstan_cache:/opt/cmdstan \
    --cleanenv \
    /Users/itpetersen/Documents/Containers/r_argon.sif \
    Rscript ./Analyses/factorScores.R

7.3.2 Using Software Stacks

#!/bin/sh

# Set working directory
cd /Users/itpetersen/Documents/Projects/Bayesian_IRT/

# Specify qsub options
#$ -pe smp 4
#$ -M isaac-t-petersen@uiowa.edu
#$ -m eas
#$ -l mf=512G
#$ -l h_vmem=512G
#$ -cwd
#$ -q UI-HM
#$ -e /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/
#$ -o /Users/itpetersen/Documents/Projects/Bayesian_IRT/Output/

# Load the environment modules
module purge
module load stack/2022.2
module load r/4.2.2_gcc-9.5.0

# Run the R script
Rscript ./Analyses/factorScores.R

7.3.3 Array Job with Variables

#!/bin/bash

# set job parameters
#$ -N tf_visualization
#$ -o /Users/mschumer/logs/$JOB_NAME_output.txt
#$ -e /Users/mschumer/logs/$JOB_NAME_errors.txt
#$ -q UI
#$ -l mf=36G
#$ -l h_vmem=36G
#$ -t 1-535
#$ -tc 16
#$ -pe smp 16

# activate mne environment
conda activate /Users/mschumer/anaconda3/envs/mne

# get list of subjects
cd /Shared/lss_PetersenLabEEGVideo/EEG/Oddball/2_TFData/1_preprocessed
subjects=($(ls))
cd /Users/mschumer

# select subject using current idx
sub=${subjects[${SGE_TASK_ID}-1]}

# set values for input and output paths
input_path=/Shared/lss_PetersenLabEEGVideo/EEG/Oddball/2_TFData/1_preprocessed
output_path=/Shared/lss_PetersenLabEEGVideo/EEG/Oddball/2_TFData/3_tf_processed

# run script
echo "${input_path} ${output_path} ${sub}" | python /Users/mschumer/tf_analysis.py

8 Stan

https://github.com/stan-dev/rstan/wiki/Installing-RStan-on-Linux (archived at https://perma.cc/89H9-L8S6)

Makevars:

\\data.hpc.uiowa.edu\argon_home\.R\Makevars

9 Troubleshooting

9.1 Running out of Memory

May need to request more memory, either by requesting more slots (to ensure more memory; i.e., -pe smp <SLOTS>), by increasing how much memory is available for the node at the moment your job starts (i.e., mf=<SIZE>), and/or by increasing the hard limit for virtual memory (i.e., h_vmem=<SIZE>); see Section 7.2.

To see how much memory was used by a killed job, run qacct -j JOBID and look at: maxvmem.

This error can generate the following error messages:

  • job JOB_ID died through signal TERM (15)
  • job [NUMBER] died through signal KILL (9)
  • Exit Status: 143
  • Exit Status: 137

9.2 Having trouble installing R packages?

9.2.1 Containers

Unless you built a sandbox container, you will not be able to install packages, because container files are not editable. To add new packages, you need to create a new container using an updated definition file with all desired packages. Alternatively, you can create a sandbox container based on your current container, add the desired packages, and convert the sandbox container to a static build:

apptainer build --sandbox newContainer/ oldContainer.sif
apptainer exec --writable newContainer R -e "install.packages(c('package1', 'package2'), repos = 'https://cloud.r-project.org')"
apptainer build newContainer.sif newContainer/

For transparency, it is preferable to create a new container from an updated definition file, which ensures all container build info is available.

9.2.2 Modules

Sometimes Argon will fail to install packages in an R workspace. We advise the following steps:

  1. Sometimes R provides an error stating something like “fatal error: modulename: No such file or directory”. In this instance, you may wish to exist R with q() and load all the modules that begin with the listed module name, as listed above in “Installing Linux Packages to Install R Packages”. Then reopen R and try to install the package again.

  2. You can also try installing the package from binary using the RStudio Package Manager (RSPM):

    remotes::install_github("cran4linux/rspm")
rspm::enable()
install.packages("PACKAGE_NAME")

  3. If that does not work, you can try downloading the .tar file directly from the CRAN repository. Then copy the file into your Argon folder and type ‘install.packages(“/Users/path/to/directory/package_name”, repos = NULL, type = “source”)’. Note that you may need to download an older version of the package (e.g., from the CRAN Archive), such that it is compatible with the version of R you are running on Argon (which it typically not the most recent R version):

    install.packages("http://cran.r-project.org/src/contrib/Archive/MASS/MASS_7.3-60.0.1.tar.gz", repos = NULL, type = "source")

  4. Check if you are installing the necessary modules within your job script:

    module load stack/2022.2
module load r/4.2.2_gcc-9.5.0

  5. Add a line to your R script to make sure it is accessing the same repository where you installed the packages manually:

    .libPaths(c("/old_Users/HAWKID/Rlibs", 
        "/cvmfs/argon.hpc.uiowa.edu/2022.2/apps/linux-centos7-broadwell/gcc-9.5.0/r-4.2.2-lv7yirk/rlib/R/library"))

  6. If the prior steps fail, email research-computing@uiowa.edu with the error, asking for help to figure out how to install the packages.

9.3 Error When Installing Packages from Source

You may have to set environment variables in each module file to help the compiler find headers and libraries. Note that if you run into any C++ code, you will need to set the CPLUS_INCLUDE_PATH variable.

9.3.1 nloptr

  1. Run the following commands
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
module load nlopt
  1. Set LIBRARY_PATH so linker can find library while launching R session (single line below):
LIBRARY_PATH=$ROOT_NLOPT/lib64:$LIBRARY_PATH R
  1. In R console, install nloptr (two lines below)
Code
install.packages(verbose = 1, "nloptr")

9.3.2 tkrplot

  1. Run the following commands
module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
  1. Set LIBRARY_PATH so linker can find library while launching R session (single line below):
C_INCLUDE_PATH=$ROOT_XPROTO/include LIBRARY_PATH=$ROOT_LIBXEXT/lib:$ROOT_LIBXSCRNSAVER/lib R
  1. In R console, install tkrplot (two lines below)
Code
install.packages(verbose = 1, "tkrplot")

9.4 Error Reading from Connection

Error in unserialize(node$con) : error reading from connection
Calls: parlmice ... FUN -> recvData -> recvData.SOCK0node -> unserialize

There likely wasn’t sufficient memory for a given core. Try increasing the max memory available and decreasing the number of cores and/or slots, so there is more memory available per core:

https://stackoverflow.com/questions/46186375/r-parallel-error-in-unserializenodecon-in-hpc (archived at https://perma.cc/MF6V-NAVS)

https://stackoverflow.com/questions/17015598/error-calling-serialize-r-function (archived at https://perma.cc/3Q75-DA2D)

https://gforge.se/2015/02/how-to-go-parallel-in-r-basics-tips/#Memory_load (archived at https://perma.cc/2JRF-8Y5F)

9.5 Error: $'\r': command not found 

/opt/sge/default/spool/argon-itf-bx47-36/job_scripts/5685660: line 2: $'\r': command not found

That often means your job script has Windows (CRLF) line endings instead of Unix (LF). You can fix the line endings with the following command:

dos2unix your_job_file.job

10 Appendix

10.1 Old Stacks

10.1.1 Stack 2022.1

module load stack/2022.1
module load r/4.1.3_gcc-9.4.0
cd path

10.1.2 Stack 2021.1

module load stack/2021.1
module load r/4.0.5_gcc-9.3.0
cd path

10.1.3 Stack 2020.2

module load stack/2020.2
module load r/4.0.2_gcc-8.4.0
cd path

10.1.4 Stack 2020.1

module load stack/2020.1
module load r/3.6.2_gcc-9.2.0
cd path

10.2 Managing packages for a project using packrat

Please note: we now use renv rather than packrat for package management

https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76514803/Adding+R+programs+to+a+personal+library (archived at https://perma.cc/3SRR-2JE7)

  1. Create a directory where you want you want to store the project.
  2. cd to the above directory
  3. Start R
  4. Load the packrat package: library("packrat")
  5. At the R prompt, initialize the packrat project on the local repository of R packages with: packrat::init(project = "/Users/itpetersen/Documents/Projects/INSERT_PROJECT_NAME/", options = list(local.repos = "/Users/itpetersen/R/x86_64-pc-linux-gnu-library/4.0"))
  6. You must restart your R session in the given project directory after running init in order for the changes to take effect!
  7. From this point on you are working in a packrat project. Installed packages will go into a library within this project. After initializing the packrat project on the local repository of R packages, packages from the local repository can be installed with packrat::install_local():
    • packrat::install_local("package_name")
  8. To restart a packrat project simply start R from the directory created in step (1). The project will initialize automatically.

To save the current state of your library: packrat::snapshot(); if that command fails due to an error when fetching sources, try packrat::snapshot(snapshot.sources = FALSE)

To disable packrat on a project: disable(restart = FALSE)

10.2.1 To install a new package for a project after packrat has been initialized

  1. First, load the R environment module (see Install R packages section above)
  2. cd to the directory of the R project
  3. launch R
    • R
  4. then install the package, (the code below uses a generic package_name):
    • install.packages("package_name", repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)
    • install.packages(c("packrat","psych","tidyverse","data.table","nlme","lme4","mirt","TeachingDemos","Amelia","mice","miceadds","abind","future","lavaan","blavaan","Rcpp","igraph","shinystan","StanHeaders","brms","rstan","rjags","renv"), repos = "http://cran.r-project.org", type = "source", dependencies = TRUE, Ncpus = 40)
  5. packrat::snapshot(); if that command fails due to an error when fetching sources, try packrat::snapshot(snapshot.sources = FALSE)

10.2.2 Build packrat environment on a compute node (only if necessary if package load fails due to issues building packages)

  1. qlogin – for more info, see here: https://uiowa.atlassian.net/wiki/spaces/hpcdocs/pages/76513454/Qlogin+for+Interactive+Sessions (archived at https://perma.cc/Y3A9-WQ3W)
  2. module load stack/2020.2-base_arch – this will ensure that the modules point to the lowest common multiarchitecture on Argon and will run on all nodes
  3. module load r
  4. cd to the directory of the R project
  5. R
  6. Create packrat project (see above)
  7. Install packages
  8. Take a package snapshot using packrat
  9. exit – to exit the qlogin session

Note: Your packrat environment will then be linked to the proper glpk library and will run on any Argon node. You do not need to use the 2020.2-base_arch module at run time, only build time.

Developmental Psychopathology Lab