acs_docs
  • Linux Fundamentals
    • Introduction to Linux
      • Navigating the Linux Shell
        • Check Your Current Shell with echo
        • List All Environment Variables with env
        • Check Your Current Directory with pwd
        • List Directory Contents with ls
        • Move Between Directories with cd
        • Visualize Directory Structure with tree
        • Locate Commands with which
      • Interacting with Files
        • Creating Directories with mkdir
        • Removing Directories with rmdir
        • Removing Files and Directories with rm
        • Viewing File Contents with cat
        • Creating Files with cat and Redirection
        • Viewing Files with head and tail
        • Renaming and Moving with mv
        • Copying with cp
        • Editing Files with nano, emacs, vi
        • Viewing File Contents by Page with more and less
      • File Permissions in Linux
        • Understanding File Permission Categories
          • File and Directory Permission Examples
          • Decimal representation
        • Changing File Permissions in Linux
          • Examples with chmod
        • Changing Group Ownership in Linux
          • Examples with chgrp
      • Access Control Lists – ACL
        • Getting ACL information
          • Get ACL information with getfacl
        • Setting ACL information
          • Set ACL with setfacl -m (modify)
        • Removing ACL information
          • Remove ACL permissions with setfacl -x
          • Remove all ACL rules with setfacl -b
    • Linux FAQs
  • Services
    • Connecting to IDSC Systems from offsite
      • How to access Pegasus from home
      • What if I’m at home with no VPN / no UMIT laptop?
      • Connecting to Advanced Computing Systems from UM networks
    • Windows
    • Mac and Linux
    • Using VS Code
    • Forwarding the display with x11
    • Connecting to IDSC Systems from offsite
    • Storage
      • GPFS storage
      • CES storage
    • File Transfers
      • Use the Pegasus Data Transfer Node for large workflows
      • Using command line utilities
        • scp
        • rsync
        • rclone
      • Using FileZilla
      • Using VS Code
    • IDSC Onboarding Training Videos
    • IDSC Adanced Computing Services
  • User Guide
    • Quickstart
      • Before You Begin
      • 1. Connect to a Cluster
      • 2. Create a Scratch Working Directory
      • 3. Prepare the Example Files Locally
        • Create the Dataset
        • Create the Python Script
        • Create the LSF Job Script
      • 4. Transfer the Files to the Cluster
        • Transfer with SCP
        • Transfer with an SFTP Application
      • 5. Verify the Uploaded Files
      • 6. Submit the LSF Job
      • 7. Monitor the Job
      • 8. Inspect the Output
      • 9. Download the Result
      • 10. Disconnect
      • Next Steps
    • Important Guidelines
      • Accounts, Projects, and Policies
      • Network Access
      • Use Login Nodes Responsibly
      • Use Scratch Storage for Active Jobs
      • Request Appropriate Job Resources
      • Manage Software with Modules and Environments
      • Use the Data Transfer Node for I/O-Intensive Work
      • Getting Help
    • Projects & Resources
    • Job Scheduling
      • LSF Overview
      • Queues
        • Pegasus Queues
        • Triton Queues
      • Commands
      • Submitting Jobs
      • Monitoring Jobs
      • Examining Jobs
      • Interactive Jobs
      • Serial/Parallel Job Scripts
      • How to Request GPUs
      • Memory Management
        • Cluster Hardware and Queue Model
          • Standard compute nodes
          • Big-memory nodes (bigmem)
          • Queues and host spanning rules
        • Units and what is enforced
          • Memory units
          • Reservation vs. limit
        • How rusage[mem=…] behaves on Pegasus
          • Per-host reservation (the most important rule)
        • Controlling placement (single host vs multiple hosts)
          • Single-host placement
          • Multi-host placement (parallel jobs)
        • A simple formula to apply (parallel queue)
        • Guidance by queue
          • Queue: normal / general (single host only)
          • Queue: parallel (multi-host allowed)
          • Queue: bigmem (96 cores / 2 TB RAM)
        • Verifying what you asked for and what happened
          • Show what resources were requested
          • Confirm placement and packing
          • Diagnose pending jobs
        • Recommended starting points (Pegasus)
          • Standard node jobs (single host)
          • Parallel jobs
          • Bigmem jobs
        • Summary
    • Software and Modules
      • Software Modules
        • Basic Commands
        • Find Software
        • Load Software
        • Modules in LSF Jobs
        • System Notes
          • Pegasus
          • Triton/Summit
        • Recommended Practice
      • Python
        • Find and Load Python
        • User Package Installs
        • Virtual Environments
        • Run Python in LSF
        • Using a Virtual Environment in LSF
        • System Notes
          • Pegasus
          • Triton/Summit
        • Recommended Practice
      • Conda and Miniforge
        • Find Conda or Miniforge
        • Create an Environment
        • Use Project or Scratch Storage
        • Use Conda in LSF
        • Export an Environment
        • Pegasus Notes
        • Triton/Summit Notes
        • Recommended Practice
      • R
        • Find and Load R
        • User R Libraries
        • Run R in LSF
        • Recommended Practice
      • JupyterHub
        • IDSC-Managed JupyterHub
          • Typical Workflow
        • Personal JupyterLab Through LSF
          • Step 1: Connect to Pegasus
          • Step 2: Start an Interactive LSF Job
          • Step 3: Create the Startup Script
          • Step 4: Start JupyterLab
          • Step 5: Create the SSH Tunnel
          • Step 6: Open JupyterLab
          • Step 7: Stop the Personal JupyterLab Session
        • GPU JupyterLab Through LSF
          • Start a GPU Job
          • Start JupyterLab
          • Check GPU Access
          • PyTorch Check
          • TensorFlow Check
        • Troubleshooting
        • When to Contact IDSC
      • MATLAB
        • Find and Load MATLAB
        • Interactive MATLAB
        • Graphical MATLAB
        • Batch MATLAB Jobs
        • Running a MATLAB Function
        • Recommended Practice
      • Machine Learning Frameworks
        • Availability
        • Start a GPU Session
        • Architecture Notes
        • Check PyTorch
        • Check TensorFlow
        • Pegasus Notes
        • Triton/Summit Notes
        • Recommended Practice
      • Workflow Tools
        • Nextflow
          • Load Nextflow
          • Complete Pegasus Example
          • Storage Requirement
          • Resuming a Workflow
        • Recommended Practice
      • Debugging and Profiling Tools
        • Allinea Forge
        • DDT
        • MAP
        • Performance Reports
        • Graphical Tools
        • Recommended Practice
      • Specialized Scientific Software
        • Gaussian
          • Triton
          • Pegasus
        • GROMACS
          • Triton
          • Pegasus
        • LAMMPS
          • Triton
          • Pegasus
        • MAPSplice
          • Pegasus
        • SAS
          • Pegasus
        • SimVascular
          • Pegasus
    • Apptainer Containers
      • Quick Start
        • Load Apptainer
        • Cluster Module Quick Reference
        • View Apptainer Help
        • Download a Container Image
        • Shell: Interactive Use
        • Exec: Run a Command
        • Run: Use the Container Default Command
        • Choosing Between Shell, Exec, and Run
        • Command Order
        • Quick Validation Checklist
      • Container Images and Builds
        • Container Image Types
        • SIF Images
        • Image Cache
        • Temporary Writable Containers
        • Sandbox Directory
        • Persistent Overlay
        • Definition Files
        • Common Definition File Sections
        • Advanced Definition File Sections
        • Example Definition File
        • Build the Container
          • Fakeroot Shell Example
        • Skip Build-Time Tests
        • Test the Container
        • Run the Container
        • View Container Help
        • Recommended Usage
      • Inspecting and Verifying Images
        • Inspect an Image
        • View Container Help
        • Verify an Image
        • Sign an Image
        • List SIF Objects
        • Fingerprints Header
        • When to Inspect or Verify
        • Quick Command Summary
      • Working with Files and Bind Mounts
        • Default File Access
        • Run a Host Script Inside a Container
        • Writing Output Files
        • Custom Bind Mounts
          • Read-Only Bind Mounts
          • Using APPTAINER_BIND
          • Project/Scratch Bind Mount Example
        • Set the Working Directory
        • Pipes and Redirects
        • More Isolated Runs
        • Common Problems
        • Recommended Practice
      • Running Jobs with LSF
        • Interactive LSF Session
        • Direct Interactive Command
        • Batch Job Example
        • Use Full Paths in Job Scripts
        • Binding Data Directories in Jobs
        • Using --cleanenv
        • Avoid Downloading Images Inside Jobs
        • GPU Jobs
        • Recommended Job Pattern
        • Common LSF Commands
        • Common Problems
      • GPU Containers
        • Workflow
        • Prepare a Basic Image
        • Start a GPU Session
        • Check GPU Access on the Host
        • Run with --nv
        • Container Compatibility
        • Framework Images
          • Triton/Summit
          • Pegasus
        • Batch Jobs
        • Common Problems
        • Recommended Practice
      • Troubleshooting Common Issues
        • Apptainer Command Not Found
        • Image Is Read-Only
        • Wrong Architecture
        • File Not Found Inside Container
        • Permission Denied for Host Files
        • Output File Is Missing
        • Fakeroot Not Available
        • Package Installation Fails During Build
        • Container Works Interactively but Fails in LSF
        • GPU Not Visible Inside Container
        • TensorFlow Shows an Empty GPU List
        • PyTorch Reports CUDA Is Not Available
        • Environment Conflicts
        • Cache Uses Too Much Space
        • Remote Image Pull Fails
        • Jupyter or RStudio Left Running
        • When to Contact IDSC
      • Practical Examples
        • Example 1: Quick Container Check
        • Example 2: Run a Python Script
        • Example 3: Use Input and Output Directories
        • Example 4: CPU Batch Job with LSF
        • Example 5: Build a Small Container from a Definition File
        • Example 6: GPU Container Check
        • Example 7: More Isolated Run with Explicit Binds
        • Example 8: Reuse Bind Mounts with APPTAINER_BIND
        • Recommended Pattern
      • Advanced Topics
        • JupyterLab
          • One-Time Setup
          • Start JupyterLab
          • Request Different Resources
          • Create the SSH Tunnel
          • Open JupyterLab
          • Verify the Session
          • Manage JupyterLab Sessions
          • Stop the Session
          • Troubleshooting
        • RStudio Server
          • One-Time Setup
          • Start RStudio
          • Request Different Resources
          • Create the SSH Tunnel
          • Open RStudio Server
          • Verify the Session
          • Manage RStudio Sessions
          • Stop the Session
          • Troubleshooting
        • Interactive R
          • Pull the R container
          • Start an interactive LSF session
          • Run R inside the container
          • Example R session
          • End the interactive job
          • Using project data
          • Troubleshooting
        • Running MCMICRO
          • Load the Modules
          • Prepare the Directories
          • Download MCMICRO and the Exemplar Dataset
          • Create the Pegasus Configuration
          • Pre-pull the ASHLAR Image
          • Run the Registration Test
          • Troubleshooting
        • Advanced Image Security
          • Build-time Verification with Fingerprints
          • Fingerprint Notes
          • Creating Local Signing Keys
          • Signing an Image
          • SIF Object IDs and Groups
          • Object-level Verify Example
          • Security Practice
    • Parallel Computing
      • Loading the Appropriate MPI Module
      • Sample parallel programs:
        • C++ source code and compilation
        • C source code and compilation:
        • Fortran 90 source code and compilation:
        • Fortran 77 source code and compilation:
      • The LSF script to run parallel jobs
    • Sylvester HPC QuickStart
      • How do I get started?
      • Sylvester Utilization
      • Sylvester Dedicated HPC Queues
        • HPC Storage
        • Tier 1 Development (sccc-dev)
        • Tier 2 General (sccc)
        • Tier 3 Big Memory (sccc-bigmem)
        • Tier 3 GPU Queue (sccc-gpu)
        • Tier 4 Premium (sccc-premium, sccc-bigmem-premium, sccc-gpu-premium)
      • What are specs of available HPC Nodes?
        • Triton (96 nodes)
        • Pegasus Compute (350 nodes)
        • Pegasus Sylvester Dedicated Compute (16 nodes)
        • Pegasus Sylvester Dedicated Big Memory GPU (2 nodes)
      • How do I reset my IDSC password?
      • How do I access the Secure Network remotely?
      • How do I run Nextflow on Sylvester HPC resources?
  • IDSC ACS Policies & FAQs
    • Policies
      • Accounts
      • IDSC Links
      • Supercomputers
      • Allocations
      • Software
      • Support
    • FAQs
      • Projects and Resources
        • How do I join a project?
        • How do I request a new project?
        • When will my project be created?
        • How can I manage my Projects and Groups?
      • Software Suites
        • What software is available?
        • How do I view my currently loaded modules?
        • How do I use software modules?
        • May I install software?
        • How do I request global software installation?
        • When will my global software request be approved/installed?
        • How can I increase Java memory?
      • Job Scheduling
        • May I run resource-intensive jobs on login nodes?
        • How do I submit jobs?
        • How do I check on my submitted jobs?
        • How do I monitor job progress?
        • Is there a limit on how many jobs I can run?
        • How can I see pending and running job counts for queues?
        • Why is my job still pending?
        • Are other users’ pending jobs slowing my job?
        • How do I submit jobs to my Project?
        • How do I submit an interactive job?
        • How do I submit an interactive X11 job?
        • Why was my job killed?
        • What about jobs in UNKWN state?
        • GPU Resource Requests on Pegasus or Triton
        • Containerization Support on Pegasus
        • Disk Quota and Storage Expansion
    • Terms and Conditions
      • Secure Storage
acs_docs
  • Linux Guides
  • Navigating the Linux Shell
  • View page source

Navigating the Linux Shell

This guide provides an introduction to essential Linux shell commands for navigating and interacting with the filesystem, especially within a typical cluster login environment. The shell is provides a command-line interface (CLI) that allows users to communicate with the system by typing commands. On Triton/Pegasus, the default shell is Bash.

Users input commands into the shell, which executes them and displays the output. Commands may include options (also called flags) to modify behavior, and arguments to specify targets or inputs. To scroll through previously entered commands, use the up/down arrow keys.

Below is a collection of commonly used Linux shell commands, along with detailed explanations and examples. Remember that Linux is case-sensitive—“name” is different from “NAME”, “Name”, “nAMe”, and so on. To stop a running process and return to the prompt, press CTRL-C (on Windows) or Command-C (on macOS).

Check Your Current Shell with echo

The echo command displays text or variable values in the terminal. In Linux, variables are referenced using a leading dollar sign ($). For instance, the variable $SHELL stores the path to your default shell. To view it, you can pass the variable to echo. Variables can also be embedded within strings, as shown below.

[username@pegasus ~]$ echo $SHELL
/bin/bash
[username@pegasus ~]$ echo "My shell is $SHELL"
My shell is /bin/bash

List All Environment Variables with env

To view a complete list of your environment variables, use the env command. To organize the output in alphabetical order, combine it with sort as shown. The pipe (|) command is used to connect multiple commands in sequence, passing the output of one as the input to the next.

[username@pegasus ~]$ env
MODULE_VERSION_STACK=3.2.10
LC_PAPER=en_US.utf8
HOSTNAME=login4
SHELL=/bin/bash
...
[username@pegasus ~]$ env | sort
...

Check Your Current Directory with pwd

Linux uses directories, which are equivalent to folders in other operating systems. When you log in, you are placed in your home directory by default. The tilde (~) symbol is a shorthand for this location. The pwd (print working directory) command outputs the absolute path of your current location in the filesystem, beginning at the root directory (/).

[username@pegasus ~]$ pwd
/nethome/username

List Directory Contents with ls

Running the ls command without any arguments displays the contents of your current directory. Note: if this is your first login, your home directory may appear empty. Items that begin with a d in their permissions are directories.

[username@pegasus ~]$ ls
example_file1  example_file2  testdir1

To inspect a specific directory, provide its path as an argument to ls. In the example below, testdir1 is located in the home directory and contains one file: testdir1_file1.

[username@pegasus ~]$ ls testdir1
testdir1_file1

You can press the TAB key while typing a name to automatically complete it. If multiple possibilities exist, a list will appear. Continue typing more characters and press TAB again until completion succeeds.

You can learn more about command flags and usage by accessing the Linux manual pages using:

[username@pegasus ~]$ man topic or command

Use the arrow keys, Page Up, or Page Down to scroll. Press SPACE to view additional content, and type q to quit.

To display more detailed file information, use ls with flags like -lh:

[username@pegasus ~]$ ls -lh
total 0
-rw-r--r-- 1 username ccsuser  54  example_file1
-rw-r--r-- 1 username ccsuser 476  example_file2
drwxr-xr-x 2 username ccsuser 512  testdir1
...

Explanation of ls -lh flags:

  • -l: Long listing format (includes size, owner, permissions, etc.)

  • -h: Human-readable file sizes (e.g., 1K, 234M, 2G)

Other useful ls options include:

  • -a: Include hidden files (starting with .)

  • -d: Show directory details instead of listing contents

  • -1 (number 1): Output one entry per line

  • -R: Recursively display subdirectory contents

  • -S: Sort files by size

  • -X: Sort files alphabetically by extension

  • -m: Display output as a comma-separated list

* Hidden files include . (current directory) and .. (parent directory). These shortcuts are useful in relative paths:

[username@pegasus testdir1]$ ls -a
.  ..  testdir1_file1
[username@pegasus testdir1]$ ls ..
example_file1  example_file2  testdir1

Move Between Directories with cd

Use the cd (change directory) command to navigate the filesystem. The provided path can be absolute (starting with /) or relative to your current location.

[username@pegasus ~]$ cd testdir1
[username@pegasus testdir1]$

Some useful cd patterns:

  • cd or cd ~: Return to your home directory

  • cd ..: Move up to the parent directory

  • cd -: Return to the previous directory and display it

Visualize Directory Structure with tree

Pegasus includes the tree utility, which shows a recursive, indented listing of files and directories. This can be more informative than ls, especially for visualizing nested directories.

[username@pegasus ~]$ tree -vC
.
|-- example_file1
|-- example_file2
|-- testdir1
    `-- testdir1_file1

1 directory, 3 files

Explanation of tree -vC flags:

  • -v: Sort contents alphanumerically by type

  • -C: Enable colored output for better visibility

Other helpful tree flags include:

  • -a: Display hidden files

  • -d: Show directories only

  • -r: Reverse the sorting order

  • -L number: Limit the depth of displayed directory levels

Locate Commands with which

The which command shows the full path of a command executable by searching directories listed in your $PATH environment variable. Use which to verify if a command is installed and to find its location.

[username@pegasus ~]$ which bash
/bin/bash
[username@pegasus ~]$ which vim
/usr/bin/vim
[username@pegasus ~]$ which python
/share/opt/python/2.7.3/bin/python
Previous Next

© Copyright 2024, UM IDSC ACS.

Built with Sphinx using a theme provided by Read the Docs.