User-Guide

Argonne National Laboratory

Mathematics and Computer Science Division

---

Prepared by

S. Balay ¹, S. Abhyankar ^1,2, M. F. Adams ³, S. Benson ¹, J. Brown ^1,10, P. Brune ¹, K. Buschelman ¹, E. M. Constantinescu ¹, L. Dalcin ⁴, A. Dener ¹, V. Eijkhout ⁶, J. Faibussowitsch ^1,18, W. D. Gropp ^1,18, V. Hapla ⁸, T. Isaac ^1,14, P. Jolivet ^12,22, D. Karpeev ¹, D. Kaushik ¹, M. G. Knepley ^1,9, F. Kong ^1,11, S. Kruger ¹⁵, D. A. May ^7,21, L. Curfman McInnes ¹, R. Tran Mills ¹, L. Mitchell ^13,20, T. Munson ¹, J. E. Roman ¹⁶, K. Rupp ^1,19, P. Sanan ^1,8, J. Sarich ¹, B. F. Smith ^1,17, H. Suh ¹, S. Zampini ⁴, H. Zhang ^1,5, H. Zhang ¹, and J. Zhang ¹

¹Mathematics and Computer Science Division, Argonne National Laboratory

²Electricity Infrastructure and Buildings Division, Pacific Northwest National Laboratory

³Computational Research, Lawrence Berkeley National Laboratory

⁴Extreme Computing Research Center, King Abdullah University of Science and Technology

⁵Department of Computer Science, Illinois Institute of Technology

⁶Texas Advanced Computing Center, University of Texas at Austin

⁷Department of Earth Sciences, University of Oxford

⁸Institute of Geophysics, ETH Zurich

⁹Department of Computer Science and Engineering, University at Buffalo

¹⁰Department of Computer Science, University of Colorado, Boulder

¹¹Computational Frameworks, Idaho National Laboratory

¹²Sorbonne Université, CNRS, LIP6

¹³NVIDIA Corporation

¹⁴College of Computing, Georgia Tech

¹⁵Tech-X Corporation

¹⁶DSIC, Universitat Politècnica de València

¹⁷Flatiron Institute, Simons Foundation

¹⁸University of Illinois, Urbana-Champaign

¹⁹Institute for Microelectronics, TU Wien

²⁰Department of Computer Science, Durham University

²¹Scripps Institution of Oceanography, University of California, San Diego

²²Toulouse INP, CNRS, Institute of Computer Science Research

---

This work was supported by the Office of Advanced Scientific Computing Research, Office of Science, U.S. Department of Energy, under Contract DE-AC02-06CH11357.

---

• Introduction to PETSc
  • About This Manual
  • Getting Started
    • Suggested Reading
    • Running PETSc Programs
    • Writing PETSc Programs
    • Simple PETSc Examples
  • Parallel and GPU Programming
    • MPI Parallelism
    • CPU SIMD parallelism
    • CPU OpenMP parallelism
    • GPU kernel parallelism
    • GPU stream parallelism
  • Compiling and Running Programs
  • Profiling Programs
  • Writing C/C++ or Fortran Applications
  • PETSc’s Object-Oriented Design
    • User Callbacks
  • Directory Structure
• The Solvers in PETSc/TAO
  • Vectors and Parallel Data
    • Creating Vectors
    • Common vector functions and operations
    • Assembling (putting values in) vectors
    • Basic Vector Operations
    • Local/global vectors and communicating between vectors
    • Low-level Vector Communication
    • Application Orderings
  • Matrices
    • Creating matrices
    • Low-level matrix creation routines
    • Assembling (putting values into) matrices
    • Basic Matrix Operations
    • Matrix-Free Matrices
    • Transposes of Matrices
    • Other Matrix Operations
    • Symbolic and Numeric Stages in Sparse Matrix Operations
    • Graph Operations
    • Partitioning
  • KSP: Linear System Solvers
    • Using KSP
    • Solving Successive Linear Systems
    • Krylov Methods
    • Preconditioners
    • Solving Block Matrices with PCFIELDSPLIT
    • Solving Singular Systems
    • Using External Linear Solvers
    • Using PETSc’s MPI parallel linear solvers from a non-MPI program
  • SNES: Nonlinear Solvers
    • Basic SNES Usage
    • The Nonlinear Solvers
    • General Options
    • Inexact Newton-like Methods
    • Matrix-Free Methods
    • Finite Difference Jacobian Approximations
    • Variational Inequalities
    • Nonlinear Preconditioning
  • TS: Scalable ODE and DAE Solvers
    • Basic TS Options
    • DAE Formulations
    • Using Implicit-Explicit (IMEX) Methods
    • IMEX Methods for fast-slow systems
    • GLEE methods
    • Using fully implicit methods
    • Using the Explicit Runge-Kutta timestepper with variable timesteps
    • Special Cases
    • Monitoring and visualizing solutions
    • Error control via variable time-stepping
    • Handling of discontinuities
    • Explicit integrators with finite element mass matrices
    • Performing sensitivity analysis with the TS ODE Solvers
    • Using Sundials from PETSc
    • Using TChem from PETSc
  • Solving Steady-State Problems with Pseudo-Timestepping
  • TAO: Optimization Solvers
    • Getting Started: A Simple TAO Example
    • TAO Workflow
    • TAO Algorithms
    • Advanced Options
    • Adding a Solver
• DM: Interfacing Between Solvers and Models/Discretizations
  • DM Basics
  • DMPlex: Unstructured Grids
    • Representing Unstructured Grids
    • Dealing with Periodicity
    • Connecting Data on Grids to its Location in arrays or Vec (PetscSection)
    • Evaluating Residuals
    • Saving and Loading DMPlex Data with HDF5
    • Metric-based mesh adaptation
  • DMSTAG: Staggered, Structured Grid
    • Terminology
    • Working with vectors and operators (matrices)
    • Coordinates
    • Numberings and internal data layout
  • Networks
    • Application flow
    • Utility functions
    • Retrieving components and number of variables
  • PetscDT: Discretization Technology in PETSc
    • Quadrature
    • Probability Distributions
  • PetscFE: Finite Element Infrastructure in PETSc
    • Using Pointwise Functions to Specify Finite Element Problems
    • Describing a particular finite element problem to PETSc
    • Assembling finite element residuals and Jacobians
• Additional Information
  • PETSc for Fortran Users
    • Basic Fortran API Differences
    • Sample Fortran Programs
  • Checking the PETSc version
    • During configure/make time
    • During compile time
    • At Runtime
  • Using MATLAB with PETSc
    • Dumping Data for MATLAB
    • Sending Data to an Interactive MATLAB Session
    • Using the MATLAB Compute Engine
    • Licensing the MATLAB Compute Engine on a cluster
  • Profiling
    • Basic Profiling Information
    • Profiling Application Codes
    • Profiling Multiple Sections of Code
    • Restricting Event Logging
    • Interpreting -info Output: Informative Messages
    • Time
    • Saving Output to a File
    • Accurate Profiling and Paging Overheads
    • NVIDIA Nsight Systems profiling
    • Using TAU
  • Hints for Performance Tuning
    • Maximizing Memory Bandwidth
    • Performance Pitfalls and Advice
  • STREAMS: Example Study
    • Detailed STREAMS study for large arrays
    • Detailed study with application
    • Application with the MPI linear solver server
  • The Use of BLAS and LAPACK in PETSc and external libraries
    • 32 or 64-bit BLAS/LAPACK integers
    • Shared memory BLAS/LAPACK parallelism
    • Available BLAS/LAPACK libraries
  • Other PETSc Features
    • PETSc on a process subset
    • Runtime Options
    • Viewers: Looking at PETSc Objects
    • Using SAWs with PETSc
    • Debugging
    • Error Handling
    • Numbers
    • Parallel Communication
    • Graphics
  • Developer Environments
    • Emacs Users
    • VS Code Users
    • Vi and Vim Users
    • Eclipse Users
    • Qt Creator Users
    • Visual Studio Users
    • Xcode IDE Users
  • Advanced Features of Matrices and Solvers
    • Extracting Submatrices
    • Matrix Factorization
    • Matrix-Matrix Products
    • Creating PC’s Directly
  • Running PETSc Tests
    • Quick start with the tests
    • Understanding test output and more information
    • Using the test harness for your own code