PetscRegressor: Regression Solvers#

The PetscRegressor component provides some basic infrastructure and a general API for supervised machine learning tasks at a higher level of abstraction than a purely algebraic “solvers” view. Methods are currently available for

Linear regressor

Note that by “regressor” we mean an algorithm or implementation used to fit and apply a regression model, following standard parlance in the machine-learning community. Regressor here does NOT mean an independent (or predictor) variable, as it often does in the statistics community.

Basic Regressor Usage#

PetscRegressor supports supervised learning tasks: Given a matrix of observed data \(X\) with size \(n_{samples}\) by \(n_{features}\), predict a vector of “target” values \(y\) (of size \(n_{samples}\)), where the \(i\)th entry of \(y\) corresponds to the observation (or “sample”) stored in the \(i\)th row of \(X\). Traditionally, when the target consists of continuous values this is called “regression”, and when it consists of discrete values (or “labels”), this task is called “classification”; we use PetscRegressor to support both of these cases.

Before a regressor can be used to make predictions, the model must be fitted using an initial set of training data. Once a fitted model has been obtained, it can be used to predict target values for new observations. Every PetscRegressor implementation provides a Fit() and a Predict() method to support this workflow. Fitting (or “training”) a model is a relatively computationally intensive task that generally involves solving an optimization problem (often using TAO solvers) to determine the model parameters, whereas making predictions (or performing “inference”) is generally much simpler.

Here, we introduce a simple example to demonstrate PetscRegressor usage. Please read Regression Solvers for more in-depth discussion. The code presented below solves an ordinary linear regression problem, with various options for regularization.

In the simplest usage of a regressor, the user provides a training (or “design”) matrix (Mat) and a target vector (Vec) against which to fit the model. Once the regressor is fitted, the user can then obtain a vector of predicted values for a set of new observations.

PETSc’s default method for solving regression problems is ordinary least squares, REGRESSOR_LINEAR_OLS, which is a sub-type of linear regressor, PETSCREGRESSORLINEAR. By “linear” we mean that the model \(f(x, \theta)\) is linear in its coefficients \(\theta\) but not necessarily linear in its features \(x\).

Note that data creation, option parsing, and cleaning stages are omitted here for clarity. The complete code is available in ex3.c.

Listing: src/ml/regressor/tests/ex3.c

#include <petscregressor.h>
int main(int argc, char **args)
{
  AppCtx         ctx;
  PetscRegressor regressor;
  PetscScalar    intercept;

  /* Initialize PETSc */
  PetscCall(PetscInitialize(&argc, &args, (char *)0, help));

  /* Initialize problem parameters and data */
  PetscCall(PetscNew(&ctx));
  PetscCall(ConfigureContext(ctx));
  PetscCall(CreateData(ctx));

  /* Create Regressor solver with desired type and options */
  PetscCall(PetscRegressorCreate(PETSC_COMM_WORLD, &regressor));
  PetscCall(PetscRegressorSetType(regressor, PETSCREGRESSORLINEAR));
  PetscCall(PetscRegressorLinearSetType(regressor, REGRESSOR_LINEAR_OLS));
  PetscCall(PetscRegressorLinearSetFitIntercept(regressor, PETSC_FALSE));
  /* Testing prefix functions for Regressor */
  PetscCall(TestPrefixRegressor(regressor, ctx));
  /* Check for command line options */
  PetscCall(PetscRegressorSetFromOptions(regressor));
  /* Fit the regressor */
  PetscCall(PetscRegressorFit(regressor, ctx->X, ctx->y));
  /* Predict data with fitted regressor */
  PetscCall(PetscRegressorPredict(regressor, ctx->X, ctx->y_predicted));
  /* Get other desired output data */
  PetscCall(PetscRegressorLinearGetIntercept(regressor, &intercept));
  PetscCall(PetscRegressorLinearGetCoefficients(regressor, &ctx->coefficients));

  /* Testing Views, and GetTypes */
  PetscCall(TestRegressorViews(regressor, ctx));
  PetscCall(PetscRegressorDestroy(&regressor));
  PetscCall(DestroyCtx(&ctx));
  PetscCall(PetscFinalize());
return 0;}

To create a PetscRegressor instance, one must first call PetscRegressorCreate():

PetscRegressorCreate(MPI_Comm comm, PetscRegressor *regressor);

To choose a regressor type, the user can either call

PetscRegressorSetType(PetscRegressor regressor, PetscRegressorType type);

or use the command-line option -regressor_type <method>; details regarding the available methods are presented in Regression Solvers. The application code can specify the options used by underlying linear, nonlinear, and optimization solver methods used in fitting the model by calling

PetscRegressorSetFromOptions(regressor);

which interfaces with the PETSc options database and enables convenient runtime selection of the type of regression algorithm and setting various various solver or problem parameters. This routine can also control all inner solver options in the KSP, and Tao modules, as discussed in KSP: Linear System Solvers, TAO: Optimization Solvers.

After having set these routines and options, the user can fit (or “train”) the regressor by calling

PetscRegressorFit(PetscRegressor regressor, Mat X, Vec y);

where X is training data, and y is target values. Finally, after fitting the regressor, the user can compute model predictions, that is, perform inference, for a data matrix of unlabeled observations using the fitted regressor:

PetscRegressorPredict(PetscRegressor regressor, Mat X, Vec y_predicted);

Finally, after the user is done using the regressor, the user should destroy its PetscRegressor context with

PetscRegressorDestroy(PetscRegressor *regressor);

Regression Solvers#

One can see the list of regressor types in Table PETSc Regressor. Currently, we only support one type, PETSCREGRESSORLINEAR, although we plan to add several others in the near future.

Table 20 PETSc Regressor#
Method	PetscRegressorType	Options Name
Linear	`PETSCREGRESSORLINEAR`	`linear`

If the particular method being employed is one that supports regularization, the user can set regularizer’s weight via

PetscRegressorSetRegularizerWeight(PetscRegressor regressor, PetscReal weight);

or with the option -regressor_regularizer_weight <weight>.

Linear regressor#

The PETSCREGRESSORLINEAR (-regressor_type linear) implementation constructs a linear model to reduce the sum of squared differences between the actual target values (“observations”) in the dataset and the target values estimated by the fitted model. By default, bound-constrained regularized Gauss-Newton TAOBRGN is used to solve the underlying optimization problem.

Currently, linear regressor has three types, which are described in Table Linear Regressor types.

Table 21 Linear Regressor types#
Linear method	`PetscRegressorLinearType`	Options Name
Ordinary	`REGRESSOR_LINEAR_OLS`	`ols`
Lasso	`REGRESSOR_LINEAR_LASSO`	`lasso`
Ridge	`REGRESSOR_LINEAR_RIDGE`	`ridge`

If one wishes, the user can (when appropriate) use KSP to solve the problem, instead of Tao, via

PetscRegressorLinearSetUseKSP(PetscRegressor regressor, PetscBool flg);

or with the option -regressor_linear_use_ksp <true,false>.

Calculation of the intercept (also known as the “bias” or “offset”) is performed separately from the rest of the model fitting process, because data sets are often already mean-centered and because it is generally undesirable to regularize the intercept term. By default, this step is omitted; if the user wishes to compute the intercept, this can be done by calling

PetscRegressorLinearSetFitIntercept(PetscRegressor regressor, PetscBool flg);

or by specifying the option -regressor_linear_fit_intercept <true,false>.

For a fitted regression, one can obtain the intercept and a vector of the model coefficients from a linear regression model via

PetscRegressorLinearGetCoefficients(PetscRegressor regressor, Vec *coefficients);
PetscRegressorLinearGetIntercept(PetscRegressor regressor, PetscScalar *intercept);