Extrinsic UQ Algorithms

Auxiliary Interval Predictor

class uq360.algorithms.auxiliary_interval_predictor.AuxiliaryIntervalPredictor(model_type=None, main_model=None, aux_model=None, config=None, device=None, verbose=True)

Auxiliary Interval Predictor 1 uses an auxiliary model to encourage calibration of the main model.

References

1

Thiagarajan, J. J., Venkatesh, B., Sattigeri, P., & Bremer, P. T. (2020, April). Building calibrated deep models via uncertainty matching with auxiliary interval predictors. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 6005-6012). https://arxiv.org/abs/1909.04079

Parameters
  • model_type – The model type used to build the main model and the auxiliary model. Currently supported values are [mlp, custom]. mlp modeltype learns a mlp neural network using pytorch framework. For custom the user provide main_model and aux_model.

  • main_model – (optional) The main prediction model. Currently support pytorch models that return mean and log variance.

  • aux_model – (optional) The auxiliary prediction model. Currently support pytorch models that return calibrated log variance.

  • config – dictionary containing the config parameters for the model.

  • device – device used for pytorch models ignored otherwise.

  • verbose – if True, print statements with the progress are enabled.

fit(X, y)

Fit the Auxiliary Interval Predictor model.

Parameters
  • X – array-like of shape (n_samples, n_features). Features vectors of the training data.

  • y – array-like of shape (n_samples,) or (n_samples, n_targets) Target values

Returns

self

predict(X, return_dists=False)

Obtain predictions for the test points.

In addition to the mean and lower/upper bounds, also returns full predictive distribution (return_dists=True).

Parameters
  • X – array-like of shape (n_samples, n_features). Features vectors of the test points.

  • return_dists – If True, the predictive distribution for each instance using scipy distributions is returned.

Returns

A namedtupe that holds

y_mean: ndarray of shape (n_samples, [n_output_dims])

Mean of predictive distribution of the test points.

y_lower: ndarray of shape (n_samples, [n_output_dims])

Lower quantile of predictive distribution of the test points.

y_upper: ndarray of shape (n_samples, [n_output_dims])

Upper quantile of predictive distribution of the test points.

dists: list of predictive distribution as scipy.stats objects with length n_samples.

Only returned when return_dists is True.

Return type

namedtuple

Blackbox Metamodel Classification

class uq360.algorithms.blackbox_metamodel.BlackboxMetamodelClassification(base_model=None, meta_model=None, base_config=None, meta_config=None, random_seed=42)

Extracts confidence scores from black-box classification models using a meta-model 4 .

References

4

Chen, Tongfei, et al. “Confidence scoring using whitebox meta-models with linear classifier probes.” The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

Parameters
  • base_model – Base model. Can be: (1) None (default mdl will be set up), (2) Named model (e.g., logistic regression ‘lr’ or gradient boosting machine ‘gbm’), (3) Base model class declaration (e.g., sklearn.linear_model.LogisticRegression). Will instantiate. (4) Model instance (instantiated outside). Will be re-used. Must have certain callable methods. Note: user-supplied classes and models must have certain callable methods (‘predict’, ‘fit’) and be capable of raising NotFittedError.

  • meta_model – Meta model. Same values possible as with ‘base_model’

  • base_config – None or a params dict to be passed to ‘base_model’ at instantiation

  • meta_config – None or a params dict to be passed to ‘meta_model’ at instantiation

  • random_seed – seed used in the various pipeline steps

fit(X, y, meta_fraction=0.2, randomize_samples=True, base_is_prefitted=False, meta_train_data=(None, None))

Fit base and meta models.

Parameters
  • X – input to the base model, array-like of shape (n_samples, n_features). Features vectors of the training data.

  • y – ground truth for the base model, array-like of shape (n_samples,)

  • meta_fraction – float in [0,1] - a fractional size of the partition carved out to train the meta model (complement will be used to train the base model)

  • randomize_samples – use shuffling when creating partitions

  • base_is_prefitted – Setting True will skip fitting the base model (useful for base models that have been instantiated outside/by the user and are already fitted.

  • meta_train_data – User supplied data to train the meta model. Note that this option should only be used with ‘base_is_prefitted’==True. Pass a tuple meta_train_data=(X_meta, y_meta) to activate. Note that (X,y,meta_fraction, randomize_samples) will be ignored in this mode.

Returns

self

get_params(deep=True)

This method should not take any arguments and returns a dict of the __init__ parameters.

predict(X)

Generate a base prediction along with uncertainty/confidence for data X.

Parameters

X – array-like of shape (n_samples, n_features). Features vectors of the test points.

Returns

namedtuple: A namedtuple that holds

y_pred: ndarray of shape (n_samples,)

Predicted labels of the test points.

y_score: ndarray of shape (n_samples,)

Confidence score the test points.

Blackbox Metamodel Regression

class uq360.algorithms.blackbox_metamodel.BlackboxMetamodelRegression(base_model=None, meta_model=None, base_config=None, meta_config=None, random_seed=42)

Extracts confidence scores from black-box regression models using a meta-model 2 .

References

2

Chen, Tongfei, et al. Confidence scoring using whitebox meta-models with linear classifier probes. The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

Parameters
  • base_model – Base model. Can be: (1) None (default mdl will be set up), (2) Named model (e.g., ‘gbr’), (3) Base model class declaration (e.g., sklearn.linear_model.LinearRegressor). Will instantiate. (4) Model instance (instantiated outside). Will be re-used. Must have required callable methods. Note: user-supplied classes and models must have certain callable methods (‘predict’, ‘fit’) and be capable of raising NotFittedError.

  • meta_model – Meta model. Same values possible as with ‘base_model’

  • base_config – None or a params dict to be passed to ‘base_model’ at instantiation

  • meta_config – None or a params dict to be passed to ‘meta_model’ at instantiation

  • random_seed – seed used in the various pipeline steps

fit(X, y, meta_fraction=0.2, randomize_samples=True, base_is_prefitted=False, meta_train_data=(None, None))

Fit base and meta models.

Parameters
  • X – input to the base model

  • y – ground truth for the base model

  • meta_fraction – float in [0,1] - a fractional size of the partition carved out to train the meta model (complement will be used to train the base model)

  • randomize_samples – use shuffling when creating partitions

  • base_is_prefitted – Setting True will skip fitting the base model (useful for base models that have been instantiated outside/by the user and are already fitted.

  • meta_train_data – User supplied data to train the meta model. Note that this option should only be used with ‘base_is_prefitted’==True. Pass a tuple meta_train_data=(X_meta, y_meta) to activate. Note that (X,y,meta_fraction, randomize_samples) will be ignored in this mode.

Returns

self

get_params(deep=True)

This method should not take any arguments and returns a dict of the __init__ parameters.

predict(X)

Generate prediction and uncertainty bounds for data X.

Parameters

X – input features

Returns

namedtuple: A namedtuple that holds

y_mean: ndarray of shape (n_samples, [n_output_dims])

Mean of predictive distribution of the test points.

y_lower: ndarray of shape (n_samples, [n_output_dims])

Lower quantile of predictive distribution of the test points.

y_upper: ndarray of shape (n_samples, [n_output_dims])

Upper quantile of predictive distribution of the test points.

Infinitesimal Jackknife

class uq360.algorithms.infinitesimal_jackknife.InfinitesimalJackknife(params, gradients, hessian, config)

Performs a first order Taylor series expansion around MLE / MAP fit. Requires the model being probed to be twice differentiable.

Initialize IJ. :param params: MLE / MAP fit around which uncertainty is sought. d*1 :param gradients: Per data point gradients, estimated at the MLE / MAP fit. d*n :param hessian: Hessian evaluated at the MLE / MAP fit. d*d

approx_ij(w_query)
Parameters

w_query – A n*1 vector to query parameters at.

Returns

new parameters at w_query

get_params(deep=True)

This method should not take any arguments and returns a dict of the __init__ parameters.

ij(w_query)
Parameters

w_query – A n*1 vector to query parameters at.

Returns

new parameters at w_query

predict(X, model)
Parameters
  • X – array-like of shape (n_samples, n_features). Features vectors of the test points.

  • model – model object, must implement a set_parameters function

Returns

A namedtupe that holds

y_mean: ndarray of shape (n_samples, [n_output_dims])

Mean of predictive distribution of the test points.

y_lower: ndarray of shape (n_samples, [n_output_dims])

Lower quantile of predictive distribution of the test points.

y_upper: ndarray of shape (n_samples, [n_output_dims])

Upper quantile of predictive distribution of the test points.

Return type

namedtuple

Classification Calibration

class uq360.algorithms.classification_calibration.ClassificationCalibration(num_classes, fit_mode='features', method='isotonic', base_model_prediction_func=None)

Post hoc calibration of classification models. Currently wraps CalibratedClassifierCV from sklearn and allows non-sklearn models to be calibrated.

Parameters
  • num_classes – number of classes.

  • fit_mode – features or probs. If probs the fit and predict operate on the base models probability scores, useful when these are precomputed.

  • method – isotonic or sigmoid.

  • base_model_prediction_func – the function that takes in the input features and produces base model’s probability scores. This is ignored when operating in probs mode.

fit(X, y)

Fits calibration model using the provided calibration set.

Parameters
  • X – array-like of shape (n_samples, n_features) or (n_samples, n_classes). Features vectors of the training data or the probability scores from the base model.

  • y – array-like of shape (n_samples,) or (n_samples, n_targets) Target values

Returns

self

get_params(deep=True)

This method should not take any arguments and returns a dict of the __init__ parameters.

predict(X)

Obtain calibrated predictions for the test points.

Parameters

X – array-like of shape (n_samples, n_features) or (n_samples, n_classes). Features vectors of the training data or the probability scores from the base model.

Returns

A namedtupe that holds

y_pred: ndarray of shape (n_samples,)

Predicted labels of the test points.

y_prob: ndarray of shape (n_samples, n_classes)

Predicted probability scores of the classes.

Return type

namedtuple

UCC Recalibration

class uq360.algorithms.ucc_recalibration.UCCRecalibration(base_model)

Recalibration a regression model to specified operating point using Uncertainty Characteristics Curve.

Parameters

base_model – pretrained model to be recalibrated.

fit(X, y)

Fit the Uncertainty Characteristics Curve.

Parameters
  • X – array-like of shape (n_samples, n_features). Features vectors of the test points.

  • y – array-like of shape (n_samples,) or (n_samples, n_targets) Target values

Returns

self

get_params(deep=True)

This method should not take any arguments and returns a dict of the __init__ parameters.

predict(X, missrate=0.05)

Generate prediction and uncertainty bounds for data X.

Parameters
  • X – array-like of shape (n_samples, n_features). Features vectors of the test points.

  • missrate – desired missrate of the new operating point, set to 0.05 by default.

Returns

A namedtupe that holds

y_mean: ndarray of shape (n_samples, [n_output_dims])

Mean of predictive distribution of the test points.

y_lower: ndarray of shape (n_samples, [n_output_dims])

Lower quantile of predictive distribution of the test points.

y_upper: ndarray of shape (n_samples, [n_output_dims])

Upper quantile of predictive distribution of the test points.

Return type

namedtuple