Extrinsic UQ Algorithms¶
Auxiliary Interval Predictor¶
- class uq360.algorithms.auxiliary_interval_predictor.AuxiliaryIntervalPredictor(model_type=None, main_model=None, aux_model=None, config=None, device=None, verbose=True)¶
Auxiliary Interval Predictor 1 uses an auxiliary model to encourage calibration of the main model.
References
- 1
Thiagarajan, J. J., Venkatesh, B., Sattigeri, P., & Bremer, P. T. (2020, April). Building calibrated deep models via uncertainty matching with auxiliary interval predictors. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 6005-6012). https://arxiv.org/abs/1909.04079
- Parameters
model_type – The model type used to build the main model and the auxiliary model. Currently supported values are [mlp, custom]. mlp modeltype learns a mlp neural network using pytorch framework. For custom the user provide main_model and aux_model.
main_model – (optional) The main prediction model. Currently support pytorch models that return mean and log variance.
aux_model – (optional) The auxiliary prediction model. Currently support pytorch models that return calibrated log variance.
config – dictionary containing the config parameters for the model.
device – device used for pytorch models ignored otherwise.
verbose – if True, print statements with the progress are enabled.
- fit(X, y)¶
Fit the Auxiliary Interval Predictor model.
- Parameters
X – array-like of shape (n_samples, n_features). Features vectors of the training data.
y – array-like of shape (n_samples,) or (n_samples, n_targets) Target values
- Returns
self
- predict(X, return_dists=False)¶
Obtain predictions for the test points.
In addition to the mean and lower/upper bounds, also returns full predictive distribution (return_dists=True).
- Parameters
X – array-like of shape (n_samples, n_features). Features vectors of the test points.
return_dists – If True, the predictive distribution for each instance using scipy distributions is returned.
- Returns
A namedtupe that holds
- y_mean: ndarray of shape (n_samples, [n_output_dims])
Mean of predictive distribution of the test points.
- y_lower: ndarray of shape (n_samples, [n_output_dims])
Lower quantile of predictive distribution of the test points.
- y_upper: ndarray of shape (n_samples, [n_output_dims])
Upper quantile of predictive distribution of the test points.
- dists: list of predictive distribution as scipy.stats objects with length n_samples.
Only returned when return_dists is True.
- Return type
namedtuple
Blackbox Metamodel Classification¶
- class uq360.algorithms.blackbox_metamodel.BlackboxMetamodelClassification(base_model=None, meta_model=None, base_config=None, meta_config=None, random_seed=42)¶
Extracts confidence scores from black-box classification models using a meta-model 4 .
References
- 4
Chen, Tongfei, et al. “Confidence scoring using whitebox meta-models with linear classifier probes.” The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.
- Parameters
base_model – Base model. Can be: (1) None (default mdl will be set up), (2) Named model (e.g., logistic regression ‘lr’ or gradient boosting machine ‘gbm’), (3) Base model class declaration (e.g., sklearn.linear_model.LogisticRegression). Will instantiate. (4) Model instance (instantiated outside). Will be re-used. Must have certain callable methods. Note: user-supplied classes and models must have certain callable methods (‘predict’, ‘fit’) and be capable of raising NotFittedError.
meta_model – Meta model. Same values possible as with ‘base_model’
base_config – None or a params dict to be passed to ‘base_model’ at instantiation
meta_config – None or a params dict to be passed to ‘meta_model’ at instantiation
random_seed – seed used in the various pipeline steps
- fit(X, y, meta_fraction=0.2, randomize_samples=True, base_is_prefitted=False, meta_train_data=(None, None))¶
Fit base and meta models.
- Parameters
X – input to the base model, array-like of shape (n_samples, n_features). Features vectors of the training data.
y – ground truth for the base model, array-like of shape (n_samples,)
meta_fraction – float in [0,1] - a fractional size of the partition carved out to train the meta model (complement will be used to train the base model)
randomize_samples – use shuffling when creating partitions
base_is_prefitted – Setting True will skip fitting the base model (useful for base models that have been instantiated outside/by the user and are already fitted.
meta_train_data – User supplied data to train the meta model. Note that this option should only be used with ‘base_is_prefitted’==True. Pass a tuple meta_train_data=(X_meta, y_meta) to activate. Note that (X,y,meta_fraction, randomize_samples) will be ignored in this mode.
- Returns
self
- get_params(deep=True)¶
This method should not take any arguments and returns a dict of the __init__ parameters.
- predict(X)¶
Generate a base prediction along with uncertainty/confidence for data X.
- Parameters
X – array-like of shape (n_samples, n_features). Features vectors of the test points.
- Returns
namedtuple: A namedtuple that holds
- y_pred: ndarray of shape (n_samples,)
Predicted labels of the test points.
- y_score: ndarray of shape (n_samples,)
Confidence score the test points.
Blackbox Metamodel Regression¶
- class uq360.algorithms.blackbox_metamodel.BlackboxMetamodelRegression(base_model=None, meta_model=None, base_config=None, meta_config=None, random_seed=42)¶
Extracts confidence scores from black-box regression models using a meta-model 2 .
References
- 2
Chen, Tongfei, et al. Confidence scoring using whitebox meta-models with linear classifier probes. The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.
- Parameters
base_model – Base model. Can be: (1) None (default mdl will be set up), (2) Named model (e.g., ‘gbr’), (3) Base model class declaration (e.g., sklearn.linear_model.LinearRegressor). Will instantiate. (4) Model instance (instantiated outside). Will be re-used. Must have required callable methods. Note: user-supplied classes and models must have certain callable methods (‘predict’, ‘fit’) and be capable of raising NotFittedError.
meta_model – Meta model. Same values possible as with ‘base_model’
base_config – None or a params dict to be passed to ‘base_model’ at instantiation
meta_config – None or a params dict to be passed to ‘meta_model’ at instantiation
random_seed – seed used in the various pipeline steps
- fit(X, y, meta_fraction=0.2, randomize_samples=True, base_is_prefitted=False, meta_train_data=(None, None))¶
Fit base and meta models.
- Parameters
X – input to the base model
y – ground truth for the base model
meta_fraction – float in [0,1] - a fractional size of the partition carved out to train the meta model (complement will be used to train the base model)
randomize_samples – use shuffling when creating partitions
base_is_prefitted – Setting True will skip fitting the base model (useful for base models that have been instantiated outside/by the user and are already fitted.
meta_train_data – User supplied data to train the meta model. Note that this option should only be used with ‘base_is_prefitted’==True. Pass a tuple meta_train_data=(X_meta, y_meta) to activate. Note that (X,y,meta_fraction, randomize_samples) will be ignored in this mode.
- Returns
self
- get_params(deep=True)¶
This method should not take any arguments and returns a dict of the __init__ parameters.
- predict(X)¶
Generate prediction and uncertainty bounds for data X.
- Parameters
X – input features
- Returns
namedtuple: A namedtuple that holds
- y_mean: ndarray of shape (n_samples, [n_output_dims])
Mean of predictive distribution of the test points.
- y_lower: ndarray of shape (n_samples, [n_output_dims])
Lower quantile of predictive distribution of the test points.
- y_upper: ndarray of shape (n_samples, [n_output_dims])
Upper quantile of predictive distribution of the test points.
Infinitesimal Jackknife¶
- class uq360.algorithms.infinitesimal_jackknife.InfinitesimalJackknife(params, gradients, hessian, config)¶
Performs a first order Taylor series expansion around MLE / MAP fit. Requires the model being probed to be twice differentiable.
Initialize IJ. :param params: MLE / MAP fit around which uncertainty is sought. d*1 :param gradients: Per data point gradients, estimated at the MLE / MAP fit. d*n :param hessian: Hessian evaluated at the MLE / MAP fit. d*d
- approx_ij(w_query)¶
- Parameters
w_query – A n*1 vector to query parameters at.
- Returns
new parameters at w_query
- get_params(deep=True)¶
This method should not take any arguments and returns a dict of the __init__ parameters.
- ij(w_query)¶
- Parameters
w_query – A n*1 vector to query parameters at.
- Returns
new parameters at w_query
- predict(X, model)¶
- Parameters
X – array-like of shape (n_samples, n_features). Features vectors of the test points.
model – model object, must implement a set_parameters function
- Returns
A namedtupe that holds
- y_mean: ndarray of shape (n_samples, [n_output_dims])
Mean of predictive distribution of the test points.
- y_lower: ndarray of shape (n_samples, [n_output_dims])
Lower quantile of predictive distribution of the test points.
- y_upper: ndarray of shape (n_samples, [n_output_dims])
Upper quantile of predictive distribution of the test points.
- Return type
namedtuple
Classification Calibration¶
- class uq360.algorithms.classification_calibration.ClassificationCalibration(num_classes, fit_mode='features', method='isotonic', base_model_prediction_func=None)¶
Post hoc calibration of classification models. Currently wraps CalibratedClassifierCV from sklearn and allows non-sklearn models to be calibrated.
- Parameters
num_classes – number of classes.
fit_mode – features or probs. If probs the fit and predict operate on the base models probability scores, useful when these are precomputed.
method – isotonic or sigmoid.
base_model_prediction_func – the function that takes in the input features and produces base model’s probability scores. This is ignored when operating in probs mode.
- fit(X, y)¶
Fits calibration model using the provided calibration set.
- Parameters
X – array-like of shape (n_samples, n_features) or (n_samples, n_classes). Features vectors of the training data or the probability scores from the base model.
y – array-like of shape (n_samples,) or (n_samples, n_targets) Target values
- Returns
self
- get_params(deep=True)¶
This method should not take any arguments and returns a dict of the __init__ parameters.
- predict(X)¶
Obtain calibrated predictions for the test points.
- Parameters
X – array-like of shape (n_samples, n_features) or (n_samples, n_classes). Features vectors of the training data or the probability scores from the base model.
- Returns
A namedtupe that holds
- y_pred: ndarray of shape (n_samples,)
Predicted labels of the test points.
- y_prob: ndarray of shape (n_samples, n_classes)
Predicted probability scores of the classes.
- Return type
namedtuple
UCC Recalibration¶
- class uq360.algorithms.ucc_recalibration.UCCRecalibration(base_model)¶
Recalibration a regression model to specified operating point using Uncertainty Characteristics Curve.
- Parameters
base_model – pretrained model to be recalibrated.
- fit(X, y)¶
Fit the Uncertainty Characteristics Curve.
- Parameters
X – array-like of shape (n_samples, n_features). Features vectors of the test points.
y – array-like of shape (n_samples,) or (n_samples, n_targets) Target values
- Returns
self
- get_params(deep=True)¶
This method should not take any arguments and returns a dict of the __init__ parameters.
- predict(X, missrate=0.05)¶
Generate prediction and uncertainty bounds for data X.
- Parameters
X – array-like of shape (n_samples, n_features). Features vectors of the test points.
missrate – desired missrate of the new operating point, set to 0.05 by default.
- Returns
A namedtupe that holds
- y_mean: ndarray of shape (n_samples, [n_output_dims])
Mean of predictive distribution of the test points.
- y_lower: ndarray of shape (n_samples, [n_output_dims])
Lower quantile of predictive distribution of the test points.
- y_upper: ndarray of shape (n_samples, [n_output_dims])
Upper quantile of predictive distribution of the test points.
- Return type
namedtuple