Regression Metrics

uq360.metrics.regression_metrics.auucc_gain(y_true, y_mean, y_lower, y_upper)

Computes the Area Under the Uncertainty Characteristics Curve (AUUCC) gain wrt to a null reference with constant band.

Parameters:
  • y_true – Ground truth

  • y_mean – predicted mean

  • y_lower – predicted lower bound

  • y_upper – predicted upper bound

Returns:

AUUCC gain

Return type:

float

uq360.metrics.regression_metrics.compute_regression_metrics(y_true, y_mean, y_lower, y_upper, option='all', nll_fn=None)

Computes the metrics specified in the option which can be string or a list of strings. Default option all computes the [“rmse”, “nll”, “auucc_gain”, “picp”, “mpiw”, “r2”] metrics.

Parameters:
  • y_true – Ground truth

  • y_mean – predicted mean

  • y_lower – predicted lower bound

  • y_upper – predicted upper bound

  • option – string or list of string contained the name of the metrics to be computed.

  • nll_fn – function that evaluates NLL, if None, then computes Gaussian NLL using y_mean and y_lower.

Returns:

dictionary containing the computed metrics.

Return type:

dict

uq360.metrics.regression_metrics.mpiw(y_lower, y_upper)

Mean Prediction Interval Width (MPIW). Computes the average width of the the prediction intervals. Measures the sharpness of intervals.

Parameters:
  • y_lower – predicted lower bound

  • y_upper – predicted upper bound

Returns:

the average width the prediction interval across samples.

Return type:

float

uq360.metrics.regression_metrics.negative_log_likelihood_Gaussian(y_true, y_mean, y_lower, y_upper)

Computes Gaussian negative_log_likelihood assuming symmetric band around the mean.

Parameters:
  • y_true – Ground truth

  • y_mean – predicted mean

  • y_lower – predicted lower bound

  • y_upper – predicted upper bound

Returns:

nll

Return type:

float

uq360.metrics.regression_metrics.picp(y_true, y_lower, y_upper)

Prediction Interval Coverage Probability (PICP). Computes the fraction of samples for which the grounds truth lies within predicted interval. Measures the prediction interval calibration for regression.

Parameters:
  • y_true – Ground truth

  • y_lower – predicted lower bound

  • y_upper – predicted upper bound

Returns:

the fraction of samples for which the grounds truth lies within predicted interval.

Return type:

float

uq360.metrics.regression_metrics.plot_picp_by_feature(x_test, y_test, y_test_pred_lower_total, y_test_pred_upper_total, num_bins=10, ax=None, figsize=None, dpi=None, xlims=None, ylims=None, xscale='linear', title=None, xlabel=None, ylabel=None)

Plot how prediction uncertainty varies across the entire range of a feature.

Parameters:
  • x_test – One dimensional ndarray. Feature column of the test dataset.

  • y_test – One dimensional ndarray. Ground truth label of the test dataset.

  • y_test_pred_lower_total – One dimensional ndarray. Lower bound of the total uncertainty range.

  • y_test_pred_upper_total – One dimensional ndarray. Upper bound of the total uncertainty range.

  • num_bins – int. Number of bins used to discritize x_test into equal-sample-sized bins.

  • ax – matplotlib.axes.Axes or None, optional (default=None). Target axes instance. If None, new figure and axes will be created.

  • figsize – tuple of 2 elements or None, optional (default=None). Figure size.

  • dpi – int or None, optional (default=None). Resolution of the figure.

  • xlims – tuple of 2 elements or None, optional (default=None). Tuple passed to ax.xlim().

  • ylims – tuple of 2 elements or None, optional (default=None). Tuple passed to ax.ylim().

  • xscale – Passed to ax.set_xscale().

  • title – string or None, optional Axes title. If None, title is disabled.

  • xlabel – string or None, optional X-axis title label. If None, title is disabled.

  • ylabel – string or None, optional Y-axis title label. If None, title is disabled.

Returns:

ax : The plot with PICP scores binned by a feature.

Return type:

matplotlib.axes.Axes

uq360.metrics.regression_metrics.plot_uncertainty_by_feature(x_test, y_test_pred_mean, y_test_pred_lower_total, y_test_pred_upper_total, y_test_pred_lower_epistemic=None, y_test_pred_upper_epistemic=None, ax=None, figsize=None, dpi=None, xlims=None, xscale='linear', title=None, xlabel=None, ylabel=None)

Plot how prediction uncertainty varies across the entire range of a feature.

Parameters:
  • x_test – one dimensional ndarray. Feature column of the test dataset.

  • y_test_pred_mean – One dimensional ndarray. Model prediction for the test dataset.

  • y_test_pred_lower_total – One dimensional ndarray. Lower bound of the total uncertainty range.

  • y_test_pred_upper_total – One dimensional ndarray. Upper bound of the total uncertainty range.

  • y_test_pred_lower_epistemic – One dimensional ndarray. Lower bound of the epistemic uncertainty range.

  • y_test_pred_upper_epistemic – One dimensional ndarray. Upper bound of the epistemic uncertainty range.

  • ax – matplotlib.axes.Axes or None, optional (default=None). Target axes instance. If None, new figure and axes will be created.

  • figsize – tuple of 2 elements or None, optional (default=None). Figure size.

  • dpi – int or None, optional (default=None). Resolution of the figure.

  • xlims – tuple of 2 elements or None, optional (default=None). Tuple passed to ax.xlim().

  • xscale – Passed to ax.set_xscale().

  • title – string or None, optional Axes title. If None, title is disabled.

  • xlabel – string or None, optional X-axis title label. If None, title is disabled.

  • ylabel – string or None, optional Y-axis title label. If None, title is disabled.

Returns:

ax : The plot with model’s uncertainty binned by a feature.

Return type:

matplotlib.axes.Axes

uq360.metrics.regression_metrics.plot_uncertainty_distribution(dist, show_quantile_dots=False, qd_sample=20, qd_bins=7, ax=None, figsize=None, dpi=None, title='Predicted Distribution', xlims=None, xlabel='Prediction', ylabel='Density', **kwargs)

Plot the uncertainty distribution for a single distribution.

Parameters:
  • dist – scipy.stats._continuous_distns. A scipy distribution object.

  • show_quantile_dots – boolean. Whether to show quantil dots on top of the density plot.

  • qd_sample – int. Number of dots for the quantile dot plot.

  • qd_bins – int. Number of bins for the quantile dot plot.

  • ax – matplotlib.axes.Axes or None, optional (default=None). Target axes instance. If None, new figure and axes will be created.

  • figsize – tuple of 2 elements or None, optional (default=None). Figure size.

  • dpi – int or None, optional (default=None). Resolution of the figure.

  • title – string or None, optional (default=Prediction Distribution) Axes title. If None, title is disabled.

  • xlims – tuple of 2 elements or None, optional (default=None). Tuple passed to ax.xlim().

  • xlabel – string or None, optional (default=Prediction) X-axis title label. If None, title is disabled.

  • ylabel – string or None, optional (default=Density) Y-axis title label. If None, title is disabled.

Returns:

ax : The plot with prediction distribution.

Return type:

matplotlib.axes.Axes

Uncertainty Characteristics Curve

class uq360.metrics.uncertainty_characteristics_curve.UncertaintyCharacteristicsCurve(normalize=True, precompute_bias_data=True)

Class with main functions of the Uncertainty Characteristics Curve (UCC).

Parameters:
  • normalize – set initial axes normalization flag (can be changed via set_coordinates())

  • precompute_bias_data – if True, fit() will compute statistics necessary to generate bias-based UCCs (in addition to the scale-based ones). Skipping this precomputation may speed up the fit() call if bias-based UCC is not needed.

fit(X, gt)

Calculates internal arrays necessary for other methods (plotting, auc, cost minimization). Re-entrant.

Parameters:
  • X – [numsamples, 3] numpy matrix, or list of numpy matrices. Col 1: predicted values Col 2: lower band (deviate) wrt predicted value (always positive) Col 3: upper band wrt predicted value (always positive) If list is provided, all methods will output corresponding metrics as lists as well!

  • gt – Ground truth array (i.e.,the ‘actual’ values corresponding to predictions in X

Returns:

self

get_AUUCC(vary_bias=False, aucfct='trapz', partial_x=None, partial_y=None)

returns approximate area under the curve on current coordinates, for each component.

Parameters:
  • vary_bias – False == varies scale, True == varies bias

  • aucfct – specifies AUC integrator (can be “trapz”, “simps”)

  • partial_x – tuple (x_min, x_max) defining interval on x to calc a a partial AUC. The interval bounds refer to axes as visualized (ie. potentially normed)

  • partial_y – tuple (y_min, y_max) defining interval on y to calc a a partial AUC. partial_x must be None.

Returns:

list of floats with AUUCCs for each input component, or a single float, if there is only 1 component.

get_OP(scale=1.0, bias=0.0)

Returns all Operating Points for original input data, on coordinates currently set up, given a scale/bias.

Parameters:
  • scale

  • bias

Returns:

list of tuples (x point, y point, unit of x, unit of y) or a single tuple if there is only 1 component.

get_specific_operating_point(req_x_axis_value=None, req_y_axis_value=None, req_critical_value=None, vary_bias=False)

Finds corresponding operating point on the current UCC, given a point on either x or y axis. Returns a list of recipes how to achieve the point (x,y), for each component. If there is only one component, returns a single recipe dict.

Parameters:
  • req_x_axis_value – requested x value on UCC (normalization status is taken from current display)

  • req_y_axis_value – requested y value on UCC (normalization status is taken from current display)

  • vary_bias – set to True when referring to bias-induced UCC (scale UCC default)

Returns:

list of dicts (recipes), or a single dict

minimize_cost(x_axis_cost=0.5, y_axis_cost=0.5, augment_cost_by_normfactor=True, search=('scale', 'bias'))

Find minima of a linear cost function for each component. Cost function C = x_axis_cost * x_axis_value + y_axis_cost * y_axis_value. A minimum can occur in the scale-based or bias-based UCC (this can be constrained by the ‘search’ arg). The function returns a ‘recipe’ how to achieve the corresponding minimum, for each component.

Parameters:
  • x_axis_cost – weight of one unit on x_axis

  • y_axis_cost – weight of one unit on y_axis

  • augment_cost_by_normfactor – when False, the cost multipliers will apply as is. If True, they will be pre-normed by the corresponding axis norm (where applicable), to account for range differences between axes.

  • search – list of types over which minimization is to be performed, valid elements are ‘scale’ and ‘bias’.

Returns:

list of dicts - one per component, or a single dict, if there is only one component. Dict keys are - ‘operation’: can be ‘bias’ (additive) or ‘scale’ (multiplicative), ‘modvalue’: value to multiply by or to add to error bars to achieve the minimum, ‘new_x’/’new_y’: new coordinates (operating point) with that minimum, ‘cost’: new cost at minimum point, ‘original_cost’: original cost (original operating point).

plot_UCC(titlestr='', syslabel='model', outfn=None, vary_bias=False, markers=None, xlim=None, ylim=None, **kwargs)

Will plot/display the UCC based on current data and coordinates. Multiple curves will be shown if there are multiple data components (via fit())

Parameters:
  • titlestr – Plot title string

  • syslabel – list is label strings to appear in the plot legend. Can be single, if one component.

  • outfn – base name of an image file to be created (will append .png before creating)

  • vary_bias – True will switch to varying additive bias (default is multiplicative scale)

  • markers – None or a list of marker styles to be used for each curve. List must be same or longer than number of components. Markers can be one among these [‘o’, ‘s’, ‘v’, ‘D’, ‘+’].

  • xlim – tuples or lists of specifying the range for the x axis, or None (auto)

  • ylim – tuples or lists of specifying the range for the y axis, or None (auto)

  • **kwargs – Additional arguments passed to the main plot call.

Returns:

list of areas under the curve (or single area, if one data component) list of operating points (or single op): format of an op is tuple (xaxis value, yaxis value, xunit, yunit)

set_coordinates(x_axis_name=None, y_axis_name=None, normalize=None)

Assigns user-specified type to the axes and normalization behavior (sticky).

Parameters:
  • x_axis_name – None-> unchanged, or name from self.axes_name2idx

  • y_axis_name – ditto

  • normalize – True/False will activate/deactivate norming for specified axes. Behavior for Axes_name that are None will not be changed. Value None will leave norm status unchanged. Note, axis==’missrate’ will never get normalized, even with normalize == True

Returns:

none

set_std_unit(std_unit=None)

Sets the UCC’s unit to be used when displaying normalized axes.

Parameters:

std_unit – if None, the unit will be calculated as stddev of the ground truth data (ValueError raised if data has not been set at this point) or set to the user-specified value.

Returns: