Regression Metrics
- uq360.metrics.regression_metrics.auucc_gain(y_true, y_mean, y_lower, y_upper)
Computes the Area Under the Uncertainty Characteristics Curve (AUUCC) gain wrt to a null reference with constant band.
- Parameters:
y_true – Ground truth
y_mean – predicted mean
y_lower – predicted lower bound
y_upper – predicted upper bound
- Returns:
AUUCC gain
- Return type:
float
- uq360.metrics.regression_metrics.compute_regression_metrics(y_true, y_mean, y_lower, y_upper, option='all', nll_fn=None)
Computes the metrics specified in the option which can be string or a list of strings. Default option all computes the [“rmse”, “nll”, “auucc_gain”, “picp”, “mpiw”, “r2”] metrics.
- Parameters:
y_true – Ground truth
y_mean – predicted mean
y_lower – predicted lower bound
y_upper – predicted upper bound
option – string or list of string contained the name of the metrics to be computed.
nll_fn – function that evaluates NLL, if None, then computes Gaussian NLL using y_mean and y_lower.
- Returns:
dictionary containing the computed metrics.
- Return type:
dict
- uq360.metrics.regression_metrics.mpiw(y_lower, y_upper)
Mean Prediction Interval Width (MPIW). Computes the average width of the the prediction intervals. Measures the sharpness of intervals.
- Parameters:
y_lower – predicted lower bound
y_upper – predicted upper bound
- Returns:
the average width the prediction interval across samples.
- Return type:
float
- uq360.metrics.regression_metrics.negative_log_likelihood_Gaussian(y_true, y_mean, y_lower, y_upper)
Computes Gaussian negative_log_likelihood assuming symmetric band around the mean.
- Parameters:
y_true – Ground truth
y_mean – predicted mean
y_lower – predicted lower bound
y_upper – predicted upper bound
- Returns:
nll
- Return type:
float
- uq360.metrics.regression_metrics.picp(y_true, y_lower, y_upper)
Prediction Interval Coverage Probability (PICP). Computes the fraction of samples for which the grounds truth lies within predicted interval. Measures the prediction interval calibration for regression.
- Parameters:
y_true – Ground truth
y_lower – predicted lower bound
y_upper – predicted upper bound
- Returns:
the fraction of samples for which the grounds truth lies within predicted interval.
- Return type:
float
- uq360.metrics.regression_metrics.plot_picp_by_feature(x_test, y_test, y_test_pred_lower_total, y_test_pred_upper_total, num_bins=10, ax=None, figsize=None, dpi=None, xlims=None, ylims=None, xscale='linear', title=None, xlabel=None, ylabel=None)
Plot how prediction uncertainty varies across the entire range of a feature.
- Parameters:
x_test – One dimensional ndarray. Feature column of the test dataset.
y_test – One dimensional ndarray. Ground truth label of the test dataset.
y_test_pred_lower_total – One dimensional ndarray. Lower bound of the total uncertainty range.
y_test_pred_upper_total – One dimensional ndarray. Upper bound of the total uncertainty range.
num_bins – int. Number of bins used to discritize x_test into equal-sample-sized bins.
ax – matplotlib.axes.Axes or None, optional (default=None). Target axes instance. If None, new figure and axes will be created.
figsize – tuple of 2 elements or None, optional (default=None). Figure size.
dpi – int or None, optional (default=None). Resolution of the figure.
xlims – tuple of 2 elements or None, optional (default=None). Tuple passed to
ax.xlim()
.ylims – tuple of 2 elements or None, optional (default=None). Tuple passed to
ax.ylim()
.xscale – Passed to
ax.set_xscale()
.title – string or None, optional Axes title. If None, title is disabled.
xlabel – string or None, optional X-axis title label. If None, title is disabled.
ylabel – string or None, optional Y-axis title label. If None, title is disabled.
- Returns:
ax : The plot with PICP scores binned by a feature.
- Return type:
matplotlib.axes.Axes
- uq360.metrics.regression_metrics.plot_uncertainty_by_feature(x_test, y_test_pred_mean, y_test_pred_lower_total, y_test_pred_upper_total, y_test_pred_lower_epistemic=None, y_test_pred_upper_epistemic=None, ax=None, figsize=None, dpi=None, xlims=None, xscale='linear', title=None, xlabel=None, ylabel=None)
Plot how prediction uncertainty varies across the entire range of a feature.
- Parameters:
x_test – one dimensional ndarray. Feature column of the test dataset.
y_test_pred_mean – One dimensional ndarray. Model prediction for the test dataset.
y_test_pred_lower_total – One dimensional ndarray. Lower bound of the total uncertainty range.
y_test_pred_upper_total – One dimensional ndarray. Upper bound of the total uncertainty range.
y_test_pred_lower_epistemic – One dimensional ndarray. Lower bound of the epistemic uncertainty range.
y_test_pred_upper_epistemic – One dimensional ndarray. Upper bound of the epistemic uncertainty range.
ax – matplotlib.axes.Axes or None, optional (default=None). Target axes instance. If None, new figure and axes will be created.
figsize – tuple of 2 elements or None, optional (default=None). Figure size.
dpi – int or None, optional (default=None). Resolution of the figure.
xlims – tuple of 2 elements or None, optional (default=None). Tuple passed to
ax.xlim()
.xscale – Passed to
ax.set_xscale()
.title – string or None, optional Axes title. If None, title is disabled.
xlabel – string or None, optional X-axis title label. If None, title is disabled.
ylabel – string or None, optional Y-axis title label. If None, title is disabled.
- Returns:
ax : The plot with model’s uncertainty binned by a feature.
- Return type:
matplotlib.axes.Axes
- uq360.metrics.regression_metrics.plot_uncertainty_distribution(dist, show_quantile_dots=False, qd_sample=20, qd_bins=7, ax=None, figsize=None, dpi=None, title='Predicted Distribution', xlims=None, xlabel='Prediction', ylabel='Density', **kwargs)
Plot the uncertainty distribution for a single distribution.
- Parameters:
dist – scipy.stats._continuous_distns. A scipy distribution object.
show_quantile_dots – boolean. Whether to show quantil dots on top of the density plot.
qd_sample – int. Number of dots for the quantile dot plot.
qd_bins – int. Number of bins for the quantile dot plot.
ax – matplotlib.axes.Axes or None, optional (default=None). Target axes instance. If None, new figure and axes will be created.
figsize – tuple of 2 elements or None, optional (default=None). Figure size.
dpi – int or None, optional (default=None). Resolution of the figure.
title – string or None, optional (default=Prediction Distribution) Axes title. If None, title is disabled.
xlims – tuple of 2 elements or None, optional (default=None). Tuple passed to
ax.xlim()
.xlabel – string or None, optional (default=Prediction) X-axis title label. If None, title is disabled.
ylabel – string or None, optional (default=Density) Y-axis title label. If None, title is disabled.
- Returns:
ax : The plot with prediction distribution.
- Return type:
matplotlib.axes.Axes
Uncertainty Characteristics Curve
- class uq360.metrics.uncertainty_characteristics_curve.UncertaintyCharacteristicsCurve(normalize=True, precompute_bias_data=True)
Class with main functions of the Uncertainty Characteristics Curve (UCC).
- Parameters:
normalize – set initial axes normalization flag (can be changed via set_coordinates())
precompute_bias_data – if True, fit() will compute statistics necessary to generate bias-based UCCs (in addition to the scale-based ones). Skipping this precomputation may speed up the fit() call if bias-based UCC is not needed.
- fit(X, gt)
Calculates internal arrays necessary for other methods (plotting, auc, cost minimization). Re-entrant.
- Parameters:
X – [numsamples, 3] numpy matrix, or list of numpy matrices. Col 1: predicted values Col 2: lower band (deviate) wrt predicted value (always positive) Col 3: upper band wrt predicted value (always positive) If list is provided, all methods will output corresponding metrics as lists as well!
gt – Ground truth array (i.e.,the ‘actual’ values corresponding to predictions in X
- Returns:
self
- get_AUUCC(vary_bias=False, aucfct='trapz', partial_x=None, partial_y=None)
returns approximate area under the curve on current coordinates, for each component.
- Parameters:
vary_bias – False == varies scale, True == varies bias
aucfct – specifies AUC integrator (can be “trapz”, “simps”)
partial_x – tuple (x_min, x_max) defining interval on x to calc a a partial AUC. The interval bounds refer to axes as visualized (ie. potentially normed)
partial_y – tuple (y_min, y_max) defining interval on y to calc a a partial AUC. partial_x must be None.
- Returns:
list of floats with AUUCCs for each input component, or a single float, if there is only 1 component.
- get_OP(scale=1.0, bias=0.0)
Returns all Operating Points for original input data, on coordinates currently set up, given a scale/bias.
- Parameters:
scale –
bias –
- Returns:
list of tuples (x point, y point, unit of x, unit of y) or a single tuple if there is only 1 component.
- get_specific_operating_point(req_x_axis_value=None, req_y_axis_value=None, req_critical_value=None, vary_bias=False)
Finds corresponding operating point on the current UCC, given a point on either x or y axis. Returns a list of recipes how to achieve the point (x,y), for each component. If there is only one component, returns a single recipe dict.
- Parameters:
req_x_axis_value – requested x value on UCC (normalization status is taken from current display)
req_y_axis_value – requested y value on UCC (normalization status is taken from current display)
vary_bias – set to True when referring to bias-induced UCC (scale UCC default)
- Returns:
list of dicts (recipes), or a single dict
- minimize_cost(x_axis_cost=0.5, y_axis_cost=0.5, augment_cost_by_normfactor=True, search=('scale', 'bias'))
Find minima of a linear cost function for each component. Cost function C = x_axis_cost * x_axis_value + y_axis_cost * y_axis_value. A minimum can occur in the scale-based or bias-based UCC (this can be constrained by the ‘search’ arg). The function returns a ‘recipe’ how to achieve the corresponding minimum, for each component.
- Parameters:
x_axis_cost – weight of one unit on x_axis
y_axis_cost – weight of one unit on y_axis
augment_cost_by_normfactor – when False, the cost multipliers will apply as is. If True, they will be pre-normed by the corresponding axis norm (where applicable), to account for range differences between axes.
search – list of types over which minimization is to be performed, valid elements are ‘scale’ and ‘bias’.
- Returns:
list of dicts - one per component, or a single dict, if there is only one component. Dict keys are - ‘operation’: can be ‘bias’ (additive) or ‘scale’ (multiplicative), ‘modvalue’: value to multiply by or to add to error bars to achieve the minimum, ‘new_x’/’new_y’: new coordinates (operating point) with that minimum, ‘cost’: new cost at minimum point, ‘original_cost’: original cost (original operating point).
- plot_UCC(titlestr='', syslabel='model', outfn=None, vary_bias=False, markers=None, xlim=None, ylim=None, **kwargs)
Will plot/display the UCC based on current data and coordinates. Multiple curves will be shown if there are multiple data components (via fit())
- Parameters:
titlestr – Plot title string
syslabel – list is label strings to appear in the plot legend. Can be single, if one component.
outfn – base name of an image file to be created (will append .png before creating)
vary_bias – True will switch to varying additive bias (default is multiplicative scale)
markers – None or a list of marker styles to be used for each curve. List must be same or longer than number of components. Markers can be one among these [‘o’, ‘s’, ‘v’, ‘D’, ‘+’].
xlim – tuples or lists of specifying the range for the x axis, or None (auto)
ylim – tuples or lists of specifying the range for the y axis, or None (auto)
**kwargs – Additional arguments passed to the main plot call.
- Returns:
list of areas under the curve (or single area, if one data component) list of operating points (or single op): format of an op is tuple (xaxis value, yaxis value, xunit, yunit)
- set_coordinates(x_axis_name=None, y_axis_name=None, normalize=None)
Assigns user-specified type to the axes and normalization behavior (sticky).
- Parameters:
x_axis_name – None-> unchanged, or name from self.axes_name2idx
y_axis_name – ditto
normalize – True/False will activate/deactivate norming for specified axes. Behavior for Axes_name that are None will not be changed. Value None will leave norm status unchanged. Note, axis==’missrate’ will never get normalized, even with normalize == True
- Returns:
none
- set_std_unit(std_unit=None)
Sets the UCC’s unit to be used when displaying normalized axes.
- Parameters:
std_unit – if None, the unit will be calculated as stddev of the ground truth data (ValueError raised if data has not been set at this point) or set to the user-specified value.
- Returns: