Compute observed coverage of posterior credible intervals.

Source: R/calculate.R

Uses ranks to compute coverage and surrounding uncertainty of posterior credible intervals. The uncertainty is only approximate (treating coverage for each interval as a set of independent Bernoulli trials, while in fact they are not independent), so for making claims on presence/ absence of detectable discrepancies we strongly recommend using plot_ecdf() or plot_ecdf_diff(). The uncertainty about the coverage can however be useful for guiding decisions on whether more SBC steps should be performed (i.e. whether we can rule out that the coverage of the given backend differs too much for our purposes from the optimal value).

empirical_coverage(stats, width, prob = 0.95, interval_type = "central")

Arguments

stats

a data.frame of rank statistics (e.g. as returned in the $stats component of SBC_results), at minimum should have at least variable, rank and max_rank columns)

width

a vector of values between 0 and 1 representing widths of credible intervals for which we compute coverage.

prob

determines width of the uncertainty interval around the observed coverage

inteval_type

"central" to show coverage of central credible intervals or "leftmost" to show coverage of leftmost credible intervals (i.e. the observed CDF).

Value

A data.frame with columns variable, width (width of the interval as given in the width parameter), width_represented the closest width that can be represented by the ranks in the input (any discrepancy needs to be judged against this rather than width), estimate - observed coverage for the interval, ci_low, ci_high the uncertainty interval around estimate (width of the interval is given by the prob argument).

Details

Note that while coverage of central posterior intervals (with the default type = "central") is often of the biggest practical interest, perfect calibration of central intervals still leaves space for substantial problems with the model (e.g. if the posterior 25% - 50% intervals contain 50% of the true values and the posterior 50% - 75% interval never contains the true value, the central 50% interval still has the ideal 50% coverage), so investigating central intervals should always be accompanied by checks with plot_ecdf() or plot_ecdf_diff() or by using type = "leftmost", because if all leftmost credible intervals are well calibrated, then all intervals are well calibrated.

See also

plot_coverage()