solarforecastarbiter.metrics.probabilistic.quantile_score

solarforecastarbiter.metrics.probabilistic.quantile_score(obs, fx, fx_prob)[source]

Quantile Score (QS).

\[\text{QS} = \frac{1}{n} \sum_{i=1}^n (fx_i - obs_i) * (p - 1\{obs_i > fx_i\})\]

where \(n\) is the number of forecasts, \(obs_i\) is an observation, \(fx_i\) is a forecast, \(1\{obs_i > fx_i\}\) is an indicator function (1 if \(obs_i > fx_i\), 0 otherwise) and \(p\) is the probability that \(obs_i <= fx_i\). [1] [2]

If \(obs > fx\), then we have:

\[\begin{split}(fx - obs) < 0 \\ (p - 1\{obs > fx\}) = (p - 1) <= 0 \\ (fx - obs) * (p - 1) >= 0\end{split}\]

If instead \(obs < fx\), then we have:

\[\begin{split}(fx - obs) > 0 \\ (p - 1\{obs > fx\}) = (p - 0) >= 0 \\ (fx - obs) * p >= 0\end{split}\]

Therefore, the quantile score is non-negative regardless of the obs and fx.

Parameters:
  • obs ((n,) array_like) – Observations (physical unit).
  • fx ((n,) array_like) – Forecasts (physical units) of the right-hand-side of a CDF interval, e.g., fx = 10 MW is interpreted as forecasting <= 10 MW.
  • fx_prob ((n,) array_like) – Probability [%] associated with the forecasts.
Returns:

qs (float) – The Quantile Score, with the same units as the observations.

Notes

Quantile score is meant to be computed for a single probability of \(n\) samples.

Examples

>>> obs = 100     # observation [MW]
>>> fx = 80       # forecast [MW]
>>> fx_prob = 60  # probability [%]
>>> quantile_score(obs, fx, fx_prob)   # score [MW]
8.0

References

[1]Koenker and Bassett, Jr. (1978) “Regression Quantiles”, Econometrica 46 (1), pp. 33-50. doi: 10.2307/1913643
[2]Wilks (2020) “Forecast Verification”. In “Statistical Methods in the Atmospheric Sciences” (3rd edition). Academic Press. ISBN: 9780123850225