CRPS

The main theme of machine learning is that we (even implicitly) assume some underlying distribution of the data. When someone grabs a L2 loss (MSE) or L1 loss (MAE), they are implicitly assuming a Gaussian or Laplace distribution for their underlying model. In this sense, I mean that we have a model parameterized by $\theta$ such that $X \sim p_\theta(X)$, where $p_\theta$ defines the underlying distribution. One unique loss that I’ve seen that actually works quite well is the CRPS loss.

CRPS stands for Continuously ranked probability score. Some people refer to it as a form of “energy score”. The official definition is associated with a CDF $F$:

To derive the discrete version, we note that:

Another identity that we need is that:

What this means is that: \begin{align*} CRPS(F, y) &= \int_{-\infty}^\infty (E[\mathbb{1}(X \geq x)] - \mathbb{1}(y \geq x))^2 dx \\ &= \int_{-\infty}^\infty (E[\mathbb{1}(X \geq x)] - \mathbb{1}(y \geq x))(E[\mathbb{1}(X’ \geq x)] - \mathbb{1}(y \geq x)) dx \\ &= E_{X, X’} \int_{-\infty}^\infty (\mathbb{1}(X \geq x) - \mathbb{1}(y \geq x))(\mathbb{1}(X’ \geq x) - \mathbb{1}(y \geq x)) dx \\ &= E_{X, X’} \frac{1}{2} |X - y| + \frac{1}{2} |X’ - y| - |X - X’| \\ &= E|X - y| - E|X - X’| \end{align*}

The finite term estimator (unbiased) will then be given by:

With this, we can train “ensembles” instead of individual models. The advantage of this metric is that it is “proper” in that CRPS is uniquely minimized when the forecast CDF is the true CDF.