Metrics¶
Common evaluation metrics for time series forecasts.
-
metrics.
acd
(data_samples, data_truth, alpha=0.05, **kwargs) → float¶ The absolute difference between the coverage of the method and the target (0.95).
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - alpha (float): percentile to compute coverage difference
- data_samples (
-
metrics.
coverage
(data_samples, data_truth, percentiles=None, **kwargs) → list¶ Computes coverage rates of the prediction interval.
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - percentiles (list): percentiles to calculate coverage for
- data_samples (
-
metrics.
mae
(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e270954e0>¶ Computes mean absolute error (MAE)
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - agg: Aggregation function applied to sampled predictions (defaults to
np.median
).
- data_samples (
-
metrics.
mape
(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095518>¶ Computes mean absolute percentage error (MAPE)
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - agg: Aggregation function applied to sampled predictions (defaults to
np.median
).
- data_samples (
-
metrics.
mase
(data_samples, data_truth, data_insample, frequencies, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e270950f0>¶ Computes mean absolute scaled error (MASE) as in the M4 competition.
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - data_insample (
np.array
): In-sample time series data (n_timeseries, n_variables, n_timesteps). - frequencies (list): Frequencies to be used when calculating the naive forecast.
- agg: Aggregation function applied to sampled predictions (defaults to
np.median
).
- data_samples (
-
metrics.
mse
(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e270950b8>¶ Computes mean squared error (MSE)
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - agg: Aggregation function applied to sampled predictions (defaults to
np.median
).
- data_samples (
-
metrics.
msis
(data_samples, data_truth, data_insample, frequencies, alpha=0.05, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095198>¶ Mean Scaled Interval Score (MSIS) as shown in the M4 competition.
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - data_insample (
np.array
): In-sample time series data (n_timeseries, n_variables, n_timesteps). - frequencies (list): Frequencies to be used when calculating the naive forecast.
- alpha (float): Significance level.
- data_samples (
-
metrics.
pinball_loss
(data_samples, data_truth, percentiles=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095160>¶ Computes pinball loss.
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - percentiles (list): Percentiles used to calculate coverage.
- data_samples (
-
metrics.
rmse
(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095128>¶ Computes mean squared error (RMSE)
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - agg: Aggregation function applied to sampled predictions (defaults to
np.median
).
- data_samples (
-
metrics.
smape
(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095080>¶ Computes symmetric mean absolute percentage error (SMAPE) on the mean
- Arguments:
- data_samples (
np.array
): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps). - data_truth (
np.array
): Ground truth time series values (n_timeseries, n_variables, n_timesteps). - agg: Aggregation function applied to sampled predictions (defaults to
np.median
).
- data_samples (