Metrics

Common evaluation metrics for time series forecasts.

metrics.acd(data_samples, data_truth, alpha=0.05, **kwargs) → float

The absolute difference between the coverage of the method and the target (0.95).

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • alpha (float): percentile to compute coverage difference
metrics.coverage(data_samples, data_truth, percentiles=None, **kwargs) → list

Computes coverage rates of the prediction interval.

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • percentiles (list): percentiles to calculate coverage for
metrics.mae(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e270954e0>

Computes mean absolute error (MAE)

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • agg: Aggregation function applied to sampled predictions (defaults to np.median).
metrics.mape(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095518>

Computes mean absolute percentage error (MAPE)

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • agg: Aggregation function applied to sampled predictions (defaults to np.median).
metrics.mase(data_samples, data_truth, data_insample, frequencies, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e270950f0>

Computes mean absolute scaled error (MASE) as in the M4 competition.

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • data_insample (np.array): In-sample time series data (n_timeseries, n_variables, n_timesteps).
  • frequencies (list): Frequencies to be used when calculating the naive forecast.
  • agg: Aggregation function applied to sampled predictions (defaults to np.median).
metrics.mse(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e270950b8>

Computes mean squared error (MSE)

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • agg: Aggregation function applied to sampled predictions (defaults to np.median).
metrics.msis(data_samples, data_truth, data_insample, frequencies, alpha=0.05, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095198>

Mean Scaled Interval Score (MSIS) as shown in the M4 competition.

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • data_insample (np.array): In-sample time series data (n_timeseries, n_variables, n_timesteps).
  • frequencies (list): Frequencies to be used when calculating the naive forecast.
  • alpha (float): Significance level.
metrics.pinball_loss(data_samples, data_truth, percentiles=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095160>

Computes pinball loss.

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • percentiles (list): Percentiles used to calculate coverage.
metrics.rmse(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095128>

Computes mean squared error (RMSE)

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • agg: Aggregation function applied to sampled predictions (defaults to np.median).
metrics.smape(data_samples, data_truth, agg=None, **kwargs) → <sphinx.ext.autodoc.importer._MockObject object at 0x7f7e27095080>

Computes symmetric mean absolute percentage error (SMAPE) on the mean

Arguments:
  • data_samples (np.array): Sampled predictions (n_samples, n_timeseries, n_variables, n_timesteps).
  • data_truth (np.array): Ground truth time series values (n_timeseries, n_variables, n_timesteps).
  • agg: Aggregation function applied to sampled predictions (defaults to np.median).