Code and summary results for "A Training-Free Overlap-Disagreement Signal for Selective Prediction and Conformal Intervals in Time-Series Forecasting"
Description
Research hypothesis. A single, already-trained deterministic time-series forecaster contains a free uncertainty signal. Evaluating it on densely overlapping lookback windows yields several forecasts for the same future timestep; their disagreement ("overlap-disagreement") tracks forecast error without extra training or labels. We test whether this signal supports (i) selective prediction (deferring high-disagreement windows) and (ii) conformal prediction (as a local nonconformity scale under online calibration), compared with deep-ensemble variance, Monte Carlo dropout variance, and split-conformal baselines. What the data is and how it was produced. No new data was collected. Experiments use six public long-term forecasting benchmarks (ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity), three forecasters (DLinear, PatchTST, and a univariate channel-independent iTransformer configuration), four horizons (96, 192, 336, 720), and five random seeds. This deposit contains the Python code (MIT license), unit tests, and summary results of these runs. The benchmark datasets themselves are not included; sources are listed in README.md. Contents. results_summary/exp-sp-h* and exp-cp-h*: per-horizon metrics.json for selective prediction (risk-coverage metrics such as AURC) and conformal prediction (coverage, interval width, Winkler score). Two summary tables, an Electricity interval-width seed sign test, and per-window selective-prediction arrays (exp-sp-h*/arrays) are also included. Conformal-prediction per-window arrays (about 1.6 GB) are not included. Notable findings. The signal gives the best selective-prediction ranking at short-to-medium horizons and remains favored in about three quarters of comparisons at horizon 720. Its conformal-interval width advantage narrows with horizon and reverses on Electricity (the highest-dimensional benchmark) at horizon 720, where intervals are wider than the constant-scale baseline. How to use. All results can be read directly from the JSON files. Tables and figures in the accompanying manuscript derive from metrics.json (and the selective-prediction arrays for the risk-coverage figure). The code in topic/A/src regenerates the experiments; see README.md for data paths and the PWW_RESULT_DIR output convention.