Public data only · no customer results yet
Evidence and benchmarks
What has been measured, including where Udgass failed.
Every test below was written down before its data was read, scored once, and is reported whatever it showed. They use public datasets. None of them shows that Udgass improves a real company’s results: that has not been measured yet.
Read this first
Four kinds of evidence, and what each can show
| Customer results | None. No company has run Udgass on its own data under an agreed protocol. Until one does, there is no evidence of savings, fewer stock-outs or better decisions in a real business. |
|---|---|
| Public held-out tests | A preregistered test on public data the methods were never developed on. Shows how a method compares with simple ones on that dataset and that period. Says nothing directly about your business. |
| Public development evidence | Results on data that was looked at while building the methods (VN1 after 26 Oct 2026). Used to decide what to test next. Optimistic by construction; never used for a claim. |
| Simulations | Inventory results from replaying a stock policy on sales history. They depend on the assumptions stated with them (lead time, lost sales, starting stock) and are always labelled as simulated. |
Scorecard
Every preregistered held-out test on public data
“Not shown” means the written rule required an improvement the data did not support. The preregistration fingerprints (SHA-256) are listed under reproducibility.
| Test | Question asked in advance | Result |
|---|---|---|
| UCI Online Retail II 24 Oct 2026 · 4,960 products · weekly | Does one fixed method (simple exponential smoothing) beat Udgass’s per-product choice, with 80% ranges covering 75–85%? | Fail. Slightly lower error (53.9% against 55.1%), but ranges covered 63.1%, and the current engine’s ranges covered only 69.5%. |
| VN1, test A 26 Oct 2026 · 15,053 series · weekly | Does a cross-learning adjustment beat the default by at least 1 point of VN1 score? | Pass (71.0% against 76.3%), but both were far worse than repeating last week’s sales (50.7%). |
| VN1, test B | Do calibrated 80% ranges cover 75–85% in every segment, with an interval score no worse than the default? | Fail. Interval score much better (3.19 against 104.7), but one segment covered 67.5%. |
| M4 Weekly, R1–R3 2 Nov 2026 · 359 series · mostly finance and macro | Are Udgass lite, LightGBM and the 0.5.0 default better than naive (mean MASE at least 2% lower, 95% interval below zero)? | Lite not shown. LightGBM not shown. Default better (6.6% lower). Two standard methods (ETS, Theta) did better than all three. |
| Car parts, R1–R5 2 Nov 2026 · 2,509 series · monthly, intermittent | Are SES and Theta better than naive and than the mean; which of the two is better? | Theta worse than naive. SES no difference shown against naive. Both better than the mean. The SES-vs-Theta rule had a flaw, described below. |
Test 1 · UCI Online Retail II · Fail
Ranges that were too narrow
One UK online retailer of gift items selling mostly to wholesale customers, December 2009 to December 2011. Cancellations, non-product codes and every customer field were dropped before use; weekly totals per product. Test origins July to November 2011.
| Measure | Udgass engine | Candidate (SES) | 4-week average | Rule |
|---|---|---|---|---|
| Error (WAPE) | 55.1% | 53.9% | 56.9% | Pass |
| 80% ranges containing the actual | 69.5% | 63.1% | — | Fail (needs 75–85%) |
| “Mostly zero” products (WAPE) | 261.8% | 287.3% | — | Fail (25.5 points worse) |
| Bias (negative = under-forecast) | −7.1% | −12.6% | — | Reported only |
What it showed about Udgass itself: on this lumpy wholesale demand, the engine’s 80% ranges held the actual only 69.5% of the time and it under-forecast by 7%. The assessment report now warns when ranges fall short like this, and later work on calibrated ranges came from it. Limits: one retailer, about five test dates, a pre-Christmas rise, and under two years of history, so no yearly seasonality could be learned.
Test 2 · VN1 · Pass on its rule, fail against naive
Far behind a simple method after the holiday peak
The VN1 Forecasting Accuracy Challenge: 15,053 weekly product-warehouse series. 183 weeks of history, then 13 test weeks (8 January to 1 April 2024) kept in a sealed folder during development. Scored with the competition’s own scorer.
| Forecast | VN1 score (lower is better) | What it is |
|---|---|---|
| Competition winner | 46.4% | Official result |
| Naive: last week repeated | 50.7% | Official benchmark |
| Udgass candidate (cross-learning) | 71.0% | Held-out at the time |
| Udgass 0.5.0 default | 76.3% | Held-out at the time |
| 12-week moving average | 80.5% | Official benchmark |
The written rule compared the candidate with the default, and it passed. The finding that matters more is that both were 20 to 26 points worse than repeating last week. The test weeks start just after the holiday peak, and Udgass’s level methods kept forecasting near the holiday level (bias +15%) while naive picked up the January drop at once.
After the test window was spent
Everything on VN1 since 26 October 2026 is development evidence, because the test weeks have been seen. Two results guided what was tested next:
- Over ten 13-week origins (October 2021 to January 2024), Udgass lite and a LightGBM model were about 6 points better than naive, at 9 of 10 origins. The 0.5.0 default was about 6 points worse, so the January failure was not specific to one window. On the high-volume, regular series that carry 65% of units, lite was slightly worse than naive (43.6% against 41.5%).
- Ranges calibrated by demand segment brought every segment within 3.5 points of its 80% target, at the cost of ranges about a fifth wider. A single pooled calibration looked right overall (81%) while the high-volume series held only 65–69%.
These rolling-origin figures were chosen after seeing the data and are optimistic. They are why the next fresh test (M4 Weekly) compared lite and LightGBM with naive. A later post-hoc run on the spent VN1 window is not a held-out result and is not shown as a ranking. Test B’s full per-segment table was not recorded; the summary above is the recorded result.
Test 3 · M4 Weekly · Two of three not shown
The VN1 ranking did not carry over
The first test on data never used in Udgass development: 359 weekly series from the M4 competition (Finance 164, Micro 112, Macro 41, Demographic 24, Other 12, Industry 6), 13 weeks ahead. Mostly business and economic series, not product demand. Forecasts were computed from training data and fingerprinted before the test file was opened.
| Model | Mean MASE (primary) | sMAPE | Series better than naive |
|---|---|---|---|
| ETS (statsforecast) | 2.420 | 8.62 | 57% |
| Theta (statsforecast) | 2.505 | 7.94 | 53% |
| Udgass 0.5.0 default | 2.594 | 9.25 | 56% |
| Naive | 2.777 | 9.16 | — |
| 4-week average | 2.923 | 9.83 | 47% |
| Udgass lite | 2.930 | 9.36 | 47% |
| LightGBM global | 3.347 | 7.68 | 56% |
| 12-week average | 3.798 | 11.94 | 33% |
| Seasonal naive (52 weeks) | 9.578 | 14.52 | 25% |
- R1 Udgass lite against naive: not shown. MASE 5.5% higher; difference +0.15, 95% interval −0.07 to +0.38.
- R2 LightGBM against naive: not shown. 20.5% higher; +0.57, interval −0.15 to +1.54.
- R3 Udgass 0.5.0 default against naive: better. 6.6% lower; −0.18, interval −0.36 to −0.03.
- Reported without a rule: Theta and ETS were better than naive; lite was worse than Theta (+0.43, interval +0.17 to +0.71).
The model that looked best on VN1 development data was not better than naive here, and the default that failed on VN1 was better here. No single forecasting method is supported by the evidence so far. That is why Udgass chooses a method for each company by replaying that company’s own history, and why SES and Theta were then added to the choice.
Test 4 · Car parts · Theta worse than naive
On intermittent demand, error metrics alone mislead
Monthly sales of 2,674 car parts (2,509 kept; 165 with missing training months excluded), 45 training months and 6 test months, mostly zeros. Chosen because it is product demand and intermittent, unlike M4. Monthly, so it tests the methods rather than Udgass’s weekly product path.
| Model | Mean MASE (primary) | VN1 score | Bias |
|---|---|---|---|
| Zero forecast | 0.747 | 200% | −100% |
| Naive | 0.981 | 152% | −12% |
| SES (Udgass’s plain-Python port) | 1.008 | 155% | +14% |
| Theta (Udgass’s plain-Python port) | 1.026 | 156% | +15% |
| TSB | 1.078 | 181% | +28% |
| Training mean | 1.144 | 203% | +35% |
| Croston | 1.223 | 203% | +38% |
- R1 Theta against naive: naive better (Theta 4.6% higher; interval +0.008 to +0.080).
- R2 SES against naive: no difference shown (+2.8%; interval −0.009 to +0.061).
- R3, R4: Theta and SES better than the training mean (10.3% and 11.9% lower).
- R5 Theta against SES: “SES better” by the rule as written, but the rule was asymmetric: its 2% margin applied to one model only, and SES was just 1.7% lower. With a symmetric margin it would read “no difference shown”. The flaw was noticed after scoring and the result is recorded as written.
A zero forecast had the lowest error, because when most months are zero, MASE rewards predicting zero, which would mean never stocking the part. Methods built for intermittent demand (Croston, TSB) were worse than naive on this measure. Error metrics alone cannot choose a stocking method for such items; Udgass’s reorder suggestions choose by replaying a stock policy that is charged for stock-outs.
Tried and removed
What did not survive development
- A segment router that picked a method per demand segment, with a launch curve for new items: worse than lite at all six VN1 origins it was run on, and worse than naive on average (65.1% against 62.4%). Stopped and the code removed.
- A fixed method for every product (the UCI candidate): rejected by its test.
- Extended per-product methods (TSB, ADIDA, Theta, longer averages added to the 0.5.0 default): no real gain on VN1 development windows (64.07% against 64.11%); not adopted.
Methodology
How the tests are run
Before the data is read
- A written plan: data and files (by SHA-256), split, models, metric, the decision rule, and what counts as no result
- The plan’s SHA-256 is recorded before the test data is opened
- Test windows are sealed in an append-only register; a sealed window cannot be scored without its plan, and a spent one is labelled “development” for ever after
- Forecasts are computed from training data and fingerprinted before the test data is read, once
When scoring
- No look-ahead by construction: at each origin every model gets copies of the series cut before that date (tested with a spy model)
- Differences between models come with a 95% paired bootstrap interval over series; it does not include uncertainty from which periods were chosen, so wins per origin are reported too
- Every result carries dataset hashes, the code fingerprint, Python and package versions, origins and horizon
- Results are reported whatever they show, including when a rule turns out to be badly written
Metrics
| VN1 score | Mean absolute error plus the absolute bias, both as a share of total sales. Large series dominate. The VN1 competition’s official metric. |
|---|---|
| MASE | Mean absolute error divided by the in-sample error of naive, averaged over series. Below 1 beats naive’s in-sample accuracy. Rewards forecasting the median, which is zero for intermittent items. |
| WAPE | Sum of absolute errors over total actual sales. |
| Coverage | Share of actuals inside the stated range. An 80% range should hold about 80%; much less means the ranges understate risk. |
| Interval score | Width of the range plus a penalty for each actual outside it, scaled. Lower is better. |
Datasets
Sources, licences and what is redistributed
| Dataset | Source and licence | Used as |
|---|---|---|
| VN1 | Vandeput, N. (2024). VN1 Forecasting – Accuracy Challenge. SupChains; data provided by Flieber. doi:10.5281/zenodo.23265103. CC BY 4.0. Repository commit 281dccfa. | Held-out test (26 Oct 2026), then development |
| UCI Online Retail II | Chen, D. (2012). Online Retail II. UCI Machine Learning Repository. doi:10.24432/C5CG6D. CC BY 4.0. | Held-out test (24 Oct 2026) |
| M4 Weekly | M4 competition, github.com/Mcompetitions/M4-methods, commit 6c1067e5. The repository has no licence file. | Held-out test (2 Nov 2026). Evidence only; data not redistributed |
| Car parts | Hyndman et al., R package expsmooth 2.3 (carparts). GPL 2 or later. | Held-out test (2 Nov 2026). Data not redistributed |
Udgass never ships benchmark data. Every dataset is identified by the SHA-256 of its files, and loading refuses a file whose hash differs.
Reproducibility
Checking these numbers yourself
- The scorer matches the official one. The benchmark harness reproduces the official VN1 Phase 2 scores to the reported precision: naive 50.70% and 12-week average 80.46%, as well as Udgass’s earlier recorded scores.
- Car parts, rerun on 2 Nov 2026 with Udgass 0.6.0 from the public data: the frozen forecasts came out with the same SHA-256 as at the test, and the result file was byte-identical.
- M4 Weekly, rerun on 2 Nov 2026 with Udgass 0.6.0: every metric, rule verdict and bootstrap interval in the result file came out identical. The frozen-forecast fingerprint differs from the one recorded at the test, because that file also records the code version, which has changed since.
- The software can be inspected under agreement. Udgass is proprietary software. Companies evaluating it can ask to review the benchmark harness and the models under confidentiality terms; the datasets, plans and fingerprints on this page are public.
| Item | SHA-256 |
|---|---|
| UCI plan (recorded; not independently checkable) | 0e8d3f084c99fd72855ee11f651b6fd526a0f338e37af96e42c20df82ea8e1d7 |
| VN1 plan | 8a4940bf25dbb43832e0dd17581322dd93adfe017bc02f3780b59cc432b2cc8f |
| M4 Weekly plan | 0dc745b2a1d2bc1048f9ec6c54612e8ab4e230bade57f38a3672e4be6316a5e4 |
| M4 Weekly frozen forecasts | bb78db12892cf5c0891871fa43f374c1d98168955089f82646ad7b0a24ff3e49 |
| Car parts plan | d904fbb251db769eabd5c50b93df702cd186704d16ce1d6ee1c7587ea038230c |
| Car parts frozen forecasts | d65f87cb7c1809b4f464445ccfe73182ec2a067e4d488d61c8257ffb47085476 |
For M4 Weekly, car parts and VN1, the plans, result files and runner scripts are available to evaluators on request at hello@kpappworx.com; for UCI, the validation record, which restates the plan, the data preparation and the results. Benchmark data is not redistributed: each plan names the public source and the file fingerprints to check. The original UCI plan file is not among the evaluator files, so its fingerprint cannot be checked independently.
Limitations
What this evidence does not show
- Nothing about your business. Four public datasets ranked the methods differently. Your data may rank them differently again; the free Health Check measures it on your own history.
- No business outcome. Forecast error is not stock, cash or service. All inventory results so far are simulated.
- Per-company method choice has not itself been tested on held-out data. It is the design response to the results above. No open dataset examined has the stock-on-hand records such a test needs.
- New products are a gap. Series with no sales before a forecast date are forecast as zero; on VN1 they carry 2.5% of volume.
- Short tests. Single origins (VN1 test, M4, car parts) and about five dates (UCI) measure one period each.
- Not yet run: ETS and ARIMA on all of VN1; pretrained forecasting models (adapters exist, weights and licences not checked); any test on a real company’s data.