Public data only · no customer results yet

Evidence and benchmarks

What has been measured, including where Udgass failed.

Every test below was written down before its data was read, scored once, and is reported whatever it showed. They use public datasets. None of them shows that Udgass improves a real company’s results: that has not been measured yet.

0companies whose own data has been evaluated
4public datasets with a preregistered held-out test
6 of 11written decision rules ended in a fail or “not shown”
0forecasting methods that won on every dataset

Read this first

Four kinds of evidence, and what each can show

Customer resultsNone. No company has run Udgass on its own data under an agreed protocol. Until one does, there is no evidence of savings, fewer stock-outs or better decisions in a real business.
Public held-out testsA preregistered test on public data the methods were never developed on. Shows how a method compares with simple ones on that dataset and that period. Says nothing directly about your business.
Public development evidenceResults on data that was looked at while building the methods (VN1 after 26 Oct 2026). Used to decide what to test next. Optimistic by construction; never used for a claim.
SimulationsInventory results from replaying a stock policy on sales history. They depend on the assumptions stated with them (lead time, lost sales, starting stock) and are always labelled as simulated.

Scorecard

Every preregistered held-out test on public data

“Not shown” means the written rule required an improvement the data did not support. The preregistration fingerprints (SHA-256) are listed under reproducibility.

TestQuestion asked in advanceResult
UCI Online Retail II
24 Oct 2026 · 4,960 products · weekly
Does one fixed method (simple exponential smoothing) beat Udgass’s per-product choice, with 80% ranges covering 75–85%?Fail. Slightly lower error (53.9% against 55.1%), but ranges covered 63.1%, and the current engine’s ranges covered only 69.5%.
VN1, test A
26 Oct 2026 · 15,053 series · weekly
Does a cross-learning adjustment beat the default by at least 1 point of VN1 score?Pass (71.0% against 76.3%), but both were far worse than repeating last week’s sales (50.7%).
VN1, test BDo calibrated 80% ranges cover 75–85% in every segment, with an interval score no worse than the default?Fail. Interval score much better (3.19 against 104.7), but one segment covered 67.5%.
M4 Weekly, R1–R3
2 Nov 2026 · 359 series · mostly finance and macro
Are Udgass lite, LightGBM and the 0.5.0 default better than naive (mean MASE at least 2% lower, 95% interval below zero)?Lite not shown. LightGBM not shown. Default better (6.6% lower). Two standard methods (ETS, Theta) did better than all three.
Car parts, R1–R5
2 Nov 2026 · 2,509 series · monthly, intermittent
Are SES and Theta better than naive and than the mean; which of the two is better?Theta worse than naive. SES no difference shown against naive. Both better than the mean. The SES-vs-Theta rule had a flaw, described below.

Test 1 · UCI Online Retail II · Fail

Ranges that were too narrow

One UK online retailer of gift items selling mostly to wholesale customers, December 2009 to December 2011. Cancellations, non-product codes and every customer field were dropped before use; weekly totals per product. Test origins July to November 2011.

MeasureUdgass engineCandidate (SES)4-week averageRule
Error (WAPE)55.1%53.9%56.9%Pass
80% ranges containing the actual69.5%63.1%—Fail (needs 75–85%)
“Mostly zero” products (WAPE)261.8%287.3%—Fail (25.5 points worse)
Bias (negative = under-forecast)−7.1%−12.6%—Reported only

What it showed about Udgass itself: on this lumpy wholesale demand, the engine’s 80% ranges held the actual only 69.5% of the time and it under-forecast by 7%. The assessment report now warns when ranges fall short like this, and later work on calibrated ranges came from it. Limits: one retailer, about five test dates, a pre-Christmas rise, and under two years of history, so no yearly seasonality could be learned.

Test 2 · VN1 · Pass on its rule, fail against naive

Far behind a simple method after the holiday peak

The VN1 Forecasting Accuracy Challenge: 15,053 weekly product-warehouse series. 183 weeks of history, then 13 test weeks (8 January to 1 April 2024) kept in a sealed folder during development. Scored with the competition’s own scorer.

ForecastVN1 score (lower is better)What it is
Competition winner46.4%Official result
Naive: last week repeated50.7%Official benchmark
Udgass candidate (cross-learning)71.0%Held-out at the time
Udgass 0.5.0 default76.3%Held-out at the time
12-week moving average80.5%Official benchmark

The written rule compared the candidate with the default, and it passed. The finding that matters more is that both were 20 to 26 points worse than repeating last week. The test weeks start just after the holiday peak, and Udgass’s level methods kept forecasting near the holiday level (bias +15%) while naive picked up the January drop at once.

After the test window was spent

Everything on VN1 since 26 October 2026 is development evidence, because the test weeks have been seen. Two results guided what was tested next:

  • Over ten 13-week origins (October 2021 to January 2024), Udgass lite and a LightGBM model were about 6 points better than naive, at 9 of 10 origins. The 0.5.0 default was about 6 points worse, so the January failure was not specific to one window. On the high-volume, regular series that carry 65% of units, lite was slightly worse than naive (43.6% against 41.5%).
  • Ranges calibrated by demand segment brought every segment within 3.5 points of its 80% target, at the cost of ranges about a fifth wider. A single pooled calibration looked right overall (81%) while the high-volume series held only 65–69%.

These rolling-origin figures were chosen after seeing the data and are optimistic. They are why the next fresh test (M4 Weekly) compared lite and LightGBM with naive. A later post-hoc run on the spent VN1 window is not a held-out result and is not shown as a ranking. Test B’s full per-segment table was not recorded; the summary above is the recorded result.

Test 3 · M4 Weekly · Two of three not shown

The VN1 ranking did not carry over

The first test on data never used in Udgass development: 359 weekly series from the M4 competition (Finance 164, Micro 112, Macro 41, Demographic 24, Other 12, Industry 6), 13 weeks ahead. Mostly business and economic series, not product demand. Forecasts were computed from training data and fingerprinted before the test file was opened.

ModelMean MASE (primary)sMAPESeries better than naive
ETS (statsforecast)2.4208.6257%
Theta (statsforecast)2.5057.9453%
Udgass 0.5.0 default2.5949.2556%
Naive2.7779.16—
4-week average2.9239.8347%
Udgass lite2.9309.3647%
LightGBM global3.3477.6856%
12-week average3.79811.9433%
Seasonal naive (52 weeks)9.57814.5225%
  • R1 Udgass lite against naive: not shown. MASE 5.5% higher; difference +0.15, 95% interval −0.07 to +0.38.
  • R2 LightGBM against naive: not shown. 20.5% higher; +0.57, interval −0.15 to +1.54.
  • R3 Udgass 0.5.0 default against naive: better. 6.6% lower; −0.18, interval −0.36 to −0.03.
  • Reported without a rule: Theta and ETS were better than naive; lite was worse than Theta (+0.43, interval +0.17 to +0.71).

The model that looked best on VN1 development data was not better than naive here, and the default that failed on VN1 was better here. No single forecasting method is supported by the evidence so far. That is why Udgass chooses a method for each company by replaying that company’s own history, and why SES and Theta were then added to the choice.

Test 4 · Car parts · Theta worse than naive

On intermittent demand, error metrics alone mislead

Monthly sales of 2,674 car parts (2,509 kept; 165 with missing training months excluded), 45 training months and 6 test months, mostly zeros. Chosen because it is product demand and intermittent, unlike M4. Monthly, so it tests the methods rather than Udgass’s weekly product path.

ModelMean MASE (primary)VN1 scoreBias
Zero forecast0.747200%−100%
Naive0.981152%−12%
SES (Udgass’s plain-Python port)1.008155%+14%
Theta (Udgass’s plain-Python port)1.026156%+15%
TSB1.078181%+28%
Training mean1.144203%+35%
Croston1.223203%+38%
  • R1 Theta against naive: naive better (Theta 4.6% higher; interval +0.008 to +0.080).
  • R2 SES against naive: no difference shown (+2.8%; interval −0.009 to +0.061).
  • R3, R4: Theta and SES better than the training mean (10.3% and 11.9% lower).
  • R5 Theta against SES: “SES better” by the rule as written, but the rule was asymmetric: its 2% margin applied to one model only, and SES was just 1.7% lower. With a symmetric margin it would read “no difference shown”. The flaw was noticed after scoring and the result is recorded as written.

A zero forecast had the lowest error, because when most months are zero, MASE rewards predicting zero, which would mean never stocking the part. Methods built for intermittent demand (Croston, TSB) were worse than naive on this measure. Error metrics alone cannot choose a stocking method for such items; Udgass’s reorder suggestions choose by replaying a stock policy that is charged for stock-outs.

Tried and removed

What did not survive development

  • A segment router that picked a method per demand segment, with a launch curve for new items: worse than lite at all six VN1 origins it was run on, and worse than naive on average (65.1% against 62.4%). Stopped and the code removed.
  • A fixed method for every product (the UCI candidate): rejected by its test.
  • Extended per-product methods (TSB, ADIDA, Theta, longer averages added to the 0.5.0 default): no real gain on VN1 development windows (64.07% against 64.11%); not adopted.

Methodology

How the tests are run

Before the data is read

  • A written plan: data and files (by SHA-256), split, models, metric, the decision rule, and what counts as no result
  • The plan’s SHA-256 is recorded before the test data is opened
  • Test windows are sealed in an append-only register; a sealed window cannot be scored without its plan, and a spent one is labelled “development” for ever after
  • Forecasts are computed from training data and fingerprinted before the test data is read, once

When scoring

  • No look-ahead by construction: at each origin every model gets copies of the series cut before that date (tested with a spy model)
  • Differences between models come with a 95% paired bootstrap interval over series; it does not include uncertainty from which periods were chosen, so wins per origin are reported too
  • Every result carries dataset hashes, the code fingerprint, Python and package versions, origins and horizon
  • Results are reported whatever they show, including when a rule turns out to be badly written

Metrics

VN1 scoreMean absolute error plus the absolute bias, both as a share of total sales. Large series dominate. The VN1 competition’s official metric.
MASEMean absolute error divided by the in-sample error of naive, averaged over series. Below 1 beats naive’s in-sample accuracy. Rewards forecasting the median, which is zero for intermittent items.
WAPESum of absolute errors over total actual sales.
CoverageShare of actuals inside the stated range. An 80% range should hold about 80%; much less means the ranges understate risk.
Interval scoreWidth of the range plus a penalty for each actual outside it, scaled. Lower is better.

Datasets

Sources, licences and what is redistributed

DatasetSource and licenceUsed as
VN1Vandeput, N. (2024). VN1 Forecasting – Accuracy Challenge. SupChains; data provided by Flieber. doi:10.5281/zenodo.23265103. CC BY 4.0. Repository commit 281dccfa.Held-out test (26 Oct 2026), then development
UCI Online Retail IIChen, D. (2012). Online Retail II. UCI Machine Learning Repository. doi:10.24432/C5CG6D. CC BY 4.0.Held-out test (24 Oct 2026)
M4 WeeklyM4 competition, github.com/Mcompetitions/M4-methods, commit 6c1067e5. The repository has no licence file.Held-out test (2 Nov 2026). Evidence only; data not redistributed
Car partsHyndman et al., R package expsmooth 2.3 (carparts). GPL 2 or later.Held-out test (2 Nov 2026). Data not redistributed

Udgass never ships benchmark data. Every dataset is identified by the SHA-256 of its files, and loading refuses a file whose hash differs.

Reproducibility

Checking these numbers yourself

  • The scorer matches the official one. The benchmark harness reproduces the official VN1 Phase 2 scores to the reported precision: naive 50.70% and 12-week average 80.46%, as well as Udgass’s earlier recorded scores.
  • Car parts, rerun on 2 Nov 2026 with Udgass 0.6.0 from the public data: the frozen forecasts came out with the same SHA-256 as at the test, and the result file was byte-identical.
  • M4 Weekly, rerun on 2 Nov 2026 with Udgass 0.6.0: every metric, rule verdict and bootstrap interval in the result file came out identical. The frozen-forecast fingerprint differs from the one recorded at the test, because that file also records the code version, which has changed since.
  • The software can be inspected under agreement. Udgass is proprietary software. Companies evaluating it can ask to review the benchmark harness and the models under confidentiality terms; the datasets, plans and fingerprints on this page are public.
ItemSHA-256
UCI plan (recorded; not independently checkable)0e8d3f084c99fd72855ee11f651b6fd526a0f338e37af96e42c20df82ea8e1d7
VN1 plan8a4940bf25dbb43832e0dd17581322dd93adfe017bc02f3780b59cc432b2cc8f
M4 Weekly plan0dc745b2a1d2bc1048f9ec6c54612e8ab4e230bade57f38a3672e4be6316a5e4
M4 Weekly frozen forecastsbb78db12892cf5c0891871fa43f374c1d98168955089f82646ad7b0a24ff3e49
Car parts pland904fbb251db769eabd5c50b93df702cd186704d16ce1d6ee1c7587ea038230c
Car parts frozen forecastsd65f87cb7c1809b4f464445ccfe73182ec2a067e4d488d61c8257ffb47085476

For M4 Weekly, car parts and VN1, the plans, result files and runner scripts are available to evaluators on request at hello@kpappworx.com; for UCI, the validation record, which restates the plan, the data preparation and the results. Benchmark data is not redistributed: each plan names the public source and the file fingerprints to check. The original UCI plan file is not among the evaluator files, so its fingerprint cannot be checked independently.

Limitations

What this evidence does not show

  • Nothing about your business. Four public datasets ranked the methods differently. Your data may rank them differently again; the free Health Check measures it on your own history.
  • No business outcome. Forecast error is not stock, cash or service. All inventory results so far are simulated.
  • Per-company method choice has not itself been tested on held-out data. It is the design response to the results above. No open dataset examined has the stock-on-hand records such a test needs.
  • New products are a gap. Series with no sales before a forecast date are forecast as zero; on VN1 they carry 2.5% of volume.
  • Short tests. Single origins (VN1 test, M4, car parts) and about five dates (UCI) measure one period each.
  • Not yet run: ETS and ARIMA on all of VN1; pretrained forecasting models (adapters exist, weights and licences not checked); any test on a real company’s data.