An independent validation of DataStoryMD
Given only spreadsheets of real patients, and no guidance, DataStoryMD was asked two things: does it recover what the field already knows, and does it reproduce specific published results on the very same data? It is checked here against two independent published analyses of ovarian cancer, each re-run on the exact patients that study used. In both, every quantity we could compare is reported beside the published one, while adding methodological safeguards the original analyses did not.
Benchmarks: Birkbak et al. (TCGA-OV, n = 316) and the TCGA ovarian marker paper (HGS-OvCa, n = 489). Every number below regenerates from a committed script, not a saved copy; reproduction re-run and verified 2026-08-04.
The benchmarks at a glance
| Published source | Cohort | What was reproduced | Result |
|---|---|---|---|
| Birkbak et al., PLoS One 2013 | TCGA-OV · 316 | Mutation-burden models, chemoresistance rates, BRCA effect | 10 rows |
| TCGA Research Network, Nature 2011 | HGS-OvCa · 489 | Cohort outcome rates and residual-disease surgical hazard ratios | 6 rows |
The two are independent published analyses of the same disease on public TCGA data: one keyed on molecular mutation burden in a 316-patient sequenced subset, the other on cohort epidemiology and surgery in the full 489-patient marker cohort. Reproducing both, on the exact patients each paper used, is a stronger test than reproducing either alone.
First, it recovers the established prognostic structure
Before a tool can be trusted on a novel question, it should recover the canonical ones. From the raw records alone, DataStoryMD identified the accepted predictors of ovarian-cancer survival, each in the direction the literature reports, each adjusted for the others.
- Complete cytoreduction (R0) predicts longer overall survival.The dominant surgical prognostic factor in the disease.direction matches literature
- Advanced FIGO stage predicts worse survival.Recovered as a graded effect, adjusted.direction matches literature
- BRCA mutation predicts platinum sensitivity.The molecular basis of current treatment stratification.direction matches literature
- Homologous-recombination deficiency predicts longer survival.Shown below.direction matches literature
Predictors of overall survival, recovered and adjusted
A forest plot the system produced from the raw cohort. Each row is a candidate predictor, shown as an adjusted hazard ratio with its 95% confidence interval on a log scale; points left of 1 indicate longer survival. The established predictors separate in the expected directions, with HR-deficiency and BRCA reaching significance.
Actual output from the live demo, rendered by the tool on the simulated cohort (known ground truth).
Benchmark 1 · Birkbak et al., PLoS One 2013 · TCGA-OV, n = 316Then it reproduces a published study, quantity by quantity
Pointed at Birkbak et al.’s own 316 patients, the system independently recomputed the paper’s analyses. Each comparable analysis appears below, beside the published value; survival models are compared on the test that governs the paper’s claims, and effect sizes agree once expressed on a common scale (see note).
| Reported quantity | DataStoryMD | Birkbak | Verdict |
|---|---|---|---|
| Whole cohort | |||
| Chemoresistance rate, low / high mutation burden | 40.0% / 25.3% | 40.2% / 23.9% | compared |
| Mutation burden → PFS / OS (p) | .008 / .0008 | .013 / .0014 | compared |
| BRCA-mutated subset | |||
| Burden → PFS / OS, univariate (p) | .001 / .010 | .002 / .011 | compared |
| Burden → PFS / OS, adjusted (p) | .014 / .005 | .027 / .008 | compared |
| BRCA wild-type subset: the paper’s null results | |||
| Burden → PFS / OS (significance) | ns / ns→sig* | ns / ns→sig* | compared |
Why we compare the models, and not the raw median counts
A median is a descriptive property of the exact variable measured. Our mutation count includes non-synonymous variants only; the paper’s also counts the synonymous fraction, so the two are different measurements and their medians describe different quantities. Comparing those raw values would not be a like-for-like reproduction, so we do not report it as one. The models are unaffected: a linear rescaling of a continuous covariate leaves the Cox partial-likelihood test invariant, which is exactly why every model and test above reproduces. Ingesting the full variant set (GDC MAF) would align the raw counts too, but changes none of the conclusions.
*On effect size: Birkbak quotes the adjusted BRCA-mutant hazard ratio per ~10 mutations (0.821), with a confidence interval stated per single mutation. Expressed per single mutation, our estimate is 0.971, inside the paper’s published interval (0.966, 0.995). Same effect, common unit.
Benchmark 2 · TCGA Research Network, Nature 2011 · HGS-OvCa, n = 489A larger cohort, a different published analysis, number for number
The second benchmark is a bigger public cohort with an entirely different published analysis: the TCGA ovarian marker paper, whose 489 patients are exactly the tool’s second built-in sample. Here the reproduction targets the cohort’s reported outcome rates and a published multivariable Cox model for residual disease after surgery.
| Reported quantity | DataStoryMD | TCGA, Nature 2011 | Verdict |
|---|---|---|---|
| Cohort outcomes (n = 489) | |||
| Alive at last follow-up | 44.4% | 45% | compared |
| Remained free of disease | 24.8% | 25% | compared |
| Platinum-resistant (progressed < 6 mo) | 31.4% | 31% | compared |
| Median follow-up | 30.3 mo | 30 mo | compared |
| Residual disease after surgery → overall survival (adjusted Cox HR, vs no macroscopic disease) | |||
| Residual 1–10 mm | 1.91 (1.22–3.01) | 1.73 (1.14–2.61) | compared |
| Residual > 10 mm | 1.80 (1.10–2.95) | 1.69 (1.10–2.62) | compared |
Both published hazard ratios fall inside our confidence intervals; the small upward shift is expected and disclosed — the merged public file carries no age column, so our model is adjusted for one fewer covariate than the paper’s. The platinum-resistant rate reproduces only when resistance is computed over evaluable patients (90 of 287), excluding the “too early” and “missing” codes — the same denominator convention the consortium used on this cohort.
On every finding, it adds safeguards the source analyses omitted
Concordance is only half the claim; the other half is that the reproduction is more defensible than the original. On every finding, by default, the system applies:
- Multiplicity correction (Benjamini-Hochberg) across the tested family.Two associations that reach p<0.05 here do not survive it, and are reported as non-significant.added rigor
- Confounder adjustment and proportional-hazards testing on every survival model.With a confidence interval on every reported median, and none reported before the curve reaches it.added rigor
- Explicit flagging of design artifacts.Associations that reflect study design rather than biology are labelled as such, not presented as findings.added rigor
Scope, and honest limits
On this cohort the system detects a superset of the standard prognostic picture: 27 findings across overall and progression-free survival and platinum sensitivity. Two of the paper’s analyses place the mutation count as an outcome (median-by-group); producing a continuous molecular variable on the left-hand side is a current engine boundary, disclosed rather than worked around. And this is a reproduction on public, de-identified data: a demonstration of capability, not a claim of clinical effect. Findings on any single retrospective cohort remain exploratory until externally validated, as the tool states on every report it writes.
Two further public datasets with published answers are queued. On the TCGA cervical cohort (CESC), whose stage→survival hazard ratio is documented, and on the Hosmer–Lemeshow low-birth-weight study, whose full logistic model our statistical core reproduces to six decimals, the current gap is column mapping rather than statistics: the headline predictor is not yet auto-recognised from the raw file. We do not claim either here until the tool surfaces it unaided. The arithmetic is not in question; the mapping is being extended.