Private synthetic data on five public clinical datasets: what held up and what didn't

Why we publish this

Every synthetic data vendor reports a quality score. Few explain what a bad score looks like, which baselines they compared against, or where their method breaks. A biostatistician deciding whether to trust synthetic data needs all three.

So we ran the MirrorCohort engine on five public, de-identified clinical datasets under one protocol and reported every number, including the weak ones. Anyone can download the same datasets and check the results with open-source metric definitions.

The datasets

We chose datasets that look like the tables research teams actually work with: hospital encounters, a population survey, vitals and labs, and two small cardiology cohorts.

Full citations and licenses are listed under Data sources.

How we tested

The protocol was the same for every dataset.

1. Split. 70% of patients were used to train the generator. The other 30% were set aside and never seen by it. Every score below is measured against those held-out patients.
2. Privacy budget. Each run used a differential privacy budget of ε = 8. The budget covers the whole pipeline, including the preprocessing step that learns column ranges and categories, not only model training. The engine chooses a private GAN for larger mixed tables and a private Gaussian copula for small or measurement-heavy ones.
3. Outcome declared up front. We told the engine which column would be predicted. It spends part of the same ε budget preserving the link between that column and the others.
4. Three measurements on each synthetic dataset:
   • Fidelity score. How closely each column's distribution and each pair of columns' relationship match the real data, from 0 to 100%. We use the open-source SDMetrics quality report definitions, so the score can be recomputed independently.
   • Predictive utility. We trained a random forest on the synthetic patients and another on the real training patients, then scored both on the held-out real patients with ROC AUC. If synthetic data preserves what matters for analysis, the two scores should be close.
   • Membership attack. An attacker who sees the synthetic data tries to tell which real patients were in the training set, by checking how close each patient is to its nearest synthetic record. An attack AUC of 0.50 means the attacker does no better than a coin flip.

Results

ε = 8 for every run. AUC is measured on the 30% of real patients held out from training. Single runs; repeated runs vary by a point or two (see Limitations).

Predictive utility

CDC survey and Cleveland: the synthetic-trained model came within 0.01 and 0.02 AUC of the real-trained one. For building and testing analysis code, that difference is unlikely to change a decision.

Hospital encounters: the gap is 0.03, but readmission is hard to predict even from real data here (AUC 0.62). On a second outcome in the same data, whether the patient was on diabetes medication, the scores were 0.898 synthetic and 0.906 real.

NHANES and heart failure: the gap was about 0.07. In NHANES the weak point is continuous lab and vital measurements: under a privacy budget, the engine keeps their shape but blurs some of their relationships with the outcome. In heart failure, 209 training patients leaves very little signal to work with once privacy noise is added. Synthetic data from these two is useful for drafting and testing analyses, but we would not tell a team to expect the same model performance as on real data.

Privacy

The attack scored 0.48 to 0.52 everywhere, including on the small cohorts, where memorising individual patients is the main risk. The synthetic records were no closer to the patients used in training than to patients the engine never saw.

This is one attack, and a fairly simple one. A passed attack is supporting evidence, not proof. The formal protection is the differential privacy guarantee, which holds against any attack, and its strength is set by ε. We explain what ε = 8 does and does not promise in a separate note.

Reading the fidelity score

A single fidelity percentage is easy to misread, so we compared it with two baselines on every dataset.

Shuffled columns. We took the real data and shuffled each column independently. Every column keeps its exact real distribution, but every relationship between columns is destroyed, including the link to the outcome. This scrambled data scored higher than our synthetic data on four of the five datasets.

That says more about the metric than about the data. The score gives a lot of weight to single-column shapes, which shuffling preserves perfectly. Shuffled data is useless for analysis, since a model trained on it has nothing to learn. So a high fidelity score alone does not show that synthetic data is useful, and a vendor quoting one without baselines is leaving out the context you need. This is why we lead with predictive utility.

A second real sample. We also scored real patients against other real patients from the same dataset. On the two small cohorts this scored only 82 to 84%, simply because 300 patients is a noisy sample. Our 81 to 83% on those datasets is close to that ceiling. On the larger datasets there is more room: real-versus-real reaches 92 to 98%.

Limitations

• Single runs. Results vary between runs. On Cleveland we repeated the run four times: fidelity ranged from 81.3% to 82.6%, synthetic-trained AUC from 0.815 to 0.853, and attack AUC from 0.48 to 0.51.
• Public, single-table data. All five are well-known public research datasets with one row per patient or encounter. Real study data is often messier, linked across several tables, and has more missing values. This benchmark does not cover multi-table data.
• NHANES version. We used the teaching version of NHANES 2009–2012 distributed with the NHANES R package, not the official CDC survey files with sampling weights.
• One privacy setting. ε = 8 is a common setting for synthetic data but is not a strong guarantee on its own terms. A lower ε gives stronger protection and lower fidelity.
• We ran it ourselves. This is our own benchmark of our own engine. The datasets and metric definitions are public so that others can check it.
• Not for final analyses. We see synthetic data as a tool for planning, code development, teaching and data sharing. Analyses for regulatory submissions or clinical decisions should run on the real data.

Data sources

• Diabetes 130-US Hospitals for Years 1999–2008. Clore, Cios, DeShazo & Strack (2014). UCI Machine Learning Repository. CC BY 4.0.
• CDC Diabetes Health Indicators. UCI Machine Learning Repository, derived from the CDC Behavioral Risk Factor Surveillance System 2015. CC BY 4.0.
• National Health and Nutrition Examination Survey 2009–2012, US Centers for Disease Control and Prevention (public domain), via the NHANES R package.
• Heart Failure Clinical Records. Chicco & Jurman (2020), BMC Medical Informatics and Decision Making 20:16. UCI Machine Learning Repository. CC BY 4.0.
• Heart Disease (Cleveland subset). Janosi, Steinbrunn, Pfisterer & Detrano (1989). UCI Machine Learning Repository. CC BY 4.0.
• Fidelity definitions: SDMetrics quality report (column shapes and column pair trends), open source.

We used random samples or subsets of columns. Use of these datasets does not imply endorsement by UCI, the CDC or the original authors.

Raja Pal, PhD is the founder of MirrorCohort. Before founding MirrorCohort, Raja spent nearly nine years as a process and quality engineer at Intel. Questions about this benchmark: raja@mirrorcohort.com · LinkedIn