PTB-XL benchmarks: what 21,837 ECGs taught us about deep learning
Strodthoff et al. set baseline scores for PTB-XL. ResNet and Inception-style CNNs win, and the paper warns that a good average can hide weak classes.
Abstract
This paper offers first benchmark results on PTB-XL, a free 12-lead ECG dataset, across statement prediction, age regression and sex classification. Among the deep time-series models tested, ResNet and Inception-based CNNs perform best, including the authors' new xresnet1d101. It also shows that pretraining on PTB-XL helps most on small target datasets, and adds analyses of hidden stratification, model uncertainty and interpretability.
Video resources
Related ECG learning videos, embedded for playback on this page.
Why I picked this
Every ECG paper I read compares against something. I wanted to see the benchmark those comparisons should rest on.
The problem
Before this paper, the field had two problems:
- No big public dataset to train on, so results were hard to compare.
- No agreed evaluation setup, so scores from different papers did not line up.
How it works
- Data. PTB-XL: 21,837 ECGs of 10 s from 18,885 patients, with 71 statements (44 diagnostic, 12 rhythm, 19 form). ICBEB2018 (6,877 ECGs, 9 classes) is used as a second dataset.
- Tasks. Statement prediction at several levels (all, diagnostic, sub-diagnostic, super-diagnostic, form, rhythm), plus age and sex prediction.
- Models. Many deep time-series networks (ResNet, Inception, LSTMs and more) and a wavelet-feature baseline. The metric is macro AUC, with ten stratified folds (ninth for validation, tenth for test).
- Extra checks. Transfer learning to ICBEB2018, hidden stratification, model uncertainty and attribution maps.
Results
| Task | Metric | Best result |
|---|---|---|
| Form statements | Macro AUC | about 0.89 |
| Diagnostic statements | Macro AUC | about 0.93 |
| Rhythm statements | Macro AUC | about 0.96 |
| Age (all / normal / abnormal) | MAE (years) | 7.16 / 6.86 / 7.38 |
| Sex (all / normal) | Accuracy | 84.9% / 89.8% |
| Sex (all / normal) | AUC | 0.92 / 0.96 |
From Tables I, III and IV of the paper. Age and sex scores are not directly comparable across datasets.
My reading notes
Averages hide problems
The class IVCD has AUC 0.74 overall. Clustering the model outputs split it into groups, one as low as 0.59, and the split follows whether NORM was also labelled.
Model doubt tracks doctor doubt
Uncertainty from an ensemble of 10 models rises as the cardiologist's stated likelihood falls, except at likelihood 100.
Transfer learning helps when data is small
Pretraining on PTB-XL gives no significant gain on large training sets, but a clear one on small ones.
Handle the feature baseline with care
Deep models beat the wavelet-feature baseline, but the authors warn those results depend heavily on feature choices.
Biggest lesson: always report per-class results, not just the overall score.
What's next
Next I will use PTB-XL scores as my yardstick. Another study reports age MAE 6.9 with about 750k ECGs, while this paper reaches 7.16 with about 20k.
Paper: Strodthoff, Wagner, Schaeffter and Samek, IEEE JBHI 25(5), 2021.
What makes PTB-XL useful?
PTB-XL provides 21,837 ten-second 12-lead ECG records from 18,885 patients, with multiple cardiologist-annotated diagnostic statements and recommended patient-wise splits. Its size and structured labels support reproducible comparison across diagnostic, rhythm and form tasks. Always check the dataset version and label mapping used by a given experiment.
How to interpret benchmark scores
The benchmark study found strong performance from one-dimensional ResNet- and Inception-style CNNs. This is a useful reminder that carefully designed baselines can outperform more complex architectures. For multi-label ECG prediction, AUROC and AUPRC can complement F1 and sensitivity; macro-averaging helps expose weaker rare classes. Calibration and uncertainty are important when models are used to support clinical decisions.
Common pitfalls
- Using random record-level splits instead of the recommended patient-wise split.
- Comparing scores produced with different label definitions, splits or metrics.
- Reporting only micro-averaged performance, which can hide rare-label weaknesses.
- Assuming results transfer across hospitals, devices and patient populations without external validation.
Primary sources
- Strodthoff et al., Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
- PTB-XL dataset and documentation
- Benchmark code repository
- IEEE JBHI publication DOI
💬 Comments
Share a thought. Your name and comment will be public. Please be kind and keep personal details private.
Say hello ✉
Questions, feedback or just a kind word? Email contact@ranjeetpandey.xyz or send a message below.
Loading comments…