Results
Every result we measured, in charts
We tested TraceRead in seven studies, S1 to S7. Each chart below shows one result, how many ECGs or photos it comes from, where they came from and which file holds the numbers. Where a result is weak, we say so.
How to read the charts
- Red is TraceRead's own result.
- Grey is what it is compared with: a published benchmark, reference ECG programs or the original recordings.
- Green means a defect was flagged, as it should be.
AUROC is a score from 0.50 to 1.00 for how well a pattern is told apart from the rest. 0.50 is a coin toss. 1.00 is perfect.
A photo made by computer is a photo of a printed page made by TraceRead's own photo simulator (tilt, light, shadow, blur).
The short version
Six results, each with the number of ECGs behind it
Every number here is repeated, with its chart, further down the page.
0 of 69
good frames wrongly flagged by the photo check, while it flagged every frame with a defect it is built to catch
n = 522 frames
Test S6, the photo check
0.908
median match between the rebuilt trace and the original recording, from the photo alone (1.00 is a perfect match)
n = 500 photos made by computer
Test S1, reading the paper
0.902
how well 71 patterns are told apart (AUROC). The published benchmark scores 0.925, so this is below it
n = 2,198 ECGs from Germany
Test S2, finding patterns
20 of 29
patterns scoring 0.90 or more on ECGs from China that TraceRead AI never trained on
n = 45,152 ECGs from China
Test S2, finding patterns
0.009
AUROC lost when TraceRead AI reads photos made by computer instead of the original recordings, across all patterns counted (0.017 across the 5 broad groups)
n = 493 photos made by computer
Test S3, photos against original recordings
7.85 s
to read 95 in 100 ECGs on a 2 vCPU server in AWS Cape Town (median 7.63 s, slowest 27.82 s), with no graphics card
n = 97 photos made by computer
Test S7, speed and cost
What every result on this page has in common
Every photo result comes from photos made by computer of public research ECGs from Germany, China and Brazil. No real phone photo has been scored and no ECG was recorded in Nigeria.
Capture
The photo check caught every defect it was built for and none of 69 good frames
Before a photo is sent, the phone checks it for blur, hand shake, glare, poor light and a page that does not fill the frame. We tested the checks on 522 frames of TraceRead test printouts, some with a defect added on purpose.
Share of frames the photo check flagged, by defect
| Row | Frames flagged | Frames tested |
|---|---|---|
| Blur of 0.5 mm of paper | 18 of 18 | 18 frames |
| Stronger blur, five sizes | 90 of 90 | 90 frames |
| Hand shake blur, three sizes | 54 of 54 | 54 frames |
| Blur, then camera noise added | 72 of 72 | 72 frames |
| Hand shake blur, then camera noise added | 18 of 18 | 18 frames |
| Large glare spot, 12% or 20% of the page width | 60 of 60 | 60 frames |
| Washed out, too bright | 18 of 18 | 18 frames |
| Too dark | 9 of 9 | 9 frames |
| Page too small in the frame (50%, 70% or in a corner) | 27 of 27 | 27 frames |
| Slight blur, 0.3 mm of paper | 0 of 18 | 18 frames |
| Small glare spot, 3% or 5% of the page width | 0 of 60 | 60 frames |
| Sharp, well lit pages (no defect) | 0 of 33 | 33 frames |
| The same good pages with camera noise added | 0 of 36 | 36 frames |
| Page filling 95% of the frame | 0 of 9 | 9 frames |
Data: Frames of TraceRead test printouts, some with a defect added on purpose. Source: spikes/s6-capture/RESULTS.md, section 2 Self test.
0 of 69
good frames flagged (33 sharp pages, 36 with camera noise added)
n = 69 frames
Test S6
366 of 366
frames with a defect it is built to catch were flagged
n = 366 frames
Test S6
Where this is weak
Blur of 0.3 mm of paper (18 frames) is below the width of the trace line, which is where the blur check starts. Small glare spots of 3% or 5% of the page width (60 frames) were left alone on purpose. Neither was flagged.
The checks were tested on pictures and in a browser with a fake camera. No real phone camera was measured.
Reading the paper
From a photo alone, the rebuilt trace matches the original recording at a median of 0.908
TraceRead finds the page, works out the layout and follows the ink of each of the 12 leads. We compared its rebuilt traces with the original recordings on 500 photos made by computer of test ECGs from Germany (PTB-XL).
Median match between the rebuilt trace and the original recording
| Row | Median match |
|---|---|
| Photo alone: TraceRead finds the page, layout and rows | 0.908 |
| Photo alone, timing shift of up to 0.1 second allowed | 0.962 |
| True page position given | 0.980 |
Data: Photos made by computer of PTB-XL test ECGs (fold 10), 6,000 lead rows. Source: spikes/s1-digitisation/RESULTS.md, part 2 section 3 Results against the targets.
0.908
median match from the photo alone
n = 500 photos made by computer
Test S1
0.980
the same match when the true page position is given
n = 500 photos made by computer
Test S1
Where this is weak
The drop from 0.980 to 0.908 is mostly in where the rows sit in time. Allowing a timing shift of up to 0.1 second brings the match to 0.962, so the shape of the trace is close and its position can be a few milliseconds off.
What TraceRead found right in the photo
| Row | Share of photos (percent) | Photos |
|---|---|---|
| Page layout read right | 99.0% | |
| Long rhythm strips read right | 99.0% | |
| Paper size read right (A4 or US Letter) | 97.6% | |
| Page found in the photo | 98.6% | 493 of 500 |
Data: Photos made by computer of PTB-XL test ECGs (fold 10), 6,000 lead rows. Source: spikes/s1-digitisation/RESULTS.md, part 2 section 3 Results against the targets.
7 of 500
pages not found, so no first read was given (1.4%)
n = 500 photos made by computer
Test S1
Where this is weak
Five of the 7 pages not found were cropped: their corners were outside the frame. TraceRead gives no first read for a page it cannot find and asks for a new photo. TraceRead knows only the page layouts it was built for, so a machine with a different page design needs a new template.
Median match for each of the 12 leads
The red dot is the median from the photo alone. The line behind it runs from the lower to the upper quarter of the 500 photos.
| Lead | Photo alone, median | Lower quarter | Upper quarter | True page position, median |
|---|---|---|---|---|
| I | 0.938 | 0.827 | 0.975 | 0.980 |
| II | 0.912 | 0.779 | 0.962 | 0.985 |
| III | 0.920 | 0.803 | 0.969 | 0.974 |
| aVR | 0.900 | 0.757 | 0.959 | 0.978 |
| aVL | 0.903 | 0.740 | 0.957 | 0.969 |
| aVF | 0.900 | 0.746 | 0.958 | 0.975 |
| V1 | 0.930 | 0.786 | 0.974 | 0.988 |
| V2 | 0.900 | 0.672 | 0.962 | 0.981 |
| V3 | 0.823 | 0.600 | 0.944 | 0.962 |
| V4 | 0.860 | 0.585 | 0.967 | 0.965 |
| V5 | 0.893 | 0.648 | 0.972 | 0.981 |
| V6 | 0.936 | 0.779 | 0.979 | 0.991 |
Data: Photos made by computer of PTB-XL test ECGs (fold 10), 6,000 lead rows. Source: spikes/s1-digitisation/RESULTS.md, part 2 section 3 Per lead medians; eval/levelC_test_per_lead.csv and levelA_test_per_lead.csv.
9 of 12
leads reach a median match of 0.90 from the photo alone
n = 500 photos made by computer
Test S1
Where this is weak
V3 (0.823) and V4 (0.860) are the weakest leads at every step, where the heart signal is tallest and rows cross. The grey dots show what the same leads reach when the page position is given.
Median match by page layout
| Row | Photo alone | True page position given |
|---|---|---|
| 3x4 with one long strip | 0.921 | 0.983 |
| 3x4 with three long strips | 0.916 | 0.981 |
| 6x2 | 0.917 | 0.982 |
| 12x1, one lead per row | 0.776 | 0.922 |
Data: Photos made by computer of PTB-XL test ECGs (fold 10), 6,000 lead rows. Source: spikes/s1-digitisation/RESULTS.md, part 2 section 3 Per layout medians; eval/levelC_test_per_layout.csv and levelA_test_per_layout.csv.
Where this is weak
The 12x1 layout is the weak spot: 0.776 from the photo alone. Its rows are only 15 mm apart, so tall beats cross the rows next to them and the trace follower sometimes follows the wrong one.
The 12 case study photos, one row each
Each row opens that case study. The filled dot is the median match as read. The hollow dot is the match when a timing shift of up to 0.1 second is allowed.
| Case study | As read | Lined up | Read wrongly |
|---|---|---|---|
| T-0010: Anterior infarct pattern | 0.880 | 0.983 | No |
| T-0018: Incomplete right bundle branch block and 1 more | 0.936 | 0.982 | No |
| T-0025: Normal ECG and 1 more | 0.801 | 0.836 | No |
| T-0045: Complete right bundle branch block and 1 more | 0.993 | 0.994 | No |
| T-0046: Unspecific wide QRS and 1 more | 0.778 | 0.983 | No |
| T-0048: Reduced blood flow pattern, bottom and 2 more | 0.766 | 0.982 | No |
| T-0053: Normal ECG | -0.004 | 0.794 | Yes |
| T-0056: Minor T wave abnormalities | 0.877 | 0.988 | No |
| T-0057: Unspecific reduced blood flow pattern and 1 more | 0.965 | 0.975 | No |
| T-0058: Inferior infarct pattern | 0.900 | 0.982 | No |
| T-0059: Normal ECG | 0.015 | 0.139 | Yes |
| T-0064: Normal ECG and 1 more | 0.910 | 0.976 | No |
Data: 12 adult pages of the TraceRead print kit (PTB-XL test ECGs), one photo made by computer each, chosen by a seeded draw. Source: apps/web/src/content/runs/index.json (the 12 case study runs).
0.879 / 0.982
median over the 12 photos, as read and lined up
n = 12 photos made by computer
The 12 case study runs
The two read wrongly
T-0053 (minus 0.004 as read, 0.794 lined up) and T-0059 (0.015 as read, 0.139 lined up) were read wrongly, and neither was held for retake. For T-0059, none of the 12 calibration pulses was read, yet the checks did not fire.
The 12 were drawn with a fixed seed, 3 from each of 4 groups of the dataset's own statements, so they do not stand for the dataset as a whole. 10 of the 12 carry the paper speed warning.
Finding patterns
TraceRead AI is level with the published scores on 2 of 6 pattern tasks and below them on 4
We scored TraceRead AI on test ECGs from Germany that it never trained on, one task at a time, the way a published benchmark does. Each task counts a different number of patterns. A score of 0.50 is a coin toss and 1.00 is perfect.
AUROC by pattern task, against the published benchmark
| Row | TraceRead AI | Published benchmark |
|---|---|---|
| 5 broad groups | 0.931 | 0.928 |
| 12 rhythm patterns | 0.955 | 0.957 |
| 19 wave shape patterns | 0.867 | 0.896 |
| 23 finer groups | 0.911 | 0.929 |
| 44 main heart patterns | 0.902 | 0.937 |
| All 71 patterns | 0.902 | 0.925 |
Data: PTB-XL test ECGs (fold 10), original recordings; patterns scored as the published benchmark scores them. Source: spikes/s2-classifier-baseline/RESULTS.md, section 3 Benchmark tasks, fold 10; results/benchmark_fold10.csv.
0.931
for the 5 broad groups, against 0.928 published
n = 2,158 ECGs from Germany
Test S2
0.902
for all 71 patterns, against 0.925 published
n = 2,198 ECGs from Germany
Test S2
Where this is weak
Rhythm is 0.002 under the published score, with the published value well inside our range: a tie at this resolution. The four wider tasks are below it, by 0.018 to 0.035.
We trained for 8 to 20 rounds where the benchmark protocol asks for 50, and could not test a longer schedule, so we cannot say how much of the gap is training time. 22 of the 71 patterns have fewer than 10 examples in the test ECGs and their scores are very uncertain.
AUROC for 29 patterns on ECGs from China
Chapman Shaoxing Ningbo ECGs, which TraceRead AI never trained on, sorted from best to worst. The line marks 0.90.
| Pattern | AUROC | Low end of the range | High end of the range | ECGs with the pattern | Label match |
|---|---|---|---|---|---|
| Complete left bundle branch block | 0.998 | 0.997 | 0.999 | 213 | Matches |
| Complete right bundle branch block | 0.994 | 0.992 | 0.995 | 1,096 | Matches |
| Sinus tachycardia | 0.992 | 0.991 | 0.993 | 7,255 | Matches |
| Right bundle branch block, complete or not stated | 0.992 | 0.990 | 0.993 | 1,745 | Matches in part |
| Atrial fibrillation or flutter | 0.985 | 0.984 | 0.986 | 9,840 | Matches |
| Left anterior fascicular block | 0.984 | 0.981 | 0.987 | 380 | Matches |
| Supraventricular tachycardia | 0.981 | 0.978 | 0.984 | 724 | Matches in part |
| Ventricular premature beats | 0.971 | 0.967 | 0.974 | 1,385 | Matches |
| Left ventricular hypertrophy | 0.963 | 0.958 | 0.968 | 647 | Matches |
| Left bundle branch block, complete or not stated | 0.960 | 0.951 | 0.969 | 453 | Matches in part |
| Incomplete right bundle branch block | 0.939 | 0.929 | 0.948 | 246 | Matches |
| Right ventricular hypertrophy | 0.936 | 0.922 | 0.948 | 110 | Matches |
| Sinus bradycardia | 0.931 | 0.928 | 0.933 | 16,559 | Matches |
| Paced rhythm | 0.926 | 0.916 | 0.936 | 1,182 | Matches |
| First degree AV block | 0.925 | 0.919 | 0.931 | 1,140 | Matches |
| Right atrial hypertrophy | 0.923 | 0.870 | 0.966 | 36 | Matches in part |
| Sinus arrhythmia | 0.923 | 0.917 | 0.928 | 2,550 | Matches in part |
| Sinus rhythm | 0.912 | 0.908 | 0.916 | 8,125 | Matches |
| Myocardial infarction pattern | 0.909 | 0.879 | 0.933 | 123 | Matches in part |
| Third degree AV block | 0.905 | 0.866 | 0.941 | 76 | Matches |
| Wolff Parkinson White pattern | 0.893 | 0.859 | 0.924 | 72 | Matches |
| Atrial premature beats | 0.881 | 0.872 | 0.889 | 1,321 | Matches |
| Prolonged PR interval | 0.877 | 0.842 | 0.907 | 52 | Matches |
| T wave inversion | 0.876 | 0.871 | 0.881 | 2,877 | Matches in part |
| ST depression | 0.826 | 0.817 | 0.834 | 1,668 | Matches in part |
| Second degree AV block | 0.822 | 0.786 | 0.853 | 97 | Matches |
| Low QRS voltage | 0.807 | 0.794 | 0.820 | 1,043 | Matches in part |
| Long QT interval | 0.735 | 0.714 | 0.756 | 394 | Matches |
| Unspecific wide QRS | 0.689 | 0.672 | 0.704 | 771 | Matches in part |
Data: Chapman Shaoxing Ningbo ECGs (China), never used to train TraceRead AI; 3x4 printout with a lead II strip. Source: spikes/s2-classifier-baseline/RESULTS.md, section 3 External test on Chapman; results/chapman_auroc_per_label.csv.
20 of 29
patterns at 0.90 or more on ECGs from China
n = 45,152 ECGs from China
Test S2
Where this is weak
Nine fall below 0.90. The weakest are unspecific wide QRS (0.689) and long QT (0.735). The patterns have between 36 and 16,559 ECGs each, so the ranges for rare patterns are wide.
AUROC measures ranking, not whether a fixed threshold would work in a new hospital. No threshold was chosen on these ECGs.
What a 3x4 printout costs, against 12 full leads
The same TraceRead AI shown the same ECGs as 12 full leads of 10 seconds, or as the common 3x4 printout with a lead II strip.
| Row | 12 full leads | 3x4 printout |
|---|---|---|
| 5 broad groups | 0.916 | 0.919 |
| 12 rhythm patterns | 0.967 | 0.969 |
| All 71 patterns | 0.902 | 0.910 |
Data: PTB-XL test ECGs (fold 10), original recordings shown to TraceRead AI as 12 full leads or as a 3x4 printout with a lead II strip. Source: spikes/s2-classifier-baseline/RESULTS.md, section 3 One model, four printout layouts; results/traceread_layouts_fold10.csv and layout_cost_fold10.csv.
+0.0075
AUROC for the 3x4 printout minus 12 full leads, all 71 patterns (range +0.0044 to +0.0108)
n = 2,198 ECGs from Germany
Test S2
Where this is weak
A 3x4 printout cost nothing we could measure. The layout that shows the least scored a little better, and the difference sits in the rare patterns. We did not find out why. These are original recordings cut to the printout layout, not photos.
Photos against original recordings
Reading a photo costs 0.009 AUROC across all patterns and 0.017 across the 5 broad groups
We scored the same test ECGs two ways: as original recordings and as rebuilt traces from photos made by computer. The gap is what the photo costs. Fine tuning TraceRead AI on rebuilt traces narrowed the gap and cost nothing on original recordings.
AUROC on original recordings and on photos
| Row | Original recordings | Photos, before fine tuning | Photos, after fine tuning |
|---|---|---|---|
| All patterns, 25 counted | 0.927 | 0.915 | 0.919 |
| 5 broad groups | 0.913 | 0.887 | 0.897 |
Data: Photos made by computer of PTB-XL test ECGs (fold 10), the same ECGs scored as original recordings and as rebuilt traces. Source: spikes/s3-clean-vs-digitised/RESULTS.md, sections 3.1 and 3.3; results/gap_baseline_test_main.csv and gap_finetuned_test_main.csv.
Reading this chart
For the 5 broad groups, the fine tuned TraceRead AI scores 0.915 on the original recordings (0.913 before). Each gap on the next chart is taken against the original recordings read by the same TraceRead AI. Fine tuning changed the original recordings by 0.000 (all patterns) and 0.001 (5 broad groups).
The AUROC lost to the photo, before and after fine tuning
| Row | Before fine tuning | After fine tuning |
|---|---|---|
| All patterns, 25 counted | 0.012 | 0.009 |
| 5 broad groups | 0.027 | 0.017 |
Data: Photos made by computer of PTB-XL test ECGs (fold 10), the same ECGs scored as original recordings and as rebuilt traces. Source: spikes/s3-clean-vs-digitised/RESULTS.md, sections 3.1 and 3.3; results/gap_baseline_test_main.csv and gap_finetuned_test_main.csv.
0.009
AUROC lost to the photo after fine tuning, all patterns counted (0.012 before)
n = 493 photos made by computer
Test S3
0.017
AUROC lost to the photo after fine tuning, 5 broad groups (0.027 before)
n = 485 photos made by computer
Test S3
Where this is weak
Fine tuning closed about a third of the gap and did not close it: for the 5 broad groups the range after (0.007 to 0.028) is still clear of zero. Doubling the fine tuning photos from 1,957 to 3,916 changed nothing.
Share of photos held for retake, by page layout
| Row | Share held (percent) | Usable photos | Photos held |
|---|---|---|---|
| 3x4 with one long strip | 4.8% (12 of 248) | 248 | 12 |
| 3x4 with three long strips | 4.2% (3 of 72) | 72 | 3 |
| 6x2 | 10.1% (10 of 99) | 99 | 10 |
| 12x1, one lead per row | 5.4% (4 of 74) | 74 | 4 |
| All layouts | 5.9% (29 of 493) | 493 | 29 |
Data: Photos made by computer of PTB-XL test ECGs (fold 10); thresholds set on the validation photos and applied unchanged. Source: spikes/s3-clean-vs-digitised/RESULTS.md, section 3.4 Abstaining; results/abstain_finetuned_test.csv.
5.9%
of usable photos held for retake (29 of 493)
n = 493 usable photos made by computer
Test S3
Where this is weak
On the photos kept, the gap to the original recordings fell from 0.017 to 0.010 (5 broad groups) and from 0.009 to 0.006 (all patterns). 6x2 pages are held most often, 10.1%.
7 of 500 photos (1.4%) could not be read at all, on top of the 5.9%. A rule set to hold 6% on the validation photos held 5.9% on the test photos, but a rule set for 10% held 13.8%, so the share has to be checked on every new set of photos.
AUROC lost to the photo, by how many leads lost their peaks
A lead counts as cut when the rebuilt trace keeps under 0.8 of the true upward height of the beats. The 5 broad groups.
| Row | AUROC lost | Low end of the range | High end of the range | Photos |
|---|---|---|---|---|
| 0 or 1 lead cut | 0.018 (0.007 to 0.029) | 0.007 | 0.029 | 350 |
| 2 or 3 leads cut | 0.009 (-0.011 to 0.030) | -0.011 | 0.03 | 87 |
| 4 or more leads cut | 0.072 (0.025 to 0.126) | 0.025 | 0.126 | 56 |
Data: Photos made by computer of PTB-XL test ECGs (fold 10); a lead is cut when it keeps under 0.8 of the true upward height. Source: spikes/s3-clean-vs-digitised/RESULTS.md, section 3.1 One cause: the digitiser cuts tall R waves; results/gap_by_cut_leads_baseline_test.csv.
0.072
AUROC lost on photos with 4 or more leads cut, 4 times the loss with 0 or 1 cut
n = 56 photos made by computer
Test S3
Where this is weak
About one lead in ten (9.8%) loses more than a fifth of the height of its beats when the trace is rebuilt, and 56 of 493 photos (11%) have 4 or more such leads. This is the largest single cause of the gap. The middle group, 2 or 3 leads cut, shows no clear difference.
Measuring
Heart rate and QRS width match the reference programs about as well as the programs match each other
We measured heart rate, PR, QRS, QT and QTc on 2,198 test ECGs from Germany and compared each with the values of reference ECG programs. The grey bar is how far apart the two reference programs are from each other, a rough floor for any measurement.
Average error in heart rate
| Row | TraceRead AI | Between the two programs |
|---|---|---|
| Heart rate (beats a minute) | 0.6 | 0.5 |
Data: PTB-XL test ECGs (fold 10), 12 full leads, against the reference ECG programs' values published with PTB-XL+. Source: spikes/s4-measurements/RESULTS.md, sections 3.3, 3.7 and 3.8.
0.6
beats a minute average heart rate error (0.5 between the two reference programs)
n = 2,198 ECGs from Germany
Test S4
Average error in the intervals, in milliseconds
| Row | TraceRead AI | Between the two programs |
|---|---|---|
| PR interval | 12.6 ms | 8.1 ms |
| QRS width | 5.6 ms | 5.2 ms |
| QT interval | 17.2 ms | 9.1 ms |
| QTc, Bazett formula | 16.8 ms | 10.4 ms |
| QTc, Fridericia formula | 16.0 ms | 9.9 ms |
Data: PTB-XL test ECGs (fold 10), 12 full leads, against the reference ECG programs' values published with PTB-XL+. Source: spikes/s4-measurements/RESULTS.md, sections 3.3, 3.7 and 3.8.
5.6 ms
average QRS width error (5.2 ms between the two reference programs)
n = 2,198 ECGs from Germany
Test S4
16.8 ms
average QTc error by the Bazett formula (10.4 ms between the two reference programs)
n = 2,198 ECGs from Germany
Test S4
Where this is weak
PR, QT and QTc are 4.5 to 8.1 ms further from the reference programs than the programs are from each other. The 95% range for QTc by Bazett reaches 17.7 ms. The error is concentrated in a few groups: wide QRS, pacing, atrial fibrillation and fast heart rates, which still have QTc errors of 30 to 50 ms.
PR is scored on the ECGs where the reference program found a P wave. These are 12 full leads; on 3x4 and 6x2 printouts the QTc error by the Bazett formula is 18.0 to 19.2 ms.
The same intervals measured from photos made by computer
Average error against the reference programs, on the original recordings and on the rebuilt traces of the same ECGs.
| Row | Original recordings | Photos |
|---|---|---|
| PR interval | 12.3 ms | 13.2 ms |
| QT interval | 16.8 ms | 18.2 ms |
| QTc, Bazett formula | 19.9 ms | 24.0 ms |
| QTc, Fridericia formula | 18.6 ms | 21.4 ms |
Data: Photos made by computer of PTB-XL test ECGs (fold 10), against the reference ECG programs' values published with PTB-XL+. Source: spikes/s3-clean-vs-digitised/RESULTS.md, section 3.2 Measurements on digitised signals.
2.6
beats a minute average heart rate error on photos (1.0 on the original recordings)
n = 492 photos made by computer
Test S3
Where this is weak
Heart rate gets worse almost entirely on 3x4 pages, where the beat counter misses beats on a long strip that lost its tall peaks. QTc by Bazett rises from 19.9 to 24.0 ms and by Fridericia from 18.6 to 21.4 ms.
Speed and cost
A median ECG takes 7.63 seconds to read, and its compute costs 0.239 US dollars per 1,000 AI first reads
We ran the whole path from photo to AI first read on servers in AWS Cape Town, with no graphics card, on photos made by computer. A vCPU is one slice of a server processor.
Seconds to read one ECG
| Row | 2 vCPU server | 4 vCPU server |
|---|---|---|
| Median ECG | 7.63 s | 4.47 s |
| 95th percentile (19 of 20 ECGs faster) | 7.85 s | 4.75 s |
| 99th percentile | 26.85 s | 15.07 s |
| Slowest ECG | 27.82 s | 15.51 s |
Data: Photos made by computer of ECGs; whole path from photo to AI first read on AWS servers in Cape Town, no graphics card. Source: spikes/s7-inference/RESULTS.md, section 3.7 The rerun with S1's real digitiser.
7.85 s
to read 95 in 100 ECGs on a 2 vCPU, 4 GB server (4.75 s on 4 vCPU)
n = 97 photos made by computer
Test S7
Where this is weak
The 95th percentile hides a tail. 3 of 97 ECGs on the 2 vCPU server took 14.0, 26.8 and 27.8 seconds, two to four times the median, because the page had to be turned and read again. The slowest was 27.82 seconds.
Cost of 1,000 AI first reads, in US dollars
| Row | US dollars per 1,000 AI first reads |
|---|---|
| Compute only, 2 vCPU server | $0.239 |
| Compute only, 4 vCPU server | $0.286 |
| Always on 2 vCPU server, at 3,000 a month | $31.14 |
| Always on 4 vCPU server, at 3,000 a month | $62.28 |
Data: Compute cost from the measured busy seconds per ECG and AWS prices in Cape Town; always on worker priced at 3,000 AI first reads a month (an assumed volume). Source: spikes/s7-inference/RESULTS.md, sections 3.3 and 3.7.
$0.239
compute cost per 1,000 AI first reads on the 2 vCPU server
n = 97 photos made by computer
Test S7
Where this is weak
The always on server is the bill, not the compute. A 2 vCPU server costs $93.42 a month, which is $31.14 per 1,000 at 3,000 a month, because it is busy under 1% of the time. The 3,000 a month is an assumed volume, not a measured one.
Round trip time from one computer in Nigeria to AWS regions
The bar is the median. The thin line runs to the 90th percentile: 9 in 10 requests were faster.
| Row | Median (ms) | Low end of the range | High end of the range | 90th percentile (ms) |
|---|---|---|---|---|
| Cape Town | 109 ms | 109 | 184 | 184 |
| London | 144 ms | 144 | 207 | 207 |
| Milan | 154 ms | 154 | 205 | 205 |
| Ireland | 193 ms | 193 | 235 | 235 |
| Frankfurt | 197 ms | 197 | 237 | 237 |
| UAE | 257 ms | 257 | 303 | 303 |
| N. Virginia | 276 ms | 276 | 360 | 360 |
| Mumbai | 359 ms | 359 | 410 | 410 |
Data: One Mac on a fixed line in Abuja, 30 requests to each AWS region (about one network round trip, in milliseconds). Source: spikes/s5-aws-region/RESULTS.md, section 3.3 Latency from the Mac in Abuja.
109 ms
median round trip to Cape Town, the nearest of nine regions (184 ms at the 90th percentile)
n = 30 requests
Test S5
Where this is weak
Bahrain did not answer in any of 30 requests. A request through CloudFront's Lagos edge took 48 ms at the median (105 ms at the 90th percentile). These come from one Mac on a fixed line. Phones on mobile networks were not measured.
Limits we measured
Where TraceRead falls short, as measured
Each of these is a measured result, shown with the number of ECGs it comes from. They are facts about what was measured, so read them before relying on any other number on this page.
How well TraceRead AI told ST elevation apart on ECGs from China
ST elevation is a sign of a heart attack. This is the nearest thing TraceRead AI learned to it.
| Row | AUROC | Low end of the range | High end of the range |
|---|---|---|---|
| ECGs from China, 12 full leads | 0.605 | 0.562 | 0.646 |
| ECGs from China, 3x4 printout | 0.623 | 0.584 | 0.665 |
| ECGs from China, read in 2.5 second windows | 0.630 | 0.59 | 0.669 |
| Inside PTB-XL (the same pattern) | 0.909 |
Data: Chapman Shaoxing Ningbo ECGs (China); the nearest thing TraceRead AI learned to ST elevation, scored against Chapman's ST elevation label. Source: spikes/s2-classifier-baseline/RESULTS.md, section 3 External test on Chapman and ST elevation.
0.61 to 0.63
AUROC for ST elevation on ECGs from China, where 0.50 is a coin toss
n = 176 ECGs from China
Test S2
Where this is weak
The same pattern scored 0.909 inside PTB-XL, so it is learnable there but does not carry over to ST elevation in China. We make no claim that TraceRead AI finds ST elevation.
Share of ST elevation ECGs a fixed measuring rule found
A rule that measures the ST segment with fixed code, run once on ECGs from China. The second line of each row is the share of other ECGs it also flagged.
| Row | Share found (percent) | Low end of the range | High end of the range | Low end | High end | Other ECGs also flagged |
|---|---|---|---|---|---|---|
| 12 full leads | 61.9% | 54.3 | 69.1 | 54.3 | 69.1 | 5.3% |
| 3x4 printout | 50.0% | 42.4 | 57.6 | 42.4 | 57.6 | 6.2% |
| Photo made by computer of a 3x4 printout | 50.9% | 43.1 | 58.6 | 43.1 | 58.6 | 6.2% |
| 12 full leads | 89.8% | 84.3 | 93.8 | 84.3 | 93.8 | 22.0% |
| 3x4 printout | 81.2% | 74.7 | 86.7 | 74.7 | 86.7 | 30.2% |
| Photo made by computer of a 3x4 printout | 80.7% | 74 | 86.3 | 74 | 86.3 | 32.0% |
Data: Chapman Shaoxing Ningbo ECGs (China): 176 labelled ST elevation and 2,000 others; photos made by computer of 3x4 printouts of 176 and 500 of them. Source: spikes/s4-measurements/RESULTS.md, section 3.5; spikes/s3-clean-vs-digitised/RESULTS.md, section 3.5.
61.9%
of ST elevation ECGs found at the definition thresholds on 12 full leads (50.0% on a 3x4 printout, 50.9% on photos)
n = 176 ECGs from China
Test S4 and test S3
Where this is weak
The more sensitive setting reaches 89.8% on 12 full leads but flags 22.0% of other ECGs; on a printout 81.2% and 30.2%; on photos 80.7% and 32.0%. Photos were measurable for 171 of the 176 ST elevation ECGs. Digitising is not the cause of the misses: the clean signal gave 50.0%.
Share of urgent heart rate and QTc patterns the safety net rules found
Simple rules on the measures, judged against the same rule on a reference ECG program's numbers, on every ECG in PTB-XL.
| Row | Share found (percent) | Low end of the range | High end of the range | Found | Out of | Low end | High end | Extra alarms | On the 2,198 test ECGs |
|---|---|---|---|---|---|---|---|---|---|
| Very fast heart rate | 73 of 91 (80.2%) | 70.9 | 87.1 | 73 | 91 | 70.9 | 87.1 | 1 | 8 of 11 |
| Wide and fast beats | 33 of 52 (63.5%) | 49.9 | 75.2 | 33 | 52 | 49.9 | 75.2 | 9 | 3 of 5 |
| Very slow heart rate | 12 of 34 (35.3%) | 21.5 | 52.1 | 12 | 34 | 21.5 | 52.1 | 1 | 2 of 5 |
| Very long QTc | 10 of 291 (3.4%) | 1.9 | 6.2 | 10 | 291 | 1.9 | 6.2 | 4 | 0 of 48 |
Data: PTB-XL ECGs from Germany, original recordings; each rule judged against the same rule on a reference ECG program's numbers. Source: spikes/s4-measurements/RESULTS.md, sections 3.6 and 3.9.
3.4%
of very long QTc patterns found (10 of 291)
n = 21,799 ECGs from Germany
Test S4
Where this is weak
The rules find 80% of very fast heart rates and 35% of very slow ones, but only 3% of very long QTc. They cannot be relied on to catch these. On the 2,198 test ECGs alone the counts are 8 of 11, 3 of 5, 2 of 5 and 0 of 48.
The two reference programs disagree most about long QTc, so there is no solid reference for that rule.
Slow drift in the baseline cost 0.174 AUROC on ten second ECGs from Brazil
Some ECGs wander up and down slowly while they are recorded. TraceRead AI as first trained, on PTB-XL only, read these ten second ECGs worse until the drift was filtered out.
| Row | AUROC | Low end of the range | High end of the range |
|---|---|---|---|
| Slow drift left in | 0.806 | 0.771 | 0.838 |
| Slow drift filtered out | 0.980 | 0.972 | 0.987 |
Data: CODE-15% ECGs (Brazil), validation exams of ten seconds; macro AUROC over the six labels CODE-15% defines, TraceRead AI trained on PTB-XL only. Source: spikes/s2-classifier-baseline/RESULTS.md, part 2 section P2.4 Baseline wander explains most of what part 1 gets wrong on CODE-15%.
0.806
AUROC with the drift left in (0.980 filtered out)
n = 3,010 ECGs from Brazil
Test S2, part 2
Where this is weak
A version trained with the Brazilian ECGs added scored 0.981 on the same ECGs without the filter, so the drift mostly stopped mattering once it was in training.
Every ECG came from a public research dataset
Every photo result on this page comes from photos made by computer of public research ECGs recorded in Germany (PTB-XL), China (Chapman Shaoxing Ningbo) and Brazil (CODE-15%). None was recorded in Nigeria. No real phone photo has been scored.
An AI first read is not the last word.
- A qualified doctor reviews every AI first read. Nothing reaches a patient until a doctor signs it.
- Do not rely on TraceRead alone in an emergency.
- A normal AI first read does not rule out heart disease.
- For adults, 18 and over.
Sources
Where every number comes from
Each chart names its result file and section under it. These are the files, in the TraceRead repository, that hold the numbers.
Spike S1, reading the paper
- spikes/s1-digitisation/RESULTS.md
- eval/levelC_test_per_lead.csv, levelC_test_per_layout.csv, levelA_test_per_lead.csv and levelA_test_per_layout.csv
Spike S2, finding patterns
- spikes/s2-classifier-baseline/RESULTS.md
- results/benchmark_fold10.csv
- results/chapman_auroc_per_label.csv
- results/traceread_layouts_fold10.csv
- results/layout_cost_fold10.csv
Spike S3, photos against original recordings
- spikes/s3-clean-vs-digitised/RESULTS.md
- results/gap_baseline_test_main.csv
- results/gap_finetuned_test_main.csv
- results/gap_by_cut_leads_baseline_test.csv
- results/abstain_finetuned_test.csv
- results/st_path.csv
Spike S4, measuring
- spikes/s4-measurements/RESULTS.md
Spike S5, the AWS region
- spikes/s5-aws-region/RESULTS.md
Spike S6, capture
- spikes/s6-capture/RESULTS.md
Spike S7, speed and cost
- spikes/s7-inference/RESULTS.md
The 12 case study runs
- apps/web/src/content/runs/index.json
The public datasets, their versions and licences are on the credits page. Every AI first read is for a qualified clinician to review.
See TraceRead read a paper ECG
We show you TraceRead reading public research ECGs, step by step, and talk through how your team prints, photographs and reads ECGs today.