Logo do repositório
 
Publicação

Large language models approach clinician performance in ESC cardiovascular risk stratification: a vignette-based benchmark study

2026-06-01Artigo de investigação Acesso aberto
Ver registo simples
dc.contributor.authorSantos, José Ferreira
dc.contributor.authorDuarte, Regina de Brito
dc.contributor.authorMota, Inês
dc.contributor.authorSantos, Rita Carvalheira
dc.contributor.authorMoreira, José Maria
dc.contributor.authorCampos, Joana
dc.contributor.authorSilva, Nuno André
dc.contributor.authorNeves, Bernardo
dc.contributor.authorLadeiras-Lopes, Ricardo
dc.contributor.authorLeite, Francisca
dc.contributor.authorDores, Hélder
dc.date.accessioned2026-06-23T16:11:50Z
dc.date.available2026-06-23T16:11:50Z
dc.date.issued2026-06-01
dc.description.abstractAims: Guideline-based cardiovascular risk stratification requires three distinct competencies: extracting risk factor data from clinical text, computing a validated risk score, and applying guideline-defined thresholds to assign a final risk category. We evaluated contemporary large language models (LLMs) on each of these tasks within the European Society of Cardiology (ESC) SCORE2 framework and compared LLM performance against a pooled individual clinician benchmark to contextualize findings against real-world human reproducibility. Methods and results: Eleven LLMs were evaluated using 30 simulated outpatient clinical vignettes presented in both Portuguese and English. For each vignette, models extracted cardiovascular risk factors, determined SCORE2 applicability, generated 10-year risk estimates where appropriate, and assigned a final three-class ESC risk category. A committee of three cardiologists established the reference standard; eight independent clinicians provided an individual-level human benchmark. Traditional risk-factor extraction was near-perfect across all models (micro-F1 0.97–0.99). Agreement with expert-assigned final risk categories was moderate and variable (best: GPT-4o, quadratic-weighted κw 0.69, 95% CI 0.44–0.84), with 10 of 11 models more often underestimating than overestimating risk. To isolate the source of classification error, post hoc deterministic recalculation of SCORE2 was performed using model-extracted variables in eligible vignettes; this markedly improved agreement across all models (κw 0.85–0.90), demonstrating that extraction was largely intact and computational execution was the primary failure mode. The pooled individual clinician benchmark showed moderate agreement with the reference standard (κw 0.52, 95% CI 0.28–0.67), indicating that the best-performing LLMs matched or exceeded the average individual clinician on this guideline-based task. Performance was broadly consistent across Portuguese and English. Conclusion: Contemporary LLMs reliably extract cardiovascular risk information from clinical text, and the best-performing systems achieved agreement within the range of average individual clinicians on this structured task. Their principal limitation lies in downstream computation and rule application.eng
dc.identifier.doi10.1093/ehjdh/ztag073
dc.identifier.eid105041854433
dc.identifier.other734eca63-ab9a-4d6d-a1b7-fb26fe12095e
dc.identifier.pmcPMC13215473
dc.identifier.pmid42211774
dc.identifier.urihttp://hdl.handle.net/10400.14/58240
dc.identifier.wos001793036100001
dc.language.isoeng
dc.peerreviewedyes
dc.publisherOxford University Press
dc.rights.urihttp://creativecommons.org/licenses/by/4.0/
dc.subjectArtificial intelligenceeng
dc.subjectCardiovascular preventioneng
dc.subjectClinical decision supporteng
dc.subjectLarge language modelseng
dc.subjectRisk stratificationeng
dc.subjectSCORE2eng
dc.titleLarge language models approach clinician performance in ESC cardiovascular risk stratification: a vignette-based benchmark study
dc.typeresearch article
dspace.entity.typePublication
oaire.citation.issue5
oaire.citation.volume7
oaire.versionhttp://purl.org/coar/version/c_970fb48d4fbd8a85

Ficheiros

Principais
A mostrar 1 - 1 de 1
Miniatura
Nome:
152988440.pdf
Tamanho:
1.14 MB
Formato:
Adobe Portable Document Format
Ver/Abrir

Coleções

CIIS - Contribuições em Revistas Científicas / Contribution to Journals