Workshops Archive

PathLlama: Domain-Adapted Language Models for Cancer Surveillance at Scale


Workshop: The 12th Computational Approaches for Cancer Workshop (CAFCW26)

Authors: Abhishek Shivanna, Patrycja Krawczuk, and John Gounley (Oak Ridge National Laboratory (ORNL)); Jamaludin Mohd-Yusof (Los Alamos National Laboratory (LANL)); and Mayanka Chandrashekar and Heidi A. Hanson (Oak Ridge National Laboratory (ORNL))

Abstract: Population-based cancer surveillance depends on converting narrative pathology reports into structured tumor characteristics. Automated coding must handle ambiguous reports and rare tumor categories while allowing difficult cases to remain under expert review [1]. We investigate whether domain-specific continual pre-training improves cancer registry coding by a general-purpose language model with tunable abstention.

We adapted Meta Llama 3 8B [2] on 16.1 million clinical and cancer-registry records containing approximately 7.0 billion words from the Surveillance, Epidemiology, and End Results (SEER) program, MIMIC-III [3], and MIMIC-CXR [4]. The supervised dataset comprised 4,054,951 pathology reports from six SEER registries, with case-level splits. Five classification heads predict site, subsite, laterality, histology, and behavior, each with an additional abstention output. Development and evaluation ran within the secure CITADEL environment on Frontier. Supervised fine-tuning used Fully Sharded Data Parallelism [5], bfloat16 precision, partial layer freezing, and 64 nodes with 256 AMD MI250X modules.

A Deep Abstaining Classifier-inspired objective [6] learns when to defer each task. During training, a validation controller adjusts each task’s abstention penalty toward a chosen accuracy target among accepted predictions. The DAC is a tunable parameter and we tested it at both the 95% and 97.5% target configurations. At inference, the highest-scoring output determines the label or abstention, without confidence thresholding.

We compared domain-adapted PathLlama-adap with PathLlama-base, the same backbone fine-tuned directly using identical supervised protocols, on 395,299 held-out reports from five registries. The 95% target configuration used random sampling; the 97.5% configuration used sequential sampling, intended to stabilize controller updates through deterministic record coverage. Target and sampler effects were not tested separately. Comparisons were made within each configuration; targets were not guarantees of test accuracy.

PathLlama-adap produced more correct automatic labels per test report in all ten task/configuration comparisons and deferred fewer reports in nine, while accuracy among accepted predictions differed by at most 0.4 percentage points. For histology at the 95% target, abstention fell from 55.8% with PathLlama-base to 53.4% with PathLlama-adap, and macro F1 over the same 566 test classes increased from 0.4541 to 0.4782. At the 97.5% target, abstention fell from 75.5% to 73.7%, and this F1 increased from 0.3145 to 0.3366. For rare histology classes (11–50 test reports each), the mean fraction correctly auto-coded per class was 22.4% with PathLlama-adap versus 3.8% with the Hierarchical Self-Attention Network (HiSAN) at the 95% target. This secondary benchmark does not isolate continual pre-training.

Domain adaptation was associated with more correct automated cancer labels while maintaining similar accuracy among accepted predictions. Tunable abstention provides a mechanism for balancing automated coding with expert review. Findings remain descriptive and internal: each arm had one run, all test registries contributed training data, corpus overlap was unquantified, and additional pre-training exposure was not controlled. External, replicated evaluation is needed before operational use.


Back to The 12th Computational Approaches for Cancer Workshop (CAFCW26) Archive Listing

Back to Full Workshop Archive Listing