The FDA has authorized roughly 1,450 AI-enabled medical devices, the large majority of them through the 510(k) pathway. Every one shares a property that rarely appears in the marketing material. The model stopped learning on the day it was submitted.
This is not an oversight. It is how the regulatory system was built. A cleared device has to remain the device that was evaluated, and a model whose weights change is arguably a different device. Freezing the model is part of what made clearance possible.
The difficulty is that surgery does not freeze alongside it.
What Freezing Actually Means
A trained model is a set of numerical weights. Those weights encode everything the model learned from its training data, and at submission they are locked. A model deployed in 2026 will return the same output for the same imaging study in 2033. It cannot notice that anything has changed, because noticing would require updating, and updating is the thing that was ruled out.
Practice Moves and the Model Does Not
Three kinds of change accumulate after clearance. Technique and technology move first: new interbody devices, different fixation strategies, minimally invasive approaches that alter the relationship between preoperative anatomy and eventual outcome. Image acquisition moves next, as scanners are replaced, sequences are retuned, and protocols are standardized across a health system. Populations move last and most quietly, as referral patterns shift, indications broaden, and the patients arriving in clinic stop resembling the ones in the training set.
Finlayson and colleagues describe this collectively as dataset shift and argue that clinicians should treat it as a routine property of deployed models rather than an unusual event [1]. The model was not wrong about the world it was trained on. That world moved.
The Decay Is Silent
What makes this dangerous is not that performance declines. It is that the decline does not appear in the number most institutions monitor.
Davis and colleagues built seven models for hospital-acquired acute kidney injury on a national cohort, then validated each of them over the following nine years. Discrimination held across every model. Calibration did not. All seven increasingly overpredicted risk as time passed [2].
Discrimination is the ability to rank, and the models kept ranking correctly, placing higher risk patients above lower risk ones. Calibration is whether the numbers mean anything, whether seventy percent corresponds to seventy percent. Area under the curve measures the first and is silent about the second. An institution watching AUC alone would have seen a flat line for nine years while the probabilities drifted steadily away from reality.
For a surgical planning tool the consequence is specific. The tool keeps ordering cases sensibly, so it continues to feel useful. Meanwhile the durability estimate quoted in a consent conversation grows progressively wrong, and nothing in the interface says so.
Why the Obvious Fix Does Not Work
The intuitive remedy is to feed the model recent cases. That introduces a different failure.
Neural networks trained sequentially on new data tend to overwrite what they previously learned, a behavior known as catastrophic forgetting. A model fine-tuned on the last two years of cases can lose competence on presentations it once handled well, especially rare ones that appear infrequently in any recent window. Kirkpatrick and colleagues showed the problem is tractable, introducing a method that slows learning on the weights most important to earlier tasks so new learning does not erase old competence [3]. Related approaches replay stored examples from earlier periods during training.
The implication for surgery is uncomfortable. Retraining on recent volume alone will improve performance on the routine case and quietly degrade it on the revision, the deformity, and the patient with poor bone quality. Those are the cases where a durability model has the most to contribute.
The Regulatory Tension, and How It Was Resolved
Until recently a manufacturer who wanted to update a cleared model faced a choice between leaving it frozen and filing a new submission for every change. Neither option produces a model that keeps pace with practice.
Congress addressed this in the Food and Drug Omnibus Reform Act of 2022, which added Section 515C to the Food, Drug, and Cosmetic Act and established the predetermined change control plan. The FDA published final guidance for AI-enabled device software functions in December 2024 [4].
A PCCP is submitted and reviewed alongside the original marketing application. It specifies three things: the modifications the manufacturer intends to make, the methodology used to develop, validate, and implement them, and an assessment of the impact those modifications will have. Changes that fall inside the approved plan can be made without a new submission. Changes outside it cannot.
This is not a loophole and it does not make a model self-updating. It moves the review earlier, requiring a manufacturer to describe in advance how the model will be permitted to change and how that change will be verified. The discipline is the point. A team that cannot specify its retraining and validation procedure precisely enough to submit it probably should not be retraining.
What Good Practice Looks Like
Several practices follow from the above, and none of them require new science.
Monitor calibration rather than discrimination alone. Given the Davis result, this is the highest yield change available and most institutions are not doing it. Monitor at the level of the individual site, because a health system average will mask a single hospital whose scanner or population has moved. Hold back a locked benchmark set that no update is allowed to touch, so that every proposed change is measured against a fixed reference rather than against the data that motivated it. Preserve rare presentations explicitly during any update, through replay or regularization, rather than assuming the model will retain them. And write the change control plan before clearance rather than discovering afterward that the model cannot legally be improved.
One practice is harder for a single institution to achieve. Distinguishing genuine drift from ordinary noise requires a comparison point, and a hospital watching only itself does not have one. Federated learning offers a structure for this, allowing each site to train locally and share only model updates rather than patient data [5]. A network assembled this way can see that a shift at one institution is local rather than universal, and it holds enough combined volume that rare presentations are not lost when the model learns something new.
How SDI Is Approaching This
Surgeon Decision Intelligence treats the update pathway as part of the submission rather than as a problem to solve after clearance. The intent is a change control plan that names the monitoring signals, the conditions that trigger a retrain, the procedure used, and the validation the updated model must pass before it reaches a surgeon.
Calibration monitoring is designed as a reported output rather than an internal diagnostic, tracked per site, so that a hospital can see whether the numbers still mean what they said when the tool was installed. Multi-institution collaboration through a federated structure is how the platform is intended to observe drift across sites and to retain uncommon presentations during updating.
References
[1] Finlayson SG, Subbaswamy A, Singh K, et al. The Clinician and Dataset Shift in Artificial Intelligence. N Engl J Med. 2021;385(3):283-286.
[2] Davis SE, Lasko TA, Chen G, Siew ED, Matheny ME. Calibration drift in regression and machine learning models for acute kidney injury. J Am Med Inform Assoc. 2017;24(6):1052-1061.
[3] Kirkpatrick J, Pascanu R, Rabinowitz N, et al. Overcoming catastrophic forgetting in neural networks. Proc Natl Acad Sci U S A. 2017;114(13):3521-3526.
[4] U.S. Food and Drug Administration. Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions. Final guidance. December 2024. Authority established by the Food and Drug Omnibus Reform Act of 2022, Section 515C of the FD&C Act.
[5] Rieke N, Hancox J, Li W, et al. The future of digital health with federated learning. npj Digit Med. 2020;3:119.