Model providers retire versions on their own schedule. When the date lands on a version your product depends on, it becomes a release you did not plan, on a deadline someone else set. Teams handle it well when they treat a model swap as routine maintenance instead of an incident.
What pinning a model version actually does
Pinning one version is still the right default. It gives you reproducible behavior and a known retirement date. It does not delay the notice. A pinned version is a countdown, and the migration work costs about the same whether it starts at ninety days or ten.
The quiet failure is an old pin that nobody owns. The feature keeps working for a year, the upgrade never gets scheduled, and the first test against the replacement happens in the last two weeks of the window, when the team has the least room to change anything.
What belongs in a frozen eval set
Keep a small set of real cases, frozen in version control, that you can run against a candidate model with one command. Fifty to a few hundred cases is usually enough to catch a bad swap, because the value is coverage of your ugly paths, not the size of the file.
Include the cases that broke before: the long input that sits near the context limit, the record with missing fields, the request that must return a fixed schema, and the one that must escalate instead of answering. Add a few ordinary cases to catch broad regressions. Score every case the same way each time, and keep the last accepted outputs so a reviewer can compare the old and new responses side by side.
What usually moves in a swap
Same prompt, new version, several things can shift at once. Output length drifts, so a summary the interface expects in two sentences arrives as six. Structured output lands close but not exact, and a parser drops rows it used to keep. Tool-call cadence changes, which moves latency and cost even when the answer still looks fine. A request that used to escalate gets answered directly.
None of that is exotic. A model revision is a real change to a component your team did not write, so it deserves the same treatment as a dependency upgrade with a version bump.
Which migration strategy fits the feature
- Pin and migrate deliberately. Old behavior holds until you choose to move. Costs an owner, a frozen set, and a migration each cycle. Fits when output shape, review, or compliance is part of the contract.
- Follow the latest version. Most of the migration work disappears. Changes then arrive without a release of yours, and every downstream step has to tolerate drift. Fits internal tools where people read the output.
- Route by task. Pinned behavior where it is contractual, current models where it is exploratory. Costs two paths to evaluate, monitor, and document. Fits products that mix customer-facing and internal AI work.
Whichever row you pick, name the person who owns the version and the date for the next move. An unowned pin is how a routine upgrade turns into a fire drill.
A migration window that works
- Log the retirement date and work backwards. Assume it can move earlier, not later.
- Run the frozen set against the replacement and diff the outputs before touching the prompt.
- Canary a small share of traffic while the old path is still reachable.
- Watch acceptance rate, review rate, latency, and cost per completed task, not just error rate.
- Keep the rollback until traffic has run on the new version long enough to cover a slow week.
Suppose a support-summary feature on a pinned version gets a ninety-day notice, and the team runs its frozen set in week two. The set flags two schema failures and one escalation that no longer fires. They fix the prompt, canary in week five, and roll forward with the rollback still in place. A team that starts the same work in week twelve finds the same failures with no room left to test a fix. The dates and results here are an illustrative scenario, not a measured outcome.
A deprecation only turns into a scramble when the swap cannot be tested or undone. Keep both of those cheap and the notice becomes a calendar item.
If this is the kind of problem your team is working through, I am always glad to compare notes.