A Hardened Release Pipeline Was Defeated In Two Hours: What The LiteLLM Supply Chain Attack Shows

On 21 October 2026, LiteLLM, the open-source AI gateway that many teams run in front of every model call they make, published a poisoned release. Version 1.107.0 of the Python package and four container tags carried compiled malware that stole credentials and pulled remote code, and the package was live on PyPI for about an hour and forty-four minutes. The company pulled it, published an incident report, and by the end of the day had a timeline of how a pipeline it had deliberately hardened six months earlier was undone.

The details are worth reading closely because they describe a class of failure that hardening alone does not prevent. The attacker never needed a vulnerability in LiteLLM's code. They needed three sets of developer credentials, a temporarily widened permission, and a repository name that looked almost right.

This article reconstructs the incident from LiteLLM's own published report, covers who is affected and how to check, walks the remediation order and why the order matters, and draws out the six lessons that transfer to any team shipping software through a modern CI pipeline. Where the vendor's own account was later revised, that is stated.

What was published, and for how long

The artefacts are documented precisely, which makes the exposure question answerable rather than a matter of guesswork.

On PyPI, litellm==1.107.0 was published as seven wheels and was live from 11:39:51 to 13:23 UTC. In the container registries, the tags v1.107.0, latest, main-latest and main-stable pointed at a compromised image index, sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e on Docker Hub from 11:46:56 to 14:20 UTC, and on GitHub's registry until the tags were made unavailable at 13:51 UTC.

Two details in that paragraph decide who is at risk. The first is the tags: latest, main-latest and main-stable are the names a deployment picks up when nobody pinned a version, so a container restart during the window was enough. The second is that LiteLLM states the official v1.107.0 source tag in GitHub does not contain the malicious files, because the compromised artefacts were built from an attacker-controlled repository. The tag was clean. The shipped binaries were not.

LiteLLM also states that enterprise customers on its private enterprise registry were not affected, since that is a separate distribution path.

How a hardened pipeline was defeated

LiteLLM had already been through a compromise in March 2026, and its account describes the hardening that followed: releases moved to a private repository, a strict set of workflows, publication through Trusted GitHub Publishers, immutable Docker release tags, and signed container images. The normal release flow is deliberately awkward, which is the point.

A release candidate branch and tag are cut on Friday at 3am and do not change. The candidate is qualified through two sets of integration tests, one public and one private. Artefacts are built by a workflow in a private repository, which only an engineer with write access can trigger. The artefacts are deployed to a testing environment and tested again. Publication itself runs through Trusted GitHub Publishers, and can only be performed by an account locked to a specific laptop physically located in the office. No Docker image is rebuilt at publication time.

Four steps undid that, and each is mundane on its own.

Permission was temporarily widened. On the Monday before the incident, LiteLLM expanded release permissions so two internal engineers could publish daily bugfix backports for customers. That is the kind of accommodation every team makes under customer pressure, and it is the window the attack used.

The attacker held three sets of developer GitHub CLI credentials. One of them could run GitHub workflows. That token was not a production secret in the usual sense; it was a developer convenience credential that happened to carry release authority.

Branch protection was overridden with an owner token. The attacker first tried to merge a change that would let the releaser point at a different source repository, and the protections blocked it. One of the tokens belonged to a GitHub organisation owner, who used it to disable branch protections and force push the change. The pipeline's protections were real, and one over-privileged identity walked around them.

Publication ran against a lookalike repository. The compromised workflow was pointed at Berri-AI/litellm instead of BerriAI/litellm, a difference of one character and one hyphen, and that repository signed and published the poisoned artefacts. Signing did what signing does: it attested that these artefacts came from the pipeline, which by then was the attacker's pipeline.

The timeline in the report records the sequence to the second, including the deletion of the production and staging workflow records at 11:53, four minutes after the wheels landed on PyPI, which limited the logs available for the investigation. That is not a detail about this attacker so much as a property of the environment: whoever can run your workflows can usually also clean up after themselves.

Where the credentials came from

The most uncomfortable part of the report is the token path, because it implicates ordinary logging.

LiteLLM states that it observed specific situations in which Claude Code embeds a user's GitHub CLI OAuth token into a response body, and that where response bodies are logged, those tokens can appear in LiteLLM logs. The company also found an unexpected, innocuously named user account in its own internal LiteLLM deployment, which was exposed to the public internet. Its assessment is that the attacker got into that deployment, pulled the GitHub CLI OAuth tokens out of its logs, and used them to compromise the release workflow. It has not confirmed how the attacker gained access to the deployment and suspects a recently patched vulnerability.

Two lessons sit in that paragraph, and neither is about LiteLLM's code. First, an AI gateway is a credential aggregation point by design: it holds provider keys, budgets and routing configuration, and it logs prompts and responses for observability, which means it can also log whatever secrets those bodies contain. Second, an internal deployment with a friendly name is still an internet-facing service if it answers on a public address. The same pattern appeared in the exposed agent dashboard campaign we covered, where the operator's own control panels were the leak.

Who is affected, and how to check

LiteLLM's guidance is direct: if you used an unpinned version between 11:39 and 13:23 UTC through pip, or started one of the four container tags between 11:46 and 14:20 UTC, assume the environment is compromised.

Three checks are published.

• pip show litellm in the environment, to see whether 1.107.0 was installed.
• The image digest, if a container ran sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e, it is compromised.
• Filesystem evidence: litellm-1.107.0.dist-info, cook.pth and sitecustomize.py. Finding one confirms exposure, and the report is explicit that not finding one does not prove the environment is clean.

That last sentence is the one to carry into any incident like this. Absence of evidence in a container that has been rebuilt, or in a temporary directory that has been cleaned, means very little.

The remediation order, and why it is not optional

The removal instructions came from LidorMachluf, the researcher who helped detect the incident, and their first line is counterintuitive: do not start with pip install.

The reason is the infection mechanism. The malicious package used Python's start-up hooks, cook.pth and sitecustomize.py, so any Python start in the affected environment runs them again, and that includes running pip. Cleaning up with the tool that re-triggers the malware is how people end up doing this twice.

The published order is:

1. From the shell, delete cook.pth, cook.py, sitecustomize.py and any matching __pycache__/cook*.pyc and sitecustomize*.pyc from every affected site-packages directory.
2. Kill the detached Python daemon, which holds /tmp/remote_code_loader.lock.
3. Reinstall pinned to litellm==1.106.2, or better, rebuild the image and the virtual environment from scratch.
4. If 1.107.0 was ever installed on a host with outbound internet access, assume the exfiltration succeeded. The marker file confirms it if present, and its absence proves nothing, because containers and temporary cleanup remove it.
5. Rotate everything the host could reach: the instance or task IAM role, every provider key stored in the LiteLLM proxy database plus the master and virtual keys, the database password, and any secret in the environment of any process on that host.
6. Check CloudTrail for GetCallerIdentity calls from the instance role around the install time, then for use of that role's credentials from outside your infrastructure.

The credential list is the part teams underestimate. A LiteLLM deployment is not a library in one project; it is a gateway holding the keys to every model provider the company uses, plus the master key that can mint new ones. If it ran the poisoned version with internet access, the correct assumption is that those keys are in someone else's hands, and the rotation is not optional.

Six lessons that transfer

Pin every dependency, including the ones you trust. The latest tags were compromised for two and a half hours, and the wheels for under two. Every deployment that resolves a moving tag has a window like that in its history. Pinning is not bureaucracy; it is the difference between a version number and a lottery.

Treat logs as a credential store. Response bodies captured for observability can contain tokens that tools embedded by accident. The specific finding here, that a coding agent wrote a GitHub CLI OAuth token into a response body which then landed in a gateway's logs, is the kind of leak no scanner will flag. Treat log access with the same care as secret access, and redact aggressively at the edge. The credential theft pattern in our coverage of breached coding agents is the same failure in a different system: the secret was never stolen from a vault, it was printed into a place nobody thought of as sensitive.

Temporary permission widening is a real attack surface. The incident window opened the day after release permissions were expanded for backport publishing. Every “just for this week” grant is an authorisation decision, and the review that would catch it usually happens after the fact.

An organisation owner is a single point of failure. Branch protection held until a token with owner rights arrived. Whatever your protections, the identity that can disable them is the real control boundary, and it should be the most tightly held credential in the company, not the most convenient.

Signing proves provenance, not intent. The poisoned artefacts were signed, because they were built by the pipeline that was supposed to build them. Signature verification tells you the artefact came from your release process; it cannot tell you that your release process was still yours.

Write the incident report while you are still wrong, then correct it. LiteLLM published its account on the day, then added a note on 23 October stating that some of its assumptions and beliefs had been inaccurate or incomplete, that the original text was being left in place as a record, and that a canonical version would follow. That is a better model than a single confident retelling: an incident timeline is a working document, and the revision history is part of the evidence.

Two more practices follow from the shape of this incident, and they are cheap. Reduce the blast radius of a gateway by scoping provider keys per team and per budget rather than issuing one master credential, which is the same argument our guide to building production-grade agent systems makes for agent design. And assume that anything you deploy will eventually run a bad package, so that recovering from one is a rehearsal you have already done.

Where the numbers stop

• The account of how the pipeline was defeated is LiteLLM's own, published the same day and explicitly revised two days later as partly inaccurate. I could not retrieve the later canonical record in this run, so this article relies on the original post as revised by its own update notes, and the vendor's own caveat is stated rather than smoothed over.
• The claim that Claude Code embeds GitHub CLI OAuth tokens into response bodies is LiteLLM's observation from its investigation. I did not verify it independently, and it is presented as a finding about specific situations rather than a general property of the tool.
• The suspected entry point into LiteLLM's internal deployment, a recently patched vulnerability, is described by the company as a suspicion, not a confirmed fact.
• The malware is described by function, credential theft and remote code loading, and this article does not reproduce any payload detail beyond the file names the vendor published for detection purposes.
• I have not tested any of the checks, and none of the remediation steps here were executed on a live LiteLLM installation.

FAQ

What happened in the LiteLLM supply chain attack?

On 21 October 2026, LiteLLM's release pipeline was compromised and published version 1.107.0 to PyPI and to Docker Hub and GHCR tags including latest. The package contained compiled malware that stole credentials and loaded remote code. The PyPI wheels were live for about an hour and forty-four minutes, the container tags for up to two and a half hours, and enterprise customers on the private registry were not affected.

Am I affected if I use LiteLLM?

You are affected if you installed an unpinned LiteLLM through pip between 11:39 and 13:23 UTC on 21 October 2026, or started one of the v1.107.0, latest, main-latest or main-stable container tags between 11:46 and 14:20 UTC. Check with pip show litellm, check the image digest, and look for litellm-1.107.0.dist-info, cook.pth and sitecustomize.py. Finding one of those confirms exposure, and not finding one does not prove the environment is clean.

What should I do first if I am affected?

Do not start with a pip command. The malware used Python start-up hooks, so any Python process in the affected environment, including pip, runs it again. Delete the hook files from every affected site-packages, kill the detached Python daemon holding /tmp/remote_code_loader.lock, then reinstall pinned to 1.106.2 or rebuild the environment from scratch, then rotate every credential the host could reach, including the IAM role, the provider keys in the proxy database, the master and virtual keys, and the database password.

The short version

LiteLLM shipped a poisoned 1.107.0 on 21 October 2026 after an attacker used three sets of developer GitHub CLI credentials, one of them an organisation owner token, to disable branch protections and run a release workflow pointed at a lookalike repository. The wheels were live for under two hours and the container tags, including latest, for up to two and a half. The entry point was LiteLLM's own internal deployment, exposed to the internet under an innocuous name, whose logs contained GitHub OAuth tokens that a coding agent had written into response bodies. The remediation order matters as much as the rotation: strip the Python start-up hooks before running anything, kill the resident daemon, rebuild pinned, then assume every credential the host could reach is compromised. The transferable lessons are pinning, treating logs as secrets, watching temporary permission grants, guarding owner identities, and remembering that a signature proves provenance rather than intent.

Sources

• LiteLLM, Contained: Oct 21, 2026 LiteLLM release v1.107.0 Compromised, covering the affected artefacts and windows, the investigation into the pipeline compromise, the token path through logs, the exposure checks and the remediation order, with a 23 October note that parts of the original account were later found inaccurate: https://docs.litellm.ai/blog/security-incident-oct-21-2026