← All posts

The Breach Wasn't Where the Headline Said It Was

On Wednesday of this week, roughly 2,500 companies learned they might be victims of what was being called the largest AI supply chain breach of the year. By Friday, a second research firm had published a timeline showing that about 95 percent of them were already compromised five days before the AI product in the headline was ever touched.

Both findings are probably right. That is the part worth your attention.

Forty Minutes That Weren't the Whole Story

Here is the version that ran everywhere first. On March 24, at 10:39 UTC, two poisoned versions of a Python package called LiteLLM went live on the public package index. LiteLLM is an AI gateway — the piece of plumbing that lets an application talk to several different AI model providers through one connection. The malicious versions sat there for about forty minutes before being quarantined. LiteLLM's own advisory tells customers to treat any install that day, right up until 16:00 UTC, as suspect.

Forty minutes does not sound like much. It was enough. One of the two versions included a file that Python runs automatically every time the interpreter starts — meaning the malicious code executed on those machines whether or not any application ever actually used LiteLLM. It swept up environment variables, SSH keys, cloud provider credentials, Kubernetes tokens, and database passwords, encrypted them, and shipped them to a domain that looked like it belonged to LiteLLM and did not.

On August 12, the threat intelligence firm CloudSEK published an analysis built from roughly 434,000 files it says the attackers captured, mapping potential exposure to more than 2,500 organizations. The names on the list are the kind that make a board sit up. CloudSEK was careful in its own language — these are reconstructed exposure figures, not confirmed compromises — but the headline number traveled faster than the caveat.

Then Someone Checked the Clocks

Two days later, on August 14, SOCRadar published a record-level look at the same incident data covering 2,188 organizations. Every record carried a first-seen and last-seen timestamp. The earliest was March 19 at 18:05 UTC. The latest was March 24 at 20:09 UTC — a span of just over five days.

For 2,085 of those organizations — 95 percent — the data collection had already finished before the poisoned LiteLLM packages ever appeared. Their credentials were gone days earlier.

The actual entry point was Trivy, an open-source vulnerability scanner from Aqua Security. On March 19, attackers used compromised credentials to publish a malicious Trivy release and tamper with the associated build actions. The first stolen data showed up eighteen minutes later. CISA added the Trivy compromise to its Known Exploited Vulnerabilities catalog on March 26, citing evidence of active exploitation. LiteLLM was downstream of that: its own build pipeline pulled in the poisoned scanner automatically, and the malicious LiteLLM releases were what came out the other end.

SOCRadar's summary is the cleanest sentence anyone has written about this incident: the forty minutes everyone reported was the closing act, not the whole play.

Why the Wrong Name on the Marquee Costs You Money

I want to be direct about the practical damage here, because it isn't reputational.

If your team scoped its investigation to the LiteLLM window — March 24, 10:39 to 16:00 UTC — you searched a four-hour slice of one day. The exposure window that mattered for most affected organizations opened on March 19 and ran through the 24th, across six different CI/CD platforms: GitHub Actions, GitLab CI, Jenkins, Bitbucket, CircleCI, and Buildkite. A company that checked the narrow window, found nothing, and closed the ticket in March has been carrying live stolen credentials for five months.

And these credentials do not expire on their own. The FBI issued a FLASH advisory on July 2 warning that actors affiliated with this campaign are likely to weaponize credentials stolen here long after the initial compromise, and telling organizations to rotate CI/CD secrets, publishing tokens, and cloud credentials that were reachable during the exposure windows. SOCRadar reports the stolen material is already being brokered on Telegram. A static cloud key copied in March works today unless somebody rotated it.

The confirmed downstream damage is not theoretical. CERT-EU concluded that the Trivy compromise gave attackers an AWS API key that led to a breach of European Commission cloud infrastructure — roughly 350 gigabytes of data, about 91.7 gigabytes compressed, touching data tied to 42 internal Commission clients and at least 29 other EU entities. The security vendor Checkmarx confirmed that credentials obtained through the Trivy attack enabled unauthorized access to its own code repositories.

Read that last sentence again. A security scanner was used to breach a security company.

The Question This Actually Raises for Your Board

I've argued for years that the most dangerous position in your technology stack is the tool nobody thinks about because it's supposed to be safe. Scanners, build systems, gateways, and monitoring agents get installed once, granted broad permissions so they can do their job, and then reviewed approximately never. They are trusted by default and audited by exception. That inversion is the whole vulnerability.

The AI angle is real, but it is not the one most coverage reached for. The story here is not that an AI product got hacked. It is that the AI layer now sits at a junction between your data, your identity systems, your compute, and increasingly your automated actions — which makes it a uniquely rich place to land. CloudSEK's own conclusion is that the next major supply chain attack will likely target AI infrastructure precisely because it is connected to everything else. That is a governance question, not a tooling question, and it belongs on an agenda rather than in a backlog.

One more thing is worth noticing. The organizations that came through this clean mostly did so for an unglamorous reason: they pinned their dependencies. LiteLLM's customers running the official Docker image were not affected, because that deployment path locks versions. There is no heroism in that. It's hygiene, and it worked.

What to Ask Your CISO This Week

Ask one question first, and don't accept a summary: did we scope our March investigation to March 24, or to March 19 through 24? If the answer is the narrower window, the investigation needs to be reopened this week, not this quarter.

Then ask what your organization did with the FBI's July 2 guidance. Were CI/CD secrets, publishing tokens, and cloud credentials rotated across that full window, or only where compromise was confirmed? "We found no evidence" is not the same answer as "we rotated anyway," and only one of those closes the exposure.

Finally, ask the structural version of the question, the one that outlives this incident: which tools in our environment run automatically, hold broad credentials, and would not trip an alarm if they started behaving differently? Then ask when anyone last looked.

The name in the headline was wrong. The credentials were still gone.