Chinese AI Firm Siphoned American AI Knowledge From Anthropic Claude By Using Millions Of Prompts

In today’s column, I examine the recent hacking-like incident reported by Anthropic that a Chinese AI firm known as Alibaba allegedly illegally siphoned digital knowledge from Claude (a highly popular generative AI and LLM of Anthropic). The siphoning consisted of using thousands of fraudulent login accounts that had been set up to access Claude. Via those fake accounts, millions of prompts and their responses were surreptitiously utilized to then draw out digital knowledge from Claude and seemingly used to train the LLM of Alibaba. Anthropic wrote a letter to Congress about this incident. In the letter, besides briefly describing what happened, the AI maker called upon Congress to take various decisive actions. I’ve previously covered how Congress and the White House have been eyeing attempts by China and Chinese firms to exploit American AI to devise their own AI; see my analysis at the link here. I will share with you the ins and outs of what has been going on and do some detailed unpacking about this recent incident that is wholly disconcerting and undoubtedly not the last time that such brazen incursions are going to be made. Let’s talk about it. This analysis of AI breakthroughs is part of my ongoing Forbes column coverage on the latest in AI, including identifying and explaining various impactful AI complexities (see the link here). AI Technique Of Distillation I will start by covering some of the crucial fundamentals about an AI technique referred to as distillation. This common technique can be used for legitimate purposes and can be used for devious purposes. I’ll start by explaining the legitimate purposes. Suppose an AI maker wants to use one of their existing full-sized AI models to enhance a smaller and less capable one of their AI models. This can be readily performed via a technique known as distillation. The typical use of distillation involves an AI maker deciding to create or enhance an SLM (small language model). They pour some of the contents of the LLM into the SLM, aiming to further fill in or pump up what the SLM can do (see my detailed explanation on how AI distillation works at the link here). You can think of AI distillation as a teacher-student type of arrangement. The LLM acts as the teacher. The SLM is the student. The larger-sized LLM shares aspects with the SLM to bolster the capabilities of the smaller AI. This is a relatively routine practice and is commonly undertaken. AI makers do this frequently, and so do AI practitioners and hobbyists. If done appropriately and legally, it is perfectly aboveboard. The twist is that distillation can be utilized in a legal way but can also be performed illegally. The illegal approach involves surreptitiously distilling from someone else’s LLM and essentially stealing their intellectual property (IP). Why would this be done? Because you can take a relatively slim or hollow LLM and pump it up to become much more full-bodied at a super low price. The LLM emerges as a robust LLM overnight. Rather than having to pay and get suitable approval, the underhanded path rips off the hard work and vast invested efforts of whoever made and owns the teaching LLM. Being Sneaky And Stay Below The Radar You might be thinking that detecting when an illegal distillation is taking place ought to be easy-peasy. All you seemingly need to do is monitor when the teaching LLM is actively giving up tons of its content. It would be akin to a water pipe that someone turned on widely or slyly tapped into, and the water is gushing out. If the contents of the teaching LLM are gushing out, voila, you’ve got an unauthorized distillation happening. The thieves are wise to such detection. They know that if they simply pumped out content at a high rate of distillation, doing so would be caught and summarily cut off. It is a much too obvious form of a cyberhack. Though an individual who isn’t in the know might try this blatant means, a sophisticated entity would be too astute to use that crude method. In the case of AI distillation thievery, here’s how a foreign entity might proceed. Keep in mind that a foreign entity could be a country or some entity that has sizable resources to devote toward cyberhacking. The entity creates thousands or perhaps millions of fake accounts in the generative AI model that is being targeted. This isn’t being done by human hand. Instead, an automated script running on a computer server will create these accounts (they become AI bot-controlled accounts). Furthermore, servers across the globe are tapped into so that the accounts appear to be geographically dispersed. It isn’t obvious where the accounts originate from. If you are wondering why an AI maker wouldn’t instantly get suspicious about perhaps millions of new accounts, the gist is that many of the major LLMs already have hundreds of millions of accounts, and new accounts by actual people are being created at an amazing pace. OpenAI has stated that ChatGPT and GPT-5 have somewhere around 1 billion weekly active users. The stats suggest that with ChatGPT, GPT-5, Google Gemini, Anthropic Claude, xAI Grok, Microsoft Copilot, and additional mainstay LLMs, the number of worldwide AI users in total is perhaps 2 billion or Don’t Need To Break Glass Does distillation break or crack the AI and, therefore, ought to be detectable? Nope, it is the mere act of submitting prompts and obtaining responses. The idea is straightforward for doing the distilling. Suppose you wanted to find out what AI can tell you about Einstein’s most famous equation. You could merely ask a question and get a response. Then, based on the response, you ask another question. Keep doing this until it seems that you’ve extracted as much as feasible from the AI about e=mc squared. Collect together all those prompts and responses. Keep them recorded as pairs. Those prompt-response pairs are then fed into the AI that you are trying to train in Einstein’s theory of relativity. By pumping in perhaps thousands or millions of such pairs, the other “student” AI patterns on the prompts and responses, ultimately becoming boosted on the topic of Einstein’s theory. No need to do anything tricky or out of the ordinary. Just submit prompts, collect responses, and do so until it seems that enough has been distilled to move on to some other topic. Distillation has the appearance of an everyday user who is interacting with the AI on a normal basis. You would be hard-pressed to discern that it was a bot that was essentially stealing from the AI. Recent Incident Reported By Anthropic On June 10, 2026, Anthropic sent a letter to Congress that made these statements (excerpts): - “Alibaba executed the largest known distillation attack on Anthropic to date.” - “Alibaba’s campaign targeted some of Claude’s most valuable capabilities, such as agentic reasoning, software engineering, and long-horizon tasks.” - “These distillation attacks are carried out illicitly, systematically, and at industrial scale to harvest US AI capabilities across frontier labs and repackage them as their own without incurring the training and R&D costs required to train US frontier models.” - “Beyond its scale, this campaign was striking for its brazen nature. Alibaba is listed on the New York Stock Exchange, maintains business operations in the United States, and is accountable to US investors and regulators.” - “Congress should advance measures that facilitate threat information sharing between US AI labs, close loopholes allowing PRC AI labs to access advanced US chips, and penalize PRC labs responsible for distillation attacks.” According to the letter, the distillation was aimed at particular areas of digital knowledge that are in Claude, focusing on AI agent reasoning, software engineering, and long-horizon tasks. Those are likely sensible to probe since they involve more complex and less readily discoverable knowledge. In other words, using distillation on an illegal basis to capture mundane knowledge about how to feed a dog or fix a car engine is probably not worth the risk. Go for valuable knowledge that is not readily found elsewhere. The Numbers Involved Let’s do some back-of-the-napkin noodling about the distillation. The idea is that thousands of fake accounts were set up to access Claude. Millions of prompts were then used to get responses from Claude. Presumably, those prompt-response pairs were then fed into the targeted LLM. Voila, Claude was indirectly used without permission to enhance the Alibaba LLM. How large was this illegal distillation? Anthropic indicated in the letter that there were 25,000 fraudulent accounts and 28.8 million exchanges. We will assume, for the sake of analysis, that an exchange consists of a prompt-response pairing. Anthropic says this distillation occurred from April to June of 2026, which I’ll approximate as a 90-day time frame (3 months at 30 days each). Dividing 28.8 million exchanges by 25,000 fake accounts means that there were around 1,152 exchanges per fake account. If we then divide the 1,152 exchanges by the 90 days of the run, this gets 12.8 exchanges per day. Thus, each fake account was apparently only doing around 12 to 13 exchanges per day (on average). The gist is that you can plainly now see why something like this might not stand out. It is a marginal daily usage and not noticeable in comparison to what an average legitimate user might be doing. Going Deeper Into The Details The next calculation considers the volume of digital knowledge that was at play. The usual average size for a prompt-response pair is about 200 to 2,000 tokens (a token is essentially a word or part of a word; see my detailed explanation at the link here). I will use the high end of 2,000 tokens since a distillation of this nature would presumably want to grab as much as feasible per each exchange. Thus, 28.8 million exchanges multiplied by 2,000 tokens per exchange gets to around 57.6 billion tokens. The total haul of this illegal distillation was about 57.6 billion tokens. Is that a lot? Well, if we were to assume that an average book consisted of around 100,000 tokens, the implication is that approximately 576,000 books worth of digital knowledge was distilled. That ought to make your hair stand on end as a large amount of stolen property. We can compare the size to the amount of data training that major LLMs undergo at the get-go. A major LLM is typically trained on around 10 to 15 trillion tokens. If we divide the 57.6 billion tokens by 10 trillion tokens, this comes to about 0.00384, or less than one-half of 1%. If we divide the 57.6B by 15T, this comes to about 0.00576, which is slightly more than one-half of 1%. The crux is that the distilled digital knowledge is a tiny fraction of how much overall training an LLM needs to be customarily trained on. That being said, we must be cautious and make sure not to compare apples and oranges. Raw data used to train an LLM from scratch is a far cry from the refined digital knowledge being distilled from an existing LLM. Also, getting access to finely tuned and hard-to-find digital knowledge might not be feasible by conventional data training. As the old line goes, the reason that people rob banks is because that’s where the money is. Likewise, stealing even a small amount of highly valuable digital knowledge would potentially be worth the risks. AI Distillation By Foreign Entities Now that you are sufficiently up-to-speed about AI distillation and this specific incident, let’s shift our focus to how American makers of AI are being habitually ripped off by foreign entities via the use of AI distillation techniques. In my analysis earlier this year, I examined in detail a publicly posted policy memorandum entitled “Adversarial Distillation of American AI Models,” by Michael J. Kratsios, Assistant to the President for Science and Technology Director in the White House Office of Science and Technology Policy (for my analysis, see the link here). These AI knowledge-stealing foreign entity activities can be likened to the types of subterfuge that took place during the Cold War era. I’m sure you know that spies would try to obtain American secrets, such as how to make certain kinds of missiles or weapons. Espionage tactics often leaned into the use of widely spread human actors, including individual researchers, governmental officials, industry practitioners, and others, to make copies of secret plans, proprietary documents, and so on. Nowadays, those same spying tradecraft precepts are being retooled as AI bots that converge in proxy swarms on a targeted LLM in an AI distillation attack. This allows scaling far beyond what human hands alone could accomplish. Deploying thousands of semi-autonomous accounts to perform coordinated queries is relatively cheap and easy to undertake. It is much less expensive than building the same content from scratch, can be done in a fraction of the time compared to the right way to do things, and is quite difficult to detect. What Can Be Done American companies working individually won’t necessarily have the wherewithal to tackle the spying tactics of AI distillation that occur on an industrial scale. Sure, they are doing what they can to devise AI safeguards around this, but the foreign entities are going after a wide swath of LLMs and can keep maneuvering as they do so. A mix of defensive tactics and strategies is being constantly crafted and advanced. Technical defenses include: - Behavioral monitoring across accounts (detect coordinated querying patterns). - Use of data watermarking or data fingerprinting to trace model lineage. - Adopt differential privacy or output perturbation (though this can degrade usefulness). - Enforce query throttling tied to aggregate signals, not just per-account limits. - Devise stronger jailbreak resistance via adversarial training. Operational controls that can be implemented include: - Establish account verification tiers to limit high-volume access. - Enact API usage auditing and anomaly escalation. - Proceed with red-teaming focused on extraction scenarios. Policy responses include: - Consider the adoption of various AI distillation-related export controls on model weights and high-end computing. - Craft legal frameworks treating large-scale AI extraction as IP theft and economic espionage. - Seek to establish agreed and enforceable international norms around AI model distillation practices. The World Ahead The Anthropic indication of the Alibaba incident is pretty much par for the course right now. The White House memorandum had offered several steps that are being undertaken, including sharing information across American AI companies about AI distillation subterfuge taking place, and having the federal government work closely with AI makers to develop best practices for identifying, mitigating, and remediating these industrial-scale efforts by foreign entities. The Anthropic letter further called for similar actions. Distillation rip-offs of American AI LLMs are going to be a never-ending cat-and-mouse game. A final thought for now. Sun Tzu famously made this remark: "There is no place where espionage is not possible." The same holds for modern-day AI. Diligence is required, and an expectation of espionage and thievery must always be on our minds.