LLMs in Security Research - AI, Standards, and Why Evidence Still Matters

No, this article is not about why you should not blindly trust what comes out of an LLM. That would be rehashing old news. It’s more about who uses which sources. Not too long ago, our analyst team received a malware sample that our solutions did not detect. This happens. Along with said sample, a link was provided to an article on the web which analyzed the malware sample we supposedly had received. Sounds convenient enough – but there’s a catch. More questions than answers Security research is often published on platforms such as dedicated company blogs, but they are also being picked up on my other media, reposted and shared. This is how information is flowing and knowledge is shared. Issues start appearing when information cannot be independently verified. This was exactly the case with the sample that was submitted. The sample we received was clean. No doubt about it. But there was this article, which at that point was already a few months old. It also did not contain a file hash of the sample that was examined. We could not say whether anything in the report was false, but we also could not verify anything because the one bit of information that would have been crucial – the file hash – was not there. The file we received was clean, so maybe it was not identical with the one examined. Maybe we viewed the exact same file as the one that was described, in which case the report would have been wrong. We have no way of knowing for sure. The article also contained IDs for involved browser extensions – the issue here is that if you access the respective IDs today, you will get a different set of files than the one the analysis was based on. So whichever way our analysts turned, they always ended up back at square one. Also, when looking at the report on the submitted malware, something felt off. The wording of the article sounded suspiciously like it was AI generated. Before you start coming at me, hold your horses: I have written articles myself, with no help from Claude, ChatGPT or the like whatsoever, and was told by others that the text sounds “kinda like AI”. I could wax philosophically about this and ask “Do I sound like AI, or does AI sound like me?” Regardless, the fact that something “feels” like it was written by AI is not necessarily a reliable enough indicator for a text that was actually written by AI, or at least heavily “inspired” by it. It is also not a good indicator for veracity or falsehood. At best, it’s a “tread carefully” indicator. Remember, LLMs are trained to output text that feels very “human”. Bias vs. Verification There are people who are strongly biased against AI generated text, and as soon as they think they see a feature that they think is “typical of LLM generated output”, such as the em-dash (which has been used by authors all over the world way before the internet – let alone AI – was a thing), they immediately dismiss it as AI generated and therefore not trustworthy or worthy of their attention. But in this case, the authorship question is beside the point. It does not matter whether the article in question was written by a human or an LLM. Maybe the English version was translated from another language using AI, because the author does not speak English very well. An inherently bad report is not made worse because of the use of AI, and a concise, well-researched report isn't of lower quality just because it was supported by an LLM. People like to get hung up on the fact that something may have been written by AI. But the real issue here is that a piece of research could not be reproduced and verified. And that is not a good thing in any field of research, IT security or otherwise. That being said, we were left with more questions than answers, as well as an uncomfortable thought: What IF someone just posted a completely AI generated malware analysis? As we have written before, AI is not infallible, especially when it comes to malware analysis. Though one must admit that substantial progress was made in recent months. And not all LLMs are created equal. Any output of an LLM therefore needs to be scrutinized and verified before publishing it. This brings up another issue: Some up-and-coming malware researchers do not do their due diligence when verifying the work of the LLMs they used. This leads to a barrage of false, or at least flawed, vulnerability reports. They sound convincing enough at first glance, but do not hold up to scrutiny. That on its own would be annoying but ultimately self-correcting, since security researchers are often very detail-oriented (to the point of being pedantic) and not known for mincing their words when it comes to calling out flawed reports. The issue starts having more substantial impact when unverified AI output is used to create vulnerability reports on the side of software vendors. Because some vendors receive automated, completely AI generated vulnerability disclosure in bulk and have sounded the alarm about this development. Because it not only creates a lot of work on their end, but also ties up resources that are needed elsewhere. Sure, AI can help here, but then it still is not a replacement for a human. This then forces vendors to make submissions more difficult, by demanding extra manual work that is objectively unnecessary but cannot be automated easily. In some cases, the influx of "AI slop" has lead to the cancellation of bug bounty programs. Public Discourse Malware reports are often freely shared under the assumption that what they state is true. This includes sharing by reputable news portals, which implicitly adds credibility to a report that might be flawed at its core. This is a good reminder that “being quoted by others” is not the same as “independently corroborated”. To put it bluntly: An incorrect claim, repeated by ten different people, does not make it factually correct. Again, we cannot prove anything in either direction here. We therefore deliberately left out the names involved in this story. Because pointing fingers is neither the goal here, nor does a cheap “Got ya!” moment help the big picture. Researchers MUST provide information which makes it possible to verify findings, lest we see disputes online that come down to a variation of “but it worked on my machine”. This is not helpful. So, dear security researchers: Please add file hashes of artifacts you examined to ALL of your reports, if you can at all (legal, confidentiality, OPSEC or similar constraints of course notwithstanding). IOCs are all well and good, and certainly helpful. But if you came to a conclusion that you published, you need to make it possible for others to reproduce your findings and verify that conclusion. It’s not like we do not WANT to trust you. But if you withhold certain details, people will begin to smell a rat and assume you have something to hide. Especially given people’s heightened sensitivity of what they perceive to be “AI slop”. At its core, the security industry has been built on trust, sharing and collaboration since its inception. Sure, we all work for different vendors and all that. But we all have a common goal. Let’s not make things murkier than they need to be. Don’t get me wrong: There is nothing inherently bad about using AI in some capacity. It can help tremendously, amplify our potential and speed up things a great deal. But it is not a panacea, and certainly no substitute for humans. The advent of AI has not changed the standards or requirements for evidence in research. But the newest and shiniest tools have in some cases caused a lapse in adherence to these standards. Moving on There is no shame in being wrong. There is a lot of shame in being wrong and not doing anything about it or doubling down on something that was proven to be false. If you are wrong at any point: own it, correct your mistake, and do better in the future. Human honesty and integrity will go further than any LLM’s beautifully phrased hallucinations. If you put your name out there together with your research, then you are fully accountable for every single word. Before publication, you have every opportunity to decide that either something needs more work, or even to say “We are not going to publish this”. If this was not enough incentive: the EU has a law called the “AI Act”, which requires publishers to tag images and texts if they were generated by AI and published with no redactional (meaning human) oversight. Vendors like Anthropic have announced text watermarking in Claude. But even if you can prove that Claude was involved in the authorship, this still does not change the underlying issue: You need to bring evidence for the findings in your research. Getting caught out using AI generated text without checking its veracity can lead to huge embarrassment, and also loss of reputation and trust – especially when the output is factually wrong. Or if someone got lazy and copied part of the LLM prompt into their layout or CMS. This has happened even to major publications such as the German magazine “Der Spiegel”, who have received pretty harsh backlash for it.