Choose your fighter: Balancing competing requirements to select models for your AI SOC

- Selecting a model for your security operations center (SOC) and digital forensics and incident response (DFIR) tasks is important, but selecting the best one is more involved than you might think. SOC tasks rely on a combination of model efficacy, analysis time, cost, and consistency of results. - Cisco Talos tested 66 model and reasoning combinations across offerings from both Anthropic and OpenAI on a log analysis task to see if we could identify a clear winner. Instead, we found a repeatable methodology that organizations can use in their own evaluations. - Reasoning effort was not a universal quality dial. More effort often cost more without improving the result. In some cases, more effort produced lower scores. - Consistency should be a major decision factor. A condition with a strong median can still produce an occasional weak run. Choosing the best model for any task involves a complex balancing act: compute/reasoning effort vs. effectiveness vs. time vs. cost vs... well, lots of other things. If you are choosing a large language model (LLM) for a security operations center (SOC) or digital forensics and incident response (DFIR) workflow, “Which model scored highest?” is almost certainly not the right question. In fact, it could even have severe negative consequences. A more useful question might be: Which model and reasoning setting gives me enough investigative quality, at a cost, speed, consistency, and failure rate my workflow can tolerate? The experiment Cisco Talos tested 66 model and reasoning combinations (the conditions) from Anthropic and OpenAI on a tool-assisted log-review task. Using only common Unix command-line tools, the reviewers had to decide whether a given dataset was real or synthetically generated. Each reviewer received an identical dataset. The dataset was synthetic, but the reviewers were told that it might be real. We chose this task because it required many of the same tools and analytic techniques used in typical incident triage and investigation, but unlike those scenarios, could easily create a single numeric score for comparison. The reviewers investigated the logs using their native agent harnesses (i.e., Anthropic models used Claude Code, OpenAI models used Codex), then assigned a synthetic-confidence score from 0 (real) to 100 (synthetic). Higher scores therefore approached the known answer more closely. Each experimental panel contained four independently prompted reviewer personas: - Threat Hunter - Detection Engineer - Network Forensics Analyst - Host/Endpoint Detection and Response (EDR) Analyst We ran five rounds per condition. A panel counted only when all four reviewers produced valid reports. We allowed a limited number of retries in the case of guardrail refusals or invalid output formats before discounting a panel. The panel score was the mean of the four persona scores, and the condition score was the median of all its complete panel scores. What we measured In addition to the review score mentioned above, we computed the following for each panel: - Cost: Total API-equivalent cost of every attempt for a condition, including failed attempts and retries, divided by the number of complete, usable panels. We calculated cost using a public list-price rate card frozen before testing began, rather than actual incurred spend. Actual costs vary by payment method, subscription plan, credits, and negotiated contract terms, making them unsuitable for consistent cross-provider comparison. The published rates were current when the study began and may differ from today’s prices. - Time: The total wall time consumed across all five planned panels for a condition, also including failures and retries, divided by the number of complete, usable four-persona panels. Within each panel, the four persona evaluations ran concurrently. Any provider-directed waits and targeted retries were included in the panel’s elapsed time, and each panel was fully resolved before the next panel began. - Downside score consistency: Some tested conditions had a wide discrepancy when it came to their efficacy scores, while some clustered tightly together. In a SOC, unexpectedly good answers are unlikely to cause problems, but unexpectedly poor answers can lead to unwelcome false positive or (worse) false negative decisions. Our score consistency is defined as the median score for the panel minus the lowest score in that panel. Smaller numbers indicate higher consistency. The data behind the tests The corpus was generated with EvidenceForge, Talos' open-source synthetic telemetry generator. We froze EvidenceForge at version 1.12.0 and used the same six-hour enterprise scenario for every condition, so the model and reasoning settings changed while the evidence did not. The reviewer-visible corpus contained 80,054 simulated log records across 20 source formats, packaged as 88 files totaling 48.0MB (45.8MiB). It combined: - Network telemetry from two Zeek sensors, including connection, DNS, HTTP, TLS, SMTP, file, certificate, OCSP, DHCP, and NTP logs - Perimeter security telemetry from a Cisco ASA firewall and Snort IDS - Endpoint telemetry, including Windows Security and Sysmon events, eCAR process, session, and flow records, Linux syslog, and shell history - Application access logs from web and proxy services - A small set of email artifacts Every reviewer received an identical copy of the data. Scenario definitions, generator information, ground truth, and other metadata generated by EvidenceForge were withheld from the model. What we learned The most important thing Talos learned was that choosing your model is not as straightforward as we had hoped. The following chart lists the top 10 conditions by median score. If we were to take the top-scoring model, we could expect to wait more than half an hour for an answer and pay about $55USD for it. While that might be acceptable for certain tasks where the need for the best possible analysis overrides any other factors, we can easily see that the “best” model here might not be the appropriate choice for workflows that execute frequently. Rank | Condition | Median score | Complete panels | Observed range | Time/panel | Cost/panel | 1 | GPT-5.6 Sol Ultra | 96.25 | 5/5 | 95.00 – 98.00 | 33.72 min | $55.48 | 2 | GPT-5.6 Sol XHigh | 92.75 | 5/5 | 92.00 – 95.75 | 24.66 min | $38.55 | 3 | GPT-5.6 Sol Max | 90.00 | 5/5 | 88.75 – 92.75 | 31.51 min | $53.88 | 4 | GPT-5.6 Sol High | 87.25 | 5/5 | 70.25 – 89.50 | 16.88 min | $28.58 | 5 | GPT-5.6 Sol Medium | 81.50 | 5/5 | 80.25 – 88.75 | 11.89 min | $15.24 | 6 | GPT-5.6 Sol Low | 73.00 | 5/5 | 57.25 – 77.50 | 5.83 min | $5.45 | 7 | GPT-5.6 Terra Max | 66.00 | 4/5 | 63.00 – 69.25 | 28.32 min | $18.27 | 8 | GPT-5.6 Terra Low | 65.00 | 5/5 | 53.00 – 76.00 | 4.72 min | $2.37 | 9 | GPT-5.6 Terra Ultra | 58.75 | 5/5 | 48.25 – 71.50 | 23.16 min | $18.56 | 10 | GPT-5.6 Luna Low | 58.25 | 5/5 | 46.00 – 74.00 | 3.24 min | $0.39 | Instead of ranking based on any single criteria, we needed a more robust, multi-variable system, so we chose to compute the Pareto frontier. Stop looking for a single winner A Pareto frontier highlights the best available tradeoffs when several measures matter, and no single measure determines the winner. A condition appears on the frontier when no other condition is at least as good across every measure and clearly better on at least one. For example, a lower-scoring condition may still belong on the frontier if it is meaningfully faster or less expensive. Conditions outside the frontier have another option that matches or improves all the measures being compared, making them less attractive under any combination of those priorities. Talos' frontier was calculated using the four primary measures discussed earlier: score, cost, time, and downside consistency. Although this produces a single frontier, a four-variable frontier is difficult to represent and interpret visually. The following graphs therefore show four two-variable views: score vs. cost, score vs. time, score vs. downside spread, and cost vs. time. The dark line in each graph marks the best observed tradeoffs for the two measures shown in that panel, while the numbered points identify conditions on the full four-measure frontier. A numbered point may fall away from a panel’s line because its frontier membership depends on one of the other measures not shown there. In the score graphs, conditions toward the upper left generally offer more attractive tradeoffs: higher scores with lower cost, time, or downside spread. In the cost-versus-time graph, the preferable direction is toward the lower left. The cost and time axes use logarithmic scales, so equal distances represent proportional rather than equal numerical changes. Together, these views help explain why each condition belongs to the frontier, but choosing among them still requires deciding which tradeoffs matter most for the intended use. A reasonable way to use this information to select the optimum condition is to begin with the conditions on the Pareto frontier, discarding all the others. Next, set acceptable thresholds for each of the four variables: - The minimum score you're willing to accept - The maximum downside consistency you can live with - The highest per-task cost you're willing to pay - The maximum amount of time you're willing to wait for an analysis task to complete From the Pareto frontier conditions, eliminate any which fail to meet at least one of those requirements. You are likely to still be left with more than one frontier condition. Choosing between those is a matter of organizational priorities and preferences. In a SOC, if all the other requirements are met, choosing the remaining condition with the highest mean score is probably a good start. Other lessons learned While our main goal was to find an effective selection methodology, we learned some other interesting things as well. In fact, some of these were rather surprising. More reasoning did not reliably mean better analysis Cost generally rose with reasoning effort. Score did not. GPT-5.6 Sol mostly improved as effort increased but max scored 90.0 while the lesser xhigh level scored 92.75. Ultra then climbed to 96.25. We saw a much more pronounced and surprising effect with GPT-5.6 Luna, where increasing the reasoning effort decreased scores at all levels. In fact, GPT-5.6 seemed to have a generally odd relationship between reasoning and score. Terra was erratic. Claude Opus 4.8 gained eight points from medium to high, then lost 9.5 points from high to xhigh. These results show why it is important to benchmark every reasoning level you might deploy. You cannot assume that a model’s performance scales according to the reasoning level you use. More effort means more cost but doesn’t always mean better results. The analyst role changed the result Talos’ results showed a measurable difference in score based on which persona was doing the evaluation. This was entirely expected (and why we chose four different personae in the first place) but it was nice to see this confirmed by data. The chart below shows every valid score produced under each of the four analyst roles across all conditions. Each dot is one evaluation. The box captures the middle half of the scores, and the line inside it marks the typical result. The Threat Hunter role produced the highest median score at 43. Network Forensics and Host/EDR both had medians of 35, while Detection Engineer had the lowest at 31. When we compared roles within the same model, reasoning setting, and test round, the largest typical difference was between Threat Hunter and Detection Engineer; Threat Hunter scored five points higher. These are tendencies, not guarantees. The distributions overlap substantially, and each role sometimes produced both high and low scores. But the results do show that changing the role and its evidence priorities could meaningfully change the model’s conclusion. For SOC workloads, the prompt should be treated as part of the system. Do not assume that one generic “SOC analyst” prompt represents every defensive workflow. If your budget allows, you might get better results by having multiple personae evaluating data according to their individual “expertise.” But watch for disagreement between the personae. Large differences may require extra human review. Higher reasoning effort sometimes reduced reliability Two failure types had the greatest effect on model selection: responses that violated the required output format and attempts blocked or declined by the model provider’s safety system. Although safeguards and model-authored refusals arise differently, both have the same immediate operational result: no usable analysis is delivered. Almost every format violation came from Claude Sonnet 4.6. Low and medium completed without any, but 10 of 27 high attempts and 15 of 29 max attempts returned invalid output. Retries recovered some cells, but high produced only two of five complete panels, and max produced none. This was not a minor formatting inconvenience; it prevented both conditions from producing enough comparable results. It doesn’t matter how good the underlying analysis is if the model can’t provide answers in the expected format. Safeguard and refusal failures followed a similar pattern at higher reasoning settings. Claude Sonnet 5 had none at low or medium, followed by one at high, four at xhigh, and five at max. We intentionally excluded Anthropic’s Fable from our experiment matrix because our early testing generated far too many refusals to get comparable scores. Safeguards blocked 21 of 31 attempts, including all eight max attempts. Ten of its 20 scheduled persona cells remained unavailable, and no reasoning level produced a complete four-persona panel. It’s worth noting that the early tests were conducted with an account which was part of Anthropic’s Cyber Verification Program (CVP) which offers relaxed safeguards for recognized cybersecurity professionals. Even with relaxed guardrails, the high refusal rate rendered the model unusable for our tests. These failures are already reflected in the optimization results. Conditions that could not produce at least three complete panels were excluded, while the cost and time of failed attempts and retries were included in the reported operational measures. However, the failure rate itself was not an axis of the Pareto frontier. These results show that reasoning effort can affect more than answer quality, cost, and completion time. It can also affect whether a usable answer arrives at all. What does this mean for your SOC? We began this work looking for the best model for a particular task. What we found instead was a set of tradeoffs. The highest-scoring condition was also slow and expensive, while several cheaper and faster conditions delivered lesser, but still useful, results. There was no single obvious winner: - Reasoning effort was not a dependable quality dial. Increasing it sometimes improved the result, sometimes made no meaningful difference, and sometimes made performance or reliability worse. - The analyst role also changed what the model concluded, confirming that the prompt is part of the system being evaluated. - Consistency and availability mattered alongside average quality. A model that occasionally produces an excellent answer may still be a poor operational choice if it also produces weak, malformed, or blocked responses too often. Rather than just using the results of our study verbatim, organizations should use it as a model for their own selection process. A focused set of representative cases and model/reasoning conditions, tested several times with the prompts and tools you intend to use in production, can reveal much more than a generic leaderboard. A spreadsheet that records quality, cost, time, consistency, and usable-answer rate is enough to expose many of the tradeoffs. The goal is not to build a perfect benchmark or discover a universally superior model. It is to replace assumptions with evidence before a system touches real investigations or starts incurring real costs. Begin with the workflows that matter most, measure what your SOC cares most about, and revisit the decision as the technology or cost changes. Model selection will still involve judgment, but it can be informed, explicit, and defensible judgment.