High-fidelity network telemetry can improve security outcomes by 2-4x across core investigation metrics, according to new research from the Provably Better Data project published by Corelight. The findings settle a persistent debate in security operations: whether AI-driven threat detection depends more on the sophistication of the large language model deployed or on the quality of the security data it analyzes. The study demonstrates that data quality outweighs model selection as the decisive factor governing successful security operations.
The research team evaluated model performance across two operational benchmarks—a 44-question capture-the-flag investigation based on a Volt Typhoon attack campaign and an incident response report generation task using a Salt Typhoon dataset. To isolate data quality as the single variable, investigators processed four distinct network telemetry sources under identical conditions: Corelight enriched logs, open source nDPI firewall logs, Snort 3 intrusion detection system alerts, and NetFlow connection telemetry. In the CTF benchmark, accuracy rates varied dramatically by source: Corelight logs achieved 95.2% accuracy, firewall logs reached 58.3%, Snort 3 alerts hit 39.4%, and NetFlow records scored just 25.8%. Corelight enabled models to answer all 44 questions with direct log evidence, whereas NetFlow supported only 15 questions, resulting in a CTF score of 4,178.3 points for Corelight compared with 970.0 points for NetFlow—more than a fourfold improvement. The incident response experiment revealed similar disparities: Corelight logs delivered 90.3% evidence coverage, firewall logs provided 61.3%, NetFlow records managed 30.9%, and Snort 3 alerts reached only 21.2%. For Tier 1 critical investigation requirements, Corelight logs enabled models to answer 91.7% of mandatory questions, whereas NetFlow logs supported just 18.3% and Snort 3 alerts only 10%—a fivefold increase in critical incident visibility. Investigation speed also improved significantly: the LLM completed the full investigation in 14.7 minutes with Corelight logs but required 27.0 minutes with NetFlow logs and 26.3 minutes with firewall logs, nearly doubling investigation times with lower-quality data.
The research emphasizes that frontier language models provide advanced reasoning capabilities, but the quality of their conclusions remains bounded by the available evidence. When telemetry lacks detailed protocol-level context, AI agents cannot infer what was never collected—the investigation can only go as far as the data allows. The study illustrates this with a CTF question requesting the NetBIOS computer name for IP address 10.110.154.113: Corelight logs provided the correct answer, FINANCE01, found in the server_nb_computer_name field after Zeek parsed the NTLM Type 2 challenge message, while firewall logs identified NTLM protocol activity but failed to parse individual fields within the challenge, leaving it unable to return the correct answer. According to the report, model upgrades and complex prompt engineering will not overcome fundamental data deficiencies, and it's worth noting that the LLMs rarely generated hallucinated outputs when prompts instructed them to mark missing evidence as unanswerable—hallucination counts remained at zero for Corelight, firewall, and NetFlow datasets.
The report recommends that security operations center leaders planning future investments should prioritize evidence quality over model selection, noting that effective automation requires complete, structured, and protocol-aware telemetry to achieve reliable outcomes. For CISOs, these outcomes offer direct operational advantages: providing concrete telemetry reduces mean time to respond, controlling token expenditure helps manage costs, and ensuring higher return on security investments helps prove security team efficacy. The findings can also help justify infrastructure investments, lower analyst turnover caused by alert fatigue, and provide defensible security metrics to executive teams and board members. Organizations betting heavily on AI-driven security automation may discover their returns hinge less on which vendor they choose and more on whether their existing infrastructure can deliver the raw material those systems need to succeed. The shift toward autonomous threat response could widen the gap between well-instrumented networks and those running on legacy visibility tools.