MITRE 2024: what’s new and should I care?

Mid-sized organizations often lack extensive, dedicated cyber security teams to run their own tests, meaning MITRE’s rigorous evaluations are a valuable tool when it comes to getting help. They provide invaluable insights into the performance of EDR tools.

Companies can compare EDR products with high MITRE ratings – particularly in terms of achieving effective threat detection with low false-positive rates – which is something that will interest mid-sized organizations. While MITRE’s evaluations are still evolving for macOS and Linux, they are valuable for Windows-based companies, which make up the majority of mid-sized businesses. 

The 2024 of MITRE ATT&CK Evaluations: Enterprise introduced several long-awaited changes that differentiate it from previous years, meaning it is now starting to resemble attacks on real-world environments instead of zero-noise lab environments:

1. The inclusion of macOS, recognizing the growing diversity of operating systems in corporate environments. This addition expands the relevance of MITRE’s evaluations, making them applicable to organizations that operate across multiple platforms.
   
   
2. A new approach to false positives. A significant portion of individual attack techniques included in the scenarios are intended as false positives, meaning that effective detection should ideally ignore these benign events. This adjustment brings the evaluation closer to real-world conditions, in which filtering out unnecessary alerts is essential to maintaining an efficient and responsive security system.
   
   
3. The elimination of telemetry. In past evaluations, the telemetry category was assigned when a system captured relevant data without triggering an alert. By removing the telemetry detection category, MITRE has shifted focus towards immediate, actionable detections.
   
   This change finally removes the raising of tens of thousands of events indented for manual threat hunting as evidence of an EDR tool’s detection efficacy.
   
   
4. MITRE has also streamlined the evidence collection process during test execution. Previously, participants were required to go through extensive review sessions covering each step of the evaluation. In this round, MITRE staff were able to provide a quicker walkthrough of the scenarios, significantly reducing the time required for feedback and allowing participants to assess their performance more efficiently.
   
   
5. The bundling of the protection test. In past years, companies had the option to participate in either the detection or protection test separately, but are now automatically entered into both unless explicitly choosing to opt out. The detection test simulates an Advanced Persistent Threat (APT) scenario with various attack steps, while the protection test is focused on attacks and ends as soon as a threat is blocked. The protection test is not expected to replace the AV-TEST or AV-Comparatives as more established independent protection tests. 

Some industry experts critique MITRE’s evaluations for being unrealistic, as the high density of attacks simulated in the test is more intense compared to what most organizations encounter on a regular basis.

However, these evaluations are designed as stress tests rather than exact replications of day-to-day conditions.

“The MITRE evaluation doesn’t exactly reflect the types of detections that are required for effective response, but rather focuses on comprehensive and detailed insights significantly beyond what responders require. In this sense, I consider MITRE to be an overall effective and ambitious stress test. When all is said and done, MITRE are responsive, the test itself works, and the process is relatively fair,” Lamarca points out.

These high standards are valuable because they push vendors to create low-noise, high-detail alerts. This helps defenders quickly understand the timeline and context of an attack without being overwhelmed by unnecessary information.

“As security specialist in a small team, I would look for a balance between EDR technology and establishing an effective partnership with an MDR service provider. I'd like to trust that my requests and suggestions are dealt with effectively. Therefore, my choice would be informed by the MITRE score along with the reactivity of my cyber security partner, to ensure I have a holistic view of my portfolio,” Lamarca concludes.