Why you’ll want to look at Mitre’s managed service evaluation for MDR

WithSecure – and before it F-Secure – has some form with MITRE Engenuity ATT&CK® Evaluations. Our Elements EDR technology – which also powers Countercept MDR - has gone all four rounds.

The ATT&CK Evaluations are a useful reference point for defenders and buyers, and provide a great point of triangulation for us, too – we already have an idea of what our tools and services do, and we hear back from customers, but a third, independent point of view is invaluable.

It goes without saying that when MITRE Engenuity announced it would run evaluations for managed services, we signed up. The results are now in, and we’ve got plenty to pick apart and digest.

As with the original ATT&CK evals, the aim is to map the capabilities of participating vendors, rather than to crown a ‘winner’. Different services benefit different types of users or customers. 

We’d urge anyone looking to buy a managed security service to go straight to the full results on MITRE Engenuity’s website and take a critical, in-depth look at both the results and assessment criteria before reaching a conclusion.

Knowing MITRE Engenuity, they’ll already be thinking hard about how they can take the managed service evaluation to the next level in 2023. We also have a few thoughts of our own as to how the evaluations could be improved in the next round which we’ll share below.

• Our results indicate our detection capability and service layer are performing well 

• We’d love to see MITRE Engenuity execute a more complex scenario on the next round of evaluations 

• The scoring mechanism does not take timeliness of escalation or accompanying context into account - which we think are oversights many users will also note 

• Containment and remediation were not in scope, but these Response phases are integral to removing the threat posed by a malicious actor 

• We compare well against competitors, but we are yet to be convinced Mitre assessments allow for true quality differentiation between service providers. 

Sophistication of Simulation

The simulation was quite simple and didn't emulate the subtlety one would expect to see from an advanced attacker. A bit more sneakiness – perhaps some memory injection techniques or the use of an advanced, intricate C2 framework - would be very welcome. This sort of skullduggery provides a better evaluation of what users will come up against from more determined types. 

Quantity doesn’t necessarily have a quality of its own

The marking mechanism was very similar to that used for the EDR assessment, and verbosity of input seems to have been used as a sign of quality. One's score would be propelled by listing everything that occurred on the machines, regardless of its relevance - providing it was listed. It’s accepted (and sometimes overcommunicated) that a high volume of false positive alerts is not a good thing, but the same could be said for throwing everything at the wall to see what sticks when describing true positive incidents.

The number of real-world incidents where we've reported this level of detail is minimal and it’s difficult to see how not listing a netstat (for example's sake) command is something that shows the effectiveness or otherwise of a MDR provider. 

MDR Evaluation of Quality

The Evaluation of Quality does not reflect the concerns we regularly hear from organizations interested in MDR. 

In our experience, the most important items customers care about are:

• timeliness of detection

• timeliness of notification

• commentary and context for a less technical audience 

• sound remediation, containment and recovery advice

• consistent communication during an incident.

This round of evaluations didn't evaluate these issues rigidly, and these are criteria our customers use consistently for discovery, if not evaluation. We don't recall an instance where not having listed every line of execution affected a customer's perception of service quality.

Timeliness and Effectiveness of Advice

Leaving it until the end of the week to get a snapshot of the tickets raised – a feature of this evaluation – seems at odds with the way we see customers, ourselves and our peer MDR providers operate. The activity executed on a Monday and reported on the same day is substantially more valuable to defenders than if that activity was reported on the Saturday.

Ultimately, what a customer cares about is whether they have sound steps to remediate and recover, and that’s something we think would improve the marking scheme.

That said, this evaluation introduces a valuable set of data points and insights where previously there were none. It has established the efficacy of Countercept Managed Detection and Response and started the process of building a set of evaluation criteria and testing that will bring insight where before there was little independent information.

Look out for the fifth MITRE Engenuity ATT&CK Evaluation results soon – and we’re relishing taking part in another round of managed service evaluations in 2023.

Now might be a good time to mention we can now cater to European Data Residency requirements and user needs with Europe-Only Countercept MDR.