Purple Team

Purple Teaming and Detection Engineering even though that as a concept exist in the information security industry for years lack of specific standardization, models and metrics. The absence of dedicated purple teams for the vast majority of organizations doesn’t enable people with collaborative mindset to focus on detection resilience. Offensive, Defensive and Threat Intelligence resources are already overloaded with their functional day to day activities which naturally deprioritize any purple team activities.

Development of detection rules with high fidelity is the end goal for detection engineering programs. However, one of the biggest mistakes that organizations are making is to segregate this department organizationally instead of embedding it under the Purple Team program which creates a silo and the breaking of this silo is depending on the communication skills and mindset of individuals. Furthermore, building sufficient detection coverage for sophisticated threats require the collaboration of different teams such as the Threat Intelligence for techniques prioritization related to the organization threat landscape, the Red Team to provide the technical context of the attack and potentially actionable defensive security measures that detection engineering teams can incorporate to their detection logic and SOC to determine how the response will look like in the event of an alert.

The ATT&CK framework has played a critical role for organizations to be able to map threat actors and attacks from adversary simulation assessments. The contribution to the industry is massive, however over-utilization of the framework for the benefits of executive measurements, metrics and impressive heatmaps (especially from security products) might lead to results which are not reliable related to the overall cyber resilience posture of the organization. The ATT&CK framework records common attacks recorded in Threat Intelligence reports. Additional techniques and procedures do exist which are not recorded in the framework and therefore without incorporate them to the overall detection strategy can create gaps which will not addressed due to the measurement of results from the ATT&CK framework mapping.

Overall, engineering detection rules require a comprehensive, systematic and strategic approach, solid understanding of the TTP (tactics, techniques and procedures) behavior and sufficient testing and documentation. This will ensure that the detection rule is aligned with the threat landscape, doesn’t introduce false positives and the organization (SOC analysts) can respond properly when the rule is triggered. The Detection Rules Development Framework can aid detection engineers towards following a structured approach when building detection’s. However, it is equally important outside of the core technical work which is associated with the rule development to record information related to data sources, detection gaps, detection confidence etc. in order to identify trends within the organization that will drive the next strategic actions.

Threat Landscape

Every organization has a different threat landscape and therefore it is essential development of new detection rules to be prioritized based on threat actors (geopolitical, sector) that could potentially target the business and their associated tactics, techniques and procedures. The ecosystem of threats and TTP’s is large and complex. SOC teams and detection engineers should leverage threat intelligence information to identify likelihood and business impact of these threats and measure them against existing controls to identify gaps which require addressing in the short term. A variation of threat intelligence scenarios could ensure that purple team operations are realistic and aligned with the threat landscape. Furthermore, in industry sectors which are heavy regulated (financial), development of these threat scenarios can aid towards organization preparation and readiness. The following image represent an example of modeling threat actor and TTP’s mapping.

Technique Metadata

Techniques should contain sufficient information to determine the focus, detection efficiency and relevance within the organization security detection strategy. Tagging metadata information to techniques aids detection engineers to gather metrics and to be able to represent and visualize the impact of the work to executives within the organization and establish trust and confidence. The detection engineering industry lacks a specific standardization and the volume of information recorded is based on the experience of the detection engineer. Defensive teams could use the following template to record information about techniques which are associated to detection rules:

id: <id>
MITRE-ID: <ID>
name: <Rule_Name>
description: <Rule_Description>
objective: <rule_objective>
author: <Rule_Author>
date: <Rule_Date>
threat-actors: <ThreatActor1,ThreatActor2>
severity: <Critical/High/Medium/Low>
version: <0.1>
interval: <Daily/Weekly/Monthly>
type: <Behavioral/Anomaly/Signature>
data-sources: <event-logs/etw/UEBA/EDR>
detection-ratio: <%>
confidence-level: <Strong/Moderate/Weak>
references:
  - <TTP-References>

Utilization of the above template enables detection engineers to map detection rules to ATT&CK framework to the associated threat actors based on threat intelligence information. Furthermore, organizations have a clear measurement about the utilization of data sources and if there are any defensive controls which are not used sufficiently during detection engineering. The detection ratio and the confidence level of the rule allows the detection engineering team to understand the rule efficiency towards the specific technique/procedure. The metrics of detection ratio and confidence level can determine if the rule should be revisited in the future to improve the detection resilience against the specific behavior.

Data Sources

The efficiency and reliability of a detection rule heavily depends on the availability of data sources. Even though detection engineers typically develop detection’s based on specific technologies, it is critical to perform a technique analysis prior of any detection rule development to understand data sources required and identify detection blind spots. Purple teaming overall should extend on the emulation of TTP’s and provide assistance and guidance to other stages of the detection process. The following table illustrates the technique mapping to data sources:

if a data source is not available, it is recommended defensive teams to conduct a review and assess whether it is feasible to enable additional logging to improve detection opportunities of the TTP in scope. Furthermore, the availability of a data source can influence the result of the overall rule confidence and detection ratio and therefore could limit false positives and remove uncertainties.

Signature vs Behavior Detection

As the level of sophistication from threat actors is increasing detection rules focus should be on suspicious activity patterns and not on rules that rely on static signatures. The main advantage is that behavioral based detection is adaptive and will most likely if done properly will enable defensive teams to respond to threats. Tools can change overtime but the underlying method of conducting a technique might not and therefore the coverage and the detection opportunities are increasing when a rule is not focused on a signature.

From the other point of view anomaly based detection produces a higher ratio of false positives since legitimate activities can be flagged as suspicious which might increase the resource overhead. It should be noted that behavioral based detection require strong understanding of how the threat operates which is challenging to decode techniques at a large scale and in a continuous capacity without sufficient amount of dedicated purple team resources.

Detection Rules Mapping

Providing specific metrics related to detection rules is challenging since rules are dynamic and can depend on a number of variables such as the environment configuration and the continuous monitoring of new developments in the offensive security industry. These variables can decrease the efficiency of a detection rule. Additionally, threat actors might utilize techniques which has not been disclosed in the public domain or used by red teams and therefore outside of the EDR machine learning capability which can aid towards identification of unknown threats, detection engineers should focus on what is known. Public resources such as the ATT&CK framework, threat intelligence and red team reports or red team intelligence should provide sufficient information to identify as many procedures as feasible per technique.

Visualization is important during threat analysis as enables detection engineers to decompile the technique to different stages such as procedures and map these procedures with the data components required to detect. The following diagram illustrates how TTP can be mapped to Data Sources.

In the event that detection gaps exist after the initial threat analysis, the data sources mapping will look like the diagram below:

Data sources are critical towards the development of detection rules. Identification of which data components are enabled versus which are not enabled cannot only aid towards building rules with high fidelity by identifying visibility constraints but can assist with a more holistic view to measure data sources availability.

Technique Simulation & Validation

Simulation of techniques and rule validation should be part of the purple team operational procedures. The purple team playbook associated with the technique should cover all the known procedures that exist on the public domain. The results will determine how the existing controls operate and if there are any detection or visibility gaps. For example the Browser Stored Credentials technique is associated with two procedures. The first procedure involves abuse of DPAPI and the second the retrieval of credentials via the PasswordReuseDetectorImpl class.

The results are captured in columns detection and visibility which will determine the current confidence towards existing detection rules and the remediation actions. These might include the development of new detection rules or adjustments to the existing and the possibility to enable additional logging to enhance visibility. For example if the current detection controls such as the Endpoint Detection and Response (EDR) or custom detection rules can detect the browser credential theft via DPAPI but there is not alert towards the PasswordReuseDetectorImpl method then the realistic detection ratio is 50% of this technique. Purple Team operators should record this as a detection gap and an investigation should be initiated to determine additional measures or the development of new detection rules. The following table maps the technique to the known procedures, associated detection rules per procedure and the detection coverage.

Building an efficient detection rule is a very broad term as the variables to determine the efficiency are the number of procedures this rule detects and the in depth coverage of simulating a behavior. A model to address the confidence level (detection efficiency) of a rule has been developed with a focus on the overall capability to detect the technique. For example the browser stored credentials has two procedures associated. A detection rule which has been sufficiently tested and validated for DPAPI provides 50% detection ratio which classifies the capability of the organization to detect this technique to Moderate. If another detection rule is developed to address the procedure gap of the PasswordReuseDetectorImpl then the capability increases to Strong. This measurement needs to come with the consideration that additional procedures might exist but defensive teams cannot build detection rules for procedures which are not known.

When discussing techniques, context is also important. For example, it should be noted that prior of executing credential theft via browsers, threat actors are required to perform some form of process injection and therefore additional detection opportunities will exist. However, building detection resilience requires a defense in-depth approach to ensure detection coverage for a wide range of techniques, procedures and response readiness for the worst possible scenario.

In the event that a purple team activity has a focus to validate a detection rule, the results should represent the confidence level and future tests should be repeated in an automated manner. There are multiple products like breach and attack simulation that offer great automation and coverage against ATT&CK framework. The constraint of executing purple team operations or detection rule validation is the lack of feasibility to extend current procedures with new methods which eventually will provide a false sense of security. However, blending in the automation elements of these products with manual executions (as techniques evolve over time) provides a more realistic and in-depth view of the defensive security posture.

False Positives

Building detection resilience is not a sprint but a marathon. Detection rules might generate false positives depending on the environment configuration, how users operate and how well defenders are aware of users behaviors on the domain. During the rule validation, purple team operators should record false positives identified and any procedure gaps which the rule cannot detect. In an ideal world the target would be to develop detection rules with zero false positives that will cover a range of procedures under the same technique. However, since this is not feasible, multiple detection rules might be required to increase detection opportunities of a specific technique. Detection rules should be treated as a living organization within the defensive ecosystem and should be evaluated for their efficiency and tuned if it is required to improve detection coverage.

Conclusion

Overall development of detection rules requires a unified approach with Threat Intelligence, Red/Purple Teams and SOC. Purple Team Leads and Operators should be responsible to record information related and bring all teams under the same mission. Purple Team programs must not classified as light version of red team assessments. Establishing a mature detection engineering program with defined methodologies and processes that is fully integrated or directly connected with a threat intelligence led purple team program can increase the defensive security posture significantly.