Digital Operational Resilience Act (DORA) and Penetration Testing: An Overview by Edgescan

Introduction

The Digital Operational Resilience Act (DORA) was introduced across EU nations to address risk management gaps and attempt to harmonise these requirements across the EU. The act specifically targets financial service entities, i.e. largely those regulated by central banks, and introduces rules around incident management and reporting, digital testing and management of third-party risk.

This document discusses the practical implications of DORA and what it means for your organisation, with a particular focus on one key element: Digital Operational Resilience Testing. This is the piece that includes requirements for establishing ongoing testing programmes.

What is DORA?

DORA (EU Regulation 2022/2554) was adopted by the European council in November 2022 and will be in full force from 17th January 2025. DORA is an EU regulation that comprehensively addresses ICT risk management in the financial sector by ensuring that all providers follow a set of standards to mitigate ICT risks for their operations. Prior to DORA different countries had different regulations and requirements when it came to ICT risk, whereas DORA aims to create a single framework that all EU member states will have to follow.

After DORA’s initial release, several clarifications on its practical implementation have been sought. Given this uncertainty about the legislation’s day-to-day elements, various European authorities have produced a number of technical documents. The first batch of these was released in January 2024, in the form of three Regulatory Technical Standards (RTS) and one Implementing Technical Standard (ITS). These documents do not contain any information on resilience testing and focus on other key DORA areas.

Five additional technical documents were published in July 2024. This set of documents included the much-anticipated “RTS on Threat Led Penetration Testing (TLPT)” (linked in references), which clarifies many aspects of resilience testing.

Who does DORA apply to?

DORA applies not only to traditional banks but also to other financial entities and ICT third parties that provide ICT-related services to financial institutions, such as cloud platforms or data analytics services.

Affected entities include traditional financial institutions, credit and payment institutions, investment firms & funds, crypto-asset providers, data reporting service providers, insurance companies and ancillary service providers, pensions providers, auditors, and some ICT third-party service providers.

Does DORA apply to US companies?

As DORA is a regulation of the European Union, it only applies to organisations within the EU. So, strictly speaking, it does not apply to US companies. However, any US firms with legal entities operating out of the EU that meet the requirements for the scope above will be subject to its effects.

DORA vs NIS2

Another similar directive, which will become law across the EU in October 2024, is the Network & Information Security 2 (NIS2) directive. This directive is aimed at a wider range of organisations and attempts to harmonise more broad levels of cyber security in the EU.

DORA is specifically focused on the financial services sector and will take precedence over any overlapping regulatory texts such as NIS2.

While similar, their purposes differ—DORA protects the financial sector, while NIS2 creates a cyber security base across a broader set of industries.

DORA Key Elements

DORA really breaks down into six crucial elements, as summarised below:

• ICT Risk Management
• ICT Incidents Management & Reporting
• Operational/Security Payment Incident Management & Reporting
• DIGITAL OPERATIONAL RESILIENCE TESTING
• Information & Intelligence Sharing
• ICT Third-Party Risk Management

The pillar ‘Digital Operational Resilience Testing’ is where Edgescan comes into the picture. This element is the focus of the rest of this document.

DORA Testing Requirements:
Digital Operational Resilience Testing

Digital Operational Resilience Testing is outlined in Articles 24, 25, 26 and 27 of the act, with further clarifications in the RTS on TLPT.

The act defines two key requirements: the notion of ‘testing’ and ‘advanced testing’.

Testing Scope
The target of these tests is on all ICT services that support critical or important functions of financial institutions.

DORA defines a critical or important function as “a function, the disruption of which would materially impair the financial performance of a financial entity, or the soundness or continuity of its services and activities, or the discontinued, defective or failed performance of that function would materially impair the continuing compliance of a financial entity with the conditions and obligations of its authorisation, or with its other obligations under applicable financial services law.”

Requirement 1: Resilience Testing

As part of the ICT risk management framework, DORA requires that financial entities define, document and maintain a thorough and comprehensive digital operational resilience testing programme. Financial entities will need to ensure that appropriate tests are conducted on all ICT systems and applications supporting critical or important functions and must do so on at least a yearly basis.

Some of the testing types that are detailed include:

• Penetration Testing
  • Regular (at least annual) testing of applications, API’s and infrastructure
• Vulnerability Assessments
  • Performing regular vulnerability scanning of networks and applications.
  • Testing should be also carried out before any deployment/redeployment of new or existing applications or infrastructure components that support critical or important functions
• Source Code Reviews
  • Source code reviews are undertaken regularly for critical or important systems
• Objective-based penetration testing
  • Akin to Red Teaming, a more focused penetration test, usually comprising of a wider range of systems, but with objectives defined in advance (i.e. flags).

Organisations will need to:

• Demonstrate that they are conducting an appropriate set of security testing on critical and important systems and applications at least annually
• ‘Fully address’ any vulnerabilities identified by this testing
  There are still some elements of ambiguity in the above requirements, such as what defines appropriate set of testing. Organisations will need to work closely with their authorities to ensure the right level of testing is established on the correct set of systems.

Edgescan Insight: Penetration testing and vulnerability management is core to what we do at Edgescan. We deliver thousands of assessments every year to Fortune 100 clients globally in the EU and the USA. We have a wide range of licencing available which covers continuous security testing and exposure management (for applications and networks), working up to more rigorous penetration testing licences (Penetration Testing as a Service).

Additionally, objective-based penetration testing (aka red teaming) and source code reviews are a capability that we have been exercising for over 10 years / since 2011.

Requirement 2: Threat-Led Penetration Testing (TLPT)

Article 26 of DORA sets out expectations in relation to the advanced testing of ICT tools, systems and processes, by way of the Threat Led Penetration Test (TLPT).

Designated significant entities (see below), must conduct a Threat-Led Penetration Test (TLPT) at least once every three years. The goal is to validate the effectiveness of the ongoing resilience testing, that each organisation is conducting. As this is a three-year requirement, it is likely that the first of these tests will not take place until closer to or during 2027. The TLPT framework being used is largely based on the TIBER-EU framework for red teaming, however a number of modifications have been made.

What is a TLPT
Conventional penetration tests provide a detailed assessment of technical and configuration vulnerabilities which are usually focused on a single system or environment in isolation. A TLPT is more akin to a traditional (albeit much larger) red team test, which focus on the entire organisational entity, including its people, processes and technologies. Additionally a TLPT is ‘intelligence-led’, which means that the testing team, works closely together with a Threat Intelligence provider, to establish a set of testing scenarios based on known current attacks and industry-specific attacks.

TLPT Authority
Each EU member state will designate an authority who is charged with all tasks and responsibilities related to TLPT being conducted in that state. The TLPT Authority will work with each organisation proactively, identifying those who must undergo a TLPT, initiating the testing process, confirming the scope of testing, confirming reporting requirements and ultimately signing off on the work completed.

Even though these Authorities are working from the same set of legislation, there will however likely be small differences between how TLTP’s are carried out in each state.

Criteria – Significant Entities
The technical standards recognise that TLPT will apply to certain organisations based on ICT maturity and overall importance of that entity. The below areas will be factors of that decision:

1. Impact-related factors: to what extent a disruption of the financial entity would impact the financial sector.
2. Possible financial stability concerns: for example, the systemic character of the financial entity at EU or national level.
3. Specific ICT risk profile: level of ICT maturity of the financial entity or the technology features involved.

The above criteria have been tested to ensure that only the biggest and most appropriate financial entities will become subject to TLPT requirements. Indeed, a specific set of financial criteria have been clarified and established in the RTS on TLPT (Article 2)

The TLPT Authority will inform the financial entities if they are required to undergo a TLPT. Given the extent of the undertaking, we would expect the TLPT Authority to provide ample notice to relevant organisations where this will apply. Once this is known, the TLPT Authority will provide three months notice to begin the actual preparation phase of the test.

TLPT Authority: This is a government authority which operates at the jurisdiction level. They will appoint individuals (a Test Manager and TLPT Cyber Team [TCT]) who will work with the financial entity in planning the TLPT and will be the ultimate audience for the test results and final reports.

Financial Entity – A number of roles exist:

• Control Team: This consists of a control team lead who coordinates all activities within the organisation. The rest of the control team should consist of any other individuals who will help coordinate the overall test. This will be the only team within the organisation that will be aware of all aspects of the TLPT and will need to exercise a high degree of secrecy.
• Blue Team: These are the defenders within the organisation. During the testing phases of the TLPT, the blue team will not be aware that the TLPT is taking place. However for final reporting, the blue team’s input will be required, to match up against the red teaming activities (i.e. to see if certain attacks were detected and/or blocked etc).

Threat Intelligence Provider: Third party provider for the threat intelligence activities. These will produce a set of test scenarios based on known current attacks and industry-specific attacks. These scenarios will be presented to the Red Team Provider for execution.

Red Team: Third party provider for the red teaming and actual testing activities. In the context of a TLPT, this would be Edgescan. See below for more details on the testing process.

Internal testers: The latest RTS clarifications now allow for the use of internal testers as part of a TLPT (see note below).

What does a TLPT look like in practice?

Testing Process
This largely follows the TIBER-EU process which is broken into standard phases of preparation, testing and closure. Rather than go into the full TLPT process, we will only highlight some of the important aspects to consider.
It should be noted however, that a TLPT will be a significant undertaking, in comparison to standard ‘business as usual’ penetration tests.

1. Preparation Phase: The control team is formed, scoping takes place, the Threat Intelligence and Red Team Provers are selected by the financial entity. The high-level scope of the test is signed off by the TLPT Authority and this is shared with the external Threat Intelligence and Red Team Providers. The control team must consult with the TLPT Authority on their risk assessments or measures, which have been identified, before any testing begins.
   
   
2. Testing Phase – This is comprised of three parts:
   1. Threat Intelligence Provider: This is where the scenarios which are to be tested are produced, based on real world threats with an appropriate provider. This provider should cover both the targets and threats relevant to the organisation being tested. At least three scenarios devised by the Threat Intelligence provider are chosen by the control team and will form the basis of the red team tests. The scenarios and plans all need approval by the TLTP Authority before actual testing. This process will likely last about four weeks. Edgescan will work closely with the Threat Intelligence provider.
      
      
   2. Red Team Test Plan: The Red Team provider (e.g. Edgescan) will prepare the test plan, according to the requirements laid down in the RTS for TLPT (Annex IV), which includes information on:
      • Communication channels
      • Tactics and techniques allowed and prohibited
      • Risk management measures
      • Detailed descriptions, plans and timelines for each test scenario
      
      
   3. Active Red Team Testing: This phase must take place over a minimum of 12 weeks, to allow for testers to mimic ‘stealthy’ threat actors. TLPTs are ultimately a covert exercise and an element of secrecy around testing activities must be maintained. The exact duration of this phase will be fine-tuned in agreement with the TLTP Authority. The Red Team must be comprised of a red team manager and at least two other red team testers. This is obviously a large and expensive undertaking and one that will evolve and take shape over the coming years. Some further guidance on the exact requirements and shape of these tests will come from the TLPT Authority within each jurisdiction. During this phase, a range of tactics and techniques are employed, such as:
      
      Reconnaissance: Collecting as much information as possible on a target
      
      Weaponization: Analysing information on the infrastructure, facilities and employees and preparing for the operations specific to the target
      
      Delivery: The active launch of the full operation on the target
      
      Exploitation: Compromising the networks, servers, applications of the financial entitle and exploiting its staff through techniques such as social engineering
      
      Control & Movement: Attempts to move from one compromised system to further vulnerable or high value ones – also known as pivoting or lateral movement.
      
      ‘Leg-ups’ can also be used during testing – this is where some extra help by the organisation is provided, such as additional access or information, in order to remove a testing blocker to facilitate continuation of testing. These are, of course, documented as such.
      
      
3. Closure Phase: Only during this phase is the actual TLPT revealed to the blue team. Initial draft reports are prepared by the Red Team for review by the control team.The blue and red teams must also come together no later than 10 weeks after the end of red team testing phase. This is in order to replay relevant actions and defences that were carried out during the test and is known as a purple teaming exercise (see note below), which is now mandatory.This phase also includes reporting by the red team, identification of any weaknesses, the attack paths, flags obtained and any other relevant technical data. Additionally, blue team reports are also prepared at this stage.

   Finally a test summary report and any remediation plans, will be shared with the TLPT Authority for review and sign off.

   Lastly there is also a cleanup phase which includes the secure deletion of data that may have been collected during testing and removal of any ‘testing artifacts’.
   
   

Purple Teaming
The red team (representing attackers) and blue team (representing defenders), working together collaboratively are known as purple teaming. Purple teaming is now mandatory in the closure phase of a TLPT.

Internal Testers
This area is one of the key differences introduced by DORA versus the TIBER-EU framework and allows for the use of internal testers, with a few stipulations. For example they should be independent, suitably experienced and their use must not negatively impact the organisation (e.g. by taking them away from other work duties). To qualify, they must also have worked at the organisation for at least one year (this requirement was eased in the last RTS). It also states that every third TLPT tests should include the use of external testers, i.e. using external testers at least once in a nine year period. The full requirements for internal testers can be found in the RTS for TLPT Article 13.

Tester Requirements
Requirements for the testers who can conduct TLPT is laid out in Article 27 of DORA and also with further clarification in the RTS for TLPT Article 5. These are similar to those requirements in the TIBER-EU framework, with some modifications.

The recent RTS document provided, has given some clarification on the initially strict minimum requirements for those conducting TLPT. The main factor which has changed, removes the chicken and egg scenario, whereby originally those conducting TLPT must have prior experience in conducting a TLPT. The requirement has been widened now to include more practical, prior experience in ‘penetration testing and red teaming’.

Given the strict requirements, the update includes provisions that the possibility that financial entities can procure providers who do not comply with some of the requirements, provided they mitigate any additional risk that may be introduced.

Testing References: The Red Team Provider must provide at least five references from previous assignments related to penetration testing and red team testing.

Insurance requirements: The Red Team and Threat Intelligence Providers must show adequate insurance provisions for professional indemnity and cover for risks of misconduct and negligence.

Red Team Manager: There must be at least one, who can demonstrate at least five years of experience in penetration testing and red team testing.

Red Team Members: There must be at least two other testers, each with penetration testing and red team testing experience of at least two years.

Additionally, the Red Team Manager and Testers must have a combined participation in at least five previous assignments related to penetration testing and red team testing.

Finally, testers should be free of conflict of interest, for example by providing services to or being employed by a provider that performs blue team tasks for an organisation which is part of this TLPT.

Joint or pooled tests
DORA includes a provision where many financial entities can come together and participate in a joint TLPT, with shared testing providers. These will allow for organisations that use the same ICT providers to undergo tests as a group. This accounts for different organisations that might be part of a wider group structure and to facilitate joined-up tests. These types of tests will likely require extra planning and management overhead.

Edgescan Insight: While the requirements around TLPT’s are now clearer, the practical execution of these tests will not happen for some time, likely starting in 2027. As we progress through the start of 2025, we should continue to receive further clarity from each Authority and will continue to provide updates around DORA on our blog.

We will be positioned to execute the testing required during a TLPT when this requirement needs to be executed (estimated closer to 2027) and we will work closely with several Threat Intelligence providers to help our clients complete this requirement.

The DORA Journey

16 January 2023: DORA entered into force
17 January 2024: Batch 1 of RTS and ITS technical standards released
17 June 2024: Batch 2 of RTS and ITS technical standards released
17 January 2025: DORA and technical standards (RTS & ITS) apply
Late 2026 / Early 2027: First TLPTs will start

What is TIBER-EU?

TIBER-EU is a European framework for conducting threat intelligence based red-teaming tests, which was introduced in 2018. It outlines the tactics, techniques and procedures (TTPs) that should be employed during testing, based on bespoke threat intelligence.

TIBER-EU brings together a number of entities within organisations themselves and outlines third party providers that are required to carry out the red-team capability and the threat intelligence capability. These providers work together with the organisation to conduct testing.

How We Can Help – Get in touch!

Testing, Reporting and Remediation
The most relevant part of DORA, is centred on resilience testing, which is mandating annual penetration testing for critical applications and systems. This is where Edgescan can help.

Penetration Testing
The good news is that most of the types of testing required by the standard are items that financial services organisations will be well familiar with and indeed, the majority of which will already have ongoing secure testing programmes that include these items. Once organisations identify their critical systems in scope, we can onboard them and start testing immediately.

We already provide world class penetration testing services globally, via our platform in the form of Penetration Testing as a Service (PTaaS). Our testing methodology more than meets the current criteria to cover the annual tests that are highlighted in DORA.

Extensive reporting via the platform, gives you an ongoing view of the security posture for your critical assets, with extensive reporting metrics and on-demand retesting.

Plus with remediation advice direct from our technical testing teams, demonstrating that any vulnerabilities identified have been remediated sufficiently and retested satisfactorily, will not be a problem.

Threat-Led Penetration Testing
We do already provide objective-based penetration testing (aka Red Teaming) capabilities using our skilled and experienced teams. The goal of standard penetration testing is to find ‘all the vulnerabilities’ that an adversary could leverage to breach a system or organisation. Red Teaming can be thought of as a narrow-scope penetration test, whereby you focus on an objective or end goal and leverage vulnerabilities to achieve this objective.

We will continue to keep a close eye on upcoming announcements from the ESAs and will issue further updates on our blog (link) for any official communication in relation to DORA.

If you are curious about our ongoing resilience testing or indeed any other aspect of TLTP or objective-based testing, please contact our sales team at sales@edgescan.com.

References

Digital Operational Resilience Act
DORA RTS on TLPT
Central Bank of Ireland – DORA
European Central Bank – TIBER-EU Framework

Useful Acronyms

DORA: Digital Operational Resilience Act
ECB: European Central Bank
EBA: European Banking Authority
ESMA: European Securities and Markets Authority
EIOPA: European Insurance and Occupational Pensions Authority
ESA:  European Supervisory Authority (comprises of EBA, EIOPA, ESMA)
ITS: Implementing Technical Standards
NIS2: Network & Information Security 2 Directive
PTaaS: Penetration Testing as a Service
RTS: Regulatory Technical Standards
TCT: TLPT Cyber Team
TIBER-EU: Threat Intelligence-Based Ethical Red teaming (EU)
TLPT: Thread-Led Penetration Testing
TTP: Tactics, Techniques, and Procedures