Kostas SecBlog

Using Windows Events to identify detections from MDE - Getting to know MDE Part 4

2026-07-18T00:00:00+00:00

Introduction

Welcome to the 4th part of the blog post series “Getting to know MDE”!

In this series, each post explores a different component, feature, or methodology for understanding and managing Microsoft Defender for Endpoint (MDE).

In this post, the focus is on Windows Events: What Event IDs to look for when investigating for blocks or detections generated by MDE, and using XPath queries to quickly filter for the needed Events.

Each section briefly describes a different MDE capability, including what Event IDs to look for, and an XPath query to use when searching for events of that capability.

If you want to get a tl;dr XPath query from this blog post which includes all the detections, check the last section.

This post was inspired by the previous post regarding Using KQL to identify detections from MDE, as well as the XPath queries provided by Microsoft, which were expanded upon to search for all Defender features.

So let’s get started.

Introduction
Contents
Event Viewer general info
- XPath queries
Demo Files
XPath Queries for Defender components
Conclusion

Event Viewer general info

Before starting with the Windows Events of each MDE capability, to get an idea of how a Windows Event is structured, an example of an Event reporting the detection and quarantine of a malicious file is provided below:


  
    
    1117
    0
    4
    0
    0
    0x8000000000000000
    
    37116
    
    
    Microsoft-Windows-Windows Defender/Operational
    DESKTOP-ABC
    
  
  
    Microsoft Defender Antivirus
    4.18.25110.6
    {CE415EB5-1414-4FDE-B466-541D57C3035A}
    2026-01-22T10:59:42.891Z
    2147536539
    Exploit:HTML/IframeRef.gen
    5
    Severe
    30
    Exploit
    https://go.microsoft.com/fwlink/?linkid=37020&name=Exploit:HTML/IframeRef.gen&threatid=2147536539&enterprise=0
    3
    
    2
    2
    System
    Unknown
    NT AUTHORITY\SYSTEM
    containerfile:_E:\phishing_datasets.zip; file:_E:\phishing_datasets.zip->phishing_datasets/private-phishing4.mbox->(IframeRefI)
    1
    Local machine
    0
    Unknown
    2
    Generic
    0
    2
    Quarantine
    0x00000000
    The operation completed successfully. 
    0
    0
    No additional actions required
    NT AUTHORITY\SYSTEM
    AV: 1.443.793.0, AS: 1.443.793.0, NIS: 1.443.793.0
    AM: 1.1.25110.1, NIS: 1.1.25110.1

It may look like the event includes a lot of information initially, but in reality most of it is not really useful for typical troubleshooting. Also, Windows Event Viewer parses these logs and presents them in a user-friendly format:

Every Windows Event log is divided into two main sections:

The Block: This contains the standard metadata generated by Windows. It tells when it happened, what computer it happened on (DESKTOP-ABC), and where the log is saved (Windows Defender/Operational).
The Block: This contains the custom payload for the specific event. For Windows Defender, this is where to find the exact details about the malware, its location, and what the antivirus decided to do about it.

XPath queries

A great thing about the Windows Event Viewer is that all of its logs are stored as XML data with the strictly enforced XML schema depicted previously. This allows to create custom views using XPath queries, where only the needed events are filtered. XPath (XML Path Language) is a query language used to find and extract specific information from an XML document.

Using XPath allows for creating shareable filtered views where only the detections and blocks done by Defender are shown, which results in similar reports to the ones created using the KQL queries shown in the previous post Using KQL to identify detections from MDE - Getting to know MDE Part 2.

All the XPath queries used here have the following format:

The only difference between the queries are Windows Event channel paths and the Event IDs searched for. To not repeat the explanation of the query under each section, a general explanation is provided here:

Query Id="0" Path="...": Default path to look for is “Microsoft-Windows-Windows Defender/Operational”. This is needed to be stated even though other paths are specified below because the schema rules dictate that this container must declare a starting point. Even if you try to skip it, when you save the Custom View in Windows Event Viewer, a path will be added automatically.
Select Path="Microsoft-Windows-Windows Defender/Operational">: Search in the path “Microsoft-Windows-Windows Defender/Operational”
*[System[(EventID=... or EventID=... or EventID=...)]]:
*: Search at any root event, i.e., grab all events in the log as a starting point.
[...]: The contents within the brackets in XPath are used to apply a filter to the item just before it, i.e., the asterisk, i.e., all the events.
System: As mentioned before, every Windows Event is split into two main XML sections: (metadata like time, provider and Event ID) and (the details of what happened). Here we are filtering the section.
The brackets again filter to the item just before, i.e., the section.
EventID=...: With this condition the specified Event IDs are filtered.

There are 2 ways to import an XPath query in Windows Event Viewer:

1) Create a Custom View

Open Event Viewer and click on Create Custom View....

Click on the XML tab and check the Edit query manually checkbox.

Click Yes on the warning pop-up:

You can now paste the XPath queries provided on this post and save them accordingly.

2) Import a Custom View

If you have saved any of the XPath queries as XML files, you can directly import them by clicking on the Import Custom View... in Windows Event Viewer:

Exporting Results

After importing the custom view, you can export the results to a CSV, txt, XML, or EVTX (Event Files) format by clicking on Save All Events in Custom View As...:

In the next sections, XPath queries are provided for each Defender feature. These can be used to facilitate troubleshooting towards identifying false positive detections and blocks. More details on the specific XPath queries are found in the sections below.

Demo Files

If there are no Events for a specific feature and there is a need to produce some for confirmation of the query working as expected, there are demo files provided by Microsoft, which can be used to force Defender detections and produce Windows Events.

XPath Queries for Defender components

MDAV Changes logging

Starting with the logging of changes to configurations, an interesting and useful event related to all Microsoft Defender Antivirus (MDAV) features is Event ID 5007 with description “Event when settings are changed”. A lot of events with Event ID 5007 are logged because it includes any change happening to the MDAV configuration, such as:

Signature/Engine updates
Policy syncs from Group Policy, Intune, SCCM
Cloud-Delivered Protection Adjustments
Any MDAV feature changing states (e.g., ASR rule switching from Audit mode to Block mode)

Investigating these logs is useful to identify suspicious tampering of the MDAV configuration, like disabling ASR rules or Real-Time protection, as well as to troubleshoot MDAV behavior that started unexpectedly, such as to identify when an ASR rule switched from Audit mode to Block mode.

In addition to the Event ID above, there are the following Event IDs related to changing settings in Defender:

Event ID	Description
5000	Real-time protection enabled: Microsoft Defender Antivirus real-time protection scanning for malware and other potentially unwanted software was enabled.
5001	Real-time protection disabled: Microsoft Defender Antivirus real-time protection scanning for malware and other potentially unwanted software was disabled.
5004	Microsoft Defender Antivirus real-time protection feature configuration changed.
5007	The antimalware platform configuration changed.
5009	Antispyware enabled: Microsoft Defender Antivirus enabled scanning for malware and other potentially unwanted software.
5010	Antispyware disabled: Microsoft Defender Antivirus scanning for malware and other potentially unwanted software is disabled.
5011	Malware protection enabled: Microsoft Defender Antivirus scanning for viruses is enabled.
5012	Malware protection disabled: Microsoft Defender Antivirus scanning for viruses is disabled.

To search for these Event IDs, which can be found under Applications and Services Logs > Microsoft > Windows > Windows Defender > Operational, the following XPath query can be used:

Attack Surface Reduction (ASR) rules

Attack Surface Reduction (ASR) rules in MDAV protect against risky software behavior commonly exploited, such as:

Running unsigned drivers or signed by an untrusted CA.
Injecting code into other processes.
Running Office Macros.
Running software that has low prevalence across organizations.

For more info about ASR rules, visit Microsoft’s relevant documentation.

When it comes to Windows Event logs, the ASR rule events are located in the Windows Event log under Applications and Services Logs > Microsoft > Windows > Windows Defender > Operational. The following event IDs are related to ASR rules:

Event ID	Description
1121	Event when an ASR rule fires in block mode
1122	Event when an ASR rule fires in audit mode
1129	Event when the user overrides an ASR rule block in warn mode

The following XPath query will search for ASR rule detections, blocks, and user overrides:

If blocks only are needed, just keep Event ID 1121:

Controlled Folder Access (CFA)

The basic idea with CFA is that a set of directories (folders) and a set of applications are defined. Only that set of apps is allowed to process in any way that defined set of directories. For more info, click here

When it comes to Windows Event logs, the CFA events are located under Applications and Services Logs > Microsoft > Windows > Windows Defender > Operational. The following event IDs are related to CFA:

Event ID	Description
1123	Blocked controlled folder access event
1124	Audited controlled folder access event
1127	Controlled Folder Access blocked an untrusted process from potentially modifying disk sectors. In general, these events are more severe, because they indicate that an attempt was made to write directly to the hard drive, bypassing the file system.
1128	Audited controlled folder access sector write block event. In general, these events are more severe, because they indicate that an attempt was made to write directly to the hard drive, bypassing the file system.

The following XPath query will filter for all detection and block events:

For blocks only, use the following:

Device Control

Device Control enables controls related to usage and installation of peripheral (USB/Bluetooth) or other devices on endpoints.

When it comes to Windows Event logs, for Device Control there is no specific set of events or a specific path where all the events reside. It is more complicated to identify block events. I could not find an exhaustive set of events to look for to ensure all Device Control events are monitored. The most relevant events are logged under

Security
Microsoft > Windows > Kernel > PnP > Configuration
Microsoft > Windows > DeviceSetupManager > Admin
Microsoft > Windows > PrintService > Admin

Event ID	Description
6423	The system blocked the installation of a hardware device (USB, Bluetooth radio, etc.) due to device installation restrictions.
871	A print job was explicitly blocked by Defender Device Control or Removable Storage Access Control.
808	The print spooler blocked a plug-in module from loading, often due to untrusted driver execution policies.
215	A printer driver failed to install due to a security policy or restriction.

The XPath query for the Event IDs above is this following:

Exploit Protection

Exploit Protection helps protect against malware that uses exploits to infect devices and spread. It consists of many mitigations that can be applied to either the whole OS or individual apps. For more info, click here.

When it comes to Windows Event logs, most Exploit Protection events are located in the Windows Event log under Security-Mitigations > Kernel Mode and Security-Mitigations > User Mode, while some are located in WER-Diagnostics > Operational and Win32k > Operational. The following event IDs are related to CFA:

Security-Mitigations > Kernel Mode and Security-Mitigations > User Mode

Event ID	Description
1	Arbitrary Code Guard (ACG) audit
2	Arbitrary Code Guard (ACG) enforce
3	Don’t allow child processes audit
4	Don’t allow child processes block
5	Block low integrity images audit
6	Block low integrity images block
7	Block remote images audit
8	Block remote images block
9	Disable win32k system calls audit
10	Disable win32k system calls block
11	Code integrity guard audit
12	Code integrity guard block
13	Export Address Filtering (EAF) audit
14	Export Address Filtering (EAF) enforce
15	Export Address Filtering+ (EAF+) audit
16	Export Address Filtering+ (EAF+) enforce
17	Import Address Filtering (IAF) audit
18	Import Address Filtering (IAF) enforce
19	Return-Oriented Programming (ROP) StackPivot audit
20	Return-Oriented Programming (ROP) StackPivot enforce
21	Return-Oriented Programming (ROP) CallerCheck audit
22	Return-Oriented Programming (ROP) CallerCheck enforce
23	Return-Oriented Programming (ROP) SimExec audit
24	Return-Oriented Programming (ROP) SimExec enforce

WER-Diagnostics > Operational

Event ID	Description
5	Control Flow Guard (CFG) Block

Win32k > Operational

Event ID	Description
260	Untrusted Font

The following XPath query will filter for all detection and block events for Exploit Protection:

For blocks only, the following XPath query can be used:

Network Protection, Web Protection, and SmartScreen

The combination of Network Protection, Web Protection, and SmartScreen provides the ability to block access to malicious websites, custom indicators, and web content filtering. For more info, click here.

When it comes to Windows Event logs, the Network Protection events are located in the Windows Event log under Applications and Services Logs > Microsoft > Windows > Windows Defender > Operational. The following event IDs are related to Network Protection:

Event ID	Description
1125	Event when network protection fires in audit mode
1126	Event when network protection fires in block mode

The following XPath query can be used to detect these events:

If only the block events are needed, the audit events can be omitted:

Tamper Protection

Tamper Protection blocks any changes attempted to several MDAV settings.

When it comes to events generated, the only Event ID that matters for Tamper Protection is located under Applications and Services Logs > Microsoft > Windows > Windows Defender > Operational, with Event ID 5013. The message of this Event ID is “Tamper protection blocked a change to Microsoft Defender Antivirus”, and states which setting change was blocked.

The XPath query for this Event ID is the following:

Microsoft Defender Antivirus (MDAV) Detections

MDAV is the core component of Defender. With it, process creation events and file download events from the internet are monitored. It not only uses its signature-based engine, but also predictive technologies such as machine learning and cloud-delivered protection to find attacks. For more info, click here

Regarding Event IDs, there is a thorough documentation here. The Event IDs which mostly relate to detections and blocks are the following:

Event ID	Description
1006/1116	The antimalware platform found malware or other potentially unwanted software.
1007/1117	The antimalware platform performed an action to protect your system from malware or other potentially unwanted software.
1008/1118	The antimalware platform attempted to perform an action to protect your system from malware or other potentially unwanted software, but the action failed.
1011	The antimalware platform deleted an item from quarantine.
1012	The antimalware platform couldn’t delete an item from quarantine.
1015	The antimalware platform detected suspicious behavior.
1119	The antimalware platform encountered a critical error when trying to take action on malware or other potentially unwanted software.

Regarding the first pair of Event IDs, they log the same event, but the 100x ones are older Event IDs for Windows 7, Windows 8, while the 111x ones are the new ones introduced in Windows 10. All these Event IDs are logged under Applications and Services Logs > Microsoft > Windows > Windows Defender > Operational.

The XPath query to filter for those events is the following:

As described in the Event IDs table, these events log also events of Potentially Unwanted Software/Applications (PUA). If the events not related to PUA are needed, an additional filter is needed to be applied in the “Category Name” field of the EventData. The following XPath query can be used:


  
   
   
    *[EventData[Data[@Name='Category Name']='Potentially Unwanted Software']]

The reason that the tag is used instead of filtering with a EventData[Data[@Name='Category Name']!='Potentially Unwanted Software'] condition with a != is that if there is an event that does not have a “Category Name”, then the condition would output false, removing that log from the query results.

Potentially Unwanted Apps (PUA)

PUA is a category of software that can:

Cause the machine to run slowly
Display unexpected apps
Install other software that might be unwanted

For more info, click here.

As mentioned in the previous section, PUA is logged under the same Event IDs as MDAV. Therefore, if events related only PUA are needed, an additional filter is needed to be applied in the “Category Name” field of the EventData. The XPath query that can be used is the following:

App Control for Business and AppLocker

AppLocker helps control which apps and files users can run. App Control for Business also known as Windows Defender Application Control (WDAC), formerly known as Configurable Code Integrity (CCI), is a newer solution for the same purpose, and provides more granular controls.

When it comes to Windows Events, AppLocker logs events under Application and Services Logs > Microsoft > Windows > AppLocker, while App Control logs its events under the AppLocker path as well as Application and Services Logs > Microsoft > Windows > CodeIntegrity > Operational.

The events are quite granular, with categories based on:

the file type of the event
whether the file that is about to run is there because of a Managed Installer (Intune, SCCM, etc.)
the feature that is controlling the file execution (AppLocker or WDAC)

More info:

For AppLocker only, the following Event IDs are of interest:

Event ID	Description
8000	AppID policy conversion failed. This indicates that the policy wasn’t applied correctly to the computer.
8001	The AppLocker policy was applied successfully to this computer.
8002	File was allowed to run. (.exe and .dll files.). When a process launches that matches a managed installer rule, this event is raised with PolicyName = MANAGEDINSTALLER found in the event Details.
8003	File was allowed to run but would have been prevented from running if the AppLocker policy were enforced. (.exe and .dll files.)
8004	File was prevented from running. (.exe and .dll files.)
8005	File was allowed to run. (script or .msi files)
8006	File was allowed to run but would have been prevented from running if the AppLocker policy were enforced. (script or .msi files)
8007	File was prevented from running. (script or .msi files)
8008	File: AppLocker component not available on this SKU. Indicates that the Windows edition does not support AppLocker.
8020	File was allowed to run. (packaged apps .appx, .msix)
8021	File was allowed to run but would have been prevented from running if the AppLocker policy were enforced. (packaged apps .appx, .msix)
8022	File was prevented from running. (packaged apps .appx, .msix)
8023	File was allowed to be installed. (packaged apps .appx, .msix)
8024	File was allowed to install but would have been prevented from running if the AppLocker policy were enforced. (packaged apps .appx, .msix)
8025	File was prevented from installing. (packaged apps .appx, .msix)
8027	No packaged apps can be executed while Exe rules are being enforced and no Packaged app rules have been configured.

For App Control only, the following Event IDs are of interest (the table information was taken from the previously provided link regarding Understanding App Control events:

Event ID	Description
8028	This event indicates that a script host, such as PowerShell, queried App Control about a file the script host was about to run. Since the policy was in audit mode, the script or MSI file should have run, but wouldn’t have passed the App Control policy if it was enforced.
8029	This event is the enforcement mode equivalent of event 8028.
8036	COM object was prevented from running.
8037	This event indicates that a script host checked whether to allow a script to run, and the file passed the App Control policy.
8038	Signing information event correlated with either an 8028 or 8029 event. One 8038 event is generated for each signature of a script file. Contains the total number of signatures on a script file and an index as to which signature it is. Unsigned script files generate a single 8038 event with TotalSignatureCount 0. These events are correlated with 8028 and 8029 events and can be matched using the Correlation ActivityID found in the System portion of the event.
8039	This event indicates that a packaged app (MSIX/AppX) was allowed to install or run because the App Control policy is in audit mode. But, it would have been blocked if the policy was enforced.
8040	This event indicates that a packaged app was prevented from installing or running due to the App Control policy.

There are also some logs under Applications and Services logs > Microsoft > Windows > CodeIntegrity > Operational, focused on App Control policy activation and the control of executables, dlls, and drivers.

Event ID	Explanation
3004	This event isn’t common and may occur with or without an App Control policy present. It typically indicates a kernel driver tried to load with an invalid signature. For example, the file may not be WHQL-signed (Windows Hardware Quality Labs) on a system where WHQL is required. This event is also seen for kernel- or user-mode code that the developer opted-in to /INTEGRITYCHECK but isn’t signed correctly.
3033	This event may occur with or without an App Control policy present and should occur alongside a 3077 event if caused by App Control policy. It often means the file’s signature is revoked or a signature with the Lifetime Signing EKU has expired. Presence of the Lifetime Signing EKU is the only case where App Control blocks files due to an expired signature. Try using option 20 Enabled:Revoked Expired As Unsigned in your policy along with a rule (for example, hash) that doesn’t rely on the revoked or expired cert. This event also occurs if code compiled with Code Integrity Guard (CIG) tries to load other code that doesn’t meet the CIG requirements.
3034	This event isn’t common. It’s the audit mode equivalent of event 3033.
3076	This event is the main App Control block event for audit mode policies. It indicates that the file would have been blocked if the policy was enforced.
3077	This event is the main App Control block event for enforced policies. It indicates that the file didn’t pass your policy and was blocked.
3089	This event contains signature information for files that were blocked or audit blocked by App Control. One of these events is created for each signature of a file. Each event shows the total number of signatures found and an index value to identify the current signature. Unsigned files generate a single one of these events with TotalSignatureCount of 0. These events are correlated with 3004, 3033, 3034, 3076 and 3077 events. You can match the events using the Correlation ActivityID found in the System portion of the event.
3090	Optional: This event indicates that a file was allowed to run based purely on ISG or managed installer.
3091	This event indicates that a file didn’t have ISG or managed installer authorization and the App Control policy is in audit mode.
3092	This event is the enforcement mode equivalent of 3091.

Lastly, for running files installed via a Managed Installer (Intune, SCCM, etc.), there are some events which are AppLocker events, but may be triggered by either AppLocker or App Control. More specifically, the Managed Installer feature was initially built for AppLocker. The following two are AppLocker components:

The SmartlockerFilter driver that watches the installations and identifies files created by authorized installers
The AppID service that verifies the tags.

These components are used to trust Managed Installers, and track applications which exist on the machine because of a Managed Installer. They can then be used either by AppLocker itself or by App Control to allow or block the execution of Managed Installer managed applications. In this case, the following events are of interest:

Event ID	Explanation
8030	ManagedInstaller check SUCCEEDED during Appid verification of . A user tried to run a program. Windows checked for the hidden stamp, found it, and confirmed that the trusted Managed Installed installed it. The program is allowed* to run.
8031	SmartlockerFilter detected file * being written by process *. The trusted deployment tool is actively downloading or installing a program. The Smartlocker filter driver is watching the tool write the files to the hard drive and stamps them with the hidden “trusted” tag.
8032	ManagedInstaller check FAILED during Appid verification of *. A user tried to run a program. Windows checked for the hidden stamp, but it wasn’t there. This means the user downloaded it themselves from the internet or brought it in on a USB drive. Because the security policy is set to Enforce, the program is blocked.
8033	ManagedInstaller check FAILED during Appid verification of * . Allowed to run due to Audit AppLocker Policy. This is the equivalent of Event ID 8032, but the program is not blocked, but only audited.
8034	ManagedInstaller Script check FAILED during Appid verification of *. A script tried to run, but it lacked the hidden stamp from the trusted Managed Installer. The script is blocked.
8035	ManagedInstaller Script check SUCCEEDED during Appid verification of *. A script tried to run, Windows found the hidden stamp by the trusted Managed Installer, and the script is allowed to run.

For AppLocker, the XPath query that can be used is the following:

For App Control, the XPath query that can be used is the following:

For blocks only, for both features, the following XPath query can be used:

MDE

When it comes to Microsoft Defender for Endpoint (MDE) itself, SENSE is the internal name used to refer to the behavioral sensor that powers MDE. The Windows Events created for SENSE can be found under the path Applications and Services Logs > Microsoft > Windows > SENSE > Operational. Information about the events created can be found in Microsoft’s documentation: Review events and errors using Event Viewer. The events created are only useful for troubleshooting purposes related to the SENSE service not operating as expected, but they are not related to any blocking activities. Therefore, they will not be documented in this post.

General XPath query for all detections

If there is no certainty as to which Defender feature may have blocked something, or the need may be to prove that Defender did not actually block anything, the following XPath query filters for any block or detection event done by any Defender feature:

For only block events, use the following XPath query:

Conclusion

Sometimes, it is complicated to understand what Defender blocked, when, and why the block happened. Having tools like Windows Events or Defender Advanced Hunting logs does help, but there is an overwhelming amount of logs.

Having a few queries ready to be used that filter for only the needed events cuts a few steps in the troubleshooting process. This post focused on making such queries for the Windows Events.

Hopefully these queries will help towards solving issues related to Defender, or to prove that Defender was not an issue.

Thank you for being here.

On to the next one!

Cyber Range as Code - Part 2 - Automating the base - Hypervisor and Firewall

2026-03-20T00:00:00+00:00

Introduction

Welcome to the 2nd part of the project “Cyber Range as Code”, where a Cyber Range is built using Infrastructure as Code and IT automation concepts.

If you want to see the concept as a whole, please take a look at part 1.

The basic idea is to use HashiCorp Packer, Terraform, and Ansible as the tools to build a fully functioning Cyber Range, for testing attack scenarios, implementing security standards, and architecting a secure infrastructure. The highlight of the concept is the IaC part. The whole point is to be able to recover seamlessly, i.e., all the infrastructure must be deployable automatically. If it is ever needed to redeploy the infrastructure on a new host server, it should be doable with minimal manual effort.

In Part 2, the base is built. This includes the Hypervisor itself, where Proxmox is used, as well as the central Firewall of the Cyber Range, where OPNSense is used.

Now that the picture is painted, let’s begin.

Introduction
Contents
Brief reminder of project architecture
Automate Proxmox
- Initial Setup
- Proxmox Ansible
Automate OPNSense
Next Steps
Conclusion

Brief reminder of project architecture

Below is a brief reminder of the project architecture. For more details, see my previous post Cyber Range as Code: Automating Security Lab with IaC - Part 1

The initial infrastructure architecture is the following. It is important to note here that this is the initial architecture idea, it is not final, and there may very well be changes during implementation.

As shown above, the architecture is a relatively simple and typical network infrastructure, with the following components:

The central Firewall, controlling the network traffic among the four network zones:
1. Demilitarized Zone (DMZ): The zone that exposes services which are to be served to the Internet. In the context of the project, these will be served to the local network, and, most importantly for the project, will be accessible by the “External Attacker”, enabling for attack scenarios initiated from the “Internet”.
2. Internal Zone: The zone where all the internal servers and services will reside. This includes:
  - Any internal servers, e.g., SQL Servers and AD DC, hosting and serving information which is destined to be consumed by internal resources only.
  - Security Tools, including the SIEM/XDR/Monitoring tools.
3. End Users: The last zone will be for the End Users, where typical workstation VMs will reside, and have defined access to specific servers/services and to the internet.
4. WAN Zone: This is where the “Internet”, in the context of the Infrastructure, lives. This is where:
  - The PC from which the Proxmox management will be done, running the Packer, Terraform, and Ansible.
  - The External Attacks will occur from.
There will also be Internal Attack Simulations, executed from within the different zones, bypassing the firewall (“assume breach”).

The code of the project can be found in the project’s GitHub Repository

Automate Proxmox

Regarding automating the configuration of Proxmox, this essentially includes:

Initial Setup

This step includes the installation of Proxmox on a host machine, setting up the network configuration, i.e., IP Address, Gateway, DNS, and setting up the root user credentials. One point of improvement here is to potentially automate the parameters during the initial installation of Proxmox. This is supported by Proxmox with the Automated Installation. This would be a very interesting improvement point towards a fully hands-off installation and configuration. But this will be left as a future exercise. For now, these steps were done manually, and can be found in my previous post.

After the initial setup and reaching the point of being able to sign in to the Proxmox GUI, the next step is to set up Proxmox so that Ansible can communicate with it.

This means:

Verify SSH Access: A rule to allow SSH access is actually created by default when enabling the Proxmox firewall, so no manual rule creation is required here, see Default firewall rules.
Create a role for all automation steps: The following permissions are required for all steps of this project:
Create user: The packer user created in the initial lab (part 1) is used. It is a simple user and the above role is assigned to it.
API key: An API key was generated and assigned to the above user, in order for Ansible to execute API calls to Proxmox.

Proxmox Ansible

In this step, the Ansible Control Node (i.e., the PC where all the code is written and executed) is prepared for Proxmox configurations.

Ansible Collection

Ansible Collections are a distribution format for Ansible content that can include playbooks, roles, modules, and plugins. Essentially, a collection is like a program library containing all the functionalities required for configuring a specific component.

The Community Proxmox Ansible Collection was used for this project, which can be found here.

To install the collection the following command was executed:

ansible-galaxy collection install community.proxmox

Additionally, the community.proxmox ansible collection requires the Python libraries proxmoxer and requests to talk to the Proxmox API. To install them, run one of the following:

# If you are on Ubuntu/Debian in WSL. Since the project is ran on WSL, this was selected:
sudo apt update
sudo apt install python3-proxmoxer python3-requests

# OR if you prefer pip:
pip3 install proxmoxer requests

Without these libraries, the following error is shown when trying to run the Ansible playbook:

Setting up SSH

Since Ansible executes SSH using certificate-based authentication, the public key of the Ansible Control Node must be added to Proxmox’s trusted keys. Since the public key is the one that exists on the physical PC currently running all the commands, the easiest way to do this task is with the following command:

ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.0.50

After running the command, it is now possible to ping Proxmox via Ansible:

In the above command, the inventory.ini file includes machines which are in the scope of configurations via Ansible. For now it only includes the Proxmox host:

[proxmox]
192.168.0.50 ansible_user=root

Initializing Ansible for Proxmox

The Community Proxmox Ansible Collection requires a few specific parameters for every task being run, and it is also supported to configure these parameters as Environment Variables:

Parameter	Comments
api_host (required)	Specify the target host of the Proxmox VE cluster. Uses the PROXMOX_HOST environment variable if not specified.
api_password	Specify the password to authenticate with. Uses the PROXMOX_PASSWORD environment variable if not specified.
api_port	Specify the target port of the Proxmox VE cluster. Uses the PROXMOX_PORT environment variable if not specified.
api_token_id	Specify the token ID. Uses the PROXMOX_TOKEN_ID environment variable if not specified.
api_token_secret	Specify the token secret. Uses the PROXMOX_TOKEN_SECRET environment variable if not specified.
api_user (required)	Specify the user to authenticate with. Uses the PROXMOX_USER environment variable if not specified.

Therefore, the following Envars are set before running Proxmox Ansible playbooks:

export PROXMOX_HOST="192.168.0.50"
export PROXMOX_USER="packer@pve"
export PROXMOX_TOKEN_ID="packer-token"
export PROXMOX_TOKEN_SECRET=""

Running Ansible to configure Proxmox

After setting up Ansible, the next step is to write and execute the Ansible Playbooks to apply the configurations to Proxmox, which were initially applied manually and described on my previous post and part 1 of this series. The Ansible playbook can be found here, and is also depicted below, with more details in the comments and afterwards:

---
#before running, run the following:
# export PROXMOX_HOST="192.168.0.50"
# export PROXMOX_USER="packer@pve"
# export PROXMOX_TOKEN_ID="packer-token"
# export PROXMOX_TOKEN_SECRET=""

- name: Configure Proxmox via API
  hosts: proxmox
  # because this play configures Proxmox via API, with the keyword below
  # the playbook is executed at the ansible control node instead of
  # SSH'ing and running it locally on the managed node.
  connection: local
  gather_facts: false

  tasks:
    # Task to configure the central DNS settings of Proxmox
    - name: Set DNS
      community.proxmox.proxmox_node:
        node_name: kkproxmox
        dns:
          dns1: 1.1.1.1
          dns2: 8.8.8.8
          search: kostas.local
      # delegate_to: localhost # this is not needed if the
      # "connection: local" is at the start of the playbook

    # Task to create the Virtual Network Bridges in Proxmox
    - name: Set Network
      community.proxmox.proxmox_node_network:
        node: kkproxmox
        autostart: true
        iface_type: bridge
        iface: "{{ item.iface }}"
        cidr: "{{ item.cidr }}"
        comments: "{{ item.comments }}"
        bridge_ports: "{{ item.bridge_ports }}"
        gateway: "{{ item.gateway }}"
      loop:
        - { iface: "vmbrWAN10", cidr: "10.0.10.2/24", comments: "WAN" , bridge_ports: "" , gateway: "" }
        - { iface: "vmbrDMZ20", cidr: "10.0.20.2/24", comments: "DMZ" , bridge_ports: "" , gateway: "" }
        - { iface: "vmbrIZ30", cidr: "10.0.30.2/24", comments: "Internal Zone" , bridge_ports: "" , gateway: "" }
        - { iface: "vmbrEUZ40", cidr: "10.0.40.2/24", comments: "End User Zone" , bridge_ports: "" , gateway: "" }
        - { iface: "vmbr0", cidr: "192.168.0.50/24", comments: "Bridge to home router" , bridge_ports: "nic1" , gateway: "192.168.0.1" }
      
    # Task to connect the physical interface to the physical network
    - name: Set Network interfaces
      community.proxmox.proxmox_node_network:
        node: kkproxmox
        iface_type: eth
        iface: nic1
        comments: "Physical Interface of Proxmox to home router"

    # Apply the above network configurations
    - name: Apply Network
      community.proxmox.proxmox_node_network:
        node: kkproxmox
        state: "apply"

    # Set firewall aliases to be used for the firewall rules below
    - name: Set Firewall Aliases
      community.proxmox.proxmox_firewall:
        level: cluster
        aliases:
          - name: subnet10
            cidr: "10.0.0.0/8"
          - name: subnet172
            cidr: "172.16.0.0/12"
          - name: subnet192
            cidr: "192.168.0.0/16"
    
    # security groups to be applied on the different Proxmox levels
    - name: Create Firewall Security Groups
      community.proxmox.proxmox_firewall:
        level: cluster
        group_conf: true # Whether security group should be created or deleted
        state: present # create the group
        group: "{{ item.group }}"
      loop:
        - { group: "allowguesttraffic" }
        - { group: "blockhomenwtraffic" }
    
    # configure rules of the above security groups
    - name: Set Firewall Security Groups rules
      community.proxmox.proxmox_firewall:
        level: group
        state: present # Create/update/delete firewall rules or security group.
        update: true # If state=present and if one or more rule/alias/ipset already exists it will update them.
        group: "{{ item.group }}"
        rules: "{{ item.rules }}"
      loop: 
        - group: allowguesttraffic # allow/not control traffic from the lab's subnets.
          rules:
            - type: in
              action: ACCEPT
              source: dc/subnet10
              pos: 0
              log: nolog
              enable: true
        - group: blockhomenwtraffic # block traffic to the home physical network.
          rules:
            - type: in
              action: ACCEPT
              source: dc/subnet192
              pos: 0
              log: nolog
              enable: false # disabled but kept just in case, because proxmox has this rule by default
              comment: "Allow ProxMox Management"
              dest: 192.168.0.50
              dport: 8006
              proto: tcp
            - type: in
              action: ACCEPT
              source: dc/subnet192
              pos: 1
              log: nolog
              enable: true
              comment: "Allow OPNSense Management"
              dest: 192.168.0.51
              dport: 443
              proto: tcp
            - type: out
              action: REJECT
              pos: 2
              log: nolog
              enable: true
              comment: "Block Local Traffic"
              dest: dc/subnet192

    # apply security groups for both cluster and node levels
    - name: Apply Security Groups
      community.proxmox.proxmox_firewall:
        level: "{{ item.level }}"
        node: "{{ item.node }}"
        update: true
        state: present
        rules:
          - action: blockhomenwtraffic # same rule for both cluster and node levels
            pos: 0
            type: group
            enable: true
      loop:
        - { level: cluster, node: "" }
        - { level: node, node: kkproxmox }

    # - name: Configure Update Repositories - left for future update

    #The below 2 tasks are useful if it is needed to print the current configuration
    # - name: Get Firewall Config
    #   community.proxmox.proxmox_node_info:
    #     # level: node
    #     # node: kkproxmox
    #   register: debug_data

    # - name: Show debug data
    #   debug:
    #     var: debug_data

- name: Configure Proxmox via SSH
  hosts: proxmox
  gather_facts: false

  tasks:
    - name: Check current Cluster Firewall status
      ansible.builtin.command: pvesh get /cluster/firewall/options --output-format json
      register: cluster_fw_status
      changed_when: false

    - name: Enable Firewall on Cluster
      ansible.builtin.command: pvesh set /cluster/firewall/options -enable 1
      # Only run this command IF the 'enable' value is not 1 (or if it doesn't exist yet)
      when: (cluster_fw_status.stdout | from_json).enable | default(0) | int != 1

    - name: Check current Node Firewall status
      ansible.builtin.command: pvesh get /nodes/kkproxmox/firewall/options --output-format json
      register: node_fw_status
      changed_when: false

    - name: Enable Firewall on Node
      ansible.builtin.command: pvesh set /nodes/kkproxmox/firewall/options -enable 1
      # Only run this command IF the 'enable' value is not 1 (or if it doesn't exist yet)
      when: (node_fw_status.stdout | from_json).enable | default(0) | int != 1

With this playbook, the following Proxmox settings are configured:

DNS Settings
Network Interfaces
Network Configuration
Firewall Rules
Enabling Firewall

One point to note is that the playbook consists of two plays:

Configuring Proxmox via API: This is the bulk of the configuration, because the Proxmox Community Ansible collection supports most of the configurations required.
Configuring Proxmox via SSH: This contains tasks that cannot be executed using the avaialble Ansible collection. It includes executing manual commands on Proxmox using its CLI pvesh.

To run this playbook, the following command is executed:

ansible-playbook -i inventory.ini proxmox_config.yml

Running the playbook outputs the following:

As shown in the screenshot, the configuration is already applied, and all tasks output status ok. The two tasks that output skipped are the ones executed via SSH. Because they manually set values on Proxmox, the when keyword is used to skip the execution if the value is already set. Otherwise, the task would always output status changed, making the total output of the playbook confusing to read.

Issues met and resolutions

While developing the Ansible playbook, the follow issues were faced:

CRLF vs LF in VS Code

In the current setup of writing the code, VS Code is used on Windows (PC) and Windows Subsystem for Linux (WSL) is used to run all the code. There is a discrepancy on how the ‘new line’ character is saved in Windows and expected to be read in Linux. More specifically:

Windows saves a new line as CRLF (\r\n).
Linux expects a new line to be LF (\n).

When Ansible (running in Linux) reads the inventory.ini file, it sees the carriage return as part of the text. It reads the group name as [proxmox\r]. The playbook, on the other hand asks for proxmox. Since proxmox does not equal proxmox\r, Ansible skips it.

To fix this, in VS Code, the new line character needs to be changed on the bottom right of the window for CRLF to LF:

After configuring Proxmox, the next step is to automatically configure the central Firewall, which is OPNSense.

Automate OPNSense

OPNSense Packer

As described on the first post of the series, the idea of this lab is to build VMs automatically in 3 steps:

Using Packer, build a relatively blank template (more on the “relatively” part later)
Use that template with Terraform to provision VMs ready to be configured by
Ansible, where the rest of the configuration and final touches happen.

Regarding OPNSense specifically, which is actually a custom machine based on FreeBSD, unfortunately it is not fully designed to be deployed and configured using automation tools. Therefore, several workarounds were needed in order to achieve an automated deployment.

First of all, in order to get the initial “blank” state of the machine, the most controllable way to do that was by configuring an OPNSense VM manually exactly up to and not any further than the point where it functions and is ready to be used by Terraform, without any more settings configured. These configurations are:

Assigning the WAN interface
Assigning the WAN interface IP with DHCP (this will be changed with Terraform)
Enabling SSH for management
Installing the QEMU agent so that Proxmox will be able to read the IP address assigned to the VM
Auto-start the QEMU agent service

After applying the above configurations manually to the OPNSense VM, the configuration of the machine was exported to a file config.xml, which can be found here. This XML file is the configuration file of OPNSense, contains all the settings previously configured, and can be imported on a clean-slate OPNSense and have the configuration applied to it.

This config.xml file contains all the information required for a VM ready to be used by Terraform.

The idea is to use the config.xml and import it on a new OPNSense VM by utilizing its Configuration Importer feature, where during boot time there is the option to select to import a config.xml file from an external drive (in this case a CD created by Packer containing the above exported config.xml).

The packer template can be found under the project’s repository, and is also presented below, with several comments to explain how it works:

packer {
  required_plugins {
    name = {
      version = "~> 1"
      source  = "github.com/hashicorp/proxmox"
    }
  }
}

# Declare variables, we will pull them later in the packer build command
variable "proxmox_api_url" { type = string }
variable "proxmox_api_token_id" { type = string }
variable "proxmox_api_token_secret" {
  type      = string
  sensitive = true
}

source "proxmox-iso" "opnsense" { #Resource type and local name
  proxmox_url = var.proxmox_api_url
  username    = var.proxmox_api_token_id
  token       = var.proxmox_api_token_secret
  # Skip TLS Verification for self-signed certificates
  insecure_skip_tls_verify = true
  qemu_agent = true # Default is true anyway
  node = "kkproxmox"
  vm_id = 1001
  vm_name = "opnsense-template"
  ssh_username = "root"
  ssh_password = "opnsense" # Default root password, can be changed later.
  ssh_timeout = "20m"
  cores = 4
  memory = 4096 # RAM must be more than 3GB, otherwise the boot_command is different and will not work
  os = "other" # for FreeBSD, you choose "other"
  cpu_type = "host"
  scsi_controller = "virtio-scsi-single"

  boot_iso {
    # type = "scsi"
    type = "ide"
    # ISO stored locally on Proxmox. In the future this can be changed to downloading from the internet.
    iso_file = "local:iso/OPNsense-25.7-dvd-amd64.iso"
    iso_checksum = "sha256:e4c178840ab1017bf80097424da76d896ef4183fe10696e92f288d0641475871"
    unmount = true
  }

  # this will created a cd in "cd1", which will be selected in the boot_command
  additional_iso_files {
    cd_content = {
    "conf/config.xml" = templatefile("${path.root}/conf/config.xml", {
      # pull the public SSH key, base64 encode it, and write it in config.xml
      dynamic_ssh_key = base64encode(file("~/.ssh/id_rsa.pub"))})
    }
    cd_label = "config"
    iso_storage_pool = "local"
  }
  
  network_adapters {
    model  = "virtio"
    bridge = "vmbr0" # Will change it in the Terraform configuration, this is only for packer.
  }

  disks {
    disk_size    = "20G"
    storage_pool = "local-lvm"
    type         = "scsi"
    ssd          = true
  }

  boot_command = [
    # There is already a 10 sec wait for boot, adding another 12.
    # Start configuration importer and select cd1 where the cd_content is stored.
    "cd1",

    # Put the default credentials to install the OS.
    "installer",
    "opnsense",
    
    # Going through the installation options:
    # Accept default Keymap and ZFS installation.
    "",
    
    # Confirm formatting (Yes) and wait 2 minutes for installation.
    "",
    
    # Do not change password and reboot.
    "",

    # Enable qemu agent service autostart and Update from console to latest version,
    # because qemu requires the latest OPNSense version.
    "rootopnsense",
    "8sysrc qemu_guest_agent_enable='YES'exit",
    "12yq"
  ]
}

build {
  sources = ["source.proxmox-iso.opnsense"]
}

Some interesting notes to point out:

1. config.xml import

In the CD that is being mounted which contains the config.xml file, the SSH public key of the host machine is being written to the config by creating a dynamic variable in the config.xml. This is done by using the templatefile function, and providing the path of the public key to it, as shown in the code snippet below:

cd_content = {
"conf/config.xml" = templatefile("${path.root}/conf/config.xml", {
  # pull the public SSH key, base64 encode it, and write it in config.xml
  dynamic_ssh_key = base64encode(file("~/.ssh/id_rsa.pub"))})
}

Note that cd_content requires one of the following tools to create the CD: xorriso, mkisofs, hdiutil, oscdimg

In this case, xorriso was installed

sudo apt install xorriso

In the config.xml file, in order to make it so that it expects a variable in the SSH key field , the dynamic variable was defined as follows:


  0
  root
  0
  system
  
  ${dynamic_ssh_key}
  
  
  $2y$10$YRVoF4SgskIsrXOvOQjGieB9XqHPRra9R7d80B3BZdbY/j21TwBfS
  
  
  
  
  
  
  
  System Administrator

As shown, for SSH keys, the dynamic variable ${dynamic_ssh_key} is provided instead of a static one.

2. boot command

Additionally, the boot_command includes all the commands executed via the keyboard during the installation of OPNSense using the configuration importer feature. These commands are described below:

Wait enough time to get to the option to use the configuration importer.
Write the external drive’s name (cd1) to pull the config.xml file from.
Install the configuration by logging in with the installer user.
Go through the installation steps (accept default keymap, select ZFS installation, select disk, confirm format and reboot)
Wait for the installation and reboot to complete.
Open a shell (press option “8”), and pass the variable qemu_guest_agent_enable='YES' to auto start the QEMU agent service on boot.
Update OPNSense to the latest version (press option “12”), because QEMU requires that.

The commands to build the Packer template are:

packer init .
packer validate -var-file="../credentials.pkrvars.hcl" .
packer build -var-file="../credentials.pkrvars.hcl" .

The file credentials.pkrvars.hcl contains the credentials used by Packer and the other components:

proxmox_api_url          = "https://192.168.0.50:8006/api2/json"
proxmox_api_token_id     = "packer@pve!packer-token"
proxmox_api_token_secret = ""
ubuntu_pw = "lab-admin"
opnsense_pw = "opnsense-admin"

The output of running the Packer template build is the following:

OPNSense Packer Issues and resolutions

During development of Packer OPNSense, the following issues were identified:

The cloud-init limitation:
- The Issue: OPNsense is built on FreeBSD and functions as an appliance where the /conf/config.xml is the absolute source of truth. It does not officially support standard Linux cloud-init for initial provisioning (like setting IPs or users).
- The Solution: OPNsense’s native Configuration Importer was utilized. By packaging a pre-configured config.xml file into a virtual CD drive, OPNsense digests the configuration during the boot sequence and permanently bakes it into the hard drive during the installation.
QEMU guest agent issues with OPNSense version:
- The Issue: Packer needs the QEMU Guest Agent to discover the VM’s dynamic IP address to connect via SSH. However, the OPNsense base ISO (25.7) was out of date with the plugin repository, meaning the agent would install but would not start its service. It was a classic ‘chicken-and-egg’ scenario: Packer couldn’t SSH in to run the update because the missing QEMU agent meant the IP was undetectable, but the QEMU agent couldn’t be installed without first running the update.
- The Solution: The setup was baked in the boot_command command list. Instead of stopping after the installation reboot, the command now continues typing in the console to drop into the shell, enabling the service to auto-start (sysrc qemu_guest_agent_enable=’YES’), and trigger a system update via console option 12. This updates the OS and pulls the agent before Packer ever tries to connect.
OPNsense Firewall Blocking Packer
- The Issue: OPNsense is a default-deny firewall that drops all WAN traffic. Packer needs to connect via the WAN interface (mapped to your Proxmox bridge) to finalize the template.
- The Solution: By ensuring no lan interface was strictly mapped initially, OPNsense’s safety mechanisms trigger the Anti-Lockout rule on the WAN interface, automatically opening port 22 and allowing Packer/Ansible to connect without the need to add a firewall rule to explicitly allow this traffic. This rule will be created on the next phases anyway.

After finishing with Packer, the result is a “blank” template, ready to be used by Terraform to provision a VM with the final settings.

OPNSense Terraform

When it comes to using Terraform for provisioning OPNSense, the process was initially straightforward, but, during development, more and more settings were added to the Terraform phase, as it was discovered that Ansible for OPNSense does not provide enough flexibility to implement what was needed. Therefore, more implementation points were moved from the “Ansible phase” to the “Terraform phase”.

The code can be found in the project’s repository and is also written below:

resource "proxmox_vm_qemu" "c-opnsense" {
    name = "c-opnsense"
    target_node = "kkproxmox"
    clone = "opnsense-template"
    agent = 1 #enable QEMU guest agent
    memory = 4096
    balloon = 4096
    bios = "seabios"
    scsihw = "virtio-scsi-single"
    os_type = "other"
    skip_ipv6 = true

    cpu {
        cores = 4
        sockets = 1
        type = "host"
    }

    disk {
        slot = "scsi0"
        cache = "none"
        discard = true
        iothread = true
        emulatessd = true
        asyncio = "io_uring"
        size = "20G"
        type = "disk"
        storage = "local-lvm"
        format = "raw"
    }

    startup_shutdown {
        order = -1
        shutdown_timeout = -1
        startup_delay = -1
    }

    # Configure the network interfaces
    network {
        id = 0
        model = "virtio"
        bridge = "vmbr0"
        firewall = true
    }
    
    network {
        id = 1
        model = "virtio"
        bridge = "vmbrDMZ20"
        firewall = true
    }

    network {
        id = 2
        model = "virtio"
        bridge = "vmbrIZ30"
        firewall = true
    }

    network {
        id = 3
        model = "virtio"
        bridge = "vmbrEUZ40"
        firewall = true
    }

    # connection used for the provisioners below
    connection {
      type = "ssh"
      user = "root"
      private_key = file("~/.ssh/id_rsa")
      host = self.ssh_host
    }

    # replace the config.xml file with the one below.
    # This one uses dynamic variables to also configure the network interfaces IPs.
    provisioner "file" {
      content = templatefile("${path.module}/template_config_opnsense_lab.xml",
        {
            dynamic_ssh_key = base64encode(file("~/.ssh/id_rsa.pub"))
            wan_if = "vtnet0"
            wan_descr = "WAN"
            wan_ip = "192.168.0.51"
            wan_subnet = "24"
            wan_gw = "WAN_GW"
            dmz20_if = "vtnet1"
            dmz20_descr = "DMZ20"
            dmz20_ip = "10.0.20.1"
            dmz20_subnet = "24"
            iz30_if = "vtnet2"
            iz30_descr = "IZ30"
            iz30_ip = "10.0.30.1"
            iz30_subnet = "24"
            euz40_if = "vtnet3"
            euz40_descr = "EUZ40"
            euz40_ip = "10.0.40.1"
            euz40_subnet = "24"
        }
      )
      destination = "/conf/config.xml"
    }

    # reboot the machine so that the new config.xml is applied
    provisioner "remote-exec" {
      inline = [ 
        "echo 'Injecting Cyber Range Topology by restarting OPNSense. VM will restart in 3 seconds...'",
        "daemon -f /bin/sh -c 'sleep 3; /sbin/reboot'"
       ]
    }
}

To run the terraform configuration:

terraform init
terraform plan -var-file=../packer/credentials.pkrvars.hcl
terraform apply -var-file=../packer/credentials.pkrvars.hcl

The var-file is the one used also in Packer and contains the connection variables so that Terraform can talk to Proxmox.

Running the terraform configuration outputs the following. Note that 2 additional test Linux VMs are also provisioned, as they were described in the previous post, but there is no reason to get into them in this post. The interesting lines are the ones starting with proxmox_vm_qemu.c-opnsense, because these ones are about the OPNSense VM:

A few interesting notes to point out:

Configuring settings using the config.xml file: As mentioned previously, OPNSense is not made to be configured “as-code” natively, i.e., by using Ansible. Therefore, several settings cannot be configured using Ansible. There is effort put by the community towards achieving that, by implementing more API calls. However, for now, a few basic settings, like configuring the network interfaces, were configured using the config.xml file of OPNSense. All these workarounds made me question whether OPNSense is the right tool for an IaC pipeline, and I thought of switching to something more programmable like VyOS, but for now I pushed through with OPNSense. The config.xml file used can be found here

Dynamic variables: As was the case with Packer, too, several variables in the config.xml file were written dynamically. The config.xml file part where these variables are expected is shown below:

 
   ${wan_if}
   ${wan_descr}
   1
   
   1
   ${wan_ip}
   ${wan_subnet}
   ${wan_gw}
 
 
   ${dmz20_if}
   ${dmz20_descr}
   1
   
   ${dmz20_ip}
   ${dmz20_subnet}
 
 
   ${iz30_if}
   ${iz30_descr}
   1
   
   ${iz30_ip}
   ${iz30_subnet}
 
 
   ${euz40_if}
   ${euz40_descr}
   1
   
   ${euz40_ip}
   ${euz40_subnet}

config.xml firewall rules needed: The Ansible collections used in the next phase require API connectivity and SSH access. Therefore, two firewall rules needed to be injected into the config.xml file:

 
   pass
   wan
   inet
   keep state
   in
   1
   1
   tcp
   
     1
   
   
     wanip
     22
   
   Allow SSH to OPNSense
 
 
   1
   keep
   
   pass
   1
   0
   wan
   in
   inet
   TCP
   wan
   0
   wanip
   0
   443
   0
   1
   0
   0
   0
   Allow HTTPS to OPNSense WebUI

config.xml API key needed: The Ansible collections used in the next phase require API connectivity, and therefore an API key. This was generated manually from OPNSense, and the configuration was added to the config.xml file:

 
   0
   root
   0
   system
   
   ${dynamic_ssh_key}
   
   
   $2y$10$YRVoF4SgskIsrXOvOQjGieB9XqHPRra9R7d80B3BZdbY/j21TwBfS
   
   
   
   
   ooyOnJ5Y7NiPxeTXkTxEgM+9steLsU+I+UehjuiXtNdBL0ckTvUQM6PWa5AxpdnUXGLGLtyRQFNCnJI8|$6$$c9GWtGrIy25Ez358v52fDrfyTfD3Q5rnzVlc7je/MKL0EzxnTOrOL0mnSh78O.t6iA8hrtd4.OfsWUvUhJgCl0
   
   
   System Administrator

Applying the configuration - Reboot the machine: After provisioning the VM, it has the final configuration written to the /conf/config.xml file, but it has not pulled and applied it yet. It needs to restart to apply it. The provisioner remote-exec was used to execute a reboot command. However, if the reboot command is executed before the script and the terraform configuration finishes, then terraform will believe that the provisioning failed. Therefore, a daemon was started that will reboot the machine in 3 seconds. So, terraform will be able to finish successfully and consider the machine fully provisioned, and then the machine will reboot to pull the configuration.
```
 provisioner "remote-exec" {
   inline = [ 
     "echo 'Injecting Cyber Range Topology by restarting OPNSense. VM will restart in 3 seconds...'",
     "daemon -f /bin/sh -c 'sleep 3; /sbin/reboot'"
    ]
 }
```

In conclusion, while there were not any noteworthy issues by using Terraform for OPNSense, Terraform was actually used to solve some issues presented by Ansible, which is described in the next section.

OPNSense Ansible

As mentioned previously, OPNSense is not designed to be configured through command line automatically. It is a GUI-first firewall. However, efforts have been put towards improving that, and an API is built through which most of its settings can be set.

Considering that the API itself is not yet fully functioning, it was even more difficult to find an Ansible Collection utilizing the API and supporting all the configurations needed to be implemented for this project.

The following Ansible collections were found which were all used in combination:

oxlorg.opnsense: most configurations supported

To install it, run:

 sudo apt install python3-pip
 python3 -m pip install --upgrade httpx
 # latest version:
 ansible-galaxy collection install git+https://github.com/O-X-L/ansible-opnsense.git
    
 # stable/tested version:
 ansible-galaxy collection install git+https://github.com/O-X-L/ansible-opnsense.git,25.7.8
 ## OR
 ansible-galaxy collection install oxlorg.opnsense # This option was selected for this project.

puzzle.opnsense: Does not have many configurations, but it supports network interface assignments.

To install it, run:
```
 ansible-galaxy collection install puzzle.opnsense
```

The Ansible playbook for OPNSense with comments can be found here and is provided below:

---
- name: Configure OPNSense via SSH
  hosts: opnsense
  # connection: local  # execute modules on firewall
  gather_facts: false
  
  tasks:
    - name: Assign Interfaces
      puzzle.opnsense.interfaces_assignments:
        description: "{{ item.description }}"
        device: "{{ item.device }}"
        identifier: "{{ item.identifier }}"
      loop:
        - { description: "DMZ20", device: "vtnet1", identifier: "opt1" }
        - { description: "IZ30", device: "vtnet2", identifier: "opt2" }
        - { description: "EUZ40", device: "vtnet3", identifier: "opt3" }
        - { description: "WAN", device: "vtnet0", identifier: "wan" }


- name: Configure OPNSense via API
  hosts: opnsense
  connection: local  # execute modules on controller
  gather_facts: false
  vars:
    # because this collection is designed to be ran on the OPNSense, it was trying to find
    # the default FreeBSD python path, instead of the one running locally. Setting this
    # variable defines the correct path for ansible.
    ansible_python_interpreter: "{{ ansible_playbook_python }}"
  module_defaults:
    group/oxlorg.opnsense.all:
      ssl_verify: false
      firewall: "{{ inventory_hostname }}" # pull the IP directly from invetory.ini
      # export of generated API keys from OPNSense
      api_credential_file: "{{ playbook_dir }}/OPNsense.internal_root_apikey.txt"

  tasks:
    # OPNSense Ansible and API do not support configuring ISC DHCP, so Kea DHCP was used instead
    - name: Enable Kea DHCP EUZ40
      oxlorg.opnsense.dhcp_general:
        enabled: true
        interfaces: ['opt3']
        fw_rules: true
        
    - name: Set Kea DHCP EUZ40 Subnet
      oxlorg.opnsense.dhcp_subnet:
        subnet: "10.0.40.0/24"
        pools: ["10.0.40.150-10.0.40.200"]
        dns: ["1.1.1.1", "8.8.8.8"]
        ipv: 4

    - name: Set Firewall Rules
      oxlorg.opnsense.rule_multi:
        rules:
        # This rule is not needed because it is needed and was added before Ansible, because
        # this Ansible collection talks via API, i.e., via port 443.
        # - description: 'Allow HTTPS to OPNSense WebUI' #name: 'Allow HTTPS to OPNSense WebUI'
        #   source_net: 'wan'
        #   destination_net: 'wanip'
        #   destination_port: 443
        #   interface: ['wan']
        #   protocol: 'TCP'
        #   action: 'pass'

        - description: 'EUZ40 allow all traffic' #name: EUZ40 allow all traffic
          source_net: 'any'
          destination_net: 'any'
          # destination_port: leave empty for all
          interface: ['opt3']
          protocol: 'any'
          action: 'pass'
          
        match_fields: ['description']
        reload: true

    # These tasks can be used to retrieve information
    # - name: Listing
    #   oxlorg.opnsense.list:
    #     target: 'rule'
    #   register: existing_entries

    # - name: Printing rules
    #   ansible.builtin.debug:
    #     var: existing_entries.data

A few interesting notes about the Ansible playbook:

API key definition: the API key used to authenticate is pulled from the path /OPNsense.internal_root_apikey.txt. This file was exported manually from an OPNSense, as described in the previous section, and saved at the path above. It is noteworthy that it is not supported to modify this file at all. The ansible tasks are looking for the format below:

Changing the format in any way results in the following error:

After configuring OPNSense with Ansible, the next step is to configure Proxmox itself for the OPNSense VM.

Configuring Proxmox for OPNSense with Ansible

In addition to configuring OPNSense itself, some settings need to be configured on Proxmox regarding the hosted OPNSense VM. More specifically, these include configuring the Proxmox firewall settings for the VM.

The Ansible playbook for this purpose is relatively short, can be found here and is the following:

---
- name: Config Proxmox for OPNSense
  hosts: proxmox
  connection: local
  gather_facts: false

  tasks:

    - name: Config Firewall OPNSense
      community.proxmox.proxmox_firewall:
        level: vm
        vmid: 101
        update: true
        state: present
        rules:
          - action: allowguesttraffic # same rule for both cluster and node levels
            pos: 0
            type: group
            enable: true
          - action: blockhomenwtraffic
            pos: 1
            type: group
            enable: true

    # These tasks can be used to retrieve the current config
    # - name: Config Firewall OPNSense
    #   community.proxmox.proxmox_firewall_info:
    #     level: vm
    #     vmid: 100
    #   register: debug_fw_opnsense

    # - name: Show debug
    #   debug:
    #     var: debug_fw_opnsense

The playbook is straightforward, and assigns the security groups (created in the previous sections) to the VM.

To run the playbook, as was the case with the playbook for proxmox itself, the envars described in the previous section need to be set firstly. To run the playbook, run:

ansible-playbook -i inventory.ini proxmox_opnsense_config.yml

Running the playbook outputs the following:

Now that Proxmox is configured for the hosted OPNSense VM, this part of the project is finished. Now let’s take a look a potential next steps.

Next Steps

The next steps include implementing some not mandatory final touches on the automation part of Proxmox and OPNSense, including:

Automating the installation process of Proxmox itself. This is supported by Proxmox with the Automated Installation.
Automating the users creation to Proxmox
Automating the changing of update repositories of Proxmox from the default ones which require a license to the free ones
Adding the ISO files to Proxmox, or alternatively switching the Packer templates to download the ISOs from the internet directly instead of trying to grab them from Proxmox locally.
Creating a file serving as the single source of truth. All the variables used in the project will be defined in that file, including IP addresses, usernames, etc. The file will be referenced by all code.

Additionally, the next steps include building out the rest of the components described in the first post.

Conclusion

In this post, the base was built under the context and concept of the project, i.e. utilizing IaC.

As it is probably understood through the post, building each component of this project proved to be more complex than expected, as each component has its own peculiarities and workarounds required in order for it to work in the context of IaC and Configuration Management.

Cyber Range as Code: Automating Security Lab with IaC - Part 1

2026-02-02T00:00:00+00:00

Introduction

What is a Cyber Range?

A Cyber Range is a virtual environment for cybersecurity training, testing, and research that simulates real-world networks and cyberattacks.

I always wanted to build a security homelab, where I will have the freedom and the infrastructure to build, secure, attack, monitor, and defend a full-on infrastructure. At the same time, I want to build all of this “as code”. I really believe in converting everything to “as code”, because this approach solves many issues of manual work, like manual maintenance, configuration drift, complex change history, and no version control. It also facilitates the ability to rebuild the infrastructure as quickly as possible. I was thinking of naming this project “Security Playground” at first or something like that, but I thought “Cyber Range as Code” was more catchy and it grasped the two main concepts that I want to focus on for this project.

The end goal of this project is to build and maintain a real-world infrastructure to:

Test attack scenarios and
- Monitor the result of the attack first-hand
- Review the generated logs to understand what happened on the backend, what could be detected and created as a detection rule
Implement security standards like ISO27001, NIS2, NIST, and CIS
Architect a secure infrastructure: This will be a never ending goal, as, even in a relatively simple infrastructure like the one which will be built under this project, there always is something to improve in terms of security architecture.
Recover seamlessly: All the infrastructure must be deployed automatically. The end goal is seamless recovery: If it is ever needed to redeploy the infrastructure on a new server, it can be done with minimal manual effort. This is where the project leans heavily into IaC, as it will be explained more in the next sections.

This project covers a lot of concepts about IaC, security hardening, security standards, blue teaming, and red teaming. Hopefully the end result will allow other security professionals to deploy this infrastructure and easily get to security testing.

In this first post, a high level design of the end result is described, of the tech stack that is planned to be used, as well as an early PoC regarding the infrastructure and its deployment using IaC.

So let’s get started!

Introduction
Contents
Automated Infrastructure
Tech stack
Architecture
Initial Configuration / PoC
Next Steps
Conclusion

Automated Infrastructure

As mentioned above, one core aspect of this project is to be able to have the whole infrastructure deployed automatically. This means that the following practices will be utilized.

Infrastructure as Code (IaC): All the infrastructure will be managed and provisioned using code and configuration files, instead of manual processes. This enables automation, consistency, version control, and swift, repeatable deployments. In the case specifically of this Cyber Range project, this will also allow quick recovery in case of an attack resulting in part of the infrastructure or the whole infrastructure being compromised and damaged unrecoverably.
For security hardening and implementing security standards, in addition to IaC, the following practices will also be utilized:
- Policy as Code (PaC): PaC (ref1, ref2 , ref3) is the practice of defining, updating, sharing and enforcing policies using code. With this approach, when compared to manual processes to manage policies, the benefits are:
  - Sandboxing: Policies provide guardrails for other automated systems. By defining Policies as Code, the verification by the policies is automated, reducing the time needed of manual work.
  - Codification: Because the policy is defined as code, it is possible to describe the logic about the policy directly on the code with comments, which results in better understand and knowledge sharing of these policies.
  - Version Control: The benefits of version control are well known (history, diffs, pull requests, etc.).
  - Testing: Because the policies are defined as code, they can be tested by utilizing automated testing such as through a CI/CD pipeline. This allows for testing if a policy will result in the expected outcome before deploying to production.
  - Automation: Similarly to IaC, with PaC, tools can be used to automatically deploy the policies to specified systems.
  - Compliance as Code (CaC) / Security as Code (SaC): This is where the concepts of CaC and SaC are also related.
  - Some examples of what rules can be enforced as policy with PaC are the following:
    - All ‘victim’ VMs must be on the isolated VLAN, not the management VLAN.
    - No single VM can be assigned more than 4 CPU cores or 8GB RAM.
    - All VMs must clone from the template X.
    - Terraform must not use the default OS users to perform actions.
    - Specific fields in Terraform must not be left empty.
    - Every VM must have a description field explaining its purpose. PaC is something that will be implemented at a later phase of the project, after the IaC phase is at a mature state.
Detection as Code (DaC): DaC enables the writing, maintenance, and automation of the threat detection logic as if it were software code, making security a built-in part of the development pipeline. Similar to PaC, DaC will start being implemented at a later phase of the project.

Tech stack

For this project, a wide variety of systems and technologies will be used, as the purpose is, among others, to simulate a real-world IT infrastructure.

The following technologies, for now, are strong contenders to be deployed. Since the project is just starting now, some technologies are only thought of as ideas, and may or may not actually be deployed when the time comes. I am looking for recommendations, though. Send me a message if you have something to recommend:

Proxmox: The whole infrastructure will be deployed on a physical server running Proxmox Virtual Environment (PVE) as a Type-1 Hypervisor. The infrastructure will be built with VMs and Containers.
For the IaC aspect, the following 3 tools will be used:
- Packer: Packer is a community tool for creating identical machine images for multiple platforms from a single source configuration. Essential, Packer:
  1. Takes an ISO file of an OS
  2. Installs it on a temporary VM
  3. Applies any defined action during the installation (language, locale, hard disk to install the OS, etc.)
  4. Applies any configuration defined (e.g., IP address)
  5. Installs any package defined and
  6. Converts that VM to a template. That template is made for the platform on which the temporary VM was created on. In the case of this project, since the VM is created in Proxmox, the template will be a Proxmox VM template. This template is then used by Terraform to provision VMs which will have that configuration ready immediately. Packer provides the freedom of utilizing any ISO and converting it to a template exactly for the needs of the task at hand.
- Terraform: HashiCorp Terraform is an infrastructure as code tool that enables the definition of both cloud and on-prem resources in human-readable configuration files that can be versioned, reused, and shared. Terraform can manage low-level components like compute, storage, and networking resources, as well as high-level components like DNS entries and SaaS features. In the context of the project, the templates created by Packer will used by Terraform to provision all the infrastructure defined. Note here that many of the configurations of the template defined by Packer can be changed during the provisioning by Terraform (e.g., RAM, CPU cores, installed packages, etc.), which provides more freedom during provisioning. But with Packer the benefit with the resulting template is that there is no need to install the packages already installed and execute any time consuming action every time a new VM is provisioned, because it was already done during the Packer VM template creation.
- Ansible: Ansible is an automation language which allows for automating essentially any IT task. For the project, Ansible is used for further configuring all the VMs provisioned by Terraform, like installing additional packages, starting services, connections, etc. Ansible uses SSH or WinRM to execute remote commands on machines. One of the benefits of Ansible is what is called Idempotency. This means that Ansible only makes changes if necessary, preventing unintended side effects. An Ansible playbook is written in YAML, and in a playbook the final desired state of the target machine is described. Then it is up to Ansible to make any changes or no changes depending on whether the target machine already has the final desired state.
OPNsense: OPNsense is an open-source next-gen grade firewall and routing platform, which brings all the features provided by commercial products to the open-source world. It is the most widely used open-source firewall, provides all the features necessary for this project, and will be used as the central firewall, controlling the traffic among the different subnets and machines.
- Suricata: Suricata is an open-source network analysis and threat detection software. Suricata will be the network-based IDS/IPS solution of this project. OPNsense provides an integration with Suricata, which will allow for an experience of a next-gen firewall in just a few steps.
SIEM/XDR: When it comes to the XDR solution to be deployed for this project, a few alternatives are taken into consideration:
- Wazuh: Wazuh is an open-source security platform, which brings a lot of features under the XDR and SIEM umbrella. This includes:
  - Configuration Assessment
  - Malware Detection
  - File Integrity Monitoring
  - Threat Hunting
  - Log Data Analysis
  - Vulnerability Detection
  - Incident Response
  - Regulatory Compliance
  - IT Hygiene
  - Containers Security
  - Posture Management
  - Workload Protection
- Microsoft Defender XDR: One of the most widely used XDR solutions in the market. It is a full commercial XDR solution, and it would require multiple posts to describe it fully. While this is not a free solution, hopefully I will figure out how to implement it in a lab environment because I would be really interested in including Microsoft Defender XDR in this Cyber Range project.
- Security Onion: Security Onion is a free platform providing a series of features, including:
  - Network visibility using Suricata
  - Intrusion detection honeypots based on OpenCanary
  - Log management with the Elastic Stack
  - File extraction with Zeek or Suricata
  - Full packet capture with Stenographer
  - File analysis with Strelka
  - Host visibility with Elastic agent
  - Centralized management with Elastic Fleet
ModSecurity + OWASP Core Rule Set (CRS): ModSecurity is an open source cross-platform Web Application Firewall (WAF). Since its version 3 release, it now works as a standalone module which provides the capability to load/interpret rules written in the ModSecurity SecRules format and apply them to HTTP content provided by the web application via ModSecurity Connectors. This will prove useful, since in its previous module it worked as an Apache module only, while now it is more independent of the web server solution that it protects. ModSecurity on its own does not provide detection/protection rules. The OWASP CRS is a set of generic attack detection rules to be used with ModSecurity.
The Actual Infrastructure that will be secured and attacked will include:
- Linux servers:
  - DVWA: Damn Vulnerable Web Application (DVWA) is a PHP/MariaDB web application which can be configured to be vulnerable against different types of web-based attacks. This will be used for simulating attacks, as well as attempting to protect the application with different measures even though it is vulnerable.
  - OWASP Juice Shop: Similar to DVWA, it is another vulnerable web application.
  - OWASP WebGoat: Another Java-based vulnerable application.
- Windows: In order to get close to a real-world infrastructure, different Windows components will also need to be configured:
  - Active Directory Domain Services
  - Windows Workstations
  - Windows Servers with different configured roles
For the Red Team side, a few potential and very useful candidates are the following:
- Atomic Red Team: It is an open source library of tests designed to test the applied security controls.
- MITRE Caldera: Caldera is a much more complex framework than Atomic Red Team. It is a cybersecurity framework which can enable the following:
  - Autonomous Adversary Emulation: It is possible to build a specific threat profile and launch it in a network to see vulnerable points.
  - Test and Evaluation of Detection, Analytic and Response Platforms: It provides automated testing of cyber defense measures.
  - Manual Red-Team Engagements: It augments existing offensive toolsets.
- PurpleSharp: It is an adversary simulation tool focused on Windows Active Directory Environments. It currently supports 47 MITRE ATT&CK techniques.
- Injection Monkey: Maintained by Akamai (originally by GuardiCore), it is an adversary emulation platform, where the basic idea is that a “worm” is dropped on a machine, and it tries to spread to every other machine in the network using common exploits and weak passwords.
Policy as Code: In the later stages of the project, after the IaC part is at a mature state, the next step will be to proceed with the Policy as Code. For now, the potential candidates for PaC are the following:
- Open Policy Agent: OPA is a general-purpose policy engine that unifies policy enforcement across the stack. It uses a high-level declarative language that allows the specification of policies for a wide range of use cases. In the case of the project, it will mostly be used for security hardening and compliance aspects.
- HashiCorp Sentinel: Sentinel is a Policy as Code framework for HashiCorp products, defining what is allowed and what is prohibited. In the context of the project, Sentinel could be used for Packer and Terraform, but cannot be extended for Ansible playbooks.
Detection as Code: Similarly to PaC, DaC is something that will be implemented later on. A few candidates identified are the following:
- Sigma Rules: Sigma rules are YAML files that contain all the information required to detect specified malicious behaviour when inspecting log files.
- YARA: YARA is a tool that assists in defining malware samples. A YARA rule includes meta-information about the malware, and a set of strings and conditions to detect the malware (signatures).

Architecture

As shown above, the architecture is a relatively simple and typical network infrastructure, with the following components:

The central Firewall, controlling the network traffic among the four network zones:
1. Demilitarized Zone (DMZ): The zone that exposes services which are to be served to the Internet. In the context of the project, these will be served to the local network, and, most importantly for the project, will be accessible by the “External Attacker”, enabling for attack scenarios initiated from the “Internet”.
2. Internal Zone: The zone where all there internal servers and services will reside. This includes:
  - Any internal servers, e.g., SQL Servers and AD DC, hosting and serving information which is destined to be consumed by internal resources only.
  - Security Tools, including the SIEM/XDR/Monitoring tools.
3. End Users: The last zone will be for the End Users, where typical workstation VMs will reside, and have defined access to specific servers/services and to the internet.
4. WAN Zone: This is where the “Internet”, in the context of the Infrastructure, lives. This is where:
  - The PC from which the Proxmox management will be done, running the Packer, Terraform, and Ansible.
  - The External Attacks will occur from.
There will also be Internal Attack Simulations, executed from within the different zones, bypassing the firewall (“assume breach”).

Initial Configuration / PoC

In this section, an initial configuration of the infrastructure is described, with only the bare minimum of components, as well as a PoC of the infrastructure working as described in the previous sections, i.e., with IaC practices.

All the code used in this PoC can be found under the project’s repository kostas-seclab.

Proxmox Configuration

The installation of Proxmox is a typical installation of any OS. Because the physical machine is connected to the home router, the following network configuration was added:

IP Address: Static 192.168.0.50/24 - Outside of the DHCP range of the home router, but in the same subnet.
Gateway: 192.168.0.1 - The IP Address of the home router.
DNS: 1.1.1.1 - Used Cloudflare’s general use DNS IP address.

Network Configuration

Proxmox VE is using the Linux network stack. A Linux bridge interface (usually named vmbrX) is needed to connect guests to the underlying physical network. It can be thought of as a virtual switch which the guests and physical interfaces are connected to.

The network configuration for the project utilized Linux Bridges to create the defined zones.

In the screenshot above, the utilized objects are the following:

Name	Type	Description
nic1	Network Device	This is the physical interface which physically connects from the Proxmox VE server to the home router, enabling connectivity to the home PC and to the internet.
vmbr0	Linux Bridge	It is through this Linux Bridge that the Proxmox VE server actually connects to the home router, as depicted in the `Ports/Slaves` column, where the `nic1` is defined. The default gateway for the Proxmox server is also defined through this bridge.
vmbrDMZ20	Linux Bridge	The Linux Bridge for the DMZ. The “20” in the name is there for convenience of knowing which subnet the zone has (10.0.20.0/24). The IP of the bridge is 10.0.20.2/24, because the .1 will be assigned to the OPNsense interface.
vmbrEUZ40	Linux Bridge	The Linux Bridge for the End Users zone. The “40” in the name is there for convenience of knowing which subnet the zone has (10.0.40.0/24). The IP of the bridge is 10.0.40.2/24, because the .1 will be assigned to the OPNsense interface.
vmbrIZ30	Linux Bridge	The Linux Bridge for the Internal Zone. The “30” in the name is there for convenience of knowing which subnet the zone has (10.0.30.0/24). The IP of the bridge is 10.0.30.2/24, because the .1 will be assigned to the OPNsense interface.

Proxmox Firewall

In addition to the OPNsense which will operate as the central Firewall of the infrastructure, the firewall of Proxmox was also enabled, with the purpose of isolating the Infrastructure from being able to access the home router local network. Think of it as a “Defence in Depth” approach.

Before explaining how firewalling works in Proxmox, let’s briefly review how the VMs and nodes are structured in Proxmox:

In the above screenshot this structure is depicted. More specifically, there is:

Datacenter: This is the top level of abstraction. Under datacenter all the proxmox servers are listed. If a Proxmox cluster was created, there would be more than one, but since in the project only one physical server is used, there is only one entry.
Proxmox VE Nodes (Host): The “kkproxmox” node is the physical server running the Proxmox VE.
VMs under each Proxmox VE Node: Under each Proxmox VE node, the different resources deployed are listed. This includes VMs, VM templates, storage, network, etc.

The Proxmox VE firewall groups the network into the above logical zones (there is also a zone related to SDN which is out of scope for this PoC). The firewall as a functionality can be enabled on any of the zones described above. For example, if the firewall is needed to be enabled on a VM, then the firewall needs to be enabled on the Datacenter, the Proxmox node hosting that VM, the VM itself, and on each virtual network interface of that VM. Also, when a firewall rule is created on one of the levels described above, it applies to all the levels under it. For example, if a rule is create on the Proxmox VE Node, then it applies to all the VMs under that Node.

In Proxmox, firewall rules can be defined for different directions:

In: Traffic that is arriving in a zone
Out: Traffic that is leaving a zone
Forward: Traffic that is passing through a zone. In the host zone this can be routed traffic (when the host is acting as a gateway or performing NAT). At a VNet-level this affects all traffic that is passing by a VNet, including traffic from/to bridged network interfaces.

There are default rules applied when enabling firewall. For In traffic the default rule is Deny and for Out traffic, the default rule is Allow.

For the project, the following configuration is applied:

Firstly, on the Datacenter zone, Aliases were created for the private subnet IP ranges:

Two Security Groups were then created, which contain groups of firewall rules, which can be enforced on any zone afterwards, for easier application to multiple zones.

The first Security Group is the following, and contains rules to allow access to the management IP of OPNsense (the creation of which is decribed on a later section), and to reject traffic to the local subnet of the home network. This is to isolate the infrastructure from being able to reach out to the local subnet directly. Access to the internet is still allowed. Rule No. 0 was initially created to allow access to the Proxmox management IP, but it was later found that Proxmox includes Default firewall rules that already permit this traffic, so the rule is now disabled.

The second Security Group is the following, and contains a rule to allow traffic between the local subnets of the infrastructure. This traffic will be controlled further by the OPNsense.

After creating the two Security Groups, they were enforced on every zone:

On the Proxmox host only the first one is needed, because traffic towards the infrastructure local subnets does not enter or leave the host directly:

On the OPNsense, the configuration of which will be shown in a later section, both rules were added.

Manual OPNsense Installation

Before moving to full IaC mode, as a PoC, OPNsense was installed manually on Proxmox.

The first step is to download the ISO from the download page. Afterwards, the ISO is needed to be uploaded to Proxmox:

VM Creation

When creating a VM in Proxmox, there are many options to select related to the OS, the system architecture, Disks architecture, etc. The configuration applied to OPNsense VM is shown in the following screenshots. Note that most selection were the default ones, because there is no significant difference to matter in the context of the project.

General Settings

OS Settings: This is where the OPNsense ISO is selected, and also the guest OS type.

System Settings

Disks Settings

CPU Settings

Memory Settings: A useful setting which will be experimented with when deploying more VMs is the “Ballooning Device” setting along with the Minimum memory. Memory ballooning allows you to have your guest dynamically change its memory usage by evicting unused memory during run time. It reduces the impact your guest can have on memory usage of your host by giving up unused memory back to the host. The Proxmox VE host can loan ballooned memory to a busy VM. The VM decides which processes or cache pages to swap out to free up memory for the balloon. The VM (Windows or Linux) knows best which memory regions it can give up without impacting performance of the VM. The Minimum memory setting defines the minimum memory in MiB which will never be freed up for use by other VMs. For now the Minimum memory is set to the actual memory size, so essentially there is no memory ballooning for the VM.

Network Settings: As mentioned before, the WAN interface of the OPNsense will be connected to the same Linux Bridge as the Proxmox host and the home router. After creating the VM, more network interfaces will be created.

VM Installation

Installation with ZFS. After reading about ZFS vs UFS, it seems the ZFS handles unexpected power loss better while UFS may lead to data corruption, but it has a slightly higher RAM usage.

ZFS Configuration: We have only 1 disk for now, so stripe it is:

Select the virtual disk created in the VM creation steps:

Changed the root password and completed the install:

After installation, through the CLI, the interface and IP address assignments were completed:

Select option 1

The interface is recommended to be assigned automatically as it is the only interface at the moment.

After the interface assignment is completed, the IP address assignment is next (select option 2).

In order for Proxmox to be able to exchange information between the host and guest, and to execute commands in the guest, the Qemu guest agent needs to be installed.

Then the service needs to be enabled:

OPNsense network interfaces

As shown in the network diagram, 3 more network interfaces were created on OPNsense, and connected to the corresponding Linux Bridges:

Then, each interface was assigned and enabled in OPNsense as well:

Each interface was assigned its corresponding IP address:

OPNsense Firewall rules

For firewall rules, since this is just a PoC, only 2 rules were created:

A rule to allow access to the Web GUI at the management IP of OPNsense.

A rule to allow machine in the End User Zone to access anything.

OPNsense DHCP

Additionally, DHCP was configured for the End User Zone:

Packer Configuration

In order for Packer to work, the following was implemented:

Prepare Proxmox for Packer

Create “Packer” Role and User on Proxmox:

For the Role:

Go to Datacenter -> Permissions -> Roles.
Click Create. Name it PackerRole.
Add these privileges: VM.Allocate, VM.Config.HWType, VM.Config.CPU, VM.Config.Memory, VM.Config.Network, VM.Config.Disk, VM.Monitor, VM.Audit, VM.PowerMgmt, Datastore.AllocateSpace, Datastore.Audit, VM.Config.Options, SDN.Use VM.Backup+Clone+Console, VM.Config.CDROM, VM.Config.CloudInit, and VM.GuestAgent.Audit

For the User:

Go to Datacenter -> Permissions -> Users.
Click Add. User name: packer, Realm: pve (more info at Proxmox VE authentication - Authentication Realms).

Assign permissions to the User:

Go to Datacenter -> Permissions.
Click Add -> User Permission.
Path: / (This gives permission for the whole datacenter).
User: packer@pve.
Role: PackerRole.

Generate API Token

This is the “Password” Packer will use to talk to the Proxmox API.

Go to Datacenter -> Permissions -> API Tokens.
Click Add.
Select User: packer@pve.
Token ID: packer-token.
Uncheck “Privilege Separation” (this ensures the token has the same rights as the user packer).
Click Add. CRITICAL: Proxmox will show you the Token Secret only at the time of its creation. If it is missed at that time, a new token must be created.

Preparing packer

To install Packer, the official method resulted in some errors, and the method describe in this stackoverflow post worked properly, which includes running the following commands and will be used to install Terraform, too.

# Source - https://stackoverflow.com/a
# Posted by Thilina Ashen Gamage
# Retrieved 2026-01-28, License - CC BY-SA 4.0

# GPG is required for the package signing key
sudo apt install gpg

# Download the signing key to a new keyring
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg

# Verify the key's fingerprint
gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint

# The fingerprint must match 798A EC65 4E5C 1542 8C8E 42EE AA16 FCBC A621 E701, which can also be verified at https://www.hashicorp.com/security under "Linux Package Checksum Verification".

# Add the HashiCorp repo
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

# apt update successfully
sudo apt update

sudo apt install packer
sudo apt install terraform

However, because I am currently running bash via Windows Subsystem for Linux (WSL), it is difficult to grab the IP address of the physical PC, which makes packer fail when opening the temporary web server to give the user-data file. Therefore, for this iteration we are running Packer on Windows, and for the next phase we will probably add the user-data file to Proxmox directly for more seamless implementation.

To install Packer for windows, you just download it from here, unzip it, and add the path of the exe to the Path environment variable of the PC:

To organize the Packer directory, a packer path was created, under which all the other files and directories were created.

For the PoC, 2 Ubuntu VMs will be created. One will be under the End user Zone and the other under the WAN Zone. Because Packer creates ready-to-provision images, an image is to be created for every OS planned to be provisioned with Terraform. Therefore, under the packer path, a sub-path was created named ubuntu-2404 for this purpose.

All the created files are found under the project’s repository kostas-seclab

Create Credentials

The following packer/credentials.pkrvars.hcl file was created, which contains credentials to be used in Packer, and later on in Terraform as well.

Create user-data file

A YAML file packer/ubuntu-2404/http/user-data was created. When running Packer, the PC becomes a temporary web server, and the machine pulls the user-data file from the PC from the http path.

To better understand how the user-data and the next files are involved during the image building using Packer, this is a great point where the tool called Cloud-init should be briefly described. Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialization. It is supported across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installations.

When a VM boots, cloud-init runs and provides the necessary glue between launching a cloud instance and connecting to it so that it works as expected. It looks for metadata provided by proxmox or terraform and performs different initialization tasks automatically, including:

Network: Sets static IP or DHCP config
Identity: changes the hostname from ubuntu-template to what is defined
Security: injects public SSH keys to log in without password
Growth: expands the disk partition to fill the size you gave it in proxmox.

Cloud-init grabs the ubuntu.pkr.hcl which is described next for instructions for proxmox and the user-data file for instructions for the guest (the OS).

But first, the user-data file is shown below:

#cloud-config
autoinstall:
  version: 1
  identity:
    hostname: ubuntu-template
    username: lab-admin
    password: "$6$wFmQrqy8bMHGTQ.O$1WWGjLd3buuOov83OY7zJbdw5Z9Gx4C3ueH04GZGHzqz6h7Jy0TelUUOisEt/1GJwQifYKYKVfj17vkd0mk0f0"
  user-data:
    disable_root: false
  locale: en_US.UTF-8
  timezone: UTC
  keyboard:
    layout: us
  ssh:
    install-server: true #install openssh
    allow-pw: true #allow password authentication in ssh
  packages:
    - qemu-guest-agent #qemu guest agent so that proxmox retrieves info like IP address.
    - cloud-init #needed so the VM is configurable via Terraform.
  storage:
    layout:
      name: direct #use the entire virtual disk as one big partition.

In the user-data file, the autoinstall directive is the set of instructions for the Ubuntu Subiquity installer. It provides all the information required during the initial installation of the ubuntu server, including:

The hostname
The admin user
Local
Timezone
Keyboard layout
SSH settings
Packages to be installed
Storage settings

Let’s look now at the ubuntu.pkr.hcl file:

packer {
  required_plugins {
    name = {
      version = "~> 1"
      source  = "github.com/hashicorp/proxmox"
    }
  }
}

# Declare variables, we will pull them later in the packer build command
variable "proxmox_api_url" { type = string }
variable "proxmox_api_token_id" { type = string }
variable "proxmox_api_token_secret" {
  type      = string
  sensitive = true
}
variable "ubuntu_pw" {
  type = string
  sensitive = true
}

source "proxmox-iso" "ubuntu-server" { #Resource type and local name
  proxmox_url = var.proxmox_api_url
  username    = var.proxmox_api_token_id
  token       = var.proxmox_api_token_secret

  # Skip TLS Verification for self-signed certificates
  insecure_skip_tls_verify = true
  # qemu_agent = true # Default is true anyway

  node    = "kkproxmox"
  vm_id   = 1000
  vm_name = "ubuntu-2404-template"

  # iso_file = "local:iso/ubuntu-24.04.3-live-server-amd64.iso"

  boot_iso {
    # type = "scsi"
    type = "ide"
    iso_file = "local:iso/ubuntu-24.04.3-live-server-amd64.iso"
    iso_checksum = "sha256:c3514bf0056180d09376462a7a1b4f213c1d6e8ea67fae5c25099c6fd3d8274b"
    unmount = true
  }

  cores = 4
  memory = 4096

  network_adapters {
    model  = "virtio"
    bridge = "vmbr0" # Will probably change it in the Terraform script, this is only for packer.
  }

  disks {
    disk_size    = "20G"
    storage_pool = "local-lvm"
    type         = "scsi"
    ssd          = true
  }

  cloud_init = true # add an empty Cloud-Init CDROM driver after the VM has been converted to a template.
  cloud_init_storage_pool = "local-lvm" # Name of the Proxmox storage pool to store the Cloud-Init CDROM on.

  boot_command = [
    "", "e",
    "",
    " autoinstall cloud-config-url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/user-data ds='nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/'",
    ""
  ]

  http_directory = "http"
  ssh_username   = "lab-admin"
  ssh_password = "${var.ubuntu_pw}"
  ssh_timeout    = "20m"
}

build {
  sources = ["source.proxmox-iso.ubuntu-server"]

  provisioner "shell" {
    # execute_command = "echo ${var.ubuntu_pw}| sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
    execute_command = "echo ${var.ubuntu_pw}| {{.Vars}} sudo -S -E sh -eux '{{.Path}}'"
    inline = [
      "echo 'Waiting for cloud-init to complete...'",
      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Still waiting...'; sleep 2; done",
      "echo 'Cloud-init completed successfully'",
      "echo 'Cleaning up...'",
      "rm -rf /var/lib/apt/lists/*",
      "rm -rf /tmp/*",
      "rm -rf /var/tmp/*",
      "cloud-init clean --logs --machine-id --seed --configs all"
    ]
  }
}

This file is called the Packer Template, and include all the information for how to build the template. It includes:

The required packer plugins for this template. In this case, the Proxmox plugin is required only, to define the rest of the information and know how to communicate with the Proxmox API.
Variable declaration to be used in the packer build in the next lines of the file. Note here that at this point only the declaration of the variables is done. The actual value of the variables will be provided via the credentials.pkrvars.hcl file during the packer build execution.
The source block is the core logic. It defines the different values that the VM should have, like CPU cores, memory size, ssh settings, the ISO file to use, the VM name, where to install the VM in proxmox, etc. The values that are not explicitly defined here have default values set provided by the proxmox plugin.
- boot_command: A very valuable part of the source block is the boot_command. In it, the actual keyboard presses for the installation are defined. With these boot commands, what is done is that the installation file is edited to pull the installation parameters from the user-data file described above. More specifically, it contains Linux Kernel Boot Parameters before the installation starts:
  - autoinstall: This tells the Ubuntu installer (Subiquity) to run in automated mode, instead of going through the manual installation process of selecting language etc.
  - The variables {{ .HTTPIP }} and {{ .HTTPPort }}: these are the IP of the PC and the port that temporarily serves the user-data file.
  - cloud-config-url: This parameter tells the installer where the file of user-data is.
  - ds: This stands for “Data Source” and Cloud-init uses its contents:
    - nocloud-net: This tells Cloud-init that the data source is not Public cloud (Azure/AWS/GCP etc.) and to look at the local network for the file.
    - s=: This stand for “seed from”. It tells Cloud-init where to find the user-data and meta-data (another file not used for this PoC) files.
The build block is the execution part of packer. The source block defines the VM, while the build block actually runs it. Here is where the Provisioners live, through which it is possible to execute actions on the machine image and configure it after booting. They can be used to install packages, patch the kernel, create users, download application code, etc. In the PoC, the shell provisioner is used to clean up the machine and reinitialize cloud-init so that it reruns during the Terraform provisioning of the VM (cloud-init runs only once on the machine unless its configuration is cleaned).

Packer pitfalls, solutions, and lessons learnt

Some tools that helped in troubleshooting Packer:

Set the envar $env:PACKER_LOG=1 to make running packer verbose. This shows many hints as to what is going wrong.
If the VM does not proceed with the autoinstaller, and reaches the manual installation flow, you can use Ctrl + Alt + F2 in the Proxmox console to jump to a root shell. This allows to
review logs of the subiquity and cloud-init. Two useful paths for checking are /var/log/installer/subiquity-server-debug.log and /var/log/syslog.
test network connections, to confirm, for example, that you can reach the temporary packer server on the PC running packer, or that the internet is reachable.
Check the hardware resource usage of the VM being installed in Proxmox. Maybe there is overuse of the resource and may need to be increased.

Some pitfalls and lessons learnt I met while setting up packer were the following:

Proxmox 500 Error: After enabling verbose mode in packer (by setting $env:PACKER_LOG=1), a repeated message of 500 “Qemu must be running to read the IP address of the machine” kept popping up. I thought this meant that something was wrong, but actually this is expected and Packer is actually waiting for Qemu guest agent to be installed so that it can read the VM’s IP Address from proxmox so it can SSH to it. If it never moves that point, then the QEMU guest agent was not installed, or, most probably, the autoinstaller did not start.
The “Interactive Menu” problem:
- The issue: The VM kept reaching the manual language selection screen instead of starting the automated install.
- The cause: The installer could not find or parse the autoinstall instructions.
- The resolution: In the boot_command, an explicit definition of the autoinstall instructions needed to be defined using the cloud-config-url parameter.
YAML formatting
- The issue: The installer found the user-data file but ignored the settings, falling back to manual installation.
- The cause: Ubuntu ignores the user-data file if it does not have the “#cloud-config” comment at the first line.
- The lesson: read the documentation of the requirements of different configuration files.
Validation crashes:
- The issue: The installer crashed with an “Unknown keyboard layout ‘en’” error (1st screenshot), and crashed while attempting to run timedatectl list-timezones (2nd screenshot).
- The cause and resolution: Some parameters in the user-data file are mandatory and do not have default values when not explicitly defined. These include the timezone and keyboard layout parameters, which were not initally defined. Once added to the user-data file, the crashes were fixed. Keyboard layout error: Timezone error:
Hardware resource constraints
- The issue: The installation appeared to hang or move extremely slowly without showing errors.
- The cause: Initially the default memory and CPU values were left in the packer template file, which means 1 CPU core and 512MB memory. Especially for the memory size, this is a very low value for an installation of Ubuntu server.
- The solution and lesson: Modern installers require more thant the default minimal resources to complete. Increase the values for 4 CPU cores and 4096MB memory resolved the issue.

SSH Handshake and authentication

The issue: The OS installed successfully, but Packer failed to connect with the following repeating error:

2026/01/26 10:59:47 packer-plugin-proxmox_v1.2.3_x5.0_windows_amd64.exe plugin: 2026/01/26 10:59:47 [INFO] Attempting SSH connection to 192.168.0.110:22...
2026/01/26 10:59:47 packer-plugin-proxmox_v1.2.3_x5.0_windows_amd64.exe plugin: 2026/01/26 10:59:47 [DEBUG] reconnecting to 
TCP connection for SSH
2026/01/26 10:59:47 packer-plugin-proxmox_v1.2.3_x5.0_windows_amd64.exe plugin: 2026/01/26 10:59:47 [DEBUG] handshaking with SSH
2026/01/26 10:59:47 packer-plugin-proxmox_v1.2.3_x5.0_windows_amd64.exe plugin: 2026/01/26 10:59:47 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
2026/01/26 10:59:47 packer-plugin-proxmox_v1.2.3_x5.0_windows_amd64.exe plugin: 2026/01/26 10:59:47 [DEBUG] Detected authentication error. Increasing handshake attempts.

The cause and resolution: the ssh_password value in the packer template is used for this connection, and was not initially defined. Upon its definition, the issue was resolved.

sudo asking password on shell provisioner in packer
- The issue: When running the “shell” provisioner in Packer build, it asked for a manual input of the user password to execute the sudo commands.
- The cause: the “shell” provisioner has a variable called execute_command. What shell provisioner does essential is that it converts all the commands provided to a script file, and then executes that script with a predetermined execute_command. This can be edited to satisfy any need.
- The solution: The execute_command value was changed to "echo ${var.ubuntu_pw}| {{.Vars}} sudo -S -E sh -eux '{{.Path}}'", which pulls the password and inputs it for when sudo asks for it.
Cloud-init status –wait not working in shell provisioner
- The issue: In the shell provisioner, when trying to clean up the cloud-init state, at first it was tried to run the “cloud-init status –wait” command, to wait for cloud-init to complete before reinitializing. This command would return status code 2 which means that the cloud-init process has not finished, while shell provisioner only accepts status code 0 as a “non-error” status code, so Packer would crash and mention this as an error.
- The Solution: The /var/lib/cloud/instance/boot-finished file is now being checked instead to verify that cloud-init finished running.

Packer build and PoC result

To run packer, the following commands are used, executed at the path of packer/ubuntu-2404/:

packer init .
packer validate -var-file="../credentials.pkrvars.hcl" . # To Validate the packer configuration before building
packer build -var-file="../credentials.pkrvars.hcl" . # To Build the template

Running the packer validate command results in the following:

Running the packer build command results in the following:

While the VM is being installed, in the Console window it is possible to review what is being run:

After successfully running Packer, the result is a Proxmox template, ready to be cloned. This template is used by Terraform, as shown in the next section.

Terraform Configuration

In order for Terraform to work, the following was implemented:

Prepare Proxmox for Terraform

Update “Packer” Role in Proxmox for Terraform

For simplicity, the same user used in Packer is used for Terraform. Therefore, the permissions required for Terraform were provided to the previously created PackerRole role. The permissions required are described here, and are the following:

Datastore.AllocateSpace
Datastore.AllocateTemplate
Datastore.Audit
Pool.Allocate
Pool.Audit
Sys.Audit
Sys.Console
Sys.Modify
VM.Allocate
VM.Audit
VM.Clone
VM.Config.CDROM
VM.Config.Cloudinit
VM.Config.CPU
VM.Config.Disk
VM.Config.HWType
VM.Config.Memory
VM.Config.Network
VM.Config.Options
VM.Migrate
VM.PowerMgmt
SDN.Use

Preparing Terraform

To organize the Terraform directory, a terraform path was created, under which all the other files and directories were created.

For the PoC, 2 Ubuntu VMs will be created. One will be under the End user Zone and the other under the WAN Zone.

All the created files are found under the project’s repository kostas-seclab

For terraform, the following files were created:

The terraform/variables.tf file contains variables to be used by terraform:

variable "proxmox_api_url" {
    type = string
    description = "The URL for the Proxmox API (e.g., https://192.168.0.50:8006/api2/json)"
}

variable "proxmox_api_token_id" {
    type = string
    sensitive = true
    description = "The API Token ID (e.g., root@pam!terraform)"
}

variable "proxmox_api_token_secret" {
    type = string
    sensitive = true
    description = "The API Token Secret (UUID)"
}

The terraform/provider.tf file contains information regarding what provider terraform will connect to (AWS, Azure Google Cloud, Proxmox, etc. - In our case it is proxmox, using the Telmate/proxmox provider), as well as the configuration to connect to the provider. The variables for the provider are pulled from the terraform/variables.tf definitions. The execution of terraform apply is where the source and the actual values of the varaibles will be defined.

terraform {
  required_providers {
    proxmox = {
        source = "Telmate/proxmox"
        version = "3.0.2-rc07"
    }
  }
}

provider "proxmox" {
    pm_api_url = var.proxmox_api_url
    pm_api_token_id = var.proxmox_api_token_id
    pm_api_token_secret = var.proxmox_api_token_secret

    pm_tls_insecure = true
}

The terraform/main.tf file contains the blocks of what exactly to build. In our case of building 2 test Ubuntu servers using the template built by Packer, the main.tf looks like this:

resource "proxmox_vm_qemu" "test_server" { # Resource type and resource name
    name = "terraform-vm-01" # Name of the VM
    target_node = "kkproxmox" # Proxmox node to build the VM on

    clone = "ubuntu-2404-template" # Clone the ubuntu template built by Packer

    agent = 1 #enable QEMU guest agent
    
    cpu {
        cores = 2
        sockets = 1
        type = "host"
    }
    memory = 2048
    scsihw = "virtio-scsi-pci"
    # bootdisk = "scsi0"

    # This is added to the default ciuser "ubuntu". "ssh-keygen -t rsa" was used to generate the key pair. This is needed for Ansible to SSH with.
    sshkeys = "${file("~/.ssh/id_rsa.pub")}"

    disk { # The cloudinit disk needs to be defined explicitly for it to be mounted temporarily for cloud-init to run.
        slot    = "ide0"
        type    = "cloudinit"
        storage = "local-lvm"
    }

    disk {
        slot = "scsi0"
        size = "20G"
        type = "disk"
        storage = "local-lvm"
        format = "raw"
    }

    startup_shutdown { # These were defined because these are the values that Proxmox puts on the VMs by default while Terraform tried to put different values, so every time that "terraform apply" would run, it would try to change them, for Proxmox to change them back right after.
        order = -1
        shutdown_timeout = -1
        startup_delay = -1
    }

    network { # This VM connects to the WAN Zone Linux Bridge
        id = 0
        model = "virtio"
        bridge = "vmbr0"
    }

    os_type = "cloud-init" # This is used together with the "cloudinit" disk to apply the cloud-init related configuration (the following commands).
    ipconfig0 = "ip=dhcp" # get IP from the DHCP
    skip_ipv6 = true # do not get IPv6
}

resource "proxmox_vm_qemu" "test_server1" { # the 2nd VM
    name = "terraform-vm-02" # Name of the VM
    target_node = "kkproxmox" # Proxmox node to build the VM on

    clone = "ubuntu-2404-template" # Clone the ubuntu template built by Packer

    agent = 1 #enable QEMU guest agent
    
    cpu {
        cores = 2
        sockets = 1
        type = "host"
    }
    memory = 2048
    scsihw = "virtio-scsi-pci"
    # bootdisk = "scsi0"

    disk { # The cloudinit disk needs to be defined explicitly for it to be mounted temporarily for cloud-init to run.
        slot    = "ide0"
        type    = "cloudinit"
        storage = "local-lvm"
    }

    disk {
        slot = "scsi0"
        size = "20G"
        type = "disk"
        storage = "local-lvm"
        format = "raw"
    }

    startup_shutdown { # These were defined because these are the values that Proxmox puts on the VMs by default while Terraform tried to put different values, so every time that "terraform apply" would run, it would try to change them, for Proxmox to change them back right after.
        order = -1
        shutdown_timeout = -1
        startup_delay = -1
    }

    network { # This VM connects to the End User Zone Linux Bridge
        id = 0
        model = "virtio"
        bridge = "vmbrEUZ40"
    }

    os_type = "cloud-init" # This is used together with the "cloudinit" disk to apply the cloud-init related configuration (the following commands).
    ipconfig0 = "ip=dhcp" # get IP from the DHCP
    skip_ipv6 = true # do not get IPv6
}

Terraform pitfalls, solutions, and lessons learnt

Some pitfalls and lessons learnt I met while setting up Terraform were the following:

Stale Cloud-Init State
- The issue: After deploying a VM with Terraform, Proxmox showed the correct name (e.g., terraform-vm-01), but the internal OS console still showed the template’s original name (ubuntu-server). The VM ignored the new configuration.
- The Cause: cloud-init is designed to run only once. During the Packer build process, cloud-init ran to set up the initial template and left a “marker file” indicating it was finished. When the template was cloned via Terraform, the new VM saw the marker and immediately went back to sleep, ignoring the new instructions from Terraform.
- The Fix: A “cleanup” shell provisioner was added at the end of the Packer configuration (ubuntu.pkr.hcl), which runs the command cloud-init clean --logs --machine-id --seed --configs all to fully reinitialized the cloud-init status.
- The Lesson: Templates must be stateless. If the state is not reset before saving a template, every clone will wake up thinking it has already been configured.
Duplicate Machine IDs
- The issue (Potential): While fixing the previous issue, a risk was identified where multiple VMs might end up with the same IP address from the DHCP server.
- The Cause: Linux generates a unique /etc/machine-id upon installation. If this isn’t reset during the Packer build, every clone shares the same ID, causing network conflicts.
- The Fix: In the cloud-init clean command in Packer, the flag --machine-id was added to reset the machine id as well.
- The Lesson: Simply changing a hostname isn’t enough. Unique system identifiers (GUIDs, Machine IDs) must be regenerated for every new instance to avoid collisions on the network level.
No Cloud-Init Drive
- The issue: Terraform was successfully creating the VM resource, but the OS configuration (hostname, user data) was never applied. When manually mounting a Cloud-Init drive fixed this issue.
- The Cause: The Packer template was missing the cloud_init = true instruction. Consequently, the template was created without a Cloud-Init drive. Without this drive, Terraform had no physical medium to insert the configuration data for the OS to read. Additionally, in the Terraform main.tf file, the “cloudinit” was also needed to be explicitly defined.
- The Fix: The source “proxmox-iso” block in Packer was updated to explicitly include the cloud-init parameter, and Terraform to include the “cloudinit” disk.

Terraform apply and PoC result

To run terraform, the following commands were used:

terraform init # To initialize terraform
terraform plan -var-file=../packer/credentials.pkrvars.hcl # To validate that the terraform config is ready to be applied.
terraform apply -var-file=../packer/credentials.pkrvars.hcl # To apply the Terraform config and build the defined VMs.

The credentials.pkrvars.hcl file was used as the var-file, because it contains the same info required for Terraform to run.

Unlike Packer, Terraform was run in WSL, because the ssh key defined in the Terraform main.tf file will be used by Ansible in the next step, and Ansible cannot be run in Windows, only in Linux.

Running terraform init results in the following:

Running terraform plan -var-file="../packer/credentials.pkrvars.hcl" outputs many lines which show the configuration to be applied. Below is a screenshot of part of the output:

Running the terraform apply -var-file="../packer/credentials.pkrvars.hcl" shows the same information as the terraform plan, which is the configuration to be applied, and asks for confirmation in the end. After typing “yes”, the VMs are being provisioned, and the final output is the following:

After 2 minutes, in Proxmox we can see the provisioned VMs up and runnning:

Now that the VMs are provisioned, the next step is to apply further configurations using Ansible.

Ansible

Ansible cannot be run in Windows PowerShell, so WSL was used.

To install Ansible, use sudo apt install ansible

Ansible uses ssh keys to SSH to machine and apply configurations. To generated the ssh key pair, run ssh-keygen -t rsa:

Then, in the Terraform main.tf file, the public key was added to one of the machines:

sshkeys = "${file("~/.ssh/id_rsa.pub")}"

For Ansible, the following files were created:

ansible/inventory.ini: This file contains the list of machines with their IP addresses for Ansible to connect to them.

[webservers]
192.168.0.104 ansible_user=ubuntu

ansible/install_nginx.yml: This file contains the state for this PoC of the configuration to be applied to the machines of inventory.ini. For the purposes of the PoC, only an Nginx server is installed on one of the two Linux Servers.

---
- name: Configure Webservers
  hosts: all
  become: true #sudo

  tasks:
    - name: Update apt cache
      apt:
        update_cache: yes
        cache_valid_time: 3600

    - name: Install Nginx
      apt:
        name: nginx
        state: present

    - name: Start Nginx service
      service:
        name: nginx
        state: started
        enabled: yes

ansible/ansible.cfg: This file override the global Ansible settings:

[defaults]
inventory = ./inventory.ini
host_key_checking = False

The ansible.cfg file had to be set by setting the environment variable (because we are running with WSL): export ANSIBLE_CONFIG=./ansible.cfg. To confirm, run ansible --version, which should show the config file value:

Running Ansible

To run Ansible with the configuration defined above, the following command was run: ansible-playbook install_nginx.yml

Due to Ansible’s idempotency, running the command again will show that everything is ok and no changes were applied:

After running Ansible, the Nginx can be reached from the browser

Next Steps

So now the flow is complete. A VM template was created using Packer, VMs were provisioned with Terraform, and further configurations were applied using Ansible. This was a PoC to verify the flow can be used for the rest of the project.

The next steps will be to reach a fully recoverable and fully IaC state from start up to the point which has been reached now manually.

This means:

Configuring all the Proxmox settings described in this post using either Ansible or any other automation tool (even a script).
Configuring OPNsense using Packer, Terraform and Ansible. A Packer template will be built using OPNsense’s ISO, Terraform will be used to provision the OPNsense with the settings which were applied manually, and Ansible will be used to apply any configuration to it, including Firewall rules, DHCP, installing Packages, etc.

Conclusion

Building a Cyber Range as Code is going to be a complex project to pull off fully “as Code”. It will require testing and several iterations in order for it work properly.

However, this project will be a great lesson towards:

Learning IaC (Packer, Terraform, Ansible).
Applying security hardening and security standards to multiple machines using IaC practices.
Developing methodologies for security testing and reviewing the results.

This post included the plan for the project. It included different ideas for potential technologies to be used, as well as the architecture to be built. In the next posts the implementation will start.

Hopefully you picked something of interest and will follow along the progress of this project.

See you in the next one.

MDE Performance Troubleshooting - Getting to know MDE Part 3

2026-01-17T00:00:00+00:00

Introduction

A frequently reported issue in MDE is its performance impact on machines. Many engineers have seen the task manager of a machine looking like the following, and are asked to fix it.

Welcome to the 3rd part of the blog post series “Getting to know MDE”!

In this series, each post contains a different component/feature/methodology when it comes to understanding and managing Microsoft Defender for Endpoint (MDE).

In this post, the focus is on troubleshooting issues that have a performance impact on machines caused by Defender processes.

All the information is based on official documentation of Microsoft, which you can find here.

So let’s get started.

Introduction
Contents
Common reasons for higher CPU by MDAV
Troubleshooting Mode
Performance Analyzer
- Run Performance Analyzer via Live Response
MPLog file parsing for performance impact
Conclusion

Common reasons for higher CPU by MDAV

There may be several reasons which cause MDAV to utilize a higher percentage of CPU power. In this section, a few common reasons are briefly described.

Binaries not signed

When a binary (exe, dll, etc.) that is not digitally signed is launched, MDAV will start a Real-Time Protection Scan.

Generally, properly identifying such cases is difficult. If you know a robust method, please let me know. The KQL table DeviceFileCertificateInfo can be used to identify certificate-related issues:

DeviceFileCertificateInfo
| where TimeGenerated > ago(30d)
| where IsTrusted == 0

The following KQL query can also be run to potentially identify cases of unsigned binaries being executed.

DeviceProcessEvents
| where TimeGenerated > ago(1d)
| where SHA1 != ""
| where FolderPath contains ":"
| project TimeGenerated, DeviceName, DeviceId, SHA1, FileName, FolderPath, ProcessCommandLine, InitiatingProcessFileName
| join kind=leftouter (
DeviceFileCertificateInfo
| where TimeGenerated > ago(30d) //putting 30 days because these events are not generated for every execution of a binary.
| project SHA1, IsSigned, IsTrusted, Signer, Issuer, DeviceId
) on SHA1 and DeviceId
| project TimeGenerated, DeviceName, DeviceId, SHA1, FileName, FolderPath, ProcessCommandLine, IsSigned, IsTrusted, Signer, Issuer, InitiatingProcessFileName
| where isnull(IsSigned) or IsTrusted == 0 or IsSigned == 0
| summarize count() by DeviceName, DeviceId, SHA1, FileName, Signer, Issuer, FolderPath, IsSigned, IsTrusted
| sort by FileName asc, DeviceName asc

The results of the detected files should then be filtered to see if they are actually signed, and what Defender mentions about their signature information. This can be done by clicking on the hashes in the resulting table, and checking the signature information:

If the binaries have certificates signed by CAs not trusted by Microsoft or need to be excluded for any reason, they can be added to the Certificates Indicators allow list.

Obfuscated scripts

Obfuscated scripts require much more CPU to be scanned, so obfuscation should be used only if necessary.

Not letting MDAV cache finish before sealing a VDI image

If creating a VDI image for a persistent or non-persistent image, ensure that the cache maintenance completes before sealing the image. To do this:

Open up the Task Scheduler mmc (taskschd.msc).
Expand Task Scheduler Library > Microsoft > Windows > Windows Defender, and then right-click on Windows Defender Cache Maintenance.
Select Run, and let the scheduled task finish.

Misspelled exclusions

Double check that the exclusions are spelled correctly.

Path exclusions only work for scanning flows

Behavior Monitoring and Network Real-time Inspection can still cause performance issues. As a workaround, one of the following can be done:

Add the exe or dll to the Indicators file hash allow list
Add the certificate to the Indicators certificates allow list
Add MDAV exclusions for the process, too.

File hash computation

If the File Hash computation feature is enabled, file hashes for every executable file that is scanned is computed, if it wasn’t previously computed. This has a performance cost especially when copying large files from a network share. Keep in mind that this feature is a prerequisite for File Hash Indicators in Defender.

If it is decided that it is not needed, it can be disabled via PowerShell or any management solution used in the organization (Intune, AD GPO, Configuration Management, etc.). To disable it with PowerShell run the following cmdlet:

Set-MpPreference -EnableFileHashComputation $false

Scheduled scanning

When it comes to scheduled scanning, the following scan settings should be checked, depending on the way they are pushed to machines:

Configure low CPU priority for scheduled scans. This lowers the scheduled scan thread priority from 9 to 8.
Specify CPU usage limit per scan from 50 to 20 or 30.
Start scheduled scan when the device is idle with ScanOnlyIfIdle.
Specify the interval to run quick scans per day to Not configured.
Specify the time for a daily quick scan to a time when the machine is least used.
Consider disabling the Scheduled Scan settings. The Daily Quick Scan is enough:
- Specify the scan type to use for a scheduled scan to Not configured.
- Specify the time of day to run scheduled scan to Not configured.
- Specify the day of the week to Not configured.

More information about how to configure the scheduled scan settings can be found in Microsoft’s documentation.

Scan after security intelligence updates

By default, a scan happens after MDAV receives its latest security intelligence updates. If scheduled scans are enabled, maybe this is not needed and can be disabled.

To disable it in Group Policy (or another management tool, such as MDM), go to Computer Configuration > Administrative Templates > Microsoft Defender Antivirus > Security Intelligence Updates, and set Turn on scan after security intelligence update to Disabled.

Conflicts with other security software

If other security software, like AV, EDR, and DLP, are used, the proper exclusions need to be defined on both MDE and the other software’s side.

More info about using third party security software with MDE can be found in Microsoft Defender Antivirus compatibility with other security products and exclusions that need to be added can be found in Microsoft’s MDE Exclusion List

Troubleshooting Mode

An easy way to investigate if Defender could be a reason for performance issues is to disable Real-Time Protection. If Tamper Protection is on, then Troubleshooting mode can be used on the machine to allow Tamper Protection to be disabled.

After Troubleshooting Mode is turned on, the following cmdlets can be run to disable Real-Time Protection:

Disable Tamper Protection: Set-MPPreference -DisableTamperProtection $true
Confirm that Tamper Protection is disabled: Get-MpPreference | fl DisableTamperProtection - The output should have a value of “True”.
Disable Real-Time Protection: Set-MPPreference -DisableRealtimeMonitoring $true
Confirm that Real-Time Protection is disabled: Get-MpPreference | fl DisableRealtimeMonitoring - The output should have a value of “True”.

Afterwards, a check should be done on the machine again to see if the issue is resolved. If it is, then Defender Antivirus should be investigated further, by following the steps described in the next sections. Otherwise, there could be another reason for the CPU usage of Defender, and probably a ticket to Microsoft should be opened for further investigation.

More info on PowerShell cmdlet for configuring Defender settings can be found here.

More info on Troubleshooting Mode can be found here.

In the following link, additional scenarios where Troubleshooting Mode may help are described Troubleshooting mode scenarios.

Performance Analyzer

Performance Analyzer is a tool that helps in determining which files, file extensions, and processes may be causing performance issues in machines during Defender Antivirus scans. This information can be used as input towards potentially defining Defender exclusions.

Running the Performance Analyzer is straight-forward, by running the following cmdlet as an admin:

New-MpPerformanceRecording -RecordTo

The RecordTo parameter defines the file where the Performance Analyzer recording is written.

The following is an example of the output of running that cmdlet:

The idea is to “turn on” Performance Analyzer, and while it is running and monitoring, the problematic scenario is reproduced. If it is a general issue which cannot be reproduced on demand, but happens sporadically, the only way to analyze that is by leaving Performance Analyzer running and hoping that the problematic behavior reappears.

After capturing the problematic behavior, the recording can be stopped by pressing Enter, which results in the following output:

The next step is to parse the Performance Analyzer’s report. To do this the cmdlet Get-MpPerformanceReport is used. This cmdlet has a lot of different available parameters, which can be viewed at Microsoft’s Performance Analyzer Reference.

Below, a series of cmdlets which can help pinpoint the issue are depicted:

To view the overview of the recording:

Get-MpPerformanceReport -Path .\recording.etl -Overview

To view the top 20 scans, paths, extensions, and processes:

Get-MpPerformanceReport -Path .\recording.etl -TopScans 20 -TopPaths 20 -TopExtensions 20 -TopProcesses 20

The screenshot is cut short because the output is too long:

After running the above command, if a specific category of top values or a specific value per-se is of interest, it is possible to then dive a bit deeper. For example, if a specific extension seems to be causing a lot of scans, to dive deeper the following cmdlet can be run:

Get-MpPerformanceReport -Path .\recording.etl -TopExtensions 20 -TopScansPerExtension 5 -TopPathsPerExtension 5 -TopScansPerPathPerExtension 5 -TopProcessesPerExtension 5 -TopScansPerProcessPerExtension 5 -TopScansPerFilePerExtension 5 -TopFilesPerExtension 5

This cmdlet uses all the parameters that end with “PerExtension”, and will produce a much longer report focused on the top extensions, and for each top extension, its top scans, paths, processes and files. This can help in pinpointing what is causing these scans, which may lead to either resolving the source of the issue or defining exclusions in MDE.

The report can get lengthy. It is possible to make the output machine readable with the -Raw parameter, and then possibly convert it to exportable formats, like JSON, which can then be exported, saved, and analyzed:

Run Performance Analyzer via Live Response

It is also possible to run Performance Analyzer via Live Response, by creating a custom script and uploading it to the Live Response library.

A prerequisite is to allow unsigned scripts to run it, by going to the MDE Portal > System > Settings > Endpoints > Live Response unsigned script execution

The following script will need to be saved in a file. For this post, the file will be named “LivePerfAnalyser.ps1”

param (
    [Parameter(Mandatory=$false)]
    [int]$Seconds = 120
)

$Hostname = $env:COMPUTERNAME
$Datetime = Get-Date -Format "yyyyMMdd_HHmmss"
$OutputPath = "C:\Temp\PerformanceRecording_${Hostname}_${Datetime}.etl"

Write-Host "Initiating recording for $Seconds seconds..."
New-MpPerformanceRecording -Seconds $Seconds -RecordTo $OutputPath

Write-Host "Performance Recording for $Hostname for $Seconds seconds was written at $OutputPath"

The steps to run Performance Analyzer via Live Response are:

Initiate a Live Response session for the endpoint that needs troubleshooting
Upload the script to library by clicking on:
Run the Performance Analyzer with the following command in Live Response: run LivePerfAnalyzer.ps1 -parameters "-Seconds 120". You can change the value of Seconds to the time period in seconds that you need to run the analyzer, or skip the parameters completely, with the default value being 120 seconds.
Try to reproduce the activity that causes performance issues on the endpoint. Wait for the designated time.
After the Performance Analyzer finishes, the following is printed:
Get the file from the path written in the script output: getfile \
After getting the file, the analysis steps described in the previous section can then be followed.

MPLog file parsing for performance impact

The Microsoft Protection Log (MPLog) file is one of a few files found under the path C:\ProgramData\Microsoft\Windows Defender\Support. Under this path, a few files which contain different Defender logs can be found, including:

MPDetection
MPDeviceControl
MPDlpLog
MPLog
MPScanSkip
MpWppCoreTracing
MpWppTracing

These files are very useful when investigating what Defender and its different detection mechanisms did. When it comes to performance troubleshooting, the MPLog file contains a few useful logs, among others, which can assist in detecting processes which experienced high scanning activity, potentially impacting their performance. Credits to the related article by CrowdStrike explaining these logs. Remember that there is always the option of getting these files via Live Response and the getfile command.

These logs have the following format:

2025-12-31T15:45:19.964 ProcessImageName: Razer Synapse Service Process.exe, Pid: 9104, TotalTime: 810, Count: 224, MaxTime: 15, MaxTimeFile: \Device\HarddiskVolume3\Program Files (x86)\Razer\Synapse3\UserProcess\Razer Synapse Service Process.exe.config, EstimatedImpact: 53%

Let’s go through this line:

Field Name	Description	Example
Date (not stated)	Timestamp of event	2025-12-31T15:45:19.964
ProcessImageName	Process Image Name	Razer Synapse Service Process.exe
Pid	Process ID of the Process	9104
TotalTime	The summary of all time periods spent in scans of files accessed by this process (milliseconds)	810
Count	The number of scanned files accessed by this process	224
MaxTime	The longest time spent in scanning a single file accessed by this process	15
MaxTimeFile	The file which was scanned for `MaxTime` milliseconds	\Device\HarddiskVolume3\Program Files (x86)\Razer\Synapse3\UserProcess\Razer Synapse Service Process.exe.config
EstimatedImpact	This value shows the impact that MDAV had on the performance of the above process. It is the percentage of (Total time spent in scans of files accessed by this process)/(Total time which this process experienced scan activity). For example, if you open a large folder in File Explorer, all the files in it are being scanned via Real-Time Protection. The time it takes until the last file is finished being scanned was 10 seconds, but the time that MDAV was actually scanning files was 3 seconds. Here the Estimated Impact is about 3/10 = 30%.	53%

Therefore, using the above logs and the EstimatedImpact value, it is possible to review processes which experience heavy scanning load. Note that this does not mean necessarily that these processes result in higher CPU for the machine, but rather that that specific process experienced a performance hit due to scanning.

The following PowerShell script parses the MPLog file, takes only the logs which include the EstimatedImpact values, and exports it all to a CSV.

$logfile = Read-Host "Enter log MPLog file"

# Extract relevant log entries, remove whitespaces, and process data
$logs = Get-Content $logfile | `
    Select-String "EstimatedImpact" | ` # Select lines that contain the string "EstimatedImpact"
    Select-String -Pattern '%$' | ` # Select lines that end with "%". Sometimes the logs are not recorded properly and 2 logs are written in the same line, which makes it difficult to parse.
    ForEach-Object {
        # Remove all whitespace from the line
        $_ -replace '\s', ''
    } | `
    ForEach-Object {
        # Add a "Date:" prefix to each log entry
        "Date:" + $_
    } | `
    ForEach-Object {
        # Insert a comma after the 30th character for consistent formatting
        $_.Insert(30, ',')
    }

# Convert the cleaned logs into structured objects for CSV export
$logObjects = $logs | ForEach-Object {
    # Split the cleaned line into fields (key-value pairs separated by ':')
    $fields = $_ -split ',' | ForEach-Object {
        $pair = $_ -split ':'
        @{$pair[0] = $pair[1]}
    }

    # Combine all key-value pairs into a single hash table
    $object = @{}
    foreach ($field in $fields) {
        foreach ($key in $field.Keys) {
            $object[$key] = $field[$key]
        }
    }

    # Remove the '%' from the "EstimatedImpact" field if it exists
    if ($object["EstimatedImpact"]) {
        $object["EstimatedImpact"] = $object["EstimatedImpact"] -replace '%', ''
    }

    # Return the final object as a PSCustomObject
    [PSCustomObject]$object
}

# Export the processed data to a CSV file
$outfile = $logfile -replace '\.log$', '.csv'
$logObjects | Export-Csv -Path $outfile -NoTypeInformation -Encoding UTF8

Write-Host "Process Logs exported to $outfile"
Read-Host -Prompt "Press Enter to exit"

Below is an example output of the above script:

After exporting the CSV file, it is useful to sort by TotalTime to find processes that tend to result in a lot of scans. For example, below, after sorting by TotalTime, it can be seen that the Process RazerCentral.exe tends to access a lot of files which are scanned for a long time:

Another option is to sort by EstimatedImpact to view processes which are affected a lot by MDAV scans:

A similar analysis can also be done by inserting a Pivot Table, and adding ProcessImageName in Rows, and TotalTime or EstimatedImpact in Values, on which you can sort again by the Sum of TotalTime or Sum of EstimatedImpact. This will produce a table like the following:

As shown above, the MPLog is yet another way of identifying processes which may impact the performance of the machine or may suffer from MDAV scans themselves.

Conclusion

Sometimes, MDE and its components may utilize a higher-than-normal CPU percentage. Identifying the root cause can prove difficult. Hopefully, with this post it will be a little bit easier.

Stick around for the next posts, where we will dive deep into looking Windows Event Logs, and also may move to other Microsoft Security products like Purview.

I hope to see you back in the next one!

Using KQL to identify detections from MDE - Getting to know MDE Part 2

2026-01-06T00:00:00+00:00

Introduction

Welcome to the 2nd part of the blog post series “Getting to know MDE”!

In this series, each post contains a different component/feature/methodology when it comes to understanding and managing Microsoft Defender for Endpoint (MDE).

In this post, the focus is on KQL and what queries to run to investigate for blocks/detections done by MDE.

In each section, a brief description of a different MDE capability is described, including what to look for when running KQL queries, and a simple KQL query searching for events of that capability.

If you want to get a tl;dr KQL query from this blog post which includes all the detections, check the last section.

All the information is based on official documentation of Microsoft, which you can find here.

So let’s get started.

Introduction
Contents
Conclusion

The DeviceEvents Table

Before starting with each MDE capability, it is important to focus on the Advanced Hunting table which includes all the MDE detection events.

The detections by MDE are reported under the DeviceEvents table in Advanced Hunting.

Two important columns of the DeviceEvents table are the following:

ActionType: The ActionType column shows what action triggered the DeviceEvents event. When it comes to MDE, the ActionType column shows which MDE detection tirggered the event.
AdditionalFields: The AdditionalFields column contains, as its name suggests, additional information regarding the event, information which does not fit any of the other columns. When it comes to MDE detections, the AdditionalFields column contains necessary information about the detection, such as whether something was blocked or audited, or what policy triggered the event. It is in JSON format and, depending on what is being searched for, the AdditionalFields JSON data will need to be parsed in order to retrieve the correct information. In the queries below, there are some examples to get a better idea of how to parse the column.

You can always refer to the DeviceEvents Schema reference directly from the Advanced Hunting page in Defender to review the different available columns and their values:

Attack Surface Reduction Rules Detections

When looking for ASR Rules detections, keep the following in mind:

For each ASR Rule detection, there can be one of three possible ActionType values in the DeviceEvents table, depending on the enabled state of the ASR Rule for the machine:

ActionType	Description
AsrAudited	ASR Rule of was triggered but did not block.
AsrBlocked	ASR Rule of was triggered and blocked.
AsrWarnBypassed	ASR Rule of was triggered in Warn mode, and the user excluded themselves from it.

In the following table the ASR Rules action types are listed and the ASR rule that they correspond to.

ActionType	ASR Rule
AsrAbusedSystemToolAudited AsrAbusedSystemToolBlocked AsrAbusedSystemToolWarnBypassed	Block use of a copied or impersonated system tools
AsrAdobeReaderChiledProcessAudited AsrAdobeReaderChiledProcessBlocked AsrAdobeReaderChiledProcessWarnBypassed	Block Adobe Reader from creating child processes
AsrExecutableEmailContentAudited AsrExecutableEmailContentBlocked AsrExecutableEmailContentWarnBypassed	Block Launching of executable content from email attachment
AsrExecutableOfficeContentAudited AsrExecutableOfficeContentBlocked AsrExecutableOfficeContentWarnBypassed	Block Office applications from creating executable content
AsrLsassCredentialTheftAudited AsrLsassCredentialTheftBlocked AsrLsassCredentialTheftWarnBypassed	Block credential stealing from the Windows local security authority subsystem (lsass.exe)
AsrObfuscatedScriptAudited AsrObfuscatedScriptBlocked AsrObfuscatedScriptWarnBypassed	Block execution of potentially obfuscated scripts
AsrOfficeChildProcessAudited AsrOfficeChildProcessBlocked AsrOfficeChildProcessWarnBypassed	Block all Office applications from creating child processes
AsrOfficeCommAppChildProcessAudited AsrOfficeCommAppChildProcessBlocked AsrOfficeCommAppChildProcessWarnBypassed	Block Office communication application from creating child processes
AsrOfficeMacroWin32ApiCallsAudited AsrOfficeMacroWin32ApiCallsBlocked AsrOfficeMacroWin32ApiCallsWarnBypassed	Block Win32 API calls from Office macro
AsrOfficeProccessInjectionAudited AsrOfficeProccessInjectionBlocked AsrOfficeProccessInjectionWarnBypassed	Block Office applications from injecting code into other processes
AsrPersistenceThroughWmiAudited AsrPersistenceThroughWmiBlocked AsrPersistenceThroughWmiWarnBypassed	Block persistence through WMI event subscription
AsrPsexecWmiChildProcessAudited AsrPsexecWmiChildProcessBlocked AsrPsexecWmiChildProcessWarnBypassed	Block Process Creations originating from PSExec & WMI commands
AsrRandomwareAudited AsrRandomwareBlocked AsrRandomwareWarnBypassed	Use advanced protection against ransomware
AsrSafeModeRebootAudited AsrSafeModeRebootBlocked AsrSafeModeRebootWarnBypassed	Block rebooting machine in Safe Mode
AsrScriptExecutableDownloadAudited AsrScriptExecutableDownloadBlocked AsrScriptExecutableDownloadWarnBypassed	Block JavaScript or VBScript from launching downloaded executable content
AsrUntrustedExecutableAudited AsrUntrustedExecutableBlocked AsrUntrustedExecutableWarnBypassed	Block executable files from running unless they meet a prevalence, age, or trusted list criteria
AsrUntrustedUsbProcessAudited AsrUntrustedUsbProcessBlocked AsrUntrustedUsbProcessWarnBypassed	Block untrusted and unsigned processes that run from USB
AsrVulnerableSignedDriverAudited AsrVulnerableSignedDriverBlocked AsrVulnerableSignedDriverWarnBypassed	Block abuse of in-the-wild exploited vulnerable signed drivers
AsrWebShellOnServerAudited AsrWebShellOnServerBlocked AsrWebShellWarnBypassed (does not have the “OnServer” substring like the other 2)	Block Webshell creation for Servers

Each ASR Rule has its own GUID. GUIDs are needed when configuring, e.g., a GPO to enable ASR rules in Audit or Block mode for machines. The ASR Rule to GUID matrix can be found in Microsoft’s Attack surface reduction rules reference

The KQL query to search for ASR Rule detections is the following:

DeviceEvents
| where ActionType startswith "Asr"
| extend RuleId=extractjson("$Ruleid", AdditionalFields, typeof(string)) //Parse
//the AdditionalFields column in order to extract the ASR Rule GUID.

Controlled Folder Access Detections

When looking for CFA detections, keep the following in mind:

CFA has only 2 ActionType possible values reported in DeviceEvents:

ActionType	Description
ControlledFolderAccessViolationAudited	Controlled folder accesses detected an attempt to modify a protected folder.
ControlledFolderAccessViolationBlocked	Controlled folder access blocked an attempt to modify a protected folder.

The KQL query to search for CFA detections is the following:

DeviceEvents
//If you want only blocks, remove the "ControlledFolderAccessViolationAudited".
| where ActionType in ("ControlledFolderAccessViolationAudited","ControlledFolderAccessViolationBlocked")

Device Control

When looking for Device Control detections, keep the following in mind:

Device Control has 7 ActionType possible values reported in DeviceEvents:

DeviceEvents ActionType	Description
BluetoothPolicyTriggered	A Bluetooth service was allowed or blocked by a device control policy.
PnpDeviceConnected	This event appears when a PnP device is connected on an onboarded device, regardless of whether Device Control is enabled or not.
PnPDeviceAllowed	Device control allowed a trusted plug and play (PnP) device. Note that when a device installation restrictions are configured and a device is installed, an event with `ActionType` of `PnPDeviceAllowed` is created.
PnPDeviceBlocked	Device control blocked an untrusted plug and play (PnP) device.
PrintJobBlocked	Device control prevented an untrusted printer from printing.
RemovableStorageFileEvent	Removable storage file activity matched a device control removable storage access control policy.
RemovableStoragePolicyTriggered	Device control detected an attempted read/write/execute event from a removable storage device.

The KQL query to search for Device Control detections is the following:

DeviceEvents
| where ActionType in ("BluetoothPolicyTriggered","PnPDeviceBlocked",
"PrintJobBlocked","RemovableStorageFileEvent","RemovableStoragePolicyTriggered")
//You may want to search for "PnPDeviceAllowed" and "PnPDeviceConnected", too, after
//configuring Device Control policies, to make sure that the desired activities are
//allowed and to investigate further activities. But for checking for Device Control
//Blocks, it is not needed.

Keep in mind that even after excluding the PnPDeviceAllowed ActionType, the other possible ActionType values still contain activities which may not indicate blocking, but allowing.

For example, take a look at the following KQL query, taken from here:

DeviceEvents
| where ActionType == "RemovableStoragePolicyTriggered"
| extend parsed=parse_json(AdditionalFields)
| extend MediaClass = tostring(parsed.ClassName)
| extend MediaDeviceId = tostring(parsed.DeviceId)
| extend MediaDescription = tostring(parsed.DeviceDescription)
| extend SerialNumberId = tostring(parsed.SerialNumber)
| extend RemovableStoragePolicy = tostring(parsed.RemovableStoragePolicy)
| extend RemovableStorageAccess = tostring(parsed.RemovableStorageAccess)
| extend RemovableStoragePolicyVerdict = tostring(parsed.RemovableStoragePolicyVerdict)
| extend PID = tostring(parsed.ProductId)
| extend VID = tostring(parsed.VendorId)
| extend VID_PID = strcat(VID,"_",PID)
| extend InstancePathId = tostring(parsed.DeviceInstanceId)
| project Timestamp, RemovableStoragePolicy, RemovableStorageAccess,RemovableStoragePolicyVerdict, SerialNumberId,VID, PID, VID_PID, InstancePathId
| order by Timestamp desc

The ActionType RemovableStoragePolicyTriggered does not mean a block or an allow action by default. It means that a policy was triggered, and more details can be found in the AdditionalFields column.

Exploit Protection

When looking for Exploit Protection detections, keep the following in mind:

Ignore DeviceEvents with ActionType ExploitGuardNetworkProtectionAudited or ExploitGuardNetworkProtectionBlocked when checking for Exploit Protection, as these Action Types correspond to detection done by Network Protection, which is a different ASR, and will be described in the next section.

Some Exploit Protection measures do not create events, because they do not detect. For example, with Mandatory ASLR and Bottom-up ASLR, a program’s code and libraries and loaded at a random memory address instead of a predictable one. This measure does not detect anything to create an event for.

Microsoft’s documentation does not include the Control Flow Guard (CFG) DeviceEvents ActionType, it only includes the Exploit Protection measures Action Types actually with the substring ExploitGuard. The complete list of the possible ActionType values for Exploit Protection is depicted in the following table.

DeviceEvents ActionType	Description	Exploit Protection Measure
ExploitGuardAcgAudited ExploitGuardAcgEnforced	Arbitrary code guard (ACG) detected or blocked an attempt to modify code page permissions or create unsigned code pages.	Arbitrary code guard (ACG)
ExploitGuardChildProcessAudited ExploitGuardChildProcessBlocked	Exploit protection detected or blocked the creation of a child process	Don’t allow child processes
ExploitGuardEafViolationAudited ExploitGuardEafViolationBlocked	Export address filtering (EAF) blocked possible exploitation activity.	Export address filtering (EAF)
ExploitGuardIafViolationAudited ExploitGuardIafViolationBlocked	Import address filtering (IAF) detected or blocked possible exploitation activity.	Import address filtering (IAF)
ExploitGuardLowIntegrityImageAudited ExploitGuardLowIntegrityImageBlocked	Exploit protection detected or blocked the launch of a process from a low-integrity file.	Block low integrity images
ExploitGuardNonMicrosoftSignedAudited ExploitGuardNonMicrosoftSignedBlocked	Exploit protection detected or blocked the launch of a process from an image file that is not signed by Microsoft.	Code integrity guard
ExploitGuardRopExploitAudited ExploitGuardRopExploitBlocked	Exploit protection detected or blocked possible return-object programming (ROP) exploitation.	Simulate execution (SimExec) Validate API invocation (CallerCheck) Validate stack integrity (StackPivot)
ExploitGuardSharedBinaryAudited ExploitGuardSharedBinaryBlocked	Exploit protection detected or blocked the launch of a process from a file in a remote shared file.	Block remote images
ExploitGuardWin32SystemCallAudited ExploitGuardWin32SystemCallBlocked	Exploit protection detected or blocked a call to the Windows system AIP	Disable Win32k system calls
ControlFlowGuardViolation	Control Flow Guard terminated an application after detecting an invalid function call	Control flow guard (CFG)

Unfortunately, it seems that for the following Exploit Protection measures, there are no Device Events created. If you know more details, please let me know:

Data Execution Prevention (DEP)
Validate exception chains (SEHOP)
Validate heap integrity
Block untrusted fonts
Disable extension points
Validate handle usage
Validate image dependency integrity
Hardware-enforced stack protection

The KQL query to search for Exploit Protection detections is the following:

DeviceEvents
| where ActionType startswith "ExploitGuard" and ActionType !contains "NetworkProtection" or ActionType == "ControlFlowGuardViolation"

Network Protection, Web Protection, and SmartScreen

Network, Web Protection, and Smart Screen are in the same section because of their co-dependency.

As mentioned in the previous section, Network and Web Protection have only 2 ActionType possible values reported in DeviceEvents:

ActionType	Description
ExploitGuardNetworkProtectionAudited	Network protection detected an attempt to access a malicious or unwanted IP address domain or URL.
ExploitGuardNetworkProtectionBlocked	Network protection blocked a malicious or unwanted IP address domain or URL.

Because Microsoft Defender SmartScreen is also involved in the detections, the following SmartScreen related ActionType values should also be searched for:

ActionType	Description
SmartScreenExploitWarning	SmartScreen warned about opening a web page that contains an exploit.
SmartScreenUrlWarning	SmartScreen warned about opening a low-reputation URL that might be hosting malware or is a phishing site.
SmartScreenAppWarning	SmartScreen warns about running a downloaded application that is untrusted or malicious.
SmartScreenUserOverride	A user has overriden a SmartScreen warning and continued to open an untrusted app or a low-reputation URL.

As shown above, some SmartScreen detections are related to URL detections, application detections, or both.

The KQL query to search for Network/Web Protection or SmartScreen detections is the following:

DeviceEvents
|where ActionType in ("ExploitGuardNetworkProtectionAudited","ExploitGuardNetworkProtectionBlocked",
"SmartScreenExploitWarning","SmartScreenUrlWarning","SmartScreenAppWarning","SmartScreenUserOverride")

Below is an example of how you can parse the JSON data in the AdditionalFields column when it comes to Network/Web Protection detections (ref. Microsoft’s Documentation):

DeviceEvents
|where ActionType in ("ExploitGuardNetworkProtectionAudited","ExploitGuardNetworkProtectionBlocked")
|extend ParsedFields=parse_json(AdditionalFields)
|project DeviceName, ActionType, Timestamp, RemoteUrl, InitiatingProcessFileName,IsAudit=tostring(ParsedFields.IsAudit),
ResponseCategory=tostring(ParsedFields.ResponseCategory), DisplayName=tostring(ParsedFields.DisplayName)
|sort by Timestamp desc

Tamper Protection

Regarding Tamper Protection, the Defender AlertInfo table can be searched for any Alert raised related to Defender tampering.

The KQL query to search for Defender alerts is the following:

AlertInfo
| where Title contains "tamper"

The DeviceEvents table can also be used to identify tampering attempts. Tampering attempts could be done either by the user of by any other process, including Group Policies enforcement, Intune, etc.

Keep in mind that when Tamper Protection is on, the following settings are blocked from being changed by any means, including GPO, Intune, Configuration Manager, etc.

Virus and threat protection remains enabled.
Real-time protection remains turned on.
Behavior monitoring remains turned on.
Antivirus protection, including IOfficeAntivirus (IOAV) remains enabled.
Cloud protection remains enabled.
Security intelligence updates occur.
Automatic actions are taken on detected threats.
Notifications are visible in the Windows Security app on Windows devices.
Archived files are scanned.
Exclusions can’t be modified or added

So, most likely, some Tamper Attempts initiated by processes like svchost.exe will be reported, with commands like svchost.exe -k GPSvcGroup. These are attempts of defender settings changes done by processes like the Service Host svchost.exe.

The KQL query to search for Tamper Protection related events is the following:

DeviceEvents
| where ActionType contains "tamper"

MDAV Detections

When it comes to detections from Microsoft Defender Antivirus, there are five different ActionType possible values:

ActionType	Description
AntivirusDetection	Windows Defender Antivirus detected a threat.
AntivirusError	Windows Defender Antivirus encountered an error while taking action on malware or a potentially unwanted application.
AntivirusMalwareActionFailed	Windows Defender Antivirus attempted to take action on malware or a potentially unwanted application but the action failed.
AntivirusMalwareBlocked	Windows Defender Antivirus blocked files or activity involving malware or potentially unwanted applications or suspicious behavior.
AntivirusReport	This should be ignored. It creates too many events without any useful info. It reports events of when Microsoft Defender Antivirus reported a threat, which can be a memory, boot sector, or rootkit threat.

The KQL query to search for MDAV detections is the following:

DeviceEvents
| where ActionType in ("AntivirusDetection","AntivirusError",
"AntivirusMalwareActionFailed","AntivirusMalwareBlocked")

Potential Unwanted Apps (PUA)

PUA detections are under the ActionType of value AntivirusDetection, and therefore its detections can be found with the KQL queries of the previous section.

If there is a focus to investigate specifically for PUA detections/blocks, the following KQL query can be used:

DeviceEvents
| where ActionType == "AntivirusDetection"
| where tostring(AdditionalFields) contains "PUA"

Generally, it was noticed that the KQL queries provided by Microsoft’s documentation sometimes have issues. For example, the KQL query provided here to view PUA events only checks for a subset of PUA events, because it checks for the “PUA:” substring instead of “PUA” (without the column), missing many events like “PUABundler”. Therefore, it is better to use the above KQL query provided in this post.

Windows Defender Application Control (WDAC) and AppLocker

WDAC and AppLocker have a lot of different possible ActionType values, which are depicted in the table below:

ActionType	Description
AppControlAppInstallationAudited AppControlAppInstallationBlocked	App control detected/blocked the installation of an untrusted app.
AppControlCIScriptAudited AppControlCIScriptBlocked	A script MSI file generated by Windows LockDown Policy was audited/blocked.
AppControlCodeIntegrityDriverRevoked	Application control found a driver with a revoked certificate.
AppControlCodeIntegrityImageAudited	Application control detected an executable file that violated code integrity policies.
AppControlCodeIntegrityImageRevoked	Application control found an executable file with a revoked certificate.
AppControlCodeIntegrityOriginAllowed	App control allowed a file due to its good reputation (ISG) or installation source (managed installer).
AppControlCodeIntegrityOriginAudited AppControlCodeIntegrityOriginBlocked	App control detected/blocked a file due to its bad reputation (ISG) or installation source (managed installer).
AppControlCodeIntegrityAudited AppControlCodeIntegrityBlocked	App control detected/blocked a code integrity policy violation.
AppControlCodeIntegrityLoaded	An app control code integrity policy was loaded.
AppControlCodeIntegritySigningInformation	App control signed information was generated.
AppControlExecutableAudited AppControlExecutableBlocked	App control detected/blocked the use of an untrusted executable.
AppControlPackagedAppAudited AppControlPackagedAppBlocked	App control detected/blocked the use of an untrusted packages app.
AppControlPolicyApplied	An app control policy was applied to the device.
AppControlScriptAudited AppControlScriptBlocked	App control detected/blocked the use of an untrusted script.
AppLockerBlockExecutable	AppLocker prevented an untrusted executable from running.
AppLockerBlockPackagedApp	AppLocker prevented an untrusted packaged app from running.
AppLockerBlockPackagedAppInstallation	AppLocker prevented the installation of an untrusted packaged app.
AppLockerBlockScript	AppLocker prevented an untrusted script from running.

The KQL query to search for WDAC and AppLocker detections is the following:

DeviceEvents
| where (ActionType startswith "AppControl" or ActionType startswith "AppLocker") and ActionType contains "block"

MDE Alerts

MDE Alerts can be queried to investigate for detections which were not reported by the above queries, with the following KQL.

AlertInfo
| where Timestamp > ago(30d)
| where ServiceSource == "Microsoft Defender for Endpoint"

General KQL query for all detections

There are almost too many Action Types to look for when investigating the behavior of Defender on a machine. In order to have a headstart, the following KQL query is written to search for most, if not all, of them. The query creates a new column named MDEMeasure based on the MDE capability, with the following possible values:

MDEMeasure Value	MDE Capability
ASRRules	Attack Surface Reduction Rules
CFA	Controlled Folder Access
DeviceControl	Device Control
ExploitProtection	Exploit Protection
TamperProtection	Tamper Protection
PUA	Potentially Unwanted Apps
MDAV	Microsoft Defender Antivirus
WDAC_AppLocker	Windows Defender App Control and AppLocker

DeviceEvents
| where DeviceId == "" //If you are investigating a specific device, then enter its Device ID in this filter
| extend MDEMeasure = case(ActionType startswith "Asr" and (ActionType endswith "Blocked" or ActionType endswith "WarnBypassed"), "ASRRules",
ActionType == "ControlledFolderAccessViolationBlocked", "CFA",
ActionType in ("BluetoothPolicyTriggered", "PnPDeviceBlocked", "PrintJobBlocked", "RemovableStorageFileEvent", "RemovableStoragePolicyTriggered"), "DeviceControl", //Device Control, these need to be checked for Allow actions
(ActionType startswith "ExploitGuard" and (ActionType endswith "Enforced" or ActionType contains "Blocked")) and ActionType !contains "NetworkProtection" or ActionType =="ControlFlowGuardViolation" , "ExploitProtection",
ActionType in ("ExploitGuardNetworkProtectionBlocked","SmartScreenExploitWarning","SmartScreenUrlWarning","SmartScreenAppWarning","SmartScreenUserOverride"),"Network_Web_SmartScreen",
ActionType contains "tamper","TamperProtection",
ActionType in ("AntivirusDetection","AntivirusError","AntivirusMalwareActionFailed","AntivirusMalwareBlocked") and tostring(AdditionalFields) !contains "PUA","MDAV",
ActionType == "AntivirusDetection" and tostring(AdditionalFields) contains "PUA","PUA",
(ActionType startswith "AppControl" or ActionType startswith "AppLocker") and ActionType contains "block","WDAC_AppLocker",
"NonMDE")
| where MDEMeasure != "NonMDE"
// | summarize count() by MDEMeasure, ActionType

Conclusion

It is always a complex task to definitively reply to the question “Did Defender block something?”. With this post, hopefully is has become at least a little bit easier.

Stick around for the next posts, where we will dive deep into looking Windows Event Logs and how to Troubleshoot Performance Issues for MDE.

I hope to see you again.

What is Microsoft Defender for Endpoint, for the Endpoint - Getting to know MDE Part 1

2025-12-17T00:00:00+00:00

Introduction

What is MDE, for the Endpoint? I always wondered what is going on on the device after MDE is installed on it and it gets onboarded to MDE. There are so many capabilities that MDE and Defender Antivirus (MDAV) provides, that it is difficult to know what is happening in the background when something is blocked or the performance of the machine is degraded when running a specific process.

In this series about MDE named “Getting to know MDE”, I will try to gather and simplify all the concepts needed to know when running MDE on an endpoint. The purpose of the series is to have a better understanding of what exactly is being run on a machine onboarded to MDE, what is being checked/scanned and how, and how to approach a troubleshooting session effectively in case that it is needed.

Many of us have heard “is it Defender?” when a machine is performing poorly or an application is not running as expected. This series is an attempt to have an assistive cheatsheet in answering quickly and effectively.

In this Blog post, the focus is on what the capabilities of MDE are, when it is running on an endpoint. The capabilities are explained on a high level, without going into many details, just to get an understanding of what each capability does. In the next posts, more details of each capability will be provided, and what to look for when troubleshooting them.

All the information is based on official documentation of Microsoft, which you can find here.

So let’s get started.

Introduction
Contents
MDE Components and Capabilities
Conclusion

MDE Components and Capabilities

1 Attack Surface Reduction (ASR)

ASR is the first line of defense in the world of MDE. ASR, as its name suggests, applies a series of security configurations and protections which aim at minimizing the potential entry points for attackers. This is done by blocking different series of actions which are known to be frequently used by attacks, but rarely used for legitimate purposes.

1.1 Attack Surface Reduction Rules

ASR Rules is one of the protection measures included under ASR. These protect against common malware behaviors and risky application actions. The ASR Rules can be separeted in a few different categories based on what protection they provide, and they are the following:

Polymorphic threats	Lateral movement & credential theft	Productivity apps rules	Email rules	Script rules	Misc rules
Block executable files from running unless they meet a prevalence (1,000 machines), age, or trusted list criteria	Block process creations originating from PSExec and WMI commands	Block Office apps from creating executable content	Block executable content from email client and webmail	Block execution of potentially obfuscated scripts	Block abuse of exploited vulnerable signed drivers
Block untrusted and unsigned processes that run from USB	Block credential stealing from the Windows local security authority subsystem (lsass.exe)	Block Office apps from creating child processes	Block only Office communication applications from creating child processes	Block JS/VBS from launching downloaded executable content	Block webshell creation for servers
Use advanced protection against ransomware	Block persistence through WMI event subscription	Block Office apps from injecting code into other processes	Block Office communication apps from creating child processes
Block use of copied of impersonated system tools	Block rebooting machine in Safe Mode	Block Adobe Reader from creating child processes
		Block Win32 API calls from office macros

1.2 Controlled Folder Access (CFA)

CFA is a measure that mostly protects against attacks related to ransomware.

With CFA essentially you define a set of directories (folders) and a set of applications. Only that set of apps is allowed to process in any way the set of directories.
There is already a default set of directories and processes in CFA which can be used immediately.
Apps are added to the list automatically when they are highly prevalent in the organization and haven’t displayed any behavior deemed malicious.
In the default folder list, the folders Documents, Pictures, Videos, Videos, Music, and Favorites are included.

1.3 Device Control

Device Control enables controls related to usage and installation of peripheral (USB/Bluetooth) or other devices with endpoints. Common scenarios include:

Control access to USB devices:
- Configure device installation restrictions
- Control access to removable media
BitLocker control: Block usage of removable media if BitLocker is disabled
Control access to printers
Bluetooth services control: Allowing advertising, discovery, preparing prompting

1.4 Exploit Protection

Exploit Protection helps protect against malware that uses exploits to infect devices and spread. It consists of many mitigations that can be applied to either the whole OS or individual apps. It protects by default some behaviors which are correlated to exploits, like an app accessing parts of memory which it should not be able to. The protection measures vary from simple which can be enabled OS-wide and in audit mode to test, all the way to complex protection which protect only per-application and cannot be turned on audit mode but only be turned on immediately. Exploit Protection provides the following protection measures:

Control flow guard (CFG): Mitigates the risk of attackers using memory corruption vulnerabilities by protecting indirect function calls.
Data Execution Prevention (DEP): Prevents memory that wasn’t explicitly allocated as executable from being executed.
Force randomization for images (Mandatory ASLR): With Address Space Layout Randomization (ASLR), every time a program runs, Windows loads its code and libraries at a random memory address, to block attackers predicting where these are located.
Randomize memory allocations (Bottom-Up ASLR): This mitigation requires Mandatory ASLR. This adds entropy to relocations, so their location is randomized.
Validate exception chains (SEHOP): Mitigation against the Structured Exception Handler (SEH) overwrite exploitation technique.
Validate heap integrity: This increases the protection level of heap mitigations in Windows, by causing the app to terminate if a heap corruption is detected.
Arbitrary code guard (ACG): Protect an application from executing dynamically generated code. It prevents memory from being marked as executable.
Block low integrity images: Prevents the app from loading files that are untrusted, because they’ve been downloaded from the internet from a sandboxed browser.
Block remote images: Prevents the application from loading files that are hosted on a remote device, such as UNC share, which could be a device controlled by the attacker.
Block untrusted fonts: Mitigates the risk of a flow in font parsing leading to the attacker being able to run code on the device.
Code integrity guard: Ensure that all binaries loaded into a process are digitally signed by Microsoft. With the use of WHQL (Windows Hardware Quality Labs) signatures, drivers developed by third parties can be signed to also become trusted.
Disable extension points: Windows lets programs hook into or extend other programs in a few ways, for example, by loading DLLs automatically or watching certain system events. These are called extension points. This function disables various extension points for an app, which might be used to establish persistence or elevate privileges of malicious content, thus only loading the program’s own code.
Disable Win32k system calls: Win32k.sys is a Windows kernel module that connects apps to the display and input system. Because it is running in kernel mode, and apps can indirectly call its functions, sandboxes/low-privilege apps could exploit it to escape the sandbox and gain admin-level privileges. So, disabling Win32k system calls for an app means that this app doesn’t need to draw windows or interact with the display, so there is no need to communicate with the Windows GUI system at all.
Don’t allow child processes: Prevents an app from creating child processes/applications.
Export address filtering (EAF): Mitigates the risk of malicious code looking at the export address table of all loaded modules to find modules that contain useful APIs for their attack.
Import address filtering (IAF): This mitigation protects against modifying the import address table (IAT), potentially redirecting to arbitrary code.
Simulate execution (SimExec): Only for 32-bit apps. It helps validate that calls to sensitive APIs return to legitimate caller functions.
Validate API invocation (CallerCheck): It is a mitigation for return-oriented programming (ROP) techniques that validates that sensitive APIs were called from a valid caller.
Validate handle usage: A handle is a reference to a protected object. This measure is a mitigation that helps protect against an attacker using an existing handle to access a protected object.
Validate image dependency integrity: Helps protect against attacks that attempt to substitute code for DLLs that are statically linked by Windows binaries.
Validate stack integrity (StackPivot): Helps protect against the Stack Pivot attack, a ROP attack where an attacker creates a fake stack in heap memory, and then tricks the application into returning into the fake stack that controls the flow of execution.
Hardware-enforced stack protection: This protects against ROP based attacks. Attackers modify the return address stored on the stack, which tells the CPU where to go back to after finishing a function.

1.5 Web and Network Protection

1.5.1 Web Protection

Web Protection in MDE includes the following capabilities:

Custom Indicators: When defined in MDE portal, with the Web Protection feature enabled, the endpoints are blocked when accessing the Blocked Indicators (URLs/IPs).
Web Threat Protection: It stops access to phishing, malicious, untrusted, or low-reputation sites.
Web Content Filtering (WCF): This provides the ability to block access of websites based on categories (e.g., gambling, torrenting, cloud storage, etc.).

1.5.2 Network Protection

Network Protection in MDE:

Protects devices by preventing connections to malicious or suspicious sites.
Expands MS Defender SmartScreen to block all outbound HTTP(S) traffic to poor-reputation destinations.
Extends Web Protection, which works only on Microsoft Edge browser, to the OS level.
Is a core component to Web Content Filtering.
Provides visibility and blocking of IoCs when used with EDR. Indicators defined in MDE portal are blocked only if Network Protection is enabled.

2 Next-Generation Protection

Under Next-Gen Protection is where most of the more “advanced” protection mechanisms reside to block emerging threats.

2.1 Microsoft Defender Antivirus (MDAV)

Here is where the famous MDAV lives. With it, process creation events and file download events from the internet are monitored. It not only uses its signature-based engine, but also predictive technologies such as machine learning and cloud-delivered protection to find attacks. If working offline, the latest dynamic intelligence from the Intelligence Security Graph is provisioned regularly throughout the day.

2.2 Cloud Protection and MDAV

To identify new threats dynamically, Next-Gen Protection technologies work with

AI systems which use machine learning models
Large sets of interconnected data in the Microsoft Intelligent Security Graph.

MDAV works with Microsoft Cloud services, also known as Microsoft Advanced Protection Service (MAPS). With these, next-gen technologies provide quick identification of new threats. This is done by MDAV uploading samples of metadata or the samples themselves to allow Cloud protection to identify if the samples are malicious.

2.3 Tamper Protection

When Tamper Protection is enabled on a machine, the following MDAV setting cannot be changed by anyone, not even by applying a new GPO or a setting in Intune:

Virus and threat protection remains enabled.
Real-time protection remains turned on.
Behavior monitoring remains turned on.
Antivirus protection, including IOfficeAntivirus (IOAV) remains enabled.
Cloud protection remains enabled.
Security intelligence updates occur.
Automatic actions are taken on detected threats.
Notifications are visible in the Windows Security app on Windows devices.
Archived files are scanned.
Exclusions can’t be modified or added

2.4 Block at first sight (BAFS)

BAFS is a simple feature in MDE. When enabled, if a file or executable is never seen before by MDE and it is identified as suspicious, the opening action or the execution of it is blocked until a verdict is received from the cloud-delivered protection.

2.5 Anti-Malware Scan Interface (AMSI) integration with MDAV

AMSI provides the capability to inspect PowerShell and other scripts, even if there is obfuscation applied on them. MDE utilizes AMSI to protect against fileless malware, dynamic script-based attacks, and other threats of this nature.

2.6 MDAV Components and Technologies

MDAV utilizes multiples engines to be able to detect and prevent a wide range of threats and attacker techniques. The following diagram shows the different engines used:

As shown above, there is a balance between engines running locally on the client which provide real time protection, and engines being provided via cloud-delivered protection which handle the heavy load when needed. With this graph the necessity of the cloud-delivered protection is also highlighted, as a machine gets a plethora of additional capabilities when it is enabled.

In the following table, the engines are described briefly:

On the Client	In the Cloud
ML engine: Lightweight ML engine. It has specialized models for specific file types commonly abused, like PE files, PowerShell, macros, JS, PDF, etc.	Metadata-based ML engine: Specialized ML models analyze a featurized description of suspicious files sent by the client. Stacked ensemble classifiers combine results to make a verdict. You can read more on how it works in this Microsoft blogpost.
Behavior monitoring engine: Monitors for potential attacks post-execution. It observes process behaviors, including behavior sequence at runtime, to identify and block activities based on predetermined rules.	Behavior-based ML engine: Suspicious behavior sequences are used to trigger to analyze the process tree behavior using ML models. Monitored attack techniques span the attack chain, like exploits, elevation, persistence, lateral movement, and exfiltration.
Memory scanning engine: Scans memory space used by a running process to expose malicious behavior that could be hiding with code obfuscation.	AMSI-paired ML engine: Client-side and cloud-side pairs analyze scripts behavior pre and post execution to detect threats like fileless and in-memory attacks.
AMSI integration engine: Enables detection of files and in-memory attacks, defeating code obfuscation. This blocks malicious behavior of scripts client-side.	File classification ML engine: Multi-class, deep neural network classifiers examine full file contents. Suspicious files are held from running and submitted to the cloud protection service for classification.
Heuristics engine: Rules identify file characteristics that have similarities with known malicious characteristics to catch new threats or modifications of known ones.	Detonation-based ML engine: Suspicious files are detonated in a sandbox. Deep learning classifiers analyze the observed behavior to block attacks.
Emulation engine: Dynamically unpacks malware and examine how they would behave at runtime. Checks both during runtime and the memory content after, finding malware packers and polymorphic malware.	Reputation ML engine: Reputation sources and models from all Microsoft are queried to block threats linked to malicious/suspicious URLs, domains, emails, and files. Sources include SmartScreen, MDO, and others through the Microsoft Intelligent Security Graph.
Network engine: Network activity is inspected.	Smart rules engine: Smart rules identify threats based on researcher expertise and collective knowledge of threats.
CommandLine Scanning engine: Scans command lines of all processes before they execute.	CommandLine ML engine: ML models scan suspicious command lines in the cloud.

2.7 Potentially Unwanted Apps (PUA)

PUA is a category of software that can:

Cause the machine to run slowly
Display unexpected apps
Install other software that might be unwanted

PUA could be:

Advertising software
Bundled software that offers to install additional software not related to the original one, and could be not signed by the same entity as the original one
Software that evades detection

PUA could be a problem because:

Increase the risk of the endpoints being infected with actual malware
Make malware harder to identify
Cost time and effort to clean up

Both MDAV and SmartScreen provide PUA protection.

2.8 Behavioral blocking and containment

This is a feature based on AI/ML to target fileless malware, polymorphic threats, and human-operated attacks. It detects attacks based on their behaviors and process trees. This feature utilizes the following:

Next-Gen Protection detects threats by analyzing behaviors.
EDR received signals across your network, devices, and kernel behavior, creating alerts and incidents.
MDE has visibility in email, data, apps, network, endpoint, and kernel behavior signals received through EDR. MDE correlates these signals, and raises alerts and incidents.

Components of Behavioral blocking and containment:

ASR Rules prevent predefines common attack behaviors.
Client behavioral blocking: As suspicious behaviors are detected on devices by MDAV, artifacts such as files or apps are blocked, checked, and remediated automatically. When suspicious behaviors are detected, MDAV sends them and their process tress to the cloud protection service to determine their maliciousness. If detected, it is blocked on the device.
Feedback-loop blocking, AKA Rapid Protection: With it, when a suspicious behavior or file is detection, information about it is sent to multiple classifiers. The rapid protection loop engine inspects and correlated the info with other signals to identify if to block the file. This results in blocking malware on a device, other devices in the org, and devices in other orgs. So there is this community approach where you will get threat intel from other orgs based on detections.

2.9 UEFI scanning in MDE

MDE now has a UEFI scanner. This helps in attempts of attackers compromosing the boot flow to achieve low-level malware behavior that is hard to detect.

Windows Defender System Guard combats this with hardware-based security features including hypervisor-level attestation and Secure Launch (aka Dynamic Root of Trust (DRTM)).

With the new UEFI scanner, firmware scanning becomes broadly available. It is built-in with MDAV. It gives MDE the ability to scan inside the firmware file system and perform security assessments. It performs dynamic analysis on the firmware it gets from the hardware flash storage. By obtaining the firmware, the scanner is able to parse the firmware, enabling MDE to inspect firmware content at runtime.

2.10 Early Launch Antimalware (ELAM) and MDAV

ELAM combats early boot threats (e.g., rootkits, or malicious drivers that can hide from detection) by using a driver named Wdboot.sys that starts before other boot-start drivers. ELAM enables the evaluation of other drivers, and helps the Windows kernel decide whether those drivers should be initialized.

3 Scheduled Scans

There are three types of scans in MDAV:

Quick Scan: Scan the locations with high probability of having malware registered to start with the system, including registry keys and known Windows startup folders.
Full Scan: Start with a Quick Scan, and then scan all mounted fixed disks and removable and network drives. A Full Scan could take a few hours or days to complete.
Custom Scan: Scan what is provided in input.

In most cases, the recommended approach is to execute only one Full Scan per endpoint, and then only run Quick Scans, because them together with Real-Time Protection being enabled covers all files in an endpoint.

4 MDAV Modes

MDAV could be operating in different modes, depending on whether there is another AV solution installed on the endpoint, or based on the policies and security settings applied on the endpoint:

Active Mode:
1. MDAV is the main AV solution on the machine.
2. Files are scanned, and actions are taken, i.e., threats are actively remediated.
Passive Mode:
1. MDAV is not the main AV solution on the machine.
2. While files are still scanned, actions are not taken, i.e., threats are not remediated.
3. Updates should still be applied, as they improve alerting and performance(!).
Endpoint Detection and Response (EDR) in Block Mode:
1. This is an enhancement to Passive Mode, where actions are taken in malicious artifacts detected by EDR capabilites. Actions are taken on post-breach, behavioral EDR detections, which were not detected by the installed AV solution.
2. Several features are still not available (Active Mode is required):
- Real-Time Protection
- Network Protection
- ASR
- Indicators
Disabled or Uninstalled:
1. MDAV is not the main AV solution on the machine.
2. Files are not scanned and actions are not taken.
3. It is recommended to have MDAV in Passive Mode for additional alerting, and if the main AV gets removed for any reason (licensing, uninstall, etc.), then MDAV will switch to Active Mode automatically, to keep protecting the machine.

Conclusion

This post got longer that I hoped for, even after cutting quite a bit of details. Hopefully it will help have more spherical knowledge around MDE when it is running on an endpoint.

I hope to see you back in the next posts.

Kostas SecBlog

Using Windows Events to identify detections from MDE - Getting to know MDE Part 4

Introduction

Contents

Event Viewer general info

XPath queries

Demo Files

XPath Queries for Defender components

MDAV Changes logging

Attack Surface Reduction (ASR) rules

Controlled Folder Access (CFA)

Device Control

Exploit Protection

Network Protection, Web Protection, and SmartScreen

Tamper Protection

Microsoft Defender Antivirus (MDAV) Detections

Potentially Unwanted Apps (PUA)

App Control for Business and AppLocker

MDE

General XPath query for all detections

Conclusion

Cyber Range as Code - Part 2 - Automating the base - Hypervisor and Firewall

Introduction

Contents

Brief reminder of project architecture

Automate Proxmox

Initial Setup

Proxmox Ansible

Ansible Collection

Setting up SSH

Initializing Ansible for Proxmox

Running Ansible to configure Proxmox

Issues met and resolutions

Automate OPNSense

OPNSense Packer

OPNSense Packer Issues and resolutions

OPNSense Terraform

OPNSense Ansible

Configuring Proxmox for OPNSense with Ansible

Next Steps

Conclusion

Cyber Range as Code: Automating Security Lab with IaC - Part 1

Introduction

Contents

Automated Infrastructure

Tech stack

Architecture

Initial Configuration / PoC

Proxmox Configuration

Network Configuration

Proxmox Firewall

Manual OPNsense Installation

VM Creation

VM Installation

OPNsense network interfaces

OPNsense Firewall rules

OPNsense DHCP

Packer Configuration

Prepare Proxmox for Packer

Preparing packer

Packer pitfalls, solutions, and lessons learnt

Packer build and PoC result

Terraform Configuration

Prepare Proxmox for Terraform

Preparing Terraform

Terraform pitfalls, solutions, and lessons learnt

Terraform apply and PoC result

Ansible

Running Ansible

Next Steps

Conclusion

MDE Performance Troubleshooting - Getting to know MDE Part 3

Introduction

Contents

Common reasons for higher CPU by MDAV

Binaries not signed

Obfuscated scripts

Not letting MDAV cache finish before sealing a VDI image

Misspelled exclusions

Path exclusions only work for scanning flows