Chapter 3–9 Effective Key Performance Indicator (KPI) Design — The Importance of Targeted Metrics
Slinging Shots at Machine Tanks — Avoiding Useless KPIs
Company management often asks, “How effective is our network security protection? Are we doing enough?”
Designing effective security KPIs is crucial for any organization to assess the effectiveness of their security measures.
Many security personnel tend to use impressive numbers to deceive themselves or management. Usually, two key performance indicators are quoted: the number of external attacks the company firewall has blocked, and the number of spam or phishing emails the company network defense has prevented.
In reality, most of the external “attacks” blocked by the firewall are harmless. Routine network scans are conducted for various purposes, not all of which are malicious, and mostly do no harm to the company defense. It is like slinging shots at machine tanks — millions of hits, but not even a dent. The same applies to spam or phishing emails, as 90% of Internet emails are spam or phishing, and can be easily filtered in the cloud before even reaching the company.
What we should focus on are the threats that have bypassed the company’s defense layers, whether it’s one layer, multiple layers, or completely undetected.
A Conceptual Design for Security KPIs:
Zone 1:
The alerts in Zone 1 are generated by internal endpoints such as laptops, servers, internal firewalls, and other security sensors, excluding local USB-related alerts. These alerts usually indicate that all perimeter defenses, including inbound firewall, WAF, IDS/IDP, and email sandbox, have failed.
By using this KPI in Zone 1, we can evaluate and address any deficiencies in our perimeter controls.
KPIs to track alerts:
Number of viruses detected.
Number of phishing emails that made it to user inboxes.
Operational KPIs to track:
Percentage of endpoints installed with security agents.
Percentage of installed security agents that are not running.
Percentage of security agents with outdated software versions or definition files as per company security policy.
I remember at a security conference when I asked the audience about the above three KPIs, almost no one raised their hand when I asked the last question.
Zone 2:
Alerts from Zone 2 are generated by outbound controls such as proxy and firewall, indicating that the threat has penetrated two layers of protection from the inbound perimeter and local endpoint protections. This will enable us to assess and address deficiencies in other layers of control.
The KPI in Zone 2 allows us to evaluate and address protection deficiencies in Zone 1.
Zone 3:
Alerts from Zone 3 are generated by sources outside the traditional company perimeter. If you subscribe to third-party posture rating services, they have a vast number of honeypots. If they detect attack traffic from your company, it will trigger those alerts.
The KPI in Zone 3 allows you to evaluate the overall company’s internal protection deficiencies. It helps you understand why threats can come and go without detection.
Zone 4:
Here, I refer to the DMZ, which hosts all applications that are accessible to the public Internet, including those hosted in the public cloud.
The most common threat vectors’ paths are from USBs, phishing emails, insecure user Internet browsing, and compromised DMZ applications.
To protect DMZ applications, I would suggest implementing the following controls and tracking these KPIs:
1. Visibility Scan and Differential Scan
These are critical tools for a security organization to identify which company applications are exposed to the outside world. Traditional scans require manual input of exposed IP addresses, but nowadays, some service providers can conduct scans using only the company domain name.
Additionally, any changes made without proper approval through Change Management should trigger immediate alerts, such as new or missing hosts, IPs, and hosts with new ports.
2. Vulnerability Scan
The key KPIs here are:
Critical vulnerabilities mitigated within X days per security policy, with a target of 0. All critical findings must be fixed within X days.
High vulnerabilities mitigated within Y days per security policy.
The security team should focus on KPI exception management.
3. Other Suggested Controls and Alerts:
Monitoring the security health (patch level, discovered vulnerabilities, security agents) of hosts inside the DMZ.
Analyzing firewall deny logs to focus on denied traffic initiated from DMZ hosts, either going outbound or inbound to the company’s intranet.
A compromised DMZ host is often used as a springboard to download new tools or scan for potential targets inside the DMZ or company intranet to attack next.
The same KPI Design Concept is written in the Cybersecurity Reference Architecture for Semiconductor Manufacturing Environments, which I co-authored.
