I have addressed with brevity pentesting in this article and mainly focused on “DDoS testing”. The reason being, I believe pentesting is a well known concept for security personnel and management in most organizations. “DDoS testing” is a relatively new concept for many people.
I’ve attempted on a macro level to give insight into the differences between the two and how they overlap. There’s a significant amount of ground covered on what “DDoS testing” is and the kind of testing which can be performed.
I submit the case of DDoS testing, for organization’s who currently have DDoS mitigation systems and cannot afford downtime for their services. I would even go further to suggest that most organizations today are not aware of their current weaknesses or implications when it comes to persistent DDoS attacks.
What is Penetration Testing?
Penetration Testing is commonly referred to as “pentesting”. This comprises of an intense and legal security assessment, done on a network by a security professional (Certified or not) to actively find vulnerabilities which could potentially be exploited to gain access to your system.
A “pentester” in some cases will try to exploit vulnerabilities found. This is done in order to validate if the vulnerability found, is in fact exploitable. The results are dependent on the type of pentest ordered and the skill of the pentesters at work. These factors will normally determine whether or not pentesters will be successful in gaining full access to the system.
Why are pentests performed?
Pentests are performed in order to strengthen your organizations cyber security posture. The logic is that if a pentester can gain access to your system and extract data or cause other damage, so can a criminal who has some agenda against your organization.
There are many types of pentests but generally they “validate” security mitigation systems and setups. Some common aspects validated are:
- Data leakage prevention systems
- Human error (Social engineering)
- Un-patched vulnerabilities
- Business logic flaws in software
- Encryption issues
The list goes on…
A pentest can be performed either from the external perimeter i.e. the internet or from the internal network itself and of course both internal and external tests could also be tested simultaneously. A pentest can be anything from a few days to a few months in length, depending on the organization being tested and the level of risk that organization wishes to mitigate and understand.
It’s important to note, by law some companies are required to have frequent pentests if they require various compliances such as PCI, HIPAA or GLBA.
What happens once vulnerabilities are found during a pentest?
Perhaps the most important phase of a pentest, is that once weaknesses are identified those weaknesses will be closed to make it tougher for a potential attacker to exploit an organization. Of course this can be a complex and timely process in reality.
Each security bug will normally be addressed with what is deemed to be the easiest bug to exploit, coupled with the highest amount of damage the exploitation of that bug could cause.
How and when security bugs are addressed is normally a practicality issue. The fact is that some security flaws may require multiple teams, as well as vendors to work together to solve a single bug. Dependent on organization structure and size this can be a lengthy process stretching out weeks or months.
Try convincing a backend web developer, and his manager, all DB (Database) access functions need to be reprogrammed in a project that’s been around for 7 years; they may decide, a WAF (Web Application Firewall) in front of the web daemon is the best option at that point :-)
They’d now also rely on that WAF for security if that’s the decision!
Such decisions, for whatever the reasons, are not uncommon and often thought of as a reasonable way forward in terms of business logic, the approach is debatable.
What is DDoS Testing?
“Standard DDoS testing”
Layer 4 attacks are normally a high rate of traffic, attacking the network layer. Layer 7 attacks are normally a lesser rate of traffic, targeting the attack towards the application layer.
A DDoS test can mean launching L4 and/or L7 attacks against known IP’s and services.
This is done by launching varying Mbps (Megabits Per second), PPS (Packets Per second) or CPS (Connections per second) of different L4 and L7 attacks against a system, by doing this the reliability of “service continuity” is evaluated under a particular or combined DDoS attack. Most systems without any DDoS mitigation infrastructure or service in place will likely have “downtime”. If you’ve never performed any DDoS testing on your infrastructure, you’ll likely be surprised (disturbed) with results at this initial phase of testing.
“APT DDoS testing”
More advanced DDoS testing can involve a prolonged APT (Advanced Persistent Threat) simulation, similar to groups like “anonymous” or other cyber criminals would do trying to attack an organization. This type of testing is more geared towards organizations with mitigation systems installed.
APT DDoS testing involves the following:
- Reconnaissance phase – Verifying parameters like available IP’s, API’s, mitigation systems in place, various “L7 challenges” and “behavioral algorithms” within the mitigation technology. This phase also to some extent would check the response of the IT staff on the receiving end of an attack (Could be done by launching a limited DDoS attack).
- Attacking phase - This can be a one off or multiple series of attacks attempting to bypass mitigation systems identified. This will be at a specific time during the assessment or at unpredictable intervals. In the latter option it’s likely you’ll be testing a staging environment (Which still may have risk associated with downtime).
Few companies have the knowledge or DDoS testing platform needed to effectively assess your risk when it comes to the APT style attacks in the area of DDoS. Similar to pentesting the quality of such an assessment is based on the DDoS tester’s knowledge of the DDoS arena.
This phase can take anywhere from a few days to a couple of weeks, which would involve customizing and creating various challenge bypasses and targeted attacking tools for the organization being tested.
Why is DDoS testing performed?
If you are an organization which has to be online 24x7 this type of assessment is crucial. Without ongoing standard and advanced DDoS testing, you’re in-fact under false illusions about your DDoS mitigation strategy. When the time comes, you will most likely be taken by surprise if you’re up against a determined attacker.
These situations happen in the largest and best funded organizations, often and in almost all cases can be avoided by APT DDoS testing and system refinements.
DDoS testing clearly illustrates to a security admin or IT manager, the likelihood of a successful DDoS attack campaign, being successful against their network infrastructure and causing “downtime” for the organization.
Without any previous DDoS testing you’ll likely have downtime, and will have to improvise and adjust on the fly your DDoS mitigation systems, this is normally a best case scenario. In a worst case scenario changes will be necessary to your network architecture, application code and the mitigation system’s configuration, while under attack.
As mentioned above the more thorough the testing, the more understanding you have in your ability to mitigate DDoS attacks. The main idea is to avoid downtime when hit by a short or prolonged DDoS attack. You need to know when you’re attacked that the necessary technology and human processes are in place to minimize the possible effects on the production environment. One would also hope to deal with the situation without creating a full blown panic throughout the entire organization and in some cases the public!
In the POC (Proof of concept) phase of a DDoS mitigation system or services being installed and purchased. A neutral 3rd party vendor must verify all claims made by the vendor and the effectiveness of the system being proposed. Various vendors make different claims about their technology. Get those claims verified, so that your environment functions as promised. Don’t wait for a DDoS attack to find out there were misunderstandings and the mitigation system only worked well under a specific attack! You can bet the next attack will be different.
Expert level DDoS consultancy a big part of the equation!
DDoS mitigation systems are often put in as a quick fix (Under a DDoS attack), this will likely change in the near future due to heightened awareness, similar to firewalls in the 90s. However quick deployments can mean extremely complex troubleshooting scenarios arise later on. Troubleshooting network issues with intermittent DDoS mitigation systems triggering can cause seemingly non related issues on your network. This is yet another reason to DDoS test during
“peace time”, you could end up troubleshooting DDoS attacks that don’t exist when mitigation systems affect your environment adversely during a real attack.
There are so many options for DDoS mitigation systems it’s a tough decision knowing which technologies to utilize for “your environment”.
In larger environments it can be a mix of different vendors or CPE and cloud technologies. Testing, coupled with consulting of a vendor neutral company, will assist you in understanding your needs prior to committing to a particular solution.
Some DDoS mitigation systems run into the hundreds of thousands of dollars. It would seem logical that the decision maker purchasing DDoS mitigation systems have a 3rd party explain aspects of the system the vendor may prefer to not mention.
In some cases it may be smart to have a DDoS consultant question the vendor or vendors proposing their respective solutions. Finding out later on, a mistake was made, will be a costly. On the other hand finding issues the vendor forgot to mention, may leave some negotiating room and in some cases may actually mean evaluating an entirely different DDoS mitigation system.
DDoS mitigation systems are very dependent upon what services and architecture you’re attempting to protect.
What happens when infrastructure and application design weaknesses are found with DDoS testing?
Depending on where the weaknesses are found, a large array of parameters would need reviewing. Some of those include:
Volumetric (L4) attack weaknesses - Parameters you need to review:
- Bandwidth –
o How much network volume until my bandwidth is saturated?
o Regardless of CPE equipment in place if bandwidth is saturated you need to have additional protection in place. One option could be a scrubbing center and in some rare cases you may need to increase your bandwidth significantly.
- Stateful devices –
o How did all my statefull devices react during the attack?
o Mitigation systems often have some amount of leakage.
- External routers –
o What is the effect of various DDoS attacks on my router?
o It may be that you had some L4 filtering which turned your router a little too statefull.
- TCP stack on servers –
o Are my TCP stacks of my external facing servers hardened?
o Leakage, even minimal, from a mitigation device or system may be enough to collapse an otherwise powerful server. This can be both on the OS level and in the particular daemon level.
o Hardening the TCP stack is an integral part of the DDoS mitigation strategy.
- Behavioral Algorithms –
o How quickly does your mitigation system identify a volumetric attack?
o Are there false positives when attacks are dynamically identified?
o Do I even have a behavioral system to detect various volumetric attacks?
o How may attack vectors is my system capable of mitigating simultaneously?
o Here you start to understand quickly how effective the behavioral aspects of a DDoS mitigation system are. You may be prepared for a 10Gbit flood but a well planned attack may take a few hundred Mbps to take down the protected daemon being targeted if the algorithm doesn’t perform as expected.
o Tuning and multiple layers of protection may be necessary for your mitigation strategy where volumetric attacks are concerned.
- General infrastructure changes – Changes to network architecture is often a necessity when reviewing weaknesses identified during DDoS testing. Especially when the network is under increased load.
- Configuration tweaking – DDoS mitigation systems in place should be tweaked and configuration changes made as needed.
Application (L7) attack weakness - Parameters you may need to review:
- API’s in use –
o Do you have API’s in use on your website or servers?
o If so, are the API’s protocols and implementation protected by the DDoS mitigation system in use?
o By default, if not, you should understand limitations and also understand how to protect proprietary services.
- L7 challenges in place –
o How easy are your layer 7 challenges to bypass?
o What services have such effective L7 challenges in place, HTTP, HTTPS, SMTP, DNS, SIP etc..?
o If there is no challenge for a particular protocol how do you mitigate a DDoS attack targeting the specific service?
o If you have HTTP challenges how easy are they to bypass with a determined attack?
o Is a captcha sufficient as a last line of defense or even feasible for your environment?
o With the “captcha” in place what is my level of protection?
o Are there other options assuming all L7 challenges fail?
o L7 challenges may also affect your production flow when triggered. The effects can be zero to severe, in some cases may contribute to the DDoS on your environment.
o It is important to understand how challenges work both with HTTP and HTTPS.
o If HTTPS is implemented on the site protected, how is detection and mitigation effected by a DDoS attack?
- Severe logic flaws –
o When Applications are attacked, do logic flaws that normally would not lead to downtime, become a bottleneck in the flow of your overall system architecture?
- Business logic flaws –
o When it comes to DDoS mitigation, are you protecting the correct services and devices in your network?
o In just one example, taking out a backend DB leaving the rest of the site up and running can effectively stop service for all customers.
- Configuration tweaking – Again for layer 7, DDoS mitigation systems in place should be tweaked and configuration changes made as needed.
The Best of Both Worlds
Pentesting checks the ability of an attacker to exploit your network and gain access to data. DDoS testing attempts to render systems unavailable or specific mitigation systems redundant (IPS/Firewalls etc.).
A relevant question is, “I have regular pentests, do I really need DDoS testing?”, The likely answer to that is “yes” however the frequency and depth of DDoS testing would depend on your environment. This analogy would be similar for pentesting with regards to depth.
It’s also important to understand that today’s APT’s (Advance persistent threats), will more often begin to use a combination of exploits, client side and DDoS as well as other targeted attacks when trying to gain access to your network data.
The reason a combination of both targeted exploits and DDoS attacks are attractive is due to the fact, a DDoS attack may be deployed as a distraction. In some cases a well targeted DDoS attack will be used to knock out any stateful devices configured to go into a “fail open” mode. This will assist in an exploit being run against a specific daemon or service while inspection of FW’s, IPS (Intrusion prevention systems) and WAF systems are down.
Contemplate, when your firewall, IPS, WAF etc. are in “fail open”. Previously hidden and protected vulnerabilities are now no longer hidden or protected. The previously protected service may become vulnerable to exploitation at that point.
- How many pentesters verify DDoS aspects when performing pentests?
- How many DDoS tests verify pentesting aspects?
- Should pentesters verify DDoS thoroughly?
- Are the pentesters able to verify DDoS mitigation systems on a good level?
- How many managers are willing to place all stateful devices to a “fail closed” state without a bypass option? This annoying little detail is well known to a seasoned attacker….
Practically, it may not be feasible to combine the two tests, due to the fact it would require an excessive amount of coordination, combined with significant disruption such a hybrid test would entail.
This doesn’t mean that pentesting or DDoS testing should be avoided; rather DDoS testing should be done separately. Troubleshooting of each DDoS security weakness will be less complex and easier to resolve.
The skill set for each test is entirely different and most DDoS testers would not necessarily be good pentesters and vice-versa.
Regular DDoS testing and pentesting should be an integral part of most medium and enterprise organizations threat assessment cycles.
The sheer volume of cyber attacks, combined with the persistence and sophistication of attacks, are complex enough to understand under normal circumstances. When a DDoS attack is thrown into the picture the complexity rises and security and IT staff need to understand their risk level as well as how to react when an APT targets their organization.
It’s well worth investing the time and money in both arenas; many organizations have a mix of new as well as legacy and outdated technologies that are relied upon for serving the organization. Use a vendor neutral testing service to verify your cyber security posture when verifying both data breaches with pentesting or DDoS susceptibility through DDoS tests.
With increasing threats and new testing methodologies on the rise, if organizations ignore either aspect the price to pay could be catastrophic. Avoid your organization becoming another statistic or headline.
Prepare your organization realistically by verifying, validating and educating yourself and your staff on how to react to both DDoS attacks and exploitation attempts.
The era of APT’s has begun and isn’t disappearing any time soon, make sure you’re protected!