Patching is important, but let's agree it takes time. It takes time to test & validate the patch in your environment, check the application compatibility with the software and the underlying services. And then, one fine day, an adversary just hacks your server due to this un-patched code while you are testing it. It breaks my heart and I wonder "what can be done in the delta period while the team is testing the patch"? Adversary on the other hand is busy either reversing the patch, or using a zero-day to attack the systems! I mean once a patch is released it's a race,
Either bad guys reverse it and release a working exploit, OR good guys test, verify and update their environment. A close game, always.
Technically, I wouldn't blame the application security team, or the one managing the vulnerable server. They have their SLA to apply updates on the OS or Application Servers. In my experience, a high severity patch has to be applied in 15 days, medium in 30 days, and low in 45 days. Now, if the criticality is too severe; it
can should be managed in 24 to 48 hours with enough testing on functionality, compatibility, and test cases with application team; or server management team. Now, what to do when there is a zero-day exploit lurking in your backyard? It used to be a low-probability gamble, but now it's getting more realistic and frequent. The recent case of Apache Struts vulnerability has done enough damage for many big companies like Equifax. I already addressed this issue in a blog-post before, and the need for alternatives such as WAF in Secure SDLC.
What shall I do if there's a 0-day lurking in my backyard?
Yes, I know there's a zero day for your web-application or underlying server, and you are busy patching but what other security controls do you have in place?
Ask yourself these questions,
- Do I have understanding of the zero-day exploit? Is it affecting my application, or a particular feature?
- Do I have a product/ tool for prevention at the application layer for network perimeter that can filter bad requests - Network WAF (Web Application Firewall), Network IPS (Intrusion Prevention System) etc.?
- Do I have a product/ tool for prevention at the application layer for host - Host based IPS, WAF etc.
- Can I just take the application offline, while I patch?
- What's the threat model and risk appetite if the exploitation is successful?
- Can I brace for impact by lowering the interaction with other components, or by preventing it to spread across my environment?
Let's understand how these answers will support your planning to develop a resilient environment,
>> Understanding of the zero-day exploit
You know there's an exploit in the wild; but does your security team or devops guys take a look at it? Did they find the exploit and understood the impact on your application? It is very important to understand what are you dealing with before you plan to secure your environment. Not all exploits are in scope of your environment due to the limitations, frameworks, plugins etc. So, do research a bit, ask questions and accordingly work on your timelines. Best case, understand the pattern you have to protect your application from.
>> Prevention at the application layer for network perimeter
If you know what's coming to hit you, you can plan a strategy to block it as well. Blocking is more effective when it's at the perimeter - earlier the better. And, if you have done good research on the exploit, or the threat-vector that can affect you; please take a note of the pattern and find a way to block it at the perimeter while you patch the application.
>> Prevention at the application layer for host
There are sometimes even when you know the pattern, and the details on the exploit but still network perimeter is incapable of blocking it. Example, if the SSL offload is on the server/ load balancer. In this case make sure the server knows what is expected; blocks everything else including an anomaly. This can be achieved by Host based protection: IPS, or WAF.
Even a small thing like tripwire can monitor the directory, and files to make sure attacker is either not able to create files; or you get the alert at the right time to react. This can make a huge difference!
Note: Make sure the IPS (network/ host) is capable of in-depth packet filtering. If the pattern can be blocked on the WAF with a quick rule, do it and make sure it doesn't generate false positives which can impact your business. Also, do monitor the WAF for alerts which can tell you if there have been failed attempts by the adversaries. Remember, the attackers won't directly use their best weapon; usually it starts with "information gathering", or uploading files, or executing known exploits before customizing the case for their needs.
You have very high chances to detect adversaries while they are gathering insights about you. Keep a keen eye on any alert from production environment.
>> Taking application offline
Is it possible to take the offline while you patch the software? This depends on the fact what's the exposure of the application, what is the kind of CIA (Confidentiality, Integrity and Availability) rating and what kind of business impact assessment has been performed. If you think that taking it offline can speed up the process, and also reduce the exposure without hurting your business; do it. Better safe than sorry.
>> Threat model and risk appetite
You have to assess & perform threat modeling of the application. The reason it is required is not every risk is high. Not every application needs the same attention, and the vulnerable application may well be internal that will substantially reduce the exposure and underlying impact! Do ask your team - is the application Internet facing, how many users are using it, what kind of data is it dealing with etc. and act accordingly.
>> Brace for impact
Finally, if things still look blurred, start prepping yourself for impact. Try to minimize it by validating and restricting the access to the server. You can perform some sanity checks, and implement controls like,
- Least privilege accounts for application use
- Least interaction with the rest of production environment
- Restricted database requests and response to limit the data ex-filtration
- Keep the incident management team on high-alert.
Incident management - Are you sure you are not already breached?
Now, what are the odds that while you reading this blog, trying to answer all the questions and getting ready - you haven't already been compromised? Earlier such statement of incidents used to begin with "What if..." but now it says "When..." so, yeah make sure all your monitoring systems are reporting the anomalies and someone is monitoring it well. These tools are only good if some human being is responsibly validating the alerts. Once an alert is flagged red; a process should trigger to analyze and minimize the impact.
Read more about incident monitoring failures in my earlier blogpost. Don't be one of them.