A Business Technology Place

The Yin and Yang of Security Patching


My computer is working, don’t change anything.

As an IT manager I observe this behavior regularly with end-users and product managers of eCommerce applications. It’s understandable. When a computer system is working and doing its job then “updates” are sources for creating failure. Updates change code. Updates rock the boat.

If a computer security update hasn’t bitten you yet, then it’s probably just a matter of time. My experience is the number of system issues related to operating system updates is growing.  It’s hard to test all the dependencies of code updates against every combination of hardware and software that exists on computing equipment. A couple of examples I can point to in 2017 are Microsoft Edge no longer working after installing the Windows 10 Creators Update.  Then there was the issue of Microsoft Outlook unable to open attachments which was later resolved with another hot fix.  

But we all know security updates are necessary. Why would we risk our personal data to thieves? In a business setting, why would put our customer’s data at risk? Why would we put the reputation of our business at risk?

Therein we find the yin and yang of security updates. We don’t want to upset the balance of a stable system, but we need to update the system so that it can remain stable in the future.

In the name of audit controls and security principles.

In the business environment, audit standards require staying up-to-date with security patches. ISO 27001/ISO 27002 and SOC2 have controls specifically addressing vulnerability patch management policies and procedures. To meet the requirements of the controls, a discipline in process and procedure is required.  These standards are there to help nudge all of us to change because we all know we resist change.

Plug those security gaps or face the consequences.

The consequences of not installing security patches can be devastating. In the worst case of cyber theft reported thus far, Equifax was robbed of information for 143 million individuals. The attackers found a weakness because Equifax failed to patch a known security vulnerability in website code they use.

Now hundreds of millions of people are exposed to the whims of criminals. The reputation of a large credit bureau is blown. The two highest ranking security officials within Equifax are out of a job. Patching known security vulnerabilities is serious business.

Complementary forces at play.

The next time someone schedules a security update for a system or application, understand the potential consequences fully. Intruders are at the gates. They make a living on our resistance to change.  But if we support the change and work with administrators to report any malfunctions, we can all help to build a safer tomorrow.  That’s how another yin and yang can make a more complete whole.

Onward and upward!

2 Server Patching Approaches

Server patching is one of those infrastructure tasks that most business owners and infrastructure managers don’t link to the eCommerce portfolio. Operating system patches, virus scanner updates, database patches, etc. are all system level software not directly related to the code line of the eCommerce team. Infrastructure managers can certainly create a patching strategy independent of the eCommerce development process, but I like two methods that keep the current version of eCommerce code in the forefront to help minimize risk and make the patching process more efficient.

1. Follow code line promotions

The premise behind this approach is to install and promote server patches in same sequence of non-production environments as the code line increments. So for example a shop might have an environment sequence for code something like this:

Development ⇒ QA ⇒ Beta ⇒ Production

Let say there is a code release version 2.2 that is currently in development. Then as version 2.2 is deployed in sequence to each environment the infrastructure manager would additionally apply all new patches.

The advantage to this approach is that as the code line is tested in each environment the patches are part of that testing. This prevents the team from having a duplicate testing event just for patch validation and should provide a more thorough set of testing before production deployment. Testing patches for potential system incompatibility is always the biggest risk to production deployments. So this approach tries to mitigate the risk by having it tested by various groups in different environments.

A disadvantage to this approach is that the need for server patching may arise independent of the release schedule. Emergency vulnerability patches may arise due to a hacker exploit or some other industry event.

2. Snapshot and test

The idea of this approach is to make a copy of the environment that you will patch and then install the patches to the copied environment. The test team will then validate that there are no code incompatibilities by testing against the new environment. Once approved, the infrastructure manager can promote the patched environment to production or patch the original production environment.

The advantage to this approach is that it can run at intervals independent of code releases. That could be useful if there is an emergency vulnerability patch and the next code release isn’t scheduled for production deployment for another four weeks.

The disadvantage to this approach is that it requires more administrative overhead and management. Depending on how it’s executed you could duplicate tasks. Additionally it creates a separate testing event for the validation team that is independent of the normal test validation associated with code installs.

In practice, it may be best to use both approaches. Following code line promotions is a risk mitigation approach that doesn’t duplicate testing efforts and would work for a routine patch update schedule. The snapshot and test approach works for off-cycle patches that have a more urgent requirement for deployment.