Upside down pyramid, why IT patching needs reform.
Get your checklist ready, send out communications, make sure we have a call tree ready and make sure we have application teams on standby to verify their systems. This sound familiar to anyone out there? It is the makings of a monthly patch cycle that most of us are familiar with, if you're not, you're the lucky one. Interesting enough patching does not make IT simple, patching adds a complexity to IT which involves millions of dollars of investment and resource overhead cost. I was raised in the dawn of the “Information Technology Golden Age” and was indoctrinated to always believe that IT should make work easier not harder.
That's the point, we should be working smarter not harder. As I think about the patching process many of you out there would agree that the patching process is excessive but necessary. Why does our cost have to be so high for something that should already be factored into our business? The one word that comes to mind is “Security”. Many of you have seen the growth of the security sector within the IT field. According to Bloomberg it costs 57,000 dollars to support security for 50 people in an organization. That means that 1,140.00 per person is what the average organization must pay to ensure that their environment is secure which tends to pose the question, what exactly are we paying for?
For data centers this question is very one sided, Organizations are now into the business of securing their data. Data has become the currency of the world, my data gets compromised and I lose business. So how does this all fit into the cost? Simply put, support teams must ensure that hardware and software are consistently maintained and updated which is commonly referred to as “Patched”. Patching allows all the software we use the ability to address security vulnerabilities by pushing updates and getting rid of the potential holes that exist; sounds really good right? Well, that all depend on who you ask, developers will tell you “No”, Engineers will tell you “Yes”, Customers will tell you “No” and Security will tell you, “you have to”. Now you start to see the problem, different stakeholders require different needs.
These needs differ based on the priorities of the Agency, Security Organization and Business Line, more importantly these needs are not aligned strategically to drive business effectively. So the question then becomes how to align business strategy across the different stakeholders especially when stakeholders have different strategic missions? Recently I attended GSA IT C2E which is a collaboration of vendors who perform work at the General Services Administration (GSA) and have created this event to discuss ways that they can improve efficiency within their customer space and broker better lines of communication amongst vendors. This same topic was one the main focal point of the evening and many vendors vocalized ideas and strategies which addressed their particular areas of focus and provided amazing alternatives to this problem. The major point we all agreed upon is that many of the maintenance activities we perform are driven solely based on the pressure to secure systems. Hacks from China, internal threats, new technologies, shellshock attacks, terrorist and many more cyber threats put the government in a state alarm. This can be understood especially when the government maintains a steady pool of personal data that could compromise individuals lives.
So then what’s next? It sounds like I am justifying the absorbent cost that we pay for securing systems and avoiding the communication gap. It’s time for an evolution in IT security and Business Automation that will streamline the way we do business today. Currently patching schedules are driven to ensure that all systems are patched within a 30 day window, which includes developing changes, testing the changes and then promoting the changes For massive systems this is just impossible. Developers require time to ensure that all dependencies in systems scale and minimize the number of changes that occur to an application in order to reduce the overall impact to the business line users. Based on the aggressive schedule teams will either take shortcuts or risk the security of the system to buy themselves more time to develop work arounds. So where does this new evolution start? DevOps, it has been around for many years but proved expensive in its early years with all the up front costs to build out the virtualized systems. With Cloud systems DevOps becomes literally inexpensive in comparison to prior years, you can now programmatically build systems without the need of having multiple full time engineers support