By now you have probably read some articles about the Meltdown and Spectre vulnerabilities but you may still be seeking guidance for how your organization should react.
First a quick recap, Meltdown and Spectre were announced early in January of 2018. Unlike most other vulnerabilities, Meltdown and Spectre exploit critical vulnerabilities in modern processors. Meltdown primarily affects Intel chips and it allows any application to access all system memory, including memory allocated for the kernel. Spectre allows an application to force another application to access arbitrary portions of its memory, which can then be read through a side channel. Spectre impacts Intel, AMD, and ARM chips.
The statements about the vulnerabilities implies that in some circumstances the security boundary between virtual machines on a common hyper-visor may not be absolute. In theory, the memory contents of virtual machines running on an unpatched physical host could be read by other virtual machines on the same physical host or read by the physical host.
Various operating system vendors have been issuing security updates to help mitigate the vulnerabilities. Also some vendors have issued security updates for specific applications. Unfortunately, none of these offer a long term general solution to the vulnerabilities since they exploit the design of modern processors. With this in mind it is fair to expect future exploits that will use the same vulnerabilities in new ways that circumvent the mitigations in place.
What’s Intel Doing
Intel has published microcode updates to address their processors, however, reboot issues have been reported and at the time of writing this post, Intel has recommended that OEMs, cloud service providers, system manufacturers, software vendors and end users stop deployment of its most recent patches. Intel expects to release new patches in the near future.
Performance issues have been reported for systems with Spectre and Meltdown mitigations in place. Performance hits ranging from 0% to 30% have been published. PostgreSQL servers in AWS have been identified as taking a 12-17% reduction in performance.
It is critical to test the security updates for these vulnerabilities in test environments before promoting the updates to production systems. It is also important to plan a response if performance is impacted and affecting service level agreements with customers.
SystemExperts recommends that mitigation plans address a number of different areas of concern:
- Systems in public clouds
- Systems in private clouds
- Physical servers
- Employee endpoint devices (workstations, laptops, smartphones, and tablets)
- Security Awareness
- Evaluating third party service providers
- Communicating with customers
Organizations that use public cloud services should start by determining what they need to do to secure their cloud environment. Amazon AWS, Microsoft Azure, and Google all had some lead time and they had all of their hyper-visors patched shortly after the initial vulnerability announcement. However, organizations that manage virtual machines in these environments still need to apply patches to the machines they manage. Check with the operating system vendor and thoroughly test any updates before promoting the patches to production environments.
Many other smaller public cloud services did not necessarily have a head start on patching their hyper-visors before the vulnerability announcement was made. Customers of these organizations need to reach out to their cloud provider to understand what has been patched, or when will all relevant physical servers be patched.
Even if an organization operates a private cloud that doesn’t enable any external party to run arbitrary software on the servers – they should evaluate their systems. The security of all of the virtual machines might not be equivalent. For example, some large private clouds might have a variety of server configurations deployed. A server to monitor environmental controls might grant remote access to an HVAC contractor, but the virtual machine might be on the same hyper-visor as database server that contains highly sensitive data or a file server that contains confidential customer data. Organizations operating private clouds should prioritize patching their hyper-visors.
Organizations with a small virtualized environment may also be at risk. Consider a small organization that has a three tier system consisting of a web server in a DMZ, a database server in subnet that can only be accessed by the webserver in the DMZ and system administrators from the internal network, and an internal server that is not directly accessible from either the Internet or the DMZ. If these servers were all on the same hyper-visor and the DMZ server were breached, it might be possible to read the memory contents of the servers in the other network tiers.
Physical Machines (servers and endpoints)
Of course, while organizations should likely prioritize updating their hyper-visors, they must not forget about testing and applying the patches to all other physical servers, desktops, laptops, smartphones, and tablets as the patches for the various operating systems and microcode updates for processes become available.
Network appliances that run Linux, Windows, or other general purpose operating systems may also be impacted by the vulnerabilities. Organizations should review their inventory of network appliances and closely monitor vendor security bulletins to learn when security updates are available, learn of any configuration work arounds, or learn if the vendor believes the appliance is vulnerable.
SystemExperts recommends that organizations notify employees and contractors about the Meltdown and Spectre vulnerabilities. Remind staff how to report suspected security incidents, and what to do if a customer asks what the organization is doing to mitigate the vulnerabilities.
Third Party Service Providers
Many companies now rely on SaaS, PaaS, and IaaS provided by third parties. Organizations should review all of its third party service providers, identify the risks, and determine what the vendors are saying about their mitigation strategies.
As an example, Box stated on January 8, 2018 that, “Box is applying patches where relevant to our infrastructure. At this time, we believe the Box service is not directly impacted, and we assess the risk as low. Though the underlying CPU and OS combination in our infrastructure may be affected by these vulnerabilities, the Box service is a closed system that does not allow customers to run custom code against our underlying infrastructure.”
Each organization will need to assess what the service providers say against its own perception of the risk and determine how it would like to proceed. Organizations should consider enabling additional security options that it has chosen not to employ in the past, or it may decide that the time has arrived to seek an alternate service provider.
Communicating with Customers
Organizations should craft its message about what it desires to communicate to its customers in response to the Meltdown and Spectre vulnerabilities and communicate to all employees exactly who can response to customer inquiries and how to refer customers to the correct information or contact point. Ideally, the response should indicate that organization has a depth of knowledge, has evaluated the risk and has taken actions to remediate the risks in a timely manner.
Paul Hill has worked with SystemExperts as a principal project consultant for more than twelve years assisting on a wide range of challenging projects across a variety of industries including higher education, legal, and financial services. He joined SystemExperts full time in March 2012 and coordinates the SMARTday practice.