PATCH MANAGEMENT BEST PRACTICES - Ecora

54 downloads 341 Views 2MB Size Report
The Fundamentals of Patching – From a Technical and Historical Perspective ...... meant not applying software patches in a piecemeal manner, but waiting for ...
PATCH MANAGEMENT BEST PRACTICES

A “How to” Guide for Securing the Enterprise

AUTHORS:

Anne Stanton President, Norwich Group

Susan Bradley Microsoft Small Business Server MVP

The Fundamentals of Patching – From a Technical and Historical Perspective

Table of Contents

INTRODUCTION ........................................................................................................................................ 3 WHAT IS PATCH MANAGEMENT?....................................................................................................... 4 HISTORY OF PATCHING ............................................................................................................................... 4 DEFINITION OF MICROSOFT PATCHES ......................................................................................................... 4 WHAT IS A PATCH? ..................................................................................................................................... 5 IDENTIFYING THE FLAW .............................................................................................................................. 6 WHY DO WE PATCH?................................................................................................................................... 6 WHAT IS INCLUDED IN A MICROSOFT PATCH? ............................................................................................ 6 HISTORICAL PATCH PROCESS WINDOW ...................................................................................................... 9 FINDING OUT ABOUT PATCHES.................................................................................................................... 9 THE ROAD AHEAD.................................................................................................................................... 11 CHANGE MANAGEMENT AND RISK ANALYSIS ............................................................................ 13 RISK AVOIDANCE – AVOID PATCHES ALL TOGETHER ................................................................................ 13 RISK AVOIDANCE – IF THEY’D ONLY WRITE BETTER CODE ........................................................................ 14 RISK ACCEPTANCE – THE REAL WORLD .................................................................................................... 14 PATCH POLICIES AND CHANGE MANAGEMENT POLICIES ........................................................................... 15 IDENTIFYING YOUR CHANGE MANAGEMENT TEAM ................................................................................... 16 OUR ASSUMPTIONS ................................................................................................................................... 17 SETTING UP A PATCH TESTING SCENARIO ................................................................................................ 17 DISCOVERY OF ASSETS ............................................................................................................................. 18 NECESSARY PATCHES VERSUS ALL PATCHES ............................................................................................ 20 SETTING UP THAT TEST LAB - UTILIZING A VIRTUAL ENVIRONMENT ......................................................... 20 THE BOTTOM LINE .................................................................................................................................... 21 IT’S ALL ABOUT RISK ........................................................................................................................... 23 THE ADMINISTRATOR’S ROLE ................................................................................................................... 23 IDENTIFY KEY FACTORS ............................................................................................................................ 24 CAN YOU PATCH? ..................................................................................................................................... 24 HANDICAPPING THE PATCHES .................................................................................................................. 25 CASE STUDY—BLASTER ARRIVES ............................................................................................................ 26 RESOURCES FOR THREAT ASSESSMENT .................................................................................................... 30 ADDITIONAL TESTING PROCEDURES ......................................................................................................... 31 TESTING PROCEDURES MAY INCLUDE EFFECTIVENESS TESTING ................................................................ 31 STANDARDIZE YOUR OFFICE ..................................................................................................................... 33 TESTING PROCESSES ................................................................................................................................. 33 ARE WE READY TO PATCH YET? ...................................................................................................... 36 PATCH “AT RISK” MACHINES FIRST ........................................................................................................... 36 USE PATCH TOOLS TO CONFIRM COMPLIANCE WITH POLICIES ................................................................... 37 CAPTURE FEEDBACK................................................................................................................................. 38 ENFORCEMENT ......................................................................................................................................... 38 SAMPLE TEST LAB SECURITY POLICY....................................................................................................... 39 TECHNICAL ENFORCEMENT ...................................................................................................................... 42 SET THE POLICY TO MAKE IT EASY TO UPDATE AND COMPLY WITH........................................................... 44 STUFF HAPPENS ........................................................................................................................................ 44 BE WILLING TO GATHER INFORMATION .................................................................................................... 45 CHANGE MANAGEMENT FOLLOW-UP ....................................................................................................... 45 SO I CAN’T PATCH, NOW WHAT DO I DO? .................................................................................................. 46 THE STORY OF TWO PATCHES ................................................................................................................... 47 Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

1

THE COMING THREATS .............................................................................................................................. 48 THE BOTTOM LINE FOR YOUR BOSS ........................................................................................................... 49 LIFE BEYOND PATCHING MICROSOFT........................................................................................... 51 PATCHING THIRD PARTY APPLICATIONS ................................................................................................... 52 PATCHING LINUX DISTRIBUTIONS ............................................................................................................. 58 BUILDING PACKAGES FROM SOURCE......................................................................................................... 66 DEPENDENCIES ......................................................................................................................................... 66 SUN SOLARIS ............................................................................................................................................ 67 JUSTIFYING PATCH MANAGEMENT—A BUSINESS TUTORIAL............................................... 69 MANAGEMENT AND PATCH MANAGEMENT .............................................................................................. 69 AN OFFICER’S PERSPECTIVE ..................................................................................................................... 70 INDUSTRY REQUIREMENTS/STANDARDS .................................................................................................. 75 COMPLIANCE WITH ISO900X ................................................................................................................... 81 COMPLIANCE WITH INDUSTRY ASSOCIATION STANDARDS AND REGULATIONS ......................................... 81 RETURN ON INVESTMENT (ROI)............................................................................................................... 82 WHAT DO WE REALLY NEED? ................................................................................................................... 84 DEFINING THE PATCH MANAGEMENT BEST PRACTICES—FIRST ROI VARIABLES PROJECT..................... 85 ONGOING MAINTENANCE AND MODIFICATION—SECOND ROI VARIABLES PROJECT ............................... 85 WHEN A BREACH OCCURS – BUSINESS CONTINUITY—THIRD ROI VARIABLES PROJECT ........................ 86 FINDING TIME IN A CRISIS MANAGEMENT ENVIRONMENT ......................................................................... 87 PURCHASING PATCH MANAGEMENT TOOLS .............................................................................................. 87 VENDOR ISSUES........................................................................................................................................ 88 CONCLUSION ............................................................................................................................................ 89 APPENDIX A– PATCH TESTING .......................................................................................................... 90 APPENDIX B—SAMPLE BUDGET REQUEST ................................................................................... 91 GLOSSARY ................................................................................................................................................ 92 NOTES ........................................................................................................................................................ 93

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

2

Introduction ver a late January weekend in 2003, ATMs for well-known financial institutions became unavailable, flights were canceled due to online ticketing and electronic check-in problems, and a major city’s emergency 911 network went down. These were just some of the issues that caused a crisis throughout the world and sent IT staffs scrambling to figure out what was happening.

O

Pretty soon, we all learned what the culprit was. “Computer worm hits the net,” screamed the headline. SQL Slammer, Sapphire, W32.Slammer whatever you want to call it, was tiny as worm files go, only 376 bytes of code designed for speed. Typical computer transactions involve a “hand shake” transmission process. One party offers a connection, another party accepts, and the transmission proceeds by means of traditional TCP/IP processes. SQL slammer, however, used another transmission standard. It transmitted UDP packets only, through a connectionless transmission. This worm did not wait for a response. It flooded all vulnerable connections it could find. That high-speed little file was looking for a port that had behind it, ready and waiting, a listening application. The Internet Assigned Numbers Authority (IANA.org) maintains the listing of computer ports used by programs, applications, and typical connections on the Internet. A computer system has almost 65,000 ports to transmit information back and forth. Typically, these ports sit there waiting, but sometimes they are in “listening” mode, waiting to be called upon. Most worms would typically try to find vulnerable systems on well-known ports, those ports from 0 to 1023. SQL slammer was different. It aimed at a port not used in previous attacks, port 1434. This is a port used by database programs such as Microsoft SQL server and something called MSDE or Microsoft SQL Database Engine. Microsoft SQL Server is a very powerful database program typically run on maintained and monitored servers. The other, MSDE, it a small but powerful database engine used by developers in many applications. Furthermore, port 1434 is unique. It is not a port used to transmit data; rather it monitors SQL transmissions. All Microsoft SQL servers listen on this port. Not all MSDE installations do however. The tiny worm’s tale includes a couple of other twists. Developers use MSDE in many applications, but do not necessarily tell purchasers that their software uses MSDE to keep track of something needed for the application’s operation. At the time Slammer struck, patching SQL server or MSDE was difficult and cumbersome, needing the patch installer to understand SQL instances. While the original patch to fix the vulnerability came out in July of 2002, the rollup service pack had just recently come out about three weeks before Slammer appeared. In early 2003, three weeks was not enough time to have server admins or application developers test and install a service pack. Database administrators were (and still are) reluctant to install patches on working databases, so installing the rollup pack was not a priority. Furthermore, all the unpatched computers equipped with MSDE were primarily in installations where the administrator had no idea that he or she had MSDE installed. Their software vendors had not informed them, nor did they have a tool to identify machines that were running MSDE. Thus, the stage was set for the worm: unpatched machines, unidentified machines that were also unpatched, a worm built for fast connections, a port never used before for worm attacks, and a port not used for data, only monitoring. Overall, we had a “perfect storm” for unleashing the worm. The news reports indicated that the major ISPs and backbone providers of the Internet knew within minutes that something was up. Realize that unlike Code Red that took 24 hours to go around the world infecting the globe, SQL slammer was around the world in 30 minutes.1

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

3

Chapter 1 – What is Patch Management?

W

e define patch management as “the act, manner, or practice of managing, handling, supervising and controlling the application of code to software in order to fix a bug, especially as a temporary correction between two releases.”

Many in our industry consider that patch management means applying security fixes that are typically not temporary corrections. In fact, patch management and its processes include the entire area of introducing “new code” into an existing environment. While this document focuses primarily on tools used for testing, supporting, and evaluating Microsoft security patches, the basic concepts are the same for all software systems. IT managers and admins must logically control the introduction of new code into a network. Computers drive world business. No longer can a financially healthy firm exist without technology. Further, maintaining, protecting, and securing computer systems is, as recent legislative initiatives make clear, simply good business. A computer, like any mechanical device, requires regular maintenance. Applying patches to systems that connect to the Internet is simply and fundamentally mandatory. Period This document focuses only on the processes and procedures for patch management. We assume that you have taken steps to harden systems,2 apply firewalls, and install antivirus protection. Patch management is only one of the steps needed to protect an environment, but it is an essential element nonetheless. We also assume that you understand the “business” reason for patching and that you have the proper buy-in from management to add patch management to your security processes. Some security vendors recommend installing the bare minimum of patches; however, that still means that you must rely upon consistent and appropriate processes and procedures. History of patching The CERT Guide to System and Network Security Practices3 has small sections on software patches. The author, Julia Allen recommends applying patches on redundant or duplicate systems before applying them to main production machines. Any vendor service level agreement (SLA) should clearly state that firewalls should remain in place during installation of operating system patches. Allen further argues that those in charge of firewall operating systems and software need to review those devices as well for updates. Not too long ago, “best practices” meant not applying software patches in a piecemeal manner, but waiting for the cumulative security patch that, presumably, was more tested. Now, delay and deferral invite crisis. Definition of Microsoft patches In the Microsoft world, patch management included all of the following types of new code introductions:4 • Critical Update – this is not a security update but a fix for an issue in broadly applied software. It is publicly available and has an accompanying knowledge base article. • Driver Update – this updates software that supports and controls hardware. Driver updates may come from either the software vendor or the hardware vendor. • Feature pack – software package that includes non-critical additions to the base software program. It typically appears between major releases. • Hotfixes – patches built to address specific issues. Recipients may not distribute hotfixes outside their organizations without written authorization from Microsoft. Get them FREE by calling Microsoft Product Support Services. Hotfixes do not receive the regular testing

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

4

process. • Security Update – this is what we traditionally refer to in patch management circles. There is a severity rating included with each update, as is a security bulletin discussing the issue and a Knowledge Base article describing the patch in detail. • Service Packs – are cumulative packages of hotfixes, security updates, critical updates, and updates. A service pack undergoes both internal and external testing. • Software Update – is any update, update rollup, service pack, feature pack, critical update, security update, or hotfix. • Update – addresses a non-critical, non-security issue. • Update rollup – cumulative package of hotfixes, security updates, critical updates, and updates, collectively tested for easy deployment. • Upgrade – this software updates and upgrades an application to a newer version while keeping the settings and data from the prior program.

Each of these categories requires the same process and procedures of testing, acceptance, and management sign off. Each must go through a process no matter the size of an organization. We will spend the bulk of our time in this document on security updates. What is a patch? Software refers to the instructions mechanical devices receive to process commands in a certain manner. Typically, developers write software to perform a process in a prescribed fashion. However, as with any process deriving from human intelligence, there can be incorrect assumptions made, and the software might not perform as intended. As Bruce Schneier and Niels Ferguson state in Practical Cryptography – Most engineers have to contend with problems like storms, heat, and wear and tear. A bridge designer only has to worry about three threats – water, gravity and wind. All of these factors affect designs, but their effect is predictable to an experienced engineer. (This is) not so in security systems. Our opponents are intelligent, clever, malicious, and devious; they’ll do things nobody had ever thought of before. They don’t play by the rules, and they are completely unpredictable. That is a much harder environment to work in.5 A software designer has to worry about threats from an unlimited number of vectors. This means that the network administrator must consider such risks from these same threat vectors when analyzing a network’s need for patches. Can the threats come from outside the organization or only from inside? What potential attack method might exploit a particular flaw? In a later chapter, we discuss resources for determining these threat vectors. Software flaws threaten an organization in various ways. The most critical give an attacker full rights to a system. Many of these flaws stem from buffer overflows. Traditionally stack-based buffer overflows have been the largest category of security issues, and are places in the software where more data enters the system than the software is asking for. If the software designer did not anticipate this, the system would “crash.” Perpetrators target buffer overflows to dump the processes to ensure that the system remains at an “administrative privilege” level, thereby forcing the system into “handing over the keys to the kingdom.” When a patch bulletin indicates that the worse case scenario is that the attacker can “run code of his choice,” this is the equivalent to that person logging in as administrator to a system. Software patches themselves are also threats. After each security bulletin release, even with the rigorous testing done by the vendors on those applications typically broken by previous security updates, issues still occur. Documentation accompanying a security update addresses any known issues likely to develop following release of the update. Remember that issues directly Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

5

caused by security updates qualify for a no-charge support call. In the United States, call 1-866PC-SAFETY.To resolve International issues contact a local Microsoft office.6 Testing patches, ensuring that they do not adversely affect systems and that they protect systems as intended, as well as applying patches, requires approval from management and may even require approval from critical line of business vendors. We discuss change management processes later and in detail. Nevertheless, you need adequate resources if these testing, ensuring, and applying processes and procedures are to work correctly. Identifying the flaw Very briefly, software vulnerability begins when someone looks at code or attempts to reverse engineer code. While Linux and its variants work under the Open Source licensing model wherein the source code accompanies the product, Microsoft does not release its source code. Thus, many security researchers use various techniques to identify flaws. In some cases, the interaction and connectivity needed between a UNIX or Linux system may provide clues to a researcher for potential flaws. Other freely available tools include Dave Aitel’s SPIKE (www.immunitysec.com), Todd Sabin’s DCE-RPC tools (razor.bindview.com), Netcat (www.atstake.com/research/tools/network_utilities), Ethereal (www.ethereal.com), and many of the utilities at the Sysinternals web site (www.sysinternals.com). If the researcher or security company agrees to responsible disclosure techniques, it contacts the software vendor ahead of time and allows the vendor to correct the flaw or respond to the issue. Eeye.com7 is one vendor that notifies but does not disclose a flaw publicly at any time prior to the release of a security update. Irresponsible vendors and security researchers post the vulnerability to listserves such as Full Disclosure8 along with information that provides a “proof of concept” that has later been exploited by others and turned into automated exploits. Once contacted, the vendor reviews the reported information to see if the issue truly is a security flaw. If there is an issue, the process of building the patch and testing the patch begins. Why do we patch? It is obvious that we patch because software is not processing commands correctly. This misprocessing could range from elevation of privilege to information disclosure. Threat Modeling, a text that explores what an adversary might attain by exploiting a flaw defines the following threat categories9 • Spoofing identity • Tampering with data (also called integrity threats) • Repudiation • Information disclosure • Denial of service • Elevation of privilege Patch management ensures that correct code replaces incorrect code. However, it is not the only way to reduce risk. The patch management process also includes mitigation techniques that are not actual patches but include additional procedures to protect networks if the patch is not available, or if admins cannot apply it to a network, or if there are other reasons that preclude applying the patch. What is included in a Microsoft patch? Let’s roll up our sleeves, get technical, and examine what is included in each type of Microsoft patch10. Security patches, critical updates, updates, update rollups, drivers, and feature packs fall into the general distribution releases (GDR) category. These go through testing across different platforms and applications to ensure proper functionality, and that the program or update that includes new features performs as intended. However, Hotfixes developed by Microsoft Product

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

6

Support Services for a specific situation are not as tested as those included in general distribution releases. Microsoft Knowledge Base articles, freely available from Microsoft Product Support Services, always accompany these QFEs. In the Windows 2003 Server environment and now in the Windows XP sp2 workstation, the product update packages may include two or more copies of the same files to support two different types of install environments for a system. When the security patch, critical update, update, update rollup, driver, or feature pack install, the installer package looks to see what files already exist on a system. Possible install environments include: • GDR environment o Original released version (RTM) o Service pack version o General Distribution release • QFE environment o Hotfix Having discovered the appropriate environment, the installer package installs the applicable file set. To see what version of a file exists in a Windows 2003 server environment, review the following formats:11 File Version

Source of file

Srv03_rtm.mmmmmmnnnn

This file is from the original RTM version of the product and has not been updated by any security patch, critical update, update, update rollup, driver, feature pack or hotfix.

Srv03_gdr.mmmmmmnnnn

This indicates that the file is from a security patch, critical update, update, update rollup, driver, or feature pack and has not been updated by a hotfix.

Srv03_spx.mmmmmmnnnn

This indicates that the file is from a SP and has not been updated by a security patch, critical update, update, update rollup, driver, and/or feature pack.

Srv03_qfe.mmmmmmnnnn

This indicates that the file is from a hotfix.

In our server, we can see that the file on our server is a GDR version. Thus, it indicates that the patch engine did not find a hotfix and instead found a GDR version. For example, let’s look at the file included in Security bulletin 04-024 (04 for the 2004 year, -024 th

meaning the 24 bulletin of the 2004 year). Find this bulletin at http://www.microsoft.com/technet/security/bulletin/MS04-024.mspx and the sample below is the patch for the Windows 2003 platform. It includes updates to one file shell32.dll. Inside the installer package are two files. One expects that the server will still have one of the original dll’s categorized as a GDR package the other anticipates a hotfix. 13-May-2004 00:07 6.0.3790.168 8,168,960 Shell32.dll RTMGDR This version is used to apply to servers that have original released version (RTM), Service packs or General distribution versions. 12-May-2004 23:29 6.0.3790.169 8,168,960 Shell32.dll RTMQFE This version is used to apply to servers that have received a hotfix version.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

7

The shell32.dll File Version Window

While security bulletin 04-024 includes an update to only one file, many patches contain a series of files that replace existing files on a system. Other security patches may include a series of files needed to correct the condition. In the Security patch Microsoft Security Bulletin MS04-022: Vulnerability in Task Scheduler Could Allow Code Execution (841873)12 the patch includes a series of files needed to remove the vulnerability from the system: Date

Time

Version

Size

File name

Folder

08-Jun-2004 08-Jun-2004 03-Jun-2004 08-Jun-2004 08-Jun-2004 08-Jun-2004 08-Jun-2004 08-Jun-2004 08-Jun-2004 18-May-2004

22:01 22:01 22:54 22:01 22:01 22:02 19:59 22:02 22:02 03:46

5.1.2600.105 5.1.2600.155 5.1.2600.155 5.1.2600.122 5.1.2600.155 5.1.2600.1564 5.1.2600.1564 5.1.2600.1562 5.1.2600.1564 5.1.2600.1555

48,640 251,392 9,728 301,568 159,232 260,096 10,752 306,688 172,544 593,408

Browser.dll Mstask.dll Mstinit.exe Netapi32.dll Schedsvc.dll Mstask.dll Mstinit.exe Netapi32.dll Schedsvc.dll Xpsp2res.dll

RTMQFE RTMQFE RTMQFE RTMQFE RTMQFE SP1QFE SP1QFE SP1QFE SP1QFE SP1QFE

Applying new executables and dll files introduces change into a stable system. As evident from the files listed above, the security update includes both executables and dynamic link library files. An exe file is a file that a computer can directly “run” or execute. A DLL file contains a range of functions accessed by other Windows applications. The standard functions in the Windows Application Programming Interface (or API) are accessed using DLL files. This standardization eases collaboration among disparate applications. Without these building blocks, applications

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

8

would look and act much differently. A DLL can have the extension of .exe, .dll, .drv, or .fon. In any case, patching introduces new files and new code into a stable system. Thus, test to ensure that you have tested the install and uninstall processes, as well as any potential rollback issues. Historical Patch Process window Until recently, an administrator could be somewhat lax in applying patches. According to Forrester Research, the average time between the release of a patch and the attack of a worm was 305 days in March 2003. However, that window of opportunity has been shortening.

Common Name SQL Slammer Bugbear Frethem Yaha Elkern Klez Badtrans

Historical window of patching in 200313 Attack Patch Advance Impact of Attack Date Issued Notice Infections doubled every 8.5 1/25/03 7/24/02 185 days seconds More than 2 million infected 9/30/02 5/16/01 502 days computers 12 variants in the first two months 7/17/02 5/16/01 427 days of activity Intercepted in one of every 268 6/22/02 5/16/01 402 days emails at peak Detected in more than 40 different 4/17/2002 5/16/01 336 days countries $9 billion worldwide productivity 4/17/02 5/16/01 336 days loss Message Labs has seen 458,359 11/24/01 5/16/01 192 days instances

Nimda

9/18/01

10/17/00

336 days

Spread worldwide in 30 minutes

Code Red

7/19/01

6/18/01

31 days

Infection doubled every 37 minutes

As the table shows, in March 2003, administrators had many opportunities to test patches and even wait until a Service Pack before deploying. However, this window has been shortening in to include instances where patches were not available. More recently, the time between the patch and the worm for an exploit commonly known as MSBlaster was 16 days. In June 2004, Microsoft’s Internet Explorer browser suffered several security issues left unpatched by Microsoft for many weeks. Therefore, while stressing patch application as the best security prevention, we include remediation techniques as well in this patch management process. Finding out about patches We next need to identify resources that help identify which patches an organization needs. Whether you patch manually or use patch management tools, ensure that your team or your Security officer is aware of the patches available for the products in your network. For most operating systems, this is rather easy to do, as all major vendors have e-mail or RSS notification. For Microsoft products, subscribe to get e-mail notifications at www.microsoft.com/security/bulletins/alerts.mspx.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

9

Sample of a Security bulletin

Another way to stay current is to sign up for RSS or “really simple syndication.” Using a feed reader, you can receive these security bulletins quickly and easily. For more information about choosing and installing an RSS reader, go to www.microsoft.com/technet/security/bulletin/secrssinfo.mspx.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

10

RSS feed of Microsoft Security bulletins (Outlook 2003 using Newsgator)

Microsoft moved to a once a month patch schedule that has patches releases on the second Tuesday of the month between 10:00 a.m. and 11:00 a.m. However, there may be times that a patch will be released “out of band” if an active exploit is “in the wild.” For the Red Hat Enterprise Linux platform, there is also many ways to obtain notifications directly from the vendor by subscribing to email or an RSS feed at the following web site: www.redhat.com/security/team/advisories.html14

RSS feed of Red Hat security bulletins (Outlook 2003 using Newsgator)

Software vendors often provide additional support resources. For some applications installed on your network, it may be more difficult to track down notification sources. Secunia.com, for example does provide security information for quite a few vendors at secunia.com/vendor/. For your “line of business” applications, you will need to contact the vendors regarding the manner in which they notify customers. The Road Ahead In the coming sections we will discuss patch methodologies, risk management as applied to patching, and development of a patch team. We will analyze a security bulletin to find resources to determine if exploit code is already on the Internet and thus having an impact on a patch timing decision. We will discuss ways to set up a lab environment for testing patches and the procedures that you should review when ensuring that the security updates perform as expected. We will point to resources for proof of concept code or other techniques you can use to confirm patch status. We will discuss ways of enforcing patch management policy. We will discuss cases where you cannot patch because of vendor limitations or the lack of a patch, and what mitigation information and resources you can look for. While we focus primarily on Microsoft technologies, we will be vendor neutral in the use and recommendation of patching tools and give you Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

11

resources to assist you in making your choice. Whatever technology or process you use to manage patches, both the application and the reporting are key elements in ensuring that systems remain safe and secure. Ensuring that the patch or mediation technique performs as expected is central to any patch strategy.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

12

Chapter 2 – Change Management and Risk Analysis

T

raditional risk analysis accepts risk as possible, calculates the associated costs, and either applies mitigation techniques and procedures to reduce exposure, or transfers the risks to a third party. This perspective bears directly on how firms approach the patch management process. To give this perspective meaning, consider that a century or more ago, when foot travel dominated, diseases replicated slowly. Historically, epidemics took time to replicate throughout the population. Today, given airline travel and international trade, we now accept that diseases travel the globe in a relatively short time. However, imagine a person contracting a highly infectious disease for which there was no cure, and that this person could hop on a plane, travel the globe, and infect people throughout the world in less than 30 minutes. This is exactly what SQL Slammer did. It was a very tiny bit of code, one “sneeze” or a “cough” (if you will) that had several things going for it. First, while the patch [or inoculation to keep with our disease analogy] was available many months before Slammer launched, the “cumulative inoculation” or Service Patch came out only about 3 weeks before, not enough time (back in 2003) for system administrators to get the patch through change management procedures. As a rule, database administrators are probably reluctant to implement change management because they are suspicious of potential impact on their systems and because, until recently, SQL Server patching was extremely cumbersome to accomplish. Add to this the fact that patches affect both the full SQL server database system AND a “lite” version of the product embedded, without fanfare, in several other products, and it becomes easier to understand why some database admins might appear to be Nervous Nellies. This reluctance exposed asset management systems to a risk that Slammer readily exploited. Essentially, Slammer was a bit of code built to transmit over UDP protocol, a “connectionless” worm that did not wait for the syn-ack to occur and merely transmitted packets without waiting for confirmation that someone or something was on the other end of those IP addresses. Then it required that port 1434 be listening. Up to that time, this had not been a targeted port. Previous large-scale attacks always probed on port 80, the one that web sites use.

Even with its small bit of code—built for speed—Slammer had a tremendous impact. Consider the pedigree of businesses whose lack of patch management was the subject of public commentary worldwide. Continental Airlines had to revert to issuing paper tickets; Bank of America had to rescue its ATM machines; and even Microsoft coped with security breaches.15 While one could argue that a lack of network segmentation contributed to the chaos (we discuss network segmentation later), the fact is that Slammer disrupted a trusted system. “Business as usual” was no longer business as usual for many companies. Whether those companies realized it or not, they accepted the risk associated with connecting to the Internet, and accepted the risk of not patching—waiting, instead, for the Service Pack approval process to work its way to completion. Currently, we have less than two to three weeks, maximum, to approve the latest patches before an exploit arrives. In the best firms, this window of risk management evaluation is less than a week. Risk avoidance – avoid patches all together Is there any way to avoid patching altogether? If your firm does not connect to the Internet and allows no introduction of outside destructive software, your firm could actually install a base image of an operating system and apply only patches needed for functionality improvements and never worry about security updates. However, can you guarantee, without ANY doubt that no software will bridge the “gap of air” between the outside infected world and the inside network?

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

13

Again, think in terms of our heath and epidemic analogy. If you have a child of school age, do you refuse to have preventive medical treatment in the forms of shots, convincing yourself that your child will never have contact with any other infected person, or do you understand that prevention (also known as shots) are a part of life and growing up and must be done. Most security benchmarks, including those from National Security Agency, Control Security Services16and the Center for Internet Security,17 recommend keeping computers and devices up to date with security patches. If you wish to use alternatives to patching, making sure that your remedial actions are effective should be part of your standard testing process. Risk avoidance – if they’d only write better code As noted in a previous chapter, the fact that software designers have to imagine threats from all sorts of vectors means that no program that interacts with humans can be completely free of issues. As we ask applications to produce more real time information and to be predictive, and as we add everything from information retrieval to real time analytics to RFID tag identification, we ask more and more of our systems and add more complexity, not less. If your attitude is that patch management issues are a temporary thing and once we get a “proper” operating system we won’t have to patch anymore, then, in my opinion, you have the wrong attitude. Risk acceptance – the real world In May 2002, CIS quoted the Gartner Group study asserting, “Over 90% of cyber attacks exploit known security flaws for which a patch is available”.18 Patching and other system hardening measures effectively mitigate 83% of security issues overall. Accepting the probability of risk and then installing all necessary patches ensures that you limit your exposure to vulnerabilities without relying on blocking mechanisms. It is also important to note that confirming patch installation is a critical part of this procedure. Here again is our risk calculation model, expressed as ARO x SLE = ALE with the components defined as follows19: Abbreviation

Defined as

Example

ARO

Annualized Rate of Occurrence

3

Multiplied by

SLE

Single Loss Expectancy

$26,000

Equals

ALE

Annual Loss Expectancy

$78,000

So we’ve done our risk calculation shown above and determined that any costs incurred less than our calculated Annual Loss Expectancy of $78,000 will be a potential cost savings to our firm. Choosing to accept the probability of risk, we choose to apply security patches. Within the U.S. health care system, the Food and Drug Administration must approve all drugs before their distribution to the public. Similar to testing and approving drugs, we must formalize change management processes in our own organizations. Every firm needs a person or a team that signs off on and authorizes change. President Truman’s desk had a plaque that read, “The buck stops here.” The same is true for patch management. The person or team must fully understand the firm’s IT environment. There must be written policies, procedures, and sign-off checklists. We discuss policies and procedures BEFORE discussing the patch analysis process for a very good reason: You must have the policy in place before implementing the technology.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

14

Patch policies and change management policies Patch policies range from informal guidelines to formal documents identifying authorized patches that appear to function as expected, and that do, indeed, protect the vulnerability they are supposed to protect. Formal procedures may also call for processes that verify and confirm patch installation. Many medium and large organizations require a procedural type of policy that guides the patch management team through the patch approval process. Samples of patch policies or change management policies are available at the following locations: • • • • • • •

Sample of change policies at the Stanford Synchrotron Radiation Laboratory http://smb.slac.stanford.edu/blu-ice/dcsAdmin/node6.html Sample of change policies at Stanford’s University ITSS department http://www.stanford.edu/services/changemgt/itss_change_policy.html State of Connecticut Management Advisory committee Change Management policies http://www.cmac.state.ct.us/policies/change.htm Texas A&M University Change Management Policy http://www.cis.tamuk.edu/help/policies/1_050_Change%20Management%20Policy.pdf IT Assessments, tools for Project Management http://www.ittoolkit.com/cgibin/itmember/itmember.cgi/reg/ITBE-ok University of Texas Health Science Change Management Policy http://www.uth.tmc.edu/itsecurity/policy/Change_Mgmt_Policy.htm Use a search engine to search on the words “Change Management Policy” for additional resources

In general, these policies include the following elements: • • • • • • • • • • • •

Identification of the assets that need patching Identification/preparation of images that properly represent either the exact systems in your firm or a close approximation Testing of impact on applications Patch installation on the test systems Patch verification of install Optional verification of effectiveness Issue management Assessment of impact Remedial actions if impact does not affect all systems Follow-up with vendors Sign off of the patch Determination of timing of change implementation

Notice that we have performed all of these procedures and have not even begun to roll out these patches into our production network. Further, after you have cleared the “patch” for installation across your network, you need to identify the best deployment mechanism. Additional discussion of change management techniques is available at http://www.microsoft.com/technet/itsolutions/techguide/msm/smf/smfchgmg.mspx. More details about these steps are forthcoming in the next sections of this document. One resource and guidance for patch procedures and timing of patching is the SANS.org @Risk newsletter that includes information regarding security vulnerabilities and how “Council sites” react to them. Find subscription information at the SANS.ORG web site: http://www.sans.org/newsletters/. As a preview, consider the following information contained in a recent newsletter issue:

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

15

Sample of the @Risk newsletter from the SANS.org organization

Ensuring that your organization has policies and procedures to introduce “change” into your organization in a controlled and managed fashion is important in managing your risk of patching as well as mitigating dangers exposed by vulnerabilities. Identifying your change management team When setting up the team that will approve and administer your patch management process, stay sensitive to the needs of different divisions, departments, applications that are key to your organization. Similar to the process that every organization should have done in advance of the Year 2000, revisit those procedures and identify those key technology assets that you rely upon. Ensure your change management team includes a representative of each critical technology. If you rely on Oracle or SQL Server as the underlying backbone of your firm’s databases, ensure that you have a technical representative on your team. For these key elements of your firm’s

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

16

infrastructure, you should also ensure that your team has vendor contact information to follow up on issues that may arise during the patching process. Our assumptions We assume that concurrent with your resolve to establish a patch management process, you have established methodology and plans for true disaster recovery. You need a tested backup procedure that gets you back to the exact copy of any system before you began the patching process. While traditional tape and file-by-file backups give you roadmaps to where you were, most computer consultants and IT professionals would recommend that you look into Drive Imaging, whereby an exact copy of the data you have is copied in a snap shot format such that you can reapply that image and be back in “business” reasonably quickly. Even if you test the patches on test boxes and they appear to work flawlessly, never apply patches to a key critical business computer asset without ensuring that you have a backup of those files mandatory to restoring that system to full operational status. We also assume that you do the minimum amount of testing of that backup and recovery process to assure yourself that the restore process of even a few files is successful. Rename a few files of varying sizes and restore those files using your restoration software program. If you have a drive image, test the restoration of these devices by reapplying the image. Just because the backup has run and the log file indicates that the backup was successful does not mean that a restoration will be successful. Setting up a Patch Testing Scenario By this time, you and your managers need to have agreed to introduce “managed change” into your organization. A first and fundamental step in this process is setting up a test lab. You will use this environment for initial testing of all patches. Do not test patches on machines that contain real or original data (normally called “production systems”). As we will illustrate throughout, a reasonable test lab is well within the reach of nearly all budgets. This test environment should mimic, as much as possible, the setup and structure of your production network. Readily available tools and procedures help you set up virtual machines that mimic your real organization. Your organization might allow for additional testing on machines run by a team of users in an actual production environment. In this situation, identify certain staff positions that use your key line of business applications to test security patches and changes in live production settings before pushing patches out to the rest of the network. To prepare a test lab to reproduce the computing assets you must patch, you need to know what assets you are protecting. This should be easy—right? All you have to do is count up all those physical computers and network devices that appear on your asset listing or depreciation schedule. Either of these shows exactly when and for how much you bought equipment that you have in your building—right? That written schedule is all you need to identify the hardware and software your firm has purchased—right? Not so fast! Multiple recent advances work against your keeping a tight rein on your computer systems. The first concerns the relative ease of bringing in a piece of equipment from Best Buy or CompUSA and attaching to the network. Thus, you may not have a Linksys wireless router that has a vulnerability that needs patching on your asset schedule, but you might very well have one inside your office. Next is the relative ease of cannibalizing parts from older systems and using them in newer systems. Your asset schedule may state that you have a 1992 Compaq but, in reality, while the chassis is long gone, parts of that system may still be in machines in your office. When we review assets schedules of companies for valuation purposes, we regularly discount as worthless computer equipment more than four years old. That does not mean the equipment is not still in your network introducing possible vulnerabilities into your system. A nearly opposite Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

17

recent phenomenon concerns virtual personal computers and servers that are inside other machines. If you see only two computers, but your physical scan of the system indicates there are four IP addresses in use, there may be operating systems you are not aware of inside of tracked and documented machines. It is also important that you not assume you know all the software and even operating systems that are on your networked machines. We have seen cases where IT staffs have introduced unofficial operating systems into a network without management’s approval. Lastly, any device that accepts a RJ45 network connection is a device that could introduce vulnerability into a network. Any device that uses TCP/IP connectivity can be an entry point to a network. Don’t assume that because a device does not have a monitor and a keyboard that it is safe from exploitation. BugBear is one example of a virus whose “footprint” was TCP/IP attached printers suddenly printing out excess paper and garbled characters. Another example is hijacking HP Webjet or other printer administrative programs for use in attacks. We assume, of course, that your organization already has a written policy that forbids any and all of these activities. Samples of these kinds of polices are available at the SANS.org policy project center at http://www.sans.org/resources/policies/. In general, such policies stipulate that management approval must precede introducing new assets into a network. One of the ways to preclude the introduction of rogue software is to standardize on an installation image of a computer system. Subsequent new assets conform precisely to what the firm expects and are thus controllable and patchable. A sample of a policy that requires that all introductions of assets be done in a controlled manner is available at the SANS.org policy project center at http://www.sans.org/resources/policies/Aquisition_Assessment_Policy.pdf. Not controlling the introduction of assets into a network greatly increases the potential for unknown vulnerabilities entering the system. Discovery of Assets Your first step in designing an effective patch management program involves understanding the environment you have to patch. In any sized firm, understanding the operating systems and all the devices susceptible to patching is pre-requisite to determining which patch tool to choose. For example, if your environment contains a mixture of Windows and Linux/Unix distributions, choose a patch tool that supports both platforms. If your environment consists mostly of remote assets, choose a tool that supports background patching on a variety of connection speeds. You don’t want the boss dialing in from home on a 56K connection as you push down a large device update. Depending on your environment, you can choose among various tools to examine assets. In the Microsoft environment, choices include tools ranging from Active Directory to identify all connected assets, to WSH for script a system inventory. There are also many freely available options, and shared scripts can remotely inventory a network and build an appropriate asset database. Consider also commercial inventory and control products such as Microsoft Operations Manager [MOM], Argent, and Tally Systems, just to name a few such tools that have inventory features. Sample asset analysis tools include: • Microsoft Corporation, Microsoft Software Inventory Analyzer http://www.microsoft.com/resources/sam/msia.mspx • Lynch, Chris, Inventory every computer in Active Directory, http://cwashington.netreach.net/depo/view.asp?Index=808&ScriptType=vbscript • Lynch, Chris, Create Hardware Inventory in Excel Format http://cwashington.netreach.net/depo/view.asp?Index=612&ScriptType=vbscript

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

18

Host Name: _____ OS Name: Microsoft Windows XP Professional OS Version: 5.1.2600 Service Pack 2 Build 2600 OS Manufacturer: Microsoft Corporation OS Configuration: Member Workstation OS Build Type: Uniprocessor Free Registered Owner: ________ Registered Organization: Product ID: ______________ Original Install Date: 9/27/2003, 8:25:41 PM System Uptime: 2 Days, 1 Hour, 21 Minutes, 26 Seconds System Manufacturer: Dell Computer Corporation System Model: XPS-Z System type: X86-based PC Processor(s): 1 Processor(s) Installed. [01]: x86 Family 6 Model 8 Stepping 10 Genuine Intel ~996 MHz BIOS Version: DELL - 20010221 Windows Directory: C:\WINDOWS System Directory: C:\WINDOWS\system32 Boot Device: \Device\HarddiskVolume1 System Locale: en-us; English (United States) Input Locale: en-us; English (United States) Time Zone: (GMT-08:00) Pacific Time (US & Canada); Tijuana Total Physical Memory: 511 MB Available Physical Memory: 114 MB Virtual Memory: Max Size: 2,048 MB Virtual Memory: Available: 1,961 MB Virtual Memory: In Use: 87 MB Page File Location(s): C:\pagefile.sys Domain: ___________ Logon Server: \\__________ Hotfix(s): 8 Hotfix(s) Installed. [01]: Q147222 [02]: Q823718 [03]: Q832483 [04]: Q828026 - Update [05]: wm817787 [06]: wm819639 [07]: wm828026 [08]: KB811113 - Service Pack Network Card(s): 3 NIC(s) Installed. [01]: 3Com Ether Link XL 10/100 PCI TX NIC (3C905B-TX) Connection Name: Local Area Connection DHCP Enabled: Yes DHCP Server: __________ IP address (es) [01]: __________ [02]: VMware Virtual Ethernet Adapter for VMnet1 Connection Name: VMware Network Adapter VMnet1 [03]: VMware Virtual Ethernet Adapter for VMnet8 Connection Name: VMware Network Adapter VMnet8 Sample of the information that can be obtained from a system using scripting

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

19

Your patch management tool may also aid in the asset identification process. If you have SNMP20 enabled on your system, use that to query the devices attached to your network. Remember that print servers and other devices can be just as vulnerable as computer systems. Necessary patches versus all patches When analyzing what to patch, determine your firm’s acceptance of patching. Do you own all of the assets you need to patch or are you bound by lease agreements to patch only on a schedule that a vendor approves? Some medical equipment vendors certify their equipment only through certain service pack levels and thus you may void your agreement if you install patches or even antivirus products on their equipment. Check your service agreements and vendors for any caveats they may have that limit your patching ability. You may need to install traffic limitations and other barrier technologies to ensure that TCP/IP traffic is only in the direction you want it to be if you are unable to install patches on leased equipment that interacts with owned equipment. You can also determine the role of the computer or server and determine the patches minimally needed for that role. Be very careful in using this minimal patch methodology unless you truly have a clear handle on the software in your network and the traffic that routes to domains, vlans and other routed networks. Again, using the example of SQL slammer, many desktop users simply did not know that they had a version of SQL called Microsoft SQL Desktop Engine that was just as vulnerable to SQL slammer as Microsoft SQL server if left unpatched. Thus, manual patch methods whereby you push only those patches that you think you need may not be a wise move at all. In some cases you might decide that the web server you are patching that does not have NetBIOS ports open to the Internet does not need the latest patch for NetBIOS information disclosure vulnerability. Nevertheless, this risk-based patching methodology needs to be discussed and approved in your organization. You and your team need to weigh and determine the exposure and risks appropriate to either patching or not patching. There are costs to both methodologies. We determine the threat vector and vulnerability risk as factors in the timing of patching, not whether or not the patch will be applied. Setting up that test lab - utilizing a virtual environment After inventorying the network, set up a lab that provides the best equivalent emulation of the actual network. Either by having duplicate machines and hardware, or by using virtual machines based on VMware21 or Microsoft Virtual PC or Virtual Server, you can test using equivalent data or even exact images of data in your network. Dr. Jesper M. Johansson offers the following discussion for using virtual PCs to obtain drive images of data used in patch testing. (More details on this method and more on network threat modeling will be included in the forthcoming book by Jesper Johansson and Steve Riley entitled Protecting Your Windows Network. It will be available from Addison Wesley, Inc. in 2005.) “There is a neat new way to get a near-perfect replica of a production system that you can use for testing. Microsoft Virtual PC 2004 includes the ability to make a new virtual disk based on an existing hard disk. Using this functionality you can install Virtual PC on a production system (mind you, probably not one that is currently serving clients) and make a replica of its disk. Then you can copy that disk image off to another computer. Now install Virtual PC on that other computer and create a new virtual machine pointing to the disk image you just created. On the first boot, you will need to remove Virtual PC from the virtual machine, and possibly install a few drivers. However, you can now set up the disk in undoable mode and have a copy of a production system on which you can test new security updates. Obviously, this does not work for all types of testing, such as testing with particular hardware [that] is not emulated by Virtual PC, but it works pretty well for testing interoperability with software. Cost-wise it is far more Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

20

efficient than purchasing and managing additional hardware since Virtual PC only costs about $150.”22 If you use virtual machines and actual data, be aware of potential security issues of using copies of data. Ensure that your organization has previously gone through the process of data stratification. Categorize data as being public, restricted, or confidential. Do not use servers or workstations that contain restricted or confidential data for testing purposes unless the change management team stipulates that testing must include this more sensitive data. When imaging such data, follow strict procedures regarding the handling of such data as detailed in the firm’s change management documentation. Destroy the imaged data after the testing procedures and have these procedures monitored and audited to ensure that data integrity occurs within the organization. You may need to include such data as credit card numbers in databases in a test patch process to ensure the patch process is successful. However, most security patches for database engines never touch the data itself. The patches, typically, restrict themselves to adjusting the code in the database engine only. Testing procedures that include database integrity (hash totals, for example) may be included in the change management procedures to ensure data integrity before and after the patching process. Patching databases is one of the most critical patch implementations in any environment as it typically involves introducing software into a critical key business asset—the database information. Thus, meet with your firm’s database software engineers and vendors and get their recommendations for patch installations and database security. In the history of worms that affected databases, an SQL database administrator could have thwarted SQL slammer if he or she had merely taken the time to remove a blank SA password.23 Here is another example of where adhering to traditional security measures could delay the timing of a patch application. Again, we recommend against not patching and relying on mitigation alone. We do grant that following some basic security guidelines will buy you time for patching. Review the resources listed here to become a smarter patch manager for databases: SQL Server • National Security Agency, SQL Server security recommendations at http://www.nsa.gov/snac/db/mssql_2k.pdf • SQLSecurity.com at http://www.sqlsecurity.com/DesktopDefault.aspx • “Best Practices Analyzer Tool for Microsoft SQL Server 2000 1.0” http://www.microsoft.com/downloads/details.aspx?FamilyId=B352EB1F-D3CA-44EE893E-9E07339C1F22&displaylang=en Oracle database • Center for Internet Security, Oracle database security benchmarks http://www.cisecurity.org/bench_oracle.html • Oracle Corporation, Security notifications http://www.oracle.com/technology/deploy/security/alerts.htm The bottom line If you are running around once a month feeling as if you don’t have a handle on what you need to patch, you need to stop and take action now. Based on this first section discussing the technical aspects of patch management your action items are as follows: • • • •

Identify key business computer assets Review the basic security procedures of your organization Ensure written policies are in place Set up a patch management team based on that analysis

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

21

• • •

Test or set up backup and disaster recovery techniques Set a patch management policy based on risk management Identify the testing lab you will use to perform patch testing

Our next section discusses ways to analyze and review security bulletins. This will help you identify patch needs based on a risk management determination and develop patching methodologies. We will also review additional techniques of enforcing patch policy to affected systems, and preview expected changes in patch management—including a look at the increasing number of tools that patch both Windows and alternative operating systems.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

22

Chapter 3 – It’s all about risk

E

arly evidence of a truly significant patch problem surfaced in mid-2003 postings to public newsgroups. Windows NT 4 server systems were dying in the water. While they were booting up and clients were able to log in, none of them could connect to the internet and the RRAS service was gone. Reports began filtering back to the newsgroups that there was a problem with a recent security patch. Posters indicated that the support representatives recommended pulling the patch off and rebooting systems. Removing the one patch confirmed restoration of RRAS and Internet services. While the support call to the vendor was a free call, placing it cost down time and investigation time for the administrator or consultant. Three weeks later, the vendor re-released the patch and all normal server functions performed as expected. The above accurately depicts what occurred to the Small Business Server 4.5 platform. One patch caused a service not to run. Microsoft Security Bulletin 03-029’s purpose was to protect Windows NT 4 from a denial of service attack. Instead, the patch caused a denial of business.24 The vendor identified the problem, repaired the patch, and removed the issue. Why did this occur? Here is some informed speculation: resources for testing on the NT platform concentrate on traditional servers and not on the integrated platform of the Small Business Server. For whatever reason, testing did not find the problem with 03-029. As a result, instead of protecting the system, the patch actually introduced failures to the system. There was less risk to a network from a potential exploit than there was benefit derived from the original patch. In this case, introducing change did more harm than good. Since that time, Microsoft has increased both internal and external patch testing to reduce these issues. A recent report card gives a grade of “C” to the current situation of Security patches,25 but commended Microsoft for developing a process for external testing to better identify patches that cause issues. Regardless of changes to vendors’ internal testing methodologies, it remains mandatory that you test patches before rolling them out to your network, your workstations, or your servers. As soon as a patch becomes available, the clock begins ticking. Miscreants begin by analyzing the patch, looking for a potential exploit, and then developing a proof of concept of the vulnerability. Therefore, the faster you obtain approval for change, the faster you can patch. You need to control patches and patch installation in your network. Only by testing and anticipating issues in advance will you be able to manage change in your system. The administrator’s role To use a sports metaphor, a patch administrator’s job is to coach on the sidelines, introducing a new player to the field. Introduce an untested player and the play misfires. Introduce a proven performer who is ready to go and ready to do his job and the team functions smoothly. To further this metaphor, it is the admin’s job to understand which player is better in front, which one needs to be a receiver, which is better at offense than defense. You may want to patch front line players faster than those who are inside the perimeter. You may want to take mitigation actions on players that, for one reason or another, are inappropriate for patching. As coach of your network, you need to know all of your team members well. Using scripts, tools, and inventory management processes, you should know exactly what computers and programs you have running. Your first job is to determine what you have and what you need to patch. Looking at a Microsoft world, you first need to determine workstations and programs. To do this, use a variety of tools or maybe just script an inventory solution. Available script sources include: • • •

http://cwashington.netreach.net/depo/view.asp?Index=808&ScriptType=vbscript http://www.microsoft.com/resources/sam/msia.mspx ftp://ftp.microsoft.com/PSS/Tools/Networking_Support/MPSReports

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

23



http://www.belarc.com/ctadvisor.html?B

Available programs include: • • •

Microsoft Systems Management Server http://www.microsoft.com/smserver/ Remedy asset management http://www.remedy.com BindView control solution suite http://www.bindview.com

If you have a mixed environment, you need to investigate other tools and programs to remotely manage and inventory your system. You must know what you have in order to control your systems. Regularly scan your network to detect any new devices present on the system. The point here is critical: If you don’t have control of your network, you don’t have control of your network. Identify key factors Recall the hysteria before the year 2000 when the world’s computers were going to come to a crashing halt. Remember also the processes you used to identify key applications that you had to keep functioning for your firm to survive. It is now time to resurrect those evaluations and examine all processes and procedures key to your organization. Determine which applications are so vital and key to your organization that impairing their operations would severely affect your company. Can you patch? So now you have identified all assets that need patching—desktops, laptops, routers, web servers, firewalls, domain controllers, database servers, SQL servers, line of business applications--every device in your network that contains computer code needs monitoring and patching. However, here’s a catch-22 many of us face: we do not truly own the assets we are in charge of patching. In the health care industry, for example, vendors that own and lease the equipment point to Federal Drug Administration rules and will not support the installation of patches. Conversely, the FDA argues that vendors can support patching of systems. Furthermore, many of us outsource our Information Technology departments and have Service Level Agreements in place. Introducing unauthorized software via patching, antivirus, or monitoring software may void agreements. Equally frustrating is the stubbornness we see among line of business vendors who will not go on record and support a patch or, worse yet, will not support a service pack or patch on systems running their software. When Microsoft released Windows XP SP2, many line of business applications refused to support the installation of the patch. As coach, then, you find yourself in a truly difficult situation: skip the patch and expose unpatched vulnerabilities to exploits, or patch and risk voiding service agreements. This is not exactly a great choice. Some time back a security researcher left the field and, in his parting post to a newsgroup, he urged those of us who remain to “Demand better security from vendors and hold them responsible”26 We need to recognize that line of business software vendors impact our security decisions. Understand this message very carefully: Use what little influence you have during contract negotiations to make your line of business vendors understand how they affect your security decisions. There is no easy answer for those of us in this position. If contractual arrangements constrain your patching responsibilities, the next section on timing of patching is irrelevant. You will instead need to concentrate on procedures to reduce exposure in your network. These procedures may include vlans [virtual local area networks], air gaps, devices that restrict traffic to flowing one-way

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

24

across your network. Bottom line: if you cannot patch, your job is to isolate, segment, defend, and minimize exposure as best as you can. Handicapping the Patches Once a vendor—Microsoft or Sun or Adobe, for example—releases a security bulletin or patch, the clock starts ticking. You have a limited about of time to review the bulletin, decide what to patch, what to mitigate, where your threats are, whether or not there are exploits in the wild and how they might enter your system. I call this process “handicapping the patches”. In my firm, I use this process to determine when I will patch. I control my machines and sometimes choose to install tested service packs before my vendors approve them. I don’t void any warranties, nor do I jeopardize any contractual arrangements. I could choose not to patch my systems if I felt that I had enough blocking mechanisms in place. I could use the methodology to identify mitigation. My small size, my fully enabled active directory environment, and my 100 percent “borg network” (all of my desktops, laptops, and even telecommuting computers are on the Windows XP platform) mean that I can quickly and easily identify, patch, and protect my assets. That does not mean I have any less highly sensitive and critical data. However, I have two issues that expose me to significant risk. First, my industry routinely handles information—names and social security numbers—targeted by identity thieves. Further, because I work in California and perform work for employees in California, I am subject to the provisions of a public law SB 1386 and a new law effective January 1, 2005 called AB 1950. SB 1386 requires that if I have an unauthorized person view data containing sensitive identity theft information, I must inform the affected parties unless authorities instruct me not to do so, or unless encryption has protected the data in question. However, none of the major business line application vendors in my industry will go on record supporting EFS or PGPdrive or any other traditional form of encryption, nor is such encryption native to the programs. The new law, AB 1950, requires “reasonable procedures” to ensure that networks are secure enough to protect identity information for California residents. Gartner Research recommends that firms follow the guidance in ISO 17799. Keep in mind that this is a California law designed to protect data of California “residents” no matter where the data resides. Thus, your firm may not be a California nexus entity, but you would be still covered under this law if your servers contain data from California residents. Second, in my environment, as may well be the case in yours, our “bread and butter” line of business applications come forward from Windows 98 time and exhibit the Windows 98 way of thinking. Thus, many of my industry’s day-to-day applications require a level of user rights called “local administrator” on the desktop. This is equivalent to removing all security settings from a Windows XP or Windows 2000 system. Using tools such as Sysinternals.com’s Regmon and Filemon, or one from PC Magazine called Inctrl5, to monitor and document permissions needed on the files, and then set group policies to push down registry settings so the programs have needed administrative rights without crippling security, takes time and consumes resources to test and implement. It would be far better to have the application run in “least privilege” in the first place. As Aaron Margolis so aptly stated in his blog on the advantages of running as User: The #1 reason for running as non-admin is to limit your exposure. When you are an admin, every program you run has unlimited access to your computer. If malicious or other “undesirable” code finds its way to one of those programs, it also gains unlimited access. When an exploit runs with admin privileges, its ability to compromise your system is much greater, its ability to do so without detection is much greater, and its ability to attack others on your network is Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

25

greater than it would be with only User privileges. When an exploit requires administrative privileges (as many do), running it as a User stops it dead. However, if you’re running as administrator, an exploit can: • • • • • • • • • • • •

install kernel-mode root kits and/or key loggers (which can be close to impossible to detect) install and start services install ActiveX controls, including IE and shell add-ins (common with spy ware and ad ware) access data belonging to other users cause code to run whenever anybody else logs on (including capturing passwords entered into the Ctrl-Alt-Del logon dialog) replace OS and other program files with Trojan horses access LSA Secrets, including other sensitive account information, possibly including account info for domain accounts disable/uninstall anti-virus cover its tracks in the event log render your machine unbootable if your account is an administrator on other computers on the network, the malware gains admin control over those computers as well and lots more27

In the end, the risks I face in my industry derive not so much from my vendors not supporting patches, but more from their not supporting users having lesser, that is “user” rights. Nevertheless, given my fiduciary duty of data protection for my clients, this means that patching as means of protection makes sense for my industry to remove the vulnerability from the system. To sum up, once the bulletins become available you need to assign resources to reading and reviewing them. I will use an historical bulletin to illustrate my methodology. You and your team need to develop your own methodology of reviewing and testing patches then scheduling patch deployment. You need a patch-testing matrix a plan for your organization. Case study—Blaster arrives Security bulletin Buffer Overrun in RPC Interface Could Allow Code Execution http://www.microsoft.com/technet/security/bulletin/MS03-026.mspx [aka MSBlaster]. Our first review of this bulletin is that is it rated by Microsoft as a critical bulletin.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

26

Graphical view of the bulletin rating system

The web page recapping the bulletin clearly shows that it is affects Windows NT, 2000, XP and 2003. The very first question that I ask is, “Does it affect my network?” If it does, I then review the rating of the bulletin. Microsoft currently uses a 4-point rating scale. The Microsoft Security Response Center (MSRC) rates the bulletins based on their impact.28 In general, the MSRC recommends ALWAYS applying bulletins labeled as critical or important to your systems. Depending on your network, they recommend that you decide whether to install moderate or low rated patches. I personally patch anything that moves and, thus, will install any patches in my network after testing. You must make this decision for your environment. You may decide that datacenters need only critical and important patches; you may decide that workstations need all patches. Remember that, for OEM systems, vendors offer patches that they deem critical, but which are not necessarily security patches. If you scan a test workstation after you patching it by using Windows Update and find that it reports it is missing a patch, look carefully. That critical patch may be for a “Dell deemed” critical driver instead.

Microsoft’s rating of the Security Bulletins

Over time, as you use your test network effectively, you will develop an innate sense of which patches the devices on your networks need depending on the roles those devices play. I strongly recommend that you set up the ability to RIS or image in your network. Further, your firm will be much more efficient if you begin to think in terms of the novel concept that states, clearly and emphatically that employees don’t own the computers that they use. Because your company owns the computers, controlling what goes on them is your responsibility. Furthermore, as we noted earlier, you can meet vendor requirements for users to have high rights levels by adjusting registry entries. The next step in reviewing the security bulletin gives an indication of the possible threat to a network:

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

27

The phrase “run code of attacker’s choice” normally indicates that the attack might be exploitable remotely. However, there may be mitigation available that is not apparent upon first reviewing the bulletin. When is critical not critical? While bulletin may indicate that the security issue is critical, you still need to review the bulletin for mitigating factors. Expand the bulletin and review the section called “Technical details.” In this section, MSRC explains the ranking. Pay close attention to this section as this will determine when or if you patch. In the “Blaster” example, the “mitigation section” reads as follows: To exploit this vulnerability, an attacker would need to send a specially formed request to the remote computer on specific RPC ports. Mitigating factors: To exploit this vulnerability, the attacker would require the ability to send a specially crafted request to port 135, 139, 445 or 593 or any other specifically configured RPC port on the remote machine. For intranet environments, these ports would normally be accessible, but for Internet connected machines, a firewall would normally block these ports. In the case where these ports are not blocked, or in an intranet configuration, the attacker would not require any additional privileges. Best practices recommend blocking all TCP/IP ports not actually in use and most firewalls including the Windows Internet Connection Firewall (ICF) block those ports by default. For this reason, most machines attached to the Internet should have RPC over TCP or UDP blocked. RPC over UDP or TCP is not appropriate in hostile environments such as the Internet. Robust protocols, including RPC over HTTP, are suitable for hostile environments. Our mitigation factors focus on whether protocol ports are open for transmission. Analyze your systems. A typical network will not have ports 135, 139, 445 and 593 open on the outside because these are usually trusted network connection ports or file sharing ports. In a typical business network configuration, these ports would not be open to the outside world. However, what is YOUR situation? Avoid problems by regularly auditing port settings to ensure that what you think is open is, indeed, the only thing that is open. Set policies to enforce control and change management on firewalls in the same manner as you set up change management on operating systems. Use tools or services to regularly scan and review the exposure on your network. Consider hiring vulnerability analysis firms to investigate and externally audit these external entry points. What your “hard shell” looks like from the outside is extremely important. Servers, mail servers, and data centers “leak” information about what operating systems they run and their patch levels. Knowing what you look like from the outside helps to identify those entry and threat points to your network. Continued analysis of our systems tells us the following: Role of System

Threat Vector

Exposure

Mandatory Patch needed

Edge device, router, firewall server, etc

Open and listening port 135, 139, 445 and/or 593

Not open to external access

Not needed

Because we do not have exposure to the outside, it seems as though we do not need to install any patches to our systems. It appears that we do not need to take any additional action whatsoever. Unfortunately, as we all know, the problem was that we are extremely vulnerable on the inside of our networks. Looking at our networks on the inside, our analysis shows: Role of System

Threat Vector

Exposure

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Mandatory Patch needed

Page

28

Internal server, workstation, Exchange server

Open and listening port 135, 139, 445 and/or 593

Wide open to inside exposure

Patch needed

If we could guarantee that all exploits would stay outside our networks, we might not need the patch inside our networks. However, unless you have network quarantine methodologies, edge filter or other blocking techniques that you have tested, TCP/IP connections from unvetted computers regularly connect back to your network. Even in my environment, my network boundary is not limited to the computers that are physically located in the firm; rather my network extends to the wireless connection in Starbucks where I am sitting using the Internet to remote back to my firm. So what’s the worst thing that can happen? In the case of the exploit of Security bulletin 03-026, there is quite a bit that can happen. Study the bulletin carefully:

It is obvious from reading the ‘more information’ section that this is serious. In addition, this bulletin rates as critical on all platforms. You need to devote resources to testing this patch. The process affected by this bulletin is something called DCOM RPC interface. So why don’t we just disable that service or process and go on with life? Because of two things: First, you may have critical line of business applications that depend on the service, and secondly, many vendors write their programs using the default enabled services and are unable to tell you if their program does or does not depend on an affected service. Thus shutting it off may not be an alternative. As is evident here, DCOM functions as glue in networking:

When you not only recognize which underlying service is vulnerable, but also know which ports are exploitable, you have clues to the level and type of testing required. Our sample patch indicates that you will need to maintain normal network connectivity. While I will discuss testing procedures in a later chapter, for this bulletin, test procedures should cover file transfers and monitor file transfer speed. In an historic footnote to this bulletin, AutoCAD files29 did indeed suffer data corruptions because of this security bulletin and a hotfix eventually corrected the issue.30 Thus, while reviewing the bulletin, you should identify key applications that you will need to test in your environment. In this instance, those applications that pulled files across the network would need additional testing time. I strongly urge that you prepare a checklist documenting the tests, indicating procedures, and identifying applications on which you have tested the patches. If an issue does arise after the fact, document the affected application, note if the files included in the patch are re-patched, and assign additional time to test against previously affected programs.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

29

Sticking with our 03-26 Security Bulletin as an example, reviewing the affected files shows that they have changed several times. Now while patches may not affect the same “bits” each time, even if you don’t have the resources to do a detailed analysis of the patch files, this provides evidence that you may need resources to test the applications in your firm that had issues with those files whenever you patched them in the past.

Files patched with Security bulletin 03-026 as listed in http://support.microsoft.com/?kbid=823980

Files included in Knowledge base hotfix 824136 (http://support.microsoft.com/?kbid=824136) that patched a corruption issue in AutoCAD files

Files included in Security bulletin 03-039 that also affected RPC and replaced 03-026 as listed in http://support.microsoft.com/?kbid=824146 31 This is just a sample of the impact of this one bulletin. For every security bulletin, you need to assign a team to review, identify your exposure, identify those services and processes most affected, and test on your standard server and workstation images. Resources for Threat Assessment Many of you may be thinking, “Well this is a fine analysis for an historical patch” but how do I find guidance for current issues? Among the chief resources for Microsoft patches is the “Threats and Countermeasures Guide: Security settings in Windows 2003 and Windows XP.” Find this at http://www.microsoft.com/downloads/details.aspx?FamilyId=1B6ACF93-147A-4481-9346F93A4081EEA8&displaylang=en. Other resources include: Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

30

• • • •

NSA Security Configuration Guides: http://www.nsa.gov/snac/index.cfm?MenuID=scg10.3.1 Center for Internet Security Guides: http://www.cisecurity.org/ Open Web Application Security Project: http://www.owasp.org/index.html IANA master listing of ports used on the Internet: http://www.iana.org/assignments/portnumbers

If you have not yet inventoried your network, take this opportunity to document what services are running and could be possible threats. One source for this can be a WMI script that pulls this information from the server. Find an example at SYDI - Script Your Documentation Instantly: http://sydi.sourceforge.net/ Remember: • You cannot protect what you do not understand. • You cannot protect what you do not know you have. • You cannot control what you do not access. Additional testing procedures Participate in patch communities to find additional resources regarding patch issues. Posters to resources such as the Patch Management32 listserve, Patch Talk33 and even newsgroups34 regularly share information and patching experiences. If you have few resources to devote to patch testing, your next best option is to find a resource for your industry. For those industries where you are restricted from patching, several online resources35 may provide guidance for properly setting up network segmentation, or may point to other resources for requesting vendors to test patches themselves. Sharing information can be a very effective resource for patch issues if you have minimal resources for testing. Testing procedures may include effectiveness testing If your testing process includes a check off for independent verification of patch effectiveness, your next task is identifying resources to obtain code you can use for testing. Testing patch effectiveness with test code involves using “live exploit” code. Be sure to do this in a segmented test lab that is nowhere near a production system. Ensure that you have an extreme air gap, meaning no electrons can go between this test network and your production network, and that you do not inadvertently introduce destructive code into your production network. Use this approach only if you have trained staff, a controlled environment, and the proper resources. This is not a trivial testing process; do not begin it without taking proper prevention and exercising caution. For example, FSecure uses an RF shielded lab when analyzing malware. This is discussed in a blog post at http://www.f-secure.com/weblog/#00000373. Typically, there are two resources for test code or exploit code. The first can be the security bulletin itself. Included in the bulletin are references to the CVE-Mitre vulnerability database. At times, this database points back to discussion of exploit code on the web. Our example bulletin of 03-026 contains a link pointing to the following:

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

31

The hyperlink, in turn, points to the web site at http://www.cve.mitre.org/cgibin/cvename.cgi?name=CAN-2003-0352 that includes a section of references. Included in this section are web sites and links. Following these links, we find that some are mere announcements on open vulnerability listserves, but some postings provide more detailed analyses and include code in their postings such as the one found at http://marc.theaimsgroup.com/?l=bugtraq&m=105914789527294&w=2. Examples of ways people share on these listserves include Jeremiah Cornelius’ analysis of the timeline between patch distribution and the first signs of exploit code on the web posted to http://lists.netsys.com/pipermail/full-disclosure/2003-August/009278.html. Ben Jurry posted the first proof of concept code, and H.D. Moore of the Metaspoit project36 refined the code. Ready-torun versions of the exploit were on the web by July 26 at http://lists.netsys.com/pipermail/fulldisclosure/2003-July/007103.html. While the process is not for the faint of heart, if your environment requires such testing, it is not long between the patch’s development37 and the exploit community's attempts to reverse engineer and develop exploit code.

MITRE Database pointing to additional resources

There are also web sites, listserves, and RSS feeds that specialize in exploit notification. Examples include: • • • • • • • •

Full Disclosure mailing list: http://lists.netsys.com/mailman/listinfo/full-disclosure (no moderation is done on this list so you have to weed through postings to find the exploits) Bugtraq mailing list: http://www.securityfocus.com/archive/1 K-otik RSS feed of their vulnerability list http://www.k-otik.com/exploits.xml K-otik - Alertes Sécurité: http://www.k-otik.com/mailing (you don’t need to read French to track vulnerabilities) Security Focus Vulnerability RSS feed: http://www.securityfocus.com/rss/vulnerabilities.xml Security focus mailing list for potential vulnerabilities: http://www.securityfocus.com/archive/82 Packet Storm Advisories: http://www.packetstormsecurity.org/advisories100.html Vulnwatch: http://www.vulnwatch.org/subscribe.html

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

32

A safe way to confirm that a patch has properly deployed into your test scenario is merely to review the location in the registry. On the other hand, if you use a patch installer program that provides verification, rerun the patch tool to ensure deployment. You might use the Microsoft Baseline Security tool (http://www.microsoft.com/technet/security/tools/mbsehome.mspx) for this or, in our sample bulletin case, open regedit and review the following regkey for installation confirmation (sample of Windows 2003 reg key entry): HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Updates\Windows Server 2003\SP1\KB823980 Also pay attention to requirements to reboot the system before the patch takes effect. There is an effort within Microsoft to keep patches from requiring reboots, but as of now, this process is not yet complete.38

Information about restarting in Security Bulletin 03-026

If you remember the history of Security bulletin 03-026, you will remember that some companies believed that they had patched. However, because they had not rebooted they failed to protect their systems. In the case of 03-026, additional, built-in tools ensured proper file replacement. You may also want to use tools located at the Sysinternals39 web site such as Listdll to perform additional testing. Standardize your office One way to simplify the patching process involves standardization of images. Using a standard image40 to deploy your systems greatly increases the supportability and predictability of your network. Documenting the approved software in your network and allowing installation of that software only helps deliver expected test and rollout results. Change management means that you approve ALL changes. A recent post to the Patch Management listserve included a summary by Dominic White41 of Sun Microsystems’ suggestions for patching.42 The guidance is universal: • • • • •

Analyze the need to apply patches or update your software based on risk, cost, availability, and timing Minimize change to your environment whenever possible Address SunAlertSM notifications (Security bulletins) and other critical issues as soon as possible Making other changes to your environment only to address known problems Keep your environment as current as appropriate for your business and application needs

To summarize—assign appropriate resources to test patches. This is crucial to maintain up time, and ensure that you manage issues that may arise. Patch testing is just one part of the change management process. Testing processes So how do you test a patch? The most basic test process involves applying the patch to samples of your normal environment of workstations, servers, data servers, and the like. This is where standardization of computer images is especially handy. The more you standardize, the faster

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

33

you can test patches against your images and roll them out to your workstations and servers. By now, you should be visualizing a patching checklist for your own environment: Testing plans Identify the operating system subject to testing. Identify the service pack level. Identify the hotfixes installed on your systems (if in addition to security fixes). Identify critical third party applications. Identify third party applications that have had patching historically. Identify those files used in patches that may have causes issues in the past. Are the included in this current patch? Assign testing resources appropriately. Study the bulletin to determine if you can uninstall the patch. If not, determine if additional resources for testing or imaging need to be in place before approving the patch. Test the installation of the patch both manually and via your automated patch technology. Can you uninstall the patch using add/remove program or your patch tool? Review the processes of your line of business applications. Are they performing as expected? Attempt to replicate a production environment using imaged data. Having an exact image provides the best testing bed. Set up performance and monitoring tools to review your testing machines43 such as PerfMon, tools from Sysinternals44 and review all log files. Confirm the installation of the patches via registry review or other means. (OPTIONAL) Confirm the effectiveness of the patch using testing code. Follow any additional procedures your situation requires. Approve the patch for release. PREPARE BACKUPS.

The Infraguard Technology Risk checklist also includes the following: • • • • • • • • • • •

When applying a patch to any system vulnerability, verify the integrity of, and test for the proper functioning of the patch Verify that the patch will not negatively affect or alter other system configurations Test patches on test beds before being released into the network Backup your system before applying patches Conduct another vulnerability test after you apply a patch Keep a log file of any system changes and updates Prioritize patches prioritized Disseminate patch update information among the organization's local systems administrators Add timetables to patch potential vulnerabilities Require that external partners deploy all non-critical patches within 30 days Require that external partners deploy critical patches to servers and clients within 48 hours45

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

34

Think about building your own test policy. In your perfect world, you will test patches on copies/images of your production systems. You will replicate your user load and your normal processes. You will identify your critical applications and related issues. That is the perfect world. Do all this and you will be ready for our next step of rolling out patches to the network in a controlled fashion—and then we will talk about reality.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

35

Chapter 4 – Are we ready to patch yet?

T

here is one key point that we need to emphasize. Even though we try to replicate our systems as best as we can, there will be times when we cannot perform an exact test to find all the expected issues that may arise in connection with deploying a patch. Thus, it is critical that you have backups and system images before you begin to roll out patches. The following are a few of the available backup and imaging solutions for Windows and Linux environments: • Symantec Ghost • Acronis True Image • Veritas • Ultrabac • Native NT backup • UniTrends • Arkeia Remember evaluate your network from a risk analysis standpoint, even if you have a proper backup or image of those stations you are patching. What systems are the most at risk for the patch in question? Carefully review security bulletins and assign patch priority based on the role a system plays in your network. Look again at patch guidelines such as Security bulletin 04-01146 for the Sasser worm. Security bulletin 04-01147 was a comprehensive bulletin that patched several security issues. It covered vulnerabilities that exposed edge devices to denial of service issues. “Chatter” on the listserves indicated that systems directly attached to the Internet and at the edge of a network needed immediate patching. For this reason, some firms, under certain circumstances, take the calculated risk of patching edge systems without pre-testing. Patch “at risk” machines first Identify regions in your network that are most at risk and assign zones. This is similar to the “Organizational Units” concept in the Active Directory setup. Zone out and patch servers separately from workstations, and build patch templates for each zone to scan and patch each zone separately. Internet Explorer, Outlook Express, or Windows Media patches may be more critical on desktops than they are for servers, as no one surfs or reads e-mail from server machines. Furthermore, in Windows 2003, the Internet Explorer enhanced lockdown ensures that most Internet Explorer patches are not highly critical. The reverse is true for desktops and machines used by information workers who surf for research purposes. Here there is a higher priority on Internet Explorer patches. Next, prepare templates for machines that you can patch more quickly than other machines due to their role in the firm or because they do not require prepatch backup procedures. Develop zones so that you can patch different machines in different ways. For those devices that do not permanently attach to your network, you may need a combination agent/agentless patch tool. We will address this in detail later, but the type of patch tool you have may dictate how you process the patches. Whether you manually script it, use Software Update Services, or use one of several available Patch Management software tools, your ability to control the patch process dramatically increases the more you automate this process. Additionally automation documents the change process and audits the installation of the patch process. Even after you have properly tested, consider doing a controlled test roll out to designated testers. This is particularly true for different types of patches such as a service pack that contains many more code changes than a security patch does. After your test team has approved the service patch for release, consider a staggered rollout. A testing environment can go only so far. It can never truly anticipate all of the issues that may arise in a production network. In testing of Windows XP SP2, we found that, even though we had no issues during testing, once deployment began, we saw issues with desktops that had digital graphics cards installed. We needed to track down an updated digital video driver. We had not deployed to the exact same hardware on which Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

36

we had tested and we encountered unforeseen issues during the rollout. Your pre-rollout testers should be computer literate and capable of clearly reporting issues to your help desk team for resolution. Once you begin patch rollout, notify employees so that they can be sensitive to the pre- and post-patch issues they may see.

Screen shot of one available patch program, HFNetChkPro

Use patch tools to confirm compliance with policies Use patch management tools to augment your firm’s policies and to audit the effectiveness of your external vendors. Additionally you may have contractual lease agreements with external vendors to manage systems. As part of these agreements, you can request that either you or they monitor the status of patches. Further, use your patch management best practices process to scan your systems periodically between patch periods to ensure that users have not added vulnerable machines to your network. Build this procedure into your agreements, and make it part of your normal control process so that you maintain proper control of your company’s systems. Clarify polices regarding patch responsibility. As we have stated earlier, if your vendors will not allow you to patch, you must scan and monitor your system so you can isolate, protect, and defend.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

37

Ecora Software graphical view of patch status

Capture feedback Every organization has or should have formal or informal help desk capturing tools48 for tracking issues relating to the patches you roll out. Ensuring that you quickly identify issues and work with your vendors is extremely important. We have seen cases where a patch application changed the way in which a third party program interacted with a system. When this happens, begin working with the vendors. In the case of Microsoft patches, any issue caused by a security patch is a free support49 call. Red Hat tracks issues in their bug database.50 It is also wise to build up tech support relationships with antivirus vendors, firewall vendors, and line of business application vendors. Determine if they have e-mail notification services for issues with security patches. With excellent cooperation and communication, we can ensure the smoothest release process. Microsoft has put in place processes to assist in testing their patches prior to release. As one industry observer has noted, “Microsoft made good on the promise to beta test security patches with a limited number of customers and partners. By testing the patches in real-world environments, this program succeeds in increasing the quality of final patches.”51 A similar feedback process in your environment contributes to patch process success. Enforcement We assume that your corporate policy mandates that all machines should have patches installed; however, without enforcement of the policy, you systems remain exposed. Here is the rule: “Policy First; Technology Second”. In the case of security policies regarding the level of patches on systems, you can find guidance on setting and developing your policies on various web sites. The SANS.org52 Security policy page highlights the following sites: •

http://www.security.kirion.net/securitypolicy/

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

38

• • • • • • • • •

http://www.network-and-it-security-policies.com/ http://iatservices.missouri.edu/security/ http://www.utoronto.ca/security/policies.html http://irm.cit.nih.gov/security/sec_policy.html http://w3.arizona.edu/~security/pandp.htm http://secinf.net/ipolicye.html http://cio.berkeley.edu/policies.html http://www.ruskwig.com/security_policies.htm http://razor.bindview.com/publish/presentations/InfoCarePart2.html

There are also numerous examples of security policies on the SANS.org web page. The key elements you can find from these documents include: From the Server Security policy53: •

Install the most recent security patches as soon as practical, the only exception being when immediate application would interfere with business requirements.

From the Acquisition policy54: • •

Install a standard image on all hosts—servers, desktops, and laptops. Business critical production servers that cannot be replaced or re-imaged must be audited and a waiver granted by InfoSec.

From the Audit policy55: •

Scanning period: and Scanning Team shall identify in writing the allowable dates for scans to take place.

From the VPN connection policy56: •

All computers connected to internal networks via VPN or any other technology must use the most up-to-date anti-virus software that is the corporate standard (provide URL to this software); this includes personal computers

Sample Test Lab Security policy Given how critical it is to your test network, we provide an entire Test Lab Security policy referenced and available from the SANS.org site. It points out the importance of separating the Lab from the rest of the network. We highlight the test lab security policy in particular, as it is imperative that you keep your test lab from infecting the rest of your production network. Isolate it accordingly.

Internal Lab Security Policy57 1.0 Purpose This policy establishes information security requirements for labs to ensure that confidential information and technologies are not compromised, and that production services and other interests are protected from lab activities. 2.0 Scope This policy applies to all internally connected labs, employees and third parties who access 's labs. All existing and future equipment, which fall under the scope of this policy, must be configured

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

39

according to the referenced documents. DMZ Labs and stand-alone, air-gapped labs are exempt from this policy. DMZ labs must comply with the DMZ Lab Security Policy. 3.1 Ownership Responsibilities 1. Lab owning organizations are responsible for assigning lab managers, a point of contact (POC), and a back-up POC for each lab. Lab owners must maintain up-to-date POC information with InfoSec and the Corporate Enterprise Management Team. Lab managers or their backup must be available around-the-clock for emergencies, otherwise actions will be taken without their involvement. 2. Lab managers are responsible for the security of their labs and the lab's impact on the corporate production network and any other networks. Lab managers are responsible for adherence to this policy and associated processes. Where policies and procedures are undefined lab managers must do their best to safeguard from security vulnerabilities. 3. Lab managers are responsible for the lab's compliance with all security policies. The following are particularly important: Password Policy for networking devices and hosts, Wireless Security Policy, Anti-Virus Policy, and physical security. 4. Lab managers are responsible for controlling lab access. Access to any given lab will be granted by the lab manager or designee only to those individuals with an immediate business need within the lab, either short-term or as defined by their ongoing job function. This includes continually monitoring the access list to ensure that those who no longer require access to the lab have their access terminated. 5. The Network Support Organization must maintain a firewall device between the corporate production network and all lab equipment. 6. The Network Support Organization and/or InfoSec reserve the right to interrupt lab connections that affect the corporate production network negatively or pose a security risk. 7. The Network Support Organization must record all lab IP addresses, which are routed within networks, in Enterprise Address Management database along with current contact information for that lab. 8. Any lab that wants to add an external connection must provide a diagram and documentation to InfoSec with business justification, the equipment, and the IP address space information. InfoSec will review for security concerns and must approve before such connections are implemented. 9. All user passwords must comply with 's Password Policy. In addition, individual user accounts on any lab device must be deleted when no longer authorized within three (3) days. Group account passwords on lab computers (UNIX, windows, etc) must be changed quarterly (once every 3 months). For any lab device that contains proprietary information, group account passwords must be changed within three (3) days following a change in the group’s membership. 10. No lab shall provide production services. Production services are defined as ongoing and shared business critical services that generate revenue streams Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

40

or provide customer capabilities. These should be managed by a organization. 11. InfoSec will address non-compliance waiver requests on a case-by-case basis and approve waivers if justified. 3.2 General Configuration Requirements 1. All traffic between the corporate production and the lab network must go through a Network Support Organization maintained firewall. Lab network devices (including wireless) must not cross-connect the lab and production networks. 2. Original firewall configurations and any changes thereto must be reviewed and approved by InfoSec. InfoSec may require security improvements as needed. 3. Labs are prohibited from engaging in port scanning, network auto-discovery, traffic spamming/flooding, and other similar activities that negatively impact the corporate network and/or non- networks. These activities must be restricted within the lab. 4. Traffic between production networks and lab networks, as well as traffic between separate lab networks, is permitted based on business needs and as long as the traffic does not negatively affect other networks. Labs must not advertise network services that may compromise production network services or put lab confidential information at risk. 5. InfoSec reserves the right to audit all lab-related data and administration processes at any time, including but not limited to, inbound and outbound packets, firewalls and network peripherals. 6. Lab-owned gateway devices are required to comply with all product security advisories and must authenticate against the Corporate Authentication servers. 7. The enable password for all lab owned gateway devices must be different from all other equipment passwords in the lab. The password must be in accordance with 's Password Policy. The password will only be provided to those who are authorized to administer the lab network. 8. In labs where non- personnel have physical access (e.g., training labs), direct connectivity to the corporate production network is not allowed. Additionally, no confidential information can reside on any computer equipment in these labs. Connectivity for authorized personnel from these labs can be allowed to the corporate production network only if authenticated against the Corporate Authentication servers, temporary access lists (lock and key), SSH, client VPNs, or similar technology approved by InfoSec. 9. Infrastructure devices (e.g. IP Phones) needing corporate network connectivity must adhere to the Open Areas Policy. 10. All lab external connection requests must be reviewed and approved by InfoSec. Analog or ISDN lines must be configured to accept only trusted call numbers. Strong passwords must be used for authentication.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

41

11. All lab networks with external connections must not be connected to corporate production network or any other internal network directly or via a wireless connection, or via any other form of computing equipment. A waiver from InfoSec is required where air gapping is not possible (e.g., Partner Connections to third party networks). 4.0 Enforcement Any employee found to have violated this policy may be subject to disciplinary action, up to and including termination of employment. 5.0 Definitions 1. Internal - A lab that is within 's corporate firewall and connected to 's corporate production network 2. Network Support Organization - Any InfoSec approved support organization that manages the networking of non-lab networks. 3. Lab Manager - The individual responsible for all lab activities and personnel 4. Lab - A Lab is any non-production environment, intended specifically for developing, demonstrating, training and/or testing of a product. 5. External Connections (also known as DMZ) - External connections include (but not limited to) third-party data network-to-network, analog and ISDN data lines, or any other Telco data lines. 6. Lab Owned Gateway Device - A lab owned gateway device is the lab device that connects the lab network to the rest of network. All traffic between the lab and the corporate production network must pass through the lab owned gateway device unless approved by InfoSec. 7. Telco - A Telco is the equivalent to a service provider. Telcos offer network connectivity, e.g., T1, T3, OC3, OC12 or DSL. Telcos are sometimes referred to as "baby bells", although Sprint and AT&T are also considered Telcos. Telco interfaces include BRI, or Basic Rate Interface - a structure commonly used for ISDN service, and PRI, Primary Rate Interface - a structure for voice/dial-up service. 8. Traffic - Mass volume of unauthorized and/or unsolicited network Spamming/Flooding traffic 9. Firewall - A device that controls access between networks; it can be PIX, that is, a router with access control lists or similar security devices approved by InfoSec. 10. Extranet - Connections between third parties that require access to connections non-public resources, as defined in InfoSec's extranet policy (link). 11. DMZ (De-Militarized Zone) - This describes network that exists outside of primary corporate firewalls, but are still under administrative control. 6.0 Revision History Technical enforcement We have our policies, our scripts, or patch tool, but how to we enforce policy compliance? Within the world of technology vendors, we are starting to see corporations adding tools to normal network services. Both CISCO58 and Microsoft59 currently provide a Network Quarantine feature to the systems that they offer. The offerings are both Server based and IP based approaches. This technology is primarily for VPN connections at this time and is estimated to include normal internal network traffic by 2005. The concept is that a computer system must pass a predefined standard before they connect to the environment. Forrester recommends that you evaluate and explore mixing technologies, however recent news has indicated that the two major players, Microsoft and Cisco, may not be working with one another and you may have to choose.60 Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

42

61

Graphical view of Microsoft Network Quarantine Services

Pete Boden, Director of US IT Corporate Security at Microsoft, used a TechNet Radio62 session to discuss ways of ensuring that systems match the policies. At Microsoft, they perform tests that Make sure the right version of Connection Manager is being used; do that to make sure that all of the subsequent checks are in place and are current. Then check for Windows XP Professional; that’s our remote-client-connection standard OS, so everybody who connects remotely to our environment should be running Windows XP Professional. Then we check that the Internet connection firewall is enabled, that Internet connection sharing is disabled, and that the antivirus product is loaded and all of the signature definitions are updated. After making that connection, we launch a full compliance scan of the machine from our network, which essentially runs through the rest of the patch and configuration compliance checks. We make sure that the machine is fully patched; we make sure that it’s configured correctly, and if it’s not, we deal directly with the asset owner, the owner of that PC, to make sure that those checks or that those standards are then met. The compliance process starts with discovery and assessment. That means, first of all we have to become aware that vulnerability exists. Typically, we use mechanisms like the Microsoft Security Response Center, at least for Microsoft vulnerabilities. Once we become aware of a vulnerability, we rate its severity to determine how critical that vulnerability is and what we need to do to remediate. If it is a critical vulnerability, we set a deadline for compliance, which is simply a date by which every system on our network needs to comply with our patch standard. The next step evaluates compliance across the network. As we set those standards, and as we set the compliance deadline, we’re performing baseline scans across our environment to see how compliant we already are. As vulnerability information is released to our client base, many clients go out and update themselves. They are allowed to update their own system and make sure it’s compliant. At the same time we test to make sure that the patch is of high enough quality to deploy, so we provide it to a very select group of application owners to make sure that they can install that patch, run their business appropriately, and uninstall it if they need to. Then we get to the deployment phase, and this is the phase where we’re doing an enormous amount of communication with our business systems owners and clients on the network to make sure that they understand when they need to be patched by and what the consequences are for noncompliance. Once we reach that patch deadline, we drop into what we call the enforcement phase. Once we get to that phase, when Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

43

we identify a vulnerable system we take one of two measures. We either forcibly patch or configure that machine or we remove it from the network. Let’s stress this point again. These computers are your company assets; they do not belong to the employees. Thus, setting policies and compliance is critical to maintain control. Set the policy to make it easy to update and comply with On the patch management listserve, people often ask for a template of all needed patches for a workstation or server. What is disconcerting in this question is that there is a desire to rely only on human interpretation of what is currently installed on a system. It’s much better to have a combination of human expectation based on a review of the published bulletins, and a reinforcement of this awareness by using a patch tool to confirm that “yes, you need to patch for this.” While this book is sponsored by a patch management software solution vendor, we cannot stress this point enough so we will make it loud and clear: If you rely solely on human judgment for patching your systems, your risks are significantly increased and your rate of failure is extremely high. Whether you use free patch tools or develop your own scanning methodology, relying solely on judgment and not following up with some sort of tool or technique to confirm your patch status is extremely foolhardy. Patch and then CONFIRM the application and effectiveness of the patch. In the windows environment confirm by checking registry keys or current, in-use DLLs to ensure patch status. Utilities such as tlist, ListDLLs63 or Process Explorer64 assist in this process. Stuff happens Even with all of our testing, things happen. Therefore, ensure that you have good vendor relationships. There is nothing so frustrating than the “ping pong” effect: Vendor A blames an issue on Vendor B; Vendor B blames Vendor A. The more a specific vendor depends on your business, the more power you have. While migrating to different vendors is not easy, the fact that we are in an increasingly competitive marketplace helps greatly. On listserves and newsgroups, we repeatedly see people post issues that they, clearly, could move to the vendor for resolution. Repeatedly they are reluctant to do so. If cost is an issue, remember that calls to resolve issues caused by a Microsoft security patch are free. When contacting any vendor, gather the following information: •





Details about the computer exhibiting the problem o Version installed o Service packs applied o Affected application ƒ Version ƒ service pack o Other applications on the box o Antivirus software, version o Processor o Memory o Hotfixes Installed Details about other computers (a client machine, for example), if applicable o Operating system and service pack o Applications o Antivirus software, version Details about the network, if applicable o Internet connection type o Number of NICs in the server o Firewall/proxy o Network topology particulars

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

44



Details about the issue o Full, complete error message wording or screenshot o Timing of first occurrence o Changes made around that time o Steps taken to reproduce the error, if any o Additional information you think might be helpful

There is a cartoon showing Bart Simpson writing on the blackboard, “I will search Google before asking dumb questions.”65 You would be surprised how many people do not know the power of the search engine. You can begin your error investigation process by using one of the greatest diagnostic tools around: A search engine such as Google. Launch the Google search engine66 and put in your "exact" error message. You may be surprised that someone else had your same problem and someone else has helped him or her find the answer. If that does not work, try searching on Google Groups67, which is the search engine of the Usenet. To narrow your search further, click on “Advanced” and limit your search to microsoft.com or redhat.com or the appropriate vendor’s domain. For narrowing down Usenet postings, click on advanced and include a wildcard, as in Microsoft.*, which will search on the public newsgroups on the Microsoft domain. Lastly, after you review the Event viewer, the web site of www.eventid.net is invaluable for quick resolution on security or even non-security issues. Subscribe to the service. Those red stop signs in your event viewer and the resulting code can point you in the direction of resolution. Be willing to gather information Without data, your vendor is dumb and blind to the issue you are seeing. If your system crashes and asks to send a “Dr. Watson” dump to Microsoft, for example, send it. The information sent to Microsoft or any vendor assists them greatly in pro-actively preventing issues. You may find that doing a Dr. Watson error gets you a link to additional information for resolution. Next, have on hand the tools that Microsoft Product Support or other vendors use in diagnosing issues. Tools such at the PSS support tools68 allow engineers to more easily see what is going on. The support engineer may also want you to use debugging programs69 to prepare a dump file for analysis. Use additional tools from the Server resource kits70 as needed. The more information you provide the faster and better will be the resolution. Finally, when you call in for support to any vendor, make sure that you get to the right division or department. Online communities offer another way to compare patch experiences. Phrase your subject line carefully when sending a question and describe the steps that you have taken. Include the relevant information to make the resolution process faster. Find additional guidance in asking effective questions the right way at http://www.catb.org/~esr/faqs/smart-questions.html Remember: There are no dumb questions. Thinking it is a dumb question is just dumb because you will not find an answer until you ask the question. Always ask, as someone else probably has the same question that you have. Change Management follow-up You’ve patched and you’ve enforced the policy to ensure that everyone patches. So now you are done, right? Wrong. First, you need a change review process to summarize and gather those items that testing missed. Can you cover an emerging issue in future testing procedures? Can you better follow up with vendors and personnel? Did the patch affect one application in particular? Use this review process to learn and make the next patch process easier.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

45

Microsoft’s representation of the Change Management process

Determine which tools did the best job of recreating your network. If there were issues with patches, what backup or imaging procedures worked the best? Did you ensure that during this process, all appropriate change control mechanisms were in place? Use this process to learn and refine your procedures as needed on an ongoing basis. So I can’t patch, now what do I do? There will be times when patching is either not an option or, in the case of zero day exploits wherein the exploit is released to the web before a patch has been built. You might find that you need to take alternative actions. Nevertheless, be sure to analyze carefully the true risk. Many businesses that didn’t patch for 03-026 erroneously believed that they were safe because they had a firewall; however, all MSBlaster (03-026) needed to shut down a business was a remote connection in from an infected remote client. The recent addition of the XP SP2 firewall71 controlled from the INSIDE of a company helps limit internal exposure. If contractual arrangements restrict you from patching, consider “air gaps” or other devices to limit and restrict TCP/IP traffic. Vlans and virtual machines may not be enough to limit such traffic as switches can be vulnerable devices as well. You can also use IPSec,72 which is “an implementation of the Internet Engineering Task Force’s Internet Protocol security (IPSec). IPSec, which is also included in Windows 2000 and Windows XP, provides network managers with a key line of defense in protecting their networks. IPSec exists below the Transport layer, so applications transparently inherit its security services. IPSec provides the protections of data

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

46

integrity, data origin authentication, data confidentiality, and replay protection without having to upgrade applications or train users.” IPSec provides defense-in-depth against vulnerabilities in upper-layer protocols and applications by protecting upper layer protocols, services, and applications. With IPSec enabled, initial packets access an application or service running on a server only if they meet the allowed policy. For example, attempts to disable applications or services on servers must first penetrate the IPSec protection. In addition, because packets will not be passed to the application or service until trust has been established through IPSec authentication and the configured protection on packets for the application or service is protected. Use IPSec policy to restrict access to servers. You can configure a server to accept specific types of traffic only. For example, you can configure an e-mail server to accept only secured e-mail traffic from client computers. The e-mail server discards all other traffic from client computers. What is the best way to find alternative to patching? Turn first to the security bulletins themselves. Using our Security bulletin 03-02673 as an example, refer to the technical information section. Scroll to the “Mitigating factors” area located in each bulletin. In this case, mitigation involved blocking the affected ports. This is easy to do on a stand-alone machine, but hard to do in a network.

The story of two patches Two patches serve as examples of “major events” for IT professionals. Both were highly critical, many administrators pushed both patches out to their network EXTREMELY fast. Moreover, while one turned into a network event, the other seemed not quite the Internet threat that it initially appeared to be. Let’s look first at the MSBlaster security bulletin 03-02674 to review why it became an event: Security bulletin 03-026 - Buffer Overrun In RPC Interface Could Allow Code Execution (823980) MSBLAST Issue Impact of vulnerability

Result Run code of attackers choice

Severity rating

Critical

Threat vectors

File and printing ports

Comment “Run code” is translation for “oh this is going to be bad” Also critical on ALL platforms [sometimes Windows 2003 has mitigation issues when there are security issues for Internet Explorer due to the IE enhanced lockdown” Network administrators thought they had more control over the parameters of their network. We don’t. Plan accordingly.

Regarding Blaster, we knew that our firewall protected us on the outside. What we did not take into account was how interconnected we are. The vulnerability for 03-026 spawned a worm called MSBlaster. This worm got inside of our networks because of VPN and other remote connections. It took down firms and put their servers and workstations offline. Even today, we remain at risk on the inside; we cannot depend on our network perimeters alone. In-depth defense through

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

47

compliance with patch policies would have prevented MSBlaster. Now let us look at our second sample patch:75

Security bulletin 04-011 - Buffer Overrun In RPC Interface Could Allow Code Execution (823980) SASSER Issue Impact of vulnerability Severity rating

Result Remote code execution Critical

Threat vectors

Primarily File and printing ports

Comment Hmmm… this also sounds bad But here’s the difference – NOT critical on Windows 2003 servers Primarily affected computers without up to date firewalls and antivirus

Of all the issues addressed in this bulletin, the malware targeted only the LSASS service. The mitigation of this one was clear: An anonymous user can remotely attack Windows 2000 and Windows XP only. While Windows Server 2003 and Windows XP 64-Bit Edition Version 2003 contain the vulnerability, only a local administrator could exploit it. Given reviews of security issues, both patches deserved very quick installation, especially when administrators compared notes and looked at the threat vectors. Many thought that the Sasser worm would have exploited the SSL vulnerability instead of the LSASS service. Nevertheless, each case highlights the same thing: The time between patch, exploit and event is shrinking. The coming threats Patching is just one source of protection. Forthcoming network protection technologies are just the latest in the arsenal of tools that we must use to protect ourselves. Remember: it’s not just about having a tool for patching; it’s about controlling the patching and enforcing patching best practices. If you do not control your systems, they will control you. We will complete our technical discussion of patches by looking at what lies ahead. A Microsoft article76 seeks to: • Establish a single set of patch related terms and mandate their use in all documentation related to patches as described in knowledge base article 824684 • Establish the role, format, and content of knowledge base articles published in conjunction with patches. The KB article will focus on installing the patch, while the bulletin will focus on the vulnerability. as described in knowledge base article 824689 • Establish a common naming convention for all patch patches, providing information such as the product the patch applies to, the language the patch was developed for, as described in knowledge base article 824685 • Establish the standard that all patches use a properties page as discussed in knowledge base article 824686 • Establish a single consistent way of recording a patch in the add/remove programs list. Microsoft states that they are examining whether it may be appropriate to develop a repository of information dedicated solely to patches • Establish a standard such that all Microsoft patches will use either of two installer technologies: o Windows Installer [MSI] will be used by patches for applications o Update.exe will be used by Windows operating systems and some applications • Establish a standard of patch size reduction by removing debugging symbols from patches. {If you receive a hotfix from Microsoft, you will notice for some hotfixes “symbol” files inside the zipped hotfixes. • Establish a uniform set of options that installer technologies will support as discussed in knowledge base 824687

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

48

• • •

Establish a standard for uniform set of return codes and log entries that all patch installers will use Establish a standard allowing for uninstalling of all patches Establish a standard of placing patches in a standard registry location

So what does the future hold for patch technology? Standardization is likely, and it can come none too soon. As we begin to mix operating systems, applications, and platforms, standards must emerge and evolve. Dig deep in your applications and work with your vendors to understand how your systems work, to examine your systems and to understand how to patch them and how to test the patches. Does this testing replicate your environment? The larger and more complex your environment, the more ability you have to ask vendors to support and test patches before you roll them out. Include such specifications in your vendor agreements. Begin to push, slowly at first, so that your applications support the latest service packs within a reasonable period. “Demand better security from vendors and hold them responsible. Use what you have, and make sure you know how to use it properly and effectively”77 Ask your vendors to step up to the plate and support and test patches as well. The bottom line for your boss If you are reading this, chances are you are not the CEO or CIO or CFO. You probably do not have budgetary control and the final say. Refer to the one-page appendix at the end of this chapter. Copy it, hand it to the people who make the decisions in your firm. Take the time to calculate the ALE formula. Discuss the impact of how the biggest security impact comes from laptops that may not get the attention they need to maintain patch level. Take the time to review the business side of patching to compile arguments you need to get the tools you must have. Effective patch management is all about good process and good control. If you do not control your server, your desktops, or your infrastructure; if you do not control who has access to your network; and if you do not control what comes into and goes out of your network, you don’t control anything. It is not YOUR network anymore. The World has changed and computing environments continue to evolve. Companies that may have considered security a minor compliance issue must now meet security standards. California already has a law on the books commonly referred to as “SB138678.” This law requires a firm to notify California residents when an unauthorized party views data sensitive to identity theft. While this is an admirable law from the standpoint of the person susceptible to identity theft, it is a potential liability and public relations nightmare. Management often requires ROI calculations on new processes and purchases, but they often ignore intangible costs. The traditional ALE equation may not give you the entire cost analysis you need but it may be all you have right now to make your point. CIO Magazine79 has predicted that the insurance industry will push for Security ROI and better industry statistics, but that has yet to happen. We are still moving toward what Kevin Soo Hoo described as “a place where computer security risks would be ‘actively and effectively managed.’”80 Do you need to be empowered to take control of your network? Refer to the US CERT research center for evidence on the survivability of the Internet.81 Consider, also, that Incidents.org, which track events on the Internet, calculate that a Windows XP workstation can run on the world wide web without protection for only twenty minutes.82 Ask yourself, if your Help Desk staff cannot get to that system within that period and your employees control their systems and not your IT staff, who really controls your network? It is certainly not your IT staff! Where is your weakest link? Look toward the person who has the ability to download what he or she wants to. Patch

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

49

management is part of the process of having your IT staff stake a claim for once again controlling YOUR network.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

50

Chapter 5 – Life beyond patching Microsoft

T

he reality of what we protect today includes products beyond those developed and supported by Microsoft. Third party software we monitor for updates included products from Apple, Macromedia, Real Networks, Adobe, WinZip, and many others. It even includes tracking vulnerabilities in antivirus agents. Many firms also track Solaris, RedHat, SuSe, Oracle, Cisco, and even vulnerabilities in devices and printers from vendors such as Ricoh and HP. While this text is about the concepts of patch management, one thing is clear. You must clearly understand what you are patching, know what information resources you have available and, just as in our extensive discussion of Microsoft patching, arm yourself with an environment and a team that is able to test, report findings, maintain good relationships with vendors, and manage Service Level Agreements tied to your goals. While this chapter covers some new ground, you will find that it will reinforce many of the concepts we have talked about before. Assessment of your network, testing procedures, policies in place, reporting of patch issues, all of these processes are identical in any software platform you maintain. The firms and service level agreements (SLAs) may be different, but the basic concept of “test to ensure you do no harm” is universal. Sun Tzu in The Art of War states, “If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle.” We would argue that in the battle for better security, we are still working our way towards knowing ourselves. If you have not yet determined the critical assets you need to defend and protect, you need to do this analysis first. Just as in our previous chapters, we assume that you have identified that the software needs a patch according to the vendor. No matter what platform, spending resources to patch, putting a business continuity plan in place, ensuring that patches will not harm systems, arranging for notification of new patches from vendors, and obtaining patches conveniently are mandatory best practices. Over the Christmas holidays, a worm exploited vulnerabilities in phpBB (an Open Source Bulletin Board or CMS) and defaced web sites. Later deemed the “Sanity” worm, it points out that every platform is susceptible. Irregularly maintained and unpatched systems are the most common denominators among successful attacks. To prevent attacks we need to patch and to patch we know we must test. Recall the basic procedures used to test patches: • • • •

Best—an exact replica (ghost) of your exact system on the exact same hardware, applying the patch to the ghosted system and monitoring processes Better—applying the patch to a mirror of the system, purposely breaking the mirror and monitoring results Better—applying the patch to a Vmware or virtual server (virtual PC will suffice with a bit of a workaround, but it is directly supported on Vmware and Virtual server platforms); monitoring applications and processes Applying the patch to a test server/workstation and monitoring the applications and processes

As always, have a testing matrix and sign off procedures. Our policies and procedures for testing and approving patches are going to be the same no matter what the platform. Identify what you have and what you need to protect; remove—do not just disable—what you do not need running; run the least amount of privileges on files, folders, servers, and workstations to limit exposure;

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

51

and audit to confirm exposure to risk. These basic concepts apply to any platform you choose to run. Patching third party applications Let us begin this chapter by examining procedures we can use to patch (or actually re-deploy) sample third party applications on the Microsoft platform. We will then discuss procedures to patch operating systems such as Linux, Solaris, Oracle, and others. For many of you, this section, hopefully, will be ‘old hat’ and just a review; but if you have not used Group Policy to deploy applications, note that patching (or deploying) a third party application on the Windows platform involves preparing a transform file and utilizing active directory. On the domain that needs to deploy an application the basic procedure is as follows: • • • •

Create a distribution point Create a group policy object for the deployment Deploy the .msi file from the shared distribution folder as machine-assigned. You can also deploy the .msi to specific security groups.

Here is an example of deploying Adobe Acrobat reader to all workstations. The concepts mimic those detailed in the white paper “Using Group Policy to Deploy Windows XP sp2” (http://www.microsoft.com/technet/prodtechnol/winxppro/deploy/gpdepsp2.mspx) with one exception. For some applications, you must prepare a file for distribution as an .msi application. In the case of Adobe, this involves a little “trickery” of pretending to install the program. When you download the full file, it is actually an executable shell around the msi program. To “expose” the .msi, you merely go through the motions of having it begin the install and then make sure you “cancel” at the proper moment.

Continue the installation “past” this screen

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

52

At this point, select “Cancel” You will now have on your local workstation system a folder that includes the “expanded files”. Copy these files up to the server installation point and the distribution point that you will use now that you have the actual msi file.

Copy these files to your “distribution” point To create a distribution point83 1. Log on to the server computer as an administrator. 2. Create a shared network folder where you are going to put the Microsoft Windows Installer package that you want to distribute. This folder is the distribution point for the software package. 3. Set permissions on the shared network folder to permit access to the distribution package. 4. Give access permissions to the following: administrators, authenticated users, and domain users. (Remember that on Windows 2003, the “Everyone” group no longer includes anonymous users.)

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

53

Setting permissions for deployment 5. Configure distributed file system (DFS) for the distribution point in larger enterprise settings. This provides you with more flexibility by ensuring uninterrupted availability of the distribution point, in case you have to replace the server. In addition, DFS makes it easier to have distribution points in multiple sites. For more information on DFS review the information at http://go.microsoft.com/fwlink/?linkid=34229 To create a Group Policy Object (GPO) for deployments84 1. Begin by ensuring that you have a copy of the Group Policy Management Console installed on a XP sp2 workstation (preferably) that has the recent updated administration files. 2. Download this console tool from http://www.microsoft.com/downloads/details.aspx?FamilyId=0A6D4C24-8CBD-4B359272-DD3CBFC81887&displaylang=en. However you can control group policy from a non-XP sp2 location with a hotfix that will be included in Windows 2003 sp1. Download the hotfix from http://support.microsoft.com/default.aspx?kbid=842933 3. On an administrative workstation in large environments, log on as the domain administrator, or in smaller environments, remote desktop to the server and open Group Policy Management Console (GPMC). This is typically done by typing in “gpmc.msc” in the start, run box as shown:

Launching the Group Policy editor 4. In the console tree, right-click the domain name in the forest in which you want to create and link a Group Policy object (GPO). 5. Click Create and Link a GPO Here.

Creating a Group Policy Object 6. In the New GPO dialog box, specify a name for the new GPO, and then click OK.

Preparing the new GPO Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

54

To edit a GPO for Deployments 1. Right click the new GPO and click Edit.

Preparing to edit the Group Policy Object 2. In the Group Policy Object Editor, click Computer Configuration, click Software Settings, and then click Software Installation. 3. On the Action menu, point to New, and then click Package.

Preparing to deploy the package [msi] 4. In the Open dialog box, in File name, type the full Universal Naming Convention (UNC) path of the shared installer package that you want to distribute. Type this path in the following format: \\ServerName\SharedFolder\Update.msi or \\ServerIP\SharedFolder\Update.msi. Make sure that you use the UNC path of the shared installer package.

Ensuring that the package can be retrieved across the domain by using the UNC path 5. Select the Windows Installer package, and then click Open. 6. In the Deploy Software dialog box, click Assigned, and then click OK. The shared installer package that you selected appears in the right pane of Group Policy Object Editor.

View of the resulting deployment To target the msi using security filtering 1. In GPMC, double-click Group Policy Objects. 2. Click the GPO to which you want to apply security filtering.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

55

Selecting the GPO 3. In the results pane, on the Scope tab, click Add. 4. In “Enter the object name to select,” type the name of the group, user, or computer that you want to add to the security filter, and then click OK. 5. If Authenticated Users appears in the Security Filtering section of the Scope tab, select this group and click Remove. This will ensure that only members of the group or groups you added can receive the settings in this GPO. If you leave this group in, staff logging into your servers will deploy the application to the servers as well. Set ONLY those groups that you want to deploy the software to in this window. Click on the settings tab to view the resulting effects of your deployment and to view the settings that the group policy will use to deploy the package:

Resulting Group policy screen It is important to realize that testing a rollout or deployment or patch methodology is very important. Businesses have suffered dearly when scripts and group policy commands did more than expected or intended. Prepare a test lab and test the process. To force the group policy application, from a command prompt at the server, enter gpupdate/force (for Windows 2003/XP machines)

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

56

GPupdate/Force In the case of “scripted” installs, you may want to add additional features to ensure that the deployment of the software is “silent,” that is, the end user need not interact to apply the software update. Typically, additional scripting ensures that the install is transparent to the end user. For additional information and guidance for using “scripted” options, look at the Tech Republic web page at http://www.techrepublic.com and in the case of Adobe you can further link to http://techrepublic.com.com/5100-6270-5054963.html. If you have prior versions of Adobe, you may need to adjust the script to uninstall the prior versions. For other applications, you can typically ask fellow administrators on the PatchManagement.org listserve (www.patchmanagement.org) for guidance or Google for recommendations. There is also additional information for using the Microsoft Command line and the Windows installer here: http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/msiexec.mspx Vendors have their own resources for enterprise deployment. The Adobe web site includes the following information for Terminal service installations, Citrix installations, and general enterprise deployment: http://partners.adobe.com/public/developer/acrobat/index_advanced.html Review your vendor’s technical guidance for enterprise deployment scenarios and suggestions.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

57

Patching Linux distributions A patch in both Windows and Linux products is a band-aid, a fix for a missed security issue or an interim update prior to a service pack release. In the Linux platforms, it is typically easier to avoid patching by not installing the application in the first place. Recent movements in the desktop applications of Linux, however, are beginning to load with “default” applications that need patching and maintenance. For example, a recent Novell/SuSe 9.2 desktop needed several critical patches. Most of the “enterprise” ready desktop versions of Linux distributions are beginning to have patch update tools built into the operating system. For example, both RedHat and SuSe come with a “windows update” style of automatic update that is useful for the desktop. The SuSe distribution requests administrator privileges before scanning for patches.

SuSe prompting for administrator rights before patching Do not, however, “keep password” on the system to maintain separation between the user account and the Superuser Do (‘sudo’) account. Once the administrator password is applied, the system begins the scan and reports on the patches needed.

Yast Online Patch window indicating critical patches needed by system Obviously similar to Windows update, these workstation auto updates work best for standalone workstations or small networks rather than the large enterprise settings. For those enterprise environments, the firm truly needs patch tools and scripts to ensure best practices. The savvy Linux administrator has an additional option. The more advanced Linux administrator will download the code and patches in source code form and compile them into a form that he will use to patch the systems before waiting for the vendor to release a patch. This manual approach should be reviewed to ensure that the procedure does not void any service line agreements (SLAs). This review takes significant manual effort to keep track of installed components and what is patched versus what needs patching. As Linux distributions proliferate into the Enterprise environment, a growing number of firms support patch distribution and inventory tools for Linux. Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

58

Here are two vendors with Linux patch management tools as a sample: • •

Shavlik with their Shavlik HFNetChkPro for Linux 2.1 – for RedHat Novell with their Linux Management console found here http://www.novell.com/products/zenworks/linuxmanagement/

When you begin a support agreement or other support arrangement with a commercial Linux vendor, discuss all the patch management tools they provide with the package itself. Manual patching and chasing down security patch information is at best tedious for any operating system. Patch information is available from each vendor. For example, each vendor normally has an email/newsletter service. Find samples at • • • • • • •

http://www.suse.com/en/private/newsletter.html http://lists.suse.com/archive/suse-security/ http://lists.suse.com/archive/suse-security-announce/ https://www.redhat.com/mailman/listinfo/enterprise-watch-list/ http://lists.debian.org/debian-security-announce/ http://www.gentoo.org/main/en/lists.xml http://www.mandrakesoft.com/security/advisories

Among others, review the offerings from the specific alternative operating system around which you choose to build. Here too, as was the case in our Microsoft-specific chapters, standardization on one platform may go a long way to helping a firm manage “change management.” If your firm had to keep up with patches for Microsoft, SuSe, RedHat, Debian, Mandrake, OpenBSD, and others, you would either need to subscribe to a notification service provided by a company such as Secunia (www.Secunia.com) or devote hours to keeping up to date. Gaining core competency surrounding a limited number of vendors is central to managing the “patch beast.” Remember that patching is mandatory for any software application. In late December 2004, the “Sanity” worm affected web sites that included code from phpBB web forum. Some other anecdotal issues in the Linux world include: • • •

The “Ramen” worm shutdown print servers. In 2000, well-known vulnerabilities in the ToolTalk database server compromised Solaris systems. “Slapper worm” attacked Apache web servers.

We could go on with more such examples, but the bottom line is each platform has vulnerabilities and needs patching. As we have stated before, the risk of introducing new code into a stable system could jeopardize that platform. Destabilizing a system will mean loss of productivity and revenue, and we need to weigh the risk against the benefit. Even after applying patches to a system, some firms wait for a “burn in period” or review patch community results before applying the patches firm wide. Vendors cannot test for all possible combinations of hardware and software and business practices and thus you must prepare your own test policies and procedures before implementing patch deployment. As in any introduction of new code, you may also want to have a “burn in” period to ensure that the patches work properly in your unique system. Begin by ensuring that you have secured your systems. The Center for Internet Security (www.cisecurity.org) has benchmarks for RedHat, Solaris, HP-Unix, and various other systems and provides auditing tools as well. Bastille (http://www.bastille-linux.org/) is another tool that

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

59

provides software-hardening information to help you proactively harden your systems against attacks. Again, it is imperative that system hardening of all operating systems complement patch management best practices. Unlike the Microsoft operating systems, Linux system administrators have the option of manually compiling patches rather than relying just on the vendor. However, this process is not a trivial one and you may wish to wait for the vendor complied packages due to unique changes with each distribution. We will discuss this later in this chapter. Ensure that whatever method you choose for patching is in line with both your operating systems and your applications. Again be aware of your vendor service level agreement requirements for commercial packages. While Linux distributions are based on the open source platform, it is your agreements with your line of business applications that are reliant on the software that you should pay attention to the most. If you have a commercial application built on a Linux distribution, has your application vendor approved the patch the question you should be monitoring the closest. You may not have any such agreements in place to worry about and can patch on your own schedule with your own methodologies. As we have previously pointed out, while the patch management process consumes time and resources to identify vendors, to develop patch testing resources and methodologies, the price of not patching is very high. Costs of loss of productivity, of reputation, and these days the potential for legislative requirements to ensure that reasonable security procedures are followed all add to the “costs” of not patching. For the RedHat distributions prior to the “Enterprise” versions, RPM (or RedHat Package Manager) files are the released updates, configurations, and documentation. Packages for Athlon and Intel processors appear along with a folder called “noarch” which apply to all systems regardless of the hardware platform.

RedHat 9 RPM locations For RedHat Enterprise, the packages organize as follows: • AS (Advanced Server) is the RedHat top of the line package for large enterprise servers that can support up to 16 CPUs and 64 gb of memory • ES (Enterprise Server) supports two CPUs and 8gigs of memory. • WS is the workstation version of RedHat for highly technical needs. • Desktop supports a single CPU.

RedHat Enterprise packages Each shop must decide whether to wait for vendor-supported packages or introduce other means for patching. Also, keep in mind that obtaining the updates by browsing to the site ftp://updates.redhat.com includes non-security updates. Before you begin the process of downloading packages from either the RedHat website or any other website, ensure you are using a trusted site and only use trusted sites. Earlier in 2004, a bogus RedHat security email attempted to entice people to download a Trojan.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

60

Compare the PGP signature (do you cover or point to documentation to support PGP on SBS or exchange and Outlook?) for emails and md5 checksums of the software you receive. For example when downloading the Fedora version of RedHat, compare the following checksums to the files downloaded: Hash: SHA1 2786a751df919f340d967a4833b63b16 0b9f23eff713db3a0d5a778e38d70299 Bd15bac386cdc30b6fcdf8e7da633403 D89ea5afcb3b94cbfaa3c275455d45a4 D177e8134ffff13e0029fe31a9f20c85 B61b0eb7e0171837aeeff4f0054a4d79 99dc12c7e8a93844a48a5675a9c07ec9 399b7ffd721ebb4244a02c34cdbb1b82 F58b1de3b880df55dbbd37d143419226

FC3-x86_64-DVD.iso FC3-x86_64-SRPMS-disc1.iso FC3-x86_64-SRPMS-disc2.iso FC3-x86_64-SRPMS-disc3.iso FC3-x86_64-SRPMS-disc4.iso FC3-x86_64-disc1.iso FC3-x86_64-disc2.iso FC3-x86_64-disc3.iso FC3-x86_64-disc4.iso

Compare the PGP signature to the RedHat public key at http://www.redhat.com/security/team/key/. To confirm a RedHat package, run the command as follows: rpm --checksig -v .rpm The process of ensuring that checksums match, confirms that you are not in a similar situation as individuals who downloaded software tools from file sharing locations that contained Trojan horses in the source code and thereby infected their systems with backdoors. Individuals who didn’t verify checksums introduced bad code into their systems. Thus follow “good common sense” when downloading patch sources. You may accept the risk for your less critical machines, but you may not want to accept the risk for more critical machines. The RedHat command can be used to determine what version of the package you have on your system. #>rpm --query –all or rpm –qa can show you a listing of packages installed. By ‘piping’ the ‘grep’ command, (rpm –qa | grep ) it is possible to easily locate particular applications.

Sample of querying for all software Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

61

You can also use this to query a specific package such as #>rpm –q openssh

Verification of Openssh package, please note I am logged in as “root” which should be limited and only done by administrators. In general, to update a package you need to follow instructions:85 To update all RPMs for your particular architecture, run: #>rpm -Fvh [filenames] where [filenames] is a list of the RPMs you wish to upgrade. Only those RPMs which are currently installed will be updated. Those RPMs which are not installed but included in the list will not be updated. Note that you can also use wildcards (*.rpm) if your current directory contains *only* the desired RPMs. Please note that this update is also available via RedHat Network. Many people find this an easier way to apply updates. To use RedHat Network, launch the RedHat Update Agent with the following command: #>up2date this will start an interactive process that will result in the appropriate RPMs being upgraded on your system. If up2date fails to connect to Red Hat Network due to SSL Certificate Errors, you need to install a version of the up2date client with an updated certificate. The latest version of up2date is available from the Red Hat FTP site and may also be downloaded directly from the RHN website: https://rhn.redhat.com/help/latest-up2date.pxt You can verify each package and see who signed it with the following command: #>rpm --checksig -v filename If you only wish to verify that each package has not been corrupted or tampered with, examine only the md5sum with the following command: #>md5sum filename To verify software integrity you can use the switch of rpm – verify or rpm –v to check packages against the local database.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

62

Verification of files To install software you can use rpm – install, rpm – upgrade or rpm – freshen. The ‘install’ flag ensures the new file is loaded but leaves the old version installed, the ‘upgrade’ flag removes a previous version that might exist, and the ‘freshen’ flag removes only the prior version of there is an older version on the system. A “defense in depth” measure is to not only disable a service but also remove the software completely, thus if you determine that you do not need a package on your system, rpm – erase is recommended to remove the software completely. Kernel updates should only be done with the ‘–I’ flag. To review what Kernel level you have on your system type in rpm –qa |grep kernel

Verification of Kernel on RedHat Enterprise 9 This will show the kernels installed on the system. You may also consider the #>uname –a command to determine which kernel is currently loaded into memory and running. Modern Linux distributions support either GRUB or LILO as a boot manager. This permits a quick and easy ‘rollback’ by just rebooting the machine, and choosing a previously used kernel. In addition, for those using RedHat/Fedora, up2date is phasing out. The replacement, called “yum,” evolves from Debian’s update utility. This new tool helps to eliminate some of the issues surrounding RPM’s poor dependency resolution problems. RPM has occasionally installed applications rather than libraries to resolve library dependencies. This just added another application of questionable value, but one that needs maintenance for patches, updates, and revisions.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

63

Another element to keep in mind is that the files downloaded by these tools can be stored in a shared directory. When other machines mount that directory, up2date need not be run on all of the machines, just the installation process. This is extremely beneficial to updating multiple workstations. Usually an administrator will not have several servers to update that all require the same program updates, this naturally excludes cluster administrators. For less skilled administrators, we still recommend using a more commercial package such as RedHat or SuSe that has a scan and update feature. For example, even on RedHat’s Fedora package, the up2date utility showcases that a recently installed ISO still needs updating:

RedHat Fedora update notification So taking our example, we first visit the Fedora RedHat web site to get the update at http://download.fedora.redhat.com/pub/fedora/linux/core/updates/ and download the needed packages we need so that we can manually update the Omni-0.9.1-7 to the more current version found on the Fedora web site to Omni-0.9.2-1.1. We would of course first check for dependencies; when we launch the download we allow the package to check for itself.

RedHat preparing system update Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

64

We can see from the next screen that the package that it depended upon was not available in the system and thus it needed another rpm package first before updating the file:

Fedora indicating dependency on another RPM package Once you have found the dependency, you run the rpm –Fvh command with the series of files together to begin the patching process

Installation of files on RedHat Fedora We have now successfully updated these files and can confirm their installation. This process can be a bit tedious at best. Thus, patch management tools for any platform that you protect can easily pay for themselves in person-hours, in more automation and confirmation. Patching for any platform is not easy and takes skill. Using advanced tools to automate this process are highly recommended.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

65

Building packages from source Because Linux distributions are modular, there is not a “rollback” feature per se, but rather redistribution, a reapplication of the necessary package to “roll back” to a working version. Besides waiting for a distribution package from a vendor, you may wish to build from source. Steve Friedl, a Security consultant, has provided detailed instructions on how to build from source packages on his web site at http://www.unixwiz.net/techtips/building-source.html. You can review the steps for more detail. Another process for distribution in the open source world is called the Gentoo distribution is gaining popularity in certain circles because the entire OS/distribution is custom built on the machine. The packages are downloaded and compiled by the installation program. The disadvantage to this is that it can be extremely time-consuming; however, the benefits include knowing exactly what has been installed on the machine, no more or less than the files needed for the system to execute its assigned tasks. This distribution is useful for environments where peak performance and/or heightened security are needed. In an enterprise environment, (large or small) Gentoo is best when deployed to the same hardware on or for which it was compiled. Clusters are a good example, as are machines purchased in ‘groups’ for team environments such as call centers. Unfortunately, due to the time involved in developing a fully optimized single machine image, this would not be a recommended solution for environments with many systems from different vendors, particularly desktops. As Steve Friedl states in his Technology tip86: • • •

It insures complete control over all the build options, including the installation pathnames. Using a pre-built package with options you do not need (say, an LDAP client) adds to bloat. It insures that cooperating packages are built with compatible options: if Postfix and Courier-IMAP both need to link with MySQL libraries, building them all yourself guarantees that the same compiler was used for all of them, with flags you can see. You develop a reusable skill that applies anywhere you install the package: apt-get is great for Debian but it won't be too helpful on Solaris or HP-UX.

Dependencies As with many software patches from many vendors, there are dependencies that must be followed to install software. You can use rpm–qf, for example, as the anaconda package is not dependent on any other package. We saw in our own example that our sample patch was dependent on another package, testing a more innocuous package indicates it does not have a dependency.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

66

Sun Solaris Sun Solaris recommends the following in their patch management strategy “The Recommended Patch Cluster reflects the current version of all of the patches required to address all Sun Alert issues. Some customers have adopted a policy of periodically applying the current Recommended Patch Cluster to a system. While this strategy does address all Sun Alert issues, it also introduces more change to the system than is necessary. Similarly, reapplying the current Recommended Patch Cluster on a scheduled basis is also not necessary. 87 For Sun Solaris, a patch report file is located at ftp://sunsolve.sun.com/patchroot/reports/public_patch_report. This lists the recommended and security patches for Solaris as of any given date.

Example of Sun Solaris public patch report The Sun Alert Pack reflects the lowest revision of all of the patches necessary to address all Sun Alert issues, and the least possible change to your system while still addressing all Sun Alert issues. Sun believes that you should apply patches only to address specific issues or needs Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

67

unless critical. According to Sun, you should not apply patches merely to keep current. There is no benefit to applying the latest revisions of patches without understanding whether those patches provide any value. Patch management tools for Solaris include • Sun Patch Manager 2.0 http://www.sun.com/software/download/products/40c8c2ad.html • Solaris Patch Manager 1.0 http://www.sun.com/service/support/sw_only/patchmanager.html • PatchPro - http://patchpro.sun.com/ • Traffic Light Patchtool - http://www.sun.com/service/preventive/index.html. • Patch utilities Coda We hope that this very brief overview of the methods and options you have when patching nonMicrosoft systems shows you that above all else you need the following key information no matter what the platform: • • • • • • • • • •

Information from the vendors for the patches needed for security issues You need the software patches You need a test network set up You need to test the patches against installed applications that you need to ensure continue to run and perform as expected. You need to ensure you have a backup of production systems You need to ensure you have a rollback plan in case the patch does not work even in the production setting You need to confirm that the patches have been applied You need a review process You need to start the process all over again each time a new patch comes out. No matter what the platform, if you have “it’ installed, patch “it.”

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

68

Chapter 6 – Justifying Patch Management—A Business Tutorial

H

e has his laptop. He has printed, bound, and carefully reviewed his customized PowerPoint and he is finally off to present to one of his biggest prospects! The day is cool; the weather perfect the feel of a sale is in the air! The last eight months have been grueling; this prospect is one of the most demanding he has ever worked with and his company has met the prospect’s every demand. What our eager sales person doesn’t realize is that his trusty laptop, the one he carries with him every where, the one that has been to the beach, and the one that has survived being dropped at the local Starbucks is about to cost him this deal. Management and Patch Management Why should any sales director be concerned with the company’s current information technology infrastructure? The sales director is responsible for a team of people who, in general, are not going to be cutting edge IT specialists. They need access to prospect information, the ability to write a letter or an e-mail, and internet access for research, but not necessarily much more. The sales director is responsible for insuring that the team is successful, efficient, motivated, and passionate about their position. Would a sales director be interested in reading a complete guide to patch management? Highly unlikely, but why does management need to be cognizant of patch management best practices? Why? A sales person visits a prospect. The sales person walks in with a portable computer with an innovative presentation that the marketing department just prepared. He sets his computer on the table and connects with the prospect’s IT person. He turns the laptop on and plugs it into a projector. During boot up, the IT person at the prospect’s site notices that the machine boots to a Microsoft Windows 98 screen. What is the first impression? This prospective vendor and his company do not know how to protect themselves from vulnerabilities and risk, “that the company is using old software,” “that the company is using an unsupported operating system!” The IT person can only feel relief that the sales person has not asked to use the internal internet connection. “We could have been infected in less time than he plans to be presenting (at best 20 minutes).” Now consider the last presentation that you attended. Did you notice the equipment used? Have you personally questioned the technical competence of presenters? Did you know that even in security-focused presentations the audience could be shocked when the presenter’s laptop has the icon in the toolbar indicating that a security patch’s status as downloaded but not installed? These two situations indicate that the decision of each person’s company “not to keep current” just cost the sales person a sale and reduced the reputation of the presenter and the company he represents. What is worse is that they do not even know what hit them! Before the presentation even started, the prospect’s IT person is projecting a negative attitude and the technical people in the audience are prejudiced even before the sales person opens his mouth, and the nontechnical attendees are picking up bad vibes. I can sympathize, as I have been in this position. Buyers judge salespeople on their looks, their knowledge, their articulation, and the tools that they carry. Keeping a sales representative’s technology current is a huge concern for the sales director. Our sample scenario involves not only a lost sale, but also seriously compromised sales staff motivation. They feel abandoned when they factors outside their control nail them. Getting a meeting can be an extremely difficult and can occasionally take years. It is critical to create the best first impressions in first meetings, as most sales managers understand. Keeping a company’s machines current and secure sends a message to the buying audience that your company cares about security! Compare it to the

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

69

“infection” impression you might get if someone wiped his nose before offering to shake your hand! Now apply our sample insecure scenario to a larger sales event such as a trade show. Most trade show exhibits rely on shipped or rented computer technology. This equipment runs slide shows, highlights product benefits, demonstrates product features, and collects prospect and client information. When did you last patch your company’s trade show equipment? Consider that larger companies often dedicate some equipment for use at shows only. Among some of the larger vendors, you would assume that the trade show equipment was well “oiled and greased” before shipping. However, a malicious worm called Slammer hit during the American Institutes of Certified Public Accounts’ 2004 annual technology conference. The vendors all had invested thousands of dollars in booth space for the audience of almost one thousand technically inclined CPAs. The show included an audience of firms working with almost every business in the United States and was an attractive venue for reaching businesses. For a vendor a trade show also includes additional costs including equipment rental, the care and purchase of the booth itself and staff expenses; and yet over half of all the vendors in attendance had no way to conduct business when Slammer hit. This included some of the largest computer vendors in the world and some of the more technically advanced vendors in the country. Vendor representatives were depending on attendees for help and vendor booths were “dark.” The impression was significant! Management was oblivious to patch management best practices. Some are still recovering. We can agree, then, that patch management best practices affect image, marketing, and sales, but they also affect other departments, as well. An Officer’s perspective We can view corporate officers’ responsibilities from several angles. There are the big picture considerations and decisions that require a forward thinking individual who constantly looks toward what ultimately should happen. Then there are getting the tasks done to reach smaller and then progressively larger goals. We refer to the above as the doers and the thinkers. The thinkers can artfully decide where a company needs to be in a month, a year, or in five years to grow and prosper. The doers have a way of using the available resources in a company, including the specialized skills of team members, to achieve the smaller goals and to move toward bigger picture items. Patch management or the lack of patch management procedures affect both of these types of leaders and their daily environments. The CEO, President, and Founder are generally big picture thinkers when it comes to a business. The CEO must constantly be aware of the environment in which a company does business. He is the captain of the ship and needs to know about the possible iceberg formation hundreds of miles away, of parallel business traffic that might interfere with the company’s plans, and of environmental variables. When it comes to security patch management, the CEO needs to know of the level of risk associated within any given period. He also must be the compliance manager! Vulnerabilities and risks, created by the very nature of technology, affect big picture decisions for moving forward and must be on the CEO’s agenda. If we consider that the global environment in which we all do business is subject to government regulations, a war, or other factors that can increase the risk, then the CEO must be extremely sensitive to any “exposed” sides of a company. Additionally today, the CEO must also be concerned with his own personal risk due to noncompliance. Given that the technology infrastructure of a company is in constant change, the exposure to risk is a changing variable that needs constant attention. To the CEO, patch management is about best practices and compliance; it is about reducing risk and vulnerabilities; and it is about protecting the corporation, the corporate officers, and the team members. Consider that Gartner predicts, “By 2005, 60 percent of security breach incident costs incurred by Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

70

businesses will be financially or politically motivated.” increases corporate risk.

88

Weak or ineffective patch management

The Chief Information Officer or Chief Technology Officer focuses on the use of technology within a company. This position emerged in the United States in the 1980s as a business-focused extension of the position of Director of R&D. Large research-oriented companies like General Electric, AT&T, and ALCOA created this position to increase the profits derived from research projects in their laboratories. During the dot-com and computer boom of the 1990s, many companies used the CTO title for their senior technical person. The MIS and IT community often use the title CTO as either synonymous with Chief Information Officer, or as a subordinate to the CIO who is more versed in the technical intricacies of the systems being deployed. There is no uniform application of the title and this causes confusion when people across domains discuss the role of this person. The roles of the CTO vary among companies and within industries, but 89 usually relate to technology. Immediately we realize that creating and rolling out a patch management best practice environment is closest in responsibility to the role of the CTO, but do patch management best practices help the actual IT departments? Does a patch management best practice make a CTO’s job easier on a day-to-day basis? Information Technology departments are traditionally environments of crisis management. Depending on the day, a set of workstations could be down, a database could be corrupt, or a network could be having trouble that requires the attention of the IT department staff. The Director of IT must also deal with limited resources to manage these crises given the erratic nature of the demand for IT professionals within a business environment. When a new application rolls out to a department, the demand for support significantly increases. On the other hand, if the application and infrastructure environment has not changed over the course of a few months the demand for IT support depends on outside variables that affect the internal infrastructure. In no other department can patch management best practices or a proven patch management process have a bigger impact then to reduce the demand on the IT staff. Small changes to applications and operating systems create efficient environments for problem resolution. If the corporate policy is to wait to install updates and patches, than the changes become bigger and bigger depending on how long the wait extends. Not only do the changes become bigger by accumulation, but the vulnerabilities increase as more and more programs that exploit vulnerabilities appear. We must also consider that the longer a company waits to role out a patch, the further away the vendor who created that patch gets from its development. This creates a situation where it is harder for the original vendor to make additional changes based on the idiosyncrasies of any one customer. Specific industries also must contend with special issues. In the medical community, for instance, the CTO balances the needs of patch management as recommended by the security provisions of HIPAA rules with the restrictions that medical equipment vendors impose to ensure that their equipment has a controlled change process in place. We will focus more on industry specific niche issues in further chapters. This brings us to the Chief Financial Officer (CFO), the accounting manager, or controller. “A financial manager, through his understanding of the company's financial health, the current 90 market, and the goals of the company, helps set direction and guides decision making.” The financial manager must understand more than numbers. He must be able to interpret data, be a researcher, and comprehend what needs to occur within his sphere of influence to help a business prosper. This includes balancing business needs/decisions/goals with investor’s needs/decisions/goals. The financial outlay and the investment that is required for rolling out any patch management best practice solution usually becomes the first consideration for the CFO; however, CFO’s are also very concerned with risk.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

71

How do you measure the impact of patching versus not patching against the investment needed in both time and dollars to roll out a proper system? There are certainly the tangible and intangible variables and we will consider these as we look at ROI, but remember that downtime costs money, sometimes so much soft money that measurement becomes extremely difficult! Downtime can also create either more of/or less of an impact depending on the nature of the business and the industry niche. Technology is also its own industry niche and technology variables affect patch management best. The CFO who compiles and creates the ROI around patch management must be more than a CFO. He must be a CFO who understands technology and the balance of risk within the technological world. We then have the Board of Directors, a group of executives often not located with or employed by the corporation. Most managers however do report to a board, but when it comes to patch management best practices a board of directors has a very different focus. They are independent of the firm but how they manage themselves, and how patch management best practices impact the board can cost a firm significant funds and loss of resources. Consider the cost of the board of directors to a corporation. Senior executives are expensive! Now consider what they are offering. Board members offer insight, experience, connections, and opportunity. They offer oversight and management support. The board has experience with the industry as a whole, with acquisitions and mergers and with the potential opportunities to a firm. With a board ranging from six to nine members, this bandwidth should be making powerful decisions and often they do—but—and the big “but” comes in the flavor of access to information and managing that access. How can the board make decisions without the appropriate information or without accurate access to current data? Printed reports a few months out of date are not easily digested, nor very applicable in an extremely fast changing environment. One of the epidemics in the corporate culture is the lack of available access to or appropriate control of digital information within a corporation. Board members are often corporate members of other companies, their internal privileges need monitoring, and their access is sensitive to changes. Yet they are often forgotten when changes need to be made. We find poorly rolled out patch management best practices can easily compromise the sensitive relationships that exist between a board and the management team, and repairing these relationships is not an easy task. If the BOD has no access to appropriate and current information then the company is not getting full benefit of their expertise. "There is a delicate balance between limiting insider access to information and crippling the ability to create revenue," said Richard Hunter, vice president for Gartner. "Generally, this conflict between security and commerce is resolved in favor of creating 91 revenue and therefore facilitating insider crime." Gartner advises that among other actions, businesses must create and enforce legal agreements defining legitimate use of proprietary 92 intellectual property by trading partners and employees. These legal agreements can also fall under the security arena best practice umbrella. Now consider, does the CTO always have deep insight into the existence of these legal agreements? When designing patch management best practices carefully consider every department within a company and every person who needs access to all data. Plan how to keep data available to those who need access and how to comply with existing legal agreements—not only how but which legal agreements apply to best practice policies. Control board member access by network and files rights management techniques such as limiting data and file access to read only or storing corporate data that is unrelated to the duties of the board in folders that are inaccessible to the BOD. To use the power and connections of a board effectively, a company must train and work with these senior level executives to ensure they can use the power of data mining and business intelligent tools.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

72

We turn our attention now to sales, marketing, and customer service. What about the every day tasks that employees and staff must accomplish to meet their job requirements? As more and more companies depend on technology to meet the needs of prospects and customers, problems can affect every department’s productivity. Marketing, consciously or unconsciously, integrates with almost every process within a corporation. Most corporations have a marketing department focused on the task of marketing externally by polishing a company’s image by preparing financial statements, producing and promoting products, or delivering compelling customer service. Marketing is part of every step of the business process. Technology is close to same role. When we call a client, we look up a telephone number (using technology). When we bill for a product or service we use accounting information systems (technology). When we market or advertise a product, we design mailings, make calls, or put advertisement in the news (using technology). We talked about the impact of poor patching on the sales process and certainly, our example plays to the perception for the new prospect. The firms impacted by the SQL Slammer and Blaster viruses realized quickly that not only did they have a “technology” issue, but a “marketing” issue. The fact that their firms’ names appeared on the nightly news as targets of these events caused issues that their marketing or public relations departments had to deal with. What other issues can an unpatched or even up to date systems cause? If a machine within the marketing department is not current, updated, or patched, it can become an issue with the outside vendors who work with that department. For instance, a publisher might not accept an electronic file needed for advertisement, because it is not in a compliant or specific format. This issue limits the marketing department’s efforts to create brochures; company t-shirts, mass mailings, or other products in compliance with outside requirements. In smaller firms, having the appropriate version of the software to produce the correct files can be a daunting task, but it is a rare magazine vendor who will accept a digital advertisement that is not compliant with the latest digital format. Having technicians to keep a firm current and up to standard increases the internal efficiency of file sharing as well as the external needs of file sharing with other vendors. When a company’s tools are current, that company is in a better position to compete. Security issues within marketing create layers of policy that affect standard practices such as emailing files. In late 2004, a security issue with image files caused some firms to begin to analyze whether or not to block such attachments. Industry wide best practices to block image file attachments slammed the marketing department’s use of e-mail to share files with publishers. A Patch Management Best practice includes knowing what types of files are in transport at any given time within a firm or department. Using policy to single out what is permissible within the marketing department increases the department’s ability to work while allowing IT to focus on the most relevant threat vectors. Blocking all attachments cripples the marketing department and crisis management would kick in to resolve the issue. Keeping systems on standards platforms and consistent with Patch Management Best Practices assists in ensuring that a business stays agile, out of crisis management and proactive in it’s ability to ensure interoperability with other business partners. Customer service is an ideal place to gather customer information and many firms fail to take advantage of this opportunity. Without patched, functioning, and current CRM software, data is continually lost, even though data accumulated equals powerful information for a firm. A dysfunctional CRM system is worse than having no CRM system because when customer service representatives run into a problem with the system they are usually on the phone with a client or prospect. Nine times out of ten the client or prospect has the impression that the corporation cannot keep its computer systems running. Talk about direct ding!

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

73

Lastly, we must consider that at the heart of every business is the supply chain and cash flow! The supply chain (inventory control, manufacturing, purchasing, etc.) is truly the blood that keeps a business alive. Consider that any company that manufactures a product relies on machinery to produce that product and, more and more, machinery relies on software to function. We must be keenly aware of what sort of problem this type of software dependency would cause. Machinery is becoming comparable to a living, breathing, and changing creature that needs careful management and maintenance, and the biggest management component of machinery software is patch management. Much of our world relies on software and this phenomenon is only going to grow. Consider the number of electronic devices used in any given environment; you have to accept that without the proper functioning of these critical devices we would be in serious trouble. “The epidemic of Windows-based worms and viruses in the past year has put hospital IT administrators on a state of high alert to protect patient-care systems that have become reliant on Microsoft operating 93 systems.” It is not just the Microsoft operating system at risk. More and more sophisticated hackers are wasting time creating havoc. The end of 2004 saw a barrage of Oracle database vulnerabilities released to the web right before the Christmas holidays. The Oracle Database infrastructure is not dependent on the Microsoft operating system. It runs on a variety of platforms with a variety of operating systems. For all firms using Oracle databases and software, crisis patching requirements increased at a time when business was busier than usual. How can businesses deal with the risks generated by external security threats and documented by researchers who discover and document these threats? How can the best practices help us reduce risks? Understand your unique environment, build strong relationships with vendors, and implement business-to-business (B2B) best practices that include high quality, partner-level relationships. Such relationships provide you with options, agility, and customized resolutions. This book focuses on many different elements of patch management best practices, the most fundamental of which is building and maintaining relationships at levels that create good flow of information, partner-to-partner win/win resolutions, and communication. To understand fully each manager’s risk assessment role, review the following guidelines provided by the National Institutes of Standards and Technology. Senior management, under the standard of due care and ultimate responsibility for mission accomplishment, must provide the necessary resources to develop the capabilities needed to accomplish the mission. They must also assess and incorporate results of the risk assessment activity into the decision making process. An effective risk management program that assesses and mitigates IT-related mission risks requires the support and involvement of senior 94 management. Senior managers have a level of power that can cut through layers of people and “peon filtering.” Senior managers need to build bridges between the appropriate people at vendors the firm depends on, allowing staff to connect with the appropriate people at other corporations. Chief Information Officer (CIO) is responsible for an enterprise’s IT planning, budgeting, and performance including its information security components. Decisions made in these areas 95 should evolve from an effective risk management program. The CIO attends trade shows and industry events, and mixes and mingles with other CIOs in the specific niche. The CIO also communicates information regarding threats to a given industry. System and information owners are responsible for ensuring that proper controls are in place to address integrity, confidentiality, and availability of the IT systems and data they own. Typically, the system and information owners are responsible for changes to their IT systems. Thus, they usually have to approve and sign off on changes to their IT systems (e.g., system Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

74

enhancement, major changes to the software and hardware). The system and information owners must therefore understand their role in the risk management process and fully support this 96 process.” System and Information owners are classic roles for participating in forums known to discuss the current risks. These forums include patch mananagement.org as well as industry specific niches focused on security risk analysis. Relationships with technical experts in the security space increase the resources a system and information owner brings to a corporation. Business and functional managers responsible for business operations and IT procurement processes must take active roles in the risk management process. These managers have the authority and responsibility for making trade-off decisions essential to mission success. Their involvement in the risk management process enables the achievement of proper security for the IT systems, which, if managed properly, will provide mission effectiveness with a minimal 97 expenditure of resources. The business and functional managers bring an outside perspective to the role of creating patch management best practices. Using their relationships, they can communicate options and case studies about how others within the corporate world are reducing risks, giving enough variable and perspective to pick the most efficient path. Information system security officers (ISSOs) and computer security officers are responsible for their organizations’ security programs, including risk management. Therefore, they play leading roles in introducing appropriate, structured methodologies to help identify, evaluate, and minimize risks to the IT systems that support their organizations’ missions. ISSOs also act as major consultants in support of senior management to ensure that this activity takes place on an 98 ongoing basis. Given that change happens so fast in the world of technology and even faster in the world of security, the best technologists and have depth of contacts unmatched by their peers. IT security practitioners, that is, network, system, application, and database administrators; computer specialists; security analysts; security consultants are responsible for implementing security requirements in their IT systems. As changes occur in the existing IT system environment (e.g., expansion in network connectivity, changes to the existing infrastructure and organizational policies, introduction of new technologies), the IT security practitioners must support or use the risk management process to identify and assess new potential risks and 99 implement new security controls as needed to safeguard their IT systems. Security awareness trainers (security/subject matter professionals) school an organization’s personnel in the use of the IT systems. Use of the IT systems and data according to an organization’s policies, guidelines, and rules of behavior is critical to mitigating risk and protecting the organization’s IT resources. To minimize risk to the IT systems, it is essential that system and application users have access to security awareness training. Therefore, the IT security trainers or security/subject matter professionals must understand the risk management process so that they can develop appropriate training materials and incorporate risk assessment into training 100 programs to educate the end users. Industry Requirements/Standards Healthcare Industry—in no other niche environment can this be better illustrated then in the world of medicine. A healthcare facility provides a heart cathode system that includes a couple of personal health Information (PHI) servers attached to the facility’s network. In security scans of the servers, the head of the IT team has noted that the manufacturer is not maintaining operating system patches and will not approve running antivirus software on these devices. The local IT team has been diligently working with the manufacturer to have the holes patched. It is not that easy, as the manufacturing vendor has made no progress. Meanwhile, the epidemic of Windowsbased worms and viruses in the past year has put hospital IT administrators on a state of high alert to protect patient-care systems.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

75

The hospital IT team has been diligent and has installed a monitoring software package on the servers to allow remote access to the systems. This software identifies a rogue dial-up modem and a few other violations of the normal system setups; however there are additional problems as the systems came with a virus scanner that the vendor updates and the vendor is updating this virus software only once a month. This standard for virus detection is outside the local IT department’s best practice policies. With the local team, the best practice standard is to install the virus definitions as they became available from the anti-virus vendor. It gets even more complicated as the manufacturing vendor has asked the local hospital IT team to remove the monitoring software and the anti-virus scanning software, stating that they cannot 101 support the system running software from other vendors, as doing so skirts FDA approval. Often, networks allowed to grow organically are not properly isolated and, as such, lie outside proper protection. All businesses need protection, but rolling out a patch does carry risks because rolling out a new patch is introducing new code and managing these risks is the goal. The manufacturing vendor is ensuring their due diligence about testing and certifying patches and the world at large needs to be diligent about expecting and requiring vendors to raise the security bar by approving patches in a reasonable period. Some vendors within the medical equipment niche have pointed to United States Federal Drug Administration rules mandating “pre-market submission tests” as the reason for their delay in approving patches. While this might make sense, it is not good enough, particularly given the online arguments against such statements. “The FDA says fingers shouldn't be pointed in its direction,” argues one poster. “The medical device manufacturers say they need FDA approval, but they don't," says another. "If this is a 102 security or virus-related patch, it doesn't require the pre-market submission,” notes a third. “Today, the delivery of healthcare increasingly relies on medical information systems (MedIS). Such systems rely on modern information technology to digitally collect, process, distribute, display, and store patient data. MedIS, like other IT-systems, are vulnerable to malware attacks. MedIS owners and operators have a special responsibility to shield their systems from malicious 103 attacks,” and patch management is a critical component of this responsibility. If we look at another example we can consider the process of medical images.” Medical scanners hit by viruses and worms can be put out of action for weeks while manufacturers decide how the problem can be fixed and whether patches to prevent future problems can be applied,” an 104 investigation by E-Health Insider has found. “The fact that many medical scanners now run on Windows operating systems connected to hospital computer networks to enable images to be stored and shared compounds the problem. This means that, unless patches are promptly applied, a scanner running Windows and connected to such a network is susceptible to viruses and worms designed to attack Microsoft software.” This system could then be down for weeks. Despite such potential devastation, “not all manufacturers have a clear policy about how and when new security patches should be applied to their medical imaging devices, and in some cases will not allow hospitals to apply security patches themselves.” The dilemma in healthcare becomes the quagmire of who is responsible for what compliance. The scenario involves “healthcare IT professionals [who want] to patch networked medical devices and run anti-virus software against them, but say their hands are tied by medical device makers and the FDA,” and the “medical device makers [who are] balancing the need to keep their products from getting infected by viruses and worms with making sure the devices meet FDA requirements” and the “FDA [which] requires medical device makers to adhere to strict 105 manufacturing processes, but wants IT pros and device makers to work out their differences.” The telecommunication industry differs from the health care industry in that it tends to deploy patches when they become available into a heavily tested inactive environment. There is no known standard, but generally rollout occurs on an as needed and when available basis. Vendors handle patches as part of the maintenance contract. Because systems carry live voice traffic, the Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

76

telecoms have to schedule the updates for non-peak times, which are generally around 2am. They bring up the inactive side of the equipment, run it through its tests, and then switch over to the active side when they are ready. Once network admins deploy a patch to a live network they test on the live network making test calls of all sorts inbound, outbound, tandem (in to out), announcement, call failure, and more. Even with these industry standards, there are historical instances where software upgrades did not go as smoothly as they should have. For instance, in 1998 AT&T experienced “a unique sequence of software flaws that caused an 106 outage” “AT&T said the problem began when a computer command to upgrade software code in one of the network switch's circuit cards created a faulty communications path that generated a large volume of administrative messages to other network switches. As a result, the other switches quickly became overloaded and stopped routing data from customer applications for periods ranging from 6 to 26 hours before the network was fully restored” The resulting impact caused disruption for AT&T’s networks and their frame relay customers resulting in a frame relay network that was out for 26 hours—yet it could have been much longer. “Praising Cisco for its cooperation in identifying the root cause of the problem and designing a fix, AT&T expressed “confidence in Cisco and its products.” AT&T will share its analysis of the network outage with the FCC, the industry-wide Network Reliability Council, and other network providers. “By sharing this information and best practices,” and AT&T spokesperson said, “we will help customers avoid similar network outages no matter which carrier provides their frame-relay service." A 2002 Agilent white paper documented the scope of the event. Disruption to customers lasted up to 26 hours and included the failure of mission-critical bank transactions. The fact that Service Level Agreements (SLAs) had promised 99.99% availability and four-hour recovery times (meant that) approximately 6600 customers were not charged, resulting in millions of dollars in lost 107 revenue alone.” Each equipment vendor to telecommunication providers has a different procedure for applying patches, and there may not be a documented procedure within a particular company. Deploying patched is just part of their maintenance procedures when they are working on the equipment. On the other hand, technicians generally deploy updates throughout the entire organization because vulnerability may affect call processing, and if there is an issue, everyone is aware of it, including customers. The vendors do as well, as the CLEC pulls down and applies the updates from the Website. It just depends on the type of equipment. The equipment may be the actual voice switching equipment, fiber optics transmission, digital loop carrier equipment, channel banks, or DSL equipment. Many of the costs to Service Providers caused by outages and downtime are tangible and easy to measure, such as • Direct costs of service restoration and equipment upgrades • Loss of revenue during outages • Penalties associated with customer SLAs However, the real costs include some losses that are harder to quantify but may be far greater. For example, • Lost revenue from dissatisfied customers moving to competitors or taking new business to competitors • The cost of a tarnished image; the lessened ability to credibly market future premium differentiated services Wall Street, Chicago Board of Exchange, and the world of finance is another world in which patch management and risk management best practices are of concern. In the world of fast decisions and even faster trading, Wall Street assesses downtime in the millions of dollars. There

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

77

is no room for vulnerabilities that can slow the trading process, for a slow down can mean millions of dollars in cash flow. The question in the world of finance is not when to update, but to maintain reports proving the system is patched and current. That the patches were tested and that best practices are in place. The financial world delegates risks to trading, while reducing the risk of downtime to the smallest possible variable. "Our organizational culture has been developed around an offensive mind-set, with innovation as a critical underlying theme,’ says Guy Chiarello, CIO of Morgan Stanley's Institutional Securities 108 Group.” In the financial industry, firms use their technology expertise to competitive advantage and as distinguisher factor. “At Morgan Stanley, one aspect of technology innovation is providing a better understanding of risks. [we have} a position, called IT risk manager that pulls together all the issues that underlie risk for a technology organization. This includes document and instant messaging retention, data security, market and credit risk, operating risk, and compliance with 109 Sarbanes Oxley and Basel II. "Whether it is data security, security around client technology offerings, or overall operating risk, risk management—and all it encompasses—now requires 110 greater focus," says Chiarello. “ There are some high profile examples of where patching and poor patch management process caused trouble. Consider, for instance, PayPal’s October 2004 incident involving a poorly tested update. “PayPal started experiencing problems with its service following a monthly software update. Officials at the company, owned by eBay Inc., blamed a software glitch for intermittent 111 outages that cut across account access as well as payment, shipping and other services. Not only did PayPal customers suffer from lack of access and a screeching halt to their business, but also PayPal had to recover from the public relations hit. Remember we mentioned the marketing aspect, and the impact on customer service – you can bet that the PayPal customer service team was working triple time this weekend. We also talked about sales, how would the above affect your sales team? Do you think “pending sales” that were about to close on Monday would close? We can see the impact of one incident is no by reviewing what came out in the news for the next few weeks about the problems with PayPal and the management of updates. What are the ties between patch management and disaster recovery? Disaster recovery is the umbrella under which patch management resides, but there is more. Mishandled patches can also cause disasters that a corporation may be ill prepared to handle efficiently. A CTO at a rather large enterprise recently revealed that backing up an Oracle Financial system and database prior to applying a patch took 10 hours. Applying and testing the patch for proper installation took another 5 hours and, if a problem developed, the restore took another 12 hours. Now consider that we just discussed at least 27 hours of potentially billable time that needs accounting for by the people who work with the Oracle Financial system. The financial concerns are huge! The time concerns are huge and the need for storing patches for recovery is critical. The software vendor perspective views our global environment and is aware of the options and risks associated with this environment. Patch management ties into security, disaster recovery, and IT best practices, but it also ties into a deeper understanding of how software vendors create software, how they create and release updates, and how to work most efficiently within a B2B model. The corporate relationship between software vendor, reseller, and business consumer is critical to businesses that depend on technology. We all know that software has bugs and will continue to have bugs. There is no known quality control process to eliminate all vulnerabilities in software because users and other applications significantly affect any specific software application. If we can’t eliminate all the vulnerabilities within the software and the people who use the software, then what better relationship to nurture than that with the team who knows the software code better than anyone else in the world! In response, the vendor who produces and releases software is generally the number one firm with Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

78

which to nurture a relationship. For the bigger software vendors we must develop bridges through the reseller or user groups to insure so that we understand vulnerabilities and stay ahead of the patch management process. Cyber crime persists. In the U.S., during the five years from 1999 to 2003, attacks on computer servers increased by over 530% to 137,000 incidents. This is partly attributable to vulnerabilities in software code, which have grown from 500 in 1995 to over 9000 in 2002. Developing countries are also susceptible, even after relying on leapfrog technology. Brazil has seen hacker attacks increase by at least 100% yearly since 2002.These growing numbers bear particularly important on the financial sector. The International Data Corporation reported that more than 57% of all hack attacks last year targeted the financial sector. The FBI has corroborated this statistic. Equally troubling, FINCEN’s Suspicious Activity Reports for Computer Intrusions have shot up more than 500% over the past year. With the growing amount of financial data stored and transmitted online, the ease of computer intrusions adds to the severity of traditional crimes such as identity theft. To put this in perspective for the digital age, over $222 billion in losses accrued 112 to the global economy because of identity theft. We can all agree that cyber crime is not going away and as more and more countries and people use computer systems, the law of numbers comes into play. We can assume that vulnerabilities will increase, that software will continue to change and that remain patched is one of many of the defenses we need to use to reduce risk. Best practices continue to spread across the growing global economy. Government regulations continue to assure that the mistakes made by a few do not compromise those within the greater expanse of the world. These mistakes can include pure malicious intent such as that which occurred at many of the large corporations famous in the news and no longer in existence today, to deficiencies in services and products from pure incompetence. Best practice methodologies cross numerous industries, though some are niche specific. They include methodologies such as Six Sigma and compliances such as ISO900x and government regulations that need to cross all industries like the Sarbanes Oxley Act of 2002. With software controlling much of the world we live in, the best practices around software maintenance mix with the methodologies in place to insure high efficiency, high profit, and low risk. Consider how Sarbanes Oxley affects a patch management best practice. Sarbanes Oxley is a complex document, but we must consider it when thinking about the network and computer systems that handle any financial data. Detailed auditing and compliance, computer glitches, hackers, and poor security are no longer acceptable errors in the business world. Accountability and compliance rule the day. If we consider the following four interpreted sections of SOX, we can quickly see that the control of internal systems is very much the responsibility of the management team for each corporation. Yes, management can delegate responsibilities, but they cannot forget them. The AICPA Business & Industry Team assembled an ad-hoc task force of members to address issues relating to management's responsibility with respect to Section 404 of the Sarbanes-Oxley Act of 2002 titled "Management Assessment of Internal Controls.” “The ad-hoc task force assembled a list of key issues to communicate to their peers in other companies that are similarly 113 impacted. The following is a list of four of those key issues and examples with regards to the risks associated with network systems and patch management best practices “Management is required to document the system of internal control over financial reporting. As required by the Sarbanes-Oxley Act of 2002 (SOX), section 404 (Management Assessment of Internal Controls), management will be required to assess the effectiveness of these controls. The ASB believes that the evidence management uses to support its assertion about the effectiveness of its internal control also should be documented. The ASB believes that a failure to document the system of controls or the evidence used in making the assessment should be considered a weakness in internal control.”

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

79

Documentation of financial data is more than a report, more than extracted data and/or compiled data and certainly contains numerous levels of detail. Most of the time base data comes from repositories on computer systems. If management is required to document the system for internal control then they must consider and then incorporate how to validate and protect the financial data on the network. How should managers control this data? How, for instance, can an admin confirm that a physical error on a hard drive has not caused a problem with the data residing in that location? How can he ensure that redundant or malicious data has not been inserted? Certainly reconcilable reports are one answer, but not all companies take this simple step. As we move away from paper, we must continue to consider internal controls and the teams responsible for these internal controls. “Positive testing of internal control must be performed to make the assessment under SOX section 404; inquiry alone is not adequate testing. Negative evidence, such as proper financial reporting, is not evidence of good internal control. On the other hand, a material misstatement detected by the auditor’s procedures that was not identified by management ordinarily is indicative of the existence of a material weakness in internal control.” Several consulting engagements have discovered data being reconciled actually crossreferenced itself. How can this be? What does this mean in real terms? Let us say you are using accounting system XYZ Software. It handles all of the financial data and accounting for your company. If you decide to run your controls using a spreadsheet software package then there is no conflict as long as the source data you are retrieving is independent of the XYZ Software product. If you are manually entering the amounts into the spreadsheet package from a report run out of the accounting system then the two are simply duplicates and reconciliation would be ridiculous. How can this happen? Quite easily, when you consider the superficial knowledge that most users have when it comes to computer systems. Controls remain valid if practitioners retrieve the data compared independently and enter it using two different systems. How does patch management come into play when talking about accounting software? The accounting software itself needs regular patching. “The documentation should encompass all significant account balances and disclosures. The company’s auditors may be utilized to help in the documentation of controls; however, management cannot abdicate its responsibility for the documentation.” How many people do you know understand the true nature of writing custom reports and are proficient in using data mining tools? What would happen if a malefactor compromised a system prior to extracting data? Do you think that the data retrieved would be accurate? If a user chose to extract only the summarized A/R data from one field within an accounting system (the summarized A/R for the month) and a transaction improperly updates the A/R transaction data but not the summary within the database will the totals for the extracted A/R data be correct? How would the user know? What if the detailed A/R was purged? Many of the controlling techniques are standards that go back to the days before computer systems, yet proper management of data has a significant impact on the ability to reconcile. The handling of data when posting, updating, and/or simply using a computer system is at the mercy of the vulnerabilities of the software itself. Since an unpatched system is at higher risk, data posted to multiple locations can be inaccurate. Consider the core operating system software or the development software used to write applications. The OS and development tools are also software packages that can have vulnerabilities. In fact, when programming during the Y2K crisis, we discovered that the core numeric fields within the software language we were using was truncating numbers to two decimal places for display purposes. This truncation was not rounding; it was simply dropping the last numbers. The displays would print and, as such, multiple displays when compared to reports were not reconciling. If as little as a penny of discrepancy can mean millions of dollars worth of differences (which is often the case in accounting) how do you think this error within the basic tool Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

80

set affected the job of the accounting manager? Numerous technical support calls and hours of research were required to report the bug, retrieve the patch, and update systems. What would happen or what does happen to the firms who are not reconciling as closely when errors appear months later? Who is impacted? The accountability is with the management team. The management team must be diligent about best practices! Policies should specify addressing vulnerabilities as they appear. When using outside service organizations (e.g., data processing, payroll processing, etc), management should consider the activities of the service organization when making an assertion about the effectiveness of the company’s internal control over financial reporting. A service organization typically engages an auditor to examine and issue a report on certain controls at the service organization, and this report has historically been a tool used by the independent auditor in planning the nature, timing, and extent of the auditor’s procedures. If a service auditor’s report on controls placed in operation and tests of operating effectiveness is available, management may consider whether this report provides sufficient evidence to support its assertion. Since the date of the service auditor’s report is not likely to correspond with the entity’s end of fiscal year, management will need to consider what changes, if any occurred at the service organization subsequent to the service auditor’s report and the effect of the time lag on their assessment. When significant changes occur, or a significant time lag exists, management may need to apply their own procedures at the service organization with respect to those subsequent changes or significant time lag. As illustrated above a company’s patch management policy can affect every department. If you do not have best practices for patch management in place, you cannot expect your partners to have them. Compliance with ISO900x ISO900x covers a wide spectrum, but we will focus on its focus on quality control and quality assurance. What is amazing is that it is not that different from a big picture view from the various other standards we have been discussing except that it has a long history of tagging a company as a quality player internationally. The ISO certification indicates an intensive process of documenting and commitment to keeping procedural process documented. With the new requirements to meet SOX government regulation, the ISO opportunities increase. While SOX is about compliance, ISO is about international opportunities not otherwise available to corporations when incorporating and DOCUMENTING Patch Management best practices you support the firm in meeting long term ISO 900x compliance goals. The world is becoming more and more global. Reviewing and understanding ISO can give the corporation a HUGE competitive advantage and can open the doors to new opportunity otherwise not possible. Consider adding an added WIN to your patch management best practices project. Look into getting ISO certified, it is not just for manufacturing plants anymore. The specific security guidelines of ISO17799 also provide a base for justification of patch and software management even though the process is not expressly stated in the documents from the ISO organization. The ultimate protection for vulnerability is to remove the vulnerability either in the form of removing the service or, if you are unable to remove the service, ensure that the flawed code is replaced by the patched ISO code. However, one should remember that while we tend to stress “security patches”, patches can also be needed for fixing software bugs. Section 6.3.3 of the ISO 17799 standards require that procedures for reporting software malfunctions be put in place. Compliance with industry association standards and regulations How can you reduce risk and increase compliance? Standards, Standards, Standards! There is some thought that “there has been progress in developing information-security risk metrics over Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

81

the past two decades, but there is still a way to go before standard metrics are established, 114 adopted, and practiced.” The following is a list of a number of institutes who offer Security standards. The International Information Security Foundation (IISF) Generally Accepted Systems Security Principles (GASSP) located at http://web.mit.edu/security/www/gassp1.html. The International Standards Organization (ISO) ISO 17799 located at http://www.iso.org/iso/en/ISOOnline.openerpage. The European Information Security Forum (ISF) Standard of Good Practice located at http://www.isfsecuritystandard.com/index_flash.htm The Institute of Internal Auditors (IIA) Systems Assurance and Control (SAC) located at http://www.theiia.org/esac/index.cfm. The Information Security Audit and Control Association (ISACA) Control Objectives for Information and Related Technology (CobIT) located at http://www.isaca.org/ How can we help the CxO or IT Technical director to summarize these standards and simplify the process? There are a few fairly simply rules of thumb: Document processes so that if the person responsible for a process is no longer available the business will continue. Update these documents for change happens fast in the transitional phase of the global economy and certainly in the world of technology culture growth. Create your standard so that it is a living document and grows with the business. This requires that more than one individual understands and can easily update said documentation. Define efficient processes including the process of pushing out updates (including testing, backing up, and adopting change). As a small business consultant, I recommend reviewing the following sample documentation and guidance: HIPAA: http://cms.hhs.gov/hipaa/hipaa2/readinesschklst.pdf Gramm Leach Bliley: “Security Check: Reducing Risks to your Computer Systems: http://www.ftc.gov/bcp/conline/pubs/buspubs/security.htm Financial Institutions and Customer Data: “Complying with the Safeguards Rule,” http://www.ftc.gov/bcp/conline/pubs/buspubs/safeguards.htm ISO 17799: http://csrc.nist.gov/publications/secpubs/otherpubs/reviso-faq.pdf Governmental engagements: http://csrc.nist.gov/pcig/cig.html SOX 404 (for small companies that are subsidiaries of publicly traded companies) http://www.pcps.org/pdf/article_mike_ramos_01.pdf http://www.isaca.org/Template.cfm?Section=home&Template=/ContentManagement/Con tentDisplay.cfm&ContentID=12406 Cross Border Privacy Issues: http://www.itgi.org/Template_ITGI.cfm?Section=Security,_Control_and_Assurance&CON TENTID=5556&TEMPLATE=/ContentManagement/ContentDisplay.cfm EUROPA Data Protection Guide: http://europa.eu.int/comm/internal_market/privacy/guide_en.htm Recent legislation in California, in the form of a law in effect as of January 1, 2005 called AB1950, requires that firms take “reasonable security measures” to protect the confidential data of California residents that could be labeled as “identity data.” I urge you to work closely with your clients if they are in one of these industries or are having these issues. In addition, obtain the resources you need, or find appropriate security consulting firms to help you stay informed. Return on Investment (ROI) What is the real return on investment to putting procedures in place? Why bother to document “another” procedure and carefully outline processes? The question is easier answered than not, but many CxO’s struggle with communicating the real understanding around the ROI of security. “Unfortunately, relatively little attention has been paid to economics, and to the applied financial practices that grow out of economics, when it comes to information security. In 2003, the annual CSI/FBI Computer Crime and Security Survey reported average losses per respondent of about Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

82

$800,000. In fact, the real headline should be that even those loss totals don't do justice to the 115 magnitude of information security crime and related costs.” "Economics--not technology-determines what security technologies get used," says Bruce Schneier, security expert and author of Beyond Fear: Thinking Sensibly about Security in an Uncertain World. "These days, I 116 feel like I do more economics than computer security." “In the commercial world, the cost of a 117 cyber security breach is measured by both ’tangibles’ and ’intangibles’.” Calculate the tangibles based on estimates of: • Lost business, due to unavailability of the breached information resources • Lost business, that can be traced directly to accounts fleeing to a “safer” environment • Lost productivity of the non-IT staff, who have to work in a degraded mode, or not work at all, while the IT staff tries to contain and repair the breach • Labor and material costs associated with the IT staff’s detection, containment, repair and reconstitution of the breached resources • Labor costs of the IT staff and legal costs associated with the collection of forensic evidence and the prosecution of an attacker • Public relations consulting costs, to prepare statements for the press, and answer customer questions • Increases in insurance premiums • Costs of defending the company in any liability suits resulting from the breached company’s failure to deliver assured information and services Not all of these tangible costs will occur with each breach; some will only occur with major, wellpublicized breaches. The intangibles refer to costs that are difficult to calculate because they are not directly measurable, but are nevertheless very important for business. Many of these intangibles are related to a “loss of competitive advantage” that results from the breach. For example, a breach can affect an organization’s competitive edge through: • Customers’ loss of trust in the organization • Failure to win new accounts due to bad press associated with the breach • Competitor’s access to confidential or proprietary information “In the military, [we must also take into consideration] the tangible costs measured in human lives, replacement costs of equipment, and prolonged military operations. The intangibles would include loss of tactical advantage, loss of international prestige, and impaired negotiating positions.” As Anita D’Amico points out, taking a close look at the break down in costs measured by tangibles and intangibles helps to understand the cost of a missed patch that causes computer down time. Certainly, security is not only about patches, but also by ignoring patch management an avoidable security breach could occur. A survey of 882 respondents determined that the MS Blaster worm cost $475,000 per company (median average - including hard, soft and productivity costs) with larger node-count companies reporting losses up to $4,228,000. MS Blaster is simply one example among many! We must then look at the constraints faced by management in budgeting for security.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

83

From Information Security Magazine July 1999, "Top Obstacle is Budget: What is the SINGLE greatest obstacle to achieving adequate info security at your organization?" Consider these top obstacles to achieving adequate security and notice that right behind budget constraints is “Lack of Senior Management Support.” "People in information security are often technicians--gear heads," the report notes. Few IS managers have come up through the ranks of 118 accounting or financial management, so they don't think in [economic] terms,” the report adds. It is critical that management focus on the economics of security and dare to go one step further and recommend a CFO who is also well versed and experienced in technology and the security niche. It is unfortunate, that more often than not it takes an event to occur before management realizes that installing a patch management tool is of value. What do we really need? According to Kathryn Korostoff of Network World, “Network security is one of the hardest technology categories for which to create an ROI analysis”119 but I doubt that it would be very hard to find a number of other situations where the ROI relies on softer variables and, as such, is as difficult. The goal is to break down the variables into manageable parts and then use the resources of the web, human experience, and knowledge to put parameters around those variables. Remember that sometimes these variables are very specific to the organization. When we understand the business processes down to the tasks preformed by specific departments and people, we can train staff to have an alternative path by default for completing the necessary jobs. If we look at a specific staff position and the task that position must accomplish we can determine the impact of having limited or no access to the internet on the completion of these tasks. A sales person making calls to prospects certainly needs access to customer relationship software, but depending on their style and the software they depend on they might not need access to the Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

84

internet. If we break down a number of sample components, we can make informed opinions and plan responses to security breaches. We might even consider building these job descriptions into the position statement stored with the HR department. When considering the ROI around patch management best practices we really have three projects. The first is that of creating and implementing a patch management best practice policy. This roll out takes time and resources and it is important to include this initial investment in our analysis. The second project is the ongoing maintenance and modification of the patch management best practices procedures, including training new staff, maintaining vendor relationships, and modifying documentation. Lastly, there is the project that defines and weighs options when a breach occurs. This last project is often bundled with the corporation’s disaster recovery planning or business continuity project. Here is an overview of some of the variables in each project. Defining the Patch Management Best Practices—First ROI Variables Project In defining patch management best practices, we need to consider that the end goal is to have a completely documented process that allows for continuous updates. This document should be a living document that, once created, evolves into a Maintenance Project. Intellectual teamwork—a team assembles the best practices. This team meets regularly to discuss the project and to make plans for completion. Who is on the team? What credentials are appropriate? Research by individual team members—having complete data on the business environment, the industry niche, and the tasks completed within the organization assures coverage of all variables. Decision-making takes time and input from the entire team. The decision-making component might also need input from higher levels of management. Documentation—writing, editing, typing, and binding the documentation require resources. Review and acceptance—once the document is completed each department, the management team, and perhaps even the board of directors needs to review and approve the procedures. Software and hardware deployment—the actual analysis of deploying the above documented best practices requires energy. A reality check, is it realistic? Who will analyze the costs of needed components? Industry specific variables can include other vendors who must approve the process such as the corporate lawyer (for industry and contract compliance), or the corporate accountant (for SEC compliance) Actual ROI—the cost and return on investment analysis should be written and then kept with the best practices patch management policy for any new manager or staff member exposed to the original documentation. These variables change with time and part of project two will be looking at the variables used at the outset. Some forward thinking firms have justified patch management practices by identifying the costs saved by not hiring help desk staff, or used the process of asset inventory to point out the unidentified risks and compliances issues. Ongoing maintenance and modification—Second ROI Variables Project Update schedule—when will the PMPB be updated and reviewed—quarterly, monthly, or yearly? Review team—who will do the review? Will there be a team, who will be on the team? Research (continued)—who will do the follow-up research? Where will they start? What new information will they look for? How do they know about this new information? Documentation (updated)—who will do the actual writing and updating of the documentation? How will he or she coordinate with the researcher? Who will manage the process? On going relationship building with key vendor contacts—this takes time and is critical to a best practice in this world of forever changing software and variables. Which people are involved? How do they meet? When do they meet? Which process assures compliance with vendor relationship policies? Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

85

Hardware and software upgrade costs—change daily. What is the policy for upgrading software and hardware? It may be necessary to implement more technology in order to provide more tools for asset inventory and control. Management—needs to set aside time to review the established procedures and policies. Testing—lab equipment, space, licensing Fire drill—does the firm support testing in a simulated situation a potential emergency. When a Breach Occurs – Business Continuity—Third ROI Variables Project Down time--what is the cost for your business to be offline? How long can you be offline? Equipment replacement—if X percentage of equipment was damaged due to a disaster, what would be the cost to replace it? How long would it take to replace? What would be the cost of that time? Backup location—if your location is destroyed, what is the cost, in time and dollars, to setup the business in a temporary location? Alternative power generator—if power is lost and all you need is power does it makes sense to borrow, rent or purchase a generator? Human impact (personnel loss that affects productivity)—even the best of us is not at our best when someone close to us has been hurt. The impact on our best asset, the people within the firm, plays a role in the ROI process. This exercise of creating a list of variables and breaking down the tasks shows that doing an ROI can be both overwhelming and very manageable. Certainly, the lists above do not include all variables you must measure for but the goal was to get you thinking about others. Make it fun, have a staff meeting where your only task is to brainstorm every possible variable appropriate to each of the above three projects. Make a list and expand the above on an ongoing basis. Build the game into the culture of the firm. Use magnetic words on the corporate refrigerator; give monthly prizes to staff who think of something new. Prizes can be something such as a two-hour lunch break, a gift certificate, or the first option on purchasing used equipment. Once you have your list of variables, break each into as many parts as possible, within reason. When you break down the ROI process into bite-size pieces, it is easy to assign values to the different components. Instead of assigning values based on an economic model, you might consider assigning points of importance. You can then assign energy to the items that rate the highest on the points scale. This look at ROI takes you out of the limiting mindset of economics and tangible items. We work in a knowledge worker’s world. Valuing knowledge workers is an art most companies have yet to master! How do I know this? Just look around at how many service companies bill by the hour. Is measuring time truly the right variable for calculating costs? Many would argue a resounding “No,” and the momentum is building. As such, how can we value the loss of time strictly on that hourly model? If you have strong feelings about the billable hour and costing by hour, check out Ron Baker’s books on the subject. He is a passionate speaker and writer and has goals of eliminating the old industrial culture of the billable hour for new models based on the knowledge! The traditional risk analysis calculation also works as a basic budget analysis and ROI calculation: SLE x ARO = ALE where SLE = Single Loss Expectancy ARO = Annualized Rate of Occurrence ALE = Annualized Loss Expectancy The accounting staff, risk management staff, and information technology staff need to cooperate on threat modeling your network. What are the biggest threats to your network? It is only in the process of identifying those risk factors of your firm that you will be able to justify to management that patch deployment is more than software deployment and asset management. As we go Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

86

forward and patching is a key process of fixing and blocking security vulnerabilities, best practices merely becomes part of compliance. Return on investment should reflect this “return” that includes intangibles that cannot be easily quantified. History has shown us repeatedly that we do not have control of our networks and that we cannot depend on our fragile “outside” perimeters anymore. We must begin the process of adding layers and building up or hardening our interior networks. Patching is one of the processes that assist in layering and hardening the defenses we use to reduce risk. Have we convinced you that patch management best practices are important? How can you as a manager work with your IT team to roll out a patch management best practice? Or, if you are the IT Manager how can you work with your management team to get the funding needed to make it all happen? How can you get buy in? When technology runs and flows smoothly and systems click along many benefits develop; however, what about the “human” factor associated with effecting a change of even the smallest magnitude? Nothing motivates a manager to buy into a concept quite like the thought of making more money and reducing personal inefficiencies that compromise available time. The number one consideration is to know your audience! When you work toward getting buy-in, you need to know what motivates your audience. Sometimes this is money considerations, but sometimes it is one of the many other variables. Knowing which variable to emphasize is the trick. For instance, for firms that make money by selling time, saving time means making money and increasing the bottom line. You can also consider the issue of compliance and risk. For firms where management is particularly exposed, such as in publicly traded companies, reporting and the potential for analysis and proof that the corporation is compliant can be the biggest motivating factor. Analysis can also be a factor for helping to direct and focus a firm on its most profitable path to best practices. This reporting component can reassure a CEO that his board of directors is not going to have problems with the projects you are proposing. Direct reports also motivate your audience. Never forget that the decisions in medium to large size firms rarely stop at one person. Speaking of one person, an ROI must reflect individual style with appropriate personal and firm-wide emphasis. ROI presentations vary widely depending on the audience. When you align your ROI with their goals, you find solutions that meet both your needs. Finding time in a crisis management environment Who has time to spend hours and hours preparing ROI presentations for top managers? Doing so takes time, but they can evolve by using bite-size pieces over time. Take the need to list all the different variables affecting an ROI analysis. Split this project into 25 different subsets and then assign each subset to a different person. You might ask the director of HR to make a list of all the things affected by computer problems. You might ask the office administrator how a phone system failure would affect clients. The marketing department is also a huge resource. They understand how crisis and unavailability can affect customer relations. Do not forget about the accounting department either. Many are numerically sensitive and can run some numbers for you. Get people involved. Make your quick list and then delegate. The beauty of the computer system is that it affects all departments and when considering variables for ROI analysis multiple perspectives and diversification can be a huge asset. Purchasing patch management tools Patch management tools play a significant part in a patch management best practices policy and in the efficiency of an IT department. These software products increase consistency and compliance in the management of corporate software and hardware updates. People who work with patch management software classify the patch management tools into two categories; the tools are either “agent” or “agentless” applications. An agentless solution does not install any software on the individual client computers. The agentless software can distribute or push updates to multiple machines without needing to first install or find an “agent” on the receiving machines. Agentless software rolls out in minutes as opposed to days. The benefits of an Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

87

agentless solution are numerous, but there are also disadvantages. An agentless system is more vulnerable. Many times agentless software will not comply with security policies that deal with distribution of encrypted data. The benefit of an agent-based system is that software updates and patches can be uniquely encrypted, delivered, and then unencrypted using the features of the agent software. Unfortunately, agent based solutions also contain the agents, which are software components that may also require updates, patches, and changes. Using an agent-based solution increases the complexity and requirements on a machine-by-machine basis. An extremely important component of patch management solutions is the reporting module. When reviewing a tool, look carefully at the built-in, pre-configured reports. Patch management software reporting can significantly decrease the amount of due diligence that must be done on an ongoing basis. The reporting component of patch management software can also indicate which patches need immediate deployment, which can wait, and which are unnecessary. Sophisticated patch management software packages offer robust reporting modules. Sophistication in patch management software also shows itself in support for multiple platforms because a patch management tool in an enterprise environment often needs to recognize and resolve patching on multiple operating systems and platforms. Even in small environments, there are more and more alternative operating systems and platforms widely available. Consider the mix of Open Source solutions and Microsoft Solutions. Many smaller companies find that having a mixed environment offers more options for less money. Patching the operating system is one consideration, but we also run numerous third party programs and each of these third party programs also has patches. The ability to recognize third party programs and pull patches from a wide variety of web resources is also a sign of a robust patch management solution. Here is a checklist of things to ask questions about when researching patch management software. 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9

Agent vs. agentless Reporting: pre-defined and custom options Rights Management: Can access to the toolset be restricted and to what level? How many clients can the package support? Are there customers using it with this many clients? What are the biggest and smallest installations? Does it depend on a database back-end engine? Which one? What operating systems are supported? What third party applications are supported? Is the list of third party applications updated regularly? When? Does it cost to get the new libraries of supported third party applications? Is the User interface easy to use? Does the user interface require technical training? Does the application require training? If yes, how much? Are there other features not related to patch management built into the core package?

In general, just as with any other due diligence review of a new software package, you want to ask as many questions as possible. A great resource for posting questions is the patchmanagement.org virtual community, but many of the security “centric” virtual communities also offer independent perspectives from people who have experience with many of the patch management tools available today. Vendor Issues Controlling the patch management process requires significant relationships with companies involved in the release of equipment and software. We stressed this earlier. In the current technology environment, a software company generally takes responsibility for their software and releases regular updates based on their methods of operation. For your company to reduce risk Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

88

and exposure it is important to understand the models in which your vendors operate. These models should also be part of the consideration process when choosing to work with a particular software or hardware vendor. Allowing vendors to manage patch management processes may compromise your work to maintain and stay within a high quality patch management best practices system. The key is to understand the vendor’s business model and then design your system around their offerings. If their offerings do not fit a good patch management best practice then consider working with a vendor who is more agile and responsive. In addition to picking vendors based on agility, you also want to create strong bridges of communication between multiple people within the vendor firm and with multiple people within your firm. Vendors can and have released incompatible patches, they can and have struggled with timing and delay issues based on their own testing schedules and their vendor relations and they can also have policies in place that could bind your hand. The software and hardware vendor have the power to threaten or even void service contracts if you touch or update their software or hardware. Conclusion As much as the world of security is becoming more important for everyone to understand, it is still a technical niche with its own language and unique set of acronyms. Extensive resources are available. This text helps focus management and staff on the three critical components of implementing and rolling out good patch management best practices. Each of these three components functions as a unique project. The first is accepting, deciding and actually rolling out a patch management best practice policy; the second is the project of maintaining and updating the patch management best practices policies so that they are viable on a continued basis; and the third and perhaps the most well known is building disaster recovery planning into the corporate culture. Patch Management is about reducing risk, increasing continuity, expanding options and relationships so that when a crisis occurs, everyone not only knows where to turn, but who to turn to for help and resources.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

89

Appendix A– Patch Testing Testing plans 9 Identify the operating system being tested 9 Identify the Service pack level 9 Identify the hotfixes installed on your systems [if in addition to security fixes] 9 Identify critical third party applications 9 Identify third party applications that have had patching historically 9 Identify those files used in patches that may have causes issues in the past. Are the included in this current patch? Assign testing resources appropriately. 9 Determine if the bulletin accurately describes patch deployment. If not, determine if additional resources for testing or imaging need to be in place before approving the patch. 9 Test the installation of the patch both manually and via your automated patch technology. 9 Can you uninstall the patch using add/remove program or your patch tool. 9 Review the processes of your line of business applications. Are they performing as expected? 9 Attempt to replicate a production environment by using imaged data. Testing tools may assist in attempting to replicate your production system but are not a substitute for an image of your system. 9 Stress tools to test a Web server: o http://support.microsoft.com/default.aspx?scid=kb;EN-US;q231282 9 Download details: Windows Application Compatibility Toolkit 3.0: http://www.microsoft.com/downloads/details.aspx?FamilyID=7fc46855-b8a4-46cd-a2363159970fde94&DisplayLang=en 120 9 Set up performance and monitoring tools to review your testing machines such as 121 PerfMon, tools from Sysinternals and review all log files. 9 Confirm the installation of the patches via registry review or other means

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

90

Appendix B—Sample budget request CONCLUSION AND RECOMMENDATION: Recommendation to the board to approve the purchase and installation of software patch management software to remotely manage all of the computing devices in the network as deemed by the IT staff. The estimated cost of this project will be in the range of ___________ to ____________. EXECUTIVE SUMMARY: The computer assets of the firm are an integral part of the revenue generation of the company. At the present time, unnecessary IT staff time is spent in attempting to control, administer and apply software patches to our current network system. We are requesting that the firm implement a more structured approach to software patching and management of the computer systems in the firm. BACKGROUND: Over the years through normal expansion and firm mergers and acquisitions, the network of the company has grown organically thus not providing the proper infrastructure to allow for the IT staff to properly control and manage the computer systems of the company, Currently the lack of software patching tools and lack of remote connectivity and control is contributing to the costs of the helpdesk in the firm. The following techniques will help reduce these costs: Standardization of desktop and server hardware and software--aids in the process of more control of the change management process we recommend that an inventory of all assets be undertaken and documentation of approved applications be prepared and only these allowed applications be installed in the firm. A virtual lab replicating these standardized desktop and server installations—aids in the quicker and faster testing of patches to deploy them quickly to the machines identified as needing these patches. Purchase of patch management software and/or tools to better aid in software patch application— currently the IT staff does not have the necessary tools to quickly and easily provide reports to management to ensure compliance with the firm’s minimum security policies. FISCAL IMPACT: Initial costs of _________ and annual maintenance fees of __________ are expected to be incurred in hard costs and staff costs are expected to range from _________ to ___________ in the testing and implementation of this solution.

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

91

Glossary Application - A program or group of programs designed for end users. Software can be divided into two general classes: systems software and applications software. Systems software consists of low-level programs that interact with the computer at a very basic level. This includes operating systems, compilers, and utilities for managing computer resources. In contrast, applications software (also called end-user programs) includes database programs, word processors, and spreadsheets. Figuratively speaking, applications software sits on top of systems software because it is unable to run without the operating system and system utilities. Critical Update – this is not a security update, but a fix for an issue in broadly applied software. They are publicly available and have an accompanying knowledge base article. Driver Update – The update in a software component that supports and controls hardware Driver updates may come from the software vendor or they may be from the hardware vendor. Feature pack – a software package that is released that includes non-critical additions to the base software program. It is typically distributed between major releases. Firewall - A system designed to prevent unauthorized access to or from a private network. Firewalls can be implemented in both hardware and software, or a combination of both. Firewalls are frequently used to prevent unauthorized Internet users from accessing private networks connected to the Internet, especially intranets. All messages entering or leaving the intranet pass through the firewall, which examines each message and blocks those that do not meet the specified security criteria Hotfixes – hotfixes are patches built to address specific issues. The Microsoft Hotfixes cannot be distributed outside of your organization without written authorization from Microsoft. They are obtained for FREE by calling Microsoft Product Support Services. They are not subjected to the regular testing process. Patch – “A piece of code added to software in order to fix a bug, especially as a temporary correction between two releases.” Patch Management - the act, manner or practice of managing, handling, supervising and controlling the application of code to software in order to fix a bug, especially as a temporary correction between two releases. Security (data) – A way of insuring data on a network is protected from unauthorized use. Service Packs (SP) – are cumulative packages of hotfixes, security updates, critical updates and updates. The SP is tested both internally and externally. Security Update – this is what we traditionally refer to in the existing patch management circles. There is a severity rating included in the updates and they are always accompanied by a Security bulletin discussing the issue and a Knowledge base article that describes the patch in more detail. Update – fixes a non-critical, non-security related issue. Update rollup – cumulative package of hotfixes, security updates, critical updates, and updates. This is tested for easy deployment. Upgrade – software that updates and upgrades a piece of software to a newer version while keeping the settings and data from the prior program

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

92

Notes Chapter 1—What Is Patch Management 1

“Analysis of the Sapphire Worm - A joint effort of CAIDA, ICSI, Silicon Defense, UC Berkeley EECS and UC San Diego CSE,” retrieved August 29, 2004 from http://www.caida.org/analysis/security/sapphire/ 2 “Threats and Countermeasures: Security Settings in Windows Server 2003 and Windows XP,” (Redmond, WA: Microsoft, Inc., 2004) from http://www.microsoft.com/downloads/details.aspx?FamilyId=1B6ACF93147A-4481-9346-F93A4081EEA8&displaylang=en 3 Allen, Julia H. The CERT Guide to System and Network Security Practices, New York: Pearson Education, 2001. 4 “Description of the standard terminology that is used to describe Microsoft software updates,” (Redmond, WA: Microsoft, Inc., 2004) from http://support.microsoft.com/default.aspx?kbid=824684 5 Ferguson, Niels and Bruce Schneier Practical Cryptography New York: John Wiley & Sons, Inc., 2003. 6 “10-Step Emergency Response Plan for Security Attacks” Redmond, WA: Microsoft Corporation, 2004 from http://members.microsoft.com/Partner/Campaign/SecurityOutreach/sell/10StepEmergencyResponsePlanforSecurityAttacks.pdf 7 “Upcoming Advisories” Aliso Viejo, CA: eEye Digital Security, 2004 from http://www.eeye.com/html/research/upcoming/index.html 8 http://lists.netsys.com/mailman/listinfo/full-disclosure 9 Swiderski, Frank and Window Snyder Threat Modeling Redmond, WA: Microsoft Press 2004. 10 While we will discuss alternative OS, due to market share and desktop impact, the bulk of the discussion will surround the Microsoft operating systems. 11 “Description of the Contents of a Windows Server 2003 Product Update Package” Redmond, WA: Microsoft Corporation, 2004 http://support.microsoft.com/default.aspx?scid=kb;en-us;824994 12 “Microsoft Security Bulletin MS04-022” Redmond, WA: Microsoft Corporation, 2004 http://www.microsoft.com/technet/security/bulletin/MS04-022.mspx 13 Koetzle, Laura, Ted Schadler, Charles Rutstein, and Robert Whiteley Can Microsoft Be Secure? Cambridge, MA: Forrester Research, March 2003 http://www.forrester.com/ER/Research/Report/0,1338,16275,FF.html 14 For purposes of this document we use only RedHat’s Enterprise Linux distribution in the examples; however all of the Linux distributions provide similar services.

Chapter 2-- Change Management and Risk Analysis 15

Farrell, Nick. “SQL Slammer hits Microsoft,” vnunet.com, January 28, 2003; found at www.networkitweek.co.uk/news/1138312. 16 “Security Configuration Guides,” found at www.nsa.gov/snac/index.cfm?MenuID=scg10.3.1 17 “CIS Benchmarks/Security Tools,” found at www.cisecurity.org/ 18 “Reducing Over 80% of Windows 2000 Professional Vulnerabilities with the Consensus Security Benchmark Settings,” Center for Internet Security, found at www.cisecurity.org/Documents/Reducing_Over_80.htm 19 Shimonsky, Robert J., “Risk Assessment and Threat Identification,” Windows Security.com, August 29, 2004, found at www.windowsecurity.com/articles/Risk_Assessment_and_Threat_Identification.html 20 “RFC 1157 - Simple Network Management Protocol (SNMP),” found at http://www.faqs.org/rfcs/rfc1157.html; “Using SNMP for Network Management,” Windows NT Resource Kit, found at http://www.microsoft.com/resources/documentation/windowsnt/4/server/reskit/enus/net/sur_snmp.mspx 21 VMware, Virtual infrastructure, http://www.vmware.com/ , Microsoft Virtual PC http://www.microsoft.com/windows/virtualpc/default.mspx, and Microsoft Virtual Server http://www.microsoft.com/windowsserversystem/virtualserver/default.mspx 22 Johansson, Jesper, “Oh Patch, How I hate thee, let me count the ways,” Microsoft Corporation, found at http://www.microsoft.com/technet/community/columns/secmgmt/sm0404.mspx 23 Boyle, Neil, “SQL Server Security Checklist,” Database Journal, June 13, 2003, found at http://www.databasejournal.com/features/mssql/article.php/2221471

Chapter 3—It’s all about risk 24

Security bulletin 03-029 retrieved from http://www.microsoft.com/technet/security/bulletin/MS03-029.mspx

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

93

25

Schultze, Eric, Microsoft’s 2004 Security report card, found at http://www.ftponline.com/wss/2004_08/magazine/departments/guestop/ 26 Demanding more from vendors, Rain Forest Puppy as extracted and placed on the following web site: http://www.sbslinks.com/really.htm 27 http://blogs.msdn.com/aaron_margosis/archive/2004/06/17/157962.aspx 28 Microsoft rating of Security Bulletins, http://www.microsoft.com/technet/security/bulletin/rating.mspx 29 File corruptions in AutoCAD files after the installation of Security bulletin 03-026, http://usa.autodesk.com/adsk/servlet/ps/item?siteID=123112&id=3431487&linkID=2476435 30 Technical information for Security bulletin 03-026 - http://support.microsoft.com/?kbid=824136 31 Keep in mind that at that time of these historical bulletins, you had to review the underlying knowledge base article; currently bulletins now include the file versions and detailed information on the security bulletin itself. 32 Patch Management listserve located at http://www.patchmanagement.org/ 33 Patchtalk listserve located at http://www.patchtalk.com/ 34 Microsoft online communities resource page - http://www.microsoft.com/communities/default.mspx 35 Workgroup for Electronic Data Interchange listserves http://www.wedi.org/listserve/ 36 HD Moore’s Metaspoint project to allow for more controlled exploit testing - http://www.metasploit.com/ 37 My analysis of the timeline between the patch release of 03-026 to the web and the resulting release of exploit code to the web - http://www.sbslinks.com/timeline.htm 38 http://www.microsoft.com/technet/security/topics/patch/stdpatex.mspx 39 http://www.sysinternals.com/ntw2k/freeware/listdlls.shtml 40 http://www.microsoft.com/technet/desktopdeployment/depprocess/depprocessris.mspx 41 http://singe.rucus.net/blog/archives/243-Sun-Recommended-Patch-Management-Policy.html 42 http://docs-pdf.sun.com/817-0574-12/817-0574-12.pdf?biga=15 43 http://www.microsoft.com/resources/documentation/WindowsServ/2003/standard/proddocs/enus/Default.asp?url=/resources/documentation/WindowsServ/2003/standard/proddocs/enus/SAG_MPmonperf_06.asp 44 http://www.sysinternals.com/ 45 Infraguard (2004) Technology Risk Checklist retrieved October 10, 2004 from http://www.infragard.net/library/pdfs/technologyrisklist.pdf

Chapter 4—Are we ready to patch yet? 46

http://www.microsoft.com/technet/security/bulletin/MS04-011.mspx http://www.microsoft.com/technet/security/bulletin/MS04-011.mspx 48 http://www.helpdesk.com/software-helpdesk.htm 49 http://support.microsoft.com/default.aspx?pr=securityitpro 50 http://bugzilla.redhat.com/bugzilla/query.cgi 51 http://www.ftponline.com/wss/2004_08/magazine/departments/guestop/ 52 http://www.sans.org/resources/policies/ 53 http://www.sans.org/resources/policies/Server_Security_Policy.doc 54 http://www.sans.org/resources/policies/Aquisition_Assessment_Policy.doc 55 http://www.sans.org/resources/policies/Audit_Policy.doc 56 http://www.sans.org/resources/policies/Virtual_Private_Network.doc 57 http://www.sans.org/resources/policies/ 58 http://www.cisco.com/application/pdf/en/us/guest/netsol/ns466/c664/cdccont_0900aecd800ce939.pdf 59 http://www.microsoft.com/downloads/details.aspx?FamilyID=fe902704-52dd-4bbe-8a75f8fbb76cd28a&DisplayLang=en 60 http://searchnetworking.techtarget.com/originalContent/0,289142,sid7_gci992761,00.html 61 http://download.microsoft.com/download/0/7/e/07ed1953-0ab5-41ea-b5da-41cf8bb9cdae/Quarantine.doc 62 http://www.microsoft.com/technet/community/tnradio/archive/tntrans_01.mspx 63 http://www.sysinternals.com/ntw2k/freeware/listdlls.shtml 64 http://www.sysinternals.com/ntw2k/freeware/procexp.shtml 65 http://dotnet.org.za/kevint/archive/2004/04/01/931.aspx 66 http://www.google.com/ 67 http://www.google.com/grphp?hl=en&tab=wg&q= 68 http://www.microsoft.com/downloads/details.aspx?FamilyID=cebf3c7c-7ca5-408f-88b7f9c79b7306c0&DisplayLang=en 69 http://www.microsoft.com/whdc/devtools/debugging/default.mspx 47

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

94

70

http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57ff-4ae7-96eeb18c4790cffd&displaylang=en 71 http://www.microsoft.com/technet/prodtechnol/winxppro/maintain/xpsp2man.mspx 72 http://download.microsoft.com/download/5/d/b/5db098e2-43d3-48bc-9e5be72ab9c65c61/IPSec_Server2003.doc 73 http://www.microsoft.com/technet/security/bulletin/MS03-026.mspx 74 http://www.microsoft.com/technet/security/bulletin/MS03-026.mspx 75 http://www.microsoft.com/technet/security/bulletin/MS04-011.mspx 76 http://www.microsoft.com/technet/security/topics/patch/stdpatex.mspx 77 http://www.sbslinks.com/really.htm 78 http://info.sen.ca.gov/pub/01-02/bill/sen/sb_1351-1400/sb_1386_bill_20020926_chaptered.html 79 http://www.cio.com/archive/021502/security.html 80 http://www.sims.berkeley.edu/resources/affiliates/workshops/econsecurity/econws/06.doc 81 http://www.cert.org/nav/index_purple.html 82 http://isc.sans.org//survivalhistory.php

Chapter 5—Life beyond patching Microsoft 83

http://www.microsoft.com/technet/prodtechnol/winxppro/deploy/gpdepsp2.mspx Find more information on the Group Policy Management Console at http://download.microsoft.com/download/a/9/c/a9c0f2b8-4803-4d63-8c323040d76aa98d/GPMC_Administering.doc and http://www.microsoft.com/windowsserver2003/gpmc/default.mspx 85 http://rhn.redhat.com/errata/RHSA-2003-280.html 86 http://www.unixwiz.net/techtips/building-source.html 87 http://docs-pdf.sun.com/817-0574-12/817-0574-12.pdf?biga=15 84

Chapter 6—Justifying Patch Management—A Business Tutorial 88

Gartner (2003, 5) 2003 Press Release retrieved October 10, 2004 from http://www3.gartner.com/5_about/press_releases/pr29may2003a.jsp 89 Smith, R (2003) "The Chief Technology Officer: Strategic Responsibilities and Relationships", Research Technology Management, July-August, 2003. and Smith, R (2003) "Maximizing the CTO's Contribution to Innovation and Growth", retrieved October 5, 2004 from www.CTOnet.org 90 American Institute of Certified Public Accountants, AICPA (2004) CPA Financial Manager's Role in a Company retrieved May 23, 2004 from http://www.aicpa.org/cefm/CPA_fincl_mgr_role.asp 91 Gartner 92 Gartner 93 Messner, E (2004, 7) “Rx for Patching Mired in Red Tape” Copyright, 1994-2004 Network World, Inc. retrieved September 18, 2004 from http://www.nwfusion.com/news/2004/070504hospitalpatch.html 94 Stoneburner, G, Goguen, A and Feringa, A (2002, July)The National Institute of Standards and Technology (NIST) Special Publication 800-30 Risk Management Guide for Information Technology Systems retrieved October 17, 2004 from http://csrc.nist.gov/publications/nistpubs/800-30/sp800-30.pdf 95 Stoneburner 96 Stoneburner 97 Stoneburner 98 Stoneburner 99 Stoneburner 100 Stoneburner 101 Cavitt, L (2004) “Questions on HIPAA and FDA approval of Biomedical devices” Online posting September 3, 2004 WEDI SNIP Security Workgroup List http://subscribe.wedi.org 102 Messner, E (2004, 7) “Rx for Patching Mired in Red Tape” Copyright, 1994-2004 Network World, Inc. retrieved September 18, 2004 from http://www.nwfusion.com/news/2004/070504hospitalpatch.html 103 Secretariat: NEMA (National Electrical Manufacturers Association) www.nema.org/medical retrieved October 10, 2004 from http://www.nema.org/DocUploads/50496076-20B1-4DADA81B8B82F808B83E/medical-defending.pdf 104 E-Health Insider is managed and maintained by E-Health-Media.com http://www.e-health-media.com/ 105 Messner, E

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

95

106

AT&T News Release (1998, 11) AT&T Announces Cause of Frame-relay Network Outage retrieved December 26, 2004 from http://www.att.com/news/0498/980422.bsb.html 107 Agilent Technologies (2002, 3) Preventing IP Network Service Outages retrieved December 26, 2004 from http://advanced.comms.agilent.com/n2x/docs/whitepapers/pdfs/sp-insight.pdf 108 Schmerken, I (2004, September 21) “Innovation In Motion” Publisher: Wall Street Technology retrieved October 17, 2004 from http://www.wallstreetandtech.com/story/riskManagement/showArticle.jhtml?articleID=47900742 109 Schmerken 110 Schmerken 111 Hicks, M (2004, October) PayPal returning to Normal eWeek retrieved October 17, 2004 from http://www.eweek.com/article2/0,1759,1676610,00.asp 112 Kellermann, Sr, T (2003, June). Data Risk Management Specialist, Integrator Unit, The World Bank retrieved October 17, 2004 from http://www.cyberpartnership.org/World%20Bank%20Erisk%20Heat%20Map.pdf 113 AICPA Business & Industry Internal Control Task Force (2003) Key Issues for Management - AICPA Implementation Guidance and Tools for Sarbanes Oxley Key Internal Control Issues for Management retrieved October 3, 2004 from http://www.aicpa.org/download/sarbanes/key%20issues%20document%20%20FINAL.pdf 114 Ozier, W (2003, May) Risk Metrics for IT Security retrieved October 3, 2004 from http://www.securitypronews.com/securitypronews2420030805RiskMetricsNeededforITSecurity.html 115 Gordon, L and Richardson, R (2004, April) Economics of Cybercrime Computer Crime Research Center retrieved October 22, 2004 from http://www.crime-research.org/articles/Economics_Cybercrime/ 116 Gordon 117 D’Amico, Anita D. Ph.D. (2000, 9) What Does a Computer Security Breach Really Cost? Retrieved October 10, 2004 from http://www.securedecisions.com/documents/CostsOfBreaches-SANSInstitute.doc 118 Gordon 119 Korostoff, K (2003, 8) The ROI of network security Network World, 08/25/03 retrieved November 29, 2004 from http://www.nwfusion.com/techinsider/2003/0825techinsiderroi.html 120 Microsoft (2004) Windows Server retrieved October 10, 2004 from http://www.microsoft.com/resources/documentation/WindowsServ/2003/standard/proddocs/enus/Default.asp?url=/resources/documentation/WindowsServ/2003/standard/proddocs/enus/SAG_MPmonperf_06.asp 121 Sysinternals (2004) retrieved November 10, 2004 from http://www.sysinternals.com/

Ecora Software Corporation Patch Management Best Practices: A “How To” Guide for Securing the Enterprise

Page

96

In the Real World... IT managers want easy to install and easy to use management software that fits within their budget and delivers immediate value right out of the box. …That’s the Ecora promise.

Ecora provides total configuration management solutions that automate multi-platform configuration reporting, change monitoring, and patch management. Ecora's solutions enhance efficiency and reduce the costs associated with IT compliance, business continuity, and vulnerability assessment while providing the means to monitor change and plan for recovery. Cost effective, easy to implement, and easy to manage IT solutions. Ecora Patch Manager – Don’t let researching and manual patching take over your time – Know You’re Patched, Today! Save time and reduce costs with a focused patch management solution featuring an optional-agent technology for the automated deployment of critical patches and unique reporting capabilities for both, IT administrators and managers. Ecora Enterprise Auditor - An indispensable tool for documenting and managing your IT environment. If you’re addressing Sarbanes-Oxley, HIPAA, CFR Part 11, or other regulatory compliance acts, Ecora automates and delivers audit-ready reports that improve accuracy and, save time and money. Ecora supports enterprise platforms: Cisco Systems, Lotus Domino, Microsoft, Novell, Oracle, HP-UX, IBM, Sun Microsystems, Red Hat, and Citrix. Ecora Dr. Wi-Fi - Continuously monitor the availability and performance of mission critical Wi-Fi networks. Dr. Wi-Fi provides IT managers real-time data with easy-to-read “dashboard” reports and proactive alerts. Ecora DeviceLock - Prevent users with USB drives from stealing your data. Halt unauthorized Wi-Fi networks from gaining access to your valuable information and manage user access to devices.

Solutions for Managing IT in the Real World. For more information about Ecora Software www.ecora.com or 1.877.923.2672

Notice to Readers

The Patch Management Reference Guide does not represent an official position of Ecora Software, The Norwich Group or Tamiyasu, Smith, Horn And Braun Accountancy Corporation and it is distributed with the understanding that the authors and publisher are not rendering legal, accounting, or other professional services in this publication. If legal advice or other expert assistance is required, the services of a competent professional should be sought.

Ecora Software Corporation Patch Management Best Practices

Copyright © 2005 by Ecora Software Corporation

All rights reserved. For information about the procedure for requesting permission to make copies of any part of this work, please call the Ecora Software Copyright Permissions Hotline at 603.436.1616. Otherwise, requests should be written and mailed to the Permissions Department, Ecora Software

Ecora Software Corporation Patch Management Best Practices

About the Authors Susan E. Bradley, CPA, CITP, MCP, GSEC is a Microsoft Most Valuable Professional for Small Business Server and Security. She is Chairman of the Technology committee of the California Society of Certified Public Accountants, Chairman of the Top Technologies committee for the American Institute of Certified Public Accountants (AICPA), and Committee Organizing member and speaker at the American Institute of Certified Public Accountants Tech 2003 and Tech 2004 Conferences. She has written numerous articles on computers and security for the AICPA InfoTech newsletter, the FCG Buzz newsletter, and the California CPA magazine, and is co-author of an upcoming book. She maintains a blog on security topics and issues affecting the Microsoft Small Business Server 2003 platform at http://www.msmvps.com/bradley. Reach her at [email protected] but if she doesn’t get back to you right away it’s because she is sorting through all of the out-of-office messages she gets when she posts to the patch management mailing list. Sign up at http://www.patchmanagement.org/. She is a principal with Tamiyasu, Smith, Horn and Braun in Fresno, Calif. She writes an ongoing column, Ebitz, for AICPA’s InfoTech Update newsletter, and is chair of the Top Technologies Task Force.

Anne A. Stanton, an enthusiastic proponent of networking, Anne A. Stanton is a consultant to consultants, specifically in the Accounting and IT consulting niches. She creates bridges between business and technology. Anne is a member of the AICPA's Top 10 Technologies Task Force, and an active participant in numerous forums including the Certified Information Technology Professionals (CITP) discussion forum, the AICPA Technology Conference Chat user's group, the Association for Accounting Marketers user's group, and the CPA Map group (CPAs discussing issues relating to the management of their practices). She also moderates the NH/VT Upper Valley Microsoft Consultant’s group and is the comoderator of the InterNETnational Microsoft SBS Leaders Group. She is a contributing writer for Accounting Software 411.com, the CPA Technology Advisor, CPA Magazine and the AICPA’s IT Section Newsletter, InfoTech Update, as well as many other industry specific journals. She is an active speaker and has presented at SMB Nation, the AICPA Technology Conference, the OK State CPA State Society conference, and the New York Technology Assurance meeting. She is the editor of the Information Technology Alliance newsletter and an active participant in numerous events sponsored by the AICPA, state CPA societies and the Information Technology Alliance. Reach her at [email protected].

Ecora Software Corporation Patch Management Best Practices