What do you mean the DNS server was being patched -- didn't you
discuss that with the server team? How could the network guys be
working on the core when you were updating the app server? Don't
you guys talk?
Does this scenario sound familiar? As a company grows, managing
change can be a challenging task. Change is necessary for a company
to respond to the needs of the business and its customers. You must
be able to adapt to change in order to thrive. The IT game itself
has changed significantly over the past ten years -- even in just
the past five years. Change can come quickly and in abundance. To
manage change, you must have a formal process in place.
Why Do You Need Change Control?
Most IT shops usually have multiple teams involved in the support
and administration of their systems. These can include teams for
network, security, servers, data center, applications, Web, and
telecom. Rarely does a change impact only a single group. With the
continued emphasis on server consolidation and the constant pressure
to reduce costs, teams are required to do more with less. As such,
a single system may now host multiple applications that used to
be individually hosted. The server team may think that a simple
patch and reboot won't have any impact on the business, but the
EDI folks will be quick to let you know that any reboot impacts
revenue stream. It is important that any and all changes be reviewed
by members of all the teams to ensure that the perceived impact
is correct and the change appropriately planned.
Lack of team communication can also lead to too many changes happening
simultaneously and, perhaps, unnecessarily. By subjecting planned
outages or changes to a peer-review process, it becomes much easier
to see potential conflicts and pitfalls. This also requires more
complete planning from the change requestor as the change must be
documented and spelled out completely for a thorough review. Proper
planning has obvious benefits, including ensuring that the change
requestor has done his homework and has confidence in the change
outcome and ensuring that the change is thoroughly understood by
other teams as well.
Symptoms of Unmanaged Change
Many shops may think there is no need for change control, especially
smaller shops that tend to "roll their own" solutions. There is
some merit to that argument since the smaller the shop is, the better
the communication tends to be. But, if you are a small company,
consider the following symptoms.
How to Get There
- No history or audit trail of changes to a system -- How many
times have you wished you could go back and know exactly when
a particular patch was applied, when the disk was last changed
out, when the operating system was upgraded? Without proper change
control, it can be difficult to keep accurate asset history with
regard to change events. Good troubleshooting practice dictates
that you always first ask "What changed?" when trying to determine
the solution to a problem.
- Excessive or unnecessary downtime -- Systems can be subjected
to unnecessary downtime if changes are not controlled. In the
introduction to this article, there was a hypothetical situation
where an admin was attempting to upgrade an application server
while the network folks were doing work in the core. When teams
use formal communication methods integrated into the change control
process, everyone is aware of what is changing and when the change
is planned. This allows all teams involved to better plan their
changes, thus avoiding conflicts and possible unplanned downtime.
Most of the shops I have worked in have had little or no formal
change control. The changes, if discussed at all, were usually communicated
in a very informal manner: email, during roundtable in staff meetings,
or by phone. To effectively manage change, you must map out a workable
process. Once you have the process mapped, you can then automate
and test it.
You can begin with something as simple as a block diagram or flow
chart. When mapping internal processes, focus on the process and
not the individual or team involved in the process. The process
must stand alone if it is going to be automated. Tying the process
to a person locks that process, and it becomes inflexible.
Think about how changes currently happen in your environment.
Who needs to know about a change; how often should we do changes;
what kind of changes do we do regularly; how often do we need a
major outage window; who needs to sign off on changes (Security,
Executive Management, Facilities...); how do we notify the user
community of our planned changes?
The main consideration in the change management process is establishing
scheduled outages windows. The overriding factor in establishing
these windows is impact to the business. You obviously want to schedule
your changes during a time that presents the least amount of impact
to your company's revenue stream. Negotiating these times with department
heads can be a challenge, however. You will hear from some departments
that no day and time is acceptable to have an outage. But these
arguments typically come from the same folks who won't pony up for
clustered or HA systems as well. Regular, predictable outage windows
are necessary for system upgrades, system patching, application
upgrades, network upgrades, and storage expansion. Without these
windows, the infrastructure cannot expand and grow to meet the business
Once you have established the outage windows, you must create
a process for managing regular change. Consider the following outline
as a simple template with which to begin:
1. Admin submits a change request through a Web interface.
2. The change description is emailed to a change coordinator.
3. All IT-related teams meet once a week with the change coordinator
to discuss the requested changes.
4. The change coordinator approves or denies the request.
5. Secondary and tertiary approvals can be added as needed (security,
6. Approved change requests are communicated to the user community
via a weekly email that summarizes the dates, times, hosts, and
This simple template can be greatly expanded to include much more
detail. See Figure 1 for a flow chart mapping an established change
Once you put a change control process into place, you will immediately
begin to see benefits to your organization and to your infrastructure.
For example, a certain large software company releases their software
patches on a monthly basis. By having regularly scheduled outage
windows for your servers, you can schedule the patches for all of
your machines throughout the month. This brings all of your servers
up-to-date from a security standpoint, users and business owners
know when to expect the outages and your planning becomes just that,
planning, instead of just throwing patches on when you get a chance.
If you include a security checklist as part of the approval process,
your changes get the benefit of being reviewed by members of the
security team, who are focused on one thing and one thing only.
Many times in our rush to get a project moving, we focus only on
the event itself and don't take security concerns into play. In
today's world, security must be involved in just about every change
With an established and enforced scheduled outage window, the
users can better plan their work schedules. It is much easier to
have regularly scheduled outages than it is to pick a date and wait
for each and every department to tell you why you can't have that
date and time.
By integrating your change control processes into a trouble-ticket
system or an asset-management database, you instantly have an audit
and history trail of changes to all of your assets. Asset change
data is invaluable for establishing and predicting trends in your
infrastructure -- from failure trends to upgrade release schedules.
This short- and long-term data is of great benefit when putting
together two- and five-year plans for your operations.
The only constant is change. And in the world of IT, you must
learn how to adapt to and how to manage change. I have presented
a brief overview of a simple change control procedure that can be
used as a template to help get your change management under control.
Peer review, predictable outage windows, management signoff, and
setting user expectations are all essential ingredients to an effective
change control process.
Bob Ess is a Senior Manager responsible for Data Center Operations
for Fujitsu Network Communications in Richardson, Texas. He is the
author of several articles related to Unix systems administration.
He has 25 years experience in the IT field and 12 years as a Unix
systems administrator. He can be reached at: email@example.com.