 Live
from Sun -- It's Upgrades!
Peter Baer Galvin
One of the most difficult, challenging, risky, and downright unpleasant
tasks a sys admin can perform is an operating system upgrade. There
is a lot to plan, and a lot that can go wrong. If problems arise,
restoring the system to its un-upgraded state also can be time consuming
and carry its own risks. When the process is all over, the poor
sys admin can look forward to testing the new operating system and
tweaking it so everything that used to work does so again. Sun has
come to the rescue with a new facility -- Solaris Live Upgrade.
This month in the Solaris Companion, I will look at Live Upgrade
-- theory and fact.
It must be frustrating to the engineers that work on an operating
system release -- they do all that work, and their users wait as
long as they can to implement the new version. Sometimes the lag
time is years. Some sites are still running SunOS 4.1.4 (and even
earlier!). Of course, there are a variety of reasons for "rev lag".
One common reason is application compatibility. If an application
is homegrown, no longer supported, or of unknown origin, it may
have problems running on the newest release. Sun works hard to assure
that each release is binary compatible with the others, so application
compatibility is likely not the case here. The next most common
delayer is support. Most sites only run supported applications in
production. If the application vendor is not yet supporting the
new operating system release, or requires an application upgrade
for the application to be supported on the new operating system
release, the operating system release stays put.
The final cause of "rev lag" is the difficulty of upgrading. No
matter how compelling the features of the new release may be, the
pain of performing the upgrade causes hesitation. Some sites need
to spend months planning an upgrade. Just consider a site that has
no downtime allowed, is clustered for high availability, and is
replicated for disaster recovery. How would you go about upgrading
that environment?
Live Upgrade
Sun introduced the Live Upgrade feature in Solaris 8 to address
the pain problem. If upgrades were easier and faster to perform,
more sites would be likely to perform them, and sooner. Overall,
this would be a win for Sun, as they would have fewer old-operating
system support calls, fewer patches to release against old releases,
and of course more sites getting to know and love the features and
benefits of the new release (and consequently less likely to switch
to other operating systems). The Solaris engineers would be happier
too, seeing people actually using the code they spent so much effort
to create.
Live Upgrade is available with Solaris 8 and 9, and can be used
on systems running Solaris 2.6 and above to upgrade the system to
Solaris 8 or 9. Its basic design creates a copy of your "critical"
partitions to a spare, bootable disk. "Shareable" partitions need
not be copied. Critical partitions include /, /usr, /opt, and /var.
The upgrade then executes against the copy of your critical partitions.
The update can be performed while the system is in production. Once
the upgrade is complete, a reboot from this new "boot environment"
(BE), brings up the system in upgraded mode. If testing shows that
all is well with the upgrade, then the system can remain as is,
and the critical partitions on the original disk will be available
for other use. If the testing fails, or if the upgrade itself was
a test and the system needs to remain running the older operating
system for a while, then a reboot off the original boot disk will
bring the system back to its original state.
The concept seems fairly simple. In fact, experienced systems
administrators perform upgrades manually using a similar set of
steps. Live Upgrade has a few advantages over the manual method
though:
- Downtime is reduced to a single reboot, rather than the upgrade
time plus the reboot.
- Reverting to the previous environment is fast and easy.
- Live Upgrade is not limited to copying critical partitions
to another disk of the same size, or even to other partitions
of the same size. It can resize, merge, and split the critical
partitions as part of its copy and upgrade function.
- Any changes made on the original critical partitions during
the copy time are duplicated to the upgraded partitions when they
are "activated" (made ready for boot use).
- Provides easy management of multiple boot environments (for
development, QA, and so on).
Prerequisites
As with any major change, before using Live Upgrade, be sure you
have a working backup copy of the boot disk partitions and the target
disk partitions. Having a third boot disk available to keep a separate
copy of the boot environment is a good idea as well, because even
if things go horribly wrong, you would have a known-good boot disk
to recovery from without resorting to tape restoration.
The Live Upgrade tool requires that the system to be upgraded
have the current patch cluster installed. So, before Live Upgrade
can be used, a patch installation may be required. Of course, this
step might require its own QA cycle.
A list of packages that are required by Live Upgrade is included
in the documentation. The host must have these packages installed
or Live Upgrade will fail.
Note that the root partition that is to be created by Live Upgrade
cannot be a Veritas Volume Manager volume or a DiskSuite metadevice.
It can be on another legal boot device, or even on the current boot
disk. Finally, the spare space left over when a 72-GB disk is used
as a Solaris root disk is useful for something besides storing MP3s!
Note also that the very useful Solaris Flash (in which an image
of a system can be captured and reinstalled via Jumpstart) can be
used in conjunction with Live Upgrade. The new BE could be a restored
archive as built by Solaris Flash, for example.
Performing a Live Upgrade
The first step (after the planning of course) is to install Live
Upgrade from the Software 2 of 2 CD of the target operating system.
The release from which Live Upgrade is installed is the only release
that the system can be upgraded to. There are two methods of using
Live Upgrade: CLI and CUI (character user interface -- curses-based
menus). The "lu" command fires up the CUI, which is quite serviceable.
For the CLI version, there are several commands, all prefaced with
"lu", such as "lustatus" and "luactivate". The command-line version
seems to have more flexibility and more options than the CUI. And,
it is scriptable and repeatable. It should be used for complex or
repetitive Live Upgrades, while the CUI is best used for one-off
situations.
To begin, a new BE is created via the CUI, via the "create" selection.
It asks for the current BE (i.e., the current critical partitions)
and a name for the new one, as well as the devices to use for the
new BE's partitions.
A major limitation of the CUI is that no multi-CD upgrades can
be accomplished that way. In this example, my system was running
Solaris 8 07/01 and was upgraded to Solaris 9 05/02. The Solaris
9 release was on multiple CDs, rather than one DVD. To perform a
Live Upgrade to Solaris 9, CLI use was required. So, to start the
upgrade, do:
luupgrade -u -n Solaris9 -s \/cdrom/cdrom0/s0
where "Solaris9" is the name of the BE to be upgraded, and the argument
to "-s" is the path to the media containing the operating system image
to be used for upgrading.
Once this first disk finished, the following command caused the
packages from the second disk to install (and an installation GUI
to pop up and ask a couple of questions):
luupgrade -i -n Solaris9 -s /cdrom/cdrom0
Upon completion, upgraded BE file systems are unmounted, even though
the output from the luupgrade command advises that the log
file be read. This takes a simple step of manually mounting the BE
root partition, and viewing the var/sadm/system/data/upgrade_cleanup
file. This file contains a list of all changes that were made, and
could be useful if, for example, you have a custom sendmail.cf file
that needs to be moved back into place.
The system knows which of the (possibly many) BE environments
should be used at the next boot time by having one specified as
"active". Switching between BEs is as simple as setting the interesting
one to active, and rebooting. This activation can be done via the
CUI or via the luactivate command, as in:
luactivate Solaris9
Once an environment is activated, a shutdown or an init 6 is required
for the activation to complete. There are shutdown scripts that must
be executed for the switch to take place. Live Upgrade automatically
changes the EEPROM boot-from settings, so the active environment is
the one that will be run next.
Also as part of an "activation", a synchronization can optionally
be requested. If requested, at boot time, any changes made to the
"old" environment between when the new BE was created and when it
was activated are copied to the new BE. Unfortunately, it is unclear
exactly what this option does (and how it does it). For example,
I created a new file in /, after I created the new BE but before
I activated it. The file was not copied to the new BE as part of
the synchronized activation. Perhaps the synchronization only occurs
to specified files (e.g., /etc/passwd). The documents do not explain
this, so we are left to guess.
Cautionary Notes
Using Solaris Live Upgrade with volume managers (Veritas Volume
Manager and Solaris Disk Suite and Volume Manager) can be a challenge.
Live Upgrade will not recreate mirrored volumes for instance. Sharable
file systems can be on managed volumes, but critical file systems
cannot be created on managed volumes. Some experimentation will
be required for Live Upgrading various combinations of volume management.
Results and Conclusions
Jay Daliparthy, Senior Product Manager at Sun for Solaris Installation
Tools, provided useful information for this column. He also commented
that he has received messages from Solaris Administrators thanking
him for saving their social lives. The power of Live Upgrade is
such that many nights and weekends can be reclaimed by those using
it. Rather than consuming many off-hours, Live Upgrade can perform
an upgrade during normal hours, with just the final reboot and testing
done during non-production hours. Combining that life improvement
with the Live Upgrade technical abilities makes this feature powerful
and welcome. Jay also imparts that improving installation and upgrade
technologies is a strategic direction for Sun. Hooray!
Thanks to Live Upgrade, the Solaris best practice for boot devices
is now to have a spare disk (or spare space on a boot disk) available
for upgrading. If pressed, you could use the mirror of a boot disk
by first breaking the mirror, but you would need to be willing to
run un-mirrored for a time.
For more details on Live Upgrade check out the Sun whitepaper
on Solaris installation in replicated and high availability environments:
http://wwws.sun.com/software/whitepapers/wp-solarisinst/ \
solaris_installation_deployment.pdf
the whitepaper on provisioning and change management tools:
http://wwws.sun.com/software/whitepapers/solaris9/prov_cmtools.pdf
and the Solaris doc set Live Upgrade manual:
ftp://docs-pdf.sun.com/806-7933/806-7933.pdf
You may also want to review The Solaris Companion column from April
2002, which provides a complete upgrade methodology:
http://www.samag.com/documents/sam0204i/
Thanks to Manny Korkodilos for providing input into this column.
Side Notes
In The Solaris Companion, November 2002, features and functions
were compared between the new Solaris Volume Manager and Veritas
Volume Manager. No performance comparisons were included in that
column, however. Veritas has released a whitepaper from a third
party regarding the performance of the two. If you are considering
SVM and VXVM in a performance-centric environment, take a look at
the article:
http://www.veritas.com/news/press/FeatureArticleDetail.jhtml?NewsId=21500
Peter Baer Galvin (http://www.petergalvin.org) is the Chief
Technologist for Corporate Technologies (www.cptech.com), a
premier systems integrator and VAR. Before that, Peter was the systems
manager for Brown University's Computer Science Department. He has
written articles for Byte and other magazines, and previously
wrote Pete's Wicked World, the security column, and Pete's Super Systems,
the systems management column for Unix Insider (http://www.unixinsider.com).
Peter is coauthor of the Operating Systems Concepts and Applied
Operating Systems Concepts textbooks. As a consultant and trainer,
Peter has taught tutorials and given talks on security and systems
administration worldwide.
|
