|
By Philip E. Courtney A
wide array of new operating systems and applications at ReliaStar Financial
Corp. (Minneapolis, Minn.) resulted in the distribution of data to the
far reaches of the enterprise. Coinciding with this distribution were
the requirements for constant, reliable application access to all business-critical
information. Another requirement was the demand for centralized management
of backups for data residing on a diverse collection of UNIX, MS-Windows
NT (Microsoft Corporation, Redmond, Wash.) and Novell (Provo, Utah) servers
as well as EMC Symmetrix (EMC Corporation, Hopkinton, Mass.) storage devices.
By taking an assessment of existing technologies, and by applying the
long-standing disciplines inherent with its mainframe heritage, the IT
organization at the company implemented a cross-platform management solution
fully exploiting the capabilities of its mainframe-based automation technologies.
"Our operations group has a strong mainframe background," explains Tom
Becchetti, senior capacity planning analyst, ReliaStar Life. "We felt
it was important to leverage our existing knowledge base as well as our
existing technologies." While some of the mainframe technol-ogies leveraged
by the company include its RACF security system (IBM Corporation), its
JOBTRAC job scheduling system and its CA-1 tape management system (both
from Computer Associates International, Islandia, NY), the bulk of its
successful solution for cross-platform data access and management came
from making the most of the capabilities of its enterprise-wide backup
and restore product. FDR/UPSTREAM along with FDRSOS with its UPSTREAM/SOS
companion product (all from Innovation Data Processing, Little Falls,
NJ) met the criteria required.
FDR/UPSTREAM is a network-based, storage management tool for centralized,
automated and unattended backup/restore and archival from PC/LAN/UNIX
systems to OS/390 MVS mainframe server tape and DASD. The product provides
full, incremental and differential backup and supports PC servers running
NT, Novell, OS/2 as well as UNIX servers and client workstations.
ReliaStar complemented FDR/UPSTREAM with FDRSOS and UPSTREAM/SOS for the
open systems data that resides on their EMC Symmetrix Array. FDRSOS works
only with open systems data and performs a physical image backup of the
SCSI or Fiber connected open systems volumes that are on Symmetrix devices.
These backups realize the greatest amount of speed because they are performed
across the ESCON channel to mainframe tape. UPSTREAM/SOS can determine
if a file has changed and back it up through specially formatted disk
volumes shared by ESCON and SCSI channels. According to Becchetti, FDR/UPSTREAM
recognizes file systems, performs incremental backup and provides file
level granularity for recovery. UPSTREAM/SOS provides greater backup speed
by avoiding the network--a crucial factor for ensuring continued cross-platform
application availability and maintaining good response times. "FDRSOS
and UPSTREAM/SOS technologies let us reserve the network for our applications
and not just the backups," says Becchetti.
Yet, for ReliaStar, expanding the use of FDR/UPSTREAM and FDRSOS products
enabled the company to meet not only the goals of centralized storage
management, but also helped deliver centralized, cross-platform access,
control and management of multi-platform tasks and jobs, as well as expanding
support for their existing firewall security features.
"We've leveraged many of the features in the products," explains Becchetti.
"They've proven they are flexible enough and strong enough to handle requirements
well beyond just cross-platform backup and recovery."
ReliaStar Financial Corp. is the eighth largest publicly held life insurance
holding company in the United States on the basis of revenues (Fortune
magazine, April 1999). ReliaStar's subsidiaries and sales offices throughout
the United States, as well as its European and Latin American reinsurance
sales operations, have over four million worldwide customers. As a comprehensive
financial services company, ReliaStar provides a wide range of value-added
investment, insurance, banking and other products and services to its
customers. The company explains the complexities of insurance and investments
using straightforward language that consumers can clearly understand.
Although IBM MVS mainframes have long provided the company's processing
horsepower, ReliaStar has recently seen the implementation of many new
cross-platform business systems beginning with simple client/server applications
to complex, N-tier, web-based, e-commerce systems. Its distributed environment
has grown from a small collection of eight Novell servers to an environment
that now includes a 1/2 terabyte of storage on nearly 70 MS-Windows NT
servers, 1.5 terabytes on approximately 60 UNIX servers supporting a mix
of Sun Solaris, IBM AIX and HP-UX operating systems, and 3/4 of terabyte
on an EMC 5430 Symmetrix device shared between MVS and UNIX operating
systems.
"Like many companies, we've evolved from a mainframe-only environment
where we once had the luxury of downtime to an environment supporting
the 24x7 needs of our business," says Becchetti. "We've installed a number
of different platforms and many different types of storage devices and
we must ensure that applications can access all of their data."
Still, several years ago, back when a web still belonged to a spider and
e-commerce was a misspelling, ReliaStar found it challenging to support
growing requirements for cross-platform application access. In many situations,
the driving reason for the challenge was a distinct lack of sophistication
in the backup and restore process.
"Simple things like missed tape mounts or bad tapes would cause a backup
to fail," says Becchetti. "The failure of a backup can result in an inoperable
application if the data cannot be restored." He also notes that a distributed,
manually operated, server-based solution lacked the intuitiveness and
automation needed to perform restores. "Not only did we have to figure
out which tape held the file," Becchetti contends, "we also had to manually
mount tapes to perform the restores. That made for some long days when
you had to perform 30-40 restores each week." However, as an organization
with strong mainframe roots seeking to extend the benefits of automation
and centralized storage management, the implementation of FDR/UPSTREAM
provided the ability to harness the resources already in place on the
mainframe. "All of our mainframe tools and procedures were already in
place," says Becchetti. "We felt it was far better to take advantage of
our scheduling system, tape management system, tape silos and mainframe
storage management system."
The implementation of FDR/UPSTREAM enabled the company to better secure
the data used by cross-platform applications. Additionally, it allowed
ReliaStar to increase its return on its investments made in scheduling
and tape management systems as well as its FDR/ABR (Innovation Data Processing)
storage management system.
"Our concerns about providing backup and recovery support for cross-platform
applications were virtually eliminated with FDR/UPSTREAM," says Becchetti.
Initial backup activities with FDR/UPSTREAM began with the retrieval of
server data from traditional hard drives. Data is backed up across the
ReliaStar network--a 100 mbit Ethernet connection using either the company's
CISCO router and CIP card or through an OSA adapter, based upon the com-pany's
differing application configur-ations. FDR/UPSTREAM first required Reliastar
to create a baseline set of jobs used for backup. These were full-volume
backups designed to capture all of the files on each of the distributed
servers--including user data as well as a mix of Sybase databases. The
distributed backup process is launched when mainframe-based JOBTRAC submits
an FDR/UPSTREAM batch job that uses a TCP/IP or LU6.2 connection to the
server environment to perform the backup task. During each process, FDR/UPSTREAM
tracks every file name and other related information such as file creation
date, last modification date, backup date/time, and server name. It also
provides immediate notification in the event of failure such as an abend,
or if the backup task was unable to retrieve specific files.
"FDR/UPSTREAM is integrated with our mainframe automation tools and manages
our distributed data regardless of the type of server or type of file
or database," says Becchetti. "Centralization means that we only need
to monitor tasks from one location."
Despite placing distributed backups under a single management umbrella,
continued growth in the distributed environment soon posed a new problem:
a dramatic increase in tape utilization. "We have a hodge-podge of systems
and applications and a diversity of specific servers for applications,
databases and networks," says Becchetti. "An increasing number of new
applications required their own servers and we could no longer get by
with using one tape per server."
Combined with the rise in tape utilization came an escalation in network
traffic as well as an increase in the amount of time needed to perform
the backups--a challenge that grew increasingly unacceptable as application
availability demands grew larger and cross-platform data access grew more
prevalent.
Responding to this challenge, ReliaStar implemented the FDR/UPSTREAM Automatic
Duplicate File Support and Forward Merge Backup facilities. Automatic
Duplicate File Support reduces backup time and storage usage by automatically
identifying duplicate files and updating just a pointer to the common
copy, thereby eliminating the overhead of transmitting duplicate files.
The Forward Merge Backup reduces network traffic by building upon the
baseline created by full volume backups and retrieving from the server
only the files that have been changed. FDR/UPSTREAM reduces tape utilization
and network traffic by constructing a new, synthesized image of each server
without sending or transmitting unchanged files. The completion of the
Forward Merge Backup results in a single tape set that is representative
of the server from the point in time of that backup. "This functionality
provides us quick access to any files that need to be restored," says
Becchetti.
Still seeking to further exploit existing technologies and improve backup
efficiency, ReliaStar also implemented another FDR/UPSTREAM feature, its
MVS Backup Data Set Migration. Launched via a JOBTRAC request, the Backup
Dataset Migration is a disk pooling technique that places backup data
directly from the multiple servers onto a mainframe DASD device where
they are subsequently consolidated onto a single tape set managed by CA-1.
"Not only does the Migration help us reduce tape utilization," Becchetti
contends, "it also allows us to improve the efficiency of the restores.
With all data from a variety of servers on a single tape set, it's easy
to group files by application."
Additionally, Becchetti emphasizes, the Backup Data Set Migration helps
speed the restore process by allowing files to be placed directly onto
the correct open systems target disk from tape without having to use an
intermediate restore or pool volume.
Demands for 24x7 application availability presented ReliaStar with the
dual challenges of maintaining cross-platform data protection while ensuring
adequate application response time.
One of the methods implemented by ReliaStar to ensure application availability
was the installation of the EMC Symmetrix devices and the creation of
independently addressable Business Continuance Volumes (BCVs) to store
data for applications running on its mainframe, UNIX or MS-Windows NT
platforms. The BCVs are mirrored copies of production volumes which can
be separated for non-disruptive backup.
According to Becchetti, because the EMC devices work with such a broad
range of operating systems, the company simplified its ability to select
the storage mechanisms for its many diverse platforms. "We were spending
too much time figuring out which volumes were needed for which machines,"
he says. "We're not in the hardware business and the EMC Symmetrix devices
allow us to store data from MVS and our distributed systems on the same
device."
The company also took advantage of the EMC Symmetrix Remote Data Facility
(SRDF) software that provides an online, host independent, mirrored data
storage solution to duplicate production data. Duplicated data can reside
on another Symmetrix in any location. This proved to be the key for successful
disaster recovery testing for ReliaStar.
ReliaStar implemented FDRSOS to support its EMC volumes and further extend
centralized storage management capabilities. By making the EMC devices
connected to the MVS operating system visible as pseudo-MVS DASD volumes,
FDRSOS performs a physical disk backup of open systems data at the Symmetrix
logical level that bypasses the network and writes data directly to MVS
tape.
For incremental backup and file level granularity ReliaStar also uses
UPSTREAM/SOS. After determining that a file has changed, UPSTREAM/SOS
performs a backup through a specially formatted disk volume (local backup)
that is shared by both ESCON and SCSI channels. However, to ensure speed
and to eliminate impact on application processing, the backup data goes
across the ESCON channel and avoids the network. Furthermore, the "local
backup" volume can be used either simply as a "pass through" data transmission
medium, with no data remaining resident on it, or can be configured for
data to remain on the local backup volume for later immediate retrieval.
"The system can also manage the space on an ongoing basis to retain as
much data as possible locally," says Becchetti.
Equally important, for the HP/UX based Sybase databases, ReliaStar has
implemented the SQL Backtrack interface of UPSTREAM/SOS that provides
throughput of eight gigabytes per hour per occurrence, or server. This
enables the backup process to avoid the network, keeping traffic at a
minimum while still providing outstanding performance.
"Using a combination of FDR/UPSTREAM across the network for some servers
and FDRSOS and UPSTREAM/SOS backups for others, we're easily backing up
90 servers each night with plenty of room for scalability," says Becchetti.
And, in addition to supporting cross- platform application data requirements,
the high throughput rate assisted in speeding the daily copy of several
pro-duction Sybase databases to the test environment. "Copies of these
production databases are needed in the test environments for several different
applications running on several different platforms," says Becchetti.
"Availability requirements for the test databases are almost the same
as the requirements for production."
ReliaStar business-critical applications depend upon Sybase databases.
So critical is this data that, according to Becchetti, it must always
remain accessible. "The data must remain available no matter what happens,"
he says. "We've built in the mechanisms so that the application can run
from the EMC BCVs, if necessary."
Just as ReliaStar sought to further exploit the functionality of its existing
mainframe-based tools, the time had come to expand usage of FDR/UPSTREAM.
In some cases, this expanded usage put the product through a number of
paces that provided unanticipated benefits. For one, the additional functionality
has helped the company overcome a layer of isolation for e-commerce applications
and file transfer mechanisms. According to Becchetti, many of these systems
are designed to have data from business partners and customers delivered
to several pre-established points outside the ReliaStar corporate firewall.
"Customer data is 'dropped-off' outside the firewall where it can be analyzed
prior to introduction into the system," he explains.
One of the original challenges experienced by the company was that of
continued data protection; while outside the firewall, the data remained
exposed to unauthorized access. Investigating methods for continued data
protection, the company discovered that it could use FDR/UPSTREAM's File
Transfer capabilities to migrate these files from the unprotected area
outside of the firewall to a secure RACF protected location.
FDR/UPSTREAM by providing integration with its RACF host security system,
operates within the confines of pre-established security rules. "We pretty
much set the access authority to allow UPSTREAM's File Transfer capability,
with its ASCII to EBCDIC translation ability, to transfer data through
the firewalls," says Becchetti. "After the data hits the mainframe, it's
protected by RACF security. And can be deleted from the outside environment."
The tradition of leveraging existing technologies continued when the company
sought a method for automating and managing enterprise-wide tasks--tasks
that resided outside of the realm of just backups and restores. While
a couple of methods were explored, ReliaStar instead opted to again look
to the functionality of FDR/UPSTREAM for an answer. "UPSTREAM provides
two-way communications between MVS and other platforms in our environment
with its pre- and post-processing capability," Becchetti explains. "Combining
that communications functionality with MVS JCL controls on the mainframe,
we felt we could effectively launch, manage and monitor multi-platform
tasks."
Starting with its MVS scheduling system, FDR/UPSTREAM jobs are submitted
into the MVS environment. Contained within those jobs are a series of
platform specific scripts that launch the tasks on the target platforms.
"The scripts direct what tasks need to be performed and the UPSTREAM communications
also provide a way to send status, error and completion messages to the
data center operations console," says Becchetti.
Also contained within these ReliaStar constructed jobs are straight-forward
MVS JCL condition code checking and operational control mechanisms. These
functions are used to guide the FDR/UPSTREAM jobs through the various
tasks, moving between job steps based on the task return codes and messages
delivered from the distributed environment.
"We're using plain old condition code checking, 'IF-THEN-ELSE', and using
the UPSTREAM MVS batch functionality 'CONV=WAIT' to control the tasks,"
Becchetti says. "Each JCL step waits for UNIX or NT to return the appropriate
message before moving onto the next step and issues the return code to
FDR/UPSTREAM MVS."
As ReliaStar continues to grow, new applications will surely be added
to support new business requirements. This growth will also see the implementation
of new servers to support cross-platform applications. According to Becchetti,
growth will drive the demand for increased access to the file information
contained within FDR/UPSTREAM from a variety of users throughout the organization.
"More and more users today want to perform their own restores," he says.
"And we'll want to provide them with that ability."
Providing users with the ability to perform their own restores can be
realized through the implementation of the new JAVA interface of FDR/UPSTREAM.
"It's a JAVA interface that shows users all of their files regardless
of where they came from or where they are stored," says Becchetti. "They
can perform their own restores based on file names. To them, it's magic."
Still, Becchetti feels confident that the organization is prepared to
further leverage its existing technologies. Besides the JAVA feature,
he also intends on taking advantage of FDR/UPSTREAM's NT SQL Server support,
and will suggest its Oracle support to the database group. Most important,
he notes, is the fact that, as an early user of the product, FDR/UPSTREAM
continued to meet the scalability challenges faced by the company during
its continued growth. "Rapid growth can sometimes be difficult but UPSTREAM
easily grew with us as we added new configurations and hardware, and I
fully expect it to scale in the future."
ABOUT THE AUTHOR
Philip E. Courtney is a freelance writer based in Bushkill, PA.
|
