to Jeroen Kuijlen, CZ's I.T. Director, Dutch health insurance companies
now face the same challenges as just about every other company in
most other industries; listen to what your customers are demanding
and then give it to them. Quickly!
now on, the choice of Health Insurance provider in The Netherlands
will no longer be based solely on price. And the companies that
eventually survive will be the ones that react quickest to customer
Already, CZ has seen an increase in requests for a faster and more
comprehensive access to information on all aspects of health care.
As a result, CZ has recently launched a new personal "My CZ"
service for its customers, allowing them to go online and get instant
access to key information, such as the status of claims payment.
The system also lets them update basic personal information.
the public domain of CZ's website (www.cz.nl), non-customers and
prospects can obtain online quotes for health care insurance, and
optionally convert those quotes into real policies. It is also possible
to check the lengths of waiting lists for specific treatment at
hospitals in The Netherlands.
The majority of CZ's client data, which forms the basis of these
new online services, is stored in a home-written application called
M.A.R.S. This system was originally developed and maintained as
a DATACOM database, but it has now been successfully converted to
IBM's DB2 database system.
Kuijlen explains the reasoning behind this move:
“The internet has now
become a major sales channel. DB2 gives us far greater flexibility,
because the information from the production M.A.R.S. DB2 database
is now heavily used within our web applications. DB2's ability to
interact with newer technologies, such as Java and Websphere helps
us to develop services that keep us one step ahead of the competition...”
example, using a Service Oriented Architecture with Webservices,
allows health providers (e.g. hospitals, doctors, dentists, etc)
to invoice CZ online for the services they provide to insured customers.
well as the conversion of the actual client data from DATACOM to
DB2, CZ also performed a migration of all its data to a new IBM
Shark ESS 2105 Dasd Subsystem. Over 1Tb of data is now stored on
the Sharks, 250Gb of which is occupied by the DB2/M.A.R.S. databases.
issue that had to be resolved before the new DB2/M.A.R.S. system
went live was the method that would be employed to perform the daily
backups of the databases. This project was headed by CZ Systems
Programmer Peter Karel...
Dutch health insurance industry will see some major changes in the
coming years, and competition between the health insurance providers
will be intensified.
costs and retaining the existing client base while at the same time
attracting new customers, are just some of the challenges that lie
insurance providers will also need to have the appropriate systems
in place to make it easy for new customers to switch between health
is ready to face these new challenges. CZ is ready for the competition!
internal M.A.R.S. customer database, (based on IBM's DB2), is at
the heart of CZ's future I.T. strategy. And as the importance of
the M.A.R.S. system continues to grow, so too does the need to provide
a reliable and secure backup & restore of the DB2 databases,
without affecting the services now being offered.
user story looks at how CZ has achieved that goal, utilising Innovation's
FDR/ABR & FDRINSTANT.
a turnover in 2002 of 2.5 billion Euros and nearly 2 million
customers, Tilburg-based CZ is currently The Netherlands'
largest independent provider of Health Care insurance products
was Peter's job to investigate the possible options for backing
up the DB2 data.
a company, CZ was new to DB2, but we had numerous DB2 consultants
on-site overseeing the development of the new databases and the
eventual conversion of the data. Of course, those guys had their
own ideas on how best to handle the backup and recovery of the databases,
but I wanted to review all of the potential options.”
was keen to take full advantage of the FlashCopy feature on the
IBM Shark ESS 2105s. A similar Dasd Replication product had previously
been employed by CZ with the daily backups of the old DATACOM M.A.R.S.
system. After quiescing activity and creating duplicate offline
point-in-time copies of the DATACOM disk volumes, those offline
copies were then brought online and backed up by DFDSS, while access
to the prime copies was handed back to the users again. However,
as Peter explains, although this process helped to reduce the "backup
window", there were several critical issues when it came to
restoring data from the DFDSS backups...
get the duplicate disks online for DFDSS to back them up, we had
to run a VTOC renaming utility. This then created discrepancies
between the VTOC, VVDS and Catalog, which made restoring individual
files (especially VSAM and SMS-managed) a real problem for us.”
CZ was to employ a similar backup technique for the DB2 system (which
makes heavy use of VSAM files), Peter knew he had to solve the problems
on restores. The eventual solution came with Innovation's FDRINSTANT.
had already purchased FDR/ABR and FDRINSTANT, and Peter was well
aware that the FDRINSTANT module could be used to backup offline
duplicate volumes created by Data Replication products like FlashCopy,
SnapShot, ShadowImage and TimeFinder, without the need for those
duplicate volumes to be renamed and/or brought online before being
backed up by FDR or ABR.
FDRINSTANT had, in fact, already been implemented on CZ's non-DB2
data. But it wasn't until Peter was discussing the DB2 restore issues
with an Innovation support technician that he realised it was also
possible to utilise FDRINSTANT on the DB2 data as well. This facility
was significantly enhanced with the DB2 LOG SUSPEND command (see
below) at Version 7 (and Version 6 with PTFs).
successfully tested the procedures and then put forward a proposal
for the utilisation of FDRINSTANT in the production DB2 backup process.
His next task was to convince the DB2 DBA consultants!
first, they weren't at all keen on the idea! To be fair, they were
familiar with IBM's Image copy utility and they were reluctant to
adopt a non-IBM tool to backup the DB2 databases. But once we showed
them what FDRINSTANT was capable of doing, they couldn't wait to
Peter says that some of those DB2 consultants have since moved on
to new assignments at other companies where they are now actively
promoting the use of FDRINSTANT with FDR/ABR for DB2 backup and
the introduction into DB2 of the Log Suspend command, the benefits
of FDRINSTANT with FDR/ABR can now also be employed in the backup
and recovery of DB2 data.
the Log Suspend has been issued to stop DB2 update activity, a combination
of FDR/ABR and FDRINSTANT, together with hardware "replication
techniques" (FlashCopy, TimeFinder, SnapShot, ShadowImage),
can create a true backup of the DB2 databases in a matter of seconds.
FDRINSTANT can then backup the offline DB2 databases to tape while
full read-write access to the 'live' copy can be given back to the
users (via the LOG RESUME command). If desired, the offline copies
can be retained and used as input to an FDR or ABR 'instant' restore.
with FDRINSTANT and Log Suspend, the DB2 backup window no longer
runs into several hours. Instead, DB2 databases can now be fully
secured with FDR or ABR without having to close DB2.
you may recall, aside from the benefits provided by FDRINSTANT on
the DB2 backups, Peter's main goal was to improve and simplify restores.
And he and his colleagues are very happy with the flexible restore
features offered by FDR/ABR, especially when restoring individual
DB2 Tablespaces (see side bar).
van den Hout is one of the CZ Storage Administrators who is responsible
for running "on request" DB2 restores:
“It's so simple. I just run a regular FDR/ABR
restore of the DB2 tablespace(s). Because the FDRINSTANT-driven
backups were taken directly from offline duplicate volumes, I don't
have to re-label disks, or rename datasets. I just restore the data
and let the DB2 guys issue the START DATABASE commands.”
is also one of the team responsible for CZ's Disaster Recovery implementation,
which is tested regularly at an IBM recovery centre in the west
of The Netherlands.
is a straight-forward and effective DR plan. We restore our
1-pack MVS system with Innovation's Stand Alone Restore (SAR). Then
we restore all our Dasd volumes (including the DB2 disks) with FDR/ABR.
Once everything is restored, we IPL our full system and away we
Karel is happy. With the combination of FlashCopy and FDRINSTANT
he can now ensure a regular daily backup of the DB2/M.A.R.S. system
with the minimal disruption to services.
van den Hout is happy. In the event of problems, he knows he can
quickly and easily restore individual DB2 databases, or the entire
DB2 system. And this can be done without requiring specialist DBA
Jeroen Kuijlen is definitely happy! He can clearly see the benefits
of the technical advances made by Peter, Johan and the rest of his
DB2/M.A.R.S. system is at the very heart of the services we are
now providing to our customers. It is essential that we fully secure
the data on a daily basis.”
has to be backed up with minimum disruption to services. And in
the event of local problems or a full disaster, we have to get the
system running again as soon as possible.”
confident that we can do all those things with FDR/ABR and FDRINSTANT.”
a variation of the DB2 RECOVER command (RECOVER LOGONLY) allows
applying of just the Log records to a Tablespace that has already
been restored outside of DB2 control-i.e. with FDR/ABR.
First, issue the STOP DATABASE SPACENAM DB2 command to stop the
Tablespace(s) to be restored.
2. Then, restore the Tablespace from the ABR backup system, by providing
it with the MVS dataset name. ABR will automatically locate the
most recent copy of this dataset, and then restore it. Note that
if a Tablespace spans more than one DB2 linear dataset, all of the
datasets involved must be restored to a consistent point-in-time.
3. Then, issue the DB2 START DATABASE command, with the ACCESS (UT)
and RECOVER TABLESPACE (LOGONLY) operand.
4. Finally, issue the DB2 START DATABASE command with the ACCESS
(RW) operand to restart the tablespace for full read-write access.
the above method, recovering DB2 Tablespaces with FDR/ABR
is significantly faster than Image copy. The restores are
quick and easy and they can be done without specialist DB2/DBA