Board index » delphi » TechTips: Database corruption - signs and symptoms

TechTips: Database corruption - signs and symptoms

A "table" in the Paradox system consists of a family of shared files,
all with the same name but different extensions, such as "DB," "MB",
"PX," "XG3" and so on.  Users who are working with these files have one
or more of these files open, are reading them directly, and may be
modifying them as well.  All of this activity is under the control of
the Borland Database Engine (BDE), which is the underlayer of software
that regulates access to this kind of database.  The file-sharing
itself, and the communication of data between your computer and the
server(s), is handled by the network operating system:  Windows, Novell,
Lantastic, and so forth.

BDE is specifically designed for this purpose and normally it does its
job quite well, as does the operating-system.  But any time you have a
group of several computers, and wires (which can inadvertantly run
underneath the wheels of desk-chairs, the edges of filing cabinets and
so on) all working together ... things CAN go wrong.  When they do,
Paradox is usually able to detect that something is the matter.

What usually happens is that someone resets their computer (because it
"locked up" ... even after just a couple of seconds!) or they are
running a virus-scanner that is set too aggressively.  [If Norton wants
to examine every file "when it is opened," and one of those files
happens to be parts of your 50,000-record database ... "!!!"  And if
several workstations are doing it at the same time, the net result is
almost worse than a virus itself.]  Now there is information that did
not get written to disk -- the files are out-of-date, and Paradox
probably knows it.

Early versions of Windows and NT contained bugs in their file-cacheing
software (which tries to reduce the amount of data traffic on a network
by hoarding local copies of data in memory, much as your Internet
browser does), but in our experience =most= of those buggy versions have
now been replaced and this is no longer an issue.  (If your company is
still using 5-year-old copies of Windows, maybe it's time to update them
a little, even if you don't go for Windows 2001.)

A corrupted file normally manifests itself as an "out of date" file that
Paradox won't open any more.  But it can ALSO be damaged, far more
severly, by a tool that is oddly-enough designed to recover from damaged
file-systems:  Microsoft ScanDisk.

Actually, that statement's not quite fair:  the file isn't damaged by
ScanDisk.  It's damaged by whatever caused your computer to crash.  And
ScanDisk does, after a fashion, "fix" the problem by restoring your disk
file-system to functional condition.  However, if one of the files that
got damaged was your "50,000 record database," well, ScanDisk does not
want to be the bearer of bad news.  So it slices the database file into
possibly several pieces, all of them now intact.  It names one of them
under the original name (BIG.DB), and any other fragments as
"C:\FILEnnnn.CHK."  Then it purrs that it "found problems on this drive
and fixed them all."  Literally true.  Not false.  But misleading.

Anyhow... what can you do about it?  

Paradox provides a simple "table repair utility" (TUTILITY/TUTIL32), and
it does a respectable but very slow job of repairing most of the things
that go wrong with a table.  It does this by trying to create a new
table with the structure of the old one, then to scan the original file
from stem to stern, copying records one-by-one into it.  In the process
it also drops passwords, may reset autoincrement field values, and may
lose some or all of your memo (MB) field values.  But, it does work and
it's free.  Lots of facilities exist to put this functionality into your
Paradox or Delphi applications.

More advanced products like our ChimneySweep? take a more detailed
approach, reconstructing table-problems in place and verifying indexes
using algorithms that take a fraction of the time.  ChimneySweep is also
fully programmable and script-driven.  But it still requires, and this
is important to remember, that "the file that got damaged is still
recognizably there."

A file *can* become destroyed.  It can get sliced into pieces or it can
have the misfortune of landing on a "bad sector."  {A momentary power
surge or power fluctuation can alter the rotational speed of a hard
drive, causing a permanent "digital smear" to occur which is from now on
a "bad sector.")  Bearings can go out without warning in several
different places on a drive.  Controllers, loose internal or external
cables, and even hair-dryers [in the other room!] can cause grief.

Therefore, you MUST have a current, usable, and tested backup plan,
where accurate copies of all your data are copied to tape or other media
every day (or several times a day) and placed in a secure location, such
as a lock-box in a nearby bank.  Your personnel must know how the backup
is run, how to verify that it did run, and how to restore files
correctly from it.  This is in addition to any and all table "repair"
mechanisms and procedures that you have in place.

A very common application of the ChimneySweep product is to rapidly
verify that a database is not damaged, just before it is backed up or
just before the day's work begins.  This is a very reasonable thing to
do.  BUT ANYHOW, "the key is vigilance."  

Never rely upon a tool -- anyone's tool -- to save your hindquarters or
your company's irreplaceable data.  {Let the record show that the two
are often linked!}  ;-) ;-)  Rely instead upon vigilance, tested and
known procedures ... forged into routine, repeatable, procedures that
any employee(s) can be trained to do:  daily.

------------------------------------------------------------------
Sundial Services :: Scottsdale, AZ (USA) :: (480) 946-8259
mailto:i...@sundialservices.com  (PGP public key available.)

Quote
> Fast(!), automatic table-repair with two clicks of the mouse!
> ChimneySweep(R):  "Click click, it's fixed!" {tm}
> http://www.sundialservices.com/products/chimneysweep

 

Re:TechTips: Database corruption - signs and symptoms


A conservative is a liberal who has recently been mugged - A person with a
vigilant backup strategy is one who recently had data loss.

...Jim

Quote
Sundial Services wrote:
> A "table" in the Paradox system consists of a family of shared files,
> all with the same name but different extensions, such as "DB," "MB",
> "PX," "XG3" and so on.  Users who are working with these files have one
> or more of these files open, are reading them directly, and may be
> modifying them as well.  All of this activity is under the control of
> the Borland Database Engine (BDE), which is the underlayer of software
> that regulates access to this kind of database.  The file-sharing
> itself, and the communication of data between your computer and the
> server(s), is handled by the network operating system:  Windows, Novell,
> Lantastic, and so forth.

> BDE is specifically designed for this purpose and normally it does its
> job quite well, as does the operating-system.  But any time you have a
> group of several computers, and wires (which can inadvertantly run
> underneath the wheels of desk-chairs, the edges of filing cabinets and
> so on) all working together ... things CAN go wrong.  When they do,
> Paradox is usually able to detect that something is the matter.

> What usually happens is that someone resets their computer (because it
> "locked up" ... even after just a couple of seconds!) or they are
> running a virus-scanner that is set too aggressively.  [If Norton wants
> to examine every file "when it is opened," and one of those files
> happens to be parts of your 50,000-record database ... "!!!"  And if
> several workstations are doing it at the same time, the net result is
> almost worse than a virus itself.]  Now there is information that did
> not get written to disk -- the files are out-of-date, and Paradox
> probably knows it.

> Early versions of Windows and NT contained bugs in their file-cacheing
> software (which tries to reduce the amount of data traffic on a network
> by hoarding local copies of data in memory, much as your Internet
> browser does), but in our experience =most= of those buggy versions have
> now been replaced and this is no longer an issue.  (If your company is
> still using 5-year-old copies of Windows, maybe it's time to update them
> a little, even if you don't go for Windows 2001.)

> A corrupted file normally manifests itself as an "out of date" file that
> Paradox won't open any more.  But it can ALSO be damaged, far more
> severly, by a tool that is oddly-enough designed to recover from damaged
> file-systems:  Microsoft ScanDisk.

> Actually, that statement's not quite fair:  the file isn't damaged by
> ScanDisk.  It's damaged by whatever caused your computer to crash.  And
> ScanDisk does, after a fashion, "fix" the problem by restoring your disk
> file-system to functional condition.  However, if one of the files that
> got damaged was your "50,000 record database," well, ScanDisk does not
> want to be the bearer of bad news.  So it slices the database file into
> possibly several pieces, all of them now intact.  It names one of them
> under the original name (BIG.DB), and any other fragments as
> "C:\FILEnnnn.CHK."  Then it purrs that it "found problems on this drive
> and fixed them all."  Literally true.  Not false.  But misleading.

> Anyhow... what can you do about it?

> Paradox provides a simple "table repair utility" (TUTILITY/TUTIL32), and
> it does a respectable but very slow job of repairing most of the things
> that go wrong with a table.  It does this by trying to create a new
> table with the structure of the old one, then to scan the original file
> from stem to stern, copying records one-by-one into it.  In the process
> it also drops passwords, may reset autoincrement field values, and may
> lose some or all of your memo (MB) field values.  But, it does work and
> it's free.  Lots of facilities exist to put this functionality into your
> Paradox or Delphi applications.

> More advanced products like our ChimneySweep? take a more detailed
> approach, reconstructing table-problems in place and verifying indexes
> using algorithms that take a fraction of the time.  ChimneySweep is also
> fully programmable and script-driven.  But it still requires, and this
> is important to remember, that "the file that got damaged is still
> recognizably there."

> A file *can* become destroyed.  It can get sliced into pieces or it can
> have the misfortune of landing on a "bad sector."  {A momentary power
> surge or power fluctuation can alter the rotational speed of a hard
> drive, causing a permanent "digital smear" to occur which is from now on
> a "bad sector.")  Bearings can go out without warning in several
> different places on a drive.  Controllers, loose internal or external
> cables, and even hair-dryers [in the other room!] can cause grief.

> Therefore, you MUST have a current, usable, and tested backup plan,
> where accurate copies of all your data are copied to tape or other media
> every day (or several times a day) and placed in a secure location, such
> as a lock-box in a nearby bank.  Your personnel must know how the backup
> is run, how to verify that it did run, and how to restore files
> correctly from it.  This is in addition to any and all table "repair"
> mechanisms and procedures that you have in place.

> A very common application of the ChimneySweep product is to rapidly
> verify that a database is not damaged, just before it is backed up or
> just before the day's work begins.  This is a very reasonable thing to
> do.  BUT ANYHOW, "the key is vigilance."

> Never rely upon a tool -- anyone's tool -- to save your hindquarters or
> your company's irreplaceable data.  {Let the record show that the two
> are often linked!}  ;-) ;-)  Rely instead upon vigilance, tested and
> known procedures ... forged into routine, repeatable, procedures that
> any employee(s) can be trained to do:  daily.

> ------------------------------------------------------------------
> Sundial Services :: Scottsdale, AZ (USA) :: (480) 946-8259
> mailto:i...@sundialservices.com  (PGP public key available.)
> > Fast(!), automatic table-repair with two clicks of the mouse!
> > ChimneySweep(R):  "Click click, it's fixed!" {tm}
> > http://www.sundialservices.com/products/chimneysweep

Other Threads