Board index » off-topic » Ye olde BDE 4GB issue

Ye olde BDE 4GB issue


2006-12-13 11:13:45 AM
off-topic0
We're working on getting rid of the BDE here, but my users are running into
this problem enough that I've had to try to deal with it in my current
BDE-based software. I've been testing out the patch from:
codecentral.borland.com/codecentral/ccWeb.exe/listing and
have learned a few things. I thought I'd share as some of this may be
useful to others.
I was confused after reading various posts about whether or not this patch
works on Win9x machines. In my testing, it caused an immediate failure on
Win98 SE. The only solution I found was to check the user's OS and exit the
patch if they didn't have an NT-based system. This leaves me with a patch
that works for WinNT machines, but does nothing for Win9x machines. I swear
I saw a post from the author of the patch somewhere saying he had fixed it
for Win9x, but I couldn't find a version that worked on Win9x.
I have also seen confusion about whether the problem happens with 4GB or
2GB. In my testing it seems that GetDiskFreeSpace on Win98 (so I assume
Win9x in general) can see a max of 2GB of free space. On WinXP/2003 it
seems to see a max of 4GB. So I'd assume that NT-based machines will see
the problem in 4GB increments, while 9x machines would see the problem at
2GB intervals.
As reported, my testing has found that GetDiskFreeSpace won't return the
free space when given a UNC path on Win98 (and I assume all Win9x) machines.
There is an upside to this though. When GetDiskFreeSpace isn't able to
return a value, then the BDE seems to assume it has enough free space. So,
if you're using UNC paths to your data with the BDE on Win9x, then you can
avoid the 2GB/4GB issue. (Although the BDE also won't know when you truly
to run out of free space).
I'm implementing this patch (modified to check that the OS so it only runs
on WinNT-based machines). I'm lucky that basically all of my users have my
Paradox tables on a file server rather than their desktop. My code already
internally converts any mapped-drive path to a UNC path before using it, so
I won't see the 2/4GB issue on Win9x machines. And the modified patch will
take care of the WinNT-based users.
 
 

Re:Ye olde BDE 4GB issue

The generally recommended 2GB/4GB patch is at www.TheDBCommunity.com;
"Downloads", "Fix for Incorrect BDE...". It should work well on all OSes,
and is also not Delphi-specific...
- If you have reviewed it - and rejected it - you might let us know.
- If you have not checked it out, and get a chance to do so, you might
also let us know.
Mike
 

Re:Ye olde BDE 4GB issue

Mike.. the patch you pointed to is "generally recommended" in the Paradox
universe.. the one that CP linked to is the one that is usually used by the
Delphi ommunity..
CP.. yes, I think the "not for Win98" comment is correct..
Diamond Software Group
www.diamondsg.com/main.htm
Paradox Support & Sales
Diamond Sports Gems
www.diamondsg.com/gemsmain.htm
Sports Memorabilia and Trading Cards
"Michael Kennedy" < XXXX@XXXXX.COM >wrote in message
Quote
The generally recommended 2GB/4GB patch is at www.TheDBCommunity.com;
"Downloads", "Fix for Incorrect BDE...". It should work well on all OSes,
and is also not Delphi-specific...
- If you have reviewed it - and rejected it - you might let us know.
- If you have not checked it out, and get a chance to do so, you might
also let us know.

Mike


 

{smallsort}

Re:Ye olde BDE 4GB issue

I did a lot of googling for BDE patches/fixes, and didn't come up with the
Paradox Community one.
I will do a bit of testing of this Paradox Group patch. It seems possible
that some people (that have both Paradox and Delphi apps), may end up with
both patches. I wonder if they'll play well together? The Paradox
Community patch is trickier in terms of deployment, but the fact that it
will work on Win9x is appealing.
I'm far from an expert in virus behaviour, but I'm a bit suprised that the
"Delphi fix" hasn't caused issues with XP/2003 or my antivirus software
(tested with both Trend Micro Office Scan and Norton AV). It would seem to
me that allowing apps to hijack (at runtime) calls made by other .dll's to
the Win32 API would be dangerous. Hopefully I'll get time to try these
patches out on Vista. I wouldn't be suprised to see only the Paradox Group
one works on Vista, since it doesn't do any runtime hijacking (at least from
the author's description). Although, maybe there's nothing wrong with
allowing apps to hijack .dll calls to the Win32 API.
 

Re:Ye olde BDE 4GB issue

I have found a bug in the Code Central patch (see
threads.borland.com/threads/threads.exe/thread -
posted as anonymous user "Craven Weasel"). Basically it means this patch
will over report free space on some OS'es (Win2k3 is the one I've tested so
far). There's an easy fix (I think), but I want to test it more on other
OS'es to be sure my fix works.
 

Re:Ye olde BDE 4GB issue

Quote
Mike.. the patch you pointed to is "generally recommended" in the Paradox
universe.. the one that CP linked to is the one that is usually used by
the Delphi ommunity..
Steve:
Rick's patch is "generic" - it extends one of the main DLLs inside the BDE,
so it should be equally applicable to ALL platforms, OSes, Paradox/Delphi,
etc, etc. Basically, the patch is embedded inside the BDE itself, so both
the user and the app-developer are not involved in any way.
- Mike
 

Re:Ye olde BDE 4GB issue

I'm thinking Rick's patch is preferable too. I've found some problems with
the code in the code central patch (which can be fixed), but it still won't
work on Win9x. Rick's patch also only impacts the BDE. The Code Central
patch will impact anything that calls GetDiskFreeSpace (while any app that
uses the patch is running). I don't like the idea of changing the behaviour
of the Win32 API for all apps. Also, I'm not sure that AV software or newer
OS'es with Vista will be ok with the API hijacking. (I haven't had a chance
to test it on Vista yet).
With Rick's patch though, you are changing idapi32.dll. Is that ok with
Borland/Corel's license agreements? I suppose given that they're not
providing an official patch, I might not worry about this too much. It
would be nice to see the source code for the patch app and the patch .dll
though. I'm always nervous about running/installing .exe's /.dll's from an
unfamiliar source. I guess Rick may be well known in the Paradox community,
but I'm not familiar with him. Also, given the problems I've found in the
Code Central patch, I'd feel better if I knew for sure which
GetDiskFreeSpaceEX function Rick was using, and how he was using it.
 

Re:Ye olde BDE 4GB issue

Quote
Is that ok with Borland/Corel's license agreements? I suppose given that
they're not providing an official patch, I might not worry about this too
much.
that's the attitude everyone else is taking.. we're doing it for
self-preservation, because Corel and Borland won't..
Quote
It would be nice to see the source code for the patch app and the patch
.dll
you'd need to contact Rick directly..
Diamond Software Group
www.diamondsg.com/main.htm
Paradox Support & Sales
Diamond Sports Gems
www.diamondsg.com/gemsmain.htm
Sports Memorabilia and Trading Cards
"C P" < XXXX@XXXXX.COM >wrote in message
Quote
I'm thinking Rick's patch is preferable too. I've found some problems
with the code in the code central patch (which can be fixed), but it still
won't work on Win9x. Rick's patch also only impacts the BDE. The Code
Central patch will impact anything that calls GetDiskFreeSpace (while any
app that uses the patch is running). I don't like the idea of changing
the behaviour of the Win32 API for all apps. Also, I'm not sure that AV
software or newer OS'es with Vista will be ok with the API hijacking. (I
haven't had a chance to test it on Vista yet).

With Rick's patch though, you are changing idapi32.dll. Is that ok with
Borland/Corel's license agreements? I suppose given that they're not
providing an official patch, I might not worry about this too much. It
would be nice to see the source code for the patch app and the patch .dll
though. I'm always nervous about running/installing .exe's /.dll's from
an unfamiliar source. I guess Rick may be well known in the Paradox
community, but I'm not familiar with him. Also, given the problems I've
found in the Code Central patch, I'd feel better if I knew for sure which
GetDiskFreeSpaceEX function Rick was using, and how he was using it.

 

Re:Ye olde BDE 4GB issue

Here is the description Rick Kelly provided on what he did if that helps.
-----------------------------------------------------------------------
Rick Kelly's original e-mail describing the fix from Original post in
"pnews.paradox-dos" with the subject "Not enough disk space"
"Rick Kelly" < XXXX@XXXXX.COM >wrote in message
OK guys...here the scoop.
1. I wrote a DLL that trapped the GetDiskFreeSpace api call and redirected
all the other kernel32 calls to, what else, kernel32.dll.
2. In the revised GetDiskFreeSpace, I used GetDiskFreeSpaceEx, checked if
free exceeded 2GB, and if so, I returned free space as close to 2GB as I
could get and being sure that any subsequent calculations would not exceed
the capacity of one dword (LongInt) or 2,147,483,647 bytes. I wasn't sure if
idapi32.dll was doing its calculations using unsigned integer math or not so
I erred on the safe side. I'll check it out for sure later on. It's real
easy to change it to 4GB if I find out that will work.
3. I made a backup copy of idapi32.dll and then, using a hex editor, changed
the two string occurrences of kernel32.dll inside of idapi32.dll to
bdeker32.dll (which is the name of my DLL).
4. Moved bdeker32.dll to windows system folder
5. Fired up PDX 8,9,10,11....and everything still works!
-------------------------------------------------------------------
"C P" < XXXX@XXXXX.COM >wrote in message
Quote
I'm thinking Rick's patch is preferable too. I've found some problems
with the code in the code central patch (which can be fixed), but it still
won't work on Win9x. Rick's patch also only impacts the BDE. The Code
Central patch will impact anything that calls GetDiskFreeSpace (while any
app that uses the patch is running). I don't like the idea of changing
the behaviour of the Win32 API for all apps. Also, I'm not sure that AV
software or newer OS'es with Vista will be ok with the API hijacking. (I
haven't had a chance to test it on Vista yet).

With Rick's patch though, you are changing idapi32.dll. Is that ok with
Borland/Corel's license agreements? I suppose given that they're not
providing an official patch, I might not worry about this too much. It
would be nice to see the source code for the patch app and the patch .dll
though. I'm always nervous about running/installing .exe's /.dll's from
an unfamiliar source. I guess Rick may be well known in the Paradox
community, but I'm not familiar with him. Also, given the problems I've
found in the Code Central patch, I'd feel better if I knew for sure which
GetDiskFreeSpaceEX function Rick was using, and how he was using it.

 

Re:Ye olde BDE 4GB issue

Thanks, I actually saw that. But in investigating the Code Central 'fix' I
found that Delphi users have 2 versions of GetDiskFreeSpaceEx available.
(One in Windows.pas, and one in SysUtils.pas). Both versions of
GetDiskFreeSpaceEx return 2 possible values for free space: the total free
space on the path, or the free space available to the caller. Depending on
which GetDiskFreeSpaceEx version, which OS you're using, and whether you're
checking total free space vs. free space to for the caller, you'll get
different results. I found the only way to get consistently correct results
was to use the SysUtils.pas version of GetDiskFreeSpaceEx.
The Windows.pas version of GetDiskFreeSpaceEx is the 'raw' API version from
Kernel32. The SysUtils.pas version is a Borland version to deal with the
fact that some OS'es don't actually have GetDiskFreeSpaceEx in their API.
The Windows.pas version was only correct for Win2000/XP/2003 (but not NT,
Win98).
I don't have disk quotas turned on, on my SBS 2003 server. With the
Windows.pas version of GetDiskFreeSpaceEx, it always said there was about
4000 petabytes of free space available to the caller, regardless of OS. One
Win2000/XP/2003, it gave the correct amount of 'total free space'. (One
WinNT/98 it always rounded 'total free space' _down_ to the nearest 4GB, so
if there was 7 GB free, it'd say there was 4. If there was 3 GB free, it'd
say there was 0). I'm guessing that if I did have disk quotas enabled, the
free space available to the caller might actually show the correct value on
Win2000/XP/2003.
For the SysUtils.pas version, I got the same, correct values for all OS'es,
for both free space available to caller, and total free space. If I had
disk quota's enabled, I'd hope to maybe see differing values for free space
available to caller, but I haven't tested this.
BTW, my testing is done with a Delphi 5 app.
I'd like to know exactly what Rick Kelly's code looks like - especially if
he used Delphi. The Code Central patch was always checking the space
available to the caller using the Windows.pas version of GetDiskFreeSpaceEx.
So it thinks there's always 4000 petabytes free, which the patch will then
always report as 4 GB free. My suspicion is that Rick Kelly's patch is
probably directly using the Win32 API version of GetDiskFreeSpaceEx. And,
as a result may not get the correct results for free space on 98 or NT. If
he's only checking the free space available to the caller, it may also be
wrong on 2000/XP/2003.
I'm posting this in part because this stuff is a bit new to me (I don't
usually deal with the Win API). It's possible that I may have done
something wrong, but I'm pretty sure my testing of the 2 GetDiskFreeSpaceEx
versions is solid. There's type conversion going on between standard Win32
API (C) types and Delphi types. Maybe the problem is actually in Delphi 5,
and not the GetDiskFreeSpaceEx Win32 API? So, if Rick used some other
language, then maybe he wouldn't see what I saw.
 

Re:Ye olde BDE 4GB issue

Rick has indicated that he tested his patch extensively, on W98/W2K/XP,
stand-alone & multi-user, various BDE builds, etc. Others also ran many
tests when the patch was released initially, and, AFAIK, nobody has reported
any issues with it - in TEST and LIVE environments.
As I stated already, I think you should assume it's "good", run your tests
on it on that basis.
- Mike
 

Re:Ye olde BDE 4GB issue

I found Rick's patch did work ok on Win98 SE, Win2K (pro, sp4), and WinXP
(pro, sp2). When used on WinNT (workstation, sp6a), my application wouldn't
start. It would show the splash screen, then just quit with no error
messages. I do check some Paradox tables while the splash screen is showing
so I am using the BDE at this point, and NT obviously doesn't like the
patch. With NT, it didn't matter how much free space was on the share on my
Win2003 server, it would just never work.
In all cases, my Paradox tables were located on the same share on my Windows
SBS 2003 (Standard) server. All machines accessed this share via a UNC
path, rather than a mapped drive. I didn't try this patch on the actual
Win2003 server. I tried the test with 3.95 GB free on the share on the
server, and also with only 6 MB free on the share on the server, and my
application was attempting to rebuild a 73 MB table. With the patch in
place the BDE knew when there was only 6 MB free on the server, and that
this wasn't enough space.
So I guess in summary my tests show...
Code Central Patch:
- will crash Win9x machines
- will always report 4 GB is free (even if much less is free) on
WinNT-based machines
Rick's Patch:
- will crash on WinNT
- correctly* detects free space on Win98, Win2000, WinXP
*According to Rick's documentation his patch will report a maximum of 2GB
free, even when more is free.
 

Re:Ye olde BDE 4GB issue

Many thanks for the excellent feedback.
Quote
>It would show the splash screen, then just quit with no error messages.
>I do check some Paradox tables while the splash screen is showing so I am
>using the BDE at this point, and NT obviously doesn't like the patch.
>With NT, it didn't matter how much free space was on the share on my
>Win2003 server, it would just never work.
Any possibility of some "permissions" issues? Did you check what happens if
you don't do the initial table checking. Could it be some "timing" issue,
whereby the BDE and/or Rick's code is not "ready" early enough... Could you
insert some "trace" facility in the app, to see if your initial table-checks
actually activated, and/or how far it got, before it blew.
Quote
I didn't try this patch on the actual Win2003 server.
I think there's no point in having the BDE, nor the Patch, installed on the
server...
Quote
Rick's Patch:
- will crash on WinNT
- correctly* detects free space on Win98, Win2000, WinXP

*According to Rick's documentation his patch will report a maximum of 2GB
free, even when more is free.
The 2GB limit is intentional - it's safer, in case any "Free-Space" calcs,
anywhere, are using limited "signed arithmetic". In Lesspace, I actually
reduced the limit to either 830MB, or 530MB (approx), to avoid similar
overflow issues...
- Mike
 

Re:Ye olde BDE 4GB issue

I found that my app crashes on NT when I start interacting with the BDE.
I'm not sure exactly what things cause the crash. I find that Delphi will
sometimes run a few lines of code after an actual line of code that causes a
crash. Here's basically what my initialization code does:
1) Set Session.NetFileDir
2) Set Session.PrivateDir
3) Call Session.ModifyDriver to modify the Paradox driver to a 16384 block
size and level 7 tables
4) Call Session.ModifyDriver to modify the dBase driver to level 5 tables
5) Creates a DataModule that in turn calls Session.AddPassword, and also
creates 2 TDatabase components
6) Creates a TTable component for a particular Paradox table
7) Checks if the table exists (and it does); then tries to open it
The application will crash at step 3. If I comment out step 3, it crashes
at step 4. If I comment out step 1 and 2 (but let 3 and 4 run) then the
code doesn't crash until step 7. I suspect this is a situation where
something has gone wrong with the BDE right away but it takes a few BDE
calls before things actually crash.
The code above runs before the application's main form is created. However,
the unit containing this code does include the dbiprocs, DB, and DBTables
units in its 'uses' clause (interface section) so I would assume that the
BDE should already be properly initialized before the code in my unit begins
executing.
I am running my tests as my network's administrator (not just the computer
admin, but the admin for the full network), so I don't think this would be a
permissions issue.
PS: I know that I recommend my users run the database restructuring and
database export (to MS-Access format) included in my app on their server so
they will see better performance. This means for me at least, users will be
using the BDE on Windows 2003 servers, although not on a daily basis.
PPS: I found some big flaws in my testing from yesterday. First of all, I
had compiled my test app with the Code Central patch, so my testing from
yesterday included both the Code Central patch, and Ricks patch. The
testing above does not include the Code Central patch, so the apparent
problem in Rick's patch is not related to yesterday's testing problem. I
will be redoing my tests on a version without the Code Central patch. I
also realized that I should be doing a test where just over 4 GB is free
(instead of just under). That way I can ensure Rick's patch does in fact
deal with the 2/4 GB issue, as well as detect when free space is actually
low. (My testing last night really only served to check if Rick's patch
could accurately detect when there really was little free space - and except
for NT, it did seem to do this). I'll post my results when they're done.
 

Re:Ye olde BDE 4GB issue

Thank you.
Quote
I found that my app crashes on NT when I start interacting with the BDE.
I'm not sure exactly what things cause the crash. I find that Delphi will
sometimes run a few lines of code after an actual line of code that causes
a crash. Here's basically what my initialization code does:

...
Though I know what all these steps are, I don't have enough experience in
them to contribute anything useful. I hope others do, and will <g>.
Quote
PPS: I found some big flaws in my testing from yesterday. First of all,
I had compiled my test app with the Code Central patch, so my testing from
yesterday included both the Code Central patch, and Ricks patch. The
testing above does not include the Code Central patch, so the apparent
problem in Rick's patch is not related to yesterday's testing problem. I
will be redoing my tests on a version without the Code Central patch. I
also realized that I should be doing a test where just over 4 GB is free
(instead of just under). That way I can ensure Rick's patch does in fact
deal with the 2/4 GB issue, as well as detect when free space is actually
low. (My testing last night really only served to check if Rick's patch
could accurately detect when there really was little free space - and
except for NT, it did seem to do this). I'll post my results when they're
done.
Without going back and re-reading the notes I have, I think if BOTH patches
are active, then the CC one will kill Rick's one (where the CC intercepts
the API calls). For all others (if there are any), Rick's will activate.
You're correct about checking with just OVER 2GB/4GB free. And it's the HEX
values (if you're into that stuff), where the Most-Significant-Bit is
discarded (on signed calcs), or all bits above the lowest 32 bits are
discarded (overflow). If you're working in plain DECIMAL values, then the
critical thresholds are:
7F FF FF FF = 2,147,483,647
FF FF FF FF = 4,294,967,295
I hope we can identify WHY Rick's patch was broken under NT - both of the
sake of other users, and in case there's some issue which "might" apply to
others OSes also.
- Mike