Board index » delphi » Memory management limitations Win x Linux

2007-06-07 03:30:33 AM
delphi47
Hi,
A friend of mine wrote a C++ program, and he has a problem.
This program depends on an algorithm he wrote that will allocate a lot of memory in
order to do some heavy calculations and produce some numbers.
He is using a linux and a native c++ compiler.
The problem lies here:
const int N1=3;
const int Ip=6500;
const int I=6500;
const int min_branch_size=1;
int main(void)
{
const int max_branch_size=max(min_branch_size,I);
double **sample, **sample_tk;
sample=new double* [Ip*max_branch_size]; <---- Here
sample[0]=new double[(Ip*max_branch_size)*(N1)]; <---- Here
....
}
The fact is he need "a little more info" for his algorithm to be more effective, and
when he sets Ip=12000 and I=12000 (would be close to ideal), there's a bad_alloc
error. From what I can see, there's a 12000 * 12000 * 8 = 1.152.000.000 bytes and
12000 * 12000 * 8 * 3 = 3.456.000.000. What's 4.5Gb of ram of data for a 500 lines
C++ program? :-)
He asked me to convert the program to delphi and make some tests, but with the
figures above, I really couldn't give him any hope. And as expected, I had a similar
problem.
I downloaded turbo c++ explorer, and tried to compiled it. I ran the program but the
error is still there. With some nice hints by the way!! (Windows XP 2.5 Gb Ram)
As the memory grows, shouldn't some virtual memory kick in under windows? and under
linux?
Anyway I wouldn't expect him to have such a problem (memory limitation) under linux.
I told him to change the "double" (64-bit) for "float" (32-bit) to see if that
affects the calculations. He is checking this write now. Ok this would cut in half
the memory consumption.
Is there any setting in the environment (or command line option, switch, flag) he can
make in order to compile and run such a memory consuming beast? Of course he wouldn't
like to make any changes in the code. But if changes should be made, would it be
possible to still use memory (instead of writing to disk as there are lots of
calculations, there will be an extra problem with the time the algorithm will take to
give an answer.)
TIA,
Clément
 

Clément Doss writes:
HTH
Jon
 

Jonathan Benedicto writes:
www.nexusdb.com/new/index.php
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at www.tamaracka.com/search.htm
 

=?ISO-8859-1?Q?Cl=E9ment_Doss?= <XXXX@XXXXX.COM>writes:
should make it work.
address space is required for such code to run. Virtual memory
only helps against lack of physical memory.
Daniël Mantione
 

"Hannes Danzl[NDD]" <XXXX@XXXXX.COM>writes:
Not adressable memory.
Daniël
 

Danikl Mantione writes:
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at www.tamaracka.com/search.htm
 

Oliver Townshend
 

"Clément Doss" <XXXX@XXXXX.COM>writes
I write programs in C and Perl which run on 64bit Suse Linux. Allocating
even 20GB of memory is not a problem and programs work beautifully (this is
bio computing stuff). All you need in 64bit compliant OS, compilers and
hardware.
P.
 

Eric
 

Another solution might be changing the algorithm so it can be divided into
several smaller steps. In each step your need for memory will be much lower
so it can be guaranteed that you won't run out of physical ram. Even if you
switch to a 64-bit system and the amount of Physical Ram is less or equal to
2 GB, the system will use virtual memory which is much slower than physical
ram. Morover portability of your code will be very low and you will have to
find a 64-bit CPU + 64-bit OS to run your program.
I don't know what algorithm yr friend uses but there is always a way to
divide a big job into several smaller tasks.
"Clément Doss"

Eric Grange writes:
No, more like this:
msdn2.microsoft.com/en-us/library/aa366527.aspx
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at www.tamaracka.com/search.htm
 

overall, it is still like EMS in that the code has to "manually" map and
manage the pages.
But at this point in time, with 64bit OSes being available and solid (at
least XP64 and Win2k3 are, been using both for quite a bit of time now),
the most interesting purpose of AWE is in a 64bit memory manager to
allocate physical memory, and thus get the benefits of "free"
reallocation of large blocks (copy-less, no need to mess with MMFs
either, and the virtually unlimited address space eliminates the issues
that approach had in 32bits).
Developper time is expensive, RAM isn't, better look twice IMO before
spending extra time (development, debugging & maintainance) to support
legacy 32bits mode, especially in a specialized application like seems
to be the OP's case.
Eric
 

I had under XP.
But event with 4Gb on my Vista 64, i can not allocate huge block of
memory due to memorry fragmentation..Windows know how to allocate huge
block in a fragmented memory even if there's enought memory left.
 

a bit more available address space, but that is just a small respite.
If you want the issue gone, you have to use 64bit code.
Eric
 

Thanks to all that replied. You all helped a lot!
For now he doesn't have much time to starting changing things, so one of the software
requirements is to run on 64-bit environment.
And I will try to help him check out his algorithm, or at least point some things
that might help.
Anyway, thanks again for your help!
Cheers,
Clément