Board index » cppbuilder » Memory allocation problems ...

Memory allocation problems ...


2007-03-19 08:36:53 PM
cppbuilder37
I've two problems about memory allocation:
Premise:
a. i've 2GigaBytes Memory RAM
b. i'm testing my sw on extreme condition
Problems:
1. if i try to allocate by "new" function 600MByte (six hundred mega
byte) the system fail and show me the following message
" ... raised exception class std::bad_alloc with message 'Exception
ObjectAddress: 0x9348AE' ..."
the instructions i use are:
char *test_mem;
test_mem = new char[600000000];
i tried to use GlobalAlloc (with different flag: GMEM_FIXED and
GMEM_MOVEABLE) too but the result is nearly the same; in this case no
exception raised but the pointer remain to NULL
/* TEST with GMEM_FIXED flag*/
char* test_mem = NULL;
test_mem = (char *) GlobalAlloc(GMEM_FIXED,600000000);
GlobalFree(test_mem);
/* TEST with GMEM_MOVEABLE flag */
char* test_mem = NULL;
test_mem = (char *) GlobalLock(GlobalAlloc(GMEM_MOVEABLE,600000000));
GlobalFree(test_mem);
2. When i use TOpenPictureDialog (or TOpenDialog )component and i want
to open some image file i see two different behaviour:
a) if i open 120 (one hundred and twenty) files each of which has
15MebaByte size, it's all OK
b) if i open 5 (five) files each of which has 250MebaBytes size, an
exception raise!
the test code i use is following:
/* start - testing code */
TPicture **Picture1;
if(OpenPictureDialog1->Execute())
{
if(OpenPictureDialog1->Files->Count>0)
{
Picture1 = new TPicture*[OpenPictureDialog1->Files->Count];
for(int i=0; i<OpenPictureDialog1->Files->Count; i++)
{
//
Picture1[i] = new TPicture;
Picture1[i]->Bitmap->HandleType = bmDIB;
Picture1[i]->Bitmap->LoadFromFile(OpenPictureDialog1->Files->Strings[i]);
}
}
}
/* end - testing code */
Questions:
1. can I allocate more than 600 MBytes and in particular as much as
memory size?
. if i can how can i do?
. if i can't how i know the memory allocation limit?
2. i belive the second problem is caused by TOpenDialog component but i
don't know which parameter i have to modify to avoid this error.
Many thanks in advance!
 
 

Re:Memory allocation problems ...

kronos < XXXX@XXXXX.COM >writes:
Quote
Questions:

1. can I allocate more than 600 MBytes and in particular as much as
memory size?
You should be able to, however, the bigger the memory request, the
harder it is to satisfy because just having a lot of RAM isn't
necessarily sufficient. A big buffer requires *contiguous* memory,
and if you have made many allocations and deallocations, your memory
usage may be fragmented across your address space, breaking your
available memory into many small blocks, with no large ones left
without "pieces" taken.
Quote
. if i can how can i do?
Sometimes if you know in advance what you'll need to do, you can
pre-allocate your memory before your program starts doing other
things.
Quote
. if i can't how i know the memory allocation limit?
There are a lot of issues, because you're using a memory allocator
from Borland, which sits between your application and the operating
system. Plus, sometimes the OS can "lie" and report that an
allocation succeeded when it actually failed, with the OS
optimistically thinking that "by the time you actually use the memory,
I'll have found the space you requested." If you actually try to use
the memory and the OS hasn't found the space, then you get a crash.
This is OS dependent, and I know it happens on linux, but I'm not sure
if Windows does that.
There are OS-specific memory calls you can make to query the OS. I
admit to using linux too much to remember the windows calls off-hand.
Good luck.
--
Chris (TeamB);
 

Re:Memory allocation problems ...

You are not just asking for 250 or 500 million bytes of memory. You are
asking for all the bytes to be in one continuous block. Your tests clearly
show that the problem which surfaced when using calls to compiler functions
remains when you re-code to use pure operating system calls. The problem
comes from within Windows. It does not have a continuous block of the
requested size to give.
Try doing a call to GlobalMemoryStatus to learn about what is currently
available and get at least a clue as to what is happening. If using XP call
GlobalMemoryStatusEx instead.
. Ed
Quote
kronos wrote in message
news: XXXX@XXXXX.COM ...

I've two problems about memory allocation:

Premise:

a. i've 2GigaBytes Memory RAM
b. i'm testing my sw on extreme condition

Problems:
1. if i try to allocate by "new" function 600MByte (six hundred mega byte)
the system fail and show me the following message

" ... raised exception class std::bad_alloc with message 'Exception
ObjectAddress: 0x9348AE' ..."

the instructions i use are:

char *test_mem;
test_mem = new char[600000000];


i tried to use GlobalAlloc (with different flag: GMEM_FIXED and
GMEM_MOVEABLE) too but the result is nearly the same; in this case no
exception raised but the pointer remain to NULL

/* TEST with GMEM_FIXED flag*/
char* test_mem = NULL;
test_mem = (char *) GlobalAlloc(GMEM_FIXED,600000000);

GlobalFree(test_mem);


/* TEST with GMEM_MOVEABLE flag */
char* test_mem = NULL;
test_mem = (char *) GlobalLock(GlobalAlloc(GMEM_MOVEABLE,600000000));

GlobalFree(test_mem);

2. When i use TOpenPictureDialog (or TOpenDialog )component and i want to
open some image file i see two different behaviour:

a) if i open 120 (one hundred and twenty) files each of which has
15MebaByte size, it's all OK

b) if i open 5 (five) files each of which has 250MebaBytes size, an
exception raise!

the test code i use is following:

/* start - testing code */
TPicture **Picture1;

if(OpenPictureDialog1->Execute())
{
if(OpenPictureDialog1->Files->Count>0)
{
Picture1 = new TPicture*[OpenPictureDialog1->Files->Count];

for(int i=0; i<OpenPictureDialog1->Files->Count; i++)
{
//
Picture1[i] = new TPicture;
Picture1[i]->Bitmap->HandleType = bmDIB;

Picture1[i]->Bitmap->LoadFromFile(OpenPictureDialog1->Files->Strings[i]);
}
}
}

/* end - testing code */


Questions:

1. can I allocate more than 600 MBytes and in particular as much as memory
size?
. if i can how can i do?
. if i can't how i know the memory allocation limit?

2. i belive the second problem is caused by TOpenDialog component but i
don't know which parameter i have to modify to avoid this error.


Many thanks in advance!
 

{smallsort}

Re:Memory allocation problems ...

Ed Mulroy ha scritto:
Quote
Try doing a call to GlobalMemoryStatus to learn about what is currently
available and get at least a clue as to what is happening. If using XP call
GlobalMemoryStatusEx instead.

Many thanks Ed,
i immediatly tried your tip and just below there are the results, i have
windows XP but the results by using GlobalMemoryStatusEx did'nt have
sense, so the result below have been obtain by using GlobalMemoryStatus
// -----------------------------------------
1. memory status after at start up program
dwMemoryLoad; // percent of memory in use // 21
dwTotalPhys; // bytes of physical memory // 2.146.545.664
dwAvailPhys; // free physical memory bytes // 1.690.284.032
dwTotalPageFile; // bytes of paging file // 4.135.329.792
dwAvailPageFile; // free bytes of paging file // 3.667.910.656 <----a
dwTotalVirtual; // user bytes of address space // 2.147.352.576
dwAvailVirtual; // free user bytes // 2.110.726.144 <--b
// -----------------------------------------
2. memory status after
char* test_mem = NULL;
test_mem = (char *) GlobalAlloc(GMEM_FIXED,500000000);
dwMemoryLoad; // percent of memory in use // 21
dwTotalPhys; // bytes of physical memory // 2.146.545.664
dwAvailPhys; // free physical memory bytes // 1.690.316.800
dwTotalPageFile; // bytes of paging file // 4.135.329.792
dwAvailPageFile; // free bytes of paging file // 3.167.444.992 <----a
dwTotalVirtual; // user bytes of address space // 2.147.352.576
dwAvailVirtual; // free user bytes // 1.610.723.328 <--b
Final consideration:
From these results i see it's all OK but it isn't so!!
To clarify: to obtain these result i make a simple one form application
with one button and one memo component.
 

Re:Memory allocation problems ...

"kronos" < XXXX@XXXXX.COM >wrote in message
Quote
b) if i open 5 (five) files each of which has 250MebaBytes size,
an exception raise!
You did not say where the exception is occuring in your code, or which
exception type is actually being thrown.
Quote
i belive the second problem is caused by TOpenDialog component
Why do you think that? Is the exception occuring before Execute() has
exited? If not, then the problem in not in TOpenPictureDialog at all.
It is very hard to diagnose this problem without more information.
Gambit
 

Re:Memory allocation problems ...

Ok, you show these results:
Quote
1. memory status after at start up program
...
dwAvailPageFile; // free bytes of paging file // 3.667.910.656 <----a
...
dwAvailVirtual; // free user bytes // 2.110.726.144 <--b
char* test_mem = NULL;
test_mem = (char *) GlobalAlloc(GMEM_FIXED,500000000);
dwAvailPageFile; // free bytes of paging file // 3.167.444.992 <----a
...
dwAvailVirtual; // free user bytes // 1.610.723.328 <--b
You did not specifically show what was done between the call to GlobalAlloc
and the second one. I suspect that at that point TApplication has created
the form and that is consistent with the compiler's memory manager grabbing
a safe default extra memory block with which to work.
You also did not say that the call to GlobalAlloc suceeded or failed but I
assume it failed as you would most likely not be back on here asking about
it if the problem had dissapeared. Note that you also have not indicated
that you called GetLastError so therefore can only guess as to what the
error is.
You asked for fixed memory. That demands physical memory in one congiguous
block. The amount of free physical memory reported is just a total, not a
guaranteed single block.
Also read my next reply to your message. I want you to try some code.
. Ed
Quote
kronos wrote in message
news: XXXX@XXXXX.COM ...

i immediatly tried your tip and just below there are the results, i have
windows XP but the results by using GlobalMemoryStatusEx did'nt have
sense, so the result below have been obtain by using GlobalMemoryStatus

// -----------------------------------------
1. memory status after at start up program

dwMemoryLoad; // percent of memory in use // 21
dwTotalPhys; // bytes of physical memory // 2.146.545.664
dwAvailPhys; // free physical memory bytes // 1.690.284.032
dwTotalPageFile; // bytes of paging file // 4.135.329.792
dwAvailPageFile; // free bytes of paging file // 3.667.910.656 <----a
dwTotalVirtual; // user bytes of address space // 2.147.352.576
dwAvailVirtual; // free user bytes // 2.110.726.144 <--b

// -----------------------------------------
2. memory status after

char* test_mem = NULL;
test_mem = (char *) GlobalAlloc(GMEM_FIXED,500000000);

dwMemoryLoad; // percent of memory in use // 21
dwTotalPhys; // bytes of physical memory // 2.146.545.664
dwAvailPhys; // free physical memory bytes // 1.690.316.800
dwTotalPageFile; // bytes of paging file // 4.135.329.792
dwAvailPageFile; // free bytes of paging file // 3.167.444.992 <----a
dwTotalVirtual; // user bytes of address space // 2.147.352.576
dwAvailVirtual; // free user bytes // 1.610.723.328 <--b


Final consideration:

From these results i see it's all OK but it isn't so!!

To clarify: to obtain these result i make a simple one form application
with one button and one memo component.
 

Re:Memory allocation problems ...

You are running 2Gig in your machine. I have only 512Meg on my laptop. Yet
I get success when I try to allocate 500 Meg (and real meg, not just 500
million as you did).
I want you to try this code and tell me how it worked.
Set up the project as
File|New|Other
Double click the Console Wizard
Uncheck the VCL
Set it for GUI
After the project is created, go under Project|Options and check Use dynamic
runtime libraries
Do a Project|Add to project and add noeh32.lib to the project. You'll have
to drill down in the directories/folders to the compiler's LIB directory to
find it.
Alternatively, skip using the IDE to build it and just run this from the
command line where "testmem.cpp" is the name of the source file.
bcc32 -WR testmem noeh32.lib
--------------------------------------------
#include <windows.h>
#include <cstring>
void WINAPI MBox(const char *msg, const char *title = "")
{
MessageBox(HWND_DESKTOP, msg, title, MB_OK);
}
DWORD WINAPI ShowLastError(const char *title_text)
{
TCHAR *err_msg;
DWORD err_value = GetLastError();
if (err_value != NO_ERROR)
{
FormatMessage(
FORMAT_MESSAGE_ALLOCATE_BUFFER |
FORMAT_MESSAGE_FROM_SYSTEM,
NULL,
err_value,
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
reinterpret_cast<LPTSTR>(&err_msg),
0,
NULL);
const char *title =
(title_text && *title_text) ? title_text : "Error";
MBox(err_msg, title);
LocalFree(err_msg);
SetLastError(0);
}
return err_value;
}
void ShowMemoryStatus()
{
MEMORYSTATUS ms;
static char buffer[256];
// buffer is made static only to remove any
// changing effect on GlobalMemoryStatus calls
GlobalMemoryStatus(&ms);
wsprintf(buffer,
"Memory Load %u%%\r\n"
"Physical Available %u\r\n"
"Page File Total %u\r\n"
"Page File Available %u\r\n"
"Virtual Total %u\r\n"
"Virtual Available %u",
ms.dwMemoryLoad,
ms.dwTotalPhys,
ms.dwTotalPageFile,
ms.dwAvailPageFile,
ms.dwTotalVirtual,
ms.dwAvailVirtual);
MBox(buffer, "Global Memory Status");
}
int WINAPI WinMain(HINSTANCE, HINSTANCE, LPSTR, int)
{
ShowMemoryStatus();
HGLOBAL hg = GlobalAlloc(GMEM_FIXED, 500 * 1024U * 1024U);
if (hg)
{
MBox("Allocation Succeeded");
GlobalFree(hg);
}
else
{
ShowLastError("Allocation Failed");
}
return 0;
}
--------------------------------------------
Note: dynamic linked and linking in noeh32.lib are to minimize the program's
footprint. DO NOT use noeh32.lib if you are using try-catch or compiler
supplied classes.
. Ed
Quote
kronos wrote in message
news: XXXX@XXXXX.COM ...
 

Re:Memory allocation problems ...

Remy Lebeau (TeamB) ha scritto:
Quote
"kronos" < XXXX@XXXXX.COM >wrote in message
news: XXXX@XXXXX.COM ...

>b) if i open 5 (five) files each of which has 250MebaBytes size,
>an exception raise!

You did not say where the exception is occuring in your code, or which
exception type is actually being thrown.
The de{*word*81} stop execution at the following instruction:
Picture1[i]->Bitmap->LoadFromFile(OpenDialog1->Files->Strings[i]);
and the message is
"... raised exception class EInvalidGraphic with message 'Bitmap image
is not valid' ..."
Quote

>i belive the second problem is caused by TOpenDialog component

Why do you think that? Is the exception occuring before Execute() has
exited? If not, then the problem in not in TOpenPictureDialog at all.
It is very hard to diagnose this problem without more information.


The program error message is : "Bitmap is not valid".
After reading the chris message i think that the problem is the system
can't allocate 250MB of contiguous block of memory; this impossibility
explain some other "strange" behaviour!
Conclusion:
you're right, the problem is not TOpenDialog component.
 

Re:Memory allocation problems ...

Hi Ed,
i've just finish to test your little, useful program, on two pc one with
2GigaB and the other with 512MegaB of memory, and the results are the
following:
PC with 2GB:
. executing "testmem600.exe" it's ALL OK
. executing "testmem700.exe" FAILED
PC with 512MB:
. executing "testmem600.exe" it's ALL OK
. executing "testmem700.exe" it's ALL OK!!!!!!!!
Premise:
On both PCs there are no application run when test has been executed,
only except for "Paint", "task manager", "explorer", "Window command"
and, only for PCs with 2GB, "Mozilla Thundembird".
Finnaly questionS:
1. WHY there aren't difference between two PCs?
2. HOW CAN IT POSSIBLE that the PC having more large amount of memory
fail with the test??!!??!!
The message box results and taskmanager i've just posted on "attachment" NG
Many thanks
 

Re:Memory allocation problems ...

kronos ha scritto:
sorry i forgot to write what i inteded for "testmem600" and "testmem700":
testmem600 means the allocation buffer is 600MegaB (six hundred)
testmem700 means the allocation buffer is 700MegaB (seven hundred)
Quote
Hi Ed,

i've just finish to test your little, useful program, on two pc one with
2GigaB and the other with 512MegaB of memory, and the results are the
following:

PC with 2GB:
. executing "testmem600.exe" it's ALL OK
. executing "testmem700.exe" FAILED

PC with 512MB:
. executing "testmem600.exe" it's ALL OK
. executing "testmem700.exe" it's ALL OK!!!!!!!!


Premise:
On both PCs there are no application run when test has been executed,
only except for "Paint", "task manager", "explorer", "Window command"
and, only for PCs with 2GB, "Mozilla Thundembird".

Finnaly questionS:
1. WHY there aren't difference between two PCs?
2. HOW CAN IT POSSIBLE that the PC having more large amount of memory
fail with the test??!!??!!

The message box results and taskmanager i've just posted on "attachment" NG

Many thanks
 

Re:Memory allocation problems ...

Quote
On both PCs there are no application run when test has been executed, only
except for "Paint", "task manager", "explorer", "Window command" and, only
for PCs with 2GB, "Mozilla Thundembird".
All I can think of is these items:
The code is a GUI app. You don't need a console window. Paint and
Thunderbird can take a significant number of memory blocks. The problem
relates to the maximum size of a contiguous memory block and not to the size
of memory itself.
We've long since established that the behavior is a function of Windows and
not of the compiler or program. It's probably time to go into Control
panel|System|Advanced and look at the Virtual Memory settings in the two
computers.
. Ed
Quote
kronos wrote in message
news: XXXX@XXXXX.COM ...

i've just finish to test your little, useful program, on two pc one with
2GigaB and the other with 512MegaB of memory, and the results are the
following:

PC with 2GB:
. executing "testmem600.exe" it's ALL OK
. executing "testmem700.exe" FAILED

PC with 512MB:
. executing "testmem600.exe" it's ALL OK
. executing "testmem700.exe" it's ALL OK!!!!!!!!


Premise:
On both PCs there are no application run when test has been executed, only
except for "Paint", "task manager", "explorer", "Window command" and, only
for PCs with 2GB, "Mozilla Thundembird".

Finnaly questionS:
1. WHY there aren't difference between two PCs?
2. HOW CAN IT POSSIBLE that the PC having more large amount of memory fail
with the test??!!??!!

The message box results and taskmanager i've just posted on "attachment"
NG

Many thanks
 

Re:Memory allocation problems ...

"kronos" < XXXX@XXXXX.COM >wrote in message
Quote
The de{*word*81} stop execution at the following instruction:


Picture1[i]->Bitmap->LoadFromFile(OpenDialog1->Files->Strings[i]);

and the message is

"... raised exception class EInvalidGraphic with message
'Bitmap image is not valid' ..."
Well, there is your answer then. You are trying to load a file that
is not in a proper bitmap format that TBitmap recognizes. You will
have to wrap the line in a try..catch block if you want to handle the
error, ie:
try
{
Picture1[i]->Bitmap->LoadFromFile(OpenDialog1->Files->Strings[i]);
}
catch(const Exception &)
{
Picture1[i]->Assign(NULL);
}
if( (Picture1[i]->Graphic) && (!Picture[i]->Graphic->Empty) )
// file was loaded successfully...
else
// file was not loaded, or was empty...
Quote
The program error message is : "Bitmap is not valid".
Like I told you earlier, this issue has absolutely nothing to do with
TOpenPictureDialog at all. You could hard-code the filename and still
have the same problem without using the dialog at all. The problem is
in the actual file itself.
Quote
After reading the chris message i think that the problem is the
system can't allocate 250MB of contiguous block of memory;
this impossibility explain some other "strange" behaviour!
That would not cause the error you are seeing. A different message
would be reported if memory could not be allocated.
Gambit
 

Re:Memory allocation problems ...

Remy Lebeau (TeamB) ha scritto:
Quote

try
{

Picture1[i]->Bitmap->LoadFromFile(OpenDialog1->Files->Strings[i]);
}
catch(const Exception &)
{
Picture1[i]->Assign(NULL);
}

if( (Picture1[i]->Graphic) && (!Picture[i]->Graphic->Empty) )
// file was loaded successfully...
else
// file was not loaded, or was empty...
I've just implement this code and work properly.
Quote
>The program error message is : "Bitmap is not valid".

Like I told you earlier, this issue has absolutely nothing to do with
TOpenPictureDialog at all. You could hard-code the filename and still
have the same problem without using the dialog at all.
About this i agree with you.
Quote
The problem is in the actual file itself.
About this i don't agree with you.
I think so because i can open the same bitmap file if i only select it
by opendialog; if i select multi large bitmap files (250 MB for each) at
the n-th (typically at the 4-th) open the exception error raise.
Quote

That would not cause the error you are seeing. A different message
would be reported if memory could not be allocated.
it would be but i'm convinced if I understand the above allocation
memory problem (see the other thread) i'll also understand this one.
Thanks for your help.
 

Re:Memory allocation problems ...

i reply addressing you at this link
tinyurl.com/35j5y3
thanks for your interest.
P.S.: i am "gildo"
 

Re:Memory allocation problems ...

Remy Lebeau (TeamB) ha scritto:
Quote

Think about it. That is ~1 GB of data being loaded into memory at one
time. There is no way you have that much contiguous memory available,
either physically or virtually. You need to seriously re-think why
you are trying to load so many large bitmaps at one time to begin
with. I also suggest you seriously consider using another image
format. PNG, GIF, or JPG would consume much less memory than BMP that
comparible dimensions.

This is a goog tip and probabily i'll do so but i belive it's important
to test the program on extreme condition so as to i'll know answer to
future users questions about manage a large size images.
Thanks again