Board index » cppbuilder » memory problem

memory problem


2003-11-06 05:25:34 PM
cppbuilder59
Hi,
I have some memory problem with my application which is extremely hard to
trace. (I have no clue so far).
Checking the heapstatus (GetHeapStatus) shows an increasing TotalAddrSpace
and a corresponding increasing TotalUncommitted.
Very important is that TotalAllocated remains constant.
TotalFree is increasing, FreeSmall is slightly increasing, FreeBig is
increasing.
TotalCommitted is increasing the same amount as TotalFree.
There are no leaks, CodeGuard is reporting nothing, MemProof doesn't report
anything, AQTime doesn't report anything. The program runs well.
But here's the problem. because TotalAddrSpace increases, at a certain
moment the system runs out of Virtual Memory. At this point things go wrong.
Where do I start tracing?
Wim
 
 

Re:memory problem

you are initiating some components with out releasing them ..
or you could be allocating memory with out releasing this.,
you could have a DLL file that is causing it and thus is hard to track with
the tools your using.
Wim Bekker wrote:
Quote
Hi,

I have some memory problem with my application which is extremely hard to
trace. (I have no clue so far).

Checking the heapstatus (GetHeapStatus) shows an increasing TotalAddrSpace
and a corresponding increasing TotalUncommitted.

Very important is that TotalAllocated remains constant.

TotalFree is increasing, FreeSmall is slightly increasing, FreeBig is
increasing.
TotalCommitted is increasing the same amount as TotalFree.

There are no leaks, CodeGuard is reporting nothing, MemProof doesn't report
anything, AQTime doesn't report anything. The program runs well.

But here's the problem. because TotalAddrSpace increases, at a certain
moment the system runs out of Virtual Memory. At this point things go wrong.

Where do I start tracing?

Wim
 

Re:memory problem

Hello all!
I'm using IBX5.04 with Delphi 5.
I'm using some querys accessing my remote Interbase database (6.02).
The problem is that the memory of my application keeps rising and rising. Is
not much, but it keeps rising. Since this is a 24/7 application, this is a
huge problem, since sooner or later it will crash.
Is anything wrong with the code below? All my querys
(selects/inserts/updates) are made like this.
Any tips?
Thanks in advance
Goncalo Martins
function DevolveNomeEmpresa:string;
var
IBSQL: TIBSQL;
Transaction: TIBTransaction;
str:string;
begin
IBSQL := TIBSQL.Create(nil);
Transaction := TIBTransaction.Create(nil);
try
Transaction.DefaultDatabase := Frmdados.IBDatabase1;
IBSQL.Database := Frmdados.IBDatabase1;
IBSQL.Transaction := Transaction;
Transaction.StartTransaction;
try
ibsql.close;
ibsql.SQL.clear;
str:='select nome from empresa';
ibsql.SQL.Add(str);
ibsql.Prepare;
ibsql.execquery;
if ibsql.recordcount>0 then
result:=ibsql.fieldbyname('nome').asstring
else
result:='';
ibsql.Close;
finally
Transaction.Commit;
end;
finally
IBSQL.Free;
Transaction.Free;
end;
end;
 

{smallsort}

Re:memory problem

Gongalo Martins wrote:
Quote
Is anything wrong with the code below?
Yes, but no memory leaks that I see.
Quote
All my querys
(selects/inserts/updates) are made like this.
Any tips?

function DevolveNomeEmpresa:string;
var
IBSQL: TIBSQL;
Transaction: TIBTransaction;
str:string;
begin
IBSQL := TIBSQL.Create(nil);
Transaction := TIBTransaction.Create(nil);
try
Transaction.DefaultDatabase := Frmdados.IBDatabase1;
IBSQL.Database := Frmdados.IBDatabase1;
IBSQL.Transaction := Transaction;
It doesn't matter a lot, but you really don't need a SNAPSHOT
transaction here. I'd suggest a read-committed, read-only transaction
(set TIBTransaction.Params) in order to minimize the load on the IB
server and keep the snapshot number from moving forward.
Quote
Transaction.StartTransaction;
try
These lines:
Quote
ibsql.close;
ibsql.SQL.clear;
...are unnecessary on a newly created TIBSQL.
Quote
str:='select nome from empresa';
ibsql.SQL.Add(str);
ibsql.Prepare;
Prepare is almost always unnecessary with IBX, unless you want to
fetch the PLAN without executing the query. IBX will automatically
prepare the query when you call ExecQuery.
Quote
ibsql.execquery;
if ibsql.recordcount>0 then
Don't use RecordCount at all with an SQL back end -- see my
Optimization I article for reasons and workarounds.
Quote
result:=ibsql.fieldbyname('nome').asstring
else
result:='';
The whole if/then here is unnecessary. You can simply write:
result:=ibsql.fieldbyname('nome').asstring;
...since AsString returns an empty string when there is a null state
or an empty dataset.
I recommend use of AQTime, MemProof, or BoundsChecker to track memory
allocations.
-Craig
--
Craig Stuntz [TeamB] . Vertex Systems Corp. . Columbus, OH
Delphi/InterBase Weblog : delphi.weblogs.com
Useful articles about InterBase and Delphi development:
delphi.weblogs.com/articles
 

Re:memory problem

Ok, thanks for the tips Craig
I guess the problem is not in IBX, since I tried another components, and the
problem remains.
Best Regards
Goncalo
"Craig Stuntz [TeamB]" < XXXX@XXXXX.COM [a.k.a. vertexsoftware.com]>
wrote in message news: XXXX@XXXXX.COM ...
Quote
Gongalo Martins wrote:

>Is anything wrong with the code below?

Yes, but no memory leaks that I see.

>All my querys
>(selects/inserts/updates) are made like this.
>Any tips?
>
>function DevolveNomeEmpresa:string;
>var
>IBSQL: TIBSQL;
>Transaction: TIBTransaction;
>str:string;
>begin
>IBSQL := TIBSQL.Create(nil);
>Transaction := TIBTransaction.Create(nil);
>try
>Transaction.DefaultDatabase := Frmdados.IBDatabase1;
>IBSQL.Database := Frmdados.IBDatabase1;
>IBSQL.Transaction := Transaction;

It doesn't matter a lot, but you really don't need a SNAPSHOT
transaction here. I'd suggest a read-committed, read-only transaction
(set TIBTransaction.Params) in order to minimize the load on the IB
server and keep the snapshot number from moving forward.

>Transaction.StartTransaction;
>try

These lines:

>ibsql.close;
>ibsql.SQL.clear;

...are unnecessary on a newly created TIBSQL.

>str:='select nome from empresa';
>ibsql.SQL.Add(str);
>ibsql.Prepare;

Prepare is almost always unnecessary with IBX, unless you want to
fetch the PLAN without executing the query. IBX will automatically
prepare the query when you call ExecQuery.

>ibsql.execquery;
>if ibsql.recordcount>0 then

Don't use RecordCount at all with an SQL back end -- see my
Optimization I article for reasons and workarounds.

>result:=ibsql.fieldbyname('nome').asstring
>else
>result:='';

The whole if/then here is unnecessary. You can simply write:

result:=ibsql.fieldbyname('nome').asstring;

...since AsString returns an empty string when there is a null state
or an empty dataset.

I recommend use of AQTime, MemProof, or BoundsChecker to track memory
allocations.

-Craig

--
Craig Stuntz [TeamB] . Vertex Systems Corp. . Columbus, OH
Delphi/InterBase Weblog : delphi.weblogs.com
Useful articles about InterBase and Delphi development:
delphi.weblogs.com/articles
 

Re:memory problem

"Craig Stuntz [TeamB]" < XXXX@XXXXX.COM [a.k.a. vertexsoftware.com]>
escribi?en el mensaje news: XXXX@XXXXX.COM ...
Quote
Gongalo Martins wrote:

>Is anything wrong with the code below?

Yes, but no memory leaks that I see.

>All my querys
>(selects/inserts/updates) are made like this.
>Any tips?
>
>function DevolveNomeEmpresa:string;
>var
>IBSQL: TIBSQL;
>Transaction: TIBTransaction;
>str:string;
>begin
>IBSQL := TIBSQL.Create(nil);
>Transaction := TIBTransaction.Create(nil);
>try
>Transaction.DefaultDatabase := Frmdados.IBDatabase1;
>IBSQL.Database := Frmdados.IBDatabase1;
>IBSQL.Transaction := Transaction;

It doesn't matter a lot, but you really don't need a SNAPSHOT
transaction here. I'd suggest a read-committed, read-only transaction
(set TIBTransaction.Params) in order to minimize the load on the IB
server and keep the snapshot number from moving forward.

>Transaction.StartTransaction;
>try

These lines:

>ibsql.close;
>ibsql.SQL.clear;

...are unnecessary on a newly created TIBSQL.

>str:='select nome from empresa';
>ibsql.SQL.Add(str);
>ibsql.Prepare;

Prepare is almost always unnecessary with IBX, unless you want to
fetch the PLAN without executing the query. IBX will automatically
prepare the query when you call ExecQuery.

>ibsql.execquery;
>if ibsql.recordcount>0 then

Don't use RecordCount at all with an SQL back end -- see my
Optimization I article for reasons and workarounds.

>result:=ibsql.fieldbyname('nome').asstring
>else
>result:='';

The whole if/then here is unnecessary. You can simply write:
False...
if you execute an ibsql and .recordcount = 0 then if you read the value of
the field you get the last value of the last not null...
Quote

result:=ibsql.fieldbyname('nome').asstring;

...since AsString returns an empty string when there is a null state
or an empty dataset.

I recommend use of AQTime, MemProof, or BoundsChecker to track memory
allocations.

-Craig

--
Craig Stuntz [TeamB] . Vertex Systems Corp. . Columbus, OH
Delphi/InterBase Weblog : delphi.weblogs.com
Useful articles about InterBase and Delphi development:
delphi.weblogs.com/articles
 

Re:memory problem

Hi ppl.
Does Bold free memory, or just mark it as free in it's structure?
Noticed that when I load something. (like 20 orders) memory is going High.
from ie. 10 to 20 megs. but when I try to unload, it unloads only 1-3 mb.
In bold system de{*word*81} you have:
Before LOAD:
Allocated : 738kb ( 755712 bytes)
InUse : 412kb ( 422800 bytes)
Overhead : 325kb ( 332912 bytes)
Total Big blks : 285kb ( 292208 bytes)
Big block count: 3594 ( 81 bytes)
Non-Bold : 802kb ( 821552 bytes)
Non-Bold Count : 29354 ( 27 bytes)
(IN TASK MANAGER 10MB)
AFTER LOAD:
Allocated : 5664kb ( 5799936 bytes)
InUse : 2727kb ( 2792828 bytes)
Overhead : 2936kb ( 3007108 bytes)
Total Big blks : 221kb ( 226484 bytes)
Big block count: 1521 ( 148 bytes)
Non-Bold : 1574kb ( 1611860 bytes)
Non-Bold Count : 48861 ( 32 bytes)
(MEMORY IN TASK MANAGER 25MB)
AFTER UNLOAD:
Allocated : 5676kb ( 5812224 bytes)
InUse : 2024kb ( 2072676 bytes)
Overhead : 3651kb ( 3739548 bytes)
Total Big blks : 204kb ( 209516 bytes)
Big block count: 1295 ( 161 bytes)
Non-Bold : 1257kb ( 1287812 bytes)
Non-Bold Count : 31482 ( 40 bytes)
(MEMORY IN TASK MANAGER 22M)
But when the program loads it has only 10MB in RAM. After loading and
unloading items. the AllLoadedObjects = 0 but the program doesn't go back to
10.
Why ?Thnx.Oleg.
 

Re:memory problem

Bold Memory management has always been a mystery. There are some people
here who claim that they know whats going on, but... they really don't :)
In our 2+ years of experience with Bold we never been able to create an
app that does not leak memory at all. There are always 20-50bytes per
BoldObject that you never get back no matter what. There are several
open issues in QC related to Bold memory management, but they just don't
get much attention today.
If you have a relatively complex order structure (order, orderitem,
item, address, customer, credit card, tax tables, etc) you have to
expect to leak at least few hundred bytes per order.
Though 500K/order as you have now is way to much.
Make sure
- You have all visual controls/grids connected to the bold system
disabled/hidden, otherwise your "unloaded" object will be loaded again
automatically
- You app compiled with BOLD_DISABLEMEMORYMANAGER compiler switch
- You do not use DBISAM, XML or BoldHTTP (or BoldExpress Synapse) for
presistence (there big leaks in these implementations)
Hope that helps
Alex
Oleg Toplionkin wrote:
Quote
Hi ppl.
Does Bold free memory, or just mark it as free in it's structure?

Noticed that when I load something. (like 20 orders) memory is going High.
from ie. 10 to 20 megs. but when I try to unload, it unloads only 1-3 mb.

In bold system de{*word*81} you have:
Before LOAD:
Allocated : 738kb ( 755712 bytes)
InUse : 412kb ( 422800 bytes)
Overhead : 325kb ( 332912 bytes)
Total Big blks : 285kb ( 292208 bytes)
Big block count: 3594 ( 81 bytes)
Non-Bold : 802kb ( 821552 bytes)
Non-Bold Count : 29354 ( 27 bytes)
(IN TASK MANAGER 10MB)

AFTER LOAD:
Allocated : 5664kb ( 5799936 bytes)
InUse : 2727kb ( 2792828 bytes)
Overhead : 2936kb ( 3007108 bytes)
Total Big blks : 221kb ( 226484 bytes)
Big block count: 1521 ( 148 bytes)
Non-Bold : 1574kb ( 1611860 bytes)
Non-Bold Count : 48861 ( 32 bytes)
(MEMORY IN TASK MANAGER 25MB)

AFTER UNLOAD:
Allocated : 5676kb ( 5812224 bytes)
InUse : 2024kb ( 2072676 bytes)
Overhead : 3651kb ( 3739548 bytes)
Total Big blks : 204kb ( 209516 bytes)
Big block count: 1295 ( 161 bytes)
Non-Bold : 1257kb ( 1287812 bytes)
Non-Bold Count : 31482 ( 40 bytes)
(MEMORY IN TASK MANAGER 22M)

But when the program loads it has only 10MB in RAM. After loading and
unloading items. the AllLoadedObjects = 0 but the program doesn't go back to
10.
Why ?Thnx.Oleg.


 

Re:memory problem

Thank you Alex.
Sorry for silly question.
How do you compile with BOLD_DISABLEMEMORYMANAGER.
I know how to use the compiler switches, but this one I'm stuck.
Please say, where to write what.
Also. I'm using Firebird IB server. I recall that somebody was saying
earlier that it,or it's components have bid memory leaks. Can this be true?
Thank you,
Oleg.
"Alex U" < XXXX@XXXXX.COM >wrote in message
Quote
Bold Memory management has always been a mystery. There are some people
here who claim that they know whats going on, but... they really don't :)

In our 2+ years of experience with Bold we never been able to create an
app that does not leak memory at all. There are always 20-50bytes per
BoldObject that you never get back no matter what. There are several open
issues in QC related to Bold memory management, but they just don't get
much attention today.

If you have a relatively complex order structure (order, orderitem, item,
address, customer, credit card, tax tables, etc) you have to expect to
leak at least few hundred bytes per order.

Though 500K/order as you have now is way to much.

Make sure

- You have all visual controls/grids connected to the bold system
disabled/hidden, otherwise your "unloaded" object will be loaded again
automatically
- You app compiled with BOLD_DISABLEMEMORYMANAGER compiler switch
- You do not use DBISAM, XML or BoldHTTP (or BoldExpress Synapse) for
presistence (there big leaks in these implementations)

Hope that helps

Alex
 

Re:memory problem

"Alex U" < XXXX@XXXXX.COM >wrote in message
Quote
Bold Memory management has always been a mystery. There are some people
here who claim that they know whats going on, but... they really don't :)
I do know what's going on...
You have two objects:
BoldLocator
BoldObject
The BoldLocator is permanent - it holds the BoldID and nothing more.
Permanent in that while your system handle is active it never gets unloaded.
This locator holds the state of your object. If you ask for a list of
objects from persistence, it doesn't create 1000 BoldObjects, just
BoldLocators (much smaller).
The Bold Locator has a pointer to the BoldObject. When you request an
object, the locator returns it to you - if it's not in memory it will get
loaded at this point.
When you call Unload, you are freeing off the BoldObject, and nilling the
pointer on the BoldLocator, but not freeing the BoldLocator. This means
that the BoldList (which lists BoldLocators, not BoldObjects) still knows
the object is in the list, in the correct position, but doesn't need to hold
it in RAM to know that.
That explains the 20 or so bytes per object that never get freed (restart
the object space by deactivating the system handle to do this).
Now the bold memory manager...
In a traditional mem manager, the memory is allocated in large blocks from
Windoze and allocated to the app in small blocks, one for each object for
example. If you free a large object, and then create a small one, you're
fragmenting the available space, leaving little holes all over it. The Bold
guys noted that all objects of a particular class are the same size. So
they allocate a block exclusively for a certain size of sub-block.
Let's say ClassA and ClassB are 50 bytes, and ClassC is 75 bytes. In the
classic model if you create two of ClassC and free off the first one, you
have a 75 byte hole at the front. Now you create a ClassA, and you now have
a 25 byte hole. None of your classes fit in this hole, so it leaks! In the
bold model, your ClassA,B space is separate from your ClassC space. So now
it doesn't fragment. However, the memory manager doesn't free off blocks
once it's allocated them, so even if you free off some ClassCs, it doesn't
give you more space for ClassA or ClassB.
Therefore the best way to manage memory is to unload objects when you're
done with them to make space for more.
If you're doing a large batch process, do it in small chunks and unload
after each chunk.
Bryan
 

Re:memory problem

Hi Brian,
Many thanks for your clear explaination. I've never had so full picture of
this before that.
/Efim
 

Re:memory problem

Also.
I'm running DB on Firebird with general components.
I heard it has a memory leak somewhere?
is it true ?
thnx.
Oleg.
 

Re:memory problem

On Tue, 22 Nov 2005 15:44:36 +0200, "Oleg Toplionkin"
< XXXX@XXXXX.COM >wrote:
Quote
Also.

I'm running DB on Firebird with general components.

I heard it has a memory leak somewhere?
is it true ?

As for me, it may be true. I had such a problem too, and
BOLD_DISABLEMEMORYMANAGER did not help.
Then I begin use ZEOS DBO Delphi components, and all troubles
disappered.
 

Re:memory problem

Hello,
Repeatedly executing a query with different params each time results in
considerable memory consumption.
I free the ADOQuery after each DB access so there should be something
left/cached in ADO.
If I do the same with identical params memory stays the same ie something is
cached.
Any idea what is happening?
Thanks
Yannis Makarounsi
 

Re:memory problem

Yannis Makarounis schreef:
Quote
If I do the same with identical params memory stays the same ie something is
cached.
Any idea what is happening?
Perhaps you are not freeing the params after use but only reassigneing them?
Danny
---