Board index » delphi » Increase in Memory Consumption Problem

Increase in Memory Consumption Problem

I have an application that increases the amount of memory it uses, and I
can't find the reason. When the application closes it frees all the memory.
It's written in Delphi 3 and uses ODBC/BDE to connect to SQL Server. I'm
using several TDatabases, TQueries, and TStroredProcs.

I'll try to explain how it works:

The program loops through a number of remote databases. It returns a dataset
from one database and writes it to another one (data server). There are
several databases scanned in each loop, and the loop executes periodically.
When the loop finishes, sometimes (not always) all the memory used isn't
freed. I use the Task Manager and Performance Monitor to control the use of
memory.

Another question related to this is the following: What exactly is the .REM
file type? What generates it and what information does it contain? Once this
program had a .REM nearly 50Mb in size. I have found any information
anywhere on the file type.

Thanks in advance,
Daniel Levy

 

Re:Increase in Memory Consumption Problem


Quote
On Wed, 12 Jul 2000 17:45:06 -0300, "Daniel Levy" <dl...@str.com.ar> wrote:
>I have an application that increases the amount of memory it uses, and I
>can't find the reason. When the application closes it frees all the memory.
>It's written in Delphi 3 and uses ODBC/BDE to connect to SQL Server. I'm
>using several TDatabases, TQueries, and TStroredProcs.

You don't mention it,  but I bet you are using BDE v5.x,  and the ODBC
drivers from MDAC 2.0SP2 (or higher).  Additionally,  I bet that you are
keeping connections open in your BDE sessions.  If you close all BDE
sessions but keep the application open,  does it release all of the memory?
If so,  then you are getting hit by the same thing that hit me about a year
ago...  The problem appears to be related to some new "connection pooling"
features added in the newer ODBC drivers.  I have not been able to
positively ID if it is in ODBC,  or if it is the BDE itself which is
"leaking" the memory,  but you lose about 4k +/- for each run of a TQuery,
maybe more if a large results set is returned...

Quote
>Another question related to this is the following: What exactly is the .REM
>file type? What generates it and what information does it contain? Once this
>program had a .REM nearly 50Mb in size. I have found any information
>anywhere on the file type.

I am not sure here.  Where are these files created,  and what are their
base file names?

Jerry Bloomfield (TeamB)
--
http://www.teamb.com                                      JersW...@wwa.com
Please do *NOT* send private e-mail without prior permission (my anti-spam
filters will probably just delete it anyway <g>)

Re:Increase in Memory Consumption Problem


Jerry Bloomfield (TeamB) <JersW...@wwa.com> wrote in message
news:sa2tmss7cj26kbn7hh91pi80jattvvgr8f@4ax.com...

Quote
> On Wed, 12 Jul 2000 17:45:06 -0300, "Daniel Levy" <dl...@str.com.ar>
wrote:

> >I have an application that increases the amount of memory it uses, and I
> >can't find the reason. When the application closes it frees all the
memory.
> >It's written in Delphi 3 and uses ODBC/BDE to connect to SQL Server. I'm
> >using several TDatabases, TQueries, and TStroredProcs.

> You don't mention it,  but I bet you are using BDE v5.x,  and the ODBC
> drivers from MDAC 2.0SP2 (or higher).  Additionally,  I bet that you are
> keeping connections open in your BDE sessions.  If you close all BDE
> sessions but keep the application open,  does it release all of the
memory?
> If so,  then you are getting hit by the same thing that hit me about a
year
> ago...  The problem appears to be related to some new "connection pooling"
> features added in the newer ODBC drivers.  I have not been able to
> positively ID if it is in ODBC,  or if it is the BDE itself which is
> "leaking" the memory,  but you lose about 4k +/- for each run of a TQuery,
> maybe more if a large results set is returned...

Yes, I am using BDE v5.00 and the ODBC drivers from MDAC v02.10.3711.0. I am
not explicitly using the Session property of the Databases. I do open and
close the database connection each time I use a table.

How did you solve the problem?

Quote
> >Another question related to this is the following: What exactly is the
.REM
> >file type? What generates it and what information does it contain? Once
this
> >program had a .REM nearly 50Mb in size. I have found any information
> >anywhere on the file type.

> I am not sure here.  Where are these files created,  and what are their
> base file names?

These files are created in the TEMP directory when the application starts y
are eliminated when the app is stopped. Apparently, they are created if the
app works with databases (not sure if only with SQL Server). Only one file
is created with each app. The base names are INMEMxxx.REM, where xxx are
digits. I have been monitoring the app's .rem file but it always has 0
bytes. That's why I need to know how, why, and when it grew so much.

Thank you for your help.
Daniel Levy
dl...@str.com.ar

Re:Increase in Memory Consumption Problem


Quote
On Fri, 14 Jul 2000 12:21:03 -0300, "Daniel Levy" <dl...@str.com.ar> wrote:
>Yes, I am using BDE v5.00 and the ODBC drivers from MDAC v02.10.3711.0. I am
>not explicitly using the Session property of the Databases. I do open and
>close the database connection each time I use a table.

>How did you solve the problem?

Well,  I actually resolved it in a rather awkward manner.  I have a certain
event which occurs that will cause all database activity to halt and wait
for the event to finish.  By doing this,  I know that all open connections
to the database are closed,  and at that point,  the memory is released...

Quote
>These files are created in the TEMP directory when the application starts y
>are eliminated when the app is stopped. Apparently, they are created if the
>app works with databases (not sure if only with SQL Server). Only one file
>is created with each app. The base names are INMEMxxx.REM, where xxx are
>digits. I have been monitoring the app's .rem file but it always has 0
>bytes. That's why I need to know how, why, and when it grew so much.

Now that you say that,  I do remember seeing them once or twice.  I am not
positive,  but I suspect that they are "local memory images of a remote
dataset".  In the most recent case I had seen one,  it was created by the
ADOExpress components using an ODBC connection to a remote Oracle server.
More than likely,  they will hold the results of the query for local cursor
browsing...

Jerry Bloomfield (TeamB)
--
http://www.teamb.com                                      JersW...@wwa.com
Please do *NOT* send private e-mail without prior permission (my anti-spam
filters will probably just delete it anyway <g>)

Re:Increase in Memory Consumption Problem


Based on what you've mentioned about connection pooling, I've been giving it
a look. The server is configured with a CPTimeout of 60 sec., while my
application loops once every 30 sec. Right now I'm testing different scan
times to determine if this is my problem.

Concerning the database activity, I do close the databases every time I
finish the loop. Do you think I should explicitly close all database
connections again before looping?

Thanks again,
Daniel Levy
dl...@str.com.ar

Jerry Bloomfield (TeamB) <JersW...@wwa.com> wrote in message
news:pmavmscqbg42cpl6mfucikn70huoif06av@4ax.com...

Quote
> On Fri, 14 Jul 2000 12:21:03 -0300, "Daniel Levy" <dl...@str.com.ar>
wrote:

> >Yes, I am using BDE v5.00 and the ODBC drivers from MDAC v02.10.3711.0. I
am
> >not explicitly using the Session property of the Databases. I do open and
> >close the database connection each time I use a table.

> >How did you solve the problem?

> Well,  I actually resolved it in a rather awkward manner.  I have a
certain
> event which occurs that will cause all database activity to halt and wait
> for the event to finish.  By doing this,  I know that all open connections
> to the database are closed,  and at that point,  the memory is released...

> >These files are created in the TEMP directory when the application starts
y
> >are eliminated when the app is stopped. Apparently, they are created if
the
> >app works with databases (not sure if only with SQL Server). Only one
file
> >is created with each app. The base names are INMEMxxx.REM, where xxx are
> >digits. I have been monitoring the app's .rem file but it always has 0
> >bytes. That's why I need to know how, why, and when it grew so much.

> Now that you say that,  I do remember seeing them once or twice.  I am not
> positive,  but I suspect that they are "local memory images of a remote
> dataset".  In the most recent case I had seen one,  it was created by the
> ADOExpress components using an ODBC connection to a remote Oracle server.
> More than likely,  they will hold the results of the query for local
cursor
> browsing...

> Jerry Bloomfield (TeamB)
> --
> http://www.teamb.com                                      JersW...@wwa.com
> Please do *NOT* send private e-mail without prior permission (my anti-spam
> filters will probably just delete it anyway <g>)

Re:Increase in Memory Consumption Problem


Quote
On Mon, 17 Jul 2000 17:10:39 -0300, "Daniel Levy" <dl...@str.com.ar> wrote:
>Concerning the database activity, I do close the databases every time I
>finish the loop. Do you think I should explicitly close all database
>connections again before looping?

Once all connections are closed (so there are not open connections at all)
the "leaked" memory will be released...  Sounds a little odd,  but that is
what I have seen on several different computers...

Jerry Bloomfield (TeamB)
--
http://www.teamb.com                                      JersW...@wwa.com
Please do *NOT* send private e-mail without prior permission (my anti-spam
filters will probably just delete it anyway <g>)

Re:Increase in Memory Consumption Problem


Jerry Bloomfield (TeamB) <JersW...@wwa.com> wrote in message
news:v1hfnsohnr45ts3i28lheg18vutg13167n@4ax.com...

Quote
> Once all connections are closed (so there are not open connections at all)
> the "leaked" memory will be released...  Sounds a little odd,  but that is
> what I have seen on several different computers...

I actually do close every database that I use when I exit the loop. I even
close the TQueries that I use before I close the databases. I prepare and
unprepare every TQuery before it runs. But I still lose memory. I've have
lowered the scan rate of the databases that I read from to 5 minutes. This
gives me time to free the connections and the memory that I use (timeout is
60 seconds). This has helped a bit, but overall I keep losing some. The leak
in the VM also grows every now and then by 4kb +/- but this never gets
released, not even with the 5 minutes scan time.

I have also tried a fix to a bug in a VM leak with MDAC 2.1, reported in the
MSDN but this hasn't helped any.

I'm not sure if the memory usage reported by the Task Manager is accurate
while debugging, but I've discovered that I lose 4kb +/- once in a while
(not very frequently) inside the loop, when a stored procedure runs. I
haven't prepared and unprepared the sp yet. I'm also reviewing the sp to
find anything wrong. I'm going to try this and let you know.

Thanks,
Daniel Levy
dl...@str.com.ar

Re:Increase in Memory Consumption Problem


You will need to remove all TSession, even the default Session before the
BDE will release.
Quote
Daniel Levy <dl...@str.com.ar> wrote in message news:3978ac3f@dnews...

> Jerry Bloomfield (TeamB) <JersW...@wwa.com> wrote in message
> news:v1hfnsohnr45ts3i28lheg18vutg13167n@4ax.com...

> > Once all connections are closed (so there are not open connections at
all)
> > the "leaked" memory will be released...  Sounds a little odd,  but that
is
> > what I have seen on several different computers...

> I actually do close every database that I use when I exit the loop. I even
> close the TQueries that I use before I close the databases. I prepare and
> unprepare every TQuery before it runs. But I still lose memory. I've have
> lowered the scan rate of the databases that I read from to 5 minutes. This
> gives me time to free the connections and the memory that I use (timeout
is
> 60 seconds). This has helped a bit, but overall I keep losing some. The
leak
> in the VM also grows every now and then by 4kb +/- but this never gets
> released, not even with the 5 minutes scan time.

> I have also tried a fix to a bug in a VM leak with MDAC 2.1, reported in
the
> MSDN but this hasn't helped any.

> I'm not sure if the memory usage reported by the Task Manager is accurate
> while debugging, but I've discovered that I lose 4kb +/- once in a while
> (not very frequently) inside the loop, when a stored procedure runs. I
> haven't prepared and unprepared the sp yet. I'm also reviewing the sp to
> find anything wrong. I'm going to try this and let you know.

> Thanks,
> Daniel Levy
> dl...@str.com.ar

Re:Increase in Memory Consumption Problem


I don't really use any additional TSession. Just the default one, and I
don't explicitly do anything with it. I am going to try the
Session.DropConnections method to see if this helps.

Thanks,
Daniel Levy
dl...@str.com.ar

Quote
Michael Harris <min...@mciworld.com> wrote in message

news:8lah3r$l3j9@bornews.borland.com...
Quote
> You will need to remove all TSession, even the default Session before the
> BDE will release.

> Daniel Levy <dl...@str.com.ar> wrote in message news:3978ac3f@dnews...

> > Jerry Bloomfield (TeamB) <JersW...@wwa.com> wrote in message
> > news:v1hfnsohnr45ts3i28lheg18vutg13167n@4ax.com...

> > > Once all connections are closed (so there are not open connections at
> all)
> > > the "leaked" memory will be released...  Sounds a little odd,  but
that
> is
> > > what I have seen on several different computers...

> > I actually do close every database that I use when I exit the loop. I
even
> > close the TQueries that I use before I close the databases. I prepare
and
> > unprepare every TQuery before it runs. But I still lose memory. I've
have
> > lowered the scan rate of the databases that I read from to 5 minutes.
This
> > gives me time to free the connections and the memory that I use (timeout
> is
> > 60 seconds). This has helped a bit, but overall I keep losing some. The
> leak
> > in the VM also grows every now and then by 4kb +/- but this never gets
> > released, not even with the 5 minutes scan time.

> > I have also tried a fix to a bug in a VM leak with MDAC 2.1, reported in
> the
> > MSDN but this hasn't helped any.

> > I'm not sure if the memory usage reported by the Task Manager is
accurate
> > while debugging, but I've discovered that I lose 4kb +/- once in a while
> > (not very frequently) inside the loop, when a stored procedure runs. I
> > haven't prepared and unprepared the sp yet. I'm also reviewing the sp to
> > find anything wrong. I'm going to try this and let you know.

> > Thanks,
> > Daniel Levy
> > dl...@str.com.ar

Re:Increase in Memory Consumption Problem


When that doesn't work. You may be able to improve on this method.

in each loop
before database connection = true;

if(sessionfree)  //  a boolean value
   {
     sessionfree=false;
     Session=new TSession( NULL );
     Session->SessionName = "default" + IntToStr(sescount);
     sescount++;  // an integer
   }

after connections are closed.

   TSession * ses = Session;
    ses->Free();
    sessionfree=true;

Quote
Daniel Levy <dl...@str.com.ar> wrote in message news:397ca317_2@dnews...
> I don't really use any additional TSession. Just the default one, and I
> don't explicitly do anything with it. I am going to try the
> Session.DropConnections method to see if this helps.

> Thanks,
> Daniel Levy
> dl...@str.com.ar

Re:Increase in Memory Consumption Problem


The DropConnections method didn't work. I also tried you solution but I
think I missed something, it gave me a runtime error. I'll check it again.

Should I do this separately for every database? Should I assign each
different TDatabase.SessionName a different session?

Thanks
Daniel Levy
dl...@str.com.ar

Quote
Michael Harris <min...@mciworld.com> wrote in message

news:8ligcu$7jl7@bornews.borland.com...
Quote
> When that doesn't work. You may be able to improve on this method.

> in each loop
> before database connection = true;

> if(sessionfree)  //  a boolean value
>    {
>      sessionfree=false;
>      Session=new TSession( NULL );
>      Session->SessionName = "default" + IntToStr(sescount);
>      sescount++;  // an integer
>    }

> after connections are closed.

>    TSession * ses = Session;
>     ses->Free();
>     sessionfree=true;

> Daniel Levy <dl...@str.com.ar> wrote in message news:397ca317_2@dnews...
> > I don't really use any additional TSession. Just the default one, and I
> > don't explicitly do anything with it. I am going to try the
> > Session.DropConnections method to see if this helps.

> > Thanks,
> > Daniel Levy
> > dl...@str.com.ar

Re:Increase in Memory Consumption Problem


I have set the connection pooling timeout at 5 min. This gives enough time
to close the connections. This hasn't solved entirely the problem because in
the longrun not all of the memory gets released, but some does get freed.

Thank's for your help,
Daniel Levy
dl...@str.com.ar

Quote
Daniel Levy <dl...@str.com.ar> wrote in message news:397365c8@dnews...
> Based on what you've mentioned about connection pooling, I've been giving
it
> a look. The server is configured with a CPTimeout of 60 sec., while my
> application loops once every 30 sec. Right now I'm testing different scan
> times to determine if this is my problem.

> Concerning the database activity, I do close the databases every time I
> finish the loop. Do you think I should explicitly close all database
> connections again before looping?

> Thanks again,
> Daniel Levy
> dl...@str.com.ar

> Jerry Bloomfield (TeamB) <JersW...@wwa.com> wrote in message
> news:pmavmscqbg42cpl6mfucikn70huoif06av@4ax.com...
> > On Fri, 14 Jul 2000 12:21:03 -0300, "Daniel Levy" <dl...@str.com.ar>
> wrote:

> > >Yes, I am using BDE v5.00 and the ODBC drivers from MDAC v02.10.3711.0.
I
> am
> > >not explicitly using the Session property of the Databases. I do open
and
> > >close the database connection each time I use a table.

> > >How did you solve the problem?

> > Well,  I actually resolved it in a rather awkward manner.  I have a
> certain
> > event which occurs that will cause all database activity to halt and
wait
> > for the event to finish.  By doing this,  I know that all open
connections
> > to the database are closed,  and at that point,  the memory is
released...

> > >These files are created in the TEMP directory when the application
starts
> y
> > >are eliminated when the app is stopped. Apparently, they are created if
> the
> > >app works with databases (not sure if only with SQL Server). Only one
> file
> > >is created with each app. The base names are INMEMxxx.REM, where xxx
are
> > >digits. I have been monitoring the app's .rem file but it always has 0
> > >bytes. That's why I need to know how, why, and when it grew so much.

> > Now that you say that,  I do remember seeing them once or twice.  I am
not
> > positive,  but I suspect that they are "local memory images of a remote
> > dataset".  In the most recent case I had seen one,  it was created by
the
> > ADOExpress components using an ODBC connection to a remote Oracle
server.
> > More than likely,  they will hold the results of the query for local
> cursor
> > browsing...

> > Jerry Bloomfield (TeamB)
> > --
> > http://www.teamb.com

JersW...@wwa.com

- Show quoted text -

Quote
> > Please do *NOT* send private e-mail without prior permission (my
anti-spam
> > filters will probably just delete it anyway <g>)

Other Threads