Board index » delphi » BP7 Heap Manager patch?

BP7 Heap Manager patch?

Is there a patch for the BP7 Heap Manager?

It returns a nil pointer from GetMem
after 3885 calls, even though there's
lots of free memory left.

Please reply to netn...@altavista.net

Thanks.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

 

Re:BP7 Heap Manager patch?


Quote
netn...@altavista.net wrote:

> Is there a patch for the BP7 Heap Manager?
> It returns a nil pointer from GetMem
> after 3885 calls, even though there's
> lots of free memory left.

Surely? Did you ask MemAvail and/or MaxAvail?
Did it deliver an error message or did you disable that?
Is it with protected mode? Asked the DPMI manager?
--
Franz Glaser, Glasau 3, A-4191 Vorderweissenbach Austria ++43-7219-7035
Muehlviertler Elektronik Glaser. Industrial control and instrumentation
-----------------------------------------------------------------------
http://members.eunet.at/meg-glaser           mailto:meg-gla...@eunet.at
http://www.geocities.com/~franzglaser  http://members.xoom.com/f_glaser

Re:BP7 Heap Manager patch?


In article <36E30CEF.8EF6D...@eunet.at>,
  "Ing. Franz Glaser" <meg-gla...@eunet.at> wrote:

Quote
> netn...@altavista.net wrote:

> > Is there a patch for the BP7 Heap Manager?
> > It returns a nil pointer from GetMem
> > after 3885 calls, even though there's
> > lots of free memory left.

> Surely? Did you ask MemAvail and/or MaxAvail?
> Did it deliver an error message or did you disable that?
> Is it with protected mode? Asked the DPMI manager?

BP7 protected mode.  There's a problem with the Heap Manager.
It runs out of pointers to assign before all the memory
is used up.  I have a memory-hungry app (numerical analysis -
large floating-point matrices).  Is there a patch available,
or a work-around, or perhaps a third-party Unit to replace
the built-in one?

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re:BP7 Heap Manager patch?


In article <36E30CEF.8EF6D...@eunet.at>,
  "Ing. Franz Glaser" <meg-gla...@eunet.at> wrote:

Quote
> netn...@altavista.net wrote:

> > Is there a patch for the BP7 Heap Manager?
> > It returns a nil pointer from GetMem
> > after 3885 calls, even though there's
> > lots of free memory left.

> Surely? Did you ask MemAvail and/or MaxAvail?
> Did it deliver an error message or did you disable that?
> Is it with protected mode? Asked the DPMI manager?

BP7 protected mode.

Appended below is a small code which illustrates the
problem.  The Heap Manager runs out of pointers before
all the memory is allocated. (I have 64Meg).
OTOH, If you set mem=32*1024,
it allocates all the memory (requires less pointers).

var
p: pointer;
l: longint;

const
mem=1024;

begin

l:=0;
repeat
write(#13,l,' ',l*mem,' ',memavail,'  ');
GetMem(p,mem); inc(l);
until p=nil;

end.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re:BP7 Heap Manager patch?


Quote
netn...@altavista.net wrote:
> [snip]

> BP7 protected mode.

> Appended below is a small code which illustrates the
> problem.  The Heap Manager runs out of pointers before
> all the memory is allocated. (I have 64Meg).
> OTOH, If you set mem=32*1024,
> it allocates all the memory (requires less pointers).
> [code snipped]

This isn't a bug in BP7.  This is a condition imposed by protected mode.  When
you allocate a pointer, DPMI provides a selector value which it uses
internally to determine where that block of memory is located and how it
should be treated.  (Working from memory) 16-bit protected mode allocates 64K
for descriptors, where each is 8 bytes, which allows for a total of 8192
maximum in theory.  Some are allocated by the system and by the RTL, reducing
that number; and if you run under Windows, it probably also swallows up a
bunch of them, reducing the total amount.

So, if you keep allocating, eventually you *are* going to run out of
selectors.

Quote

> -----------== Posted via Deja News, The Discussion Network ==----------
> http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own

--
Scott Earnest            | SPAM protection in effect. Remove  |
setech@_ix.netcom.com    | "_" as needed for true addresses.  |
earnests@_homenet.lm.com |    UIN:1136443  EFnet:pale_blue    |
sinykal@_{*word*104}space.org  | URL: http://www.netcom.com/~setech |

Re:BP7 Heap Manager patch?


Quote
Scott Earnest wrote:
> netn...@altavista.net wrote:
> > [snip]
> > BP7 protected mode.
> > Appended below is a small code which illustrates the
> > problem.  The Heap Manager runs out of pointers before
> > all the memory is allocated. (I have 64Meg).
> > OTOH, If you set mem=32*1024,
> > it allocates all the memory (requires less pointers).
> > [code snipped]
> This isn't a bug in BP7.  This is a condition imposed by protected mode.  When
> you allocate a pointer, DPMI provides a selector value which it uses
> internally to determine where that block of memory is located and how it
> should be treated.  (Working from memory) 16-bit protected mode allocates 64K
> for descriptors, where each is 8 bytes, which allows for a total of 8192
> maximum in theory.  Some are allocated by the system and by the RTL, reducing
> that number; and if you run under Windows, it probably also swallows up a
> bunch of them, reducing the total amount.

> So, if you keep allocating, eventually you *are* going to run out of
> selectors.

So what's the solution? Can you avoid using Borland's heap manager, and
take care of memory management yourself?

How about allocating a block of memory, then using linear addressing
instead of pointers. Each linear address is 4 bytes, allowing twice as
many items. You could start at the top of the block and work down, and
put the data at the bottom of the block and work up. The available memory
is the difference between the last two entries minus 4 bytes.

Best Regards,

Michael R. Monett,
Automated Production Test
mailto:a...@csolve.net

Re:BP7 Heap Manager patch?


Quote
>Scott Earnest wrote:

>> This isn't a bug in BP7.  This is a condition imposed by protected mode.  When
>> you allocate a pointer, DPMI provides a selector value which it uses
>> internally to determine where that block of memory is located and how it
>> should be treated.  (Working from memory) 16-bit protected mode allocates 64K
>> for descriptors, where each is 8 bytes, which allows for a total of 8192
>> maximum in theory.  Some are allocated by the system and by the RTL, reducing
>> that number; and if you run under Windows, it probably also swallows up a
>> bunch of them, reducing the total amount.

>> So, if you keep allocating, eventually you *are* going to run out of
>> selectors.

Mike Monett <a...@csolve.net> wrote:
>So what's the solution? ....

You might try modifying HeapLimit and HeapBlock.  These should be
changed before any allocations are made.  If any units initializations
use the heap, you might want to modify these variables in a unit that
initializes first.  Declare the unit first in the program's uses clause.

The values from my heap initialization are:

BEGIN
   {$IFDEF DPMI}
       HeapBlock := $FFF0;
       HeapLimit := $4000;
   {$ENDIF}
END.

If you still fail, you might want to increase the size of HeapLimit
since it determines the size of an allocation that will obtain its own
selector.

    ...red

Re:BP7 Heap Manager patch?


Quote
netn...@altavista.net wrote:

> Is there a patch for the BP7 Heap Manager?

> It returns a nil pointer from GetMem
> after 3885 calls, even though there's
> lots of free memory left.

Well, I recently have used around 4850 pointers, assigned with NEW.
Does this strange thing happen only when using GetMem?
Could it be a solution to use a kind of 'faked' NEW instead of GetMem?

Arno Fehm

Re:BP7 Heap Manager patch?


Quote
In article <7bvbf0$vs...@nnrp1.dejanews.com>,  <netn...@altavista.net> wrote:

>BP7 protected mode.

>Appended below is a small code which illustrates the
>problem.  The Heap Manager runs out of pointers before
>all the memory is allocated. (I have 64Meg).
>OTOH, If you set mem=32*1024,
>it allocates all the memory (requires less pointers).

Could you please indent the code next time you post.

Quote

>var
>p: pointer;
>l: longint;

>const
>mem=1024;

>begin

>l:=0;
>repeat
>write(#13,l,' ',l*mem,' ',memavail,'  ');
>GetMem(p,mem); inc(l);
>until p=nil;

>end.

This has to do how the PM heap manager works. If the allocation is large
it allocates a new segment. However, if the allocation is small,
it allocates (if necessary) a large block and then suballocates
the request from that block. The threshold between the large and small
block is controlled by variable HeapLimit and the size of the large
block that is allocated by variable HeapBlock. Now the default for
HeapBlock is 8192 and the default value for HeapLimit is, surprise,
surprise, 1024.

So add the following lines at the beginning:

  heaplimit:=4096;
  heapblock:=32768+12;

The 12 comes from the observation that the first suballocation has
offset 12, so there will be no waste when the allocations are exactly
1024 bytes. If they are random length it does not matter.

Osmo

Re:BP7 Heap Manager patch?


In article <36E41E7C.A36C3...@bigfoot.de>, Arno Fehm  <af...@bigfoot.de> wrote:

Quote
>netn...@altavista.net wrote:

>> Is there a patch for the BP7 Heap Manager?

>> It returns a nil pointer from GetMem
>> after 3885 calls, even though there's
>> lots of free memory left.

>Well, I recently have used around 4850 pointers, assigned with NEW.
>Does this strange thing happen only when using GetMem?
>Could it be a solution to use a kind of 'faked' NEW instead of GetMem?

No, it makes no difference whether one uses new or getmem. The limit is
not on pointers but on segments.

Osmo

Re:BP7 Heap Manager patch?


Quote
Osmo Ronkanen wrote:

[...]

Quote
> This has to do how the PM heap manager works. If the allocation is large
> it allocates a new segment. However, if the allocation is small,
> it allocates (if necessary) a large block and then suballocates
> the request from that block. The threshold between the large and small
> block is controlled by variable HeapLimit and the size of the large
> block that is allocated by variable HeapBlock. Now the default for
> HeapBlock is 8192 and the default value for HeapLimit is, surprise,
> surprise, 1024.

> So add the following lines at the beginning:

>   heaplimit:=4096;
>   heapblock:=32768+12;

> The 12 comes from the observation that the first suballocation has
> offset 12, so there will be no waste when the allocations are exactly
> 1024 bytes. If they are random length it does not matter.

Are there good ways to find out the best values of heaplimit and
heapblock for a given program? Or at least some rules of thumb?

Is it allowed to temporary change those values, like

function OneSeg(int size): pointer;
 var
  p      : pointer;
  oldlim : word;
 begin
  oldlim:=heaplimit;
  heaplimit:=1; { force allocation of extra segment }
  MemAlloc(p, 1);
  heaplimit:=oldlim;
  OneSeg:=p
 end;

Re:BP7 Heap Manager patch?


In article <36E68B63.1FD8A...@physik.tu-muenchen.de>,
Christopher Eltschka  <celts...@physik.tu-muenchen.de> wrote:

Quote
>Osmo Ronkanen wrote:

>[...]

>> This has to do how the PM heap manager works. If the allocation is large
>> it allocates a new segment. However, if the allocation is small,
>> it allocates (if necessary) a large block and then suballocates
>> the request from that block. The threshold between the large and small
>> block is controlled by variable HeapLimit and the size of the large
>> block that is allocated by variable HeapBlock. Now the default for
>> HeapBlock is 8192 and the default value for HeapLimit is, surprise,
>> surprise, 1024.

>> So add the following lines at the beginning:

>>   heaplimit:=4096;
>>   heapblock:=32768+12;

>> The 12 comes from the observation that the first suballocation has
>> offset 12, so there will be no waste when the allocations are exactly
>> 1024 bytes. If they are random length it does not matter.

>Are there good ways to find out the best values of heaplimit and
>heapblock for a given program? Or at least some rules of thumb?

If the program allocates relatively small fixed size blocks, like 1024K,
make sure that heaplimit is larger than that. In the worst case one can
use about 3000*heaplimit bytes. I think you could keep heapblock close
to 64KB. Then heaplimit could vary according to available memory. If
there is little memory, keep it smaller, on larger ones keep it larger.

Quote
>Is it allowed to temporary change those values, like

>function OneSeg(int size): pointer;
> var
>  p      : pointer;
>  oldlim : word;
> begin
>  oldlim:=heaplimit;
>  heaplimit:=1; { force allocation of extra segment }
>  MemAlloc(p, 1);
>  heaplimit:=oldlim;
>  OneSeg:=p
> end;

I do not know. If it was done properly that would not be a problem.
Especially in the way you use I cannot see why there would be a
problem. I could understand that if the designers had been drunk and
they used the current value of heapblock as the size of every block
allocated for small allocations instead of storing the size then there
could be some problems.

IMO it would be a serious bug to require that they cannot be changed.
This because one could always accidentally use a unit that could do some
memory allocations before the

Osmo

Re:BP7 Heap Manager patch?


In article <36E39581.5...@csolve.net>,
        Mike Monett <a...@csolve.net> wrote:

Quote
>Scott Earnest wrote:

>> netn...@altavista.net wrote:

>> > [snip]

>> > BP7 protected mode.

>> > Appended below is a small code which illustrates the
>> > problem.  The Heap Manager runs out of pointers before
>> > all the memory is allocated. (I have 64Meg).
>> > OTOH, If you set mem=32*1024,
>> > it allocates all the memory (requires less pointers).
>> > [code snipped]

>> This isn't a bug in BP7.  This is a condition imposed by protected mode.  When
>> you allocate a pointer, DPMI provides a selector value which it uses
>> internally to determine where that block of memory is located and how it
>> should be treated.  (Working from memory) 16-bit protected mode allocates 64K
>> for descriptors, where each is 8 bytes, which allows for a total of 8192
>> maximum in theory.  Some are allocated by the system and by the RTL, reducing
>> that number; and if you run under Windows, it probably also swallows up a
>> bunch of them, reducing the total amount.

>> So, if you keep allocating, eventually you *are* going to run out of
>> selectors.

>So what's the solution? Can you avoid using Borland's heap manager, and
>take care of memory management yourself?

>How about allocating a block of memory, then using linear addressing
>instead of pointers. Each linear address is 4 bytes, allowing twice as
>many items. You could start at the top of the block and work down, and
>put the data at the bottom of the block and work up. The available memory
>is the difference between the last two entries minus 4 bytes.

>Best Regards,

>Michael R. Monett,
>Automated Production Test
>mailto:a...@csolve.net

Hello Mike,

That is one way.  The method is called suballocation.  I used a variation on
your idea for creating a string space handler for the compiler I am writing.

Basiclly, I created an array [1..10] type of type String.  I now allocate heap
space for this array type using New.  Now, as I need to allocate a string
I suballocate one of the 10 elements of this array.  When this array is
full at the time I need a new string, I just New another array.  (This
explanation is somewhat simplified, because I have to keep track of
each array, but you can figure that part out.)

Also, in order to help avoid the proverbial brick wall, I create a list of
available space.  Each time I am done with one of the above mentioned
arrays, instead of disposing it, I put it on the list of available space.  Now,
when I need to allocate a new array, before using New, I check if there is
an array on the list of available space.  I use mark and release to take care
of the heap.  Of course, there is a down-side to this approach.  It takes
a little more time and processing to make it work, and does tie up memory
in the availible space list, but this avoids the allocation limt and helps
avoid the heap crashing due to fragmention.

Hope this helps,
Charles

Charles          cbr...@ix.netcom.com
"For God So Loved The World, That He Gave His
 Only Begotten Son, That Whosoever Believeth
 In Him Should Not Perish, But Have Everlasting
 Life"John3:16  * http://pw2.netcom.com/~cbrtjr/wrdthing.html *

Re:BP7 Heap Manager patch?


In article <7cfl64$...@dfw-ixnews3.ix.netcom.com>,
  Charles E. Bortle, Jr. <cbr...@ix.netcom.com> wrote:

Quote
> In article <36E39581.5...@csolve.net>,
>    Mike Monett <a...@csolve.net> wrote:
> >Scott Earnest wrote:

> >> netn...@altavista.net wrote:

> >> > [snip]

> >> > BP7 protected mode.

> >> > Appended below is a small code which illustrates the
> >> > problem.  The Heap Manager runs out of pointers before
> >> > all the memory is allocated. (I have 64Meg).
> >> > OTOH, If you set mem=32*1024,
> >> > it allocates all the memory (requires less pointers).
> >> > [code snipped]

> >> This isn't a bug in BP7.  This is a condition imposed by protected mode.
When
> >> you allocate a pointer, DPMI provides a selector value which it uses
> >> internally to determine where that block of memory is located and how it
> >> should be treated.  (Working from memory) 16-bit protected mode allocates
64K
> >> for descriptors, where each is 8 bytes, which allows for a total of 8192
> >> maximum in theory.  Some are allocated by the system and by the RTL,
reducing
> >> that number; and if you run under Windows, it probably also swallows up a
> >> bunch of them, reducing the total amount.

> >> So, if you keep allocating, eventually you *are* going to run out of
> >> selectors.

> >So what's the solution? Can you avoid using Borland's heap manager, and
> >take care of memory management yourself?

> >How about allocating a block of memory, then using linear addressing
> >instead of pointers.

Because BP7/DPMI doesn't work that way.  The max chunk of memory you can
allocate at once is 64K.  If you allocate 2 64K blocks one right after the
other, you cannot assume they are physically contiguous.  That is to say, you
cannot take the pointer from the first allocation and perform pointer
arithmetic on it to form a pointer to memory in the second 64K block.  You
have to use the pointers as they are assigned.

Take the following real-life example.   I want to create a data structure in
memory for the lower-triangular part of a 4000x4000 array of floating point
numbers:

type
float = double;
row = array[1..4000] of float;
ary = array[1..4000] of ^row;
aptr = ^ary;

var
n : aptr;
i : word;

begin

new(n);

for i:=1 to 4000 do getmem(n^[i],i*sizeof(float));

end.

Even though I may have plenty of free memory, the above code crashes because
there are not enough pointers; e.g. Getmem returns nil even though there is
plenty of memory left.

Now, I COULD calculate how much memory I need:

8*(4000*4000 + 4000)/2 bytes

...and then allocate 64K chunks to cover it, which would require far fewer
pointers.  But then I would have to write a couple of kludgy procedures
GetN(i,j) and PutN(i,j) (instead of an array N) to access the data, but that
would be far too slow (and ugly).

There does not seem to be any way around the limitation.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re:BP7 Heap Manager patch?


Charles, E., Bortle, Jr. wrote:

[...]

Quote
> Hello Mike,

> That is one way.  The method is called suballocation.  I used a variation on
> your idea for creating a string space handler for the compiler I am writing.

> Basiclly, I created an array [1..10] type of type String.  I now allocate heap
> space for this array type using New.  Now, as I need to allocate a string
> I suballocate one of the 10 elements of this array.  When this array is
> full at the time I need a new string, I just New another array.  (This
> explanation is somewhat simplified, because I have to keep track of
> each array, but you can figure that part out.)

> Also, in order to help avoid the proverbial brick wall, I create a list of
> available space.  Each time I am done with one of the above mentioned
> arrays, instead of disposing it, I put it on the list of available space.  Now,
> when I need to allocate a new array, before using New, I check if there is
> an array on the list of available space.  I use mark and release to take care
> of the heap.  Of course, there is a down-side to this approach.  It takes
> a little more time and processing to make it work, and does tie up memory
> in the availible space list, but this avoids the allocation limt and helps
> avoid the heap crashing due to fragmention.

> Hope this helps,
> Charles

  Hi Charles,

  I stay  away from pointers and don't use the heap. I simply  use int
  21, function  48h to allocate memory, and function 49h  to  free the
  allocated memory.

  This way,  I get all the available DOS memory and can do  anything I
  want with  it. I don't have to release memory when it  is  no longer
  needed - I just load the new data over the old.

  Simple arrays have a single longint as an entry point to  tell where
  the array  starts. If I need to move the array, the only  thing that
  needs updating is the base entry point.

  I never  have problems with the old 64k limit. The arrays can  be as
  big as available DOS memory.

  It is  very easy to keep track of the memory remaining.  It  is also
  easy to check if enough memory is available to load the next item in
  an array.

  I never  have the problems I used to have with pointers. I  find the
  code very easy to debug and understand years after it was written. I
  cannot do that with pointers.

  Also, I  find the time needed to access memory is a very  small part
  of the time needed by most routines. I have not compared this method
  with code using Borland's heap manager, simply because  memory acces
  turns out to be insignificant.

  I don't know why more programmer's don't use this method.

Best Regards,

Michael R. Monett,
Automated Production Test
mailto:a...@csolve.net

Other Threads