Board index » cppbuilder » Sorting a virtual TListView and caching

Sorting a virtual TListView and caching


2006-03-09 05:47:51 PM
cppbuilder46
I need to optimise the memory usage of a file viewing application. At
present we use a virtual TListView but we load all the information for
all the objects at the start into our own random access structure.
The problem is that we now have a need to reduce the memory footprint
of this viewer (basically because we want to populate it with 2.5
million objects, lol). I can fairly easily change things so that the
data is only loaded when it's needed but presumably this will cripple
the performance of OnCompare. It's going to run slow enough as it is
without adding the overhead of continually loading/unloading a couple
of hundred bytes of data per object.
Does anyone think there would be useful milleage in providing
rudimentary caching of object details? Any other hints?
--
Andrue Cope [TeamB]
[Bicester, Uk]
info.borland.com/newsgroups/guide.html
 
 

Re:Sorting a virtual TListView and caching

"Andrue Cope [TeamB]" < XXXX@XXXXX.COM >wrote in message
Quote
The problem is that we now have a need to reduce the memory
footprint of this viewer (basically because we want to populate it
with 2.5 million objects, lol).
I have an application that does exactly that. It is a log file viewer, and
has to be able to display up to millions of records at one time, complete
with full sorting and filtering (the user can choose which types of records
to display).
Quote
I can fairly easily change things so that the data is only loaded when
it's needed but presumably this will cripple the performance of OnCompare.
Why are you using the OnCompare event in a virtual TListView? You should be
doing all of your operations on the physical data directly, not through
TListItem.
Quote
Does anyone think there would be useful milleage in providing
rudimentary caching of object details? Any other hints?
In my app, I keep a cache of about 200 records in memory at any given time.
This way, there is enough data for both the visual display, and the
surrounding areas above and below the visual area. When the user scrolls
the display, the OnDataHint event tells me the indexes of the items that are
likely to be displayed soon, and the cache is updated accordingly. For
small movements, the display is quite fast since the cache is not changed
very much or very often. However, if the user presses the PGUP, PGDOWN, or
END keys, the cache management gets a little bit more complicated
(particularly for me, since I have to filter the records before they are
placed in the cache, as well as when calculating the number of total items
that need to be displayed)
In your case, I don't know if you are doing any filtering or the like. If
you list item indexes coorespond to physical file offsets (such as the file
having fixed-length structures for each item), then maintaining a cahce is
very straight-forward. In the OnDataHint event, take the provided indexes,
convert them into file offsets, and then read the file data into the cache
as needed. You should also have the cache keep track of the indexes that it
actually contains, so that you can compare them to the OnHintData indexes,
to minimize unnecessary loading/unloading operations if the items are
already in the cache.
Gambit
 

Re:Sorting a virtual TListView and caching

Remy Lebeau (TeamB) wrote:
Quote
>I can fairly easily change things so that the data is only loaded
>when it's needed but presumably this will cripple the performance
>of OnCompare.

Why are you using the OnCompare event in a virtual TListView? You
should be doing all of your operations on the physical data directly,
not through TListItem.
Historical reasons. Yesterdays's investigations had highlighted that as
an area that could be improved :)
Quote
However, if the user presses the PGUP, PGDOWN, or
END keys, the cache management gets a little bit more complicated
(particularly for me, since I have to filter the records before they
are placed in the cache, as well as when calculating the number of
total items that need to be displayed)
We do have filtering but I'm pretty sure I can place the cache after
that point because the filtering is not something that the user changes
on the fly (it's kind of a global filter). As long as my cache is
transparent (misses get services correctly anyway) I should be fine.
--
Andrue Cope [TeamB]
[Bicester, Uk]
info.borland.com/newsgroups/guide.html
 

{smallsort}

Re:Sorting a virtual TListView and caching

"Andrue Cope [TeamB]" < XXXX@XXXXX.COM >wrote in message
Quote
We do have filtering but I'm pretty sure I can place the cache after
that point because the filtering is not something that the user changes
on the fly (it's kind of a global filter). As long as my cache is
transparent (misses get services correctly anyway) I should be fine.
If your cache is large enough to hold all of the items that the ListView
displays, then handling the PGUP, PGDOWN, and END keys is a no-brianer,
since the OnHintData index will coorespond to items that are already in the
cache. I don't know how you manage your filtering, but if the cache does
not have the items yet when the TListView needs them, then you will likely
have to start at the beginning of your source data, run through the items
one at a time filtering them into your cache as needed until the indexes are
satisfied. I haven't been able to find a better way to handle it yet
(although I am experimenting with writing a custom TDataSet descendant so
that I can use a TDBGrid instead of a virtual TListView).
Gambit
 

Re:Sorting a virtual TListView and caching

Remy Lebeau (TeamB) wrote:
Quote
I haven't been able to find a better way to handle it yet
(although I am experimenting with writing a custom TDataSet descendant so
that I can use a TDBGrid instead of a virtual TListView).
What about TDrawGrid ?
Jonathan
 

Re:Sorting a virtual TListView and caching

"Jonathan Benedicto" < XXXX@XXXXX.COM >wrote in message
Quote
What about TDrawGrid ?
The purpose of writting a custom TDataSet class is because the DB
architecture manages all of the buffering, filtering, and searching code
automatically (or with little manual effort). There is no point in
switching from TListView to TDrawGrid, as both would require the same
ammount of effort to manage everything manually.
Gambit
 

Re:Sorting a virtual TListView and caching

Remy Lebeau (TeamB) wrote:
Quote
The purpose of writting a custom TDataSet class is because the DB
architecture manages all of the buffering, filtering, and searching code
automatically (or with little manual effort). There is no point in
switching from TListView to TDrawGrid, as both would require the same
ammount of effort to manage everything manually.
Oh I see. Thank you for telling me.
Jonathan