Board index » cppbuilder » ProcessMessages() in a loop

2005-03-04 01:37:06 AM
cppbuilder19
Hi,
I have a database (with data, of course) which structure
was not adequate to me. So I decided to transform it's
data in the way I want. The database itself is pretty big
(for MS Access ;) - a couple of tables every with about
400,000 records.
I haven't used the standard DB mechanisms for seraching
for matches (Lookup, Locate), because I found working with
TStringList-s signifficantly faster.
Then, I made a utility which:
1) transfers the data to TStringList-s
2) processes that TStringList-s
3) transfers data from TStringList-s back to DB.
The problem is that utility worked 98% correct which
is rediculous. The 2% of data was processed wrong.
I'm wondering if such code skeleton can in some
circumstances become bad, because of ProcessMessages(),
or something else:
for ( int i=0; i<400000; i++ )
{
Application->ProcessMessages();
// Here, I had few TStringList objects' manipulations
// on very big (400,000 items) objects of TStringList
// like SomeIndex = SL1->IndexOf( ... ),
// SL2->Strings[ SomeIndex ] = ...,
// etc, ...
}
At the beginning, loop processed about 2000 recors / second
but as the destination TStringList become bigger, it processed
only a few records / second (because IndexOf() now had more
job to do)
Once, the destination TStringList was full of duplicated entries,
like:'sometextsometext' instead of 'sometext'. Sometimes, the
destination TStringList was short for maybe of 1000 consecutive
records. There were also some other misterious results.
Can ProcessMessages() be dangerous in this case? Should I use
CriticalSections in this transformations, like:
for ( int i=0; i<400000; i++ )
{
Application->ProcessMessages();
EnterCS
Processing ...
LeaveCS
}
--
Best regards,
Vladimir Stefanovic
 

A few questions:
1. Is your loop below being performed in a message handler on the main
thread, or on another thread?
2. Using TStringList for this is a bad idea; its results will gradually
increase as the size of the list grows because it's doing a sequential
search through the data. I would consider using the STL map class, with
the first part of the template being an AnsiString and the second part
being a pointer to the record.
3. I must be missing something, what is TStringList-s as opposed to
TStringList?
4. What else is happening in that loop?
Vladimir Stefanovic wrote:

can be changed in two ways:
1) dest_string_list->Add( Something )
2) dest_string_list->Strings[ SomePosition ] =
destination_string_list->Strings[ SomePosition ] + ";" + Addition;
... so it has to search the whole string list object sequential.
not of any future importance, so I wanted something simple.
TStringList is involved in processing.
and I reactrivate it, the loop stopped working.
--
Best regards,
Vladimir Stefanovic
 

{smallsort}

Vladimir Stefanovic wrote:
of your algorithm.
question: remove the Processmessages statement and see if
the result is 100%.
Working with TStringList with a large amount of data is quite
possible. But yes: avoid IndexOf() as it might need more time
to find the strings at the end.
By the way. Is this the algoritm we discused lately for your
dictionary ?
Hans.
 

Hi Hans!
still there.
the techniques (IndexOf, Strings[]) I used.
right now. The utilitiy is just now working at my work place, and
I'll check what weird has happend tomorrow morning.
BTW, The procedure which I mentioned then had not string lists
involved, and even then there was something weird:
The complete wordlist beginning with letter 'W' was somehow
missing ?! And I had *NO CODE* which checked anything
similar. The algorythm was general for all words.
--
Best regards,
Vladimir Stefanovic
 

Vladimir,
I would look for a pointer problem. Can you explain how the transfer is
going wrong? Can you enable code guard? Please show the code.
John
"Vladimir Stefanovic" < XXXX@XXXXX.COM >wrote in message