Board index » delphi » Couple of questions...kinda long

Couple of questions...kinda long

Hi,

I have a couple of problems with an app I am working on.

This app reads text files and displays the information in a ListView.
The text files are in a simple format; they are names of people all on a
single line by themselves.

Problem number one is that the text files I need to read are up to 10MB
in size and take FOREVER to read into the ListView. For instance, a 7MB
text file takes almost 2 hours to load using the following code:

procedure TheList;
var
  iListItems:  Integer;
  ListItems:   TListItem;
begin
  Addresses := TStringList.Create;
  Addresses.LoadFromFile(fAddressList);
  Addresses.Sorted := TRUE;
  iListItems := Addresses.Count;

//I know I can use 'with' to eliminate the
//Form1.blah blah...  but i am too lazy
  Form1.ListView1.Items.BeginUpdate;
    Form1.ListView1.Items.Clear;
  Form1.ListView1.Items.EndUpdate;

  i := iListItems;
  Prog := 0;

  Form1.Progress.Min := 0;
  Form1.Progress.Max := i;

  Form1.ListView1.Items.BeginUpdate;
  while (Prog < i) do
  begin
    repeat
     Application.ProcessMessages;
     ListItems := Form1.ListView1.Items.Add;
       ListItems.Caption := IntToStr(Prog);
       ListItems.SubItems.Add(Addresses.Strings[Prog]);

      if (i > Prog) then
      begin
      inc(Prog);
        Application.ProcessMessages;
        Form1.Progress.Step := Form1.Progress.Max div i;
        Form1.Progress.StepIt;
        Form1.Label1.Caption := IntToStr(Form1.Progress.Position * 100
div i) + '% complete';
       end
      else
     Form1.Progress.Step := Form1.Progress.Max;
    until Prog = i;
  end;
end;

This works, but is EXTREMELY slow. I thought about using streams, but
was not sure if that would be any faster. That and I do not know how to
work with streams yet.

My second problem here is the ListView. It is a multi-select ListView.
Reason; so the user can highlight the names they want, and delete them
from the list. Not from the file, only from the list. Here is what I am
using to attempt this, but it ONLY removes the top name in the list for
some odd reason.

procedure TForm1.btnClearClick(Sender: TObject);
var
  iSelected:  Integer;
begin
  for iSelected := 0 to ListView1.Items.Count -1 do
    if ListView1.Items.Item[iSelected].Selected then
      ListView1.Items.Delete(iSelected);
//Boolean value letting me know that names were removed
  ItemsRemoved := TRUE;
end;

Heh, in my own world...  this *should* work. But, it obviously does not.
To the trained eye, it is probably very bad coding. Perhaps you can see
what I am doing wrong.

Last question...

I looked and looked. I even went to several web sites (delphi3000,
Torry, programmersheaven, etc.) but could NOT find a way to track
elapsed time and display it on a label in the format '00:00:00'  for
some reason. I think this should be easy to do...  but it is completely
escaping me.

Anyway..  if you can help, thanks very much. Even if you cannot help,
thanks for at least reading this lengthy post.

Thanks again, in advance...

Ryan

 

Re:Couple of questions...kinda long


Quote
Ryan Styles wrote in message

<1tl32uc91b75hdi7fo1ec7ebja4rs5t...@4ax.com>...
Quote
>Hi,

>I have a couple of problems with an app I am working on.

>This app reads text files and displays the information in a ListView.
>The text files are in a simple format; they are names of people all on a
>single line by themselves.

>Problem number one is that the text files I need to read are up to 10MB
>in size and take FOREVER to read into the ListView. For instance, a 7MB
>text file takes almost 2 hours to load using the following code:

<code snipped>

I do not see offhand what's wrong with it. That's one thing
wrong with it: I don't understand it. Perhaps if you explained
what it's supposed to do.

Quote
>This works, but is EXTREMELY slow. I thought about using streams, but
>was not sure if that would be any faster. That and I do not know how to
>work with streams yet.

>My second problem here is the ListView. It is a multi-select ListView.
>Reason; so the user can highlight the names they want, and delete them
>from the list. Not from the file, only from the list. Here is what I am
>using to attempt this, but it ONLY removes the top name in the list for
>some odd reason.

>procedure TForm1.btnClearClick(Sender: TObject);
>var
>  iSelected:  Integer;
>begin
>  for iSelected := 0 to ListView1.Items.Count -1 do
>    if ListView1.Items.Item[iSelected].Selected then
>      ListView1.Items.Delete(iSelected);
>//Boolean value letting me know that names were removed
>  ItemsRemoved := TRUE;
>end;

>Heh, in my own world...  this *should* work. But, it obviously does not.
>To the trained eye, it is probably very bad coding. Perhaps you can see
>what I am doing wrong.

This is an old one. When you delete an item, the ones
after it are move up a spot. You should either count
down, or loop over an if statement that either removes
an item or increments the counter. If the latter is
not clear, this is what I mean:

i:=0; n:=a.count;
while (i<n)
do begin
  if (a[i].selected)
  then a.remove(i)
  else inc(i);
end;

Quote
>Last question...

>I looked and looked. I even went to several web sites (delphi3000,
>Torry, programmersheaven, etc.) but could NOT find a way to track
>elapsed time and display it on a label in the format '00:00:00'  for
>some reason. I think this should be easy to do...  but it is completely
>escaping me.

Elapsed:=Now-StartTime; { in days }
Days:=Trunc(Elapsed);
Elapsed:=Elapsed*24; Hours:=Trunc(Elapsed); Elapsed:=Frac(Elapsed); { in
hours }
Elapsed:=Elapsed*60; Minutes:=Trunc(Elapsed); Elapsed:=Frac(Elapsed); { in
minutes }
Elapsed:=Elapsed*60; Seconds:=Trunc(Elapsed); Elapsed:=Frac(Elapsed); { in
seconds }
Format('%02.2d:%02.2d:%02.2d', [Hours, Minutes, Seconds]); { spot the
omission }

Groetjes,
Maarten Wiltink

Re:Couple of questions...kinda long


"Maarten Wiltink" <maar...@kittensandcats.net> skrev i melding
news:9vspk4$g37$1@news1.xs4all.nl...

Quote
> Ryan Styles wrote in message
> <1tl32uc91b75hdi7fo1ec7ebja4rs5t...@4ax.com>...
> >This works, but is EXTREMELY slow. I thought about using streams, but
> >was not sure if that would be any faster. That and I do not know how to
> >work with streams yet.

> >My second problem here is the ListView. It is a multi-select ListView.
> >Reason; so the user can highlight the names they want, and delete them
> >from the list. Not from the file, only from the list. Here is what I am
> >using to attempt this, but it ONLY removes the top name in the list for
> >some odd reason.

> >procedure TForm1.btnClearClick(Sender: TObject);
> >var
> >  iSelected:  Integer;
> >begin
> >  for iSelected := 0 to ListView1.Items.Count -1 do
> >    if ListView1.Items.Item[iSelected].Selected then
> >      ListView1.Items.Delete(iSelected);
> >//Boolean value letting me know that names were removed
> >  ItemsRemoved := TRUE;
> >end;

> >Heh, in my own world...  this *should* work. But, it obviously does not.
> >To the trained eye, it is probably very bad coding. Perhaps you can see
> >what I am doing wrong.

> This is an old one. When you delete an item, the ones
> after it are move up a spot. You should either count
> down, or loop over an if statement that either removes
> an item or increments the counter. If the latter is
> not clear, this is what I mean:

> i:=0; n:=a.count;
> while (i<n)
> do begin
>   if (a[i].selected)
>   then a.remove(i)
>   else inc(i);
> end;

ListView1.Items.BeginUpdate;
// delete them
ListView1.Items.EndUpdate;

..will speed things up, as no visual representation of the ongoing process
needs to take place.

--
Bjoerge Saether
Asker, Norway
bjorge@hahaha_itte.no (remve the obvious)

Re:Couple of questions...kinda long


You have a huge text file - well 10mb

You want to display it in sorted order.

Personally I would make a sorted copy of the file on disk
For a 10mb file this takes about 40 Secs

www.iss.u-net.com/dsortv.htm

I would rapidly scan the file to make an Index of start and end of
each text line so that I could treat the file as random access.

I would then look at creating a Virtual Listbox that just displays the
lines in view on the screen

Quote
On Thu, 20 Dec 2001 12:36:28 GMT, Ryan Styles <r...@r.ru> wrote:
>Hi,

>I have a couple of problems with an app I am working on.

>This app reads text files and displays the information in a ListView.
>The text files are in a simple format; they are names of people all on a
>single line by themselves.

>Problem number one is that the text files I need to read are up to 10MB
>in size and take FOREVER to read into the ListView. For instance, a 7MB
>text file takes almost 2 hours to load using the following code:

>procedure TheList;
>var
>  iListItems:  Integer;
>  ListItems:   TListItem;
>begin
>  Addresses := TStringList.Create;
>  Addresses.LoadFromFile(fAddressList);
>  Addresses.Sorted := TRUE;
>  iListItems := Addresses.Count;

>//I know I can use 'with' to eliminate the
>//Form1.blah blah...  but i am too lazy
>  Form1.ListView1.Items.BeginUpdate;
>    Form1.ListView1.Items.Clear;
>  Form1.ListView1.Items.EndUpdate;

>  i := iListItems;
>  Prog := 0;

>  Form1.Progress.Min := 0;
>  Form1.Progress.Max := i;

>  Form1.ListView1.Items.BeginUpdate;
>  while (Prog < i) do
>  begin
>    repeat
>     Application.ProcessMessages;
>     ListItems := Form1.ListView1.Items.Add;
>       ListItems.Caption := IntToStr(Prog);
>       ListItems.SubItems.Add(Addresses.Strings[Prog]);

>      if (i > Prog) then
>      begin
>      inc(Prog);
>        Application.ProcessMessages;
>        Form1.Progress.Step := Form1.Progress.Max div i;
>        Form1.Progress.StepIt;
>        Form1.Label1.Caption := IntToStr(Form1.Progress.Position * 100
>div i) + '% complete';
>       end
>      else
>     Form1.Progress.Step := Form1.Progress.Max;
>    until Prog = i;
>  end;
>end;

>This works, but is EXTREMELY slow. I thought about using streams, but
>was not sure if that would be any faster. That and I do not know how to
>work with streams yet.

>My second problem here is the ListView. It is a multi-select ListView.
>Reason; so the user can highlight the names they want, and delete them
>from the list. Not from the file, only from the list. Here is what I am
>using to attempt this, but it ONLY removes the top name in the list for
>some odd reason.

>procedure TForm1.btnClearClick(Sender: TObject);
>var
>  iSelected:  Integer;
>begin
>  for iSelected := 0 to ListView1.Items.Count -1 do
>    if ListView1.Items.Item[iSelected].Selected then
>      ListView1.Items.Delete(iSelected);
>//Boolean value letting me know that names were removed
>  ItemsRemoved := TRUE;
>end;

>Heh, in my own world...  this *should* work. But, it obviously does not.
>To the trained eye, it is probably very bad coding. Perhaps you can see
>what I am doing wrong.

>Last question...

>I looked and looked. I even went to several web sites (delphi3000,
>Torry, programmersheaven, etc.) but could NOT find a way to track
>elapsed time and display it on a label in the format '00:00:00'  for
>some reason. I think this should be easy to do...  but it is completely
>escaping me.

>Anyway..  if you can help, thanks very much. Even if you cannot help,
>thanks for at least reading this lengthy post.

>Thanks again, in advance...

>Ryan

Re:Couple of questions...kinda long


Quote
> I have a couple of problems with an app I am working on.

> This app reads text files and displays the information in a ListView.
> The text files are in a simple format; they are names of people all on a
> single line by themselves.

> Problem number one is that the text files I need to read are up to 10MB
> in size and take FOREVER to read into the ListView. For instance, a 7MB
> text file takes almost 2 hours to load using the following code:

You're addressing the wrong problem.  You  probably should  rethink the
design - 10mb worth of names would be somewhere between 100K and a million
names.  Is anyone really going to browse this list manually and maintain it?
Based on what criteria?  Get the criteria somehow and at least partially
automate the process, maybe presenting your proposed changes for
edit/approval.  Or subdivide the file by first letter, count or something.
That way several people could work on smaller files at the same time.
Database browsers avoid the problem  by loading only a page full at a time -
and text files aren't that hard to put into a database.   Or any other
approach that avoids trying to display 10mb of data.

Just my opinion.

_________________________
Gary
http://www.delphiforfun.org
_________________________

Re:Couple of questions...kinda long


Quote
Ryan Styles <r...@r.ru> wrote in message <news:1tl32uc91b75hdi7fo1ec7ebja4rs5t98e@4ax.com>...

Hello again, Ryan.

Quote
> My second problem here is the ListView. It is a multi-select ListView.
> Reason; so the user can highlight the names they want, and delete them
> from the list. Not from the file, only from the list. Here is what I am
> using to attempt this, but it ONLY removes the top name in the list for
> some odd reason.

you can do it like this:
procedure TForm1.btnDelSelectedClick(Sender: TObject);
begin
  while (ListView.SelCount > 0) do
   ListView.Items.Delete(ListView.Items.IndexOf(ListView.Selected));
end;

Quote

> I looked and looked. I even went to several web sites (delphi3000,
> Torry, programmersheaven, etc.) but could NOT find a way to track
> elapsed time and display it on a label in the format '00:00:00'  for
> some reason. I think this should be easy to do...  but it is completely
> escaping me.

procedure TForm1.btnStartProcessClick(Sender: TObject);
begin
  StartTickCount := GetTickCount;
  while true do
    UpdateElapsedTime;
end;

procedure TForm1.UpdateElapsedTime;
var
  ATickCount     : Cardinal;
  hh, mm, ss, ms : Word;
begin
  ATickCount := GetTickCount - StartTickCount;
  ms := (ATickCount mod 1000);
  ATickCount := Trunc(ATickCount / 1000);
  ss := (ATickCount mod 60);
  ATickCount := Trunc(ATickCount / 60);
  mm := (ATickCount mod 60);
  ATickCount := Trunc(ATickCount / 60);
  hh := (ATickCount mod 60);
  ATickCount := Trunc(ATickCount / 60);
  Label1.Caption := IntToStr(hh) + ':' + IntToStr(mm) + ':' +
IntToStr(ss) + ':' + IntToStr(ms);
  Application.ProcessMessages;
end;

Good luck...

--- markus.
[ Old programmers never die. They just terminate and stay resident]

Re:Couple of questions...kinda long


For the Elapsed Time problem try this - TDateTime is realy a Double.

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms,
Dialogs,
  StdCtrls;

type
  TForm1 = class(TForm)
    Button1: TButton;
    Label1: TLabel;
    procedure Button1Click(Sender: TObject);
  private
    { Private declarations }
    OldTime : TDateTime ;
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.DFM}

procedure TForm1.Button1Click(Sender: TObject);
   Var
   Elapsed : TDateTime ;
begin
   If OldTime = 0 Then
      OldTime := Now
   Else
      Begin
        Elapsed := Now - OldTime ;
        Label1.Caption := TimeToStr( Elapsed ) ;
      End;

end;

end.

Re:Couple of questions...kinda long


Just wanted to say thanks for all of the help from everyone on the
elapsed time issue. All of the suggestions really helped me out.

Ryan

Re:Couple of questions...kinda long


Quote
"Ryan Styles" <r...@r.ru> wrote in message

news:1tl32uc91b75hdi7fo1ec7ebja4rs5t98e@4ax.com...

Quote
> Hi,

> I have a couple of problems with an app I am working on.

> This app reads text files and displays the information in a ListView.
> The text files are in a simple format; they are names of people all on a
> single line by themselves.

> Problem number one is that the text files I need to read are up to 10MB
> in size and take FOREVER to read into the ListView. For instance, a 7MB
> text file takes almost 2 hours to load using the following code:

A stringlist is not very efficient for storing and sorting large numbers of
strings.
Loading a stringlist one at a time can have severe delays if it runs out of
physical ram and
starts disk thrashing.
Listviews takes ages to update for more than a few thousand items.

Assuming you are using report then use a virtual listview.

var sl1:Tstringlist;

procedure TForm1.ListView1Data(Sender: TObject; Item: TListItem);
begin
   //called for every mousemove and column update
   if sl1 <> nil then
   begin
     if item.index < sl1.Count then
      item.caption:=inttostr(item.index)+' '+sl1.Strings[item.index]
     else
       item.caption:='unfound '+inttostr(item.index);
   end;
end;

procedure TForm1.LoadandsortClick(Sender: TObject);
var tstart,tend:cardinal;
    NewColumn: TListColumn;
begin
  freemem(allocmem(20000000)); // clear some RAM space
  sl1:=tstringlist.create;
  sl1.capacity:=1000000;//preallocate for count of strings indexes
  tstart:=gettickcount;
  sl1.LoadFromFile('10megofstrings.txt');
  tend:=gettickcount;
  memo1.Lines.add('load elapsed ms='+inttostr(tend-tstart));
 // ~2 seconds(varies by disk io)
  beep;
  tstart:=gettickcount;
  sl1.Sorted:=true;
  tend:=gettickcount;
  memo1.Lines.add('sort elapsed (ms)='+inttostr(tend-tstart));
 //~60 seconds (varies by cpu and virtual memory thrashing)
  beep;
  with listview1 do
  begin
    ViewStyle := vsReport;
    NewColumn := Columns.Add;
    NewColumn.Caption := 'Whatever';
    items.count:=sl1.Count;
   end;
 application.processmessages;
end;

Other Threads