Board index » delphi » TFileStream is so sloooooow

TFileStream is so sloooooow

Is it just me or is the TFileStream class unfeasibly slow.

I am writing a program that needs to extract data from a file 1 byte at a
time, perform a simple algorithm on it and then write it into another file.
The algorithm is not very complex and should cause the operation to take so
long, it takes several seconds to complete even an a small file that is only
a few K in size.

I am using the TFileStream class because my program could be expected to
have to deal with files of  >2Gb, I am using the form:

var
  SourceFile: TFileStream;
  DestFile: TFileStream;
  C: Byte;
begin
   repeat
        SourceFile := TFileStream.Create(sFilename,  fmOpenRead);
        DestFile := TFileStream.Create(sDestFile,  fmOpenWrite);
        // Do transfer
        SourceFile.ReadBuffer(C, 1);
        ......
        ........
        ........
        DestFile.WriteBuffer(C, 1);
   until SourceFile.Position = SourceFile.Size;
end;

Any comments are welcome.

Chris

 

Re:TFileStream is so sloooooow


Quote
Chris Morrison wrote:
> I am using the TFileStream class because my program could be expected to
> have to deal with files of  >2Gb, I am using the form:

If you're using D6, the TFileStream supports files larger than 2GB. If
you're using D5, then I suggest you consider using Stream64, by Tomas
Koutny. I'm using it on very large files, and the performance seems good.

http://www.rawos.com/qlinks/st64.htm

--
Bill
To review: A free browser is bad, but a free OS is good.....
Come again?
- Posted with XanaNews -

Re:TFileStream is so sloooooow


Try reading and writing larger buffers.  You should notice a large
performance improvement.
Jason Swager

Chris Morrison <[REMOVE]chris[SPAMBLOCK]-morrison@dsl[TO].pipex.com[REPLY]>
wrote in message news:3cfd150b_1@dnews...

Quote
> Is it just me or is the TFileStream class unfeasibly slow.

> I am writing a program that needs to extract data from a file 1 byte at a
> time, perform a simple algorithm on it and then write it into another
file.
> The algorithm is not very complex and should cause the operation to take
so
> long, it takes several seconds to complete even an a small file that is
only
> a few K in size.

> I am using the TFileStream class because my program could be expected to
> have to deal with files of  >2Gb, I am using the form:

> var
>   SourceFile: TFileStream;
>   DestFile: TFileStream;
>   C: Byte;
> begin
>    repeat
>         SourceFile := TFileStream.Create(sFilename,  fmOpenRead);
>         DestFile := TFileStream.Create(sDestFile,  fmOpenWrite);
>         // Do transfer
>         SourceFile.ReadBuffer(C, 1);
>         ......
>         ........
>         ........
>         DestFile.WriteBuffer(C, 1);
>    until SourceFile.Position = SourceFile.Size;
> end;

> Any comments are welcome.

> Chris

Re:TFileStream is so sloooooow


Quote
> Is it just me or is the TFileStream class unfeasibly slow.

> I am writing a program that needs to extract data from a file 1 byte at a
> time
>         SourceFile.ReadBuffer(C, 1);

Each call to ReadBuffer is causing a disk access.  You'd be better off
reading, say 8K, into a memory buffer and reading one byte at a time from
there.

-Mike

Re:TFileStream is so sloooooow


Quote
> Each call to ReadBuffer is causing a disk access.  You'd be better
off
> reading, say 8K, into a memory buffer and reading one byte at a time
from
> there.

I think this screams for an implemenation of a "buffered" stream: a
TStream descendant that can sit on top of another TStream object and
buffer reads and writes. I know BP7 had such an object. Does anyone
know where somebody an find one?

Re:TFileStream is so sloooooow


Quote
On Wed, 5 Jun 2002 12:24:11 +0300, "Constantine Yannakopoulos" <k...@singular.gr> wrote:
>> Each call to ReadBuffer is causing a disk access.  You'd be better
>off
>> reading, say 8K, into a memory buffer and reading one byte at a time
>from
>> there.

>I think this screams for an implemenation of a "buffered" stream: a
>TStream descendant that can sit on top of another TStream object and
>buffer reads and writes. I know BP7 had such an object. Does anyone
>know where somebody an find one?

See Julian Bucknall's article on TBufferedFileStream in issue 29 of The Delphi Magazine -
souce code from http://www.thedelphimagazine.com/disks/dmag29.zip

John Leavey

Re:TFileStream is so sloooooow


In article <3cfd1a53$1_2@dnews>, "Mike Williams" <mi...@remove.aps-soft.com>
wrote:

Quote
> Each call to ReadBuffer is causing a disk access.  You'd be better off
> reading, say 8K, into a memory buffer and reading one byte at a time from
> there.

I optimised a routine (reading a file, not a stream) recently, and found
that using a 1K buffer made a big difference, but increasing it to 8K did
not make any noticeable further improvement. FWIW.

-Iain.

Re:TFileStream is so sloooooow


Quote
> I optimised a routine (reading a file, not a stream) recently, and found
> that using a 1K buffer made a big difference, but increasing it to 8K did
> not make any noticeable further improvement. FWIW.

I think it depends on the hard drive / file system / cluster size /etc.  8K
might be more than needed in your situation.  However, unless memory usage
is a big concern, your app might perform better on another machine with an
8K buffer even though it makes no difference on your machine.

-Mike

Re:TFileStream is so sloooooow


Hi Chris,

Chris Morrison <[REMOVE]chris[SPAMBLOCK]-morrison@dsl[TO].pipex.com[REPLY]>
wrote in message 3cfd150b_1@dnews...

Quote
> Is it just me or is the TFileStream class unfeasibly slow.

> I am writing a program that needs to extract data from a file 1 byte at a
> time, perform a simple algorithm on it and then write it into another
file.
> The algorithm is not very complex and should cause the operation to take
so
> long, it takes several seconds to complete even an a small file that is
only
> a few K in size.

> I am using the TFileStream class because my program could be expected to
> have to deal with files of  >2Gb, I am using the form:

> var
>   SourceFile: TFileStream;
>   DestFile: TFileStream;
>   C: Byte;
> begin
>    repeat

are you sure it works as you like?While do you create SourceFile and
DestFile each time?
Another thing: at the end you have to free the two objects

By Katiuscia

- Show quoted text -

Quote
>         SourceFile := TFileStream.Create(sFilename,  fmOpenRead);
>         DestFile := TFileStream.Create(sDestFile,  fmOpenWrite);
>         // Do transfer
>         SourceFile.ReadBuffer(C, 1);
>         ......
>         ........
>         ........
>         DestFile.WriteBuffer(C, 1);
>    until SourceFile.Position = SourceFile.Size;
> end;

> Any comments are welcome.

> Chris

Re:TFileStream is so sloooooow


Chris,

You might want to try using memory-mapped files.  Deborah Pate has made
Delphi
interface:

  http://www.djpate.freeserve.co.uk/GetCode.htm

You should buffer your file operations to larger chunks than one byte if
possible - try 4096 bytes or larger if possible.

Cheers,
David

Quote
----- Original Message -----
From: "Chris Morrison"

<[REMOVE]chris[SPAMBLOCK]-morrison@dsl[TO].pipex.com[REPLY]>
Newsgroups: borland.public.delphi.vcl.components.using
Sent: Tuesday, June 04, 2002 8:29 PM
Subject: TFileStream is so sloooooow

> Is it just me or is the TFileStream class unfeasibly slow.

> I am writing a program that needs to extract data from a file 1 byte at
a
> time, perform a simple algorithm on it and then write it into another
file.
> The algorithm is not very complex and should cause the operation to take
so
> long, it takes several seconds to complete even an a small file that is
only
> a few K in size.

> I am using the TFileStream class because my program could be expected to
> have to deal with files of  >2Gb, I am using the form:

Re:TFileStream is so sloooooow


Chris,

Because your code is bad. <g> You're re-creating the Source
and Dest streams once for each byte of the file you read (because
it's inside the repeat loop.

Try this:

var
   SourceFile: TFileStream;
   DestFile: TFileStream;
   C: Byte;
begin
    SourceFile := TFileStream.Create(sFilename,  fmOpenRead);
    DestFile := TFileStream.Create(sDestFile,  fmOpenWrite);
    try
      repeat
         // Do transfer
         SourceFile.ReadBuffer(C, 1);
         ......
         ........
         ........
        DestFile.WriteBuffer(C, 1);
     until SourceFile.Position = SourceFile.Size;
  finally
    DestFile.Free;
    SourceFile.Free;
  end;
end;

Ken
---
Ken White
kwh...@adpsi.com

Clipper Functions for Delphi and C++ Builder
http://www.adpsi.com

"Chris Morrison"
<[REMOVE]chris[SPAMBLOCK]-morrison@dsl[TO].pipex.com[REPLY]> wrote in
message news:3cfd150b_1@dnews...

Quote
> Is it just me or is the TFileStream class unfeasibly slow.

> I am writing a program that needs to extract data from a file 1 byte at a
> time, perform a simple algorithm on it and then write it into another
file.
> The algorithm is not very complex and should cause the operation to take
so
> long, it takes several seconds to complete even an a small file that is
only
> a few K in size.

> I am using the TFileStream class because my program could be expected to
> have to deal with files of  >2Gb, I am using the form:

> var
>   SourceFile: TFileStream;
>   DestFile: TFileStream;
>   C: Byte;
> begin
>    repeat
>         SourceFile := TFileStream.Create(sFilename,  fmOpenRead);
>         DestFile := TFileStream.Create(sDestFile,  fmOpenWrite);
>         // Do transfer
>         SourceFile.ReadBuffer(C, 1);
>         ......
>         ........
>         ........
>         DestFile.WriteBuffer(C, 1);
>    until SourceFile.Position = SourceFile.Size;
> end;

> Any comments are welcome.

> Chris

Re:TFileStream is so sloooooow


You're re-creating the Source
Quote
> and Dest streams once for each byte of the file you read (because
> it's inside the repeat loop.

I typed this example in off the top of my head to demonstrate what I was
trying to achieve. That was a typing mistake and I apologise for any
confusion caused (well spotted).

I have to read the data, one byte at a time because my program has to deal
with variable amounts of data and files that may be split across an unknown
number of floppy diskettes. I'm also keeping a progress bar up to date in
real time as the data is read.

Thank you everyone for your help.

Chris

Re:TFileStream is so sloooooow


"Chris Morrison"

Quote
<[REMOVE]chris[SPAMBLOCK]-morrison@dsl[TO].pipex.com[REPLY]> wrote:

|I'm also keeping a progress bar up to date in
|real time as the data is read.

Do you update the progress bar for each byte?  If so, this may be your
problem.  One of the progress bars goes through a very time-consuming
update procedure for each update call, even if the update increment is
too small to make a visual change.  Try running without the progress
bar to see if this speeds things up.

Phil

Re:TFileStream is so sloooooow


I once made a parser for large SQL files.
This parsr used simple strings but I found an interesting effect using large files. I may be of use.

for example

procedure ParseFile(AFile: string);
var
  sl_File: TStringList;
  s1, s2: string;
  i: integer;
begin
  sl_File := TStringList.Create;
  try
    sl_File.LoadFromFile(AFile);
    s1 := sl_File.Text;

   for i := 1 to Length(s1) do
     s2 := s2 + s1[i];

   (* Better is
     SetLength(s2, Length(s1)) // <<-- makes a great difference!
     for i := 0 to Length(s1) do
       s2[i] := si[i];
   *)
  finally
    sl_File.Free;
  end;
end;

The performance speed was from minutes to seconds on large files ( > 1Mb)
The reason has everything to do with the way in wich memeory is allocated. Its more efficient to allocate all memory at once.

What I would do:
  Use a TMemoryStream to temporaly store the result.
  Try to set the size of this MemoryStream in advance if
  possible.
  Then write the result to file by copying the memoryStream to
  a TFileStream, in one operation, instead of per byte.
  You may get an even better result using a MemoryStream
  to store the initial stream as well.
  This means the read to file and write to file operations
  should be much faster.

Quote
"Chris Morrison" <[REMOVE]chris[SPAMBLOCK]-morrison@dsl[TO].pipex.com[REPLY]> wrote:
>Is it just me or is the TFileStream class unfeasibly slow.

>I am writing a program that needs to extract data from a file 1 byte at a
>time, perform a simple algorithm on it and then write it into another file.
>The algorithm is not very complex and should cause the operation to take so
>long, it takes several seconds to complete even an a small file that is only
>a few K in size.

>I am using the TFileStream class because my program could be expected to
>have to deal with files of  >2Gb, I am using the form:

>var
>  SourceFile: TFileStream;
>  DestFile: TFileStream;
>  C: Byte;
>begin
>   repeat
>        SourceFile := TFileStream.Create(sFilename,  fmOpenRead);
>        DestFile := TFileStream.Create(sDestFile,  fmOpenWrite);
>        // Do transfer
>        SourceFile.ReadBuffer(C, 1);
>        ......
>        ........
>        ........
>        DestFile.WriteBuffer(C, 1);
>   until SourceFile.Position = SourceFile.Size;
>end;

>Any comments are welcome.

>Chris

Re:TFileStream is so sloooooow


And that why two TMemoryStream should be used to temporaly store the input and the output.
This will mean that the file will have to be read and written only once.

var
  SourceFile: TFileStream;
  DestFile: TFileStream;
  m1,m2: TMemoryStream;
  C: Byte;
begin
  SourceFile := TFileStream.Create(sFilename,  fmOpenRead);
  m1 := TMemoryStrea,Create;
  m2 := TMemoryStream.Create;
  m1.LoadFromStream(SourceFile);
      repeat
         // Do transfer
        m1.ReadBuffer(C, 1);
         ......
         ........
         ........
        m2.WriteBuffer(C, 1);
     until m2.Position = m2.Size;
  DestFile := TFileStream.Create(sDestFile,  fmOpenWrite);
  DestFile.LoadFromStream(m2);

  // and finally free al streams!
end;

This is stille not efficient enough because m2 will grow one byte at a time. This means the heap will have to grow one byte at a time. Thats realy inefficient.
This could be solved by reading and writing chunks of, say , 1 Kb at a time from m1 to m2.
Reading and writing the file only once should be a great improvement already.

Quote
"Ken White" <kwh...@adpsi.com> wrote:
>Chris,

>Because your code is bad. <g> You're re-creating the Source
>and Dest streams once for each byte of the file you read (because
>it's inside the repeat loop.

>Try this:

>var
>   SourceFile: TFileStream;
>   DestFile: TFileStream;
>   C: Byte;
>begin
>    SourceFile := TFileStream.Create(sFilename,  fmOpenRead);
>    DestFile := TFileStream.Create(sDestFile,  fmOpenWrite);
>    try
>      repeat
>         // Do transfer
>         SourceFile.ReadBuffer(C, 1);
>         ......
>         ........
>         ........
>        DestFile.WriteBuffer(C, 1);
>     until SourceFile.Position = SourceFile.Size;
>  finally
>    DestFile.Free;
>    SourceFile.Free;
>  end;
>end;

>Ken
>---
>Ken White
>kwh...@adpsi.com

>Clipper Functions for Delphi and C++ Builder
>http://www.adpsi.com

>"Chris Morrison"
><[REMOVE]chris[SPAMBLOCK]-morrison@dsl[TO].pipex.com[REPLY]> wrote in
>message news:3cfd150b_1@dnews...
>> Is it just me or is the TFileStream class unfeasibly slow.

>> I am writing a program that needs to extract data from a file 1 byte at a
>> time, perform a simple algorithm on it and then write it into another
>file.
>> The algorithm is not very complex and should cause the operation to take
>so
>> long, it takes several seconds to complete even an a small file that is
>only
>> a few K in size.

>> I am using the TFileStream class because my program could be expected to
>> have to deal with files of  >2Gb, I am using the form:

>> var
>>   SourceFile: TFileStream;
>>   DestFile: TFileStream;
>>   C: Byte;
>> begin
>>    repeat
>>         SourceFile := TFileStream.Create(sFilename,  fmOpenRead);
>>         DestFile := TFileStream.Create(sDestFile,  fmOpenWrite);
>>         // Do transfer
>>         SourceFile.ReadBuffer(C, 1);
>>         ......
>>         ........
>>         ........
>>         DestFile.WriteBuffer(C, 1);
>>    until SourceFile.Position = SourceFile.Size;
>> end;

>> Any comments are welcome.

>> Chris

Other Threads