Board index » delphi » Opening very large text files

Opening very large text files

Hi,

I have to open and process a text file that is about 15-20 Meg in size
with a max record length of 1000 characters. When I open the file with
assignfile it takes for ever and eventually runs out of memory, I triued
using the ttextdata component to read the data and the same thing
happens. It looks like everything is attempting to read the entire file
into memory on the open statement, is that true ?

Is there anyway that I can read this text file ?

Thanks
Allan

 

Re:Opening very large text files


On Thu, 19 Oct 2000 10:18:44 +1000, Allan Morris

Quote
<allan.mor...@actfs.com.au> wrote:
>Hi,

>I have to open and process a text file that is about 15-20 Meg in size
>with a max record length of 1000 characters. When I open the file with
>assignfile it takes for ever and eventually runs out of memory, I triued
>using the ttextdata component to read the data and the same thing
>happens. It looks like everything is attempting to read the entire file
>into memory on the open statement, is that true ?

Assignfile should be fine.  Could you post the code you were using?

TTextData I'm not so sure about.

Duncan Murdoch

Re:Opening very large text files


The Borland community page has a link to "Intelligently reading a file one
line at a time "

on the following site
http://www.delphi3000.com/article.asp?ID=1363

Mike
--
Mike Best Programming
Brisbane
Australia

Quote
Allan Morris <allan.mor...@actfs.com.au> wrote in message

news:39EE3DE4.8F70D87D@actfs.com.au...
Quote
> Hi,

> I have to open and process a text file that is about 15-20 Meg in size
> with a max record length of 1000 characters. When I open the file with
> assignfile it takes for ever and eventually runs out of memory, I triued
> using the ttextdata component to read the data and the same thing
> happens. It looks like everything is attempting to read the entire file
> into memory on the open statement, is that true ?

> Is there anyway that I can read this text file ?

> Thanks
> Allan

Re:Opening very large text files


Im Artikel <39EE3DE4.8F70D...@actfs.com.au>, Allan Morris
<allan.mor...@actfs.com.au> schreibt:

Quote
>Is there anyway that I can read this text file ?

Use TStream instead of the legacy File interface. I'm currently working with an
68 MB file and a TFileStream, without any problems.

Of course you should avoid to read the whole file into memory, unless you have
an appropriate amount of free RAM available. Otherwise the chunks read from the
file are swapped out to disk almost immediately, and the swapping mechanism of
Windows results in a much slower processing, as if you were reading only the
actually interesting line from the file.

DoDi

Re:Opening very large text files


Yup - DoDi is right.

What I do is read the file in large chunks using a TStream, as I
encounter CrLf I record the position and string length in a temporary
'index' file

     [Position] [Length]

This gives random read access to any line in the file

Depending on your application you may want to process the whole file
in one go, or just process say 1 large block at a time.

A text viewer is better processing the lot (even if it is spoofed with
a Thread or on a Timer)  if you only want sequential access then just
pull it off a buffer as needed - and refill the buffer when required.

The trick is to minimize the number of Disk Accesses.

On 21 Oct 2000 00:51:47 GMT, vb...@aol.com (VBDis) wrote:

Quote
>Im Artikel <39EE3DE4.8F70D...@actfs.com.au>, Allan Morris
><allan.mor...@actfs.com.au> schreibt:

>>Is there anyway that I can read this text file ?

>Use TStream instead of the legacy File interface. I'm currently working with an
>68 MB file and a TFileStream, without any problems.

>Of course you should avoid to read the whole file into memory, unless you have
>an appropriate amount of free RAM available. Otherwise the chunks read from the
>file are swapped out to disk almost immediately, and the swapping mechanism of
>Windows results in a much slower processing, as if you were reading only the
>actually interesting line from the file.

>DoDi

Other Threads