How to open a big text file

MOHAMMED AL_HEELA

Delphi Developer

Tue, 02 Apr 2002 03:00:00 GMT

How to open a big text file

How to open a big text file.

Bruce Robert

Delphi Developer

Tue, 02 Apr 2002 03:00:00 GMT

Re:How to open a big text file

There are numerous ways to work with text files:

1. Use a tStringList and LoadFromFile. Probably not a great idea for
large multi-mega byte files since the entire file is loaded into memory. It
is the simplest way to work with files and I would use it for files < 4MB
(maybe event larger if my computer had more than 64MB RAM).
2. Use standard Pascal Readln/Writeln - see also AssignFile, Reset,
Rewrite.
3. Use a tFileStream.
4. Use handle based routines see FileOpen and related routines.

Quote

MOHAMMED AL_HEELA <al_heela2...@hotmail.com> wrote in message

news:38070794.18232FCE@hotmail.com...

Quote

> How to open a big text file.

th..

Delphi Developer

Thu, 04 Apr 2002 03:00:00 GMT

Re:How to open a big text file

In article <38070794.18232...@hotmail.com>,
MOHAMMED AL_HEELA <al_heela2...@hotmail.com> wrote:

Quote

> How to open a big text file.

Depends what you want to do with it? If it is quite large, several
megs or more, you should avoid reading it into memory if at all possible
as it can take a lot of time eg. if you read it into a StringList using
load from file, the system will appear to lock up while it read it in -
definately NOT the way to go. If you have to read it all into memory,
write you own procedure to do so. That way you can at least display
some sort of status message and give an option to abort. Perhaps using
a stream would be the best way to go because you could then read in part
of the file at a time. If the user scrolls to another part, you can
load in the next block. Could be a problem if they need to edit it. I
hate how most editors eg. NotePad, Visual C++ editor etc. just blindly
load in a huge file, say 100M, and don't give you an option to cancel
(picked the wrong file!) or at least to see some of the file while it
loads the rest. It is also a good way to blow away all your resources
(memory that is).

I wrote a programmable text editor to allow me to edit unlimited text
files (100's of megs or more) without using up memory. I just read a
line at a time, do the processing on it (removing stop words, converting
strings, changing case, doing lookup conversions, tokenise and
reordering tokens) and save the result to a new file. I only load a
section of the file into a list box if the user requests it! I use this
program extensively for processing of data files, better than a heap of
command lines on a unix box!

trevor

Sent via Deja.com http://www.deja.com/
Before you buy.

Sundial Service

Delphi Developer

Thu, 04 Apr 2002 03:00:00 GMT

Re:How to open a big text file

Quote

th...@my-deja.com wrote:
> I wrote a programmable text editor to allow me to edit unlimited text
> files (100's of megs or more) without using up memory. I just read a
> line at a time, do the processing on it (removing stop words, converting
> strings, changing case, doing lookup conversions, tokenise and
> reordering tokens) and save the result to a new file. I only load a
> section of the file into a list box if the user requests it! I use this
> program extensively for processing of data files, better than a heap of
> command lines on a unix box!

When faced with a similar objective, long ago, I created a class that
would basically treat the file as a series of "regions," some of which
might have been pre-loaded into memory, some of which still resided only
in the input file. Of the regions that had been loaded, some had been
modified (these I kept in memory, although I could have spilled them out
to a scratch-file); the others had not and were therefore discardable or
reusable.

Later on I started to discover some of the many "container classes" for
Delphi that implemented much the same thing. They're out there.

If you think about it, there is no particular reason why you need to
read a file into memory in order to process it.

------------------------------------------------------------------------
Sundial Services :: Scottsdale, AZ (USA) :: (480) 946-8259
mailto:i...@sundialservices.com (PGP public key available.)

Quote

> Got Paradox/Delphi database headaches? ChimneySweep{tm} can help, FAST!
> http://www.sundialservices.com/cs3web.htm

stev

Delphi Developer

Fri, 05 Apr 2002 03:00:00 GMT

Re:How to open a big text file

I think what you want are win32 memory-mapped files. This allows you to access
the file as if it is in memory, and the operating system handles all file
region page swapping for you. You can access the file sequentially or
randomly, both for reading and writing. This is very handy as you can use the
file by just reading and writing a pointer to memory and all file handling is
dealt with for you. Try looking at the win32 help on CreateFileMapping. You
create a file handle with CreateFile, and then map it using CreateFileMapping.
I have seen an article somewhere that created a delphi class to encapsulate
the functionality but can't remember where - perhaps you could search for it,
unless someone else in the newsgroup knows.

Quote

Sundial Services wrote:
> th...@my-deja.com wrote:
> > I wrote a programmable text editor to allow me to edit unlimited text
> > files (100's of megs or more) without using up memory. I just read a
> > line at a time, do the processing on it (removing stop words, converting
> > strings, changing case, doing lookup conversions, tokenise and
> > reordering tokens) and save the result to a new file. I only load a
> > section of the file into a list box if the user requests it! I use this
> > program extensively for processing of data files, better than a heap of
> > command lines on a unix box!

> When faced with a similar objective, long ago, I created a class that
> would basically treat the file as a series of "regions," some of which
> might have been pre-loaded into memory, some of which still resided only
> in the input file. Of the regions that had been loaded, some had been
> modified (these I kept in memory, although I could have spilled them out
> to a scratch-file); the others had not and were therefore discardable or
> reusable.

> Later on I started to discover some of the many "container classes" for
> Delphi that implemented much the same thing. They're out there.

> If you think about it, there is no particular reason why you need to
> read a file into memory in order to process it.

> ------------------------------------------------------------------------
> Sundial Services :: Scottsdale, AZ (USA) :: (480) 946-8259
> mailto:i...@sundialservices.com (PGP public key available.)
> > Got Paradox/Delphi database headaches? ChimneySweep{tm} can help, FAST!
> > http://www.sundialservices.com/cs3web.htm

Engard Feren

Delphi Developer

Fri, 05 Apr 2002 03:00:00 GMT

Re:How to open a big text file

Quote

On Sun, 17 Oct 1999, Sundial Services wrote:
>> I wrote a programmable text editor to allow me to edit unlimited text
>> files (100's of megs or more) without using up memory. I just read a
>> line at a time, do the processing on it (removing stop words, converting

Could somebody suggest an url or something about the algorithm how to
do a text-editor working like this? That is, I can make changes,
insert and delete things, browse the file and I need not to move
100's of megabytes until I finally say 'save'.

Quote

>When faced with a similar objective, long ago, I created a class that
>would basically treat the file as a series of "regions," some of which
>might have been pre-loaded into memory, some of which still resided only
>in the input file. Of the regions that had been loaded, some had been
>modified (these I kept in memory, although I could have spilled them out
>to a scratch-file); the others had not and were therefore discardable or
>reusable.

Er... It sounds complicated... :) An easier way? I am lazy... :-)

Thx:
Circum

stev

Delphi Developer

Sat, 06 Apr 2002 03:00:00 GMT

Re:How to open a big text file

Try looking at this article:

http://www.borland.com/delphi/news/zd/1999/apr99/

Quote

Engard Ferenc wrote:
> On Sun, 17 Oct 1999, Sundial Services wrote:

> >> I wrote a programmable text editor to allow me to edit unlimited text
> >> files (100's of megs or more) without using up memory. I just read a
> >> line at a time, do the processing on it (removing stop words, converting

> Could somebody suggest an url or something about the algorithm how to
> do a text-editor working like this? That is, I can make changes,
> insert and delete things, browse the file and I need not to move
> 100's of megabytes until I finally say 'save'.

> >When faced with a similar objective, long ago, I created a class that
> >would basically treat the file as a series of "regions," some of which
> >might have been pre-loaded into memory, some of which still resided only
> >in the input file. Of the regions that had been loaded, some had been
> >modified (these I kept in memory, although I could have spilled them out
> >to a scratch-file); the others had not and were therefore discardable or
> >reusable.

> Er... It sounds complicated... :) An easier way? I am lazy... :-)

> Thx:
> Circum

> __ @
> / \ _ _ Engrd Ferenc
> l | ( \ / | | (\/) mailto:s-f...@kkt.sote.hu
> \__/ | | \_ \_/ I I http://pons.sote.hu/~s-fery

Other Threads

1. Open big text file

2. (Fwd) Re: Saving big arrays of char in text files

3. Saving MS Word files / Bmp files/ text / rich text files in SQL Server

4. how2 TMemo - Automatic scroll bars appear when text too big, disappear when text fits

5. Database File Size keeps getting bigger and bigger

6. Opening header files without opening the same .CPP file

7. Opening very large text files

8. Open text file

9. Is a text file open or closed ???

10. Opening a Text File (Delphi 1)