Board index » delphi » "Wraparound" text files

"Wraparound" text files

Hi all,

Here's what I'm trying to achieve (in TP7 for DOS):

I have a "log" (text) file that grows until a certain size (number of
lines) and then starts overwriting the start again, in effect giving a
list of the last n events that occured.  

I don't want to rewrite the entire file every time I remove the first
line nor do I want it to be a file of records coz I want to be able to
view it as a normal text file (ie in the order it occured).  At
present  I have implemented it as a file of fixed size strings padded
with spaces and with lengths forced to 0x20 (so that they appear as
spaces when viewing) but this is obviously wasteful of space.  Also,
an "*** end of file ***" string record can appear anywhere in the
middle of the file coz it only "logically" wraps around which is a bit
of a pain!

What I was thinking of trying was to use the way DOS stores files as a
list of clusters.  Instead of rewriting the entire file, isn't there
some way to get at these cluster allocations and simply remove the
first and continue appending as a normal text file ?  Obviously such
drastic action plays severely with internals but MUST be compatible on
past and future (if any :) versions of DOS.

I'd be interested to hear any comments.

 

Re:"Wraparound" text files


Quote
In article <51gg5j$...@dodo.global.co.za> lest...@global.co.za (Lester Hanger) writes:
>Hi all,
>Here's what I'm trying to achieve (in TP7 for DOS):
>I have a "log" (text) file that grows until a certain size (number of
>lines) and then starts overwriting the start again, in effect giving a
>list of the last n events that occured.  
>I don't want to rewrite the entire file every time I remove the first
>line nor do I want it to be a file of records coz I want to be able to
>view it as a normal text file (ie in the order it occured).  At
>present  I have implemented it as a file of fixed size strings padded
>with spaces and with lengths forced to 0x20 (so that they appear as
>spaces when viewing) but this is obviously wasteful of space.  Also,
>an "*** end of file ***" string record can appear anywhere in the
>middle of the file coz it only "logically" wraps around which is a bit
>of a pain!

Some ideas:

1.)  Load the file into an array of strings (no space padding necessary).  
Use the image in memory as the log, and write it back when the program is
finished.  You'll get minimal memory consumption with something like:

const
  NUMENTRIES = 40;

type
  Pstring = ^string;
  Tstringarry = array[0..NUMENTRIES-1] of Pstring;

var
  entrycount : byte;
  f : text;
  tmp : string;
  log : Tstringarray;

begin
  entrycount := 0;
  fillchar (log,sizeof(log),0); {initializes all pointers to nil}
  assign (f,'my_file.log');
  reset (f);
  while not eof(f) and (entrycount<40) do
    begin
      readln (f,tmp);
      getmem (log[entrycount],length(tmp)+1);
      log[entrycount] := tmp;
      inc (entrycount);
    end;
end;

Writing the file back and deallocating is fairly similar (freeing memory is
easy because the string has the information indicating roughly how much
memory is allocated to it).  Adding to the log isn't the most
straightforward, but if you keep track of where you are in the log, keeping
a rolling record is easy.

2.)  Just occasionally trim the log back by copying to a temporary file,
t{*word*220}, then copying over the original.  This is the simplest and doesn't
require any great amount of precision or trickery.  Besides, is it really
going to matter that much if you have a few extra entries in your file?

Quote
>What I was thinking of trying was to use the way DOS stores files as a
>list of clusters.  Instead of rewriting the entire file, isn't there
>some way to get at these cluster allocations and simply remove the
>first and continue appending as a normal text file ?  Obviously such
>drastic action plays severely with internals but MUST be compatible on
>past and future (if any :) versions of DOS.

No, that's not a good idea.  Clusters represent large blocks of space -- 1,
2, or 4K on small hard drives and floppies, 8, 16, or even 32K on drives
with very large partitions.  This proposal causes far more problems than it
solves.

Quote
>I'd be interested to hear any comments.

--
Scott Earnest          | We now return you to our regularly scheduled |
siny...@{*word*104}space.org | chaos and mayhem. . . .                      |

Re:"Wraparound" text files


Oh, I forgot, you should use Seek(), for move the file pointer before
doing the ReadBlock() and WriteBlock().

Peter

Re:"Wraparound" text files


On Sun, 15 Sep 1996 16:15:20 GMT, lest...@global.co.za (Lester Hanger)
wrote:

Quote
>Hi all,

>Here's what I'm trying to achieve (in TP7 for DOS):

>I have a "log" (text) file that grows until a certain size (number of
>lines) and then starts overwriting the start again, in effect giving a
>list of the last n events that occured.  

Yes, I have a similar log file...

Quote
>I don't want to rewrite the entire file every time I remove the first
>line nor do I want it to be a file of records coz I want to be able to
>view it as a normal text file (ie in the order it occured).  At
>present  I have implemented it as a file of fixed size strings padded
>with spaces and with lengths forced to 0x20 (so that they appear as
>spaces when viewing) but this is obviously wasteful of space.  Also,
>an "*** end of file ***" string record can appear anywhere in the
>middle of the file coz it only "logically" wraps around which is a bit
>of a pain!

>What I was thinking of trying was to use the way DOS stores files as a
>list of clusters.  Instead of rewriting the entire file, isn't there
>some way to get at these cluster allocations and simply remove the
>first and continue appending as a normal text file ?  Obviously such
>drastic action plays severely with internals but MUST be compatible on
>past and future (if any :) versions of DOS.

Personally I would strongly advise against the latter method. Although
it could, in theory, save the rewrite, it does require you to go deep
into the DOS BIOS. You must also consider how much time it would take
you to write such routines (is that why you ask the question?), and
how much it would be used (not much!). It's better to do the rewrite
within the file itself. Simply open the file as a general FILE type;

VAR MyLogFile:FILE;

and set the record size to 1;

Assign(MyLogFile,'....');
Reset(MyLogFile,1);

Then ReadBlock() and WriteBlock() to move the desired portion of the
file to the front (Use at least 16Kb for the buffer to get some speed)
and then truncate the file. This really won't take that long... and it
is normal pascal! You can write this routine in half an hour... or
faster.

It is of course very wasteful to use spaces within such a log file.
Actually it would be best to approach the log file, under normal
circumstances, as an ordinary test file. Simply use a, normally not
used character, to seperate the different loggings from eachother.
Before you wrap around you should, of couse, search for that character
so that you do not split a log line into two.

Anyway, I don't see any need for mind-boggling speed when all you want
to do is to truncate a log file every now and then...

Much luck,

Peter

Re:"Wraparound" text files


Quote
siny...@{*word*104}space.org (Scott Earnest) wrote:
>Some ideas:
>1.)  Load the file into an array of strings (no space padding necessary).  
>Use the image in memory as the log, and write it back when the program is
>finished.  You'll get minimal memory consumption with something like:
>const
>  NUMENTRIES = 40;

This is where the problem is!  Try NUMENTRIES = 40000!!!  (An average
of one event per minute, 24 hours a day with results for a month!)

Quote
>2.)  Just occasionally trim the log back by copying to a temporary file,
>t{*word*220}, then copying over the original.  This is the simplest and doesn't
>require any great amount of precision or trickery.  Besides, is it really
>going to matter that much if you have a few extra entries in your file?

This is possible but I don't like the idea of rewriting over a meg of
data every time.

Quote
>>What I was thinking of trying was to use the way DOS stores files as a
>>list of clusters.  Instead of rewriting the entire file, isn't there
>>some way to get at these cluster allocations and simply remove the
>>first and continue appending as a normal text file ?  Obviously such
>>drastic action plays severely with internals but MUST be compatible on
>>past and future (if any :) versions of DOS.
>No, that's not a good idea.  Clusters represent large blocks of space -- 1,
>2, or 4K on small hard drives and floppies, 8, 16, or even 32K on drives
>with very large partitions.  This proposal causes far more problems than it
>solves.

Only if you don't have a way of implementing it easily!

So far, I'll stick to treating it as a file of records (strings) and
just path the correct entry.  

I thank you for your commens Chris.

Cheers,
Lester

Re:"Wraparound" text files


Quote
siny...@{*word*104}space.org (Scott Earnest) wrote:
>Some ideas:
>1.)  Load the file into an array of strings (no space padding necessary).  
>Use the image in memory as the log, and write it back when the program is
>finished.  You'll get minimal memory consumption with something like:
>const
>  NUMENTRIES = 40;

This is the problem: Try NUMENTRIES = 43200!!!  (Assume one event per
minute with a log for the last month).

Quote
>2.)  Just occasionally trim the log back by copying to a temporary file,
>t{*word*220}, then copying over the original.  This is the simplest and doesn't
>require any great amount of precision or trickery.  Besides, is it really
>going to matter that much if you have a few extra entries in your file?

This could be done whenever the user exits the program but if the
program is left to run (over a number of days), this could be a
problem too.

Quote
>No, that's not a good idea.  Clusters represent large blocks of space -- 1,
>2, or 4K on small hard drives and floppies, 8, 16, or even 32K on drives
>with very large partitions.  This proposal causes far more problems than it
>solves.

I agree!  If it was easy to achieve I wouldn't have a problem.  Thanx
for your thoughts Chris.

Cheers,
Lester

Other Threads