Board index » delphi » From TMemoryStreams to String Problem.

From TMemoryStreams to String Problem.

Hi everybody,

Problem: I need to read an encrypted file from disk, decode it into memory, do
something with it then write it again as an encoded file. The encoding/decoding
routing works with TMemoryStreams.

I have stumbled across this strange (to me) behaviour.

Let's call 'DecodedStream' a TMemoryStreams containing the decoded contents of
the file in memory. This is already decoded and plain ascii.

I can do this: DecodedStream.SaveToFile('decoded.txt') and I get a perfectly
decoded ASCII file on disk.

If I do this: Memo1.Lines.Add(PChar(DecodedStream.Memory)) however, I get the
same perfect ASCII file with about 2480 extra bytes of garbage appended. Since
I have a routine which has to work with that file and expects it to start and
end with a precise set of tags, that routine also is screwed. In other words, I
need a way to convert the 'DecodedStream.Memory' contents into a clean string,
the same way that DecodedStream.SaveToFile does, and pass it as parameter to
another function.

The only solution so far has been to save the decoded file to disk and then
have the routine read that file from there but this completes defeats the
purpose of encrypting the file in the beginning, since I'm saving a clear text
copy of it on disk. I would like my program to work with the decoded copy in
memory only.

Any suggestions ?

Thank you.

--
Posted from  [80.17.43.149]
via Mailgate.ORG Server - http://www.Mailgate.ORG

 

Re:From TMemoryStreams to String Problem.


Quote
Ettore Carrara wrote in message

<86814087f7e75f437b1ad3647d84786c.34...@mygate.mailgate.org>...

Quote

>Hi everybody,

>Problem: I need to read an encrypted file from disk, decode it into
memory, do
>something with it then write it again as an encoded file. The
encoding/decoding
>routing works with TMemoryStreams.

>I have stumbled across this strange (to me) behaviour.

>Let's call 'DecodedStream' a TMemoryStreams containing the decoded
contents of
>the file in memory. This is already decoded and plain ascii.

>I can do this: DecodedStream.SaveToFile('decoded.txt') and I get a
perfectly
>decoded ASCII file on disk.

>If I do this: Memo1.Lines.Add(PChar(DecodedStream.Memory)) however, I get
the
>same perfect ASCII file with about 2480 extra bytes of garbage appended.
Since
>I have a routine which has to work with that file and expects it to start
and
>end with a precise set of tags, that routine also is screwed. In other
words, I
>need a way to convert the 'DecodedStream.Memory' contents into a clean
string,
>the same way that DecodedStream.SaveToFile does, and pass it as parameter
to
>another function.

>The only solution so far has been to save the decoded file to disk and
then
>have the routine read that file from there but this completes defeats the
>purpose of encrypting the file in the beginning, since I'm saving a clear
text
>copy of it on disk. I would like my program to work with the decoded copy
in
>memory only.

I haven't looked, but I think SaveToFile passes a buffer address
(Memory) and a byte count (Length?) to some OS routine. This is
the proper way; a buffer address is just a starting address and
the length of the buffer is related but independent information.
Casting Memory to a PChar follows it until the first NUL character,
which has nothing to do with anything. In a TStringStream you may
expect the data to be zero-terminated, but in a TMemoryStream
that's just optimism bound to be bitten by reality. Until now,
you've gotten garbage, but at some point you're going to get
access violations, too. You're reading memory that isn't yours
in the first place.

This routine that expects delimited data, that sounds wrong, too.
If you can find the delimiters, obviously the data ends at the
terminator. Any data coming after that may indicate a logical
error, but should never cause your code to crash.

The final question is why you are doing things in this manner at
all. Wouldn't it be much cleaner to derive a new TStream that
is plaintext in memory and encrypted on disk, and encrypts and
decrypts when loading/saving?

Groetjes,
Maarten Wiltink

Re:From TMemoryStreams to String Problem.


As Maarten pointed out - you are treating a Stream as a block of
memory with no regard to its internal structure.

Personally I would get the Stream Size, set its Position to 0 and make
an AnsiiString of the correct size.

Then I would:
       DecodedStream.Read( S[1], Length(S) )

The fact that the DecodedStream is really a memory buffer is just
chance - it could be getting data from anywhere.

On Thu, 20 Dec 2001 10:07:13 +0000 (UTC), "Ettore Carrara"

Quote
<penn...@poboxes.com> wrote:

>Hi everybody,

>Problem: I need to read an encrypted file from disk, decode it into memory, do
>something with it then write it again as an encoded file. The encoding/decoding
>routing works with TMemoryStreams.

>I have stumbled across this strange (to me) behaviour.

>Let's call 'DecodedStream' a TMemoryStreams containing the decoded contents of
>the file in memory. This is already decoded and plain ascii.

>I can do this: DecodedStream.SaveToFile('decoded.txt') and I get a perfectly
>decoded ASCII file on disk.

>If I do this: Memo1.Lines.Add(PChar(DecodedStream.Memory)) however, I get the
>same perfect ASCII file with about 2480 extra bytes of garbage appended. Since
>I have a routine which has to work with that file and expects it to start and
>end with a precise set of tags, that routine also is screwed. In other words, I
>need a way to convert the 'DecodedStream.Memory' contents into a clean string,
>the same way that DecodedStream.SaveToFile does, and pass it as parameter to
>another function.

>The only solution so far has been to save the decoded file to disk and then
>have the routine read that file from there but this completes defeats the
>purpose of encrypting the file in the beginning, since I'm saving a clear text
>copy of it on disk. I would like my program to work with the decoded copy in
>memory only.

>Any suggestions ?

>Thank you.

>--
>Posted from  [80.17.43.149]
>via Mailgate.ORG Server - http://www.Mailgate.ORG

Re:From TMemoryStreams to String Problem.


"Ettore Carrara" <penn...@poboxes.com> skrev i melding
news:86814087f7e75f437b1ad3647d84786c.34776@mygate.mailgate.org...

Quote
> Problem: I need to read an encrypted file from disk, decode it into memory,
do
> something with it then write it again as an encoded file. The
encoding/decoding
> routing works with TMemoryStreams.

> I have stumbled across this strange (to me) behaviour.

> Let's call 'DecodedStream' a TMemoryStreams containing the decoded contents
of
> the file in memory. This is already decoded and plain ascii.

> I can do this: DecodedStream.SaveToFile('decoded.txt') and I get a
perfectly
> decoded ASCII file on disk.

> If I do this: Memo1.Lines.Add(PChar(DecodedStream.Memory)) however, I get
the
> same perfect ASCII file with about 2480 extra bytes of garbage appended.
Since
> I have a routine which has to work with that file and expects it to start
and
> end with a precise set of tags, that routine also is screwed. In other
words, I
> need a way to convert the 'DecodedStream.Memory' contents into a clean
string,
> the same way that DecodedStream.SaveToFile does, and pass it as parameter
to
> another function.

i BELIEVE THE MEMORY STREAM ALLOCATES ONE PAGE OF MEMORY AT A TIME (NORMALLY
4096 BYTES). (sorry for shouting).
The simplest would be to use a string as a temp storage:

var
  Tmp: string;
begin
  SetLength(Tmp, DecodedStream.Size);
  DecodedStream.Seek(0, soFromBeginning);
  DecodedStream.Write(Tmp[1], DecodedStream.Size);
  Memo1.Lines.Add(Tmp);

- Show quoted text -

Quote
> The only solution so far has been to save the decoded file to disk and then
> have the routine read that file from there but this completes defeats the
> purpose of encrypting the file in the beginning, since I'm saving a clear
text
> copy of it on disk. I would like my program to work with the decoded copy
in
> memory only.

> Any suggestions ?

> Thank you.

> --
> Posted from  [80.17.43.149]
> via Mailgate.ORG Server - http://www.Mailgate.ORG

--
Bjoerge Saether
Asker, Norway
bjorge@hahaha_itte.no (remve the obvious)

Re:From TMemoryStreams to String Problem.


Quote
"Maarten Wiltink" <maar...@kittensandcats.net> wrote in message

news:9vsf7n$5ap$1@news1.xs4all.nl...

Quote
> I haven't looked, but I think SaveToFile passes a buffer address
> (Memory) and a byte count (Length?) to some OS routine. This is
> the proper way; a buffer address is just a starting address and
> the length of the buffer is related but independent information.
> Casting Memory to a PChar follows it until the first NUL character,
> which has nothing to do with anything. In a TStringStream you may
> expect the data to be zero-terminated, but in a TMemoryStream
> that's just optimism bound to be bitten by reality. Until now,
> you've gotten garbage, but at some point you're going to get
> access violations, too. You're reading memory that isn't yours
> in the first place.

The starting problem was the following: I have a encryption/decryption routine
which works with TMemoryStream as input and as output. I have looked at other
implementations of different algorithms (blowfish, rsa, des, rindael (sp?),
etc. but more or less they work the same (in terms of parameters)). Since my
encrypted file decodes to a plain ascii file, I needed a way to translate the
output of the decryption routine (which is a TMemoryStream) to something that
the subsequent procedure would accept, without resorting to saving the file to
disk.

So my clumsy attempts were intended to find a quick and dirty way to achieve
that. Thanks to yours and others suggestions, I found that the following way
seems to work correctly:

  OutputText := TStringStream.Create('');
  DecodedStream.SaveToStream(OutputText);
  Memo1.Lines.Add(OutPutString.DataString); // or whatever

Quote
> The final question is why you are doing things in this manner at
> all. Wouldn't it be much cleaner to derive a new TStream that
> is plaintext in memory and encrypted on disk, and encrypts and
> decrypts when loading/saving?

I'm sure it is but you would need to evaluate how much more work it would be
compared to the three lines above. I will think about it anyway. Thank you.

CiAo! Et

--
Posted from  [80.17.43.149]
via Mailgate.ORG Server - http://www.Mailgate.ORG

Re:From TMemoryStreams to String Problem.


On Thu, 20 Dec 2001 13:39:50 +0100, "Bj?rge S?ther"

Quote
<REMOVE_bsaether@THIS_online.no> wrote:

<snip>
Quote
>The simplest would be to use a string as a temp storage:

>var
>  Tmp: string;
>begin
>  SetLength(Tmp, DecodedStream.Size);
>  DecodedStream.Seek(0, soFromBeginning);
>  DecodedStream.Write(Tmp[1], DecodedStream.Size);

    Er, won't that just fill the stream with uninitialized garbage?
Quote
>  Memo1.Lines.Add(Tmp);

Re:From TMemoryStreams to String Problem.


    DecodedStream.Position := 0;
    Memo1.Lines.LoadFromStream (DecodedStream);

Another option is to use a tStringStream instead of a tMemoryStream. Then
you could do

    Memo1.Text := DecodedStream.DataString;

Re:From TMemoryStreams to String Problem.


"J French" <je...@iss.u-net.com> skrev i melding
news:3c21f7eb.17860129@news.u-net.com...

Quote
> On Thu, 20 Dec 2001 13:39:50 +0100, "Bj?rge S?ther"
> <REMOVE_bsaether@THIS_online.no> wrote:

> <snip>
> >The simplest would be to use a string as a temp storage:

> >var
> >  Tmp: string;
> >begin
> >  SetLength(Tmp, DecodedStream.Size);
> >  DecodedStream.Seek(0, soFromBeginning);
> >  DecodedStream.Write(Tmp[1], DecodedStream.Size);
>     Er, won't that just fill the stream with uninitialized garbage?

OOPS! It should be Read(....

Thanks, Jerry ! It's a comfort seeing that one's scribbles *are* actually
being read ! In fact, the distinction Read() <> Write() with streams allways
make me think a second time: "Is this read or write...", but here I only
thought once. I believe the problem is what direction "read" and "write" is
working. ;-)

Quote
> >  Memo1.Lines.Add(Tmp);

--
Bjoerge Saether
Asker, Norway
bjorge@hahaha_itte.no (remve the obvious)

Re:From TMemoryStreams to String Problem.


Quote
"Bj?rge S?ther" <REMOVE_bsaether@THIS_online.no> wrote in message

news:%KDU7.6501$KQ3.82679@news1.oke.nextra.no...

Quote
> Thanks, Jerry ! It's a comfort seeing that one's scribbles *are* actually
> being read ! In fact, the distinction Read() <> Write() with streams allways
> make me think a second time: "Is this read or write...", but here I only
> thought once. I believe the problem is what direction "read" and "write" is
> working. ;-)

I read your scribbles too, don't feel alone. :-)

In my continuos struggle with TMemoryStreams, I keep finding some things a bit
puzzling. For example:

  Buffer: array of byte;
  MyStream: TMemoryStream;
  FileSize: int64;

  [..]

  MyStream := TMemoryStream.Create;
  MyStream.LoadFromFile('whatever');
  MyStream.Seek(0,0);
  FileSize := MyStream.Size;
  MyStream.Read(Buffer, FileSize);

  At this point, FileSize reports as expected but If I have a peek at the
buffer contents, it's an endless plain of zeros.. Shouldn't it be the binary
content of 'whatever' ?

By the way, I have to use the FileSize variable because if I access the
'MyStream.Size' property directly, it always reports zero (!).

In other words, how do I load something from a file (being a binary or text
file) and convert it to a stream of bytes ? it should be obvious but it isn't
to me.. :-(.

Thanks.

--
Posted from  [80.17.43.149]
via Mailgate.ORG Server - http://www.Mailgate.ORG

Re:From TMemoryStreams to String Problem.


On Thu, 20 Dec 2001 15:36:35 -0500, "Bruce Roberts"

Quote
<b...@bounceitattcanada.xnet> wrote:
>    DecodedStream.Position := 0;
>    Memo1.Lines.LoadFromStream (DecodedStream);

Yes - I considered that - but I think the OP wanted to Add a load of
data to a TMemo

Quote

>Another option is to use a tStringStream instead of a tMemoryStream. Then
>you could do

>    Memo1.Text := DecodedStream.DataString;

Memo1.Lines.Add( DecodedStream.DataString )

By far the best/cleanest idea - I confess I had never found
tStringStream

- Thanks

Re:From TMemoryStreams to String Problem.


Ettore,

beats me why the  Stream.Size is playing up for you, but the main
problem is that Buffer is a pointer to an array - the start location
of the data is Buffer[0]

Personally, unless you have good reasons for it, why not use Strings
instead of 'arrays of byte' ?

Anyway - this works

procedure TForm1.Button1Click(Sender: TObject);
  Var
  Buffer: array of byte;
  MyStream: TMemoryStream;

  Begin

  MyStream := TMemoryStream.Create;
  MyStream.LoadFromFile('c:\autoexec.bat');
  MyStream.Position := 0 ; //Not Needed
  SetLength( Buffer, MyStream.Size ) ;
  MyStream.Read(Buffer[0], MyStream.Size );
  MyStream.Free ;
end;

On Fri, 21 Dec 2001 10:12:59 +0000 (UTC), "Ettore Carrara"

Quote
<penn...@poboxes.com> wrote:
>"Bj?rge S?ther" <REMOVE_bsaether@THIS_online.no> wrote in message
>news:%KDU7.6501$KQ3.82679@news1.oke.nextra.no...

>> Thanks, Jerry ! It's a comfort seeing that one's scribbles *are* actually
>> being read ! In fact, the distinction Read() <> Write() with streams allways
>> make me think a second time: "Is this read or write...", but here I only
>> thought once. I believe the problem is what direction "read" and "write" is
>> working. ;-)

>I read your scribbles too, don't feel alone. :-)

>In my continuos struggle with TMemoryStreams, I keep finding some things a bit
>puzzling. For example:

>  Buffer: array of byte;
>  MyStream: TMemoryStream;
>  FileSize: int64;

>  [..]

>  MyStream := TMemoryStream.Create;
>  MyStream.LoadFromFile('whatever');
>  MyStream.Seek(0,0);
>  FileSize := MyStream.Size;
>  MyStream.Read(Buffer, FileSize);

>  At this point, FileSize reports as expected but If I have a peek at the
>buffer contents, it's an endless plain of zeros.. Shouldn't it be the binary
>content of 'whatever' ?

>By the way, I have to use the FileSize variable because if I access the
>'MyStream.Size' property directly, it always reports zero (!).

>In other words, how do I load something from a file (being a binary or text
>file) and convert it to a stream of bytes ? it should be obvious but it isn't
>to me.. :-(.

>Thanks.

>--
>Posted from  [80.17.43.149]
>via Mailgate.ORG Server - http://www.Mailgate.ORG

Re:From TMemoryStreams to String Problem.


Quote
"J French" <je...@iss.u-net.com> wrote in message

news:3c231469.2637041@news.u-net.com...

Quote
> On Thu, 20 Dec 2001 15:36:35 -0500, "Bruce Roberts"
> <b...@bounceitattcanada.xnet> wrote:

> >    DecodedStream.Position := 0;
> >    Memo1.Lines.LoadFromStream (DecodedStream);
> Yes - I considered that - but I think the OP wanted to Add a load of
> data to a TMemo

Which is what the second line does.

Re:From TMemoryStreams to String Problem.


In article <a4295fbf80217c63dd230ab6e3934d67.34...@mygate.mailgate.org>,

Quote
"Ettore Carrara" <penn...@poboxes.com> writes:
>So my clumsy attempts were intended to find a quick and dirty way to achieve
>that. Thanks to yours and others suggestions, I found that the following way
>seems to work correctly:

>  OutputText := TStringStream.Create('');
>  DecodedStream.SaveToStream(OutputText);
>  Memo1.Lines.Add(OutPutString.DataString); // or whatever

You could also code ...

const
  NilByte : byte = 0;

  DecodedStream.Seek(0, soFromEnd);
  DecodedStream.Write(NilByte, 1);
  Memo1.Lines.Add(string(PChar(DecodedStream.Memory)));

Alan Lloyd
alangll...@aol.com

Re:From TMemoryStreams to String Problem.


On Fri, 21 Dec 2001 12:47:41 -0500, "Bruce Roberts"

Quote
<b...@bounceitattcanada.xnet> wrote:

>"J French" <je...@iss.u-net.com> wrote in message
>news:3c231469.2637041@news.u-net.com...
>> On Thu, 20 Dec 2001 15:36:35 -0500, "Bruce Roberts"
>> <b...@bounceitattcanada.xnet> wrote:

>> >    DecodedStream.Position := 0;
>> >    Memo1.Lines.LoadFromStream (DecodedStream);
>> Yes - I considered that - but I think the OP wanted to Add a load of
>> data to a TMemo

>Which is what the second line does.

Sorry I meant 'Apend'  -  surely the LoadFromStream replaces any
existing lines.

Anyway this thread has been very interesting - I shall be creating
some useful descendents of TStringStream.

Quote

Re:From TMemoryStreams to String Problem.


Quote
"J French" <je...@iss.u-net.com> wrote in message

news:3c2485db.489712@news.u-net.com...

Quote
> Sorry I meant 'Apend'  -  surely the LoadFromStream replaces any
> existing lines.

Then a string stream is probably the easiest:

Memo1.Text := Memo1.Text + aStringStream.DataString;

Sticking with memory streams would probably involve using another one:

var tmpStrm : tMemorStream;

tmpStrm := tMemoryStream.Create;
try
    Memo1.Lines.SaveToStream (tmpStrm);
    tmpStrm.CopyFrom (DecodedStream);
    tmpStrm.Position := 0;
    Memo1.Lines.LoadFromStream (tmpStrm);
finally
    tmpStrm.Free;
    end;

Other Threads