Board index » cppbuilder » TClientDataSet::LoadFromStream seeks to end of stream?

TClientDataSet::LoadFromStream seeks to end of stream?


2008-01-25 11:48:18 AM
cppbuilder91
I am attempting to store a TClientDataSet in a TFileStream, followed by some other data (in my case, a second TClientDataSet). Saving all of the data works perfectly (verified by examining data in file).
When I call TClientDataSet::LoadFromStream() to read the data from a TFileStream (opened in dfOpenRead mode), the file position is set to the end of the file when LoadFromStream() returns (verified by checking TFileStream::Position after every operation). The TClientDataSet is loaded correctly, but reading any more data from the stream fails since it's at EOF now.
I can load the remaining objects from the file by calling TFileStream::Seek to move to the end of the TClientDataSet, but that only works if I know the exact size of the data in the file, which I don't under normal circumstances.
What is going on? How can I make this not happen?
Here is my test program. It's a C++ console application with VCL support:
//---------------------------------------------------------------------------
#include <stdio.h>
#include <vcl.h>
#include <DBClient.hpp>
#include <System.hpp>
#pragma hdrstop
//---------------------------------------------------------------------------
#pragma argsused
void DumpDataSet (TClientDataSet *ds, const char *name) {
printf("dataset '%s' (%i records):\n", name, ds->RecordCount);
for (ds->First(); !ds->Eof; ds->Next()) {
int i0 = ds->Fields->Fields[0]->AsInteger;
int i1 = ds->Fields->Fields[1]->AsInteger;
printf("{ %li %li }\n", i0, i1);
}
}
// creates tclientdataset with 2 columns, possibly filling with
// some initial data.
TClientDataSet * BuildDataSet (bool populate) {
static int a = 0;
TClientDataSet *ds = new TClientDataSet(NULL);
ds->FieldDefs->Add("first", ftInteger, 0, true);
ds->FieldDefs->Add("second", ftInteger, 0, true);
ds->CreateDataSet();
if (populate) {
ds->InsertRecord(ARRAYOFCONST((a+1, a+2)));
ds->InsertRecord(ARRAYOFCONST((a+3, a+4)));
ds->InsertRecord(ARRAYOFCONST((a+5, a+6)));
a += 6;
}
return ds;
}
int main (int argc, char **argv) {
// create and populate 2 datasets
TClientDataSet *DS1 = BuildDataSet(true);
TClientDataSet *DS2 = BuildDataSet(true);
DumpDataSet(DS1, "DS1 [new]");
DumpDataSet(DS2, "DS2 [new]");
// save them both to a stream, one after the other
TFileStream *data = new TFileStream("testdata.dat", fmCreate);
printf("%i\n", (int)data->Position);
DS1->SaveToStream(data, dfBinary);
printf("%i\n", (int)data->Position);
DS2->SaveToStream(data, dfBinary);
printf("%i\n", (int)data->Position);
delete data;
// now, create brand spankin' new datasets
delete DS2;
delete DS1;
DS1 = BuildDataSet(false);
DS2 = BuildDataSet(false);
DumpDataSet(DS1, "DS1 [reset]");
DumpDataSet(DS2, "DS2 [reset]");
// then load them both from that file, same order:
data = new TFileStream("testdata.dat", fmOpenRead);
printf("%i\n", (int)data->Position);
DS1->LoadFromStream(data);
printf("%i\n", (int)data->Position); // <--- position ends up at eof
DS2->LoadFromStream(data);
printf("%i\n", (int)data->Position);
delete data;
DumpDataSet(DS1, "DS1 [loaded]"); // <--- loads ok.
DumpDataSet(DS2, "DS2 [loaded]"); // <--- doesn't load.
//
delete DS2;
delete DS1;
printf("finished\n");
getchar();
return 0;
}
=======================
Here is the output I get:
dataset 'DS1 [new]' (3 records):
{ 1 2 }
{ 3 4 }
{ 5 6 }
dataset 'DS2 [new]' (3 records):
{ 7 8 }
{ 9 10 }
{ 11 12 }
0
140
280
dataset 'DS1 [reset]' (0 records):
dataset 'DS2 [reset]' (0 records):
0
280 <<<<< ****** position is at EOF!!!!! ******
280
dataset 'DS1 [loaded]' (3 records):
{ 1 2 }
{ 3 4 }
{ 5 6 }
dataset 'DS2 [loaded]' (0 records):
finished
 
 

Re:TClientDataSet::LoadFromStream seeks to end of stream?

Hi Jason Cipriani
Jason Cipriani says:
Quote

I am attempting to store a TClientDataSet in a TFileStream, followed by some other data (in my case, a second TClientDataSet). Saving all of the data works perfectly (verified by examining data in file).

When I call TClientDataSet::LoadFromStream() to read the data from a TFileStream (opened in dfOpenRead mode), the file position is set to the end of the file when LoadFromStream() returns (verified by checking TFileStream::Position after every operation). The TClientDataSet is loaded correctly, but reading any more data from the stream fails since it's at EOF now.

I can load the remaining objects from the file by calling TFileStream::Seek to move to the end of the TClientDataSet, but that only works if I know the exact size of the data in the file, which I don't under normal circumstances.

What is going on? How can I make this not happen?
I don't think the intension of LoadFromFile was to load more
then one item Object from the stream, in other words it expect
that all the data in the stream is to be read.
E.g. TStrings::SaveToStream just writes TStrings->Text to file.
snip from the source:
procedure TStrings.SaveToFile(const FileName: string);
var
Stream: TStream;
begin
Stream := TFileStream.Create(FileName, fmCreate);
try
SaveToStream(Stream);
finally
Stream.Free;
end;
end;
procedure TStrings.SaveToStream(Stream: TStream);
var
S: string;
begin
S := GetTextStr;
Stream.WriteBuffer(Pointer(S)^, Length(S));
end;
I use TStrings as example because I don't have TClientDataSet
I think You need to make Your own read and write function where
You start by writing the size of the data to be read and then
copy that to a buffer.
There might be a difference if You use a TMemoryStream instead
of a TFileStream.
P.s. I sure can follow Your logic and it would be nice if
SaveToStream worked the way You expected.
P.s.p.s. please break Your text lines.
Kind regards
Asger
 

Re:TClientDataSet::LoadFromStream seeks to end of stream?

"Jason Cipriani" < XXXX@XXXXX.COM >wrote in message
Quote
When I call TClientDataSet::LoadFromStream() to read the
data from a TFileStream (opened in dfOpenRead mode), the
file position is set to the end of the file when LoadFromStream()
returns
That is where it is supposed to be. Unless the TStream data payload
explicitally states how many bytes to read (which is not the case in
LoadFromStream() for most classes), LoadFromStream() has to read to the end
of the TStream. It cannot make any assumptions about how you will use the
TStream after it exits, so it has to leave the Position where the reading
finished. It is your responsibility to seek the TStream manually if needed
before doing other things with it.
Quote
I can load the remaining objects from the file by calling
TFileStream::Seek
to move to the end of the TClientDataSet, but that only works if I know
the exact size of the data in the file, which I don't under normal
circumstances.
Yes, you do - the TStream::Size property. But what what else are you trying
to load exactly? You can't have non-TClientDataSet data in the same TStream
following the TClientDataSet data, unless you are managing streaming
manually and can thus insert a size value into the payload. Otherwise, as
you have already seen, TClientDataSet::LoadFromStream() loads data to the
very end of the TStream, leaving nothing left to load for anything else.
Quote
Here is my test program. It's a C++ console application with VCL support:
You are saving two separate TClientDataSet payloads to the same TStream.
There is no delimiter stored between the two payloads. When you load the
TStream back later, there is no way for TClientDataSet::LoadFromStream() to
know where one payload ends and the next begins. As far as TClientDataSet
is considered, the entire TStream is one big record set, so the first
TClientDataSet will end up receiving all of the records and the second
TClientDataSet will receive none at all.
To do what you are attempting requires a lot more manual work to manage the
TStream content more selectively. Now, it just so happens that
TClientDataSet already has functionality to read/write the actual payload
size from/to the TStream, but that only applies to DFM streaming. You don't
have access to invoke those extra functions directly, so you have a choice:
1) use TStream::ReadComponent() and TStream::WriteComponent() (which will
also store/load the TClientDataSet properties in addition to the record
payload) instead of TClientDataSet::LoadFromStream() and
TClientDataSet::SaveToStream():
int main (int argc, char **argv)
{
// create and populate 2 datasets
TClientDataSet *DS1 = BuildDataSet(true);
TClientDataSet *DS2 = BuildDataSet(true);
DumpDataSet(DS1, "DS1 [new]");
DumpDataSet(DS2, "DS2 [new]");
// save them both to a stream, one after the other
TFileStream *data = new TFileStream("testdata.dat", fmCreate);
printf("%Li\n", data->Position);
data->WriteComponent(DS1);
printf("%Li\n", data->Position);
data->WriteComponent(DS2);
printf("%L\n", data->Position);
delete data;
// now, create brand spankin' new datasets
delete DS2;
delete DS1;
DS1 = BuildDataSet(false);
DS2 = BuildDataSet(false);
DumpDataSet(DS1, "DS1 [reset]");
DumpDataSet(DS2, "DS2 [reset]");
// then load them both from that file, same order:
data = new TFileStream("testdata.dat", fmOpenRead);
printf("%Li\n", data->Position);
data->ReadComponent(DS1);
printf("%Li\n", data->Position);
data->ReadComponent(DS2);
printf("%Li\n", data->Position);
delete data;
DumpDataSet(DS1, "DS1 [loaded]");
DumpDataSet(DS2, "DS2 [loaded]");
delete DS2;
delete DS1;
printf("finished\n");
getchar();
return 0;
}
2) use additional TStream objects to manually mimic what the DFM does
internally:
int main (int argc, char **argv)
{
// create and populate 2 datasets
TClientDataSet *DS1 = BuildDataSet(true);
TClientDataSet *DS2 = BuildDataSet(true);
DumpDataSet(DS1, "DS1 [new]");
DumpDataSet(DS2, "DS2 [new]");
TMemoryStream *tmp = new TMemoryStream;
__int64 size;
// save them both to a stream, one after the other
TFileStream *data = new TFileStream("testdata.dat", fmCreate);
printf("%Li\n", data->Position);
DS1->SaveToStream(tmp);
size = tmp->Size;
data->WriteBuffer(&size, sizeof(__int64));
data->CopyFrom(tmp, 0);
tmp->Clear();
printf("%Li\n", data->Position);
DS2->SaveToStream(tmp);
size = tmp->Size;
data->WriteBuffer(&size, sizeof(__int64));
data->CopyFrom(tmp, 0);
printf("%L\n", data->Position);
delete data;
// now, create brand spankin' new datasets
delete DS2;
delete DS1;
DS1 = BuildDataSet(false);
DS2 = BuildDataSet(false);
DumpDataSet(DS1, "DS1 [reset]");
DumpDataSet(DS2, "DS2 [reset]");
// then load them both from that file, same order:
data = new TFileStream("testdata.dat", fmOpenRead);
printf("%Li\n", data->Position);
data->ReadBuffer(&size, sizeof(__int64));
tmp->Clear();
tmp->CopyFrom(data, size);
tmp->Position = 0;
DS1->LoadFromStream(tmp);
printf("%Li\n", data->Position);
data->ReadBuffer(&size, sizeof(__int64));
tmp->Clear();
tmp->CopyFrom(data, size);
tmp->Position = 0;
DS2->LoadFromStream(tmp);
printf("%Li\n", data->Position);
delete data;
delete tmp;
DumpDataSet(DS1, "DS1 [loaded]");
DumpDataSet(DS2, "DS2 [loaded]");
delete DS2;
delete DS1;
printf("finished\n");
getchar();
return 0;
}
3) store each TClientDataSet in its own file instead of trying to merge them
together.
Gambit
 

{smallsort}

Re:TClientDataSet::LoadFromStream seeks to end of stream?

Hey, great info, thanks guys. So I guess what it comes down to is:
"Asger Joergensen" < XXXX@XXXXX.COM >wrote:
Quote
I don't think the intension of LoadFromFile was to load more
then one item Object from the stream, in other words it expect
that all the data in the stream is to be read.
"Remy Lebeau (TeamB)" < XXXX@XXXXX.COM >wrote:
As far as TClientDataSet is considered, the entire TStream is one big
record set.
I can clearly tell where one TClientDataSet starts and stops, both in
dfBinary and dfXML formats, but it makes sense why it's not implemented that
way. I am a little sad it doesn't store the payload size in the stream,
though.
Quote
1) use TStream::ReadComponent() and TStream::WriteComponent() (which will
also store/load the TClientDataSet properties in addition to the record
payload) instead of TClientDataSet::LoadFromStream() and
TClientDataSet::SaveToStream():
I'm assuming the main disadvantage to this, then, is the properties in the
data file would end up overriding whatever I have set in the designer, so if
I change something about the TClientDataSet and a user opens a data file
saved from a previous version of the application, it will mess with all my
changes?
Quote
2) use additional TStream objects to manually mimic what the DFM does
internally:
That's an interesting example. I guess you could also accomplish the same
thing playing some tricks with Seek/Position as long as the TStream
supported it. For me, the big disadvantage with this one is I'm looking into
TClientDataSets as sort of a good all-purpose way to -simplify- storing
application data, because I'm sick of always coming up with my own weird
little file formats and writing the code to deal with them.
This info really helps me decide what to do, then. I guess I'll just have to
find other solutions and handle things case-by-case depending on what kinds
of data I end up storing in the files.
As far as the TClientDataSets go, I was basically trying to store multiple
database "tables" in a single data file. But, I just finished working
through a tutorial that talks about nested data sets, which let me store
multiple TClientDataSets in a single one, and it will handle writing all the
data to a single file with just one SaveToFile() call, which is pretty
handy. That makes more sense for this particular application.
Thanks a lot again, I really appreciate the time you took to answer that so
thoroughly. If I run into you at a bar some day, I'll buy you a drink.
Jason
"Asger Joergensen" < XXXX@XXXXX.COM >wrote:
Quote
P.s.p.s. please break Your text lines.
Oops... sorry. I used the web interface to post that one and I guess it
didn't put line breaks in.
 

Re:TClientDataSet::LoadFromStream seeks to end of stream?

"Jason Cipriani" < XXXX@XXXXX.COM >wrote in message
Quote
I'm assuming the main disadvantage to this, then, is the properties in
the data file would end up overriding whatever I have set in the designer,
so if I change something about the TClientDataSet and a user opens a
data file saved from a previous version of the application, it will mess
with all my changes?
Potentially, yes.
Gambit