Board index » off-topic » TClientDataSet truncated data

TClientDataSet truncated data


2004-05-17 06:38:03 AM
off-topic19
Hiya.
Perhaps someone here will have some comments or suggestions on a problem
that we have encountered at work. We are caching data using a
TclientDataSet. It works well, except when it fails :( This happens when
the returned result set is over a size limit, somewhere around 47Mb. The
failure is quiet - no exception, just a truncated results file,
sometimes truncated in mid-row. We are using Delphi 7, but have not yet
applied the patch. Sample code is below. Any comments?
unit uCDSTest;
{ the problem is that there is an upper limit to the size of data
that you can store this way. Somewhere between 45 and 50Mb it cuts off
in mid data row. No error message, no exception, no indication
that anything has gone wrong
But the file is incomplete. Data is Missing.
10000 rows in clientdataset, expected 10000
200000 rows in clientdataset, expected 200000
487163 rows in clientdataset, expected 500000
478157 rows in clientdataset, expected 800000
478157 rows in clientdataset, expected 900000
there are more rows than this in the query
}
interface
uses
Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls,
Forms,
Dialogs, MemDS, VirtualTable, Provider, DB, DBClient, StdCtrls, DBAccess,
Ora, DBTables;
type
TForm1 = class(TForm)
ClientDataSet1: TClientDataSet;
DataSetProvider1: TDataSetProvider;
OraSession1: TOraSession;
OraQuery1: TOraQuery;
btnGo: TButton;
mResults: TMemo;
eRows: TEdit;
Label1: TLabel;
procedure FormCreate(Sender: TObject);
procedure btnGoClick(Sender: TObject);
procedure eRowsChange(Sender: TObject);
private
public
end;
var
Form1: TForm1;
implementation
{$R *.dfm}
function DataSetManualRowCount(aDataset: TDataset): integer;
begin
Result := 0;
aDataSet.First;
while not aDataSet.Eof do
begin
inc(Result);
aDataSet.Next;
end;
end;
procedure TForm1.FormCreate(Sender: TObject);
begin
OraSession1.Connect;
eRowsChange(nil);
end;
procedure TForm1.btnGoClick(Sender: TObject);
begin
OraQuery1.Open;
DataSetProvider1.DataSet := OraQuery1;
ClientDataSet1.ProviderName := DataSetProvider1.Name;
ClientDataSet1.Open;
ClientDataSet1.SaveToFile('c:\temp\tempcds.txt', dfXML);
Application.ProcessMessages;
mResults.lines.Add(IntToStr(DataSetManualRowCount(ClientDataSet1)) +
' rows in clientdataset, expected ' + eRows.Text);
ClientDataSet1.Close;
OraQuery1.Close;
end;
procedure TForm1.eRowsChange(Sender: TObject);
begin
{ a random big query, cartesian product
on doing the equivalent count(*) queries,
select count(*) from table1
select count(*) from table2
got a count of 584414 * 20391 = 11 916 785 874
}
OraQuery1.SQL.Text := 'select s.* from table1 s, table2 e where
rownum <= ' + eRows.Text;
end;
end.
object Form1: TForm1
Left = 250
Top = 107
Width = 421
Height = 436
Caption = 'Form1'
Color = clBtnFace
Font.Charset = DEFAULT_CHARSET
Font.Color = clWindowText
Font.Height = -11
Font.Name = 'MS Sans Serif'
Font.Style = []
OldCreateOrder = False
OnCreate = FormCreate
PixelsPerInch = 96
TextHeight = 13
object Label1: TLabel
Left = 24
Top = 16
Width = 22
Height = 13
Caption = 'rows'
end
object btnGo: TButton
Left = 24
Top = 80
Width = 75
Height = 25
Caption = 'Go'
TabOrder = 0
OnClick = btnGoClick
end
object mResults: TMemo
Left = 24
Top = 120
Width = 345
Height = 265
TabOrder = 1
end
object eRows: TEdit
Left = 24
Top = 40
Width = 121
Height = 21
TabOrder = 2
Text = '100000'
OnChange = eRowsChange
end
object ClientDataSet1: TClientDataSet
Aggregates = <>
Params = <>
ProviderName = 'DataSetProvider1'
Left = 104
Top = 264
end
object DataSetProvider1: TDataSetProvider
Constraints = False
Exported = False
Options = [poReadOnly, poDisableInserts, poDisableEdits,
poDisableDeletes]
Left = 72
Top = 264
end
object OraSession1: TOraSession
ConnectPrompt = False
Username = 'merit_test'
Password = 'merit_test'
Server = 'cms_dev3'
Connected = True
Left = 72
Top = 208
end
object OraQuery1: TOraQuery
Session = OraSession1
SQL.Strings = (
'')
Left = 112
Top = 208
end
end
 
 

Re:TClientDataSet truncated data

Bill Todd (TeamB) wrote:
Quote
TClientDataSet was designed to hold a few hundred, or at most a few
thousand records. It is not intended for the amount of data that you
are trying to load.
But all the same - there was no exception, no error state, just the
object silently discarding and corrupting data. Are we really doing
nothing wrong in this example?
Quote
Many users recommend KbmMemTable for larger
volumes of data.

I will look at that, thanks.
Anthony