cmSrcInvert different with 16 and 256 colours

I am creating an application where I need to do XOR copying of bitmaps
from one canvas to another. I have been developing on a machine using 256
colours and have just gone to test on a standard 16 colour VGA machine.
The copyrect function using the cmSrcInvert gives different results with
the different colour depths.

Using the code below, I get what I consider to be the correct result on
the 256 colour machine, ie the text turns out blue, but on the 16 colour
machine it turns out yellow.

Can anybody point me towards the cause of this?

var
  rc: TRect;
begin
  with image1.canvas do begin
    brush.style := bsSolid;
    brush.color := clBlack;
    rc := Rect(0,0,Width,Height);
    fillrect(rc);
    {pen.style := psClear;}
    font.color := clOlive;
    textout(10,10,'Sample text');
  end;
  with image2.canvas do begin
    brush.style := bsSolid;
    brush.color := clSilver;
    fillrect(Rect(0,0,Width,Height));
    copymode := cmSrcInvert;
    CopyRect(rc, image1.canvas, rc);
  end;

end;

--
David Willis