scale/pan using mapping modes: VCL Issues

I have put together a small app using mapping modes / viewport / window
extents to effect panning and zooming.  It works just fine when I draw
directly on the canvas that I'm manpulating, but any VCL components placed
on the panel are not affected by the changes.  Is it possible to modify the
behavior of the canvas / panel / etc to apply the scaling to the VCL

PS - I do NOT want to use the scaleBy method to effect zooming.
Implementation is too slow.  Same goes for using a scroll box for panning.

ANY help would be GREATLY appreciated.

- Steve