Board index » cppbuilder » Re: parallel bcc32 build tool

2005-08-18 01:07:05 AM
cppbuilder106
"Roddy Pratt" <roddy at spam fritter dot com>writes:
sure if I should have been trying to cheer you up or not. If you're
not using it, then I was trying to cheer you up. <g>
Seriously, all I know is that slow IDE builds are due to a bug in the
IDE. I'm also not sure if it's limited only for background compiles
or not, though background compilations certainly seem worse, IIRC, so
it could be.
The IDE compiles differently, because it uses a .dll version of the
compiler, and doesn't create any new process. It seems this has
potential of being faster, because it avoids all the startup time and
can keep the PCH cached. However, for whatever reason, this improved
performance is not realized.
--
Chris (TeamB);
 

"Rodrigo Gómez" < XXXX@XXXXX.COM >wrote in message
Now to fix it. :)
Jonathan
 

Jonathan Benedicto wrote:
all three do not consume CPU time. And the compilation is on a still stand.
--
Regards,
Andreas Hausladen
 

{smallsort}

Jonathan Benedicto wrote:
headers many times it makes more sense to figure out how to assign source files to
the two compiler instances.
Some ideas toward that end:
- Measure of the compile times for all files. Run a Build as I previously described.
Do it twice with the first time to get the precompiled headers generated. Then on the
second pass record the times. Save those weightings to use to assign files in an
algorith to give the processors approximately equal amounts of work to do.
- Check .cpp file sizes on the disk to figure out which are bigger. But this misses
out on the cost of STL headers and other things that can eat into performance.
- Let the user assign files in lists to groups.
- Let the user create a table of weightings for how long each file takes to compile.
You could generate a compile time report on each file to help them do that.

Well.. it's the same... :(
Now I see the two threads:
C:\tsts\3.0\kernel>make -fDataModuleTS3.mak -B -d BCC32="mtbcc32 -debug"
MAKE Version 5.2 Copyright (c) 1987, 2000 Borland
C:\CBuilder6\BIN\mtbcc32 -debug -Od -H=c:\temp\objs\tsts3\datamodule\pch
.csm -Hc -Q -Vx -Ve -ff -X- -r- -a8 -6 -b- -k -y -v -vi- -c -tWM -w-par -Ic:\F
astReport3\LibBCB6;C:\ts\GenLicencias;..\mensajeria;..\programacion;..\varios;..
\ventanas;K:\LockBox\hpp6;C:\CBuilder6\Projects;C:\CBuilder6\include;C:\CBuilder
6\include\vcl;K:\NexusDB1\CBuilder6;K:\RxLib275BCB6\Units;k:\TMSPack\hpp6;K:\Del
phiGems\VirtualTreeview\hpp6;"K:\DelphiGems\Theme
Manager\hpp6";k:\bc2\tol;k:\bc
2\fechas;C:\jcl\include\c6;c:\ts\hasp;C:\HASPCD12\win32api\C;C:\DevExp.VCL\Libra
ry\CBuilder6;c:\kbmMemTable4\c6;"k:" -D_DEBUG;USAR_HASP;_RTLDLL;NO_STRICT;USEPAC
KAGES -nc:\temp\objs\tsts3\datamodule\ .\DataModuleTS3.CPP .\AutoDebug.CPP
.\Aud
itoria.CPP .\VErrorDebug.CPP .\VDataModulePrincipalTS3.CPP
..\ventanas\VEntrada.
CPP ..\varios\UtilVarios.CPP .\ConsErrores.CPP
..\varios\UtilConfiguracion.CPP .
\UtilUsuarios.CPP ..\mensajeria\UtilMensajeria.CPP .\ManejadorLicencias.CPP
.\VD
ataModuleReportes.CPP .\PermisosUsuarios.CPP
Compiling...
Thread count: 2
Debug: true
Equalize: false - 1 files at a time.
Thread 2368:
bcc32 -Od -H=c:\temp\objs\tsts3\datamodule\pc-0h.csm -Hc -Q -Vx -Ve
-ff -X- -r- -a8 -6 -b- -k -y -v -vi- -c -tWM -w-par -Ic:\FastReport3\LibBCB6;C:
\ts\GenLicencias;..\mensajeria;..\programacion;..\varios;..\ventanas;K:\LockBox\
hpp6;C:\CBuilder6\Projects;C:\CBuilder6\include;C:\CBuilder6\include\vcl;K:\Nexu
sDB1\CBuilder6;K:\RxLib275BCB6\Units;k:\TMSPack\hpp6;K:\DelphiGems\VirtualTreevi
ew\hpp6;K:\DelphiGems\Theme
Manager\hpp6;k:\bc2\tol;k:\bc2\fechas;C:\jcl\include
\c6;c:\ts\hasp;C:\HASPCD12\win32api\C;C:\DevExp.VCL\Library\CBuilder6;c:\kbmMemT
able4\c6;k: -D_DEBUG;USAR_HASP;_RTLDLL;NO_STRICT;USEPACKAGES -nc:\temp\objs\tsts
3\datamodule\ .\DataModuleTS3.CPP .\AutoDebug.CPP .\Auditoria.CPP
.\VErrorDebug.
CPP .\VDataModulePrincipalTS3.CPP ..\ventanas\VEntrada.CPP
..\varios\UtilVarios.
CPP
Thread 204:
bcc32 -Od -H=c:\temp\objs\tsts3\datamodule\pc-1h.csm -Hc -Q -Vx -Ve
-ff -X- -r- -a8 -6 -b- -k -y -v -vi- -c -tWM -w-par -Ic:\FastReport3\LibBCB6;C:\
ts\GenLicencias;..\mensajeria;..\programacion;..\varios;..\ventanas;K:\LockBox\h
pp6;C:\CBuilder6\Projects;C:\CBuilder6\include;C:\CBuilder6\include\vcl;K:\Nexus
DB1\CBuilder6;K:\RxLib275BCB6\Units;k:\TMSPack\hpp6;K:\DelphiGems\VirtualTreevie
w\hpp6;K:\DelphiGems\Theme
Manager\hpp6;k:\bc2\tol;k:\bc2\fechas;C:\jcl\include\
c6;c:\ts\hasp;C:\HASPCD12\win32api\C;C:\DevExp.VCL\Library\CBuilder6;c:\kbmMemTa
ble4\c6;k: -D_DEBUG;USAR_HASP;_RTLDLL;NO_STRICT;USEPACKAGES -nc:\temp\objs\tsts3
\datamodule\ .\ConsErrores.CPP ..\varios\UtilConfiguracion.CPP
.\UtilUsuarios.CP
P ..\mensajeria\UtilMensajeria.CPP .\ManejadorLicencias.CPP
.\VDataModuleReporte
s.CPP .\PermisosUsuarios.CPP
But after this, no other message. In the task manager there is no bcc32
running now, but it still doesn't return from the call to make... and
mtbcc32 is still present in the task manager.
Changing for threads1, 2, 4 etc doesn't seem to help... bcc32 ends from the
task list but mtbcc32 seems to fail realizing that.
Regards,
--
Rodrigo Gómez
www.rodrigogomez.com.mx/gallery/
"Jonathan Benedicto" < XXXX@XXXXX.COM >escribi?en el mensaje

Thats what I get:
z:\Qt\bor\qt-4\src\tools\rcc>mtbcc32 -debug -c -q -tWR -5 -d -O2 -w -w-hid
-tWC -DUNICODE -DQT_RCC -DQT_LITE_UNICODE -DQT_NO_DATASTREAM
-DQT_NO_THREAD -DQT_NO_QOBJECT -DQT_NO_UNICODETABLES -DQT_NO_COMPONENT
-DQT_NODLL -I"..\..\corelib\arch\generic" -I"Z:/Qt/bor/qt-4/include" -I"."
-I"Z:/Qt/bor/qt-4/include/QtCore" -I"Z:/Qt/bor/qt-4/include/QtXml"
-I"..\..\3rdparty\zlib" -I"." -I"Z:\Qt\bor\qt-4\mkspecs\win32-borland"
-ntmp\obj\debug_shared\ ..\..\3rdparty\zlib\trees.c
Compiling...
Thread count: 2
Debug: true
Equalize: false - 1 files at a time.
Thread 7416: bcc32 -c -q -tWR -5 -d -O2 -w -w-hid -tWC -DUNICODE -DQT_RCC
-DQT_LITE_UNICODE -DQT_NO_DATASTREAM -DQT_NO_THREAD -DQT_NO_QOBJECT
-DQT_NO_UNICODETABLES -DQT_NO_COMPONENT -DQT_NODLL
-I..\..\corelib\arch\generic -IZ:/Qt/bor/qt-4/include -I.
-IZ:/Qt/bor/qt-4/include/QtCore -IZ:/Qt/bor/qt-4/include/QtXml
-I..\..\3rdparty\zlib -I. -IZ:\Qt\bor\qt-4\mkspecs\win32-borland
-ntmp\obj\debug_shared\ ..\..\3rdparty\zlib\trees.c
****{*word*154} **** CTRL+C
--
Regards,
Andreas Hausladen
 

I think I know why it's failing, at least the compilation.
If you see the command line passed, there is an include path "Theme manager"
that is not beign passed with ", so it
1) screws the include paths, so the errors reported by the 1.1 version
2) tryes to compile a Manager\... unit, that obviously doesn't exist.
The strange thing here is that it's not reported anywhere.
I'll see if I can do a testcase to send to you so you can try.
It's really easy to reproduce this. In any makefile, add to the include
paths a path surrounded by "" that contain spaces. That at least will show
this problem. The other that there is nothing reported back I guess it's
going to be a little bit difficult to reproduce :-/
Regards,
--
Rodrigo Gómez
www.rodrigogomez.com.mx/gallery/
"Rodrigo Gómez" < XXXX@XXXXX.COM >escribi?en el mensaje

The main "problem" is in make.exe which mtbcc32.exe cannot control.
A very complex solution would be to replace the make.exe so independent
dependency lists could be executed twice, like the "gnumake -j2" command.
--
Regards,
Andreas Hausladen
 

Great!
Let me know when you have it so I can try here :-)
Regards,
--
Rodrigo Gómez [NDX]
rodrigo.gomez at ndx dot nexusdb dot com
www.rodrigogomez.com.mx/gallery/
"Jonathan Benedicto" < XXXX@XXXXX.COM >escribi?en el mensaje

--
Rodrigo Gómez
www.rodrigogomez.com.mx/gallery/
"Andreas Hausladen" < XXXX@XXXXX.COM >escribi?en el
mensaje news: XXXX@XXXXX.COM ...

"Chris Uzdavinis (TeamB)" < XXXX@XXXXX.COM >wrote in message
The times quoted weren't mine. But I also see a>5x speedup if I build from
the command line rather than from the IDE, and I don't use BG compiles.
If the bccide interface was even half-documented, it might be possible to
get the DLL to use the command line compiler instead....?
- Roddy
 

Hi Roddy,
Bye, Oliver
 

"Roddy Pratt" <roddy at spam fritter dot com>wrote in message
have altered my settings while I slept.
 

I have written an alternative tool (fbcc32.exe) which uses another file
distribution algorithm. (and does not hang on my computer). Because I'm
more advanced in writing Delphi application, it is written in Delphi but
that should be no problem.
It tries to catch the problem with files that take long to compile by not
dividing the all files to the threads. It divides only 75% of the files.
The rest is given to idle thread. So fast threads can get more files while
slow threads can do their job.
If there is only one file or one thread the main thread does the job
itself. The compiler output is written to the console when one file is
compiled (parses the compiler's output) or the thread terminates.
See b.p.attachment for the download (includes executable, source code and
a build.bat). Qt-4 is now compiled really fast with fbcc32.
--
Regards,
Andreas Hausladen
 

This works fine with my projects, it compiles the projects I've having
problems with mtbcc32 without problems.
Now... I'm seeing that for a project, one of the bcc32 running can't write
te precompiled headers, maybe changing the filename as mtbcc32 does solves
this problem?
Regards,
--
Rodrigo Gómez
www.rodrigogomez.com.mx/gallery/
"Andreas Hausladen" < XXXX@XXXXX.COM >escribi?en el
mensaje news:4303bbc3$ XXXX@XXXXX.COM ...