Board index » delphi » Multi-threaded app on a multi-processor system exiting mysteriously

Multi-threaded app on a multi-processor system exiting mysteriously

Hi, I have an application that spawns about 15 threads, each thread having a
SQL connection via the BDE.  I'm running my app on a dual processor machine.
At first I had problems, I beleive due to the fact that Windows 2000 was
allocating the 15 threads onto different processors, and I understand the
BDE doesn't like that.  I used a call to SetProcessAffinityMask() and
SetThreadAffinityMask() in my main program and threads, respectively, to
force them to all use processor 1.  This seemed to work good, looking at the
task manager show that the first processor was peaking while the second
looked pretty idle.

Unfortunately, my application still exits mysteriously after running for a
period of time.  I'm basically load testing it now, forcing it to try to
insert a duplicate record into a MS SQL table over and over, about 10,000
times.  Before my Affinity call changes, the app would exit out after about
a minute.  With the Affinity calls in, it can go through about 7,000 of the
insert fails, then it seems to finally give up and exit.

So it seems that I've almost corrected my problem, but not quite.  I've
since tried moving two of the threads that don't access the BDE to use the
second processor, and I'm testing that now.

Needless to say, this application runs fine on a single processor machine.

Anybody have any thoughts/comments/suggestions?  Any help will be greatly
appreciated, thanks.

Martin Binder

 

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Quote
Martin Binder wrote:

 > Needless to say, this application runs fine on a single processor
 > machine.

        My guess is that it doesn't, and that you just haven't found the problem
yet (or pushed your app hard enough).  Threading conflicts show up
*much, much* more quickly on multiprocessor machines.

Quote
> Anybody have any thoughts/comments/suggestions?  Any help will be greatly
> appreciated, thanks.

        It does sound like a threading problem.  My suggestion would be to try
and reduce it to the tightest possible test case.  When several people
did that with the IB driver they discovered that the driver itself was
not thread-safe.  I haven't heard any SQL Server complaints, but if your
test case doesn't work try substituting the ADO components and see if
things improve.

        I presume you are already using a separate TSession for each thread.  It
would be worth spending some time to make absolutely certain that each
DB component is hooked up to the appropriate session for its thread.  I
recommend doing this in code.

        HTH,

        -Craig

--
  Craig Stuntz (TeamB) Vertex Systems Corp. Columbus, OH
      Delphi/InterBase WebLog: http://delphi.weblogs.com
      InterBase PLANalyzer (Free IB optimization tool):
           http://delphi.weblogs.com/IBPLANalyzer

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Quote
> Threading conflicts show up *much, much* more quickly on

multiprocessor machines.

Unfortunately, we found this out the hard way a few years ago.  If you
are going to write a multi-threaded application, you MUST test it on a
single- AND multi-processor system.  A single-processor system behaves
completely different when it comes to dealing with threading than a
multi-processor system.  If you are going to do development for a
mulit-threaded app, you have to get a multi-processor system for the
programmer to use.  This is why Danny Thorpe and others in R&D at
Borland have a multi-processor box for development.

I'm assuming you are using Delphi.  What version are you using?  There
were some serious bugs in D4 threading [they were fixed in D5] that you
didn't really see until you ran on a multi-processor box.  In our case
with our web app server, we could beat the crud out of it (stress/load
testing) on a single-procesoor box, but it would fail with 2 users on a
multi-processor box.

Quote
> > Anybody have any thoughts/comments/suggestions?  Any help will be

greatly appreciated, thanks.

In addition to what Craig said, I'd also check your BDE settings.
You'll probalby want to increase the MaxBufSize, MemSize, SharedMemSize
(especially this one), etc.  See the BDE help for details.

HTH,

David R.

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Quote
David R. Robinson wrote:
> I'm assuming you are using Delphi.  What version are you using?

We are using Delphi 5 Update Pack 1.

Quote
> In addition to what Craig said, I'd also check your BDE settings.
> You'll probalby want to increase the MaxBufSize, MemSize, SharedMemSize
> (especially this one), etc.  See the BDE help for details.

I will check these settings.  Thanks a lot for the help!

Martin Binder

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Quote
Craig Stuntz wrote:
> My guess is that it doesn't, and that you just haven't found the problem
> yet (or pushed your app hard enough).  Threading conflicts show up
> *much, much* more quickly on multiprocessor machines.

So are you saying that an app with multiple threads, all using the BDE, is
OK on a multi-processor machine?  I've read a lot of posts that say that
isn't true.  From what I've read, all threads within an app that use the BDE
MUST use the same processor.

Quote
> I presume you are already using a separate TSession for each thread.  It
> would be worth spending some time to make absolutely certain that each
> DB component is hooked up to the appropriate session for its thread.  I
> recommend doing this in code.

Yes, I've gone through checking the Session about a hundred times, I don't
think that's the problem.

Martin Binder

Quote
"Craig Stuntz" <cstuntz@no_spam.vertexsoftware.com> wrote in message

news:3C1FCE8D.1000605@no_spam.vertexsoftware.com...
Quote

> Martin Binder wrote:

>  > Needless to say, this application runs fine on a single processor
>  > machine.

> My guess is that it doesn't, and that you just haven't found the problem
> yet (or pushed your app hard enough).  Threading conflicts show up
> *much, much* more quickly on multiprocessor machines.

> > Anybody have any thoughts/comments/suggestions?  Any help will be
greatly
> > appreciated, thanks.

> It does sound like a threading problem.  My suggestion would be to try
> and reduce it to the tightest possible test case.  When several people
> did that with the IB driver they discovered that the driver itself was
> not thread-safe.  I haven't heard any SQL Server complaints, but if your
> test case doesn't work try substituting the ADO components and see if
> things improve.

> I presume you are already using a separate TSession for each thread.  It
> would be worth spending some time to make absolutely certain that each
> DB component is hooked up to the appropriate session for its thread.  I
> recommend doing this in code.

> HTH,

> -Craig

> --
>   Craig Stuntz (TeamB) Vertex Systems Corp. Columbus, OH
>       Delphi/InterBase WebLog: http://delphi.weblogs.com
>       InterBase PLANalyzer (Free IB optimization tool):
>            http://delphi.weblogs.com/IBPLANalyzer

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Quote
Martin Binder wrote:

> So are you saying that an app with multiple threads, all using the BDE, is
> OK on a multi-processor machine?  

        No, that isn't what I'm saying.  What I said is that if something fails
on a multiprocessor machine, it will probably fail on a single processor
machine sooner or later.

Quote
> I've read a lot of posts that say that
> isn't true.  From what I've read, all threads within an app that use the BDE
> MUST use the same processor.

        At least one of the SQL Links drivers for the BDE (the InterBase driver,
to be specific) is not thread safe.  There could be others.  The BDE
itself could have multithreading problems.  Affining the app to a single
processor could hide these problems, if they exist.

        If you want to do multithreaded access to a given DB, you need to start
by writing a simple test case that is designed to surface multithreading
issues and convince yourself that your test case is safe by running it
on a multiprocessor machine.  I've done this with IBX and I'm convinced
that IBX, used properly, is safe.

        HTH,

        -Craig

--
  Craig Stuntz (TeamB) Vertex Systems Corp. Columbus, OH
      Delphi/InterBase WebLog: http://delphi.weblogs.com
      InterBase PLANalyzer (Free IB optimization tool):
           http://delphi.weblogs.com/IBPLANalyzer

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Hello Martin !

I am just back from development of multy threading
application <g>. Some thoughts to share:
1) If you does not have threading problems on single
processor box - it is time to debug application of multy
processor box. In my case app. simply dies after few
seconds ...
2) To test and proof, that threading application works
well - use stress testing, which will be bounce thread
priorities, send requests as fast as it is possible, etc
3) To analyze threading problems i found just single
solution, which work for me - tracing. Sometimes i have
tracing points after each line of code <g>. If trace will
have good structure, then you can use Excel to analyze
your trace. Yes, Excel is my debugging tool <g>
4) So full debugging process will be:
- add tracing facilities to your application
- run stress tests
- analyze logs
- fix bugs
- and repeat from stress testing ...
5) Interact with main thread (user interface) from other
threads only through synchronize method. Sometimes,
single not synchronized call will kill your application.

SetProcessAffinityMask will not make you happy, because:
- some system services will still run on another processors.
And if your application interacts with them, then - you will have
still many processors
- good threading application must work on any number of
processors

Regards,
Dmitry

--
===========================================
Dmitry L. Arefiev, director of gs-soft.ru ltd.
Solutions for successful companies

Author of NCOCI8 - Freeware Delphi to Oracle8i direct access

ICQ: 50741007
EMail: daref...@gs-soft.ru
Company: http://www.gs-soft.ru
NCOCI8: http://www.da-soft.com

Quote
"Martin Binder" <mbin...@gatewayticketing.com> wrote in message news:3c1fba70$1_2@dnews...
> Hi, I have an application that spawns about 15 threads, each thread having a
> SQL connection via the BDE.  I'm running my app on a dual processor machine.
> At first I had problems, I beleive due to the fact that Windows 2000 was
> allocating the 15 threads onto different processors, and I understand the
> BDE doesn't like that.  I used a call to SetProcessAffinityMask() and
> SetThreadAffinityMask() in my main program and threads, respectively, to
> force them to all use processor 1.  This seemed to work good, looking at the
> task manager show that the first processor was peaking while the second
> looked pretty idle.

> Unfortunately, my application still exits mysteriously after running for a
> period of time.  I'm basically load testing it now, forcing it to try to
> insert a duplicate record into a MS SQL table over and over, about 10,000
> times.  Before my Affinity call changes, the app would exit out after about
> a minute.  With the Affinity calls in, it can go through about 7,000 of the
> insert fails, then it seems to finally give up and exit.

> So it seems that I've almost corrected my problem, but not quite.  I've
> since tried moving two of the threads that don't access the BDE to use the
> second processor, and I'm testing that now.

> Needless to say, this application runs fine on a single processor machine.

> Anybody have any thoughts/comments/suggestions?  Any help will be greatly
> appreciated, thanks.

> Martin Binder

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Quote
Craig Stuntz wrote:
> If you want to do multithreaded access to a given DB, you need to start
> by writing a simple test case that is designed to surface multithreading
> issues and convince yourself that your test case is safe by running it
> on a multiprocessor machine.

Thanks for the suggestions, I will write a test program to prove if the BDE
is truly multi-processor, multi-thread safe.

I don't think I mentioned this before, but my app only seems to go down on a
multi-processor machine after suffering many BDE exceptions.  I have it so
I'm having multiple threads try to insert a record into a SQL table that
violates the primary key.  Repeated violations of this primary key is what I
believe is really causing my app to exit mysteriously.  Otherwise, without
violating that primary key, I think my app runs fine on a multi-processor
machine.

Perhaps my test program will shed some light on this.  Thanks for all the
help, I'll let you know.

Martin Binder

Re:Multi-threaded app on a multi-processor system exiting mysteriously


I had a multi-threaded, BDE-utilizing application that ran for weeks on end
on a Win95 workstation.  As soon as it went to a multi-processor machine on
NT4.x, it couldn't go half a day without a crash.

As soon as I updated the BDE from 5.x to 5.10, it worked for weeks straight
and I haven't heard a complaint since (two + years ago).  I would update the
BDE.  I can vouch for version 5.10 and its suitability for multi-threaded
apps on multi-processor machines.

Regards,
Rob

Quote
"Craig Stuntz" <cstuntz@no_spam.vertexsoftware.com> wrote in message

news:3C20DD84.1000807@no_spam.vertexsoftware.com...
Quote

> Martin Binder wrote:

> > So are you saying that an app with multiple threads, all using the BDE,
is
> > OK on a multi-processor machine?

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Quote
Robby Tanner wrote:
>I would update the BDE.  I can vouch for version 5.10 and its
>suitability for multi-threaded apps on multi-processor machines.

Thanks Robby, but we're running BDE 5.1.1.

Martin Binder

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Quote
"Martin Binder" <mbin...@gatewayticketing.com> wrote in message

news:3c21f0d1$1_1@dnews...

Quote
> Robby Tanner wrote:

> >I would update the BDE.  I can vouch for version 5.10 and its
> >suitability for multi-threaded apps on multi-processor machines.

> Thanks Robby, but we're running BDE 5.1.1.

Which is not to say that a bug wasn't reintroduced.  Possibly move up to
5.2?
The symptoms you describe are identical to the ones we encountered.

Rob

Quote
> Martin Binder

Re:Multi-threaded app on a multi-processor system exiting mysteriously


I would just like to add some closure to this thread, and thank everyone who
participated.  We have determined that using the BDE with a MS SQL back-end
on a multi-processor machine using multiple threads is NOT stable when
encountering multiple key violations.  To test this, we wrote a VERY simple
program that spawns 15 threads, each of which had its own session, database
and query.  Each one of these threads were continually trying to insert a
record into an MS SQL table that violated the unique index on the Table.
Running the simple test program on a dual processor machine, the application
closed within 30 seconds every time.

Now let me say that BDE with MS SQL in multiple threads on a multi-processor
machine works fine if you don't encounter a lot of key violations.  That
said, this article on Borland's Community site was the last nail in the
coffin:

    http://community.borland.com/article/0,1410,27266,00.html

We have decided to convert our program to use ADO, which when used in the
simple test program described above, worked flawlessly on a multi-processor
machine.

Again, thank you to all who posted.

Martin Binder

Re:Multi-threaded app on a multi-processor system exiting mysteriously


Quote
Robby Tanner wrote:
> The symptoms you describe are identical to the ones we encountered.

        The symptoms he describes are indicitive of threading problems in
general, FWIW.

        -Craig

--
  Craig Stuntz (TeamB) Vertex Systems Corp. Columbus, OH
      Delphi/InterBase WebLog: http://delphi.weblogs.com
      InterBase PLANalyzer (Free IB optimization tool):
           http://delphi.weblogs.com/IBPLANalyzer

Other Threads