Board index » delphi » Connection being disconnected.

Connection being disconnected.


2006-09-07 01:46:53 PM
delphi84
Hello,
We are using Indy 9, D7. Both client and server are developed by us.
Clients connect to the server using IdTCPClient and they should stay
connected (may be up to 2/3 hours).
The data transmitted over the connection is always from server->clients.
So the clients have a "reader thread" and are always reading and processing
the data.
The data however comes in different intervals of time. Between each data
packet there may be 2, 10, 15 or more minutes.
Sometimes we noticed that a few number of our clients are disconnected from
the server.
The server triggers an OnException event with the message "Disconnected"
This only happens when the interval between the data packets is long enough.
My guesses are that "something" causes the connection to disconnect when no
activity.
I also guess that this is some kind of a router or a firewall. Am I right
???
Is this likely to happen when no activity over a TCP connection?
It does not happen for all of the client but only for a small percentage
(about 10-15%)
Will a ping/pong mechanism solve the problem?
Any suggestions are welcome.
Kind Regards
 
 

Re:Connection being disconnected.

"York" <XXXX@XXXXX.COM>writes
Quote
The data however comes in different intervals of time. Between
each data packet there may be 2, 10, 15 or more minutes.
That is not a good design choice. If the server is not going to send actual
data for long periods of time, then you should have the server send
keep-alive packets at regular intervals during the idle periods. This way,
the client will be able to tell if the server is still active. If the
client does not receive any packets from the server after a suitable period
of time, then the client can assume that the server is dead so that it can
reconnect the connection.
However, I should also mention that if the client sends no data to the
server at all, then the server has no way of detecting dead clients in a
timely manner. If you can, you should have your clients send keep-alive
packets to the server (and have the server reply to them). This way, the
server knows that each client wants to keep the connection active.
Quote
Sometimes we noticed that a few number of our clients are disconnected
from the server. The server triggers an OnException event with the
message "Disconnected"
Without knowing exactly what your code is doing, I can only guess that maybe
you probably had a network failure for those connections.
Quote
This only happens when the interval between the data packets is long
enough.
Are you using timeouts in your reading/writing?
Quote
My guesses are that "something" causes the connection to disconnect
when no activity.
Sockets do not do that. However, if the connections are going through a
firewall or a network router, they might be closing idle connections to
reclaim unused resources.
Quote
Is this likely to happen when no activity over a TCP connection?
Under normal situations, no.
Quote
It does not happen for all of the client but only for a small percentage
(about 10-15%)
Can you isolate what is different about those clients and/or their
environments?
Quote
Will a ping/pong mechanism solve the problem?
Most likely, yes.
Gambit
 

Re:Connection being disconnected.

Remy, thanks.
Here is a more detailed explanation
D7, Indy 9.0.18, Problemous clients - Windows XP or 98, Server - Win 2003
Std
Each client establishes 2 connections to the server (there are two different
thread classes on the server)
There are 2 IDTCPservers listening on two different ports.
The application is actually a chat/voice classroom.
The first connection established has some control purposes and it has a
ping/pong mechanism
the second connection carries voice over tcp
So the one with the ping/pong never gets disconnected unless a network
problem but in case in a network problem the two of them get disconnected
wich is OK.
The second one, however as described (this is the audio) has no ping/pong
and what actually happens
When a client connects to the server with the control connection it is
authenticated and short after that the "audio" connection is established.
The reader thread enters a it "while not terminated" loop
audioconn.readbuffer(....)
There is no read timeout. It reads untill disconnected or data arrives.
Usually initially there is no activity data transfered over the "audio"
connection. (Think of students entered the classroom and waiting for the
teacher to start).
When the server starts sending data over this connection some of clients
indicate their "audio" connection is disconnected.
It is hard to trace but the server logs show the exception "Disconnected"
and the client log shows that the thread has exited because of the client
socket has disconnected.
In a session with 50 users there are usually 3-5 of them with this problem.
The logs show that it usually takes 14 or 15 minutes from the time they
first connect till the time they get disconnected.
To mention here that their "audio" connection never gets disconnected while
data is transfered. i.e. when not idle. It happens that a presenter may
speak for 1 or 2 hours and they have no problems. Only when nothing is sent
for more than 14/15 minutes (i.e. when he turns off his micorphone) and
starts again there are complaints about that.
Sorry for involving you with these details but I just want to give more lite
on this.
The problemous connections report remote ports in the high range above 50000
and IP locations show different countries (India, Australia, China). Never
USA, Europe. (but I doubt it has something in common)
So to summarize:
The connection with a ping/pong never gets disconnected
The connection without a ping/pong gets disconnected when idle for more than
14/15 minutes but never disconnected when not idle no matter how long
Most likey there is a router and/or a proxy or a firewall.
I hope this give more lite.
As far as the architecture is concerned. I mentioned above
Two servers litening on two ports. Each one has a custom thread class.
Per user queue is available. A separate "writer" thread for each client
connection on the server to send data (to avoid blocking). Well there is
nothing special about design. Actually one sends to server. Data is queued
for each client and then each client's writer thread takes care about
sending. So I really think it is not a design issue but has something to do
with the routers.
Yesterday we implemented ping/pong and will be monitoring this.
Thanks
"Remy Lebeau (TeamB)" <XXXX@XXXXX.COM>writes
Quote

"York" <XXXX@XXXXX.COM>writes
news:44ffb24a$XXXX@XXXXX.COM...

>The data however comes in different intervals of time. Between
>each data packet there may be 2, 10, 15 or more minutes.

That is not a good design choice. If the server is not going to send
actual
data for long periods of time, then you should have the server send
keep-alive packets at regular intervals during the idle periods. This
way,
the client will be able to tell if the server is still active. If the
client does not receive any packets from the server after a suitable
period
of time, then the client can assume that the server is dead so that it can
reconnect the connection.

However, I should also mention that if the client sends no data to the
server at all, then the server has no way of detecting dead clients in a
timely manner. If you can, you should have your clients send keep-alive
packets to the server (and have the server reply to them). This way, the
server knows that each client wants to keep the connection active.

>Sometimes we noticed that a few number of our clients are disconnected
>from the server. The server triggers an OnException event with the
>message "Disconnected"

Without knowing exactly what your code is doing, I can only guess that
maybe
you probably had a network failure for those connections.

>This only happens when the interval between the data packets is long
enough.

Are you using timeouts in your reading/writing?

>My guesses are that "something" causes the connection to disconnect
>when no activity.

Sockets do not do that. However, if the connections are going through a
firewall or a network router, they might be closing idle connections to
reclaim unused resources.

>Is this likely to happen when no activity over a TCP connection?

Under normal situations, no.

>It does not happen for all of the client but only for a small percentage
>(about 10-15%)

Can you isolate what is different about those clients and/or their
environments?

>Will a ping/pong mechanism solve the problem?

Most likely, yes.


Gambit


 

Re:Connection being disconnected.

"York" <XXXX@XXXXX.COM>writes
Quote
Each client establishes 2 connections to the server (there
are two different thread classes on the server)
I would be careful with that approach. It suffers from the same problem
that the FTP protocol suffers from. The more connections you use, the more
likely that firewall/routers will interfere with your program.
Quote
Usually initially there is no activity data transfered over the "audio"
connection. (Think of students entered the classroom and waiting
for the teacher to start).
Because you need to broadcast audio to a lot of clients, you might consider
using UDP instead of TCP. Unfortunately, UDP is also usually blocked by
firewalls/routers as well :-(
Quote
When the server starts sending data over this connection some of
clients indicate their "audio" connection is disconnected.
Please elaborate. I'd guess that you are probably crashing the
broadcasting code on the server side, causing the connections to get dropped
unexpectedly.
Quote
It is hard to trace but the server logs show the exception "Disconnected"
What does the exact log look like? What is the exact class type of the
exception?
Quote
The connection without a ping/pong gets disconnected when idle for more
than 14/15 minutes but never disconnected when not idle no matter how long
Like I said earlier, you should be sending a ping packet across the audio
connection during idle periods so that it is not truely idle.
Gambit
 

Re:Connection being disconnected.

Remy - the problem I described is now solved.
A sysadmin at a company using the software helped me with this.
All the users connecting from that company were behind a proxy server. The
proxy server was set up to kill all the tcp connections idle more than 15
minutes.
We implemented a ping/pong each 60 secs and this does not occur any more...
(as of now) at least up until someone does not set the idle timeout less
then a minute on the proxy.
you wrote below :
Quote
Because you need to broadcast audio to a lot of clients, you might
consider
using UDP instead of TCP. Unfortunately, UDP is also usually blocked by
firewalls/routers as well :-(
- yes originally we did so but really had many many problems with
routers/firewalls. So we gave up because we were about to loose clients and
we decided to give TCP a chance though it is slower.
Thanks
Kind Regards
"Remy Lebeau (TeamB)" <XXXX@XXXXX.COM>writes
Quote

"York" <XXXX@XXXXX.COM>writes
news:XXXX@XXXXX.COM...

>Each client establishes 2 connections to the server (there
>are two different thread classes on the server)

I would be careful with that approach. It suffers from the same problem
that the FTP protocol suffers from. The more connections you use, the
more
likely that firewall/routers will interfere with your program.

>Usually initially there is no activity data transfered over the "audio"
>connection. (Think of students entered the classroom and waiting
>for the teacher to start).

Because you need to broadcast audio to a lot of clients, you might
consider
using UDP instead of TCP. Unfortunately, UDP is also usually blocked by
firewalls/routers as well :-(

>When the server starts sending data over this connection some of
>clients indicate their "audio" connection is disconnected.

Please elaborate. I'd guess that you are probably crashing the
broadcasting code on the server side, causing the connections to get
dropped
unexpectedly.

>It is hard to trace but the server logs show the exception "Disconnected"

What does the exact log look like? What is the exact class type of the
exception?

>The connection without a ping/pong gets disconnected when idle for more
>than 14/15 minutes but never disconnected when not idle no matter how
>long

Like I said earlier, you should be sending a ping packet across the audio
connection during idle periods so that it is not truely idle.


Gambit