There are some delicate relationships between the event poller server side timeout, the event poller viewer side timeout and the child agent inactivity server side timeout. Problems in this area have been discovered before. I think I just got to the bottom of another one.

This is not something that needs an urgent fix, but if any work is done on the event poller, this issue needs to be considered.

Pepper Linden has told me that there's a 50-secoind timer which causes the server to disconnect a child agent for inactivity. I've seen that happening. If the viewer's long poll timeout is longer than 50 seconds, there is a possibiilty of child agents timing out and a DisableSimulator being sent.

I've done some tests in Sharpview. A 90 second timeout causes an unexpected DisableSimulator for an active child region followed by a viewer disconnect about once every hour. With a 15 second timeout, that does not happen in 24 hours. So I now have a workaround inside Sharpview.

Here is what I think is happening. I can only see viewer side, of course.

Pepper set up an island atop a region corner at Blake Sea - Nelson on aditi. I set up a train track loop which goes right through the region corner, and a small train which endlessly circles the loop. We put avatars on the train and run unattended region crossing tests for hours or days. This is a useful stress test and has brought out some region crossing bugs, which have been fixed. By now, the LL and Firestorm viewers can run that test for a full day without error.

But when I run one of the avatars from my Sharpview viewer, it was failing in 1-3 hours. I was using a long HTTPS poll with a 90 second timeout. That works fine until you start running multi-hour tests.

Why would the timeout matter? Because, apparently, no activity on the event channel for 50 seconds causes a DisableSimulator. An event channel timeout seems to count as "activity". So if the event channel timeout is longer than the DisableSimulator timeout, and there's no real traffic on the event channel, and it's a child region, and the timers line up, the region disables itself for that viewer for inactivity.

Henri Beauchamp has experimented with various timeouts in his viewer, and notes that behavior changes with the timeout value. Since both server and viewer can time out, whomeever has the shorter timeout times out first. If both have the same timeout, there's a race. We're still not sure about all the possible race conditions.

I'd suggest a solid fix that gets rid of HTTPS timeouts. When there are no events to send, the servers should send an empty array of events over the event channel every 15-20 seconds to keep the connection alive. Existing viewers should ignore such an empty array. so this should not affect existing viewers.

All HTTPS transactions on the event channel then become OK 200 transactions, the happy path. Overhead drops, because re-establishing the HTTPS connection several times a minute means a full cryptographic handshake, while sending a short message over an existing persistent connection is much cheaper. If there are any remaining unexplained event poll problems, this is probably worth doing.

It's so easy to create intermittent problems by changing one of those numbers.

Changes to libcurl, Apache (which, I think, front-ends the HTTPS caps on sim servers), the sim servers themselves, or viewers can affect this. It's a huge pain to diagnose such problems, because the failure rate is so low and you need a test fixture like the little train on its track loop. This took weeks of work, because the test is so slow.