Tech Robonaught Sadly that comes as no surprise to me. Your vehicle is being replicated in the new region and resuming its scripts before the avatar has arrived and been re-sat on it. It is tragic but true that with the current system the more efficient you make the vehicle, the more likely are the error pop-ups.

[ Part 2 ]

* Vehicle and attachment payloads are decompressed and reconstructed into executable scripts in a definite but not documented order. The time necessary to reconstruct everything is not bound. AVs entering and leaving simultaneously as well as other activities will spread this work out over time. It can be minutes in particularly bad cases.

* When a vehicle is reconstructed and its scripts are again in a runnable state, its scripts can be enabled to run.

* A particularly interesting and rarely mentioned aspect of this is that an event handler invocation that begins in the source region will resume execution in the destination region. That is, code at the top of a 'changed()' handler may run on Region A but by the time it nears completion, it's running on Region B or even Region C. How often might you be exposed to this switch in the middle of a single invocation of code that naturally assumes no such thing is possible?

* This may be especially relevant for SLua where various event handling and coroutine schemes may combine for longer-lived event handler invocations. Pay attention to the details.

* Vehicle reconstruction continues with the vehicle attempting to re-sit the avatars that were sent over with it. Avatars are not seatable until they are fully reconstructed. (This condition may be the source of the 'avatar not found' failures.) For reasons unknown to me, there are various timeouts in effect on this re-sit effort. One of two seconds, another of three-seconds - these between certain points in the processing (these are very magic numbers). If the re-sit doesn't happen in that time, it won't happen by automatic means. Avatar has left the vehicle leaving it up to scripting to re-join them.

* Once an avatar has been fully reconstructed, it joins the set of tasks in the destination region and it can be located by agent reference and its attachments' scripts can run.

* The viewer has been getting updates during this process and at some point, it will send a UDP CompleteAgentMovement message to sync up with the destination simulator. Simulator will return a UDP AgentMovementComplete message to acknowledge synchronization (logged in the viewer as process_agent_movement_complete, I think). Dropping either of these messages is fatal to the session and the viewer will declare itself disconnected and go the the Black&White screen. These messages do go missing... a lot. And other message pairs are equally critical.

Monty Brandenberg Thank you for the response and under-the-hood insight - it is clearly a complex process and very hard to work on.

As region crossing code is clearly a very difficult thing to change, I guess those scripting long distance physical vehicles are also looking if there is anything we can do to improve our scripts for better crossing reliability. We already poured over the permissions code to make sure we got that right, but still get the error pop-ups regardless, and pop-ups often progress to become unsits.

It seems a lot of residents have already given up trying to make multi-region journeys, as for many these days crossing fails are just too frustrating for a journey to be fun.

From reading your notes I guess my train locomotive changes are working by easing the burden of avatar deconstruction and reconstruction : it removes any active pairing of animation, camera and controls between avatar and object before the RC is begun. It holds the permission to do so, but just is not actually doing any until the avatar is confirmed re-sat. Perhaps it reduces the chance of those reconstruction timeouts - it is doing something that helps, whatever that might be. It seems to work remarkably well - I recently went twice around the Heterocera rail line (150+ crossings) with zero error pop-ups and zero unsits. I just can't figure a viable/reliable way to implement that crossing precognition into a physical vehicle. Hyrum's Law aside, it would be nice if I could make it work today, even if I have to change it again another day.

Atomic Infinity: You can change the probabilities quite a bit but getting to 100% is very difficult for a vehicle. (Disclaimer: I don't spend that much time with the script system internals so don't trust everything I say - I discover new and surprising things all the time and try to understand and communicate them when I can.) No-script avatars will shift the probabilities a good bit but will still probably not be good enough for 100% success rate.

animats Resident's comments really are a critical element, however. That detection of the reseating is essential to approach perfection. The vehicle scripts simply aren't in control until that happens.

The LSL API and behaviors really do not lend themselves to hitting that 100% goal. Error handling, environment sensing, APIs, and other things could be changed to make it possible. And using worst-case vehicle tours as the design center (vehicles with 40 riders with 1000 full-data scripts each while doing donuts around four-corners locations) would make everything else in SL just work.

Monty Brandenberg can I ask that you clarify :

Detection of the reseating by the server? or detection by a script?

It is quite clear that scripts are running regardless of reseating, and will error pop-up if the avatar is not there. A script that is checking for reseated agents can pop a perms error while it waits. Also detection of re-sit does nothing about 'trying to stop animation' errors that occur in the source region between the unsit of the agent and the deletion of the old vehicle.

Or is it that we should be aiming to engineer that the script holding perms is essentially tied up mid-event as the crossing starts and keeps the script within that event until the avatar is confirmed reseated? basically, keep it busy so it can't react that the target of its permissions activity is 'not found'.

There is no call to llStopAnimation in my script on leaving a region, and no fresh call to llStartAnimation in a new region. Those calls are being made entirely 'back-of-house', not by the script. My own scripts have the animation start/stop triggers inside the scope of a CHANGED_LINK, and during crossing the server unsits/resits the avatar in a way that does not trigger a CHANGED_LINK to happen, which is fine and expected behavior. At this time though, the only reliable way to avoid animation perm error pop-ups, is to not animate.

Atomic Infinity: detection by a script. Server is the entity performing sit/unsit - it knows the state of things.

I'm convinced some of these problems are race conditions in the scripting. LSL is driven by events passing through queues. State changes in the system can happen long before an event informing a script of a change is delivered as a running handler. Additionally, other events between those two points may have handlers coded to rely on the local (and now incorrect) state. That's what event-based programming looks like. Review of the code by other eyes can help greatly.

There may be genuine bugs in here, as well. Teasing those out from other causes and taking action on them requires proper identification. There are a lot of suspect areas. But the vehicle disassembly prior to transport isn't one of them. I don't trust it 100% but I trust it a lot more than other things.

(I'm ignoring most of the disconnects here... Those are mostly caused by other things and happen to correlate because more systems are involved during RC/TP.)

Monty Brandenberg I'll do some debug in one of my physical vehicles, but I'm pretty sure that if I trundle over a border in cruise, the only event in the queue is a recurring timer() that propels it along. There is literally nothing else going on, not even a control event unless I am also steering (which everyone knows is a bad idea to steer at crossing, if you want any control over your direction after the border). If there is a race condition, I have no idea what is racing or where it can be hiding.

My thought is that the only race is the one in the server - the race between avatar and object changing regions. If the server knows the state of things, it shouldn't try to stop anims if it has already sent me off region, and it shouldn't try to start anims again until it has re-sat me. It is not my script making those function calls, it is the server, and any in-world script detecting re-sits is not going to change that.

Yep, I'd consider that a bug and we'd want to look at that one in depth.

Events debug test results : In all instances the recurring timer() is the only event in the queue. There are no animation or camera function calls in the timer() event.

-

-

-

Soon after this, a bit of rubberband camera on a crossing led me into a zero-seconds ‘security orb’ parcel and I got yeeted into the unknown, reminding me of the other reason I gave up building vehicles and long distance travel.

Just to be clear, this is not about my vehicles, this is about all permission holding sat-on objects that cross borders. This canny was opened by the sailing community, not me. We just share in the same problem. This issue affects hundreds of objects, and thousands of residents. Sure there will be some 'resident scripter' issues that cause some errors, but the vast majority of these border problems are entirely server caused/generated, and utterly out of our control. Border crossing server code needs a total overhaul. I have stated all I can about my observations in these canny postings, some have been read, others glossed over. I can't make LL believe it or even pay attention to it, and I imagine for the most part they are sick of reading my name. I'll stop prodding it now, there is nothing more I can do. Either the LL decision makers will take this years-old problem seriously, or they won't. My hopes are not high.

Atomic Infinity: First thing, I would _really_ like the viewer log from that session. Some more precise timestamps would help me align the notifications with events in the backend.

The comments and debug work are very much appreciated, at least in engineering. And the voting backs that up.

As for the test run, the three problematic crossings all experienced the two-second timeout situation I mentioned. The crossings into Burnott and Carmine were as expected. Whether this is cause or effect I don't yet know. Understanding those llRequestPermissions, etc. calls should help.

Monty Brandenberg I will IM inworld later when I get chance to get you yesterdays log stuff.

For better timestamps - my logs seem to be in GMT not SLT, so 12:42 SLT is 19:42 by the log :

The script floater opens at 19:42:52 and 19:46:36 with 'control not found', and then seems to be at 19:49:25 for the anim errors, though it doesn't have a note like the control one does, so I'm just assuming that 8 lines of 'script floater opening' notes at Carmine/Riddings is that double error popping up.

animats Resident I'm not using KFM because it has too many caveats - I find it far more reliable to just do the math and place the object with llSLPPF. Doing that means I can literally pin down the exact frame that the border handover will begin, which is how I came to my conclusions listed. So far I never ended up in the wrong region even when doing 4-way region corners at speed, so I guess I'm not doing whatever that bug is (or I'm lucky).

continued...

I believe the key part that makes this succeed is that :

The script continues to hold the permissions to act on the avatar, but is not actually animating, nor controlling cam, nor taking controls at the moment of crossing the border. If it is not to do these things, it can not generate an erroneous and unjustified script error when it can't find the avatar.

So my LSL re-write is effectively a workaround for the synchronization problem of the suspend/resume of scripts not matching when the avatar and/or vehicle actually do change region and re-unite.

The script does not lose its permissions status at any point - what it 'loses' is the avatar it is supposed to be acting upon. The scripts are running when they are not supposed to be.

My code does not fix unsits, it avoids unjustified script errors. Avoiding these script errors is what fixes most unsits.

I do still get some script errors occasionally, so somewhere my workaround is not 100% reliable at suspending / resuming function in the brief moment of border crossing, or detecting the avatar fast enough.

The users viewer also seems to be an influencing factor. When I do get script errors now on crossing a region border, it is most often when my viewer framerate is compromised, for example when approaching a built up area with a lot of content to load and render. Maybe the delay in my lower framerate still allows some stray updates to be sent while the script is trying to release cam & controls before the border, but this is just guesswork.

I was thinking I might try to do a similar process in some of my physical vehicle projects, but seeing what it took to improve in a non-physical, I’m not sure that is going to be possible. The non-physical movement is like precognition of when the crossing will happen, whereas physical movement is not so predictable. I guess I'll try it and see how that goes.

Anyway, hopefully this info is useful in the search to help fix border crossings for everyone.

Addendum - These are the errors, and why they occur :

The object is still in the OLD region, the script is running. The avatar is not sitting on it, and has already left the old region.

The object is in the NEW region, the script is running. The avatar is not sitting on it, and has not arrived in the new region.

The object is still in the OLD region, the script is running. The avatar is NOT sitting on it, but is also still in the old region. (Anim perm is auto granted when sitting, therefore avatar is not sitting)

The object is in the NEW region, the script is running. The avatar has arrived in the new region, but is NOT yet sitting on it. (if the avatar was sitting, anim perms are auto granted)

To me this one is the most obvious example of the problem. I was sitting on it in the old region, and I’ll be sitting on it in the new region. The only time I am not sitting, is while the server hands me over.

The common denominator in all these, is that the scripts are still running when the avatar is unseated, and have started up again before the avatar is re-seated.

And then there’s that random pop up box that happens sometimes :

[OK]

umm.... whut? yeah ok, but I have no idea what that means :D

Atomic Infinity Scripts for vehicles have to recognize that being in transit across a region boundary takes time. There is an in-transit period during which some things won't work. Upon arrival in the new region, you have to detect that all avatars are properly seated (vehicle is parent of avatar, avatar is child of vehicle). Then re-acquire permissions. Then re-establish controls, animations, etc. See my code at https://github.com/John-Nagle/lslutils/tree/master/motorcycle

A few years ago, one of the server Lindens suggested stopping vehicle script execution during region crossings. That way, scripts never see what's going on during a region crossing. But if the server stops scripts, you can't steer or throttle during a crossing, and slow crossings result in runaway vehicles. That would make things worse.

animats Resident I can see it is complicated yes, but it needs to happen in a controlled order. If they can't be in a controlled order then they should disable the script error pop-ups that have no basis in fact. Its highly likely that the server is basically causing unsits by tripping over itself sending wrong script errors out that are entirely out of the scripters control.

Plus it is massively disheartening to have a vehicle you holds correct permissions constantly telling you on border crossings that it doesn't, and suggesting your work is substandard....

edit : maybe I am not expressing it that well, but I am talking about suspending scripts at the moment of actual handover only at server level, not some time before you get to the border. Control inputs 'during crossing' are moot anyway if the avatar is not seated, unless you asked for and secured perms manually from the avatar before they sat down.

So I figured do some lsl investigating, to see if scripts really do lose permission and need it re-acquired after a crossing, and how long it takes for the process to happen, how long is the avatar removed for, etc. I added this script to a physical vehicle, then drove it over a lot of borders. 3 seconds after each border crossing it reports how many timer events happen with each of the 3 conditions tested, then resets the values.

Typical Results -

In typical results there was always 5 more ‘not sat on’ events than there are ‘no avatar’ events. If I recall, a timer event is likely to throttle at 0.044 seconds per event. If that is correct, then per crossing on average the avatar is not seated for 11 x 0.044 = 0.484 seconds, and the object and avatar are in different regions for 6 x 0.044 = 0.264 seconds.

Extreme results with an unsit -

In other unsits that became viewer disconnects, the results were lost in the crash. A delayed arrival of the avatar in the new region perhaps makes an unsit more likely.

So it seems re-acquiring permission is not needed, but the script has time available to produce errors because the avatar whom it needs contact with is not present & seated.

I originally thought scripts were stopped in a crossing, but I guess that assumption is wrong. It would certainly be worth testing crossings that stop scripts in 'sat on' objects before separating the avatar - 4 homesteads in a square on the beta grid so you can drive round in circles and through a 4-way corner, to see what happens.

Since investigating all that ^ I have since read this page https://wiki.secondlife.com/wiki/Region_crossing from which I learned something :

Sat-on objects do not cross region borders, ever.

The object does not move across, instead it is replicated in the new region, and the old copy in the previous region is deleted. Accordingly, there is no such thing as steering or throttle during a crossing, so there is nothing to break by stopping the scripts. Any active physical velocity in the old copy will exist in its new copy upon creation, but it won't be taking any new control inputs anyway while the avatar is not seated.

The more I look at this issue, the more convinced I become that if the scripts in the object are suspended before removing the avatar, and then re-started after re-seating them, that it would solve a huge number of bad crossings.

But all the theory is for nothing without an experimental set of regions on Beta grid ....

Skyler Pancake Well, that does happen to me sometime, but thankfully Kokua Viewer has "Refresh Scene" menu item that forces a refresh of the visible region. Anyways, I think there's another Canny feedback about this ...

AWalphaomega Resident Soon?

It has been like this since they introduced the vehicle system in 2004.

I would LOVE it if they would do something about this, but they have an extremely long and reliable record (2+ decades) of not fixing it.

Philip Linden I hope you'll tell the teams that, and keep checking in until they do it. It has been like this for 21 years, since the days of hoverboards and Busy Ben's. I'm not saying they can't fix it, but I am saying there appears to be a monumental inertia behind NOT fixing it.

If they don't keep hearing that it's a priority, from you personally, what could plausibly turn that around that hasn't already proven ineffective?

What your people will learn from this, not just technically but from an organizational behavior standpoint, will be invaluable. Fixing this will be like pulling Excalibur out of the stone.

Philip Linden I really hope you do. Driving on mainland, which was a huge thing for my partner and I, has become almost impossible as of late. The constant crashes and being thrown on sim crossing has really taken the fun and enjoyment out of driving.