Maestro Linden
tracked
Iexo Bethune
Tarawa Atoll, 16:44 SLT on 1/1/24. Region lagged noticeably (avatar movement and physics slowed) when a resident entered/logged on.
Firestorm 7.1.11 (76496) Oct 22 2024 21:16:39 (64bit / AVX2) (Firestorm-Releasex64) with Havok support
Release Notes
You are at 161.5, 232.8, 23.1 in Tarawa Atoll located at simhost-0f114d65427958349.agni
(global coordinates 262,050.0, 236,521.0, 23.1)
Second Life Server 2024-11-25.12013542687
Release Notes
CPU: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz (3600 MHz)
Memory: 32602 MB (Used: 4369 MB)
Concurrency: 16
OS Version: Microsoft Windows 10 64-bit (Build 19045.5247)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: NVIDIA GeForce GTX 1080/PCIe/SSE2
Graphics Card Memory: 8059 MB
Graphics Card Memory (Detected): 8059 MB
Graphics Card Memory (Budget): Unlimited
Windows Graphics Driver Version: 32.0.15.6590
OpenGL Version: 4.6.0 NVIDIA 565.90
RestrainedLove API: RLV v3.4.3 / RLVa v2.4.2.76496
libcurl Version: libcurl/7.54.1 OpenSSL/1.1.1w zlib/1.3.1.zlib-ng nghttp2/1.62.1
J2C Decoder Version: KDU v8.4.1
Audio Driver Version: FMOD Studio 2.02.20
Dullahan: 1.14.0.202408091639
CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54
Chromium: 118.0.5993.54
LibVLC Version: 3.0.21
Voice Server Version: Not Connected
Settings mode: Phoenix
Viewer Skin: Vintage (Classic)
Window size: 1366x745 px
Font Used: Deja Vu (96 dpi)
Font Size Adjustment: 0 pt
UI Scaling: 1
Draw distance: 512 m
Bandwidth: 1500 kbit/s
LOD factor: 3
Render quality: High-Ultra (6/7)
Disk cache: Max size 9984.0 MB (88.0% used)
Built with MSVC version 1941
Packets Lost: 25/141,448 (0.0%)
January 01 2025 16:47:07 SLT
Monty Brandenberg
Iexo Bethune: thanks! Grabbed logs for both events. You don't happen to know the offending avatars in these cases, perhaps?
Iexo Bethune
Monty Brandenberg The second was EvilJR Resident. The first was Umaeril Resident, I think.
Monty Brandenberg
Ah, thanks, saves guessing. Quick scan looked like the simulator was handling things well (and so pointing at viewer or network for blame). Will be reviewing this in more detail.
Monty Brandenberg
Iexo Bethune: Looking more closely, this seems pretty healthy. Halcyon arrival took 128 frames over 4.8s or 37ms/frame for 23 attachments. Tarawa Atoll took 160 frames over 4.6s or 29ms/frame for 27 attachments. Frame rate moderated but didn't go crashing into a wall and updates queued up to you without obvious problems at this end.
(This is a scenario when I really want logs on both ends. Simulator thinks it's doing its job, viewer might have another view of the effort. But half the story will be in the viewer log - if I'm lucky. Sadly, Canny isn't the vehicle for sharing viewer log files. If you happened to set them aside, I can look at the bigger story.)
Iexo Bethune
Halcyon, in the Shop & Hop, at around 16:16:30 SLT on 12/26/24. Extreme lag (unable to move) for about 15-20 seconds upon arrival of a single avatar.
Firestorm 7.1.11 (76496) Oct 22 2024 21:16:39 (64bit / AVX2) (Firestorm-Releasex64) with Havok support
Release Notes
You are at 38.8, 216.1, 52.5 in Halcyon located at simhost-03aefb341631db06d.agni
(global coordinates 125,735.0, 306,136.0, 52.5)
Events 2024-11-25.12013542687
Release Notes
CPU: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz (3600 MHz)
Memory: 32602 MB (Used: 3305 MB)
Concurrency: 16
OS Version: Microsoft Windows 10 64-bit (Build 19045.5247)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: NVIDIA GeForce GTX 1080/PCIe/SSE2
Graphics Card Memory: 8059 MB
Graphics Card Memory (Detected): 8059 MB
Graphics Card Memory (Budget): Unlimited
Windows Graphics Driver Version: 32.0.15.6590
OpenGL Version: 4.6.0 NVIDIA 565.90
RestrainedLove API: RLV v3.4.3 / RLVa v2.4.2.76496
libcurl Version: libcurl/7.54.1 OpenSSL/1.1.1w zlib/1.3.1.zlib-ng nghttp2/1.62.1
J2C Decoder Version: KDU v8.4.1
Audio Driver Version: FMOD Studio 2.02.20
Dullahan: 1.14.0.202408091639
CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54
Chromium: 118.0.5993.54
LibVLC Version: 3.0.21
Voice Server Version: Not Connected
Settings mode: Phoenix
Viewer Skin: Vintage (Classic)
Window size: 1366x745 px
Font Used: Deja Vu (96 dpi)
Font Size Adjustment: 0 pt
UI Scaling: 1
Draw distance: 384 m
Bandwidth: 1500 kbit/s
LOD factor: 3
Render quality: High-Ultra (6/7)
Disk cache: Max size 9984.0 MB (55.5% used)
Built with MSVC version 1941
Packets Lost: 10/720,543 (0.0%)
December 26 2024 16:20:24 SLT
Jackson Redstar
At a crowded event that spanned 2 sims - the 'audience' sim had about 80 avatars and the 'performers' sim about 60 avatars. I was in the audience sim cammed into the performers to video the event. The performers were sitting - this was an orchestra. I wasn't experiencing any real lag - but about half of the performers, their mesh heads would not load. Bodies, body skin clothing, eyes, teeth, hair, and parts of the face with the system skin. Right click to 'refresh texture' did nothing. I tried my usual trick of de rendering the avatar, then removing it from the blacklist, switch groups, to force it to reload, but no luck getting the head to render. If there was just some way when right clicking on a avatar, to dump the cache for that avatar and force a re load/ re fetch, I really don't know what would work. for reference my system RAM was at about 25g of 32g with FS 7.1.11 and OBS running and VRAM was at about 9g of 16g with the GPU at about 35% utilization
This is the system at the time of the event:
Firestorm 7.1.11 (76496) Oct 22 2024 21:16:39 (64bit / AVX2) (Firestorm-Releasex64) with Havok support
Release Notes
System Specs
Second Life Server 2024-11-25.12013542687
Release Notes
CPU: Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz (under load runs 4.90 ghz)
Memory: 32669 MB (Used: 1897 MB)
Concurrency: 20
OS Version: Microsoft Windows 11 64-bit (Build 22621.4317)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: NVIDIA GeForce RTX 4070 Ti SUPER/PCIe/SSE2
Graphics Card Memory: 16063 MB
Graphics Card Memory (Detected): 16063 MB
Graphics Card Memory (Budget): Unlimited
Windows Graphics Driver Version: 32.0.15.6614
OpenGL Version: 4.6.0 NVIDIA 566.14
RestrainedLove API: (disabled)
libcurl Version: libcurl/7.54.1 OpenSSL/1.1.1w zlib/1.3.1.zlib-ng nghttp2/1.62.1
J2C Decoder Version: KDU v8.4.1
Audio Driver Version: FMOD Studio 2.02.20
Dullahan: 1.14.0.202408091639
CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54
Chromium: 118.0.5993.54
LibVLC Version: 3.0.21
Voice Server Version: Not Connected
Settings mode: Firestorm
Viewer Skin: Firestorm (Grey)
Window size: 2560x1417 px
Font Used: Deja Vu (96 dpi)
Font Size Adjustment: 0 pt
UI Scaling: 1
Draw distance: 176 m
Bandwidth: 15000 kbit/s
LOD factor: 3.625
Render quality: High-Ultra (6/7)
Disk cache: Max size 19968.0 MB (75.1% used)
Built with MSVC version 1941
Packets Lost: 0/3,912 (0.0%)
December 23 2024 13:26:42 SLT
animats Resident
I went looking for that last week. It really is better in Firestorm 7.1.11 than it was in 7.1.9. Avatars stuck in cloud mode at crowded clubs are now rare, but not nonexistent. 1 or 2 broken avatars out of 60 to 70 at the busiest clubs and London City. Relogging shows me different broken avatars, so that's an attachment loading problem for observing viewers, not a problem on the avatar owner's side.
Relogging helps distinguish between attachment loading problems, user-side wardrobe malfunctions, and people being nude on purpose, all of which can be seen at WelcomeHub.
Monty Brandenberg
There's been more noise lately that there is server-induced lag, either new or returned since the server release earlier this year. I want to capture those reports in this issue as they suggest regressions or missed factors. But I suspect many of these will actually be viewer-induced problems. We've already identified a number of prioritization problems with avatar content in the viewer and improvements are showing up in 'develop'. Unfortunately, we're also finding coding decisions that make it challenging to approach a theoretical ideal such as: https://github.com/secondlife/viewer/issues/3223 (There's the bug itself as well as having sequential queuing systems which make re-ordering very difficult.)
But viewer issues will get sorted elsewhere. Really want the server candidates here and I'll try to sort out the server from the viewer where the guilty party isn't obvious.
animats Resident
Hm. My own Sharpview viewer still renders avatars as blocks, so I don't have any insight from that side. What do the mobile viewer and the Crystal viewer do? Those are different code bases.
A test region on the beta grid with a hundred bots would force the issue. The problem seems to appear somewhere around 50 avatars in a region.
Back on the main grid, it might be worthwhile, as a test, to put a few very busy regions on the new big event server instances. Don't increase the avatar limit, just provide more compute power. If that fixes London City, that would help new user onboarding, too.
N
Nya Jules
Monty Brandenberg Hi Monty, I'm currently investigating constant small lag spikes in a sim. On my very early findings I see Sim FPS and Scripts Run % both dropping at the same time when an avatar enters the sim. The drops only lasts for a very short time. Could this be the issue that is being talked about?
Monty Brandenberg
Nya Jules: That's a marker that may indicate a problem and can be the starting point for an investigation. But first I need to be clear about expectations here.
Last year, we released simulator changes that reduce the impact of avatar arrival on avatars that are already resident. Unfortunately, what many heard was "Region crossings are fixed! Everything's perfect now!" and that's not the case. :) So we're listening to questions about this work at the same time as we're having a lot of viewer churn and separating concerns is critical (and Linden's responsibility).
For the simulator, I'm trying to see if the 2024 changes are functioning as intended (as well as prioritizing 2025 work for TPs/RCs). So far, what I'm seeing is that they are working and most of the experiential lag lies with the viewer and network. For example, I extracted some logging using the data Iexo Bethune supplied and found an amusing avatar logging into an unrelated region (but on a shared simhost). It took 31s and over 1300 simulation frames for the attachments to fully realize. Previously, that might have been something like 20 frames of 0.5-2s per
frame
. So that is a success. The processing for RCs/TPs can't be reduced to zero (though it badly needs optimization) but it can be spread more fairly over frames. Which data mostly shows it is.So there is still impact on arrival and it will show up in the simulator frame rates. But it will be improved relative to Summer 2024. And other areas haven't been addressed yet such as RC/TP out of a region as well as other frame rate-destroying events. So I'm interested in good reports of any such events and I'll sort out the relevant details. At the viewer, it's difficult to even determine whether 'lag' is coming from viewer, network, or simulation. (I would love to improve that situation.)