After 22 years... why do textures take so long to rez?
tracked
Eren Padar
"If the 2D don't work, the 3D don't work".
I was at Sci Fi Con 17. All day long, no matter how many or how few avatars were on a region, textures took ruddy ages to load. Now one might say "There are so many textures at a big event it just takes a long time." But I'd walk into an enclosed room... where textures should be limited, yet the vendors on the wall just sat there and sat there and sat there. Textures RIGHT NEXT TO ME fail to load. One can stand there and stand there and stand there... and they still don't rez.
I would click on a texture and tell it TEXTURE REFRESH... and it would ignore that instruction.
I first reported texture issues and bottlenecks back in 2005. Now my computer and graphics card is hundreds of times more powerful... and textures still load at a snail pace. I will say the same thing I said back then: There is a texture bottleneck on SL. (Or, several texture bottlenecks, and it affects EVERYTHING.)
With respect, I'm not asking for an explanation, nor excuses. I'm making a simple observation: After 22+ years online, Second Life should be able to very quickly load 2D textures in an enclosed room. Is this not correct?
I suspect there will be people all too ready to give reasons why textures load slowly. To any and all such reasoning, I have one answer: Second Life needs to make textures load quickly. There's no argument, debate, discussion or excuse to that simple statement.
Users need textures to load quickly enough that we don't get bored standing in a store (waiting for vendor textures to load)... and decide to go do something else. Like Netflix. ;D
Because that's what it is: Second Life vs Netflix. Users have options on how to spend their time. When Netflix can stream entire videos realtime, surely Linden Lab should be able to send 2D textures across the Internet... yes?
As a note: I'm not going to debate nor discuss or argue with those who are always ready to argue on these feedbacks. The feedback is posted, I've said what I came to say... and I have to believe the vast majority of SL users will agree with this: textures on SL need to load faster.
Log In
Dan Linden
Thank you for the report!
Issue tracked at https://github.com/secondlife/viewer/issues/4274. We have no estimate when it may be implemented. Please see future updates here.
Dan Linden
tracked
Eren Padar
Dan Linden Thank you Dan. As a small addition to this, a common issue: We upload a texture from our hard drive and it is put in our inventory. We double-click on the texture we just uploaded, from our own inventory... and a gray box comes up that can take 15 seconds, 30 seconds, an entire minute or even more to load. This "texture on demand" issue (the system failing to rez textures that are the direct focus of the user) can be especially annoying, because by all rights they should be immediately available.
If we upload a texture, it should be at immediate access, yes? If we go to a vendor and click on a vendor button, the texture should load as a priority, because it is the direct-focus, texture-on-demand of an active user (as opposed to 500 textures that are somewhere in the distance behind me).
True story: I brought this to the attention of the devs at Kitely grid (Oren there is an especially gifted dev). I provided them with considerable research data proving there was a texture bottleneck. Oren found that bottleneck (and more) in just a few days, and after those bottlenecks were repaired, texture rezzing increased in speed 15x to 30x +. The difference was amazing.
Oren did say that there were other texture bottlenecks, but he was afraid to touch them because they were undocumented in the Viewer and he wasn't sure of the domino effect they might cause. He did say however, it was a combination of Viewer, Sim Server and Asset Server communications. (Beyond that, I don't know the precise nature.)
What this did prove however was that there were bottlenecks, they were able to discover them and fix them, and performance improved significantly. It gives us the assurance that if they could be fixed on Kitely, they can surely be fixed on SL (which is an integrated company rather than separate dev teams from different companies). So offering that experience as encouragement: the problem does exist, it can be fixed (although it is multi-fold), and the result is impressive improvements in speed.
Imagine getting a texture that once took 30 seconds to load... to load in 1 or 2 seconds instead... especially in "on-demand" textures.
Zoom zoom. : )
Christina Riederer
Alot of the loading issues could get resolved by switched over to Vulkan, LL really need to start modernizing the way rendering works, its a great platform but the lag and load times even on high end machines really puts off new players.
Pazako Karu
I agree fully that we need a drastically different approach to texture handling and caching. The ll viewer is improving, surely, but there's likely something fundamental going on here. I've got my own nginx backup cache just so SL can cache thrash all it wants without causing me many issues. There's a great chance that if the Lindens are testing from offices or VPNs - those systems have precached things in ways we are not privy to.
Dan Linden
Thank you for the report, Eren.
In my experience texture loading is significantly faster and complete on a recent viewer than on a viewer from 2 years ago. So there may be a difference between what you see and I see.
The best way to get a bug fixed is to get the bug to reproduce on a Developers machine. To do that, we need to get the bug to reproduce on a QA machine. To do that, QA needs the exact steps that trigger the bug on your machine.
Please share the location of the screenshot https://gyazo.com/fb30ede4a4a86aec9fd2df3909e2112a or IM it to me if you don't want it public.
Please turn on the Texture console and take another photo there and attach it to this bug. What I want to see is if the texture console thinks everything is loaded. To turn on the Texture console, first enable Preferences > Advanced > Show Developer Menu. Then hit ctrl-shift-3 to enable the texture console.
Please do the same at http://maps.secondlife.com/secondlife/LagLand/4/128/85 I would be interested to see if there are textures there that do not load for you.
Does texture loading work better (or worse) at the store location with the SecondLife viewer?
Eren Padar
Dan Linden I'll be happy to do that for you Dan, soon as I get a chance. I can give you part of the information now: that sign wall was taken at Sci Fi Con 17 at my DragonForge booth, SLURL:
There were only a tiny handful of avatars on the region at the time. Additional information to follow later, as I'm a bit swamped this evening.
As a note, I am a member of the Firestorm Preview Testers group, so I'm using the very latest Firestorm BETA Viewer. That said, as the OP mentions these issues have been going on for 22 years (the beginning of SL). So these issues are nothing new, unique, or isolated to a specific location. They happen every day, all the time, everywhere, to thousands of users.
Eren Padar
Dan Linden Dan I ran the test with the TEXTURE CONSOLE as you requested. I still have tests to run and will provide info as I get it... but one thing I noticed is this: if I look at one wall full of textures... the texture console will show them as loaded (and they are if I stand there long enough). But it took a full minute for that wall of about 52 textures to load... and nothing else on my viewer screen. Then once the wall was loaded, I noticed when I zoomed in on the textures, or moved my camera from one side of the wall to the other... the exact same textures would load again, over and over and over, every time I moved my camera. That is what Balpien reported back in 2007 (only at that time we didn't have to move the camera. The textures just kept reloading on their own).
So I wonder why the asset server is reloading textures that should already exist in my cache... just because I move my camera along the same wall. Perhaps there may be a logical and rational explanation, but it seems up-front to be a glitch. Or I could be reading the chart wrong. Dunno exactly how that chart works.
Here's another observation: I can look at a wall in my store... have it fully rez... then visit the store next door. Same region. All textures should be cached. But when I go back to my store, all the textures are gray and fuzzy, and start reloading all over again, no faster than before (ie, apparently not from cache).
Those are my initial observations without extensive testing. I know from experience this will happen repeatedly, pretty much every time. I could go from one store to the other over and over and each time the textures will refresh, although I'm still in the same region. Surely there aren't enough textures in one region to totally fill 12gb GRAM and 9gb disc cache.
Third observation. I've been standing here, staring at the same exact wall, and the textures are loading over and over and over... again appearing to verify Balpien's findings.
Eren Padar
Dan Linden I finished the test you asked for and visited the location you requested. This is the result I got... for a full 60 seconds. I noticed again that if I zoomed in and out on those textures, every time it would go through what appears to be the same reload (but I understand I could be misinterpreting the readout, since I don't really know what all that does). ;D
Once this screen stopped, I zoomed in...
and...
Eren Padar
Dan Linden One last test and note: when I left the SLURL you gave me (that texture wall) and returned to the exact same sim I was on before, looking at the exact same wall... ALL of the textures started out as gray and appeared to be reloading from the asset server all over again. Why the asset server I thinks? Because if those textures were still on my graphics card or disk cache they should have loaded lightning fast. Instead they grayed, then fuzzed, then blurry, and a minute later that 52-texture wall was loaded.
Thanks for your attention to this Dan. Appreciated.
Monty Brandenberg
Eren Padar: If you want to know more about the texture console display, there is some dated documentation available on the wiki: https://wiki.secondlife.com/wiki/Texture_Console It needs some updates but it is close to correct for the SL viewer. There is quite a bit of information so make a cup of tea/pour a cold one first. At the bottom is a 'Some patterns to look for' section which parallels your own questions and may help guide your reading.
Caching has many layers in the texture and mesh systems and I'll list them briefly for reference purposes:
* The asset store itself is a set of S3 buckets in AWS.
* Above this is a Linden asset cache built on the CARP protocol.
* Above that is the CDN origin server which requests assets, caches them, and delivers them to remote CDN endpoints (PoPs or Points-of-Presence).
* Above that are fleets of CDN caching PoPs, one of which will be serving your session.
* Above that are the texture, mesh, and header on-disk caches managed by the Viewer.
* Above that are the in-memory (application) caches for textures and meshes.
* Above that are the on-graphics (system) cache of active visual elements.
The console metrics are mostly about the last three bullets. In doing repeatable testing, we usually try to eliminate the other caches by 'pre-heating' the PoP. But we'll leave that out for now.
Dan Linden
open
Eren Padar
This is what I'm speaking of above and below. This is my shop, textures that I see every day, several times a day. I've been standing here for a full minute, in an enclosed room looking at one wall... and this is what I see.
It took a full 2 minutes for those few textures to fully rez. All textures there are 512x512.
Maestro Linden
needs info
Hi guys, a few notes about texture loading:
- For the past decade or so, the SL Viewer has been fetching texture and mesh assets over HTTP through a global Content Delivery Network (CDN). The SL simulator is not involved in this process, aside from communicating the asset_ids of an objects or textures on in-world objects or avatars.
- If the viewer is able to render a prim at all, it already has all the information it needs to load the object's textures. It is up to the viewer to fetch that asset from the CDN.
- When the viewer fetches a texture to render, it stores it in memory and also on the local disk cache.
- If the viewer's disk cache is full (has reached the cache size set in viewer preferences), the viewer should be purging the assets which were accessed least recently to make room for the new assets.
Because of this design, you're probably seeing a client-side issue, so I'm moving this to viewer bug reports.
Some common causes of poor texture loading performance on the viewer are:
- Some custom setting is active in the viewer, which is reducing performance.
- Antivirus software is blocking (or slowing) the viewer's access to its cache files.
To debug these likely causes, please do the following and report back if the performance issue persists:
- Clear your viewer settings by temporarily renaming the settings file. On Windows, this will be at %APPDATA%\Firestorm_x64\user_settings\settings.xml or %APPDATA%\SecondLife\user_settings\settings.xml , depending on your viewer. Renaming it to something like settings.xml.bak should work.
- If you have Antivirus software running, whitelist specific folders used by the viewer in the antivirus settings. The FireStorm wiki has a guide to do that: https://wiki.firestormviewer.org/antivirus_whitelisting
Maestro Linden
As a benchmark of raw texture loading performance, you can try the following. This benchmark involves the viewer arriving in a scene with many textures, and measuring the total time for the viewer to load them.
- Login to and visit http://maps.secondlife.com/secondlife/LagLand/4/128/85
- Set your viewer's draw distance to 512m, and have your avatar face east to view the walls of textures
- Clear your viewer cache, and exit the viewer
- Launch the viewer, and click to 'Login to Last Location'
- As soon as you click to login, immediately start a stopwatch timer (I recommend using one's phone)
- Keep the viewer widow in focus, so that it runs at full speed
- Open the texture console (Develop -> Consoles -> Texture Console, or Ctrl+Shift+3 ) to view the texture-fetching status
- The 6th line of the texture console will read "Textures: [total texture count] Fetch: [remaining texture count] ...". In my attached screenshot, it shows 5763 total textures with 1724 remaining to fetch.
- The texture console also shows a dynamic list of all the texture assets the viewer is currently fetching, with a progress bar for each texture. The leftmost column contains the first 7 digits of each texture's asset ID.
- Watch the progress in the texture console, and stop the stopwatch when the "Fetch" stat gets to 0 (indicating that all textures have loaded).
- Repeat steps (3) - (8) a few times, to get a few samples.
I did 3 runs with the Second Life Release 7.1.14.15192634334 on Mac OS, and the total times I got were 53.8s, 57.8s, 55.6s. Since there were 5763 textures in the scene and the measured time also included the login process, the viewer was loading textures at a rate of at least 100 assets/s. I would expect some variation based on internet connection speed, proximity to your local CDN node, and viewer performance, but that loading speed should be achievable for many SL users.
Nerxual Oh
Maestro Linden Or you know. You guys could LOOK INTO THE ISSUE. Instead of giving everyone a cookie cutter response "Have you tried turning it on and off?"
This has been the topic in most tiny groups. We're very tired of this.. Personally I did clear cache, I did firewall, DD is set and all of that. It doesn't work. This is a feedback, stop glossing over it please. And look into the issue. That is all we'd like from y'all.
Eren Padar
Maestro Linden I have to agree with Nerxual. First, I'm too old for this stuff. Honestly, in my younger days I would have done all that step by step, but I'm just too old and tired to jump through a dozen hoops for someone else's company.
This is especially the case since frankly, it doesn't matter how many textures you're loading per second if numerous textures FAIL TO LOAD. I provided a picture of my shop, in which my shop name sign failed to rez for 60 seconds. That has nothing to do with textures per second loaded; that has to do with a texture BOTTLENECK (as I've been telling Linden Lab since 2004, when I was still working as a computer consultant).
Consider please (and I'm going to exaggerate to make the point): The Viewer has to load 2,000 textures. Out of those 2,000 textures, 1.800 load in 1/2 a second. WOW! What speed. However, the remaining 200 textures don't load at all and remain either gray or fuzzy.
Would you call that successful texture handling?
So what Nerxual says above. I filed the initial JIRA in 2005. Balpien Hammerer joined in 2007 and we had a HUGE debate with Linden Lab over these issues. (As a note: Balpien is a pro-level tech.)
Balpien built a device which tracked what textures were being loaded and how often, and discovered the same textures were being reloaded from the asset server as many as 30 times in a row. Now multiply that by millions of textures an hour. DDoSA on your own servers? Linden Lab ignored Balpien's findings, claiming textures were working fine (with respect, kind of like the replies we're seeing above, only direct denial of any problem).
In 2010 LL announced it had "discovered" a "texture bottleneck" in its system and lo and behold... it was the same bottleneck Balpien had nailed in 2007.
Seriously, if the user has to jump through a dozen hoops to whitelist software, I have to wonder what's going on software-side (NONE of my other software titles require whitelisting, no matter how complex).
When we stand in a CLOSED ROOM and stare at a wall of three vendors with maybe 25 textures total showing on those vendors (ie, the only thing in my Viewer window) and those textures are all either gray or fuzzy and remain that way for up to a minute or forever... then the number of textures per second have nothing to do with this issue. Again:
Monty Brandenberg
Okay, 22 years is a bit slow to rez a texture. Even for us...
Beth Ghostraven
Sometimes I turn around in my own shop, and when I turn back seconds later the textures have to re-rez - it's ridiculous!
Eren Padar
Beth Ghostraven I have experienced that sooo many times Beth. Back in 2007 Balpien Hammerer ran some tests and discovered that SL was reloading the same texturs over and over again, basically creating a DDoS attack on their own asset system. Something was / is obviously wrong with the Cache system.
What I have to wonder is why it is that we can face a wall... with nothing but that wall in our Viewer view... and the textures on that wall aren't given high priority in loading.
Both "field of current view" and "texture on demand" (ie, clicking on a vendor or object) should be given top texture priority. I don't know how SL handles textures, but as I stated in the example above, when companies can stream live video to millions of users across the Net, one would think that LL could send out 2D textures far more efficiently and cache them properly.
Load More
→