After 22 years... why do textures take so long to rez?
Eren Padar
"If the 2D don't work, the 3D don't work".
I was at Sci Fi Con 17. All day long, no matter how many or how few avatars were on a region, textures took ruddy ages to load. Now one might say "There are so many textures at a big event it just takes a long time." But I'd walk into an enclosed room... where textures should be limited, yet the vendors on the wall just sat there and sat there and sat there. Textures RIGHT NEXT TO ME fail to load. One can stand there and stand there and stand there... and they still don't rez.
I would click on a texture and tell it TEXTURE REFRESH... and it would ignore that instruction.
I first reported texture issues and bottlenecks back in 2005. Now my computer and graphics card is hundreds of times more powerful... and textures still load at a snail pace. I will say the same thing I said back then: There is a texture bottleneck on SL. (Or, several texture bottlenecks, and it affects EVERYTHING.)
With respect, I'm not asking for an explanation, nor excuses. I'm making a simple observation: After 22+ years online, Second Life should be able to very quickly load 2D textures in an enclosed room. Is this not correct?
I suspect there will be people all too ready to give reasons why textures load slowly. To any and all such reasoning, I have one answer: Second Life needs to make textures load quickly. There's no argument, debate, discussion or excuse to that simple statement.
Users need textures to load quickly enough that we don't get bored standing in a store (waiting for vendor textures to load)... and decide to go do something else. Like Netflix. ;D
Because that's what it is: Second Life vs Netflix. Users have options on how to spend their time. When Netflix can stream entire videos realtime, surely Linden Lab should be able to send 2D textures across the Internet... yes?
As a note: I'm not going to debate nor discuss or argue with those who are always ready to argue on these feedbacks. The feedback is posted, I've said what I came to say... and I have to believe the vast majority of SL users will agree with this: textures on SL need to load faster.
Log In
Dan Linden
Thank you for the report, Eren.
In my experience texture loading is significantly faster and complete on a recent viewer than on a viewer from 2 years ago. So there may be a difference between what you see and I see.
The best way to get a bug fixed is to get the bug to reproduce on a Developers machine. To do that, we need to get the bug to reproduce on a QA machine. To do that, QA needs the exact steps that trigger the bug on your machine.
Please share the location of the screenshot https://gyazo.com/fb30ede4a4a86aec9fd2df3909e2112a or IM it to me if you don't want it public.
Please turn on the Texture console and take another photo there and attach it to this bug. What I want to see is if the texture console thinks everything is loaded. To turn on the Texture console, first enable Preferences > Advanced > Show Developer Menu. Then hit ctrl-shift-3 to enable the texture console.
Please do the same at http://maps.secondlife.com/secondlife/LagLand/4/128/85 I would be interested to see if there are textures there that do not load for you.
Does texture loading work better (or worse) at the store location with the SecondLife viewer?
Eren Padar
Dan Linden I'll be happy to do that for you Dan, soon as I get a chance. I can give you part of the information now: that sign wall was taken at Sci Fi Con 17 at my DragonForge booth, SLURL:
There were only a tiny handful of avatars on the region at the time. Additional information to follow later, as I'm a bit swamped this evening.
As a note, I am a member of the Firestorm Preview Testers group, so I'm using the very latest Firestorm BETA Viewer. That said, as the OP mentions these issues have been going on for 22 years (the beginning of SL). So these issues are nothing new, unique, or isolated to a specific location. They happen every day, all the time, everywhere, to thousands of users.
Eren Padar
Dan Linden Dan I ran the test with the TEXTURE CONSOLE as you requested. I still have tests to run and will provide info as I get it... but one thing I noticed is this: if I look at one wall full of textures... the texture console will show them as loaded (and they are if I stand there long enough). But it took a full minute for that wall of about 52 textures to load... and nothing else on my viewer screen. Then once the wall was loaded, I noticed when I zoomed in on the textures, or moved my camera from one side of the wall to the other... the exact same textures would load again, over and over and over, every time I moved my camera. That is what Balpien reported back in 2007 (only at that time we didn't have to move the camera. The textures just kept reloading on their own).
So I wonder why the asset server is reloading textures that should already exist in my cache... just because I move my camera along the same wall. Perhaps there may be a logical and rational explanation, but it seems up-front to be a glitch. Or I could be reading the chart wrong. Dunno exactly how that chart works.
Here's another observation: I can look at a wall in my store... have it fully rez... then visit the store next door. Same region. All textures should be cached. But when I go back to my store, all the textures are gray and fuzzy, and start reloading all over again, no faster than before (ie, apparently not from cache).
Those are my initial observations without extensive testing. I know from experience this will happen repeatedly, pretty much every time. I could go from one store to the other over and over and each time the textures will refresh, although I'm still in the same region. Surely there aren't enough textures in one region to totally fill 12gb GRAM and 9gb disc cache.
Third observation. I've been standing here, staring at the same exact wall, and the textures are loading over and over and over... again appearing to verify Balpien's findings.
Eren Padar
Dan Linden I finished the test you aske for and visited the location you requested. This is the result I got... for a full 60 seconds. I noticed again that if I zoomed in and out on those textures, every time it would go through what appears to be the same reload (but I understand I could be misinterpreting the readout, since I don't really know what all that does). ;D
Once this screen stopped, I zoomed in...
and...
Eren Padar
Dan Linden One last test and note: when I left the SLURL you gave me (that texture wall) and returned to the exact same sim I was on before, looking at the exact same wall... ALL of the textures started out as gray and appeared to be reloading from the asset server all over again. Why the asset server I thinks? Because if those textures were still on my graphics card or disk cache they should have loaded lightning fast. Instead they grayed, then fuzzed, then blurry, and a minute later that 52-texture wall was loaded.
Thanks for your attention to this Dan. Appreciated.
Monty Brandenberg
Eren Padar: If you want to know more about the texture console display, there is some dated documentation available on the wiki: https://wiki.secondlife.com/wiki/Texture_Console It needs some updates but it is close to correct for the SL viewer. There is quite a bit of information so make a cup of tea/pour a cold one first. At the bottom is a 'Some patterns to look for' section which parallels your own questions and may help guide your reading.
Caching has many layers in the texture and mesh systems and I'll list them briefly for reference purposes:
* The asset store itself is a set of S3 buckets in AWS.
* Above this is a Linden asset cache built on the CARP protocol.
* Above that is the CDN origin server which requests assets, caches them, and delivers them to remote CDN endpoints (PoPs or Points-of-Presence).
* Above that are fleets of CDN caching PoPs, one of which will be serving your session.
* Above that are the texture, mesh, and header on-disk caches managed by the Viewer.
* Above that are the in-memory (application) caches for textures and meshes.
* Above that are the on-graphics (system) cache of active visual elements.
The console metrics are mostly about the last three bullets. In doing repeatable testing, we usually try to eliminate the other caches by 'pre-heating' the PoP. But we'll leave that out for now.
Dan Linden
open
Eren Padar
This is what I'm speaking of above and below. This is my shop, textures that I see every day, several times a day. I've been standing here for a full minute, in an enclosed room looking at one wall... and this is what I see.
It took a full 2 minutes for those few textures to fully rez. All textures there are 512x512.
Maestro Linden
needs info
Hi guys, a few notes about texture loading:
- For the past decade or so, the SL Viewer has been fetching texture and mesh assets over HTTP through a global Content Delivery Network (CDN). The SL simulator is not involved in this process, aside from communicating the asset_ids of an objects or textures on in-world objects or avatars.
- If the viewer is able to render a prim at all, it already has all the information it needs to load the object's textures. It is up to the viewer to fetch that asset from the CDN.
- When the viewer fetches a texture to render, it stores it in memory and also on the local disk cache.
- If the viewer's disk cache is full (has reached the cache size set in viewer preferences), the viewer should be purging the assets which were accessed least recently to make room for the new assets.
Because of this design, you're probably seeing a client-side issue, so I'm moving this to viewer bug reports.
Some common causes of poor texture loading performance on the viewer are:
- Some custom setting is active in the viewer, which is reducing performance.
- Antivirus software is blocking (or slowing) the viewer's access to its cache files.
To debug these likely causes, please do the following and report back if the performance issue persists:
- Clear your viewer settings by temporarily renaming the settings file. On Windows, this will be at %APPDATA%\Firestorm_x64\user_settings\settings.xml or %APPDATA%\SecondLife\user_settings\settings.xml , depending on your viewer. Renaming it to something like settings.xml.bak should work.
- If you have Antivirus software running, whitelist specific folders used by the viewer in the antivirus settings. The FireStorm wiki has a guide to do that: https://wiki.firestormviewer.org/antivirus_whitelisting
Maestro Linden
As a benchmark of raw texture loading performance, you can try the following. This benchmark involves the viewer arriving in a scene with many textures, and measuring the total time for the viewer to load them.
- Login to and visit http://maps.secondlife.com/secondlife/LagLand/4/128/85
- Set your viewer's draw distance to 512m, and have your avatar face east to view the walls of textures
- Clear your viewer cache, and exit the viewer
- Launch the viewer, and click to 'Login to Last Location'
- As soon as you click to login, immediately start a stopwatch timer (I recommend using one's phone)
- Keep the viewer widow in focus, so that it runs at full speed
- Open the texture console (Develop -> Consoles -> Texture Console, or Ctrl+Shift+3 ) to view the texture-fetching status
- The 6th line of the texture console will read "Textures: [total texture count] Fetch: [remaining texture count] ...". In my attached screenshot, it shows 5763 total textures with 1724 remaining to fetch.
- The texture console also shows a dynamic list of all the texture assets the viewer is currently fetching, with a progress bar for each texture. The leftmost column contains the first 7 digits of each texture's asset ID.
- Watch the progress in the texture console, and stop the stopwatch when the "Fetch" stat gets to 0 (indicating that all textures have loaded).
- Repeat steps (3) - (8) a few times, to get a few samples.
I did 3 runs with the Second Life Release 7.1.14.15192634334 on Mac OS, and the total times I got were 53.8s, 57.8s, 55.6s. Since there were 5763 textures in the scene and the measured time also included the login process, the viewer was loading textures at a rate of at least 100 assets/s. I would expect some variation based on internet connection speed, proximity to your local CDN node, and viewer performance, but that loading speed should be achievable for many SL users.
Nerxual Oh
Maestro Linden Or you know. You guys could LOOK INTO THE ISSUE. Instead of giving everyone a cookie cutter response "Have you tried turning it on and off?"
This has been the topic in most tiny groups. We're very tired of this.. Personally I did clear cache, I did firewall, DD is set and all of that. It doesn't work. This is a feedback, stop glossing over it please. And look into the issue. That is all we'd like from y'all.
Eren Padar
Maestro Linden I have to agree with Nerxual. First, I'm too old for this stuff. Honestly, in my younger days I would have done all that step by step, but I'm just too old and tired to jump through a dozen hoops for someone else's company.
This is especially the case since frankly, it doesn't matter how many textures you're loading per second if numerous textures FAIL TO LOAD. I provided a picture of my shop, in which my shop name sign failed to rez for 60 seconds. That has nothing to do with textures per second loaded; that has to do with a texture BOTTLENECK (as I've been telling Linden Lab since 2004, when I was still working as a computer consultant).
Consider please (and I'm going to exaggerate to make the point): The Viewer has to load 2,000 textures. Out of those 2,000 textures, 1.800 load in 1/2 a second. WOW! What speed. However, the remaining 200 textures don't load at all and remain either gray or fuzzy.
Would you call that successful texture handling?
So what Nerxual says above. I filed the initial JIRA in 2005. Balpien Hammerer joined in 2007 and we had a HUGE debate with Linden Lab over these issues. (As a note: Balpien is a pro-level tech.)
Balpien built a device which tracked what textures were being loaded and how often, and discovered the same textures were being reloaded from the asset server as many as 30 times in a row. Now multiply that by millions of textures an hour. DDoSA on your own servers? Linden Lab ignored Balpien's findings, claiming textures were working fine (with respect, kind of like the replies we're seeing above, only direct denial of any problem).
In 2010 LL announced it had "discovered" a "texture bottleneck" in its system and lo and behold... it was the same bottleneck Balpien had nailed in 2007.
Seriously, if the user has to jump through a dozen hoops to whitelist software, I have to wonder what's going on software-side (NONE of my other software titles require whitelisting, no matter how complex).
When we stand in a CLOSED ROOM and stare at a wall of three vendors with maybe 25 textures total showing on those vendors (ie, the only thing in my Viewer window) and those textures are all either gray or fuzzy and remain that way for up to a minute or forever... then the number of textures per second have nothing to do with this issue. Again:
Monty Brandenberg
Okay, 22 years is a bit slow to rez a texture. Even for us...
Beth Ghostraven
Sometimes I turn around in my own shop, and when I turn back seconds later the textures have to re-rez - it's ridiculous!
Eren Padar
Beth Ghostraven I have experienced that sooo many times Beth. Back in 2007 Balpien Hammerer ran some tests and discovered that SL was reloading the same texturs over and over again, basically creating a DDoS attack on their own asset system. Something was / is obviously wrong with the Cache system.
What I have to wonder is why it is that we can face a wall... with nothing but that wall in our Viewer view... and the textures on that wall aren't given high priority in loading.
Both "field of current view" and "texture on demand" (ie, clicking on a vendor or object) should be given top texture priority. I don't know how SL handles textures, but as I stated in the example above, when companies can stream live video to millions of users across the Net, one would think that LL could send out 2D textures far more efficiently and cache them properly.
Eren Padar
Follow up to the OP:
This is my store sign. one that I myself made. I stood in front of my store for a full minute before the sign rezzed. All of the rest of the store had rezzed, but that store sign took an extra minute. The question here: why is the rest of my store rezzed, and the store sign isn't? Answer: bottleneck. Texture rezzing needs fixed.
In other locations I had several signs that remained gray for an excessive period of time. There was no pattern, rhyme or reason. Bottlenecks.
Boots Shamrock
Totally with you Eren. I recently upgraded my pc and now have a great CPU and GPU but I’ve hardly noticed any difference.
Nerxual Oh
Because folks abusing 1024/2048 textures in small vendors and ADs. If the frame was large, use 1024. If it's just normal sized frame or small, people got to use something smaller than 1024.
So we end up getting everyone and their grandmother using 2048 for everything. And that can build up lag for everyone.
There needs to be a fix for this and I do hope LL comes through with it. Textures blurring repeatedly, refusing to load is real annoying.
Eren Padar
Nerxual Oh Agreed. In truth, the 2048 textures just added an extra burden to an already texture-lagged system... because this has been going on since before I joined SL in 2004. About 22 years evidently.
Beth Ghostraven
Nerxual Oh this was happening before 2048 textures came out, although that's probably making it worse
Eren Padar
Beth Ghostraven Indeed. And not to beat a living horse, but PBR has doubled that problem when an object now contains BOTH PBR and BP textures, doubling the texture workload. One has to wonder sometimes if LL even considers the big picture when they do something like that.
Shadoskill Heckroth
Nerxual Oh Just to throw this out there, this is not really the case anymore.
You get served a down scaled version of the texture depending on the size it takes up on your screen.
So if some one was using a 2k texture on a earring you would not download that 2k texture unless you zoomed REAL close to it.
Nerxual Oh
Beth Ghostraven oh I know. It's def making it so much worse. I'm glad for 50L$, to give bad actors/uneducated that moment halt and think. So there's that i guess.