Bitwise operators from Lua 5.3 | Voters

Bitwise operators from Lua 5.3

tracked

Wulfie Reanimator

Traditional LSL requires bitwise operators quite often: event parameters (changed
), function call flags (llRezObjectWithParams
), ...
Lua 5.3 introduced native bitwise operators: https://www.lua.org/manual/5.3/manual.html#3.4.2
Having these usable in SLua would be a significant quality-of-life improvement in the face of Luau's bit32
 library, though the library does still include other very useful features like btest
, replace
, etc. The library should stay regardless.
To be fair, Luau's reasoning for not including bitwise operators is decent, but LSL context adds quite a heavy weight on this decision. I don't know the work that would go into implementing these operators, but hopefully they can be implemented in such a way that makes use of the same VM built-ins as the bit32
 library to avoid any performance penalties.

April 1, 2025

Harold Linden

marked this post as

tracked

We're tracking this internally, if we do end up implementing this it will likely be during the beta phase or after due to the hairiness of modifying the AST / parser passes.

If we do end up implementing this,

and

are unlikely to be included due to parsing difficulties with Luau's lexer / parser design.

Wulfie Reanimator

Thinking about the LSL use cases specifically, I wonder if implementing &
, |
 and ~
 (unary and binary) would be enough?
The shift operators do
 come up, but not nearly as often as The Big Three, all of which can show up in a single control
 event. (Just look at the wiki example: https://wiki.secondlife.com/wiki/Control) It would cover the vast majority of cases without having to touch the issue of parsing >>
.
Having to do shifts, replacements, etc. through the existing library doesn't seem like the end of the world to me, as someone who does bit-packing in weapon scripts quite often.

Harold Linden

Wulfie Reanimator: That seems reasonable to me, those are all sigils that exist within Luau already but have no meaning outside of type expressions.

Frionil Fang

I very much would like bitwise ops, but from my experience the bit32 performance is already debatable and in many cases it's just plain more efficient to use arithmetic instead (e.g. rshift(band(value, 0xff00), 8) generally ends up being worse than

(value//0x100)%0x100

, and if you're just combining flags that you know can't overlap, + is much better than bor), so if they were implemented I'd rather have them be first-class bytecode citizens and not just function calls with a prettier syntax.

Harold Linden

Frionil Fang:
>bit32 performance is already debatable [...] I'd rather have them be first-class bytecode citizens and not just function calls with a prettier syntax.
At least part of that can be explained by us just not being very smart about function calls to builtins right now. Currently Luau forces them to go through a VM_INTERRUPT()
 check, which is generally a no-op in Luau but ends up calling out to the script scheduler in SLua to see if the script has overrun its time slice. We could likely eliminate that perfomance gap by designating certain FASTCALL
 indices as exempt from scheduler checks on the basis of them being trivial and constant-time. That's essentially what LSL/Mono does for a number of operations.

Frionil Fang

Harold Linden

Agaiin, fair enough, thanks. I'll stay tuned for changes.

Harold Linden

Frionil Fang: Created https://feedback.secondlife.com/slua-alpha/p/relax-interrupt-checks-for-constant-time-cheap-library-functions for this so I don't forget :)

Frionil Fang

Harold Linden

Excellent news for all assembly-C-brains (all three of us).

Harold Linden

Frionil Fang As an aside, do you find that

bit32.extract()

is any faster than

bit32.rshift(bit32.band(val, mask), num_bits)

or floor division + mod? It should do basically the same thing in a single op.

Frionil Fang

Harold Linden
In earlier tests (and to be sure, I checked and it still applies), extracting bits with floor div + modulo is roughly 60% faster than bex. Band+rshift is naturally even slower than extract, with its 2 calls.
Other bit ops have their specialized moments where doing things with arithmetic and compares would end up losing, especially bit32.btest thanks to giving you a true/false to work with, but for basic bit manipulations they generally end up being slower.
As a side note a bit32.popcount(value) -> number of 1-bits for dense flag storage could be neat... boolean tables get kinda memory hungry quick (something like 56 bytes for the table and 16 for each bool is not sustainable for long assuming we're going to stick with the classic 64 k limit).

Harold Linden

We ended up (mostly) resolving this out-of-band. bit32.extract()
 is equivalent or faster than floor div + mod, and always faster than shift + mask.
> As a side note a bit32.popcount(value) -> number of 1-bits for dense flag storage could be neat... boolean tables get kinda memory hungry quick (something like 56 bytes for the table and 16 for each bool is not sustainable for long assuming we're going to stick with the classic 64 k limit).
That seems reasonable to me. I'd submit that as a feature request to Luau first, and if they don't accept we'll revisit that as a feature request.
But yeah, in general you never want to use a table of boolean
s because every
 scalar in Lua uses 16 bytes due to them all being boxed in TValue
s. Part of why we had to make quaternion
 a garbage-collectable userdata unlike vector
 is we would have had to make all TValue
s wide enough to fit 4 floats.

SuzannaLinn Resident

What about leaving it to the preprocessor?

Harold Linden

SuzannaLinn Resident: A preprocessor can't easily do it because you need to be able to parse enough of the code to decide whether

is part of a type expression or a binary operation, it's not enough to just tokenize the code like a C preprocessor would do. blindly replacing

with

bit32.rshift(lhs, rhs)

would break valid Luau code like

local foo: Not<cls & Not<Unrelated>> = something

If that's a grammatical ambiguity that a preprocessor could resolve easily and enough people want that behavior, the logic should just live in the compiler's lexer / parser.

Harold Linden

One other thing to consider would be if adding

as an operator would mess with Luau's type expression parsing. I don't know if we can treat

as a separate lexeme easily without breaking them, similar to C++'s previous issues of requiring a space for

std::vector<std::vector<whatever> >

so the template closing brackets weren't treated as right shift.

If anyone's interested in having a look before we're able to get around to it (might be a bit) these would be the relevant places to look https://github.com/luau-lang/luau/blob/6b33251b89a11b7d945dcb77e77604e4b208b524/Ast/src/Parser.cpp#L3086 and https://github.com/luau-lang/luau/blob/6b33251b89a11b7d945dcb77e77604e4b208b524/Ast/src/Lexer.cpp#L819

Harold Linden

marked this post as

under review

Harold Linden

We're not ideologically opposed to adding bitwise operators in the way that Roblox is (we don't really care if we don't support __shl
 and __shr
 metamethods,) but it would have to be done in a way that didn't add a ton of maintenance burden on us and people maintaining tooling for SLua.
Particularly, We're concerned about it breaking compatibility with the existing Luau ecosystem (luau-lsp and friends.)
If it could be added as a de-sugaring pass that converts lhs >> rhs
 into bit32.lshift(lhs, rhs)
 before the usual type-checking pass that might be okay, but that wouldn't necessarily solve the tooling problems.

Kadah Coba

For what its worth, I have made various uses of the shift operators in the past, though I'm pretty sure bit32.lshift/bit32.rshift would work fine even in the stranger uses cases I've done in the past.