InWorldz Halcyon – Taking A Look Back At 2016
It’s always interesting to try to predict where a new year will take us, and sometimes interesting to look back at the previous year. December and January were very busy times at InWorldz, so I haven’t had a chance to reflect on 2016 until now. That is, along with intentional delays in upgrading servers to avoid disrupting holiday events, is a second reason why there was no Halcyon release in December. And to be honest, we needed a bit of a lighter month just to recover from the October and November monster-sized releases.
But looking back at 2016, it has been a busy year. Yet, we haven’t accomplished everything we had hoped to in 2016. That was partly due to investing so much time and energy into improving crossing reliability, and I think partly due to having a planned three-week development cycle: two weeks for coding new changes, fixes and improvements, and 1 week for testing in a staged rollout and release. Having only two weeks for development just isn’t enough time to solve some of the larger work items, and it is likely that you will see that release cycle increase slightly in 2017. Still, we did manage to do several significant upgrades during the year.
This is part one of a two-part post. The second post will discuss plans for 2017 and beyond.
There are more details on each of these below, but first short high-level summary of each update.
There were two updates in January. The first was a fairly normal incremental improvement. Our goal in that month was to establish a new baseline server version, a stable foundation upon which to build further improvements during the coming year. It mostly included improvements to the performance of parcel border crossing checks and user profiles and some server-side workarounds to viewer issues.
The second (0.9.20) release was a bit of a builder’s dream; it introduced support in InWorldz for SL-style “materials” (normal and specular maps, giving the texture a more 3-D appearance). It also included LSL scripting support for materials (something OpenSim does not yet, and may never be able to support), as well as projection lighting, increased LSL script limits (128K), and lower LI cost.
In February, an update saw another incremental update: the implementation of “Restore To Last Position”, fixed storage of per-prim “scripted” status, major fixes for bot outfits, fixes to parcel management LSL functions such as llAddToLandBanList and llGetLandOwnerAt, and a server-side fix for a viewer issue (auto-cleaning the Current Outfit folder on outfit changes).
March saw another server update with significant performance improvements to parcel border crossing, region crossings, and key fixes to outfits, rezzing, and more.
April‘s update included one critical security fix that we also passed on to the OpenSim folks who were also affected. It also included many improvements to LSL scripting functions, and key bug fixes, to teleport routing, region restarts, and a lot more.
In May, we introduced support for both hard and soft region limits on the number of avatars. The soft limit does not apply to Estate Owner, Estate Manager, or anyone whitelisted (e.g. club/event staff or performers). So the people who need to be there are never locked out from a “full” sim region. We also added iwDeliverInventory and iwDeliverInventoryList, which allow vendors to know if a delivery was a success or not, and if not, why. This update also included other bug fixes, such as plugging a memory leak, attachments, and viewer compatibility.
For June, the update was more of an incremental improvement and included fixes for bots (especially the initial appearance of bots), an SL-compatible llDetectedGrab() implementation for InWorldz, and several other bug fixes.
I was away travelling for the second half of June, so the July update was smaller, with most of the work focused on changes to help identify asset and inventory delays, along with a couple of fixes for maps, JSON handling in LSL scripts.
We had reports of problems with logins to the “last location” so that code was given a complete overhaul in the August update. We fixed problems with Home location, last login, and added support for diverting you to your home region if your desired one was down. If your home region was down, only then did you end up at a default location like the IDI welcome region. Several bug fixes, workarounds for viewer issues, and a couple of security changes (e.g. IP network bans and “teleport user” console command). Also fixed a serious problem where repeated attempts by a banned user to enter a region could cause the region to be marked unavailable to all users. LSL scripting fixes for llGetObjectDetails, llGiveMoney, llMoveToTarget, botFollowAvatar, and added region size returns for llGetEnv.
Much of August and September was spent working on crossings. Unfortunately, those changes were significant and not fully tested in time for the September release which was already bursting at the seams in size. We delayed the September release trying to get sufficient testing in but in the end, we had to defer those crossing fixes until the subsequent (October) release.
In spite of delaying the crossings changes, September‘s 0.9.28 update was a major upgrade. It added support for JSON Web Token (JWT) logins, a feature the MOSES team needed in order to move to InWorldz’ Halcyon servers. Also, after years of requests and demand for “avatar-as-a-prim” support, InWorldz servers were finally updated to include this. This was a major change to behavior and hand to be done very delicately in order to minimize the impact on existing content. In the InWorldz Halcyon implementation, the server assumes the older InWorldz behavior (avatars move with the prim) but automatically switches to the avatar being controllable from LSL scripts if the script makes an attempt to position the avatar directly. This enables all kinds of seated avatar positioning, without, for example, the need for furniture to rez pose balls to sit on (which are then repositioned instead of the avatar). Again, this was a very much requested (demanded) feature supported by SL and missing from InWorldz, and one of the major outstanding work items needed to be completed before Halcyon can be declared version 1.0. Sit targets were also much improved, both in terms of performance and in terms of SL compatibility. Also, many avatars can now sit on a single prim. If there is an unoccupied prim with a sit target, the avatar will be seated there first, but otherwise, the presence of a seated avatar no longer produces the “That seat is taken by another user.” error. There were also lots of LSL scripting improvements, and a couple of unrelated bug fixes.
So in October, InWorldz saw the 0.9.29 update, which finally included the substantial changes to crossings. This was a major update with numerous crossing-related fixes and improvements, as well as significant LSL scripting improvements. It included experimental changes to crossings, where the destination region confirms the crossing, rather than the users’ viewer(s). Fixes several crossing failures and timeouts. There were many other crossing reliability fixes, to try to ensure that — absent an actual Internet communications problem or down region — that the crossing would succeed. In particular, the case of “corner crossings” (two crossings that occur very close together) was handled correctly and without errors. Added the ability to toggle “/!debug crossings” and preserve that status on crossings for end-user crossing reports, as well as other diagnostics to help track down any issues that remained. The October update also included some cleanup fixes for sit target issues after the avatar-as-a-prim changes in the September update. Some LSL script functions were also implemented: llGetAttachedList, iwStandTarget, iwLinkStandTarget, iwGiveMoney and llTransferLindenDollars, including support for transaction_result events. It also included significant changes to parcel ban behaviors: parcel bans to not be enforced for Estate Managers, and prevented Estate Managers from being banned from that estate. Changes to match SL’s banned parcel implementation, as outlined in Mantis 3250. The minimum height over a parcel where an avatar is on the ban list is now 5000m above the terrain height (the new limit is much higher). The minimum height over a parcel with “Public access” disabled is now 50m above the terrain height (generally lower than it was). This update also included several other bug fixes, including a notable problem shift-copying objects, and several Halcyon project improvements.
The November upgrade was a quick update focused on cleaning up the loose ends from the previous two major upgrades. The Halcyon 0.9.30 release only included two changes. The first was a quick but important fix for problems preserving sit targets after some operations, identified in the 0.9.29 update. And as an aid to testing, the llGetPrimitiveParams part of Linden Lab’s recent PRIM_SIT_TARGET parameter addition was implemented.
We wanted to avoid any further disruption of in-world holiday events, so the next update was postponed until early January. But instead of working through December behind the scenes on that update, just before the holidays, we found that the world map on the main InWorldz website was completely broken after apparent changes by Google to their v2 Google Maps support.
The world map is critical for region purchases, so this became a priority to resolve. As a result, the world map was completely rewritten. To avoid a future repeat of this issue, a remote reference to the Google Maps API was replaced with a local reference to a Leaflet.js-based map implementation, which also provides a much leaner, faster maps implementation, much more under IW control.
Here is a more detailed summary of the changes in Halcyon servers for 2016. For an even more detailed description, follow the links to the individual release notes below.
InWorldz Halcyon Server 0.9.19 (Release Notes)
• significant improvements to user profile caching, network delays
• server-side protection against the many repeated duplicate calling cards being created due to a viewer bug
• some fixes to online/offline status
• more complete implementation of parcel access checks
• performance improvements to region crossings and parcel crossings
• cross-region access checks entering a parcel across a region border.
• Major improvements to llGetEnv features
• Halcyon developers grow, include some fixes for loose ends we don’t have time to complete
InWorldz Halcyon Server 0.9.20 (Release Notes)
• full support for materials, related viewer operations
• LSL script support for materials and projection lighting functions
• increased LSL script memory limits (128K per script)
• lower Land Impact (LI) calculations
• *many* reliability and performance improvements (too many to list, see link).
InWorldz Halcyon Server 0.9.21 (Release Notes)
• Implemented the “Restore To Last Position”
• fixed storage of per-prim “scripted” status
• major fixes for bot outfit
• fixes to parcel management LSL functions such as llAddToLandBanList, etc
• llGetLandOwnerAt more reliable
• auto-clean Current Outfit folder on outfit changes (server-side fix for some viewer issues)
InWorldz Halcyon Server 0.9.22 (Release Notes)
• Major crossing optimization: Replaced a very expensive call that was invoked once on a crossing, once for every prim in an object, if there were any seated avatars.
• Additional crossing optimizations, improving crossing performance.
• Fixed rezzing by attachments on group land.
• Support for Current Outfit folder (COF) versions for viewers.
• Fixed the materials code to do a deep copy of on a duplicate operation. This fixes problems with materials after an object shift-copy.
• A very long list of reliability and performance improvements. (See link.)
• Fixed online friend visibility tests which were inverted.
InWorldz Halcyon Server 0.9.23 (Release Notes)
• Fixed a VERY serious vulnerability in llRemoteLoadScriptPin which was also quietly passed on to OpenSim developers.
• LOTS of LSL scripting improvements, fixes, additional functions:
– Significant improvements to llHTTPRequest, llRemoteLoadScriptPin, llJsonSetValue and llDialog.
– Tweaks to botGetPos, llGetMemoryLimit, llJsonSetValue, llSay, llRegionSay, llOwnerSay, llShout, llWhisper, etc.
• Fairly significant fixes to the memory management of bot clients, avatars and client scenes to address memory leaks.
• Changes to XML file handling to do commits out-of-place. This avoids cases where a 0-length Region.XML file was created preventing the region from restarting.
• Fixed the implementation of the teleport routing value: “Blocked”.
• Fixed the server to recognize a region restart time of -1 as a region restart abort request.
• Viewer protocol fixes to add support for missing AlertInfo parameters. Proper implementations for HomePositionSet and RegionSecondsRestart alerts.
InWorldz Halcyon Server 0.9.24 (Release Notes)
• Region Avatar Limits: There are two limits on avatars in a region. The first is a hard limit set by InWorldz (currently 100 avatars). That limit will deny even the Estate Owner and Estate Managers. This update adds support for the second, soft limit, which can be set or changed on the Region/Estate form. The InWorldz implementation differs from the implementation on other grids such as SL, in that it will still allow the Estate Owner, Estate Managers, and anyone on the region access whitelist to enter the region. This works for cases where a club or performance group may always want its club staff or performers to get in after a crash, but otherwise, apply an agent limit to the region as a whole.
• Implemented iwDeliverInventory and iwDeliverInventoryList, as well as the IW_DELIVER_* series of error return code constants (more info at the link above).
• Fixed a problem with bots sometimes not actually deleting attachments when removed
• Fixes to the initial viewer COF version numbering.
• Fixed appearance changes to avatar layers incorrectly resetting the COF version number to 0.
• Several references were not being cleared in session Close methods, holding references and prolonging object memory usage (memory leak).
InWorldz Halcyon Server 0.9.25 (Release Notes)
• Fixed the initial appearance of bots to show correctly. Also fixed a second case where full updates for bots were not sent correctly (e.g. botChangeOutfit).
• Fixed a case-sensitivity issue with all users’ name lookups. Because MySQL is case-insensitive, lookups of normal user profiles would be successful but would bypass the user profile cache. This was making them slower than they needed to be, and adding unplanned load on the core database.
• Fix a regression in llGiveInventory and the other Link and List variants that added unwanted new delays when the delivery was directed to an object.
• Fixed some race conditions related to removing bots. Fixed two cases of null reference exceptions if botRemoveBot is called at the wrong time.
• Provides an SL-compatible llDetectedGrab() implementation for InWorldz.
• Added a new InWorldz-specific constant for llGetObjectDetails and iwGetAgentList, named IW_OBJECT_SCRIPT_MEMORY_USED.
InWorldz Halcyon Server 0.9.26 (Release Notes)
• Fixed the infinite “Loading…” message on the World Map when clicking on invalid region locations (now displays “Invalid Location”).
• Fixed problems related to the llJson2List handling of a JSON object with a null value for an element.
• Implemented detailed statistics gathering on asset-related operations on cloud storage.
• Added an implementation for the “show assets” console command to dump these statistics, or to reset them with “show assets reset”.
• Added support in the Cassandra-related inventory code (internal inventory browser tool) for purging and recreating empty folders.
• Several Halcyon project enhancements from the Halcyon developers, useful for development.
InWorldz Halcyon Server 0.9.27 (Release Notes)
• Several login-related and Home location-related fixes, as well as LSL improvements.
• Fixed a problem with stale connection info being used when fetching user profiles. Fixes “last location” logins not going to last logoff location, and probably other cases where the cached info was stale.
• Fixed a problem where repeated attempts to enter the region by a banned user could cause the region to be marked as unavailable to all users.
• If the specified login location is not available, the User login service now tries user’s Home location before falling back to alternative default locations (like IDI).
• Fixed a null reference exception that could occur if the user has no home region at all.
• Added support for IP network bans in the central User login server itself (rather than region servers).
• Added support for the “HomeLocation” CAP (region capability). This is a server-side fix for the Firestorm Viewer regression in the Firestorm 4.7.9 update.
• Eliminated the superfluous “Notecard saved.” messages.
• Fixes to llGetObjectDetails, llGiveMoney, llMoveToTarget, botFollowAvatar, and added region size returns for llGetEnv.
• Added a “teleport user” console command to allow a grid administrator to force-teleport a user.
• Lots of Halcyon project updates.
InWorldz Halcyon Server 0.9.28 (Release Notes)
• This was a *major* update. Please see the link above for more detail. Highlights below.
• Support for remote console logins based on JSON Web Token (JWT). This is a feature that allows the MOSES team to move to InWorldz Halcyon-based servers for their core production systems.
• Avatar-as-a-Prim (AaaP) changes, which allow a script to treat a seated avatar as if it were another prim on the end of the link set. This enables all kinds of seated avatar positioning, without, for example, the need for furniture to rez pose balls to sit on (which are then repositioned instead of the avatar). Was a very much requested (demanded) feature supported by SL and missing from InWorldz.
• Sit target handling is much improved, both in terms of performance and in terms of SL compatibility.
• The most noticeable change is that many avatars can now sit on a single prim. If there is an unoccupied prim with a sit target, the avatar will be seated there first, but otherwise, the presence of a seated avatar no longer produces the “That seat is taken by another user.” error.
• Various improvements to the LINK_ “wildcards” handling to allow them to be used in more cases now, more consistently. These also now include seated avatars where they match.
• Unlike SL, llGetLinkPrimitiveParams supports the above LINK_ wildcards, so for example, that you can pass LINK_SET, [LINK_NAME, LINK_POSITION] and get a list with prim and seated avatar names and positions. Any reasonable combination is supported. (This differs from SL where the response is not useful; SL always returns an empty list.)
• Where a list of object links are provided in some form, the links are now ordered by link number.
• Updates to llGetLinkName and llGetLinkKey to be consistent with the other similar functions, as well as to support seated avatar “links” now.
• Fixed a long-standing bug that could cause updates from an avatar to not be sent to other avatars.
InWorldz Halcyon Server 0.9.29 (Release Notes)
• This was a *major* update with *numerous* crossing-related fixes and improvements, as well as significant LSL scripting improvements.
• This update included experimental changes to crossings, where the destination region confirms the crossing, rather than the users’ viewer(s). Fixes several crossing failures and timeouts.
• Fixed to make sure the agent (avatar) is only released from the original region once. There’s a race between a fast viewer response and the region server response.
• Fixed a deadlock regression seen on InWorldz Sandbox 1 during developer testing there. There was a very long “position info” lock that included some heavy calls that included nested locks.
• Fixed it so that flying avatars cannot bounce back into the previous region on corner crossings. Avoids the case where bounce-back during a corner crossing puts you in a seemingly random location.
• Fixed the “connection established” check to directly use the destination region from the transit arguments where possible. Avoids a consistency ambiguity and eliminates an error case.
• Fixed to not allow a prim crossing to even be [i]started[/i] if seated avatars weren’t yet in the region. These are the same pre-checks that were done later on each avatar, [i]after[/i] the object was already committed to leaving the region (or had). This one fixes several crossing failures.
• Added the ability to toggle “/!debug crossings” and preserve that status on crossings for end-user crossing reports.
• Several additional changes to help diagnose crossing performance.
• Fixed a problem where a new call to llSitTarget was clearing the reference to any avatar that was still seated. This was causing llAvatarOnSitTarget to return NULL_KEY if called, even while an avatar was seated, if llSitTarget was called while the avatar was seated.
• Implemented llGetAttachedList as per SL.
• Implemented llTransferLindenDollars, including support for transaction_result events.
• Implemented iwGiveMoney and factored the code that is common with llTransferLindenDollars. The main difference is that iwGiveMoney does not require (or generate) a transaction_result event, but returns the transaction ID from the transaction history directly, or one of the error tags on an error (both as strings).
• Added support for stand targets and LSL functions: iwStandTarget and iwLinkStandTarget.
• Changed parcel bans to not be enforced for Estate Managers.
• Prevent Estate Managers from being banned from that estate, but allow removal of bans (for EO and EM).
• Changes to match SL’s banned parcel implementation, as outlined in Mantis 3250. The minimum height over a parcel where the avatar is on the ban list is now 5000m above the terrain height (the new limit is much higher). The minimum height over a parcel with “Public access” disabled is now 50m above the terrain height (generally lower than it was).
• Fixed runtime permissions to avoid changing to 0 when a new permissions popup is sent to the user. (It needs to only be updated on the response (which it is already doing), or if there is nobody present to respond (as per SL).
• Fixed a shallow copy (reference copying) in region objects. Changes to things like sit target lists (e.g. llSitTarget), or LSL scripted targets (e.g. llTarget), bounding boxes, and other references in one copy of an object affected all copies of that object that were shift-copied.
• Several Halcyon project improvements.
InWorldz Halcyon Server 0.9.30 (Release Notes)
• This was a quick fix release for problems preserving sit targets after some operations, identified in the 0.9.29 update.
• As an aid to testing, the llGetPrimitiveParams part of Linden Lab’s recent PRIM_SIT_TARGET parameter addition has been implemented.
World Maps Work (See world map)
• The world map on the main InWorldz website was completely rewritten after apparent changes by Google to their v2 Google Maps support broke the site.
• To avoid a repeat of this issue, a remote reference to the Google Maps API was replaced with a local reference to a Leaflet.js-based map implementation.
• In addition to ensuring that we aren’t bitten by this problem again, it also provides a much leaner, faster maps implementation, much more under IW control.