matter.js

033110a3 - New high-level API for 0.8 with full device support (#715)

New high-level API for 0.8 with full device support (#715) * Squashed commit of the following: commit f77735696c37c9268089c290add690aa3d592c87 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Jan 5 19:06:02 2024 -0800 Tweaks & fixes - More reasonable name for PartStoreService - Improved number assignment - Fix state deserialization commit 0ca1b0bfa8190a8a6c64c0940b563e06b7519ae6 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Jan 5 13:08:34 2024 -0800 Squashed commit of the following: commit 9f1958ecf2593388c04a70629ee6915c85118bf8 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Jan 5 13:06:15 2024 -0800 More polish, logging, bug fixes commit 1eb10f4b56471e188b31115dd16387677e743237 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Jan 5 04:02:33 2024 -0800 Hopefully final pass on persistence Fixes several parts of persistence workflow that were broken and handles a bunch of edge cases related to asynchronous initialization. commit 72a90d3735c96f5e05f3fce6f7b043621f1009c5 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Jan 4 23:24:59 2024 -0800 Server now starts. Lots more fit & finish and bug fixes. Proceeding to working through diagnostics flow. commit bf11ca046607f9cd2cfd3e4a6f3044fa54bbc138 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Jan 4 16:04:14 2024 -0800 Significant polishing and tightening Focused primarily on state management commit 5e0ed8775a083e028e03c1dda716e0d3bec312e2 Author: Greg Lauckhart <greg@lauckhart.com> Date: Wed Jan 3 17:00:08 2024 -0800 continued work commit 6b8ebd1b4a97c7efcc7a2ae109007a5164309ea4 Author: Greg Lauckhart <greg@lauckhart.com> Date: Wed Jan 3 14:07:28 2024 -0800 WIP - streamlining lifecycle management + bug fixes commit 6e9bbb45e2c9eda85fe4177f830c8df9e1cbbd2a Author: Greg Lauckhart <greg@lauckhart.com> Date: Mon Jan 1 04:25:29 2024 -0800 WIP - Back to bringing node up commit 7fec7c20f4bf8349e2e8e2e5785ead13b77a99da Author: Greg Lauckhart <greg@lauckhart.com> Date: Mon Jan 1 03:32:06 2024 -0800 Fix up tests commit 339471894e03bdadd5ca618bd2ef5f58b86b40f2 Author: Greg Lauckhart <greg@lauckhart.com> Date: Sun Dec 31 16:54:42 2023 -0800 WIP: Refactor cluster lifecycle handling. Adds uniqueness checks for part ID and number. Corrects descriptor behavior root and aggregator endpoints with "flat" namespaces. Adds index for retrieving parts from flat namespaces by ID or name. Simplifies logic and implementation related to lifecycle and descriptor. commit cdd03b7b4b19856f72525a89217931a79f126696 Author: Greg Lauckhart <greg@lauckhart.com> Date: Sat Dec 30 18:34:59 2023 -0800 Squashed commit of the following: commit 8854828a515161a9cd37018cbe577a64e793ef8c Author: Greg Lauckhart <greg@lauckhart.com> Date: Sat Dec 30 18:22:06 2023 -0800 Storage refactor complete. commit f7efbec28565cf42e9ac97096a5b38f93ac26d65 Author: Greg Lauckhart <greg@lauckhart.com> Date: Sat Dec 30 11:23:47 2023 -0800 WIP - fix up rest of matter.js tests commit 728f7b6a01b46d2fe03bb8263982a81c1c906376 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Dec 29 23:47:31 2023 -0800 Fix AttributeServerTest Old tests started version numbers at 0, 1 and sometimes 3 so needed to adjust assertions to match. Some of the tests also used post-increment for increasing the version. Logic seemed OK but assertions were asserting erronious behavior. Fixed those too. Also fixed an actual bug or two in test rewrite. commit 19aa92cfbb47ec46686ef0f8b48d72c676500880 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Dec 29 17:04:47 2023 -0800 WIP Remove redundant code in several tests, refactor v0.8 API for changes in storage management. commit a8aa7c7d634474fe2a309cb27f540312f6e5c7c5 Merge: 5e211144 ac956483 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Dec 28 01:50:50 2023 -0800 Merge remote-tracking branch 'chip/device-gen' into device-gen commit 5e211144482b8e664df622f2e17c5dd4f4db1f06 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Dec 28 01:46:55 2023 -0800 WIP Refactor storage handling for CommissioningServer, InteractionServer and ClusterServer. This decouples storage from clusters somewhat and allows for more flexibility in how NodeServer handles storage. commit c128816dad1dd430d013e501493233a1ed986af2 Author: Greg Lauckhart <greg@lauckhart.com> Date: Sat Dec 23 19:27:32 2023 -0800 Bring back initialization options This decomposes server options and associates them with behaviors. Also move some commissioning related node functionality from the node to a root endpoint behavior. commit 0db5e354582a36669981171b386a07b2222ea319 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Dec 22 12:05:24 2023 -0800 Commissioning WIP commit 6c22b67e77b2ca9ebb95c6ec2bc382f489c01a77 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Dec 22 02:51:13 2023 -0800 Working through commissioning commit ac9564838fe185d127019e34cb7b2ad8cfbc437e Author: Ingo Fischer <github@fischer-ka.de> Date: Sat Dec 23 00:02:02 2023 +0100 Declare current examples Legacy by renaming files (#593) * Rename old examples Files to *Legacy * add deprecation notice commit 702ee9ea40be3619361aa075a58580e179de2feb Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Dec 21 22:03:22 2023 -0800 Remove temporary kludges from OnOffServer commit 68e18af335ee3d04ba99c9bbcb4e8cc0862d0bfb Merge: afc50ec7 62dbb83a Author: lauckhart <greg@lauckhart.com> Date: Thu Dec 21 21:56:02 2023 -0800 Merge pull request #591 from project-chip/onoff-revamp Revamp OnOff Cluster logic to specs commit afc50ec79f1535d383f6aeae02503a33c9efadcf Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Dec 21 21:10:11 2023 -0800 Fixes & tests Updates for TS 5.3, additional testing, fix various bugs in (heretofor untested) node startup logic. commit 62dbb83a9edd986cfe696cf64169f685f7705f37 Author: Ingo Fischer <github@fischer-ka.de> Date: Thu Dec 21 22:45:53 2023 +0100 Make OnOff a Complete Cluster Handler commit c6b894e9eaa0d9a79434cb8fb4bd81eaa75a1318 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Dec 21 09:40:31 2023 -0800 wip commit 35acb66a279973fa38a35b8d75cb1dd92b46aa92 Author: Ingo Fischer <github@fischer-ka.de> Date: Thu Dec 21 18:25:37 2023 +0100 Make sure no node_modules is committed unplanned commit 6f8308778ec7caa7a94899108166ca4a4e8e8d6c Author: Ingo Fischer <github@fischer-ka.de> Date: Thu Dec 21 18:11:40 2023 +0100 adjust to use on/off only for easier inheritance commit dd4ae4bb0aa3125530139bf5c4904a7aae9bb6d7 Author: Ingo Fischer <github@fischer-ka.de> Date: Thu Dec 21 14:49:58 2023 +0100 prettier commit 1f2de5cbeab8c136406d7a876b7cc07f2d90b281 Author: Ingo Fischer <github@fischer-ka.de> Date: Thu Dec 21 14:47:54 2023 +0100 Revamp OnOff Cluster logic to specs ... still 1.1ish commit fce98fc692974bb67a149b4891e11287fa90de51 Author: Greg Lauckhart <greg@lauckhart.com> Date: Wed Dec 20 18:42:58 2023 -0800 Tweaks, some fixes commit 7827a5c555b1963430b091ee27d9c5c1efdb9d99 Author: Greg Lauckhart <greg@lauckhart.com> Date: Wed Dec 20 13:08:19 2023 -0800 Build project + dependencies on test commit 2ea09bb507766640db39a0bbe177bd0ef325d53d Author: Greg Lauckhart <greg@lauckhart.com> Date: Wed Dec 20 01:55:42 2023 -0800 Reject illegal write attribute paths Also added a note for one place I think the behavior is still incorrect. commit e6265a8eee37d3bedf85830e0d0f6651fb43f723 Author: Greg Lauckhart <greg@lauckhart.com> Date: Wed Dec 20 00:48:36 2023 -0800 Fix matter-node.js tests Includes a couple of places I need to follow up on commit e28373fadeb18086cea211b64027a1f785076894 Author: Greg Lauckhart <greg@lauckhart.com> Date: Tue Dec 19 02:52:18 2023 -0800 New device API. * Fix persistence of newly assigned number * Diagnostic framework & wiring Upgrades logging, adds diagnostic framework and wires into various modules. * Fixes * Minor tweaks * WIP - Refactoring, diagnostics & node API progress Finished refactoring of BaseNodeServer Added diagnostics for tracking generic object lifecycle and applied to behaviors Added tracking of construction promises Tweaked various aspects of NodeServer commissioning (WIP) * WIP - error handling, diagnostics, behavior fixes * More error handling & bug fixes Usual bunch of new functionality too * Hierarchical part storage * Remove MatterServer as dependency for Host Various other fixes. * Excessive prettiness, better error handling, misc * Ignore global attributes + better defaults. * Various fixes & hygiene * More updates for device API Implemented semantic factory reset (still needs testing). Fixed some bugs. Made AsyncConstruction state mutable and observable so you can use it to broadcast lifecycle changes and perform reinitialization. Improved logging. Etc. Also made Observable async capable and allow for return values from observers, neither of which I ended up using yet but will be useful in the future. * Fixes * Invocation ACL enforcement, fixes Fixed feature flag matching for the conformance validator. Implemented command ACL enforcement and loosened enforcement during invocation. A few other minor things. * Matter.js fixes, test fixes & code gen tweaks Now generating requirements as independent definitions so they can be individually documented and referenced more easily. * Device code gen Resurrected package.json mods that got lost in a merge. Fixed and reran code gen. * Fix a bit of JS weirdness * Better time reset logic Integration test leaks the "real" time integration if it crashes... Rather than add a bunch of finallies I just made the test runner ensure MockTime is installed before each suite. * Revert 0.7 APIs Now that so little is shared between NodeServer and CommissioningServer it no no longer makes sense to have a common base class. Many of the changes now were just moving things around, and there were breaking differences in the API that are no longer necessary. This reverts most of the changes to MatterNode, CommissioningServer, CommissioningController and MatterServer. CommissioningServer still has a number of internal changes as it integrates components that do have changes. But I don't think any of these changes affect the public API. * Start of list & struct management tests Bit of testing infrastructure. Rewrote handling of fabric-scoped lists to handle non-transactional mutation from different fabrics simultaneously. Not necessarily of real-world use but more complete and makes writing tests simpler. * New device API. * Tweaks, some fixes * Fixes & tests Updates for TS 5.3, additional testing, fix various bugs in (heretofor untested) node startup logic. * Squashed commit of the following: commit 8854828a515161a9cd37018cbe577a64e793ef8c Author: Greg Lauckhart <greg@lauckhart.com> Date: Sat Dec 30 18:22:06 2023 -0800 Storage refactor complete. commit f7efbec28565cf42e9ac97096a5b38f93ac26d65 Author: Greg Lauckhart <greg@lauckhart.com> Date: Sat Dec 30 11:23:47 2023 -0800 WIP - fix up rest of matter.js tests commit 728f7b6a01b46d2fe03bb8263982a81c1c906376 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Dec 29 23:47:31 2023 -0800 Fix AttributeServerTest Old tests started version numbers at 0, 1 and sometimes 3 so needed to adjust assertions to match. Some of the tests also used post-increment for increasing the version. Logic seemed OK but assertions were asserting erronious behavior. Fixed those too. Also fixed an actual bug or two in test rewrite. commit 19aa92cfbb47ec46686ef0f8b48d72c676500880 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Dec 29 17:04:47 2023 -0800 WIP Remove redundant code in several tests, refactor v0.8 API for changes in storage management. commit a8aa7c7d634474fe2a309cb27f540312f6e5c7c5 Merge: 5e211144 ac956483 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Dec 28 01:50:50 2023 -0800 Merge remote-tracking branch 'chip/device-gen' into device-gen commit 5e211144482b8e664df622f2e17c5dd4f4db1f06 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Dec 28 01:46:55 2023 -0800 WIP Refactor storage handling for CommissioningServer, InteractionServer and ClusterServer. This decouples storage from clusters somewhat and allows for more flexibility in how NodeServer handles storage. commit c128816dad1dd430d013e501493233a1ed986af2 Author: Greg Lauckhart <greg@lauckhart.com> Date: Sat Dec 23 19:27:32 2023 -0800 Bring back initialization options This decomposes server options and associates them with behaviors. Also move some commissioning related node functionality from the node to a root endpoint behavior. commit 0db5e354582a36669981171b386a07b2222ea319 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Dec 22 12:05:24 2023 -0800 Commissioning WIP commit 6c22b67e77b2ca9ebb95c6ec2bc382f489c01a77 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Dec 22 02:51:13 2023 -0800 Working through commissioning * WIP: Refactor cluster lifecycle handling. Adds uniqueness checks for part ID and number. Corrects descriptor behavior root and aggregator endpoints with "flat" namespaces. Adds index for retrieving parts from flat namespaces by ID or name. Simplifies logic and implementation related to lifecycle and descriptor. * Fix up tests * Squashed commit of the following: commit 9f1958ecf2593388c04a70629ee6915c85118bf8 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Jan 5 13:06:15 2024 -0800 More polish, logging, bug fixes commit 1eb10f4b56471e188b31115dd16387677e743237 Author: Greg Lauckhart <greg@lauckhart.com> Date: Fri Jan 5 04:02:33 2024 -0800 Hopefully final pass on persistence Fixes several parts of persistence workflow that were broken and handles a bunch of edge cases related to asynchronous initialization. commit 72a90d3735c96f5e05f3fce6f7b043621f1009c5 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Jan 4 23:24:59 2024 -0800 Server now starts. Lots more fit & finish and bug fixes. Proceeding to working through diagnostics flow. commit bf11ca046607f9cd2cfd3e4a6f3044fa54bbc138 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Jan 4 16:04:14 2024 -0800 Significant polishing and tightening Focused primarily on state management commit 5e0ed8775a083e028e03c1dda716e0d3bec312e2 Author: Greg Lauckhart <greg@lauckhart.com> Date: Wed Jan 3 17:00:08 2024 -0800 continued work commit 6b8ebd1b4a97c7efcc7a2ae109007a5164309ea4 Author: Greg Lauckhart <greg@lauckhart.com> Date: Wed Jan 3 14:07:28 2024 -0800 WIP - streamlining lifecycle management + bug fixes commit 6e9bbb45e2c9eda85fe4177f830c8df9e1cbbd2a Author: Greg Lauckhart <greg@lauckhart.com> Date: Mon Jan 1 04:25:29 2024 -0800 WIP - Back to bringing node up * Tweaks & fixes - More reasonable name for PartStoreService - Improved number assignment - Fix state deserialization * No sparse array for nocs/fabrics * change handler creation needs clusterServer to be set on backing object * Enhance proxy for array handling * Cloning must also support objects * Allow non fabricFiltered reads * Log "undefined" * Support sparse arrays in storage and restore "undefined" (8)and update) * Use current message/session for initial subscription data reads * Fixes and tests * More datasource tests, fix versions Fleshed out datasource tests with version handling and transactional commit tests. Fixed bug whereby version wasn't save when initialized with no changes and when only non-volatile values changed. * Optional version persistence Disable version persistence for non-cluster behaviors so they can initialize synchronously. Also fixed up tests that were broken by async initialization of cluster behaviors. * ActionContext and transactionality changes Formalizes online/offline ActionContext variants. Removes support for non-transactional state mutation. * checkpoint * Checkpoint * Test fixes, formalized behavior reactivity Adds new Behavior.reactTo() functionality for reacting to other behaviors. Fixes up a bunch of tests for API changes though not quite all yet. Also added MaybePromise.finally(). Should also do catch() for completeness but haven't yet. * Fix build and some tests and bugs * API changes & test fixes - Removed Host, BaseNodeServer, NodeServer, NodeClient, PartOwner and node/options/* - Added ServerNode and ClientNode which extend Part - Create new matter.js/environment module with generic environmental services: - MdnsService manages broadcaster & scanner - RuntimeService manages long-lived workers (nodes & MDNS being two) - StorageService manages storage - VariableService manages configuration - Node networking is moved to NetworkBehavior - Added context-free access to (read-only) state and events via part.state.behaviorName and part.events.behaviorName - Environment now supports simple hierarchical delegation; replaces old node serviceFor - Move non-matter behaviors from behaviors/definitions to behaviors/system - Modify diagnostic location management a bit to convey location along with reference - Fix up matter.js tests - Promise API updates - Added CancellablePromise interface to support general cancellation of long-running workers - Added proper catch and finally helpers for MaybePromise and AsyncConstruction - Updates to transaction management - Added "purpose" to action contexts to improve diagnostic messages - Convert shared transactions from snapshot isolation to read committed as it's more intuitive - Reactors may now specify locks required for reaction - Fix several bugs in state management, in particular unwrapping managed values on insert * Update node.js environment, examples and exports * Runtime rewrites + ServerNode tests * Fix storage root * Test fixes * docs on RuntimeService * Low-level API for endpoints + fixes Creates "EndpointStructuralAdapter" which is a very minimal API for interacting with endpoints. Includes an implementation for EndpointInterface and for Part. Made events emit once transaction commit is complete rather than within the transaction. Various other fixes. * Format * Add missing event handler wiring * Add timed info to context * Read initial SUbscription data with correct context * Fix typing error * Fix shutdown * Fixes and Excelsior 1000 EZ-Nite A composed device example and various fixes, partially untested. * Fix botched merge * Fixes - Add dependency definitions for behaviors to allow for proper destruction order - Do not increment version number on construction and destruction of behaviors if unless non-volatile fields change - Fix static ID assignment - Import matter-node.js in IlluminatedRollerShade.ts because otherwise it is not loaded Partially tested -- IlluminatedRollerShade starts and shuts down cleanly but has one error I need to track down. But out of time tonight. * Various fixes. Tests are broken but out of time! * Bit of additional formality for lifecycle Three new documented interfaces: Startable = supports start() Cancellable = supports cancel() Destructable = supports destroy() and optionally tc39 "explicit resource management" Made ServerNode Startable and Cancellable and tweaked environment support a bit to cover new interfaces. Modified NetworkRuntime and ServerRuntime so network does not go online until node is started. * Tweak advertisement handling Standardize on terminology "advertisement window" and "advertise" to cover both commissioning announcement and operational discovery broadcasts. Add comments to better describe behavior. Made "online" event trigger after the advertisement window is open unless the node isn't configured for automatic advertisement. * Fixes, one api extension API extension is "callback" method on Behavior. This method wraps a callback so it can be invoked on an offline behavior. Works as an adapter for non-Observable APIs with callbacks. Fixes: - Cleaned up AdministratorCommissioningServer a bit and made callbacks run in proper context - Do not drop the timeout promise in AdministratorCommissioning (we do this a lot with timers) - Make NetworkBehavior.start() properly a no-op if already started - Fixed Part validating child ID too early upon insertion Also better documented expected behavior of Startable and Cancellable interfaces. * Forgot to commit * Fixes Fixes to AdministrationCommissioningServer were incomplete. Use callback() for timers in OnOffServer. * Bridged nodes emit redundant reachability changes * Minor additions Couple of utilities on DescriptorServer and Agent. Modified BridgedNodeBasicInformationServer to ensure its Part is a BridgedNodeEndpoint owned by an AggregatorEndpoint. * Improved configurability of Node.js process * Fixes Mostly fixes to tests, fix one bug, minor cleanup. * Action tracing infrastructure * Run format, remove circular reference. * Couple of fixes Botched "unwrapping" of managed values when inserting into object. Also fix issue w/ array refs when array is newly assigned, may have only been a test issue. Added additional ManagedStruct tests. * WIP * Tests & fixes. Removed StructuralEndpointAdapter and instead extracted state management out of MatterDevice and FailsafeManager. * Format * Renamed TimedOperation and loosened requirements TimedOperation -> FailsafeContext FailsafeManager -> FailsafeTimer ClusterBehaviors now validate successfully so long as a cluster is supported regardless of base class. * Factory reset, fixes, quality of life - Completed testing of factory reset and reworked a few bits of it - Fixed various issues turned up during testing - Implemented a variety of quality-of-life updates Factory reset test blocked by a synchronization issue with MatterDevice with no obvious workaround so I probably need to address. Test is commented out for now, probably would pass otherwise. * Initialization resilience * Fix dropped lifecycle changes * Additional lifecycle change event testing * Handle premature exit during await Logs an error when node would otherwise exit silently with error code 13 but we are still awaiting the runtime. * Improved initialization error logging Awaiting construction would result in errors that point to the original source of the error and did not include the trace of the awaiter. It would also potentially result in redundant log messages. Instead we now log the original error once and throw a new error anywhere waiting on construction. * Do not write to immutable value. * Terminate on construction errors by default If a behavior crashes the part now crashes. Inconvenient override point by subclassing Part but not sure there's a use case for this so not exposing at higher level. If a part crashes the node now crashes. May override by subclassing or adding observer to "partCrash" event and returning false. * Async create for nodes and add for parts * Part.set This is an asynchronous method for updating Part state. Input is an "patch" object that includes only fields to modify. It is applied recursively to each behavior and the behavior's subfields. Applies patches atomically. * Do not generate type definitions for examples This confuses TS and ends up with TS2742 because @project-chip/matter.js is not a direct dependency for matter-node.js-examples. Also formatting. * Rename "offline" to "act" and better document * Fixes and more tests - Add tests for Part.set - Fix Part.set bugs turned up by tests - Make Part.set install default values when applying patch where no object was previously present - Implement ownKeys and getOwnPropertyDescriptor for managed list proxies (used by for...in loops) - Add functional tests for ClusterServerBacking, only partially complete ATM - Fix bugs in fabric-scoping turned up by previous * Finish fabric-filtered read * Load shims for Node < 19 in direct mocha * Just set globalThis.crypto Using node-shims doesn't work when run as ESM. * Import webcrypto * Define isGlobalAttribute differently Previous definition was causing the property to serialize. * Fix cluster composition Features flags weren't set if they ingested no elements * Fix extended cluster when no elements * Fix variable load order * Devices-Gen test Fixes and Updates (#694) * Cluster adjustments from tests Some of them are device-gen fixes, some adoptions from master * needed to prevent locking issues I got an error when it was executed by the #commissioningTimeout worker * Faabric scoped !== sensitive * Correctly initialize isCommissioned flag * Destroy ServerStore on factory Reset because will get reinitialized * Add convenient "startUp" method for ServerNode it is like "run" but resolves already when node is online. We can change the name but "start" was already used, so I decided for this for now. Lets talk * Add info on react/callback * Prevent danging promise ... Or do we have a better way in the meantime? * Removed unused settings * Initialize Part Name * Separate Init- and Defaultvalue in Attributeservers ... and handle initialization errors with fallback * Adjust test to adjusted BasicInformation definition In a former commit we adjusted the BasicInformation type to include the events ... with this these testcases do not work anymore. I did npt found any way to get them working again. * Needed Adjustments to Legacy cluster impl * Finalize/Fix OnOffServer implementation * Implement Identify logic (no legacy) * Add has method to VariableService * Streamline mdns interface settings to one * Document Environment variables * Set log format to ANSI when tty and no config provided ... as legacy logic * use getter for hasSignalSupport * A Closed Json storage can be reinitialized * Former commit needs Data model version exported * Enhance logging * Updated package-lock * Add test scripts for devices-gen logic * Fix * Adjust Nocs/Fabrics to be fabric scoped * Fixes * Address review feedback * Update NodeJsEnvironment.ts * Fixes First, behaviors now derive from: *.Cluster.with(<all non-exclusive features>) instead of *.Complete The complete cluster was kind of a hack and doesn't have the proper extension metadata to support with() correctly. This would result in us incorrectly indicating support for elements. It also includes conditional rules that are redundant and sometimes incorrect based on the limited metadata structure. Second, in ValidatedElements detection of unsupported commands was a bit broken. We'd warn a command wasn't supported if it was selected by feature but was optional. And we'd then incorrectly register as supported. * Tests + fixes Made servers just use the same base cluster as the behaviors. Added test for ServerNode announcement and expiration but not quite working yet as operationalPort is still read too early Run script now conveys error code if child process exits with error code instead of throwing unhandled exception. * Fix double announce and incorrect port * Examples and stuff (#696) * Rename startUp->bringOnline * Examples and Readme * Linter * Also register SIGTERM * Readme * 0.8 Migration guide, changelog tasklist * 0.8 Migration guide, changelog tasklist * Adjust commissioning config ble was not used because now on network, automaticAnnouncement was hard pinnxed to true and not iverwritable. And there seems to be a state value issue currently - means: optional state values will never be taken from config into state! * Respect automaticAnnouncement false * Fix example * Revert change because wrong place * Fix automaticAnnouncement * Fix BLE only announcement with late mdns init * Various updates Rename ServerRuntime to ServerNetworkRuntime. Rename destroy() to close() Change Symbol.asyncDispose to close in several places Add close() to MdnsInstanceBroadcaster and block MdnsBroadcaster.close() until all instance broadcasters close Add InstanceBroadcaster guards to prevent use of closed instances A few quality-of-life updates to tooling Make server reset default to mindless destruction Separate "soft" reset (memory-only) from factory reset Move reset logic from NodeLifecycle into ServerNode Add "mutex" class and use it to simplify and improve reliability of ServerNode workers Fix spurious warning about missing node ID Fix a couple of diagnostic values that weren't incremented correctly Convert a few more classes to new private variable convention * Forgot to add * Fix state restoration * Fixes - Close server store when node is closed - Do not exit commissioning mode in ServerNodeNetwork until the node officially emits commissioned - Do not consider node commissioned so long as there is an active failsafe context * Do not track "is new" state of stores More trouble than it was worth which was a single warning that the persisted endpoint ID was not present * d'oh * Test fixes The awaits I added yesterday invalidated the tests but a bug was hiding this fact. The bug was causing crashes in web tests so they're fixed now too. * Format * Add Custom cluster * Adjustments * Readme and migration guide update * Fixes and convenience utilities. Fixes cover a few of the lint promise warnings. Convenience utilities allow for simpler attribute and event access on parts for behaviors that are not present in the endpoint's TS type. * Rename "Part" to "Endpoint" * Bug fixes - Apparent but in Node's crypto requires macrotask yield rather than microtask. Added that to MockTime.resolve and CryptoSubtle.deriveBits manages to complete. - An untracked promise in MatterDevice made it impossible to ensure internals were no longer in use at close(). Stuck the promise into a Mutex and awaited the mutex on close. - Also renamed MatterDevice.stop() to MatterDevice.close() * Added aggregator test Doesn't capture the error I was going for though * Add Tx sanity check and fix bugs Transaction now rejects participants if participant identity is already present. This prevents multiple participants from mutating the same data which is indicative of a logic error. Fixed Behavior initialize/destroy where this could turn up and transaction "roles" which was probably just an optimization but also not working correctly. Fixes serverList for bridged endpoints in BridgedDeviceNode.ts. * Better handle recursive behavior initialization * Remove resolved stuff * Add NetworkCommissioning Spec fix * Change noble/bleno package to better maintained one * BLE optimizations * Introduce ChannelNotConnectedError To stop sending retries and accept session as closed. Is used for BLE and will in future be used for TPC * Consider an added bridged device as reachable ... if not set else by developer * Fix Default state value * Remove automaticAnnouncement flag ... because will produce issues with Factory reset * Split MDNS enablement and BLE removal ... and introduce info on the commissioningChanged callback about the action (Add, Remove, Update) * Restore ble default to undefined * Simplify examples * Migration doc adjustments * Prettify * Adjust testing with changes * Session behavior & refactoring - Added "SessionBehavior", a lightweight interface to session metadata - Remove MatterDevice as intermediary for session creation - Make session lifecycle observable and used to replace single-use callbacks - Make environmental service lifecycle observable - Place FabricManager and SessionManager in environment; remove MatterDevice - Rename "UnsecureSession" to "InsecureSession" - Migrate Session, SecureSession, InsecureSession and SessionManager to ES privates + getters - Rename "networking" directory to "network" - Remove ServerRootEndpoint which was redundant with ServerNode.RootEndpoint * Formatting again * add FAQ to docs * Handle theoretical case * Initialize SessionBehavior before announcing device * Finalize SessionsBehavior * Make sure service events are reusable * type and rename exposed fabrics getter * COmplete rename unsecure -> insecure * Add all events to DeviceFull Example * Enhance DevicesFull docs * Formatting * Update migration docs * fix * Consider console.log always being non-fabric-filtered! * Add Readme info for shell * Squashed commit of the following: commit da5056979a08e069d6fc632c170ed876e8aa12b5 Author: Greg Lauckhart <greg@lauckhart.com> Date: Thu Feb 29 15:53:27 2024 -0800 Fix tests & lint errors - Another overhaul of reactor implementation to simplify and better track activity - Remove promise tracking as node activity tracking covers a similar need at a higher level - Make internal state accessible without an active context - Revert to unconditional (maybe) promise return on act() methods because there's no way to avoid async commit - Resolve lint errors - Omit validators for "any" schema; allows state properties to initialize later (and small optimization) commit ee5abe8c9c7527405c73c8426112a30d0de937a9 Author: Greg Lauckhart <greg@lauckhart.com> Date: Wed Feb 28 16:29:34 2024 -0800 Lint fixes + quality of life - Add endpoint to context available in Val.Dynamic - Use above to move pairing codes into CommissioningServer.State - Add callback equivalents for AsyncConstruction promise methods so eslint doesn't expect an await - Add a flag to Observable that conveys runtime information regarding async support - Add AsyncObservable that injects above flag and designates return value as MaybePromise - Make reactors omit the promise return for async reactors on non-async observables - Convert Transaction promise/destroyed to onShared/onClose methods for installing callbacks so eslint doesn't expect an await - Various other small eslint errors fixed - Update tests to conform with above (code complete but not all tests passing yet) commit d54e01a70dd12e22e1adc46046e3d70c3c2aae8e Author: Greg Lauckhart <greg@lauckhart.com> Date: Tue Feb 27 12:08:32 2024 -0800 Simplify node activity, reactor tests NodeActivity now only tracks a single "actor" type and added elapsed time. Implemented reactor tests and fixed a cyclical promise error tests turned up. commit 5c2047aa9a5f86b5ae8dec23be2b95bd45f0df27 Author: Greg Lauckhart <greg@lauckhart.com> Date: Tue Feb 27 00:17:46 2024 -0800 Misc - Code formatting - Choose test package based on specs if supplied - Workaround for Mocha bug that would silently drop certain errors - Test flag for enabling Mocha debugging - Include reactors in activity tracking - Rewrite reaction to make eslint happy and fix a few gaps in logic - Add reactor tests (WIP) - Tweak new logging features of StructManager for mocha/chai compatibility - Hide group & scene warning spam - Report count of locks released instead of resource names - Make list methods throw on write and return empty data on read if expired - Fix some floating promises commit eae2dcc3eb04320501053481dd64a34a9e6833df Author: Greg Lauckhart <greg@lauckhart.com> Date: Mon Feb 26 03:04:50 2024 -0800 Track node activity Tracks active contexts associated with a node. Use this information to make create() wait until all initialization activity is complete. commit 8b726ef9c6bd3f0cd3c19d89a328b06de7ee9048 Author: Greg Lauckhart <greg@lauckhart.com> Date: Mon Feb 26 00:11:24 2024 -0800 More "quality of life" stuff - Flatten stack with more removal of MaybePromise utility methods - Render structs and lists like normal objects and arrays on node - toString for structs and lists that displays contents - Lock state manually in DescriptorServer to avoid flooding log on startup - Perform change detection when updating PartsList to reduce unnecessary locks - Better name/toStringTag for struct classes - Rewrap more comments commit 3d80978f4ae40ffede623349a822552dee4ba4b5 Author: Greg Lauckhart <greg@lauckhart.com> Date: Sat Feb 24 18:25:04 2024 -0800 Tests, quality of life Quality of life improvements mainly focus on improving stack traces: - Give names to various functions in critical paths that were previously anonymous - Give names to class-like objects that do not use classes and were previously anonymous - Greatly reduce use of MaybePromise.* utility functions in critical code paths. Makes code slightly harder to read but makes stack traces easier to comprehend so I think it's worth the tradeoff - Remove a handful of functions in critical paths that increased stack size but didn't add anything logically significant - Make async transaction commit1's (which are not currently common) run in parallel (removed a couple of functions from stack) Tests are for DescriptorServer and are WIP (currently failing) * Post-merge fixes - Workaround for a bit of new eslint silliness - Rename "CommissiongServerFailsafeContext.js" to English - Merge in @Apollon77's log changes, skipped in rebase as was too messy - Replaced package-lock.json * Fix botched merge * A bit of stack shortening & deanonymization * Bridge tests, a few niceties - Added tests for bridge startup - Fixed bug in bridge startup - Tweaked logging & cleaned up stack - Added a few convenience methods on Endpoint and Parts * Needed to add this else had build errors. Unsure if this is future proof and generic enough * make sure the Mem storage verifies initialization logic * There might be no session when ending commissioning ... eg when done by timeout * There might be no failsafe context ... * Update docs * Fix test * Adjust Mig guide * Fix example * Fix uninitialized error messages * Fix storage tests. * Fix hangs in DescriptorServer test When run as part of full suite some lingering background process was throwing off timing and causing initialization to complete earlier than expected. * Fix matter-node.js storage tests * Update copyright * More copywrite updates * Set test cwd to package root * revert package change * Make PairedNode reconnect attempts in a loop Previous recursive method made very long stack traces * Fix IntegrationTest CommissioningServer needs to open new MdnsInstanceBroadcaster after closing now that MdnsInstanceBroadcaster has a "closed" state. * Misc - More complete copyright update - Allow observers to be async even if emitter type is not async - General "multiplex" sibling interface to "mutex" implemented by RuntimeService - Do not exit "run" process on SIGINT until child process exits * Fix dangling reference to "parts" in codegen * Commissioning events and Sensor example (#710) * Todo update * Fix storage init flag * Fix storage init flag * new matter 1.2+ tests not compatible to us * update mig guide * Move Fabric events to FabricManager * Rewire Fabrics logic and always rebuild To prevent out of sync issues we rewire the new OperationalCredentials cluster for now and also rebuild the data structures on fabric changes instead of trying to build a own copy. This change is also always commited directly because in fact it already is realized in fabric Manager * Rewire Commissioning and add change event * Finish Rewiring * no announcements when we close down * Enhance Device Full example * Update mig docs * Update readme with Google change * Add Sensor device example * Update Sensor device example * Address review feedback * Fixes for Sensor example * Fixes for Sensor example * Update network state recursively - Update network state for any endpoint that supports NetworkCommissioningBehavior - Modify Endpoint.visit() to support async visitor - Change prettier config to only ignore behavior files that are not human editable * Doc updates. - Push interfaces higher in the behavior files so documentation appears closer to top of file - Add a series of small "tutorial" examples - Add additional documentation on Node API that includes new examples * Missing example file * Start work on Environment enabled Controller * Minor tweaks * Lint fixes * Fabrics adjustment and Environment aware Controller (#713) * Readme and Changelog * Adjust imports in examples * Only send fabric update event if it is really an update This can be removed again once we remove the fabric scoped storage there (soon when we update Gropups and add scenes again). * Only send fabric update event if it is really an update This can be removed again once we remove the fabric scoped storage there (soon when we update Gropups and add scenes again). * Adjust Failsafe for auto updating fabric structures * Adjust monkey patching in legacy code path * make CommissioningController startable in new Environment * Restore some code in documented way as reference * Use peer-dependencies * revert peer-deps and prepare 0.8 * finalize * [execute-chiptests-long] Format *Server.ts files * [execute-chiptests-long] Do not include Zigbee's "query" feature on identify * [execute-chiptests-long] remove en0 limitation from local tests * [execute-chiptests-long] remove device composition test for new API for now ... because it do not like the ScenesCluster we put in * [execute-chiptests-long]Update deps and adjust Fabric sensitive reads in Legacy API * [execute-chiptests-long] Fix Fabric sensitive read and lint error * [execute-chiptests-long] Revert promises finally change * [execute-chiptests-long] Fix Fabric Filtered read * Workaround and copious documentation of eslint bug * [execute-chiptests-long] Docs and disable CodeQL for problematic file * [execute-chiptests-long] disable CodeQL for test files * [execute-chiptests-long] disable CodeQL for test files * Address codeql comments --------- Co-authored-by: Ingo Fischer <github@fischer-ka.de>