spencer's cafe

Search

SearchSearch

Managing UI State With a Reactive Relational Database

Jan 10, 2024, 16 min read

Metadata

interoperability and data-first programming rabbit hole:

Takeaways

  • A lot of incidental complexity in creating software comes from managing and propagating state
  • Reactive models for programming are critical to having understandable programming environments and removing a lot of unnecessary complexity in creating software that all comes from state management
    • This gets to a world where we can create software operate at speed of thought and most of the incidental complexity with creating software disappears. People who have ideas can actualize their ideas
  • Relational data models are state of the art for a reason
    • they are declarative, which decouples performance optimization from the way you write things (in theory, not quite true with all implementations)
    • they allow for complex management of relations
    • leverage years of deep research into database optimizations
  • local-first data builds in interoperability at the foundation. Combining that with a reactive model, you can have external agents change data state and have it all reflected in the app
  • concept of a reactive relational database that ties all of this together, so apps are queries

Highlights

  • We’re exploring a new way to manage data in apps by storing all app state—including the state of the UI—in a single reactive database. Instead of imperatively fetching data from the database, the user writes reactive queries that update with fresh results whenever their dependencies change.
  • apps as queries
  • In data-centric apps, much of the complexity of building and modifying the app comes from managing and propagating state.
  • However, we’ve observed that the primary use of database queries is to manage peristence: that is, storing and retrieving data from disk. We imagine a more expansive role for the relational database, where even data that would normally be kept in an in-memory data structure would be logically maintained “in the database”
  • However, database queries are often not included in the core reactive loop. When a query to a backend database requires an expensive network request, it’s impractical to keep a query constantly updated in real-time; instead, database reads and writes are modeled as side effects which must interact with the reactive system.
  • This limits the scope of reactivity: the UI is guaranteed to show the latest local state, but not the latest state of the overall system.
  • The developer can register reactive queries, where the system guarantees that they will be updated in response to changing data
  • Low latency is a critical property for reactive systems. A small spreadsheet typically updates instantaneously, meaning that the user never needs to worry about stale data; a few seconds of delay when propagating a change would be a different experience altogether. The goal of a UI state management system should be to converge all queries to their new result within a single frame after a write;
  • With a fast database close at hand, this split doesn’t need to exist. What if we instead combined both “UI state” and “app state” into a single state management system? This unified approach would help with managing a reactive query system—if queries need to react to UI state, then the database needs to somehow be aware of that UI state. Such a system could also present a unified system model to a developer, e.g. allow them to view the entire state of a UI in a debugger.
  • This would make it easy to decide to persist some UI state, like the currently active tab in an app. UI state could even be shared among clients—in real-time collaborative applications, it’s often useful to share cursor position, live per-character text contents, and other state that was traditionally relegated to local UI state.
  • Our prototype is implemented as a reactive layer over the SQLite embedded relational database. The reactive layer runs in the UI thread, and sends queries to a SQLite database running locally on-device. For rendering, we use React, which interacts with Riffle via custom hooks.
  • A typical React solution might be to introduce some local component state with the useState hook. But the idiomatic Riffle solution is to avoid React state, and instead to store the UI state in the database.
  • Finally, UI state is persistent by default. It’s often convenient for end-users to have state like sort order or scroll position persisted, but it takes active work for app developers to add these kinds of features. In Riffle, persistence comes for free, although ephemeral state is still easily achievable by setting up component keys accordingly.
  • It’s unusual to send user input through the database before showing it on the screen, but there’s a major advantage to this approach. If we can consistently achieve this performance budget and refresh our reactive queries synchronously, the application becomes easier to reason about, because it always shows a single consistent state at any point in time
  • The Riffle model produces a highly structured app. Each component contains: local relational state reactive queries that transform data a view template for rendering DOM nodes and registering event handlers
  • the sources are the base tables, the sinks are the DOM templates, and the two are connected by a tree of queries. All dependencies are known by the system at runtime.
  • Data-centric design encourages interoperabilityOne quality we found particularly intriguing in our prototype was the ability to control the app from the outside, using the database as an intermediary.
  • This kind of editing doesn’t need to be done manually by a human using a generic UI; it could also be done programmatically by a script or an alternate UI view. We’ve effectively created a data-centric scripting API for interacting with the application, without the original application needing to explicitly work to expose an API. We think this points towards fascinating possibilities for interoperability.
  • Unfortunately, verb-based APIs create an unfortunate n-to-n problem : every app needs to know how to call the APIs of every other app. In contrast, data-based interoperability can use the shared data directly: once an app knows how to read a data format, it can read that data regardless of which app produced it.
  • Many users who are familiar with standard UNIX tools and conventions speak wistfully of “plain text” data formats, despite its disadvantages. We feel that plain text is an unfortunate way to store data in general, but recognize what these users long for: a source of truth that is legible outside of the application, possibly in ways that the application developer never anticipated.
  • We’ve taken a different approach of modeling this as a problem of shared state: both our application and Spotify are reading/writing from the same SQLite database. When the user performs an action, we write that action to the database as an event, which is then synced by a background daemon using the imperative Spotify APIs. Conversely, when something happens in Spotify, we write an event to our local database, and the app updates reactively as it would with an app-created write. Overall, we think shared state is a better abstraction than message passing for many instances of integrations with external services.
    • Note: this needs to handle reconciling conflicts. shared state across multiple clients
  • We found it nice to treat all data, whether ephemeral “UI data” or persistent “app data”, in a uniform way, and to think of persistence as a lightweight property of some data, rather than a foundational part of the data model.
  • In most apps, closing a window is a destructive operation, but we found ourselves delighted to restart the app and find ourselves looking at the same playlist that we were looking at before. It made closing or otherwise “losing” the window feel much safer to us as end-users.
    • Note: persistence by default means you’re always safe to close and not lose state
  • This did lead to another observation, though: in this model, we can decouple restarting the app from resetting the state.
  • In principle, declarative queries should be a step towards good app performance by default. The application developer can model the data conceptually, and it is up to the database to find an efficient way to implement the read and write access patterns of the application. In practice, our results have been mixed.
  • Our prototype stores all state, including ephemeral UI state that would normally live exclusively in the main object graph, in the database, so any change to the layout of that ephemeral state forced a migration
  • Taken to the extreme, we’ve ended up with a strange model of an interactive app, as a sort of full-stack query. Users take actions that are added to an unordered set of events, and then the DOM minimally updates in response. The entire computation in between is handled by a single relational query.
  • Many systems for incremental maintenance work by tracking data provenance: they remember where a particular computation got its inputs, so that it knows when that computation needs to be re-run.
  • Imagine a browser-style developer console that allows you to click on a UI element and see what component it was generated from. In a system with end-to-end provenance, we could identify how this element came to be in a much deeper way, answering questions not just questions like “what component template generated this element?” but “what query results caused that component to be included?” and even “what event caused those query results to look this way?“.
  • outline of an approach where user interfaces are expressed as queries, those queries are executed by a fast, performant incremental maintenance system, and that incremental maintenance gives us detailed data provenance throughout the system. Together, those ideas seem like they could make app development radically simpler and more accessible, possibly so simple that it could be done “at the speed of thought” by users who aren’t skilled in app development.

title: “Managing UI State With a Reactive Relational Database” author: “riffle.systems” url: ”https://riffle.systems/essays/prelude/” date: 2023-12-19 source: hypothesis tags: media/articles

Managing UI State With a Reactive Relational Database

rw-book-cover

Metadata

Highlights

  • We’re exploring a new way to manage data in apps by storing all app state—including the state of the UI—in a single reactive database. Instead of imperatively fetching data from the database, the user writes reactive queries that update with fresh results whenever their dependencies change.
  • apps as queries
  • In data-centric apps, much of the complexity of building and modifying the app comes from managing and propagating state.
  • However, we’ve observed that the primary use of database queries is to manage peristence: that is, storing and retrieving data from disk. We imagine a more expansive role for the relational database, where even data that would normally be kept in an in-memory data structure would be logically maintained “in the database”
  • However, database queries are often not included in the core reactive loop. When a query to a backend database requires an expensive network request, it’s impractical to keep a query constantly updated in real-time; instead, database reads and writes are modeled as side effects which must interact with the reactive system.
  • This limits the scope of reactivity: the UI is guaranteed to show the latest local state, but not the latest state of the overall system.
  • The developer can register reactive queries, where the system guarantees that they will be updated in response to changing data
  • Low latency is a critical property for reactive systems. A small spreadsheet typically updates instantaneously, meaning that the user never needs to worry about stale data; a few seconds of delay when propagating a change would be a different experience altogether. The goal of a UI state management system should be to converge all queries to their new result within a single frame after a write;
  • With a fast database close at hand, this split doesn’t need to exist. What if we instead combined both “UI state” and “app state” into a single state management system? This unified approach would help with managing a reactive query system—if queries need to react to UI state, then the database needs to somehow be aware of that UI state. Such a system could also present a unified system model to a developer, e.g. allow them to view the entire state of a UI in a debugger.
  • This would make it easy to decide to persist some UI state, like the currently active tab in an app. UI state could even be shared among clients—in real-time collaborative applications, it’s often useful to share cursor position, live per-character text contents, and other state that was traditionally relegated to local UI state.
  • Our prototype is implemented as a reactive layer over the SQLite embedded relational database. The reactive layer runs in the UI thread, and sends queries to a SQLite database running locally on-device. For rendering, we use React, which interacts with Riffle via custom hooks.
  • A typical React solution might be to introduce some local component state with the useState hook. But the idiomatic Riffle solution is to avoid React state, and instead to store the UI state in the database.
  • Finally, UI state is persistent by default. It’s often convenient for end-users to have state like sort order or scroll position persisted, but it takes active work for app developers to add these kinds of features. In Riffle, persistence comes for free, although ephemeral state is still easily achievable by setting up component keys accordingly.
  • It’s unusual to send user input through the database before showing it on the screen, but there’s a major advantage to this approach. If we can consistently achieve this performance budget and refresh our reactive queries synchronously, the application becomes easier to reason about, because it always shows a single consistent state at any point in time
  • The Riffle model produces a highly structured app. Each component contains: local relational state reactive queries that transform data a view template for rendering DOM nodes and registering event handlers
  • the sources are the base tables, the sinks are the DOM templates, and the two are connected by a tree of queries. All dependencies are known by the system at runtime.
  • Data-centric design encourages interoperabilityOne quality we found particularly intriguing in our prototype was the ability to control the app from the outside, using the database as an intermediary.
  • This kind of editing doesn’t need to be done manually by a human using a generic UI; it could also be done programmatically by a script or an alternate UI view. We’ve effectively created a data-centric scripting API for interacting with the application, without the original application needing to explicitly work to expose an API. We think this points towards fascinating possibilities for interoperability.
  • Unfortunately, verb-based APIs create an unfortunate n-to-n problem : every app needs to know how to call the APIs of every other app. In contrast, data-based interoperability can use the shared data directly: once an app knows how to read a data format, it can read that data regardless of which app produced it.
  • Many users who are familiar with standard UNIX tools and conventions speak wistfully of “plain text” data formats, despite its disadvantages. We feel that plain text is an unfortunate way to store data in general, but recognize what these users long for: a source of truth that is legible outside of the application, possibly in ways that the application developer never anticipated.
  • We’ve taken a different approach of modeling this as a problem of shared state: both our application and Spotify are reading/writing from the same SQLite database. When the user performs an action, we write that action to the database as an event, which is then synced by a background daemon using the imperative Spotify APIs. Conversely, when something happens in Spotify, we write an event to our local database, and the app updates reactively as it would with an app-created write. Overall, we think shared state is a better abstraction than message passing for many instances of integrations with external services.
    • Note: this needs to handle reconciling conflicts. shared state across multiple clients
  • We found it nice to treat all data, whether ephemeral “UI data” or persistent “app data”, in a uniform way, and to think of persistence as a lightweight property of some data, rather than a foundational part of the data model.
  • In most apps, closing a window is a destructive operation, but we found ourselves delighted to restart the app and find ourselves looking at the same playlist that we were looking at before. It made closing or otherwise “losing” the window feel much safer to us as end-users.
    • Note: persistence by default means you’re always safe to close and not lose state
  • This did lead to another observation, though: in this model, we can decouple restarting the app from resetting the state.
  • In principle, declarative queries should be a step towards good app performance by default. The application developer can model the data conceptually, and it is up to the database to find an efficient way to implement the read and write access patterns of the application. In practice, our results have been mixed.
  • Our prototype stores all state, including ephemeral UI state that would normally live exclusively in the main object graph, in the database, so any change to the layout of that ephemeral state forced a migration
  • Taken to the extreme, we’ve ended up with a strange model of an interactive app, as a sort of full-stack query. Users take actions that are added to an unordered set of events, and then the DOM minimally updates in response. The entire computation in between is handled by a single relational query.
  • Many systems for incremental maintenance work by tracking data provenance: they remember where a particular computation got its inputs, so that it knows when that computation needs to be re-run.
  • Imagine a browser-style developer console that allows you to click on a UI element and see what component it was generated from. In a system with end-to-end provenance, we could identify how this element came to be in a much deeper way, answering questions not just questions like “what component template generated this element?” but “what query results caused that component to be included?” and even “what event caused those query results to look this way?“.
  • outline of an approach where user interfaces are expressed as queries, those queries are executed by a fast, performant incremental maintenance system, and that incremental maintenance gives us detailed data provenance throughout the system. Together, those ideas seem like they could make app development radically simpler and more accessible, possibly so simple that it could be done “at the speed of thought” by users who aren’t skilled in app development.

Graph View

Backlinks

  • No backlinks found