High-level UI philosophy

[B-Reif]

There's been some chat in the Discord about how to direct the API for Bevy's UI system. I'm a React dev by day and I wanted to post about some high-level UI dev concepts, and discuss Bevy's take on these things. I'll also post some links to articles and prior art for webdev for further reading.

UI is a pure function of state

For any given app state, we can describe a UI as a pure function that maps the state to an interface. For example:

const Counter = state => 
  <h1>Count: ${state.count}</h1>

This React function component maps the application state to a header that displays the count. We can project this into a wide variety of different interfaces. We could add buttons to increment and decrement the count, a text box to type the count in directly, or something else.

This is a pure function, one that only operates on its arguments and without side effects. This makes sense: we always want the UI to reflect the app state, and a pure function is a kind of reflection.

State updates are data

Events that happen in our application are values describing something that happened. Suppose we add a button to increment the Counter:

const Counter = state => 
  <div>
    <h1>Count: ${state.count}</h1>
    <button onClick={ /* ?? */}> 
      +
    </button>
  </div>

Since we're not mutating the DOM directly, this event needs to signal that something happened. In Elm or Redux, we would create a value (a Message or Action, respectively) that describes what happened in terms of application logic, and pass that to a dispatcher to kick off the state updates:

const Counter = ({ state, dispatch }) => 
  <div>
    <h1>Count: ${state.count}</h1>
    <button onClick={e => dispatch({ type: "Increment" })}> 
      +
    </button>
  </div>

This value is passed into a pure 'reducer' function that returns the next state. This has a few advantages over direct mutation:

• The update values can be serialized, inspected, rewound, and replayed. This helps in creating development tools, makes state easier to debug, and provides easy time travel/action history.
• It's easy to test and make assertions about both the messages themselves and the state emitted by our reducer function.
• This model is decoupled from the actual UI implementation. F#'s Elmish, for example, is completely agnostic about how its "view" values are actually consumed or rendered. This allows developers to change their UI framework without totally rewriting the state management logic.

Imperative view updates are hard

Before React, front-end web dev often took on a kind of imperative form. Here's a snippet of jQuery code:

function counter(state) {
  let counter = $( "#counter-header" );
  counter.html(state.count);
}

This code fetches a DOM element from the document and updates its html content. With jQuery (or pure js), the developer must ensure that the UI reflects state at all times by manually mutating the document each time state changes. This brings problems:

• This impure function assumes the existence of the #counter-header element in the DOM. If the output of $( "#counter-header" ) was null, the application would crash when we tried to invoke .html. (Or in Rust, as an Option type, we'd have to contend with the None case somehow.)
• We have to know when and how often to invoke this mutation function. If we don't invoke it on time, or too late, or not often enough, our application won't reflect state correctly. If we invoke it too often, we're wasting resources.
• We can only use this function in the context of #counter-header. To reuse this, we'd have to parameterize the element id (and at that point we may as well just put the mutation inline).
• This function is difficult to test. The test framework needs to provide a window, a document, an element #counter-header, and any other implicit state this function relies on.

By contrast, declarative frameworks like Elm and React will diff the output of your functions and reconcile the DOM on your behalf. This eliminates the above classes of problems! This is a huge win in development ergonomics over imperative development like IMGUI, jQuery, or Unity's UiElements (and if you don't believe me, take it from John Carmack!)

Declarative view updates need a backend

Any declarative solution needs a framework to do mutations. React's reconciler does a lot of work to diff and update the DOM to reflect state. The React team's work on these internal algorithms is an interesting topic unto itself, and React's design document includes ideas about scheduling updates:

React is not a generic data processing library. It is a library for building user interfaces. We think that it is uniquely positioned in an app to know which computations are relevant right now and which are not.

We can think of a backend like this as a kind of programming runtime, built with its own tradeoffs in mind.

Implications

This post is getting long! To wrap things up, here are a few considerations:

• The ideal building block of a UI is a pure-ish function. In React, this is a component; in Elm, it's simply a function. It's easier to think of the UI as it is at any given slice in time.
• A declarative API incurs some cost as the framework diffs and patches the UI tree.
• An imperative API subjects developers to classes of bugs around when, where, and how often to mutate the UI. It also makes code more difficult to test.
• Bevy is well-positioned to understand game data and when it changes. I think we have a good opportunity to develop a powerful framework to support a declarative API.
  • For instance, we could expose an API that accepts pure functions which accept kinds of game data (entities, components, resources) and returns values of a UI DSL (similar to JSX).
  • This differs from using systems, which implement their own arbitrary behavior. Instead, Bevy itself would implement the behavior, and the game developer would provide a mapping from game data to UI values.
• React's reconciler operates in O(view size), and not O(model size). I think we can do something similar to support a declarative UI without expensive operations on large numbers of entities.

Thanks for reading! Here are some resources that I've linked above for additional reading:

• The React docs: https://reactjs.org/
• The Elmish docs: https://elmish.github.io/elmish/
• About React's internal architecture, called Fiber: https://github.com/acdlite/react-fiber-architecture
• "UI is a function of state": https://www.kn8.lt/blog/ui-is-a-function-of-data/
• "React as a UI Runtime": https://overreacted.io/react-as-a-ui-runtime/

[Nilirad]

It would be cool to have something like a jsx! macro to write declarative code with the possibility to add rust snippets inside.

[cart]

I've done a decent amount of professional webdev and I am generally sold on "reactive" ui programing (I did a reasonable amount of React and Angular 2-X dev at Microsoft). I also think things like Jetpack Compose and Elm are worth learning from. Theres also "immediate mode" UI which has a lot of similarities with "reactive" programming (although the implementation details end up being very different).

Given that we probably want "Bevy ECS components" to be our primary UI datastructure (to integrate nicely with the Bevy app model, take advantage of built in change detection, and work with Bevy Scenes), that means we've already broken ourselves into "retained state jail". It would be hard to make "immediate mode ui" feel integrated in that context. That means we already have a lot of parallels with the web world and it makes sense to learn from it.

I agree that JQuery-style "imperative updates" won't be ideal for complex UI apps. Resolving this problem in a way that works "harmoniously" with the "Bevy App Model" is going to be the hardest problem we face when designing Bevy UI. I have some ideas here (hierarchical Bevy system executors, adding "observables" to bevy ECS, "ecs systems as ui components", etc), but I'll need to do some experimentation to see if they're actually viable. I won't start on that until I've wrapped up my renderer work, but others are free to start experimenting with "reactive bevy ui" designs if they feel inclined. If you find a design you believe in, feel free to create an RFC. Also check out the existing ui rfcs to see what other people are designing.

[cart]

Yeah I think you're effectively describing making bevy ecs "reactive". My general idea for "observables" is basically what you described: register systems that run when a component is mutated. We could even do these updates "within systems" if Queries inherit the access of the observable logic connected to the mutable components they reference. This creates a lot of problems (running the logic immediately on deref is unsound for normal iterators), so we would need to move to for_each for those cases.

We could also just run the "reactions" when commands are applied at the end of a stage. But I think "immediacy" would solve a lot of problems (such as keeping Parent and Children components in sync).

[alice-i-cecile]

The second-kind of system reminds me of some of my early thoughts on event-triggered systems (#1273). I definitely think something similar could be done with change detection. I would love to see this; the only reason my initial RFC on system wiring didn't have this was because I wanted to see how far we could get without needing to add major new features :)

@concave-sphere this lines up nicely with some of your thoughts on data-oriented scheduling; I would love to hear your input on this topic to the extent that you're able to give it.

[alice-i-cecile]

(FYI, this reactivity would be absurdly useful for indexes, as discussed in #1205, as it would help dramatically with exactly the same sort of data-sync challenge.)

[concave-sphere]

@alice-i-cecile

The categorization of systems seems sensible. I don't have a strong enough intuition for the gamedev space, but I would be hesitant to bake the "three kinds" too strongly into the API -- it feels more to me like a good set of use cases that we should make sure are served well by the Bevy ECS. There may be other interaction patterns out there that straddle or sit beside the cases listed.

I do think that the pattern of having a "root-of-truth" and derived state is a really fruitful way to design systems of all kinds, and it has nice properties when you start to layer as well (e.g, my current project has a UI layer and a simulation layer, which combine to feed into a rendering layer; the sim layer really "should" have multiple layers inside it, but that would require too much fiddling with Bevy system scheduling, so I just picked a design that doesn't suffer too badly from being single layered).

This actually gets you part of what was requested, without leaving the ECS "system" model:

Bevy itself would implement the behavior, and the game developer would provide a mapping from game data to UI values.

Here, the "UI values" are just derived state of the "game data", and the "mapping" is a system that computes the UI values. This works great for my toy project, but you shouldn't take my word for it until my UI is more complicated than a system that splats something on the screen when the mouse is clicked!

The tricky part is how to handle incremental updates of derived state. If you can recompute the derived state from scratch every time, this is easy and fits very nicely into data oriented scheduling. If you can't (e.g, there's too much of it), then things become messier and you're subject to various kinds of glitches (e.g, if you modify an Entity but don't update its index entries). Data oriented scheduling can make things more consistent, by ensuring that at least you're clear on whether you're using data from before or after a given update, but it can't fix glitches caused by the code doing the wrong thing after it's scheduled (such as indexing entities that were spawned, but not entities that were updated).

I know that in the "big data" universe, there have been some efforts (e.g, Apache Beam) to unify the programming model for "full recomputation" and "incremental update". I don't know how they do it, though, and my guess is that the performance tradeoffs might not make sense for something like Bevy. Folks over there tend to solve throughput problems by throwing more computers at them, and the latency bounds are measured in minutes.

So we might be "stuck" with explicit incremental updates, at least for now.

[concave-sphere]

I've been chewing on this a bit more. Earlier, I went off on a discussion of incremental updates.

There's a whole separate question here of how you model the dataflow. I generally understand systems best when I can understand them in terms of dataflow (this isn't always possible, but if it isn't possible it often is an architectural flaw, so it's still an interesting exercise).

For the "Observables" or "indexes" that were postulated earlier, like I said in my earlier comment, we're looking at some kind of source / derived data relationship. But I think there's a nuance here that I didn't notice at first reading. I've generally been thinking about dataflow as plugging together a bunch of Systems that receive inputs and produce outputs. With indexes, though, do you really want to say "and my indexing system runs after all other systems have made all their changes to component T"? And what if you need to update T, then run another system that reads T's index, and then update T again? Do you need two copies of the "indexer" system?

(note: I'm going to keep talking about "indexing" and "indexer" as a convenient shorthand, but this really applies to any case where you have some derived data)

I mean, maybe this is OK. I actually would kind of prefer this more manual approach, because it lets the indexer batch up as much work as possible, and avoids accidentally fragmenting the system's execution just because you added a new mutation. But I don't know if it will satisfy all use cases.

There's another option: you can say that the index of T is a sort of "virtual" data item, produced "on demand" by an indexer system. From a versioning point of view, as far as other systems can see, the index is updated atomically with T. In other words, every version of T has a corresponding version of the index. I think this is similar to what Cart was suggesting above with "observables".

But! If we define indexing in terms of dataflow ("run this system to bring the index up-to-date") instead of imperatively ("run this system after this component changes"), something interesting happens. We don't actually need to run the indexer every time that T changes. We only need to run it if something is going to read that version of the index, and the scheduler can choose to defer running the indexer until a version is ready to be read. The other versions of the index exist conceptually, but we don't have to actually calculate them.

This theoretically should allow the scheduler to allocate resources more efficiently. As an example, if the indexed value is written 100 times and the index is read only once, after all 100 mutations have completed, the scheduler could choose to run the indexer once, after all 100 writes, or to run it once per write, or anything in between. Having more options means that there's more flexibility to choose a good option (although finding the right algorithm to do so might be hard).

My main concern about this idea is that it may be pushing things a bit too far -- it adds some complexity to the API and implementation, and it needs to be justified. Before I designed and implemented it, I'd want to understand what the requirements are in terms of semantics and performance. Are there use cases that really need to interleave reads and writes on a single index? And are there use cases where there's a large efficiency difference (in either direction) between incremental and batch updates to an index? I don't have a sense for this, but would be interested in hearing more.

(oh, and there's a tricky issue of versioning -- if the system reads more than one component/resource, what version gets assigned to the output? I have to think about this a bit more, but I suspect some restrictions will be necessary to make this work)