This week I did more work on the demo engine/editor of course :)

This includes nailing down more architecture-related stuff; basically I think I have enough now to call the architecture stuff "finished" and say "the rest is details", more or less. So, I thought it'd be a good time to finally write some of this stuff out.

Note: I wrote this writeup initially in the repo's internal wiki, and I'm basically going to just copy/paste it here. Unfortunately this means you don't have the code to go with it (probably), so some things might seem a bit vague, but there's enough there to describe it back to myself in the future I think, and that's what matters. And of course I've made some small edits here where in the wiki there were links to other pages, etc.

So, here we go :)

General architecture

Major Components

Before diving into the project's architecture, it's necessary to understand the following high-level terms:

Node

Nodes are the fundamental building blocks of everything we're making in the tool. Really, the various nodes we have, the way they're put together, and animation data for certain node properties represent the entire value of the project. The tool's sole existence is to make these things easy to put together and tweak.

Each node has a list of properties, a list of children (represented as indexes into a list, which we'll get to in a moment), and a render function that will be used to draw the actual node.

Nodes are expected to be stored in an indexed list. This reduces references to specific nodes to integer indexes, as well as makes the model a bit more 64k-friendly. The NodeList type exists specifically for this use case. The root of the scene graph is always the first node in this list (node 0).

Additionally, node names are expected to be unique. There are cases where, for example, an editor might need to find a node by name. If there are ambiguities, this can't work.

Each node property has a type (currently only represented by the actual data type itself), a string name, its current value, and a list of decorators. These decorators help the editor display the properties properly; for example, the DisplayName decorator represents a prettified name to be displayed in the editor for the property rather than its normal name (which follows the Rust snake-case naming convention; more on that later). All decorators are optional and are not checked for compatibility with a particular property. The editor will simply look for certain decorators in contexts where it makes sense; that's it :) .

Engine

The engine project is a standalone project that contains just about everything needed to represent and draw nodes, minus lower-level things like window setup. It contains GL bindings and abstractions the nodes can use (such as textures, vertex buffers, etc), the actual node representation, and an asset manager that can be used to resolve resources at runtime. This allows assets to be reloaded if changed on disk primarily.

Context

This is a small data object that's passed to each node as the scene graph is traversed. It contains information like which render target is bound, the current world/view matrices, current time, etc. These values are maintained as stacks, and nodes can push/pop new values here as they please. Really, this is just a very basic mechanism to keep track of state that's not node-specific as the scene is being rendered/traversed.

Editor

This is a tool to edit actual projects. It contains editors for specific parts of the project (the scene graph, various properties, etc) and a render window to show the current output as these values are changed. In the future it will also likely contain additional render windows, an animation suite, etc.

Most of the work put in thus far has been in finding the right architecture to fit the various parts of the editor together nicely. The rest of this writeup will deal with the editor architecture specifically.

Editor Architecture

Because the editor will deal with almost all parts of project creation/editing (in fact the only thing it doesn't touch is actual text editing), nailing the architecture of this is paramount to making sure the project never gets out of hand and changes/additions are always possible. A great deal of effort has been put forward to come up with an architecture that satisfies all of our design constraints, while at the same time trying to be elegant (though there are some cases where elegance is dropped in favor of actually working, but these aren't terribly common and I wouldn't call any part of the project a "mess" [yet :P]).

Design Constraints

These constraints must always be upheld when working on the tool:

• It must be cross-platform. Though the actual executables produced in the end will likely only be for Windows (as per demoscene convention), parts of our team (namely myself) aren't always using that OS, and it should be possible to fire up the tool whenever and start playing. So, we at least need multiple OS support.
• As many actions as humanly possible should be undo-able. This is extremely important to be able to freely edit projects without worry, and is something that's hard to retrofit, so this has to be considered from the beginning.
• It must be performant. It has to feel silky smooth when doing things like editing properties etc.
• Adding new node types should be a smooth experience. Generally, the editor should never have to change to incorporate new node types, and all properties on these nodes should "just work".
• The editor must be compatible with the Rust engine/data model. Since the engine is written in Rust, the major parts of the domain model probably should be as well, and the editor needs to be able to work with this as simply as possible.
• The editor should look at least remotely pretty. This is probably the weakest of all constraints, but still one that's going to matter if we plan on staring at the tool for hours on end.

Languages/Frameworks

These should probably be covered first, as they're the decisions that will affect most everything else in the project.

• Qt: Pretty much the only native cross-platform toolkit that provides a performant, well-documented way to do user interfaces. Also provides a visual editor for orchestrating widgets/layouts. This is used for basically the entire UI layer.
• C++: Because we're using Qt, C++ is necessary for working at this layer.
• Rust: Since the engine is written in Rust, major parts of the tool are as well.
• C: Used to form a thin bridge layer between C++ and Rust. Also helps to enforce some of the tool's policies, as it can be quite difficult to do things over this bridge.

Major Components

What follows is a somewhat high-level overview of the most important components in the editor.

Views

These are the actual specific editors used to modify various parts of the domain model. This includes the inspector, the animation editor, etc.

Domain Model

The domain model contains basically all state the editor exists to modify. This includes the nodes, animation data, history, and some editor state such as currently selected node index.

The domain model is made up of a few different parts:

Editor Context

This is the actual owner of all of the domain model state, minus some immutable data objects that might be floating around (such as strings and change items). Generally, there should only be one editor context instance for the entire tool. This is implemented in Rust with a C++ abstraction (through a C API). Almost everything the tool can see or can do will eventually go through this object.

While this object by definition is a "God object", it makes sense to have as it represents a hard boundary between the editor and the data parts of the tool. This not only helps enforce certain editor policies, but also allows us to swap our either the tool or the domain model while keeping the other side in-tact, given this object's API doesn't change.

Because this object contains the project's nodes, it's necessary that all method calls on the object are synchronous and happen on the same thread. There is no internal sync provided for this. Generally, this is ok though, as most UI frameworks are single-threaded by default anyways (including Qt).

Data Model

This represents the actual state of the editor context. All introspection and mutation of the state eventually gets delegated to this object. It includes the project's node list, history, and some editor state such as which node is selected.

Change item

Change items are immutable POD's that describe specific changes to the data model, such as a change of a float property, selection of a node, etc.

The primary reason for having change items is to represent the project's history so that everything can support undo/redo.

These change items are also used to notify the UI when changes occur. In most tool architectures, this is handled by using the observer pattern to have the UI components observe changes on the model. However, in the past I've found managing observer subscriptions to be somewhat difficult in C++, and when considering the fact that our data model is in Rust, implementing the observer pattern would mean at the very least having to pass handlers and managing subscriptions over the C boundary, which I assumed would be complex and error prone. Instead, we can take advantage of data structures we already have in our system to notify the UI of the same data. While it could be argued that we may have more propagation overhead, since all controls will be notified for all changes (instead of just the subscribed controls for particular changes in the observer pattern), the amount of controls in the editor should never be enormous and the actual handling of change items by the editors isn't expensive.

They can also be inverted; for example a change item representing a change of a float property's inverse should represent changing the property to the value it was previously. This means that wherever a change item would be propagated, we only have to care about the change item's new values; a change item being undone for example is simply a propagation of the change item's inverse.

On the C++ side, there exists an IHandleChangeItem interface that implementors can use to signify they handle incoming change items. All views are expected to implement this to handle incoming data model changes, either from the current editor, other editors, or undo/redo.

History

The history is simply a list of change items that have occurred in the project while the tool's been running. This allows for undo/redo.

Component Interactions and Policies

In order to ensure that all changes on the model play nicely with the undo/redo and editor update mechanisms, certain policies must be enforced.

Luckily, there are actually only a few:

1. Views/editors have full visibility of the data model, but cannot make changes directly.
2. All changes to the data model must be delegated to functions on the Rust side that have more complete domain knowledge than any specific view/editor might have. The editor can decide whether or not these changes are committed to the history (which allows certain interactions, such as sliding a slider, to update the model with intermediate values before a final change is committed to the history when an interaction ends).
3. All change items produced by the domain model are expected to be propagated to all editors in the UI. This includes editors being expected to propagate change items to their child editors.
4. Editors must not make changes to the data model as a result of an incoming change item. When data model changes are requested, they are assumed to be complete and correct (which is why specific editors with limited domain model knowledge can't make these changes). For example, if an editor wishes to remove a node from the node graph, it's not correct for the editor to simply remove that node, because other nodes might also have references to the node being removed. However, semantically, "removing the node" also includes clearing the references to that node, so the editor should instead call a domain model function that represents just removing the node and assume this function with complete domain model knowledge will also do the other necessary work and produce appropriate change item(s).

All of these policies except #4 can be enforced by simply providing abstractions to the editors whose implementations ensure correct operation. Primarily, this is done with the following objects:

• ISceneGraph: Represents an immutable view of the scene graph.
• SceneGraphMutator: Exposes an interface for making changes to the scene graph.

Under the hood, SceneGraphMutator also handles change item propagation to the root of the UI. Therefore, if all the editors/views require an ISceneGraph and a SceneGraphMutator (not the editor context directly), expect the SceneGraphMutator's implementation to update the data model wholly, and they implement IHandleChangeItem to handle all relevant change items (and possibly route them to any child editors/views they might have) without making any additional data model changes in response to them, all of our policies are upheld and the editor works smoothly and stays fast.

Some Implemention Notes

I figured I should mention some implemention details, particularly regarding some GL-specific things and memory management.

Ownership of the editor context is managed only by the tool C++-style with no additional mechanisms necessary. This is because there will likely only ever be one per editor instance; this means we can always just take a pointer to the context and assume it will be live for the entire lifetime of the app. This works in most cases, however, one thing to be aware of is that our GL context is created by the viewport's GL control instance. So, while we can assume this will be around basically until our app closes, we do have to wait to actually feed this context to some editors until it's actually created. This is handled very easily, but important to write a note about I think.

The C bridge really only consists of some functions that take and/or return simple data types, such as integers, floats, etc. However, there are a couple higher-level data types they can accept/return, such as strings and change items. These higher-level data types are all managed on the Rust side using reference-counting, which is safe because they're immutable. In C++, they're wrapped in simple classes that are meant to be passed around by value that will increment/decrement the reference count automatically, so management of these objects is trivial.

There are a few cases where interfaces and objects are used in mixed company. Generally, with a small codebase like this, I don't like to introduce interfaces until we actually will get some reuse or abstraction out of them, as otherwise they would really just introduce more things I need to touch when they change (which so far has been somewhat often, as I've been working towards this architecture in the first place). At the end of the day, less code is less code, and that's usually better than more code, even if that more code follows more rules :) . I mean, really "don't be an idiot" is always implied, and that works out pretty well.

Conclusion

All of these small bits together keep the project relatively small and maintainable, while still satisfying all of our initial constraints. This architecture so far has been pretty straightforward to implement and quite performant, sometimes at the cost of being a bit verbose (mainly because of the C bridge), but even the verbose code is straightforward and manageable enough.

Last Edited on Sat Apr 16 2016 20:11:13 GMT-0400 (EDT)