PL

A New Home for VoodooPad

December 14, 2017, by Rebecca Bratburd

We are pleased to announce that the team at Primate Labs has taken over sales and development of VoodooPad.

Earlier this year, a significant member of our team resigned, which, along with other changes in direction for our cooperative, made it difficult to provide continued development and support of VoodooPad. Thus, we're passing the VoodooPad torch to a company better suited to support it now and in the future.

We trust Primate Labs, a company we've known for about a decade, to take over the reins. Primate Labs develops performance analysis software for desktop and mobile platforms, namely Geekbench. We think they'll make excellent custodians of VoodooPad alongside it.

Existing users of VoodooPad will be able to continue using the app as they have been. We will transfer the apps, but users won't need to take any action. Support requests will be handled by Primate Labs, and we'll be available to them during a transition period to help them get up to speed.

We’re grateful to the customers who wrote in with feature requests and bug reports that we were able to learn from and strengthen VoodooPad. We’re sad to let VoodooPad go, but at the same time we’re excited to see the product live on with Primate Labs as its new stewards. We wish them, and you, all the best!

PLRelational: Query Optimization and Execution

October 3, 2017, by Mike Ash

This is our latest entry in a series of articles on PLRelational. For more background, check out these other articles in the series:


We've been talking a lot about PLRelational lately and what you can do with it. Today I want to talk about some of its internals. In particular, I want to talk about how it optimizes and executes queries, which is one of the most interesting components of the framework.

Simple Query Execution

PLRelational's relations are divided into two kinds: source relations that contain data directly, and intermediate relations that produce data by operating on other relations. (The latter are implemented in the IntermediateRelation type.)

To get data out of a relation, PLRelational executes a query. For source relations, this is simple: ask it for its data, and that's it. Executing a query on intermediate relations gets more complicated.

Executing a query on intermediate relations is easy to implement if speed isn't a concern. Fetch the contents of the operands, perform the operation on those contents, and provide the result. For example, here's pseudocode for the intersection operation:

for row in operand[0] {
    if operand[1].contains(row) {
        provide(row)
    }
}

Difference is almost identical, just with the condition reversed:

for row in operand[0] {
    if !operand[1].contains(row) {
        provide(row)
    }
}

Union is slightly more complicated due to the need to avoid providing duplicate rows, but still straightforward:

for row in operand[0] {
    provide(row)
}
for row in operand[1] {
    if !operand[0].contains(row) {
        provide(row)
    }
}

Early versions of PLRelational implemented operations in this way. It works fine, but can be really slow.

To see why, let's build up a small example. We'll start off with a list of pets:

let pets = MakeRelation(
    ["id", "name",   "animal"],
    [1,    "Mickey", "cat"],
    [2,    "Fido",   "dog"],
    [3,    "Nudges", "cat"],
    [4,    "Rover",  "dog"])

Let's pull out the cats:

let cats = pets.select(Attribute("animal") *== "cat")

How is the select method implemented? A really simple implementation might look like this:

for row in operand[0] {
    if expression.valueWithRow(row).boolValue {
        provide(row)
    }
}

Of course, this requires iterating over all of the pets. This is fine when there's only four of them, but what if there are thousands?

We could improve this by adding some code to the underlying Relation type produced by MakeRelation. It could implement select to return a new Relation which knows how to efficiently perform the operation.

What if the situation is more complicated, though? For example, maybe pets is loaded from disk, and then we make in-memory modifications by adding and deleting rows. We can express that by creating Relations for the added and removed rows, then using a union and a difference to combine it all:

let added = MakeRelation(
    ["id", "name", "animal"],
    [5, "Pointy", "cat"])
let removed = MakeRelation(
    ["id", "name", "animal"],
    [6, "Nudges", "cat"])

let currentPets = pets.union(added).difference(removed)

Then we pull out these cats:

let currentCats = currentPets.select(Attribute("animal") *== "cat")

Unfortunately, we're back to iterating over all of the pets.

We should be able to do better. Filtering the output of a union produces the same results as filtering the inputs, then performing the union on the filtered results. The same is true for intersections and differences. If the system could take the filter and push it down to the storage relations, each one could efficiently perform the select and we'd avoid iterating over everything.

How do you make that happen? You could override select in unions and differences to do this, but then what if you derive multiple relations from a union? What about other operations? What about more complicated derived relations? What we really want is a separate system that can see the big picture and optimize it all.

Query Execution Structure

PLRelational implements such a system using three high-level components.

The query planner is responsible for translating a set of Relations into a more execution-oriented representation. Relation objects in memory are easy to work with in code, but a bit challenging to work with for query execution. The QueryPlanner class takes an array of Relations and output callbacks (which will be provided with the data from those Relations), and builds a Node for each Relation passed in and all of the Relations they depend on. Each Node contains things such as an operation describing what it does, an approximate count, and pointers to parent and child nodes.

Next, the query optimizer takes the nodes that the planner produced and performs optimizations on them.

After the optimizer is done, the nodes are sent to the query runner. It performs all the work of actually fetching data, computing results, and invoking the output callbacks. The runner is also able to perform optimizations which are better suited to being performed as the query runs rather than in advance.

Ahead-of-Time Optimizations

The distinction between optimizations performed in advance and optimizations performed while running the query mirrors the world of compilers, where you have ahead-of-time and just-in-time compilers. As I implemented this system, the best optimizations came from the query runner, and the query optimizer ended up with a pretty small role.

The current implementation only optimizes a couple of degenerate cases with unions. Unions with only one operand, which are used internally as placeholders in some situations, are eliminated. Nested union operations are compressed into a single operation, within limits. Finally, a garbage collection pass removes nodes that have been orphaned by these optimizations.

Just-in-Time Optimizations

The query runner is responsible for some key optimizations which can dramatically speed up. Most importantly, it tracks filters which can be applied to each node, and pushes those filters to the children when possible. When those filters make it all the way to a child which provides data directly, the filter can be used to efficiently fetch only the needed data.

First, it finds all of the select operations in the graph and applies the select expressions as filters to the children. This is done before running the query, and so strictly speaking could be done as part of the query optimizer instead. However, the query runner tracks the nodes' filter states, which makes it a more suitable place to implement this step.

The query runner tracks two important pieces of information about each node's filter state: the current filter expression, and the number of parent nodes which have not yet provided a filter. As long as there are parents which have not provided a filter, the node must provide its data unfiltered, because those parents may need it all. When a parent node provides a filter, the filter expression is ORed with the current filter expression, and the number of parents which haven't provided a filter is decremented. Once that number reaches zero, the filter expression can be used, and it's propagated to the node's children.

Once select operations are applied, the query runner begins to run the query and process data. At this point, another big optimization kicks in for join operations.

A join works a lot like a select, where the expression is dynamically derived from the contents of the operands. For example, let's take the currentPets data above, and track the IDs of the selected animals stored in another Relation:

let selectedPetIDs = MakeRelation(
    ["id"],
    [5])

If we need the selected pet's name and animal type, we could do this with a select(Attribute("id") *== 5), but we'd have to manually manage that. It's much easier to do a join:

let selectedPets = selectedPetIDs.join(currentPets)

The fact that selectedPetIDs contains a single row with id: 5 makes it act like a select on currentPets, returning only the rows where id is 5.

Equivalently, we could look at currentPets as providing a massive select expression based on all of its id values, which then applies to selectedPetIDs. This produces the exact same output, but is far more expensive to compute, which will be crucial!

When a join node receives all data for one of its operands, it will attempt to turn it into a filter which it can push to the other operand. It creates the filter by constructing a filter from each input row and ORing them all together. To avoid making a massive, inefficient filter that wouldn't actually speed anything up, it only does this if the number of rows is below a certain threshold, currently set at 100 rows.

Ordering Operations

The order in which data is fetched is crucial. Joins where one operand is small can be wonderfully optimized if the small side is computed first, but will be woefully inefficient if the large side is computed first. Figuring out which side is the small side ahead of time can be really tough, since each operand can be an arbitrarily complex operation. Instead, the query runner attempts to make this happen as it fetches and computes data.

The query runner operates by choosing a node which can produce data (called an "initiator node") and asking it for some rows. It then propagates those rows through the graph until as much processing as possible has been done. It then goes back to the initiator and asks for more data, and this repeats until the initiator has provided all of its data. The query runner then chooses the next initiator, and repeats everything again until all initiators have been drained.

Initiator nodes are able to estimate the number of rows they contain which match a certain filter. It's important for this operation to be fast, which is why it's not an exact number. If the estimate is incorrect, the query will still produce correct results, just potentially slower. If the initiator can't come up with an estimate at all, it can return nil, in which case the query runner considers it to be bigger than anything else.

When the query runner chooses a new initiator node to pull data from, it picks the one with the smallest estimated size at the time. This usually results in the small sides of joins being filled first, which then allows the query runner to filter the other join operand. Estimated sizes change as filters are pushed down, so this can result in a chain reaction as one small initiator is used, filtering another initiator making it small, which then filters even more.

Subtree Copying

A child node must receive filters from all parents before it can apply that filter to itself. If one parent doesn't have a filter, then it must be assumed that this parent needs all of the data. This is problematic when one parent provides a filter, and then that parent's own data provides an input to the other parent which would generate a filter. Subtree copying fixes this seeming chicken-and-egg problem.

This is a lot easier to understand with an example. Let's consider a Relation which contains metadata for a document hierarchy. Each document gets an ID and a name. It also gets a parent ID, which expresses the nesting relationship. Documents at the top level have a null parent ID:

let documentMetadata = MakeRelation(
    ["id", "name",               "parent_id"],
    [1,    "A Story",            .null],
    [2,    "Cake Recipe",        .null],
    [3,    "A Story: Chapter 1", 1],
    [4,    "A Story: Chapter 2", 1])

We'll track the notion of the currently selected document ID in another Relation:

let selectedDocumentID = MakeRelation(
    ["id"],
    [3])

We can derive a Relation containing the metadata of only the currently selected document by performing a join:

let selectedDocumentMetadata = selectedDocumentID.join(documentMetadata)

Now let's imagine that we want the metadata of the currently selected document's parent as well. We can obtain this by projecting selectedDocumentMetadata on the parent_id attribute (removing the ID and name data), renaming parent_id to id, and then joining the result to documentMetadata:

let selectedDocumentParentID = selectedDocumentMetadata
    .project("parent_id")
    .renameAttributes(["parent_id": "id"])
let selectedDocumentParentMetadata = selectedDocumentParentID
    .join(documentMetadata)

Let's take a more graphical look at this setup. PLRelational has code which can dump a Relation into a Graphviz dot file, which we can then convert to SVG for display on the web. Here's what selectedDocumentParentMetadata looks like:

relation_graph _10241aa00 IntermediateRelation op: equijoin([id: id]) debugName: selectedDocumentParentMetadata _10241a7d0 IntermediateRelation op: rename([parent_id: id]) debugName: selectedDocumentParentID _10241aa00->_10241a7d0 operands_0 _102415100 MemoryTableRelation scheme: [id, name, parent_id] debugName: documentMetadata _10241aa00->_102415100 operands_1 _10241a620 IntermediateRelation op: project([parent_id]) debugName: nil _10241a7d0->_10241a620 operands_0 _10241a570 IntermediateRelation op: equijoin([id: id]) debugName: selectedDocumentMetadata _10241a620->_10241a570 operands_0 _102419cb0 MemoryTableRelation scheme: [id] debugName: selectedDocumentID _10241a570->_102419cb0 operands_0 _10241a570->_102415100 operands_1

Note that in this graph, the selectedDocumentParentMetadata is at the top, and the arrows point from operations to their operands.

After going through the query planner, but not yet running the query, the query planner nodes look like this:

query_planner_graph node_0 Node 0 selectedDocumentParentMetadata Scheme: [name, id, parent_id] equijoin([id: id]) 1 output callbacks Active buffers: 2 Rows input: 0 node_1 Node 1 selectedDocumentParentID Scheme: [id] rename([parent_id: id]) Active buffers: 1 Rows input: 0 node_1->node_0 child 0 node_2 Node 2 documentMetadata ~1000.0 rows Scheme: [parent_id, id, name] selectableGenerator((Function)) Active buffers: 0 Rows input: 0 node_2->node_0 child 1 node_4 Node 4 selectedDocumentMetadata Scheme: [name, id, parent_id] equijoin([id: id]) Active buffers: 2 Rows input: 0 node_2->node_4 child 1 node_3 Node 3 Scheme: [parent_id] project([parent_id]) Active buffers: 1 Rows input: 0 node_4->node_3 child 0 node_3->node_1 child 0 node_5 Node 5 selectedDocumentID ~1.0 rows Scheme: [id] selectableGenerator((Function)) Active buffers: 0 Rows input: 0 node_5->node_4 child 0

This graph is inverted from the last one. selectedDocumentParentMetadata is now at the bottom, and the arrows point in the direction of data flow.

In this graph, we can see the problem visually. selectedDocumentID has the smallest number of rows, so the query runner will fetch its data first. That will go into the join, which will convert the input to a filter and push that filter to documentMetadata. However, because documentMetadata has two parents and only one filter has been applied, that's not enough to activate a filter, so it produces all of its rows. That means that potentially thousands of rows flow through this graph just to produce one output row, which is terribly inefficient.

Subtree copying fixes this problem. When certain criteria are met, a node and all of its parent nodes are copied when pushing a filter to a node. In this example, documentMetadata is duplicated, with one copy providing input to selectedDocumentMetadata and the other copy providing input to selectedDocumentParentMetadata. This is the state of things when the query runner completes, and it looks like this:

query_planner_graph node_0 Node 0 selectedDocumentParentMetadata Scheme: [name, id, parent_id] equijoin([id: id]) 1 output callbacks Active buffers: 0 Rows input: 2 node_1 Node 1 selectedDocumentParentID Scheme: [id] rename([parent_id: id]) Active buffers: 0 Rows input: 1 node_1->node_0 child 0 node_2 Node 2 documentMetadata ~1000.0 rows Scheme: [parent_id, id, name] selectableGenerator((Function)) Parental select: (id) = (1) - 0 remaining Active buffers: 0 Rows input: 0 node_2->node_0 child 1 node_3 Node 3 Scheme: [parent_id] project([parent_id]) Active buffers: 0 Rows input: 1 node_3->node_1 child 0 node_4 Node 4 selectedDocumentMetadata Scheme: [name, id, parent_id] equijoin([id: id]) Active buffers: 0 Rows input: 2 node_4->node_3 child 0 node_5 Node 5 selectedDocumentID ~1.0 rows Scheme:  [id] selectableGenerator((Function)) Active buffers: 0 Rows input: 0 node_5->node_4 child 0 node_6 Node 6 documentMetadata ~1000.0 rows Scheme: [parent_id, id, name] selectableGenerator((Function)) Parental select: (id) = (3) - 0 remaining Active buffers: 0 Rows input: 0 node_6->node_4 child 1

The documentMetadata node has been duplicated. One of them now points into selectedDocumentMetadata, and the other one points into selectedDocumentParentMetadata. When selectedDocumentMetadata receives input from selectedDocumentID, it creates a filter and pushes it to its copy of documentMetadata.

Since that node only has one parent, the filter is immediately activated, which lets it efficiently produce a single row. The query runner pulls data from that node next, since it's now the smallest. That then allows selectedDocumentMetadata to compute the output of the join. That data then flows up the chain until it reaches selectedDocumentParentMetadata. That join is then able to compute a filter and push it to the other copy of documentMetadata. This allows both fetches from documentMetadata to be performed efficiently, and avoids the chicken-and-egg problem posed by the original structure.

I mentioned that certain criteria must be met for the query runner to copy a subtree. Those criteria are:

  1. The total number of nodes in the subtree must be no more than a certain limit, currently set at 100 nodes.
  2. The node at the top of the subtree must have at least one other parent that hasn't yet provided a filter.
  3. There must be at least one initiator node in the subtree which can efficiently produce less data when given a filter.

#1 is an arbitrary limit put in place to avoid doing the optimization if it's going to require a lot of work. Really complicated subtrees may cost more to duplicate than the time saved by the operation. It's hard to figure out exactly where the tradeoff is no longer worth it, but 100 is a workable limit for now.

#2 avoids pointless work. If the node isn't waiting on any other parents, then it can propagate the filter directly and copying won't help anything.

#3 likewise avoids pointless work. If none of the initiators in the subtree can efficiently produce filtered data, then there's no point in trying to ensure that they can be filtered. We'll pay the cost one way or another, and we might as well skip the cost of copying the subtree on top of that.

Subtree copying can make a big difference. Testing with 1,000 rows in memory, the query runs in about 30 milliseconds on my computer with subtree copying disabled, and about 7 milliseconds with it enabled. On 100,000 rows, it takes 2.4 seconds to run without subtree copying, and only 18 milliseconds to run with it.

Conclusion

That's the basic tour of PLRelational's query infrastructure. One of the interesting aspects of building the framework around relational algebra is the ability to optimize complicated data flow graphs constructed by the programmer on the fly based on the actual data being used at the time. Our optimizer is still fairly basic and there's a lot of room for improvement and new optimizations, but what we currently have is already enough to provide many orders of magnitude improvement on real-world data.


Learn Swift from the experts at Plausible Labs! We offer on-site corporate workshops with hands-on instruction in the deep and mysterious ways of Swift. Check out our training offerings for more information, or get in touch!

Let's Build with PLRelational, Part 2

September 28, 2017, by Chris Campbell

This article picks up where Part 1 left off, continuing our look at building a real application using PLRelational.

For more background on PLRelational, check out these other articles in the series:

In Part 1 of this article, we used PLRelational to build a portion of a to-do application for macOS. In this article, we will follow the same process and focus on building out the right-hand side of the application, specifically the detail view.

The detail view allows the user to change the status or text of the selected to-do item, apply tags, add notes, or delete the item. Click the "Play" button below to see the detail view in action:

As in Part 1, we've broken the right half of the application down into distinct functional requirements that describe how each piece of the UI should behave.

The following interactive screenshot shows the right half of the app in its completed state. Each red dot corresponds to a single functional requirement. Hover over a dot to read a summary of the requirement, then click to see how we implemented it using PLRelational.

6 7 8 9 10 11 12 13

Let's explore how we implemented this half of the application. (To follow along with the complete source code in Xcode, clone the PLRelational repository and open Example Apps > TodoApp in the project.)

REQ-6: Selected Item Detail

A detail view on the right side that allows the user to change information about the to-do item that is currently selected in the list. If there is no selection, the detail view should be hidden and replaced by a gray "No Selection" label.

There are two parts to this requirement.

First we need to bind the selection state of the list view to our selectedItemIDs relation. Every time the user selects an item in the list, the identifier of that item will be poked into the selectedItemIDs relation. In later steps, we will join that relation with others to derive data related to the selected item, such as its title.

Once we have the selection binding in place, we can expose a hasSelection property that resolves to true whenever there is an item selected in the list view. This makes it easy to set up bindings at the view level to toggle between showing the "No Selection" label and the detail view:

class Model { ...
    /// Resolves to the item that is selected in the list of to-do items.
    lazy var selectedItems: Relation = {
        return self.selectedItemIDs.join(self.items)
    }()

    /// Resolves to `true` when an item is selected in the list of to-do items.
    lazy var hasSelection: AsyncReadableProperty<Bool> = {
        return self.selectedItems.nonEmpty.property()
    }()
}

class ChecklistViewModel { ...
    /// Holds the ID of the to-do item that is selected in the list view.  This
    /// is a read/write property that is backed by UndoableDatabase, meaning that
    /// even selection changes can be undone (which admittedly is taking things
    /// to an extreme but we'll keep it like this for demonstration purposes).
    lazy var itemsListSelection: AsyncReadWriteProperty<Set<RelationValue>> = {
        return self.model.selectedItemIDs
            .undoableAllRelationValues(self.model.undoableDB, "Change Selection")
    }()
}

class ChecklistView { ...
    init() { ...
        // Bidirectionally bind list view selection to the view model
        listView.selection <~> model.itemsListSelection
    }
}

class AppDelegate { ...
    func applicationDidFinishLaunching() { ...
        // Toggle the "No Selection" label and detail view depending
        // on the selection state
        detailView.visible <~ model.hasSelection
        noSelectionLabel.visible <~ not(model.hasSelection)
    }
}

Note that we define itemsListSelection as an undoable property, just to show that even things like selection state can be modeled using relations and can participate in the undo system. (Lightroom is an example of an application where selection changes are part of the history stack.)

REQ-7: Selected Item Completed Checkbox

A checkbox at the top-left of the detail view. This should reflect whether the selected to-do item is pending (unchecked) or completed (checked). The behavior of this checkbox is the same as described in REQ-3, except that it controls the selected item's position in the list.

Hooking up the checkbox in the detail view is almost identical to the way we did it for list cells in REQ-3. In fact, we use the same itemCompleted function from Model, except this time we pass it a Relation that focuses on the status value of the selected item:

class DetailViewModel { ...
    /// The completed status of the item.
    lazy var itemCompleted: AsyncReadWriteProperty<CheckState> = {
        let relation = self.model.selectedItems.project(Item.status)
        return self.model.itemCompleted(relation, initialValue: nil)
    }()
}

class DetailView { ...
    init() { ...
        // Bidirectionally bind checkbox state to the view model
        checkbox.checkState <~> model.itemCompleted
    }
}

REQ-8: Selected Item Title Field

A text field to the right of the checkbox in the detail view. This should reflect the title of the selected to-do item. If the user changes the title in the detail view and presses enter, the title should also be updated in the selected list cell.

Just like with the last step, wiring up the text field in the detail view follows the same approach used in REQ-4 (list cell text field):

class DetailViewModel { ...
    /// The item's title.  This is a read/write property that is backed
    /// by UndoableDatabase, so any changes made to it in the text field
    /// can be rolled back by the user.
    lazy var itemTitle: AsyncReadWriteProperty<String> = {
        let titleRelation = self.model.selectedItems.project(Item.title)
        return self.model.itemTitle(titleRelation, initialValue: nil)
    }()
}

class DetailView { ...
    init() { ...
        // Bidirectionally bind title field to the view model
        titleField.string <~> model.itemTitle
    }
}

REQ-9: Assign Tag Combo Box

A combo box with placeholder "Assign a tag". The pull-down menu should include a list of available tags (all tags that haven't yet been applied; if a tag has been applied it should not appear in the menu). The user can also type in an existing tag or new tag. If the user clicks or enters a tag, that tag should be added to the set of applied tags for the selected to-do item.

Combo boxes are complex controls (they are part menu and part text field), so let's break this up into a few parts. First we'll show how we populate the menu of available tags:

class Model { ...
    /// Resolves to the set of tags that are associated with the selected to-do
    /// item (assumes there is either zero or one selected items).
    lazy var tagsForSelectedItem: Relation = {
        return self.selectedItemIDs
            .join(self.itemTags)
            .join(self.tags)
            .project([Tag.id, Tag.name])
    }()

    /// Resolves to the set of tags that are not yet associated with the
    /// selected to-do item, i.e., the available tags.
    lazy var availableTagsForSelectedItem: Relation = {
        // This is simply "all tags" minus "already applied tags", nice!
        return self.tags
            .difference(self.tagsForSelectedItem)
    }()
}

class DetailViewModel { ...
    /// The tags that are available (i.e., not already applied) for the selected
    /// to-do item, sorted by name.  We use `fullArray` so that the entire array
    /// is delivered any time there is a change; this helps to make it
    /// compatible with the `EphemeralComboBox` class.
    lazy var availableTags: AsyncReadableProperty<[RowArrayElement]> = {
        return self.model.availableTagsForSelectedItem
            .arrayProperty(idAttr: Tag.id, orderAttr: Tag.name)
            .fullArray()
    }()
}

class DetailView { ...
    init() { ...
        // Bind combo box menu items to the view model
        tagComboBox.items <~ model.availableTags
    }
}

Next we'll show how to handle when an existing tag is selected in the menu:

class Model { ...
    /// Applies an existing tag to the given to-do item.
    func addExistingTag(_ tagID: TagID, to itemID: ItemID) {
        undoableDB.performUndoableAction("Add Tag", {
            self.itemTags.asyncAdd([
                ItemTag.itemID: itemID,
                ItemTag.tagID: tagID
            ])
        })
    }
}

class DetailViewModel { ...
    /// Adds an existing tag to the selected to-do item.
    lazy var addExistingTagToSelectedItem: ActionProperty<RelationValue> = ActionProperty { tagID in
        self.model.addExistingTag(TagID(tagID), to: self.itemID.value!!)
    }
}

class DetailView { ...
    init() { ...
        // When a tag is selected, add that tag to the selected to-do item
        tagComboBox.selectedItemID ~~> model.addExistingTagToSelectedItem
    }
}

Finally we'll show how to handle when a new tag is entered in the combo field:

class Model { ...
    /// Creates a new tag and applies it to the given to-do item.
    func addNewTag(named name: String, to itemID: ItemID) {
        let tagID = TagID()

        undoableDB.performUndoableAction("Add New Tag", {
            self.tags.asyncAdd([
                Tag.id: tagID,
                Tag.name: name
            ])

            self.itemTags.asyncAdd([
                ItemTag.itemID: itemID,
                ItemTag.tagID: tagID
            ])
        })
    }
}

class DetailViewModel { ...
    /// Creates a new tag of the given name and adds it to the selected to-do item.
    lazy var addNewTagToSelectedItem: ActionProperty<String> = ActionProperty { name in
        // See if a tag already exists with the given name
        let itemID = self.itemID.value!!
        let existingIndex = self.model.allTags.value?.index(where: {
            let rowName: String = $0.data[Tag.name].get()!
            return name == rowName
        })
        if let index = existingIndex {
            // A tag already exists with the given name, so apply that tag
            // rather than creating a new one
            let elem = self.model.allTags.value![index]
            let tagID = TagID(elem.data)
            self.model.addExistingTag(tagID, to: itemID)
        } else {
            // No tag exists with that name, so create a new tag and apply
            // it to this item
            self.model.addNewTag(named: name, to: itemID)
        }
    }
}

class DetailView { ...
    init() { ...
        // Add a new tag each time a string is entered into the combo field
        tagComboBox.committedString ~~> model.addNewTagToSelectedItem
    }
}

REQ-10: Selected Item Tags List

A list of tags that have been applied to the selected to-do item.

Setting up this list view follows roughly the same process that we used for the list of to-do items in REQ-2 through REQ-5:

  1. Define the ListViewModel. In this case, the list will contain the array of tags for the selected to-do item, sorted by name.
  2. Define a property to hold the selection state of the list.
  3. Define properties for each element of the list cell. In this case, there is a single text field for the tag name, and we want to allow the user to edit the name so we set it up as a bidirectional binding.
  4. Bind the views to those properties we defined in the View Model layer.

Here's what that process looks like in code form:

class Model { ...
    /// Returns a property that reflects the tag name.
    func tagName(for tagID: TagID, initialValue: String?) -> AsyncReadWriteProperty<String> {
        return self.tags
            .select(Tag.id *== tagID)
            .project(Tag.name)
            .undoableOneString(undoableDB, "Change Tag Name", initialValue: initialValue)
    }
}

class DetailViewModel { ...
    /// The tags associated with the selected to-do item, sorted by name.
    private lazy var itemTags: ArrayProperty<RowArrayElement> = {
        return self.model.tagsForSelectedItem
            .arrayProperty(idAttr: Tag.id, orderAttr: Tag.name)
    }()

    /// The model for the tags list, i.e., the tags that have been applied to the
    /// selected to-do item.
    lazy var tagsListViewModel: ListViewModel<RowArrayElement> = {
        return ListViewModel(
            data: self.itemTags,
            cellIdentifier: { _ in "Cell" }
        )
    }()

    /// Returns a read/write property that resolves to the name for the given tag.
    func tagName(for row: Row) -> AsyncReadWriteProperty<String> {
        let tagID = TagID(row)
        let name: String? = row[Tag.name].get()
        return self.model.tagName(for: tagID, initialValue: name)
    }

    /// Holds the ID of the tag that is selected in the tags list view.
    lazy var selectedTagID: ReadWriteProperty<Set<RelationValue>> = {
        return mutableValueProperty(Set())
    }()
}

class DetailView { ...
    init() { ...
        // Bind outline view to the tags list view model
        tagsListView = ListView(model: model.tagsListViewModel,
                                outlineView: tagsOutlineView)

        // Bidirectionally bind list view selection to the view model
        tagsListView.selection <~> model.selectedTagID

        tagsListView.configureCell = { view, row in
            // Bidirectionally bind tag name field to the view model
            let textField = view.textField as! TextField
            textField.string.unbindAll()
            textField.string <~> model.tagName(for: row)
        }
    }
}

REQ-11: Selected Item Notes

A text view that allows the user to type in notes about the selected to-do item.

This step is straightforward. There is a single read/write property that reflects the notes value for the selected to-do item, and we bidirectionally bind it to the TextView:

class Model { ...
    /// Returns a property that reflects the selected item's notes.
    lazy var selectedItemNotes: AsyncReadWriteProperty<String> = {
        return self.selectedItems
            .project(Item.notes)
            .undoableOneString(self.undoableDB, "Change Notes")
    }()
}

class DetailViewModel { ...
    /// The item's notes.
    lazy var itemNotes: AsyncReadWriteProperty<String> = {
        return self.model.selectedItemNotes
    }()
}

class DetailView { ...
    init() { ...
        // Bidirectionally bind content of notes text view to the view model
        notesTextView.text <~> model.itemNotes
    }
}

The TextView class will take care of delivering changes via its text property whenever the user is done editing (as a result of a change in first responder, for example).

REQ-12: Selected Item "Created On" Label

A read-only label that shows when the selected to-do item was created, e.g. "Created on Sep 1, 2017".

This step is also relatively simple. It's a read-only label, so we only have to worry about transforming the raw created timestamp string, as stored in the items relation, to a display-friendly string:

class DetailViewModel { ...
    /// The text that appears in the "Created on <date>" label.  This
    /// demonstrates the use of `map` to convert the raw timestamp string
    /// (as stored in the relation) to a display-friendly string.
    lazy var createdOn: AsyncReadableProperty<String> = {
        return self.model.selectedItems
            .project(Item.created)
            .oneString()
            .map{ "Created on \(displayString(from: $0))" }
            .property()
    }()
}

class DetailView { ...
    init() { ...
        // Bind "Created on <date>" label to the view model
        createdOnLabel.string <~ model.createdOn
    }
}

Let's dissect that createdOn declaration:

// Start with the `selectedItems` relation
self.model.selectedItems

// Take (`project`) the `created` attribute of the `selectedItems` relation
.project(Item.created)

// Derive a `Signal` that carries the raw timestamp string when there is a
// selected to-do item (or an empty string when there is no selection)
.oneString()

// Convert the raw timestamp string into a display-friendly one
.map{ "Created on \(displayString(from: $0))" }

// Lift the `Signal` into an `AsyncReadableProperty` that offers the latest value
.property()

This is another example of the Relation -> Signal -> Property pattern that we discussed earlier in REQ-5. You will encounter this pattern frequently when working with PLRelationalBinding, so it's a good idea to familiarize yourself with it.

REQ-13: Delete Selected Item Button

A delete button. If the user clicks this button, the selected item should be deleted (removed from the list entirely) and the list selection should be cleared (no item selected).

We're almost done! Deletion is an interesting case because we need to be able to delete not just the row in the items relation but also the associated rows in itemTags and selectedItemIDs. If we had the ID of the selected item handy we could simply issue three asyncDelete calls and be done with it, but where's the fun in that? Nowhere, that's where. So instead, let's demonstrate the fancy cascadingDelete function:

class Model { ...
    /// Deletes the row associated with the selected item and
    /// clears the selection.  This demonstrates the use of
    /// `cascadingDelete`, which is kind of overkill for this
    /// particular case but does show how easy it can be to
    /// clean up related data with a single call.
    func deleteSelectedItem() {
        undoableDB.performUndoableAction("Delete Item", {
            // We initiate the cascading delete by first removing
            // all rows from `selectedItemIDs`
            self.selectedItemIDs.cascadingDelete(
                true, // `true` here means "all rows"
                affectedRelations: [
                    self.items, self.selectedItemIDs, self.itemTags
                ],
                cascade: { (relation, row) in
                    if relation === self.selectedItemIDs {
                        // This row was deleted from `selectedItemIDs`;
                        // delete corresponding rows from `items`
                        // and `itemTags`
                        let itemID = ItemID(row)
                        return [
                            (self.items, Item.id *== itemID),
                            (self.itemTags, ItemTag.itemID *== itemID)
                        ]
                    } else {
                        // Nothing else to clean up
                        return []
                    }
                }
            )
        })
    }
}

class DetailViewModel { ...
    /// Deletes the selected item.  This demonstrates the use of
    /// `ActionProperty` to expose an imperative (side effect producing)
    /// action as a property that can easily be bound to a `Button`.
    lazy var deleteItem: ActionProperty<()> = ActionProperty { _ in
        self.model.deleteSelectedItem()
    }
}

class DetailView { ...
    init() { ...
        // When the button is clicked, delete the selected item
        deleteButton.clicks ~~> model.deleteItem
    }
}

This was a pretty straightforward use of cascadingDelete, but it offers other features (not used in this particular example) that really come in handy when deleting lots of interconnected data and repairing connections affected by the deletion. Additionally, there is a companion to cascadingDelete called treeDelete that simplifies deletion of hierarchical data. For more on these, check out MutableRelationCascadingDelete.swift in the repository along with some example uses in RelationTests.swift.

Quiet Victories

While most of this article was focused on showing how succinctly and easily we can build a user interface using PLRelational, it's worth noting the things you didn't see. These are just a few of the things you no longer have to worry about, because PLRelational does the hard work for you:

  • Objects, object graphs, observers: In a traditional application, we would have defined classes or structs for things like Item and Tag, and we would have had to maintain some sort of manager/controller to handle loading them from disk (and saving them back again), maintaining the relationships in a complex graph, updating those objects and sending out notifications to observers when something has changed, and so forth. With PLRelational, all of this gets handled through Relations and their reactive extensions; in our to-do application, we never explicitly loaded data into objects, and data flows from disk up to the user interface and back again through those Relations.

  • Undo/Redo: In a traditional application, we would have had to register with the UndoManager separate imperative methods for each undoable action: one to apply the change, and one to roll it back (along with the required support in the user interface layer to correctly react to those changes). With PLRelational, deltas can be automatically computed with each change, so implementing undo/redo support in your application is as easy as providing a "forward" transaction. When the user performs an undo or redo operation, the framework simply reverts the relations to a particular snapshot, and the user interface layer automatically updates by virtue of the reactive bindings.

  • Table updates: In a traditional Cocoa application, we would have had to implement the NSTableViewDataSource and NSTableViewDelegate protocols, writing fragile logic to coordinate updates to the table view in response to some changes in the object graph. With PLRelational, the ArrayProperty and ListView classes work in concert, tracking changes made in the relations and delivering them in a nice bundle to the underlying NSTableView with almost no custom logic needed on your part.

Wrapping Up

Over the course of this two-part article, we demonstrated how to use PLRelational to model source relations and then use a reactive-relational style of programming to build a working macOS application.

The vocabulary offered by PLRelational makes it easy to declare — and reason about — the relationships between data in your application and how it is presented in the user interface. Although we focused on building a macOS application in this article, the same techniques can be used to simplify development of iOS applications as well. (And perhaps we will demonstrate just that in a future article!)

The complete source code for this application is available in the PLRelational repository under the Examples/TodoApp directory and can be run directly from the Xcode project.

Finally, if you've been experimenting with PLRelational and have thoughts to share, or have a topic related to these frameworks that you'd like us to cover, please tweet at us or mail it in!


Learn Swift from the experts at Plausible Labs! We offer on-site corporate workshops with hands-on instruction in the deep and mysterious ways of Swift. Check out our training offerings for more information, or get in touch!