Reactive Relational Programming with PLRelational

August 10, 2017, by Chris Campbell

While working on the next major version of VoodooPad [1], an observation was made: user interfaces are basically just bidirectional transformation functions, taking data from some source and presenting it to the user, and vice versa. It sounds so simple when boiled down like that, but the reality of app development is a different story, often filled with twisty, error-prone UI code, even with the help of functional programming aids (e.g. Rx and its ilk). A question then emerged: can we build a library that allows us to express those transformations in a concise way, with an eye towards simplifying app development?

PLRelational is our (exploratory) answer to that question. At its core, PLRelational is a Swift framework that allows you to:

• declaratively express relationships using relational algebra
• asynchronously query and update those relations
• observe fine-grained deltas when a relation's data has changed

Perhaps some example code can help explain what that all means:

// Use combinators to `join` our two on-disk relations and `project`
// a subset of attributes
let allEmployees = employees
    .join(depts)
    .project([Employee.id, Employee.name, Dept.name])

// The following...
print(allEmployees)
// will print:
// "emp_id", "name",  "dept_name"
// 1,        "Alice", "HR"
// 2,        "Bob",   "Sales"
// 3,        "Cathy", "HR"

// Observe changes made to Cathy's record.  When the underlying relation
// changes, extract the changed name and append to `empThreeNames`.
var empThreeNames = [String]()
let empThree = employees.select(Employee.id *== 3)
empThree.addAsyncObserver(...)

// Update all employees in the HR department to have the name Linda
allEmployees.asyncUpdate(Dept.name *== "HR", [Employee.name: "Linda"])

// (Async updates and queries are processed in the background; results
// will be delivered on the main queue when ready...)
...

// Once the update is processed, our observer will see the change
print(empThreeNames) // will print: [Linda]

In the above example, we used some basic relational algebra operations (e.g. join and select) that are familiar if you've spent time with SQL. PLRelational includes a full set of combinators, including aggregate functions (e.g. max, count) and other operations like leftOuterJoin that are considered extensions to the relational algebra.

For example, we can find the number of employees in the HR department that like chocolate ice cream:

let chocoholicsInHR = employees.
    .select(Employee.dept *== "HR")
    .join(favoriteFlavors)
    .select(Flavor.name *== "chocolate")
    .count()

Or suppose we are building an application to manage the company's sports teams. We can use relational algebra to find out which teams a certain employee hasn't yet joined:

// Figure out which teams the employee *has* already joined
let selectedEmployeeTeams = employeeTeams
    .join(selectedEmployeeID)
    .join(teams)
    .project([Team.ID, Team.Name])

// Use `difference` to determine which teams are available to join
let availableTeams = teams
    .difference(selectedEmployeeTeams)

Note that in these examples, we're not actually forcing any data to load. We simply declare how the data is related, and then PLRelational takes care of efficiently loading and transforming the data as it is needed. For example, that hypothetical application might have a table view to display the available teams, and the data wouldn't need to be pulled until that table view is loaded for the first time.

One other great thing about PLRelational is that you can use the same Relation operations to interact with data from different sources. Out of the box, PLRelational has support for data stored in memory, in an SQLite database, or from plist data stored in a single file or spread out over multiple directories. There is also support for layering additional functionality, such as caching and change logging, through simple composition. In other words, the Relation API looks the same regardless of how or where the data is stored underneath.

Taken on its own, there are a lot of interesting things happening behind the scenes in PLRelational -- efficiently pulling and storing data, using relational algebra to compute derivatives, and so forth. But things get really interesting once we have a way to bind relations at the UI level, and that's where our PLRelationalBinding library comes in.

PLRelationalBinding is a separate Swift framework that builds on PLRelational and adds reactive concepts. Think of it as a layer for massaging raw, low-level deltas produced by PLRelational into something that's easier for a UI toolkit (e.g. AppKit or UIKit) to digest. And to assist with the latter, we have a third framework, PLBindableControls that is not so much an exhaustive UI toolkit but rather a handful of extensions to existing AppKit and UIKit controls that allow for binding to a Relation or Property.

With PLRelationalBinding, you can take an MVVM-like approach, using Properties to build an easily testable Model/ViewModel layer, and binding to those Properties from a separate View layer. While PLRelational mainly operates in terms of relational concepts like attributes and rows, PLRelationalBinding adds operators like map and zip that are familiar from the world of functional programming.

The various Property implementations in PLRelationalBinding take care of transforming data from those low-level rows into Swift-ly typed values. For example, if you have a single-string Relation, PLRelationalBinding lets you easily expose that as an AsyncReadableProperty<String> that can be bound to an NSTextField. Likewise, if you have a multi-row Relation, you can easily create an ArrayProperty or TreeProperty from it that can be bound to, say, an UITableView in one step.

Building on our example from above:

class FlavorsViewModel {
    private let selectedEmployee: Relation
    private let selectedEmployeeFlavors: Relation

    init(...) {
        ...
        self.selectedEmployee = employees.join(selectedEmployeeID)
        self.selectedEmployeeFlavors = favoriteFlavors.join(selectedEmployeeID)
    }

    lazy var employeeName: AsyncReadableProperty<String> = {
        return self.selectedEmployee
            .project(Employee.name)
            .oneString()
            .property()
    }()

    lazy var labelText: AsyncReadableProperty<String> = {
        return self.employeeName
            .map{ "\($0)'s Favorite Flavors" }
    }()

    lazy var favoriteFlavors: ArrayProperty<RowArrayElement> = {
        return self.selectedEmployeeFlavors
            .arrayProperty(idAttr: Flavor.id, orderAttr: Flavor.name)
    }()

    lazy var favoriteFlavorsModel: ListViewModel<RowArrayElement> = {
        ...
    }()
}

class FlavorsView: NSView {

    @IBOutlet var nameLabel: Label!
    @IBOutlet var flavorsOutlineView: NSOutlineView!
    private var flavorsListView: ListView<RowArrayElement>!

    init(model: ViewModel) {
        // Load the xib to connect the outlets
        ...

        // Bind our UI controls to the ViewModel layer
        nameLabel.text <~ model.labelText
        flavorsListView = ListView(model: model.favoriteFlavorsModel,
                                   outlineView: flavorsOutlineView)
    }    

    ...
}

With typical ORM-ish frameworks like CoreData and Realm, you shuttle objects back and forth and often have to worry about things like "if this view over here changes part of this object will that other view over there see those changes?" and so forth. PLRelational and friends take a less object-oriented approach, instead favoring a more dataflow-like strategy that maintains a traceable connection between a UI component that displays/edits some data, the source of that data (e.g. a table in an SQLite data store), and all the transformations in between. In other words, it's relations all the way down, which makes it possible for the query optimizer to determine what is changing and how to efficiently deliver those changes up to the UI layer.

In addition, thinking with a reactive-relational mindset instead of an object-oriented one leads to some rather nifty benefits for application developers. There are a number of traditionally tedious or tricky tasks that can be made elegant with PLRelational. Here are just a few examples:

Undo/Redo: When using a ChangeLoggingRelation, PLRelational automatically computes the minimal set of deltas that result each time a relation is mutated. Those deltas are bundled up into a transactional database snapshot, so implementing undo/redo at the app level becomes as simple as using a custom UndoManager that reverts to a particular snapshot. This works even in cases where a user action triggers many complex changes to multiple relations. From the perspective of the UI controls, the undo or redo action just looks like any other relation change (inserts, deletes, and updates), so no additional logic is required to handle undo/redo.

Table Updates: If you've ever written logic to update a table view in response to drag-and-drop reordering or changes to underlying data, you probably know how tricky it can be to get things just right. With the help of ArrayProperty (from PLRelationalBinding), insertions/deletions/updates made to a Relation are translated into changesets that can be applied directly to an NSTableView or UITableView; you no longer have to write that error-prone update logic yourself. Similarly, the TreeProperty class can do the same for structured, hierarchical data of the type that is typically displayed in an NSOutlineView.

Full Text Search: Full text search results (with highlighted snippets and all) can be treated as just another Relation and easily bound to, say, a UITableView. The RelationTextIndex class takes care of keeping an SQLite index updated when the associated searchable content is changed. It also maintains a Relation that contains search results and automatically refreshes its contents when the search query string is changed.

More depth on these and other perks is best left as fodder for future articles.

PLRelational is still in its nascent stages. There's a lot that works (well), but plenty that we'd still like to improve — one thing in particular would be stronger typing at the Relation level. That said, we're pleased with the way it is coming together, so much so that we've been happily using it as the foundation for the next version of VoodooPad on macOS and iOS.

This has been an intentionally brief introduction to PLRelational. If your interest has been piqued, we recommend checking out the sources at the official PLRelational repository. There are working example apps for macOS in the Xcode project that serve as a more complete companion to the simplified code examples in this article. All feedback is welcome!

Also of note: Plausible Labs offers consulting services for software engineering. If you'd like some professional assistance, whether with PLRelational or for something entirely different, consider us. More information can be found on our consulting page.

[1] Yes, we're still working on it, and no, it's not quite ready for public consumption — sorry!