Reactive Relational Programming with PLRelational

August 10, 2017, by Chris Campbell

While working on the next major version of VoodooPad [1], an observation was made: user interfaces are basically just bidirectional transformation functions, taking data from some source and presenting it to the user, and vice versa. It sounds so simple when boiled down like that, but the reality of app development is a different story, often filled with twisty, error-prone UI code, even with the help of functional programming aids (e.g. Rx and its ilk). A question then emerged: can we build a library that allows us to express those transformations in a concise way, with an eye towards simplifying app development?

PLRelational is our (exploratory) answer to that question. At its core, PLRelational is a Swift framework that allows you to:

• declaratively express relationships using relational algebra
• asynchronously query and update those relations
• observe fine-grained deltas when a relation's data has changed

Perhaps some example code can help explain what that all means:

// Use combinators to `join` our two on-disk relations and `project`
// a subset of attributes
let allEmployees = employees
    .join(depts)
    .project([Employee.id, Employee.name, Dept.name])

// The following...
print(allEmployees)
// will print:
// "emp_id", "name",  "dept_name"
// 1,        "Alice", "HR"
// 2,        "Bob",   "Sales"
// 3,        "Cathy", "HR"

// Observe changes made to Cathy's record.  When the underlying relation
// changes, extract the changed name and append to `empThreeNames`.
var empThreeNames = [String]()
let empThree = employees.select(Employee.id *== 3)
empThree.addAsyncObserver(...)

// Update all employees in the HR department to have the name Linda
allEmployees.asyncUpdate(Dept.name *== "HR", [Employee.name: "Linda"])

// (Async updates and queries are processed in the background; results
// will be delivered on the main queue when ready...)
...

// Once the update is processed, our observer will see the change
print(empThreeNames) // will print: [Linda]

In the above example, we used some basic relational algebra operations (e.g. join and select) that are familiar if you've spent time with SQL. PLRelational includes a full set of combinators, including aggregate functions (e.g. max, count) and other operations like leftOuterJoin that are considered extensions to the relational algebra.

For example, we can find the number of employees in the HR department that like chocolate ice cream:

let chocoholicsInHR = employees.
    .select(Employee.dept *== "HR")
    .join(favoriteFlavors)
    .select(Flavor.name *== "chocolate")
    .count()

Or suppose we are building an application to manage the company's sports teams. We can use relational algebra to find out which teams a certain employee hasn't yet joined:

// Figure out which teams the employee *has* already joined
let selectedEmployeeTeams = employeeTeams
    .join(selectedEmployeeID)
    .join(teams)
    .project([Team.ID, Team.Name])

// Use `difference` to determine which teams are available to join
let availableTeams = teams
    .difference(selectedEmployeeTeams)

Note that in these examples, we're not actually forcing any data to load. We simply declare how the data is related, and then PLRelational takes care of efficiently loading and transforming the data as it is needed. For example, that hypothetical application might have a table view to display the available teams, and the data wouldn't need to be pulled until that table view is loaded for the first time.

One other great thing about PLRelational is that you can use the same Relation operations to interact with data from different sources. Out of the box, PLRelational has support for data stored in memory, in an SQLite database, or from plist data stored in a single file or spread out over multiple directories. There is also support for layering additional functionality, such as caching and change logging, through simple composition. In other words, the Relation API looks the same regardless of how or where the data is stored underneath.

Taken on its own, there are a lot of interesting things happening behind the scenes in PLRelational -- efficiently pulling and storing data, using relational algebra to compute derivatives, and so forth. But things get really interesting once we have a way to bind relations at the UI level, and that's where our PLRelationalBinding library comes in.

PLRelationalBinding is a separate Swift framework that builds on PLRelational and adds reactive concepts. Think of it as a layer for massaging raw, low-level deltas produced by PLRelational into something that's easier for a UI toolkit (e.g. AppKit or UIKit) to digest. And to assist with the latter, we have a third framework, PLBindableControls that is not so much an exhaustive UI toolkit but rather a handful of extensions to existing AppKit and UIKit controls that allow for binding to a Relation or Property.

With PLRelationalBinding, you can take an MVVM-like approach, using Properties to build an easily testable Model/ViewModel layer, and binding to those Properties from a separate View layer. While PLRelational mainly operates in terms of relational concepts like attributes and rows, PLRelationalBinding adds operators like map and zip that are familiar from the world of functional programming.

The various Property implementations in PLRelationalBinding take care of transforming data from those low-level rows into Swift-ly typed values. For example, if you have a single-string Relation, PLRelationalBinding lets you easily expose that as an AsyncReadableProperty<String> that can be bound to an NSTextField. Likewise, if you have a multi-row Relation, you can easily create an ArrayProperty or TreeProperty from it that can be bound to, say, an UITableView in one step.

Building on our example from above:

class FlavorsViewModel {
    private let selectedEmployee: Relation
    private let selectedEmployeeFlavors: Relation

    init(...) {
        ...
        self.selectedEmployee = employees.join(selectedEmployeeID)
        self.selectedEmployeeFlavors = favoriteFlavors.join(selectedEmployeeID)
    }

    lazy var employeeName: AsyncReadableProperty<String> = {
        return self.selectedEmployee
            .project(Employee.name)
            .oneString()
            .property()
    }()

    lazy var labelText: AsyncReadableProperty<String> = {
        return self.employeeName
            .map{ "\($0)'s Favorite Flavors" }
    }()

    lazy var favoriteFlavors: ArrayProperty<RowArrayElement> = {
        return self.selectedEmployeeFlavors
            .arrayProperty(idAttr: Flavor.id, orderAttr: Flavor.name)
    }()

    lazy var favoriteFlavorsModel: ListViewModel<RowArrayElement> = {
        ...
    }()
}

class FlavorsView: NSView {

    @IBOutlet var nameLabel: Label!
    @IBOutlet var flavorsOutlineView: NSOutlineView!
    private var flavorsListView: ListView<RowArrayElement>!

    init(model: ViewModel) {
        // Load the xib to connect the outlets
        ...

        // Bind our UI controls to the ViewModel layer
        nameLabel.text <~ model.labelText
        flavorsListView = ListView(model: model.favoriteFlavorsModel,
                                   outlineView: flavorsOutlineView)
    }    

    ...
}

With typical ORM-ish frameworks like CoreData and Realm, you shuttle objects back and forth and often have to worry about things like "if this view over here changes part of this object will that other view over there see those changes?" and so forth. PLRelational and friends take a less object-oriented approach, instead favoring a more dataflow-like strategy that maintains a traceable connection between a UI component that displays/edits some data, the source of that data (e.g. a table in an SQLite data store), and all the transformations in between. In other words, it's relations all the way down, which makes it possible for the query optimizer to determine what is changing and how to efficiently deliver those changes up to the UI layer.

In addition, thinking with a reactive-relational mindset instead of an object-oriented one leads to some rather nifty benefits for application developers. There are a number of traditionally tedious or tricky tasks that can be made elegant with PLRelational. Here are just a few examples:

Undo/Redo: When using a ChangeLoggingRelation, PLRelational automatically computes the minimal set of deltas that result each time a relation is mutated. Those deltas are bundled up into a transactional database snapshot, so implementing undo/redo at the app level becomes as simple as using a custom UndoManager that reverts to a particular snapshot. This works even in cases where a user action triggers many complex changes to multiple relations. From the perspective of the UI controls, the undo or redo action just looks like any other relation change (inserts, deletes, and updates), so no additional logic is required to handle undo/redo.

Table Updates: If you've ever written logic to update a table view in response to drag-and-drop reordering or changes to underlying data, you probably know how tricky it can be to get things just right. With the help of ArrayProperty (from PLRelationalBinding), insertions/deletions/updates made to a Relation are translated into changesets that can be applied directly to an NSTableView or UITableView; you no longer have to write that error-prone update logic yourself. Similarly, the TreeProperty class can do the same for structured, hierarchical data of the type that is typically displayed in an NSOutlineView.

Full Text Search: Full text search results (with highlighted snippets and all) can be treated as just another Relation and easily bound to, say, a UITableView. The RelationTextIndex class takes care of keeping an SQLite index updated when the associated searchable content is changed. It also maintains a Relation that contains search results and automatically refreshes its contents when the search query string is changed.

More depth on these and other perks is best left as fodder for future articles.

PLRelational is still in its nascent stages. There's a lot that works (well), but plenty that we'd still like to improve — one thing in particular would be stronger typing at the Relation level. That said, we're pleased with the way it is coming together, so much so that we've been happily using it as the foundation for the next version of VoodooPad on macOS and iOS.

This has been an intentionally brief introduction to PLRelational. If your interest has been piqued, we recommend checking out the sources at the official PLRelational repository. There are working example apps for macOS in the Xcode project that serve as a more complete companion to the simplified code examples in this article. All feedback is welcome!

Also of note: Plausible Labs offers consulting services for software engineering. If you'd like some professional assistance, whether with PLRelational or for something entirely different, consider us. More information can be found on our consulting page.

[1] Yes, we're still working on it, and no, it's not quite ready for public consumption — sorry!

VoodooPad Encryption Vulnerability

July 6, 2015, by Landon Fuller

After Plausible Labs’ acquisition of VoodooPad, a cryptography audit was performed and VoodooPad’s document encryption implementation was found to use weak or improperly employed cryptographic primitives. The discovered issues include weak key derivation, use of known-weak ciphers, predictable RNG seeding, and improper IV re-use.

As a result, an attacker with access to an encrypted document may be able to decrypt the document’s contents without access to the original encryption password.

To resolve these issues, we’ve invested heavily in a complete redesign and rewrite of VoodooPad’s encryption implementation; as of VoodooPad 5.1.4:

• All VoodooPad releases now ship with an encryption implementation based on industry standards and best practices.
• VoodooPad will display a warning upon opening an insecurely encrypted document, and will optionally perform an in-place upgrade of the document’s encryption.
• We have published complete technical specifications documenting VoodooPad’s new encryption implementation; refer to “Additional Resources” below.

Due to the sweeping implementation changes that were required in VoodooPad’s document storage:

• Encrypted documents produced in VoodooPad 5.1.4 and later will not be readable in earlier releases of VoodooPad.
• We are initially releasing VoodooPad 5.1.4 as a public beta to allow for further testing while allowing affected customers to upgrade immediately. We do recommend that affected customers upgrade now.

Common Questions

I’m a VoodooPad user but I don’t use encryption. I didn’t even know VoodooPad had encryption! Do I have anything to worry about?

Nope! This issue only applies to documents using encryption.

I have some encrypted VoodooPad documents. How could this impact me?

Unfortunately, anyone that is able to gain access to your VoodooPad-encrypted documents can potentially decrypt any document that was encrypted with a previous VoodooPad release, even if they do not know the document password.

When using a cloud file or backup service, encrypted VoodooPad documents could be decrypted by anyone with access to your account (including the cloud service provider).

For documents stored locally on your computer, an attacker would require access to your computer, or access to your local files via another means — such as unencrypted backups.

OK — So how do I secure my documents?

The first step is to upgrade to the VoodooPad 5.1.4 Beta release. If you’re a Mac App Store customer, you’ll need to download the beta from the provided link instead of the App Store, but you won’t need to purchase a separate license. If you are an iOS user, reading securely encrypted documents will also require VoodooPad 5.1.4 for iOS, available as a free upgrade via the App Store.

Next, open your encrypted documents (or documents containing encrypted pages) with VoodooPad 5.1.4 on Mac OS X — VoodooPad will offer to upgrade the documents immediately in-place. An unmodified copy of your document will also be placed in the Trash — you may wish to empty the trash after you’ve verified that your document has been upgraded successfully.

Lastly, be aware that cloud services like Dropbox may store backup copies of files, and those backups may include an insecurely encrypted version of your document. You can request that Dropbox permanently delete a file, but just to be safe, we recommend saving a local backup of the file on your own computer first.

What if I’m using VoodooPad 4 or earlier?

Previous releases of VoodooPad used a different design for document and page encryption; unfortunately, this was also found to use weak or improperly employed cryptographic primitives. We recommend that all customers upgrade to VoodooPad 5.

Discounted upgrade pricing is available to direct-purchase customers via the Plausible Store. For Mac App Store customers, Apple does not support discount upgrade pricing via the Mac App Store – if you previously purchased VoodooPad 4 through the Mac App Store, please contact us directly to arrange for upgrade pricing.

Additional Resources

We’ve published additional technical details on the design and implementation of VoodooPad 5.1.4′s document encryption implementation, including:

• VoodooPad Cryptography Overview - A high-level technical overview of how VoodooPad utilizes cryptography and the overall design of the VoodooPad’s document encryption.
• VoodooPad Cryptography Specification – The concrete specification of VoodooPad’s document encryption, including file formats, keying, ciphers, algorithms, and parameters.
• VoodooPad Encrypt-then-MAC AEAD Specification – Defines VoodooPad’s ETM-AEAD composition of AES-CBC, PKCS#7, and HMAC used for all authenticated encryption.

Plausible 2.0

December 15, 2014, by Landon Fuller

Seven years ago, we founded Plausible Labs as a worker-owned cooperative with the aim of building a company focused on sustainability, operating for the benefit of its owner-employees, and standing as a vibrant example of an alternative to the venture capital model in Silicon Valley.

I’m incredibly happy to announce that starting next year, for the first time in the co-operative’s history, our entire team will be focused on self-directed, self-funded product development. I look forward to sharing more about what we’re working on over the coming months, and the results of our labors over the coming years.

This has been a long road — founded with a $10,000 personal loan, we’ve neither borrowed money nor received investment since. The success of the co-operative has instead been entirely the product of the hard work of our employee-owners; including our current team of Terri Kramer, Chris Campbell, Mike Ash, and Rebecca Bratburd — all of whom I could not be more proud to work with. Relying heavily on consulting work, it took enormous dedication and stamina on everyone’s part to continue to invest our income, where we could, in our long-term goal of becoming a full-time product company.

We’ve accomplished a great deal to be proud of: early projects such as PLJukebox, a re-implementation of Apple’s Cover Flow that helped fund our initial growth, our consulting work on products such as comiXology’s mobile Comics applications, and on our ongoing stewardship of open source projects like PLCrashReporter, a library that powers most of the crash reporting services available for Apple’s platforms today.

Now we’ll demonstrate what Plausible Labs can do with time, resources, and a singular focus.