PL

Let's Build with PLRelational, Part 1

September 18, 2017, by Chris Campbell

This is our latest entry in a series of articles on PLRelational. For more background, check out these other articles in the series:

Since we started opening up about PLRelational, a number of developers have asked us how it compares to some particular existing technology, e.g. Core Data, Rx, Cocoa Bindings, or SQLite. It's a good question, but before we can clearly explain the differences, we should first dissect a working application that was built using PLRelational.

Our goal with this article is to give a big picture look at PLRelational and how it can be used to build an actual application. We will show how you can do things the PLRelational Way, and that will give us a baseline to help compare and contrast to existing technologies in a future article.

We believe PLRelational[+Binding] gives you a unique, expressive vocabulary for declaring how your UI is connected. Once you've declared those relationships, the frameworks can do a lot of heavy lifting, simplifying the way you build an application. As we will demonstrate here, there are a few patterns commonly used when working with PLRelational that can help bring a sense of order and sanity to application development.

What Should We Build?

To get started, we need to think of a relatively constrained type of application to build, and there's no better (or more clich├ęd) example than a to-do app, so let's run with that. In a nutshell, our application will allow the user to:

  • enter new to-do items
  • mark an item as completed
  • change the item text
  • add tags to an item
  • add notes for an item
  • delete an item

When the application is finished, it should look something like the following, except a bit prettier:

Additionally, our application should automatically save everything to disk, and it should allow the user to undo or redo any change.

How Do We Organize It?

When building applications using PLRelational, we use an approach that is very similar to MVVM (Model / View / View Model).

In traditional MVVM-based apps, it is often the case that the Model layer is based on some sort of in-memory object graph that is pulled from (and stored back to) some source, e.g. files on disk, a database, or the network.

With PLRelational, we forego the traditional object graph and instead build the Model by declaring how our source relations are laid out (most likely backed by some on-disk storage, e.g. plists or SQLite) and defining some base operations that modify those relations.

As seen in this diagram, when building with PLRelational the implementation of each piece of the user interface is typically spread across three layers:

  • In the Model layer, we declare our source Relations and define transactions that modify those Relations.

  • In the View Model layer, we declare Relation / Signal / Property chains that express how the data from our Model layer is to be transformed for display. If the interaction is two-way, as is the case for something like a text field, we'll also declare how changes made by the user should be transformed before being stored back via the Model layer.

  • In the View layer, we define the views and bind them to the Property chains that we expressed in our View Model layer.

Note that in a larger application, you would probably consider splitting up the Model layer over multiple classes for improved separation of concerns and modularity. Our to-do application on the other hand is relatively simple, so a single Model class will be sufficient to encapsulate all the relations we care about.

Declaring Relations

Let's begin building our Model layer by declaring the attributes and layout of our 4 relations, which will be stored on disk using property list files:

/// Scheme for the `item` relation that holds the to-do items.
enum Item {
    static let id = Attribute("item_id")
    static let title = Attribute("title")
    static let created = Attribute("created")
    static let status = Attribute("status")
    static let notes = Attribute("notes")
    fileprivate static var spec: Spec { return .file(
        name: "item",
        path: "items.plist",
        scheme: [id, title, created, status, notes],
        primaryKeys: [id]
    )}
}

/// Scheme for the `tag` relation that holds the named tags.
enum Tag {
    static let id = Attribute("tag_id")
    static let name = Attribute("name")
    fileprivate static var spec: Spec { return .file(
        name: "tag",
        path: "tags.plist",
        scheme: [id, name],
        primaryKeys: [id]
    )}
}

/// Scheme for the `item_tag` relation that associates zero
/// or more tags with a to-do item.
enum ItemTag {
    static let itemID = Item.id
    static let tagID = Tag.id
    fileprivate static var spec: Spec { return .file(
        name: "item_tag",
        path: "item_tags.plist",
        scheme: [itemID, tagID],
        primaryKeys: [itemID, tagID]
    )}
}

/// Scheme for the `selected_item` relation that maintains
/// the selection state for the list of to-do items.
enum SelectedItem {
    static let id = Item.id
    fileprivate static var spec: Spec { return .transient(
        name: "selected_item",
        scheme: [id],
        primaryKeys: [id]
    )}
}

We use enums to provide a namespace for the Attributes associated with each Relation. For each one we also declare a Spec (typealias for PlistDatabase.RelationSpec) that tells our PlistDatabase how the relations will be stored on disk in property list format. In the case of ItemTag and SelectedItem, we use identifier attributes that act as foreign keys referring to the Item and Tag relations.

Preparing the Model

The next step is to use these "specs" to initialize a PlistDatabase by loading existing data from disk or creating a new one. We wrap it in a TransactionalDatabase so that we can capture and apply snapshots. That class also has a handy saveOnTransactionEnd feature that, when enabled, gives us auto-save functionality. Finally, we wrap the TransactionalDatabase in an UndoableDatabase which will help us coordinate undoable/redoable operations:

class Model {
    let items: TransactionalRelation
    let tags: TransactionalRelation
    let itemTags: TransactionalRelation
    let selectedItemIDs: TransactionalRelation

    ...

    init(undoManager: PLRelationalBinding.UndoManager) {
        let specs: [Spec] = [
            Item.spec,
            Tag.spec,
            ItemTag.spec,
            SelectedItem.spec
        ]

        // Create a database or open an existing one (stored on disk using plists)
        let path = "/tmp/TodoApp.db"
        let plistDB = PlistDatabase.create(URL(fileURLWithPath: path), specs).ok!

        // Wrap it in a TransactionalDatabase so that we can use snapshots, and
        // enable auto-save so that all changes are persisted to disk as needed
        let db = TransactionalDatabase(plistDB)
        db.saveOnTransactionEnd = true
        self.db = db

        // Wrap that in an UndoableDatabase for easy undo/redo support
        self.undoableDB = UndoableDatabase(db: db, undoManager: undoManager)

        // Make references to our source relations
        func relation(for spec: Spec) -> TransactionalRelation {
            return db[spec.name]
        }
        items = relation(for: Item.spec)
        tags = relation(for: Tag.spec)
        itemTags = relation(for: ItemTag.spec)
        selectedItemIDs = relation(for: SelectedItem.spec)
    }
}

Wiring Up the User Interface

Now that we've laid out our primary relations, let's set our sights on building out the user interface.

In this article we will be focusing on the left side of the application, namely the "Add a to-do" field and the list of to-do items. (There is a lot to share about the implementation of the right side too, but in the interests of time and space we will save that for Part 2, coming soon.)

Click the "Play" button below to see this part of the application in action:

To keep things organized, I decided to break the application down into distinct functional requirements. Each requirement corresponds to a specific portion of the UI and explains how we want it to behave.

The following interactive screenshot shows the left half of the app in its completed state. Each red dot corresponds to a single functional requirement. Hover over a dot to read a summary of the requirement, then click to see how we implemented it using PLRelational.

1 2 3 4 5

In each of the sections that follow, we will show the relevant portions of code (really, a cross section of the Model, *ViewModel, and *View classes) that were used to implement a functional requirement. To explore the sources in Xcode, feel free to clone the PLRelational repository and follow along with the sources under Example Apps > TodoApp in the project.

REQ-1: New Item Field

A text field at top-left that allows the user to enter new to-do items. When the user types a non-empty string, a new pending item should be added at the top of the list, and the text field should be cleared.

Let's translate this requirement into some code. The first step is to define an undoable transaction in the Model that adds a single row to the items relation. We expose this as an ActionProperty in the ChecklistViewModel, and then set up a binding in ChecklistView:

class Model { ...
    /// Adds a new row to the `items` relation.
    private func addItem(_ title: String) {
        // Use UUIDs to uniquely identify rows.  Note that we can pass `id` directly
        // when initializing the row because `ItemID` conforms to the
        // `RelationValueConvertible` protocol.
        let id = ItemID()

        // Use a string representation of the current time to make our life easier
        let now = timestampString()

        // Here we cheat a little.  ArrayProperty currently only knows how to sort
        // on a single attribute (temporary limitation), we cram two things -- the
        // completed flag and the timestamp of the action -- into a single string of
        // the form "<0/1> <timestamp>".  This allows us to keep to-do items sorted
        // in the list with pending items at top and completed ones at bottom.
        let status = statusString(pending: true, timestamp: now)

        // Insert a row into the `items` relation
        items.asyncAdd([
            Item.id: id,
            Item.title: title,
            Item.created: now,
            Item.status: status,
            Item.notes: ""
        ])
    }

    /// Adds a new row to the `items` relation.  This is an undoable action.
    func addNewItem(with title: String) {
        undoableDB.performUndoableAction("Add Item", {
            self.addItem(title)
        })
    }
}

class ChecklistViewModel { ...
    /// Creates a new to-do item with the given title.
    lazy var addNewItem: ActionProperty<String> = ActionProperty { title in
        self.model.addNewItem(with: title)
    }
}

class ChecklistView { ...
    init() { ...
        // Add a new item each time a string is entered into the text field
        newItemField.strings ~~> model.addNewItem
    }
}

Note that the EphemeralTextField class in PLBindableControls takes care of delivering the text via its strings signal and clearing out the text field when the user presses enter.

Here we've established a pattern — breaking things down across the three components — that we'll see again and again in the implementation of our app. This layout encourages isolation between individual pieces of the UI and also allows for easy testing (a topic that we'll cover in a future article).

REQ-2: Items List

A list view on the left side that contains all to-do items. The list should be sorted such that the first part of the list contains pending items, and the second part contains completed items. Pending items should be sorted with most recently added items at top. Completed items should be sorted with most recently completed at top.

Table and outline views are a frequent source of headaches when building iOS and macOS applications. PLRelationalBinding and PLBindableControls include the ArrayProperty and ListView classes, respectively, that do a lot of heavy lifting so that we can focus simply on how the data is related. Here we use arrayProperty to lift the contents of our items relation into a form that can track rows by their unique identifier and keep them sorted by their status value:

class ChecklistViewModel { ...
    /// The model for the list of to-do items.
    lazy var itemsListModel: ListViewModel<RowArrayElement> = {
        return ListViewModel(
            data: self.model.items.arrayProperty(idAttr: Item.id,
                                                 orderAttr: Item.status,
                                                 descending: true),
            cellIdentifier: { _ in "Cell" }
        )
    }()
}

class ChecklistView { ...
    init() { ...
        // Bind outline view to the list view model
        listView = CustomListView(model: model.itemsListModel,
                                  outlineView: outlineView)
    }
}

The ListView class reacts to changes in the ArrayProperty, taking care of animating insertions and deletions (as seen in the animation at the top of this section). As you can see, it takes very little code to set this up; no custom NSTableViewDataSource or NSTableViewDelegate implementation required.

REQ-3: Item Cell Completed Checkbox

Each list cell will have a checkbox on the left side indicating whether the item is pending (unchecked) or completed (checked). If the user clicks the checkbox such that it becomes checked, the item should animate down the list to sit at the top of the completed section. If the user clicks the checkbox such that it becomes unchecked, the item should animate to the top of the list.

In the previous step, we used ArrayProperty to describe how the rows of to-do items are organized, but we still need to break each list cell down into three parts: checkbox, text field, and (tags) label.

For each of these cell components, we need to set up a conduit that takes data from a specific part of the underlying relation and delivers it to that part of the list cell.

Click the following to see an animation that helps visualize how, for each checkbox, we define a Property (in this case, an AsyncReadWriteProperty<CheckState>) that serves as a two-way transform:

Now, let's translate this into code. In Model we define a bidirectional transform that converts the checkbox state to our custom status string and vice versa. ChecklistViewModel creates an instance of that transform for a given Row (i.e., a list cell), and then in ChecklistView we bind the checkbox state to the view model:

class Model { ...
    /// Returns a property that reflects the completed status for the given relation.
    func itemCompleted(_ relation: Relation, initialValue: String?) -> AsyncReadWriteProperty<CheckState> {
        return relation.undoableTransformedString(
            undoableDB, "Change Status", initialValue: initialValue,
            fromString: { CheckState(parseCompleted($0)) },
            toString: { statusString(pending: $0 != .on, timestamp: timestampString()) }
        )
    }
}

class ChecklistViewModel { ...
    /// Returns a read/write property that resolves to the completed status for
    /// the given to-do item.
    func itemCompleted(for row: Row) -> AsyncReadWriteProperty<CheckState> {
        let itemID = ItemID(row[Item.id])
        let initialValue: String? = row[Item.status].get()
        let relation = self.model.items.select(Item.id *== itemID).project(Item.status)
        return self.model.itemCompleted(relation, initialValue: initialValue)
    }
}

class ChecklistView { ...
    init() { ...
        listView.configureCell = { view, row in ...
            // Bidirectionally bind checkbox state to the view model
            let checkbox = cellView.checkbox!
            checkbox.checkState.unbindAll()
            checkbox.checkState <~> model.itemCompleted(for: row)
        }
    }
}

There are a few things worth highlighting here:

  • In Model, our itemCompleted transform is set up using undoableTransformedString which provides built-in support for registering an action with the underlying UndoManager. UndoableDatabase will take care of reverting to the previous state if the user performs an "Undo" action, or reapplying the change after a "Redo" action.

  • In ChecklistViewModel, note how we use select and project to hone in an individual value in a specific row of a relation. We are effectively setting up a live connection to that value, but note that we never have to explicitly load or store an object.

  • In ChecklistView, the <~> operator means "set up a bidirectional binding between these two things": changes initiated by the user flow back to the model, and vice versa.

REQ-4: Item Cell Title Field

Each list cell will have the to-do item title to the right of the checkbox. The user should be able to change the title by clicking in the list cell's text field. The title field should be updated if the user changes it in the detail view, and vice versa.

The process we use to hook up the text field for each list cell is almost the same as what we did for the checkboxes in the previous step. Click to visualize:

Once again, let's translate this into code. The implementation of this requirement is very similar to the previous one, so it should all look familiar:

class Model { ...
    /// Returns a property that reflects the item title.
    func itemTitle(_ relation: Relation, initialValue: String?) -> AsyncReadWriteProperty<String> {
        return relation.undoableOneString(undoableDB, "Change Title", initialValue: initialValue)
    }
}

class ChecklistViewModel { ...
    /// Returns a read/write property that resolves to the title for the given
    /// to-do item.
    func itemTitle(for row: Row) -> AsyncReadWriteProperty<String> {
        let itemID = ItemID(row[Item.id])
        let initialValue: String? = row[Item.title].get()
        let relation = self.model.items.select(Item.id *== itemID).project(Item.title)
        return self.model.itemTitle(relation, initialValue: initialValue)
    }
}

class ChecklistView { ...
    init() { ...
        listView.configureCell = { view, row in ...
            // Bidirectionally bind title field to the view model
            let textField = cellView.textField as! TextField
            textField.string.unbindAll()
            textField.string <~> model.itemTitle(for: row)
        }
    }
}

The itemTitle transform is even simpler than the itemCompleted transform that we saw in the previous step. The undoableOneString convenience gives us a two-way (read/write) property that resolves to the title string value of the to-do item, and then writes updates back to the source relation when the user changes that string in the UI (again the undo support is handled for us).

REQ-5: Item Cell Tags Label

Each list cell will have a read-only label containing applied tags on the right side. The tags should be comma-separated and in alphabetical order. The label should be updated whenever the user adds or removes a tag for that item in the detail view.

For the third and final piece of our list cells, we will display a string representation of the list of tags applied to that item. Unlike the previous two steps (checkbox and text field), this one doesn't accept input from the user, so it's just a matter of creating a read-only property. Click to visualize:

Here's what that looks like in code form:

class Model { ...
    /// Returns a property that resolves to a string containing a comma-separated
    /// list of tags that have been applied to the given to-do item.
    func tagsString(for itemID: ItemID) -> AsyncReadableProperty<String> {
        return self.itemTags
            .select(ItemTag.itemID *== itemID)
            .join(self.tags)
            .project(Tag.name)
            .allStrings()
            .map{ $0.sorted().joined(separator: ", ") }
            .property()
    }
}

class ChecklistViewModel { ...
    /// Returns a property that resolves to the list of tags for the given
    /// to-do item.
    func itemTags(for row: Row) -> AsyncReadableProperty<String> {
        return self.model.tagsString(for: ItemID(row))
    }
}

class ChecklistView { ...
    init() { ...
        listView.configureCell = { view, row in ...
            // Bind detail (tags) label to the view model
            let detailLabel = cellView.detailLabel!
            detailLabel.string.unbindAll()
            detailLabel.string <~ model.itemTags(for: row)
        }
    }
}

It's worth dissecting that tagsString declaration; it's an interesting one. Let's break it down step by step:

// Start with the `itemTags` source relation (Item.id : Tag.id pairs)
self.itemTags

// Select the rows corresponding to the given to-do item
.select(ItemTag.itemID *== itemID)

// Join with the `tags` relation to get the tag names
.join(self.tags)

// Take just the tag names (we can drop the IDs)
.project(Tag.name)

// Derive a `Signal` that carries the tag names as a `Set<String>`
.allStrings()

// Convert the `Set<String>` into a sorted, comma-separated string
.map{ $0.sorted().joined(separator: ", ") }

// Lift the `Signal` into an `AsyncReadableProperty` that offers the latest value
.property()

This demonstrates a common practice when working with PLRelational:

  • Start with one or more Relations
  • Slice and dice (i.e., select, join, and project) until you've narrowed your focus onto a small set of data
  • Derive a Signal that extracts Swift-ly typed values
  • Transform the values using Signal operators like map
  • Lift to an AsyncReadableProperty for easier binding (and unit testing)

By declaring the relationships in this manner, any time the list of tags changes for that to-do item (by virtue of the user adding/removing tags, or as a result of undo/redo, etc), the list cell will automatically be updated to display the latest string value.

Wrapping Up

In this article, we demonstrated how to use PLRelational and its reactive-relational style of programming to build a real, working macOS application.

Building the PLRelational Way takes a slightly different mindset as compared to more imperative, object-oriented approaches. However, once you've learned the few core patterns that we presented here, the reactive-relational approach becomes second nature, and using the vocabulary offered by PLRelational can help simplify the way you build applications.

In Part 2, we will continue our deep dive and explore the right half of the to-do application. In that article we will cover topics such as modeling list selection, building with complex controls like combo boxes, using cascading deletion, and more.

The complete source code for this application is available in the PLRelational repository under the Examples/TodoApp directory and can be run directly from the Xcode project.


Learn Swift from the experts at Plausible Labs! We offer on-site corporate workshops with hands-on instruction in the deep and mysterious ways of Swift. Check out our training offerings for more information, or get in touch!

The Best New Features in Swift 4

September 13, 2017, by Mike Ash

Swift 4 is here, and it's bringing some nice changes. We're not getting a radical rework of the syntax like we did last year, nor are we getting a breathtaking pile of new features like we did for Swift 2, but there are some nice additions you can use to improve your code. Let's take a look!

Multi-Line String Literals

Sometimes you want long, multi-line strings in your code. It might be an HTML template, a blob of XML, or a long message for the user. Either way, they're painful to write in Swift 3.

You can write them out all on one line, which gets ugly fast:

let message = "Please disable your Frobnitz before proceeding.\n\nTo do this, visit Settings -> Frobnitz, then toggle the switch to \"off\".\n\nIf you need the Frobnitz to remain enabled, tap \"Proceed Anyway\" below."

You can split it onto multiple lines by concatenating strings:

let message =
    "Please disable your Frobnitz before proceeding.\n\n"
  + "To do this, visit Settings -> Frobnitz, then toggle the switch to \"off\".\n\n"
  + "If you need the Frobnitz to remain enabled, tap \"Proceed Anyway\" below."

There are other ways to do it too, but none of them are all that good.

Swift 4 solves this problem with multi-line string literals. To write a multi-line string literal, use three quote marks at the beginning and end:

let message = """
    Please disable your Frobnitz before proceeding.

    To do this, visit Settings -> Frobnitz, then toggle the switch to "off".

    If you need the Frobnitz to remain enabled, tap "Proceed Anyway" below.
    """

If you've used Python, then this new syntax will look familiar. However, it's not quite the same. There are some interesting limitations and features of this syntax in Swift.

This triple-quote syntax cannot be used on a single line. Something like the following will not compile:

// Will not compile
label.text = """Put your text in "quotes" to make them look quoted."""

This could be handy to avoid having to escape quotes, but it's not allowed. The content of the string must be on separate lines between the """ marks.

Multi-line strings can be indented in your code without indenting the final result. The multi-line string above indents each line in the code, but the string placed into message has no leading whitespace. This is really nice, but what if you want some indentation? This feature is based on the indentation of the closing """ mark. Its indentation will be stripped off all of the other lines. If for some reason you needed the contents of message to be indented, you can do so by indenting the text farther than the closing """ mark:

let message = """
        Please disable your Frobnitz before proceeding.

        To do this, visit Settings -> Frobnitz, then toggle the switch to "off".

        If you need the Frobnitz to remain enabled, tap "Proceed Anyway" below.
    """

To avoid confusion, each line of text must be indented at least as much as the closing """ mark. A line with less indentation will produce an error.

You may want to split your text onto multiple lines without producing multiple lines in the output. You can remove a line break from the resulting string by adding a \ at the end of the line:

let message = """
    Please disable your Frobnitz before proceeding. \
    To do this, visit Settings -> Frobnitz, then toggle the switch to "off". \
    If you need the Frobnitz to remain enabled, tap "Proceed Anyway" below.
    """

One-Sided Ranges

This is a nice, small change that's mostly self explanatory. Ranges can now be one-sided, and the "empty" side is implied to be the minimum or maximum value that makes sense in context.

When subscripting a container, this means you can leave off things like string.endIndex or array.count. For example, if you want to split an array into halves:

let middle = array.count / 2
let firstHalf = array[..<middle]
let secondHalf = array[middle...]

Or if you want to get a substring up to a particular index:

let index = string.index(of: "e")!
string[..<index]

It can also be handy in switch statements:

switch x {
case ..<0:
    print("That's a negative.")
case 0:
    print("Nothing!")
case 1..<10:
    print("Pretty small.")
case 10..<100:
    print("Bigger.")
case 100...:
    print("Huge!")
default:
    // Unfortunately, the compiler can't figure out
    // that the above cases are exhaustive.
    break
}

For one-sided ranges up to a given value, you can use ..< for an exclusive range or ... for an inclusive range, just like two-sided ranges. For one-sided ranges starting at a given value, only ... is allowed, since the distinction between ... and ..< makes no sense there.

Combined Class and Protocol Types

Sometimes you need an object which both subclasses a class and conforms to a protocol. For example, you might need a UITableViewController that also implements KittenProvider. Swift 3 had no way to express this idea, requiring various ugly workarounds. Interestingly, Objective-C is able to express this idea:

UITableViewController<KittenProvider> *object;

Swift 4 can now express this concept as well by using the & symbol. This could already be used to combine multiple protocols into a single type, and now it can also be used to combine protocols with a class:

let object: UITableViewController & KittenProvider

Note that only one class can be included in any such type, since you can't subclass more than one class at a time anyway.

Generic Subscripts

Swift has supported generic methods forever, but before Swift 4 it did not support generic subscripts. You could overload subscripts by implementing more than one with different types, but you couldn't use generics. Now you can!

subscript<T: Hashable>(key: T) -> Value?

Generics are fully supported, so you can use things like where clauses:

subscript<S: Sequence>(key: S) -> [Value] where S.Element == Key

Just like methods, the generic type can be used as a return value as well:

subscript<T>(key: Key) -> T?

This could be really handy for dynamically-typed containers, such as when dealing with JSON objects.

Codable

Speaking of JSON, perhaps the biggest new feature in Swift 4 is the Codable protocol. The compiler will now auto-generate serialization and deserialization code for your types, and all you have to do is declare conformance to Codable.

Imagine you have a Person type:

struct Person {
    var name: String
    var age: Int
    var quest: String
}

If you wanted to read and write Person values to and from JSON, you previously had to write a bunch of annoying repetitive code to do so.

In Swift 4, you can make this happen by adding half a line:

struct Person: Codable {

If for some reason you only want to support encoding or decoding, but not both, you can declare conformance to Encodable or Decocable separately:

struct EncodablePerson: Encodable { ... }

struct DecodablePerson: Decodable { ... }

Conforming to Codable is just a shortcut for conforming to both.

Using a Codable type requires an encoder or decoder, which determines the serialization format and how Swift values are translated to and from serialized values. Swift provides encoders and decoders for JSON and property lists, and Foundation's archivers also support Codable types.

To encode something as JSON, create a JSONEncoder and call its encode method:

let jsonEncoder = JSONEncoder()
let data = try jsonEncoder.encode(person)

To decode, create a JSONDecoder and call decode, passing it the type you want to decode and the data to decode from:

let jsonDecoder = JSONDecoder()
let decodedPerson = try jsonDecoder.decode(Person.self, from: data)

Note that encoding and decoding methods are marked as throws because there are a lot of potential errors that can occur during the process, such as type mismatches or incomplete data, so you'll need to add try to these calls and catch the errors they throw.

Since JSON doesn't natively support dates or binary data, those values need to be converted to/from some other JSON representation. For example, it's common to use base64 encoding for data. JSONEncoder and JSONDecoder can be customized with different strategies for handling these values. For example, if you want to encode dates as ISO-8601 and data as base64:

let jsonEncoder = JSONEncoder()
jsonEncoder.dateEncodingStrategy = .iso8601
jsonEncoder.dataEncodingStrategy = .base64
let data = try jsonEncoder.encode(person)

And on the decode side:

let jsonDecoder = JSONDecoder()
jsonDecoder.dateDecodingStrategy = .iso8601
jsonDecoder.dataDecodingStrategy = .base64
let decodedPerson = try jsonDecoder.decode(Person.self, from: data)

They also provide the option for providing a totally custom strategy by writing your own code.

Property list coding is similar. Use PropertyListEncoder and PropertyListDecoder instead of the JSON coders. Since property lists can natively represent dates and binary data, the property list coders don't provide those options.

If you're already using NSCoding then you can mix and match it with Codable so that you don't have to change everything at once. NSKeyedArchiver provides a new encodeEncodable method which takes any Encodable type and encodes it under the given key. NSKeyedUnarchiver provides a corresponding decodeDecodable which can decode any Decodable type.

Codable is a flexible protocol with lots of room for custom behavior. The compiler provides a default implementation, but you can provide your own if you need different behavior. This makes it straightforward to write implementations that migrate old data into new types, use different names in the serialized representation than in the source code, or make other customizations.

A full discussion of all the possibilities of Codable is beyond the scope of this article, but you can read more about it in my Friday Q&A post about Swift.Codable.

Wrapping Up

Swift 4 hasn't brought us the dramatic changes we've seen in earlier years, but it's a solid improvement. Codable will make some really common tasks a lot easier, and it's probably my favorite feature out of the bunch. Other features like multi-line string literals and generic subscripts won't have the same impact, but put together they should make for nice improvement in the code we write.


Learn Swift from the experts at Plausible Labs! We offer on-site corporate workshops with hands-on instruction in the deep and mysterious ways of Swift. Check out our training offerings for more information, or get in touch!

PLRelational: Storage Formats

September 7, 2017, by Mike Ash

This is our latest entry in a series of articles on PLRelational. For more background, check out these other articles in the series:

We've been talking a lot lately about PLRelational and all the fancy stuff it does. However, we've been glossing over a fundamental part of it: how it actually stores data. After all, PLRelational is a data persistence framework at its core.

PLRelational's relations break down into two categories. There are relations which store rows, and there are relations which derive their contents by performing some operation on other relations. Many of the relations which store rows do so by persisting them to disk.

Memory Relations

Let's start with one that doesn't persist its contents, just for simplicity. The basic interface is the same, so it provides a good foundation.

The MemoryTableRelation class is a relation which, as the name indicates, stores its data in memory. You create one with a scheme:

let people = MemoryTableRelation(scheme: ["id", "name", "quest", "favorite-color"])

Once you have that, you can add, update, and delete rows:

people.asyncAdd(["id": 1, "name": "Lancelot",
                 "quest": "to find the holy grail", "favorite-color": "blue"])
people.asyncAdd(["id": 2, "name": "Galahad",
                 "quest": "to find the holy grail", "favorite-color": "blue"])
people.asyncUpdate(Attribute("id") *== 2, newValues: ["favorite-color": "yellow"])
people.asyncDelete(Attribute("id") *== 2)

Of course, this is purely in memory, and the data is lost once the people relation is destroyed. It can still be useful to represent transient data, but doesn't persist anything.

Property List Files

We might imagine taking this and building our own persistence on top of it by saving data to a property list file. We'd define a simple way to turn rows into property list objects. To save, we'd fetch the rows in the relation, convert them to property list objects, and save the whole thing into a file. To load, we'd load the property list file, convert it to rows, and add them to the relation.

This is essentially what PlistFileRelation is. It's an in-memory relation that's backed by a property list file. We can change people to use PlistFileRelation by changing the initialization:

let people = try PlistFileRelation.withFile(plisturl,
    scheme: ["id", "name", "quest", "favorite-color"],
    primaryKeys: ["id"], create: false
).orThrow()

It can then be used just like the in-memory version of people. It will automatically load data from the property list file, and when you want to save changes back to that file, call save:

let result = people.save()
if let error = result.err {
    // handle error
}

You may have noticed the primaryKeys parameter in the code that creates the relation. Specifying primary keys allows PLRelational to optimize queries involving those attributes, at the expense of some overhead when adding, removing, or changing rows. MemoryTableRelation is assumed to always contain relatively small amounts of data and doesn't have this option.

Property List Directories

A single property list file works well for moderate amounts of data, but it requires reading the entire file at once, and rewriting the entire file on save. This overhead can become significant once you start working with large amounts of data.

PlistDirectoryRelation solves this problem by storing the rows in individual files within a directory. This allows reading and writing individual rows separately.

We can use this by once again changing the initializer for people:

let people = try PlistDirectoryRelation.withDirectory(dirurl,
    scheme: ["id", "name", "quest", "favorite-color"],
    primaryKey: "id", create: true
).orThrow()

As with PlistFileRelation, changes are not saved automatically. Instead, a delta is kept in memory, and then changes can be flushed to disk by calling save:

let result = people.save()
if let error = result.err {
    // handle error
}

When creating a PlistDirectoryRelation, it takes a primaryKey parameter, singular. Unlike PlistFileRelation, PlistDirectoryRelation can only have a single primary key. This is due to how it efficiently locates the files for each row.

The filename for a row's plist is determined from the value for the primary key in that row. The filename is derived from that value by converting it to a canonical representation in raw bytes and then taking the SHA-256 hash of that representation. The result is something like:

4cd9b7672d7fbee8fb51fb1e049f690342035f543a8efe734b7b5ffb0c154a45.rowplist

Because the mapping is deterministic and based only on the value of the primary key, equality queries for the primary key can be done quickly. In this example, the deletion at the end will compute the filename for the ID of 2 and then mark that file for deletion. Joins or selects involving the primary key will efficiently look up only the data they need.

Property List Databases

Your app will probably use multiple stored relations for different kinds of data. This may be a mix of property list files and directories. For example, you might place metadata that's needed everywhere in property list files, while large data that should only be loaded on demand is in a directory.

The PlistDatabase class handles this. You create it with a list of relation specs, which describe each relation's scheme, primary keys, path, and name. It then handles creating, loading, and saving all of those relations for you, and bundles them all into a single directory structure. This example creates a people relation stored in a file and a people-images relation stored in a directory, since images are probably large:

let db = try PlistDatabase.create(rootURL, [
    .file(
        name: "people", path: "people.plist",
        scheme: ["id", "name", "quest", "favorite-color"],
        primaryKeys: ["id"]
    ),
    .directory(
        name: "people-images", path: "people-images",
        scheme: ["id", "image-data"],
        primaryKey: "id"
    )
]).orThrow()

let people = db["people"]!
let images = db["people-images"]!

people and images will contain whatever data currently exists there, and can be manipulated with the standard Relation methods:

people.asyncAdd(["id": 1, "name": "Lancelot",
                 "quest": "to find the holy grail", "favorite-color": "blue"])
people.asyncAdd(["id": 2, "name": "Galahad",
                 "quest": "to find the holy grail", "favorite-color": "blue"])
people.asyncUpdate(Attribute("id") *== 2, newValues: ["favorite-color": "yellow"])
people.asyncDelete(Attribute("id") *== 2)

images.asyncAdd(["id": 1, "image-data": imageData])

To save changes back to disk, call saveRelations:

let result = db.saveRelations()
if let error = result.err {
    // handle error
}

This will save all of the individual relations.

Dropbox/Cloud Sync

Property list storage works well for syncing to Dropbox and other cloud storage, which tend to work with file granularity. Placing a file relation in cloud storage allows the whole relation to be synced as a single unit, and using a directory relation allows individual rows to be added, updated, and deleted independently.

PLRelational doesn't directly talk to cloud storage, but it does have some special facilities to accommodate it. Ultimately there are two kinds of actions that need to be handled: local changes to files which need to be synced to the server, and remote changes to files which need to be synced locally.

Local changes are handled using PlistDatabase's addSaveObserver method. Pass it a function which will be invoked any time any of the relations in the database change a file on disk. It receives the URL of the file that was changed. It can then take whatever action is needed, like making an API call, to sync that file:

db.addSaveObserver({ url in
    cloudAPI.syncLocalFile(url)
})

Local changes are handled by asking the PlistDatabase to make a change to a local URL. To replace a local file with a new version, call replaceLocalFile. This can also be used to add a new file, by giving it the URL to where the new file should go:

func gotNewFile(tmpURL: URL, localFileURL: URL) {
    db.replaceLocalFile(url: localFileURL, movingURL: tmpURL)
}

func gotNewFileVersion(tmpURL: URL, localFileURL: URL) {
    db.replaceLocalFile(url: localFileURL, movingURL: tmpURL)
}

Note that the new file must first be saved to a temporary location, and the PlistDatabase takes care of moving it to its final location. This allows PlistDatabase to read any old data that was in the file previously, which is necessary to generate proper change notifications.

To delete a local file, call deleteLocalFile:

func deleteFile(url: URL) {
    db.deleteLocalFile(url: url)
}

These methods understand the layout of the database's files on disk and will look up the appropriate Relation for any given local URL. When applying the changes, the Relation in question will generate the appropriate change notifications, meaning that your UI remains automatically in sync with any changes generated by cloud sync activity. It's also smart enough to know which files are not part of the database, and will refuse to perform the operation if asked to operate on a URL that doesn't belong to the database. The return value of these methods will tell you whether the action was performed or not, making it easy to sync files outside the PlistDatabase too.

SQLite Databases

SQLite and PLRelational fit together well. SQLite was our first target for persistent storage, and PLRelational's data types match what SQLite provides.

PLRelational's SQLite support starts with the SQLiteDatabase class. As the name indicates, this represents an SQLite database, and it provides the individual tables in the database as Relation objects.

To create an SQLiteDatabase instance, initialize it with a path to the database file:

let db = try SQLiteDatabase(path)

If a database already exists at that location, it will open the existing database. Otherwise it will create a new one.

To create a new table, use the createRelation method:

let people = try db.createRelation(
    "people",
    scheme: ["id", "name", "quest", "favorite-color"]).orThrow()

To fetch an existing table, use subscripting with the table name:

let images = db["people-images"]

For the common case where you want to create the table if it doesn't exist and fetch it if it does, use the getOrCreateRelation method:

let people = try db.getOrCreateRelation(
    "people",
    scheme: ["id", "name", "quest", "favorite-color"]).orThrow()
let images = try db.getOrCreateRelation(
    "people-images",
    scheme: ["id", "image-data"]).orThrow()

As before, once you have the Relations, you can manipulate them with the usual calls:

people.asyncAdd(["id": 1, "name": "Lancelot",
                 "quest": "to find the holy grail", "favorite-color": "blue"])
people.asyncAdd(["id": 2, "name": "Galahad",
                 "quest": "to find the holy grail", "favorite-color": "blue"])
people.asyncUpdate(Attribute("id") *== 2, newValues: ["favorite-color": "yellow"])
people.asyncDelete(Attribute("id") *== 2)

images.asyncAdd(["id": 1, "image-data": Data(imageData)])

Unlike property list storage, the SQLite storage saves changes immediately. The asyncAdd call translates directly to an SQLite INSERT statement, for example. If you want to buffer changes in memory and only flush them to disk with an explicit save, you can accomplish this by wrapping the SQLiteDatabase in a ChangeLoggingDatabase:

let sqliteDB = try SQLiteDatabase(path)
_ = try sqliteDB.getOrCreateRelation(
    "people",
    scheme: ["id", "name", "quest", "favorite-color"]).orThrow()
_ = try sqliteDB.getOrCreateRelation(
    "people-images",
    scheme: ["id", "image-data"]).orThrow()

let db = ChangeLoggingDatabase(sqliteDB)
let people = db["people"]
let images = db["people-images"]

people.asyncAdd(["id": 1, "name": "Lancelot",
                 "quest": "to find the holy grail", "favorite-color": "blue"])
people.asyncAdd(["id": 2, "name": "Galahad",
                 "quest": "to find the holy grail", "favorite-color": "blue"])
people.asyncUpdate(Attribute("id") *== 2, newValues: ["favorite-color": "yellow"])
people.asyncDelete(Attribute("id") *== 2)

images.asyncAdd(["id": 1, "image-data": Data(imageData)])

ChangeLoggingDatabase will record all changes made to its relations rather than passing them directly to the SQLite layer. When you want to persist all changes, call save:

let result = db.save()
if let error = result.err {
    // handle error
}

This passes all of the changes to the SQLite layer, which writes them out all at once.

Advanced: Raw SQL in SQLite

For advanced uses, it's also possible to execute SQL queries directly against the database using the executeQuery method. This takes an SQL string and an optional array of RelationValue parameters and returns Rows for the results. PLRelational uses this to implement RelationTextIndex, which is a full text search API backed by SQLite's fts4 module.

If you use this, it's important to note that there is not a perfect mapping between RelationValue types and SQLite data types. The standard SQL NULL has a lot of weird behaviors. For example, comparing NULL for equality or inequality always produces NULL, which is evaluated as false. That means that, for example, a SELECT statement checking for == NULL or != NULL will always produce zero results regardless of the contents of the table.

SQLite matches this weird behavior in order to be compatible with other SQL databases. PLRelational doesn't have that constraint and so we decided to make NULL behave more consistently, and act like any other value. PLRelational's RelationValue.null is equal to itself, not equal to other values, gets sorted consistently, etc. However, this means that we can't use SQLite's NULL while still translating PLRelational select operations into SQLite SELECT statements. To work around this, we decided not to use SQLite's NULL at all, and instead translate RelationValue.null differently.

Instead, RelationValue.null is translated into an SQLITE blob containing the ASCII bytes "NULL". In order to ensure that actual blobs are never mistakenly interpreted as null, all blobs are prefixed with the ASCII bytes "BLOB". This works fine for normal PLRelational use and this translation step is invisible unless you go digging through the database by hand. However, if you're submitting raw SQL with executeQuery, it might get in the way. The optional bindBlobsRaw parameter allows you to control whether blobs are translated like this while going into and out of SQLite. It's false by default, meaning that translation is performed. By setting it to true, blobs will be passed in and out as-is. Note that this parameter does not influence how NULL is translated.

Try It Out

If you'd like to try out these code snippets or otherwise explore the available options, grab a copy of the official PLRelational repository (or GitHub mirror). The project includes example apps which you can examine and modify, and adding PLRelational to your own project is as easy as dropping the framework in and writing import PLRelational in your code.


Want to learn more? Plausible Labs offers consulting and training services for software engineering. If you'd like some professional assistance, whether with PLRelational or for something entirely different, consider us. We also offer on-site corporate training in addition to periodic, hands-on Swift workshops. Our next workshop will be held in New York City on September 29th; hope to see you there! More information can be found on our consulting and training pages.