Archive for April, 2018

Evolve Swift: Part 1 – Building your Tools

Equipment

If you’ve made the slightly rash but equally rewarding decision to dive into developing the Swift language then make sure you have access to a fast computer with good, fast storage. What you don’t want to do is be running an old Mac mini using an old external hard drive because Apple hasn’t updated the mini line in a few forevers.

If your “About this Mac” panel looks like this, seriously question why you’re trying to do this in the first place. For the sake of uncomplicated narrative, I’m going to assume that just about everyone out there in the real world has a better Mac with better memory and primary disk than I do:

For those interested in the naming from these screen shots: Esopus Spitzenburg., Kiku, and Broxwood Firewhelp. I have others but I think those are the names you might see in screen shots and text examples. Hopefully this will be the first of several posts about the topic, in honor of Paul Cantrell.

Embrace the Wait

Before you can change the compiler, you must get the compiler downloaded, compiled, and working in its most recent form. For me, this is multi-day event. You probably will have slightly fewer roadblocks. Still, keep your day open and focus on other work as your Mac may or may not wind tunnel its way through the process.

Even after you’ve got your toolchain under control and build, you’ll continue to have many many waits in your development process as you’ll want to keep updating your repo to match the current state of Apple’s master and its supporting tools. Each time you do this, you can expect a morning of recompiling before you can start working on anything significant.

During this time, you cannot reasonably explore, test, and build unless you do so on a separate machine or for distinct installs. It’s a big pain in the ass but it’s part and parcel of developing the language.

If you plan to do this casually, and a week  has passed between the last pull from upstream/master and now, don’t expect to be productive until the afternoon at the earliest.

Although Swift’s toolset provides a quick iteration tool (called “Ninja”), that adds fast incremental builds, getting up to date and staying up to date means a large time commitment.

Further, don’t be shocked when your compiler or tools simply fail to build. They do that sometimes. It’s utterly frustrating to be stuck with a broken build for whatever reason. Incorporating workarounds into your toolset is a big part of swifting.

If you can’t embrace the wait, you don’t want to be developing swiftlang.

Selecting Xcode

Ensure that your default Xcode is set correctly. Use xcode-select -p to check which version is selected. If it isn’t your main app, use the utility to establish the right one. I use the production (not beta) version of Xcode:

sudo xcode-select -s /Applications/Xcode.app/Contents/Developer

Package Managers

Make sure you have a good package manager installed. I use Homebrew. I know there was some kind of controversy about that a while ago but I can’t remember why and it does the job for me.

Update your package manager to the most recent version:

brew update

If you run into any issues, you can do a full remove and re-install. These calls require bash and will misfire in csh/tcsh:

Uninstall Brew (removes all existing packages):

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall)"

Install:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Get cmake and ninja

Grab a copy of cmake and ninja if you do not already have them installed on your system. CMake provides a cross-platform build/test/package solution that generates appropriate Make files, specialized to your environment. Ninja offers a speedy solution for incremental builds. It’s really fast, so once you’ve built out your libraries, you can go in and ninja swift for a really fast update when you’re doing edit/build/test cycles.

brew install cmake ninja

Go ahead and install sphinx now too. Sphinx “makes it easy to create intelligent and beautiful documentation” :

brew install sphinx

Notifications

Because of the long, long nature of setup and update process, consider some tools to help keep you on top of tasks, so you know when they finish and can move on to your next steps.

I use a command-line utility called noti to announce when each stage of downloading and compilation completes. Noti is an open source tools that triggers feedback as a command line process finishes.

You can have the utility speak, create a macOS system notification, and more. It’s helpful for when you’re running long tasks in background windows. Thanks goes to Scott Robbins for his recommendation.

To install noti, follow the directions in the repo’s Readme file. If you elect to curl a copy to your system, use bash, not csh (or in my case tcsh), for copy/pasting to your command line. The utility runs fine in tcsh but its escaping assumes bash is running.

Once installed, just prefix any command line command with noti followed by flags. I prefer using noti -b -s, which gives me both a banner alert and spoken text.

% noti --help
Usage:
  noti [flags] [utility [args...]]

Examples:
noti tar -cjf music.tar.bz2 Music/

Flags:
  -t, --title string     Set notification title. Default is utility name.
  -m, --message string   Set notification message. Default is "Done!".
  -b, --banner           Trigger a banner notification. This is enabled by default.
  -s, --speech           Trigger a speech notification.
  -c, --bearychat        Trigger a BearyChat notification.
  -i, --hipchat          Trigger a HipChat notification.
  -p, --pushbullet       Trigger a Pushbullet notification.
  -o, --pushover         Trigger a Pushover notification.
  -u, --pushsafer        Trigger a Pushsafer notification.
  -l, --simplepush       Trigger a Simplepush notification.
  -k, --slack            Trigger a Slack notification.
  -w, --pwatch int       Monitor a process by PID and trigger a notification when the pid disappears. (default -1)
  -f, --file string      Path to noti.yaml configuration file.
      --verbose          Enable verbose mode.
  -v, --version          Print noti version and exit.
  -h, --help             Print noti help and exit.

Fork Swift and Clone

Starting on Github, log into your account and then fork Apple’s Swift repo. Navigate to github.com/apple/swift, and click fork:

My fork uses my erica github account. Adjust the following directions accordingly so they correspond to your fork.

Hop over to a nice fast drive, and create a development directory. Enter the directory (cd swiftdev, for example) and clone the Swift repo:

git clone https://github.com/erica/swift.git

If you want to get a notification and time how long this took, your command may look like this:

noti -b -s time git clone https://github.com/erica/swift.git

I’ll skip the noti and time references from here on so they don’t clutter up these instructions. I like how time-ing gives you a sense of how long each task takes.

Install Supporting Utilities

Grab copies of the supporting utilities using the built-in update-checkout utility. There’s quite a few supporting folders that need to be built, including llvm, clang, and more. If you’re following these instructions, make sure you’re in the same top level directory as your swift install and not in the swift repo. Otherwise, adjust the path for the update-checkout to drop the initial swift:

% pwd
/Volumes/Kiku/Development
% ls
swift
% ./swift/utils/update-checkout --clone
Skipping clone of 'swift', directory already exists
Running ``obtain_additional_swift_sources`` with up to 8 processes.
Cloning 'compiler-rt'
Cloning 'llvm'
Cloning 'swift-xcode-playground-support'
Cloning 'swift-corelibs-foundation'
Cloning 'clang'
Cloning 'llbuild'
Cloning 'cmark'
Cloning 'lldb'

Now wait for everything to clone. There’s another way to clone using ssh, but I’ve never done it. I just know that it exists and is documented on Apple’s Swift repo Readme file.

Build

You should now be ready to perform a release build. I’m differing here from what’s recommended on Apple’s page because I got help early on from people who didn’t use that method. I’m sticking with my notes because they’re tried and trusted, not because I know what I’m doing.

% cd swift
% noti -s time utils/build-script -R

Building Swift (and, this first time, all its supporting utilities) is not…swift. So go do something else for a significantly long time. Unlike your initial Swift build and when you perform updates by pulling from Apple’s upstream master, incremental changes can be relatively quick. Getting things done the first time and after each update is long and tedious.

Test

If you’ve built things properly, you’ll find a newly build swift compiler waiting for you in the build folder. This directory is located at the top level of your development folder:

% ls

./                       lldb/
../                      llvm/
.DS_Store                ninja/
Hold/                    swift/ [1]
Notes.txt                swift-corelibs-foundation/
build/ [2]               swift-corelibs-libdispatch/
clang/                   swift-corelibs-xctest/
cmark/                   swift-integration-tests/
compiler-rt/             swift-xcode-playground-support/
llbuild/                 swiftpm/

Your cloned swift repo [1] sits along the build folder [2]. Navigate to build > Ninja-ReleaseAssert > swift-macosx-x86_64 > bin to find the swift executable. Check the modification date and run it with the -version flag to confirm it is the right build:

% ls -l build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin/swift 
-rwxr-xr-x  1 ericasadun  staff  96294356 Apr 30 09:23 build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin/swift*

% build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin/swift -version
Swift version 4.2-dev (LLVM bb2a4e8df7, Clang 56384a48b9, Swift 5f9bf115b3)
Target: x86_64-apple-darwin17.5.0

Go ahead and use it to compile a file. Use the path to this build to ensure you don’t accidentally use the default compiler:

% cd ~/Desktop/
% lns ~/github/apple/swift/build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin/swift swift
Created symbolic link at /Users/ericasadun/Desktop/swift
% echo 'print("Hello World")' > test.swift
% ./swift test.swift
Hello World

Conclusion

Getting your initial toolset built and running is the first step on the path to Swift evolution. Next up: ninja builds, creating branches, stashing differences, and managing test builds.

Tap tap, hey is this thing on?

tl;dr: Erica’s site gets hacked repeatedly. Erica’s account is closed by Bluehost. Erica wails into the void. Played with DNS, with github.io, nuked wordpress install, re-installed wordpress, re-installed data, reinstalled plugins, scanned for malware, attempted to restore DNS, wailed into void, some semblance of site possibly restored. Maybe.

postscript: I’m posting this as a test to see if my site is back and alive. If so, please make sure to use https and not http to connect. Fingers crossed.

 

Playing with Strides

Some stride-related discussions took place on the forums today. I was inspired to play with strides as I took a break between my kids’ appointments. If any of this resonates with you as a potential user for it, let me know.

Index Strides

The following extension allows you to stride over a collection’s indices.

extension BidirectionalCollection where Index: Strideable {
    /// Returns the sequence of collection elements (`self[start]`,
    /// `self[start + stride]`, `self[start + 2 * stride]` ... *last*
    /// where *last* is the last element in the collection whose index
    /// is less than or equal to `end`.
    ///
    /// - Note: There is no guarantee that the element at index `end`
    ///   is part of the sequence
    ///
    /// - Parameter start: The starting index
    /// - Parameter end: The upper index bound
    /// - Parameter distance: The stride for each successive index
    /// - Returns: A lazy sequence of collection members
    public func stride(from start: Index, through end: Index, by distance: Index.Stride) -> LazyMapSequence<StrideThrough<Index>, Element> {
        return Swift.stride(from: start, through: end, by: distance)
            .lazy.map({ self[$0] })
    }
    
    /// Returns the sequence of collection elements (`self[start]`,
    /// `self[start + stride]`, `self[start + 2 * stride]` ... *last*
    /// where *last* is the last element in the collection whose index
    /// is strictly less than `end`.
    ///
    /// - Parameter start: The starting index
    /// - Parameter end: The upper index bound
    /// - Parameter distance: The stride for each successive index
    /// - Returns: A lazy sequence of collection members
    public func stride(from start: Index, to end: Index, by distance: Index.Stride) -> LazyMapSequence<StrideTo<Index>, Element> {
        return Swift.stride(from: start, to: end, by: distance)
            .lazy.map({ self[$0] })
    }
}

Here’s an example that uses this approach to form a sequence that strides through the collection:

let myArray = ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"]
for value in myArray.stride(from: 0, through: myArray.count, by: 3) {
    print(value) // a, d, g, j
}

Range Operators

The next snippet uses the division operator to create its stride. This was pitched by “Tony Y” in a forum thread.

extension CountableClosedRange {
    static func / (range: CountableClosedRange, divisor: Bound.Stride) -> StrideThrough<Bound> {
        let distance = range.distance(from: range.startIndex, to: range.endIndex) / divisor // deprecated
        return Swift.stride(from: range.lowerBound, through: range.upperBound, by: distance)
    }
}

In this example, you just “divide” the range:

for idx in (0 ... 100) / 5 {
    print(idx) // 0, 20, 40, 60, 80, 100
    // yes that's 6 not 5, but isn't that what 
    // you want? 5 subranges between each index?
}

Because I wrote this fairly quickly, I used the deprecated range.distance function as straight index arithmetic seemed to error:

error: binary operator '-' cannot be applied to two 'ClosedRangeIndex' operands
        let distance = (range.endIndex - range.startIndex) / divisor

As usual, I probably messed up somewhere (or several somewheres) or you might see ways I could improve the code. Let me know!

Update: After some thought, I decided I preferred a sequence of subranges (which should probably be a type and not like I do it here):

extension CountableClosedRange {
    static func / (range: CountableClosedRange, divisor: Bound.Stride) -> UnfoldSequence<CountableClosedRange<Bound>, (CountableClosedRange<Bound>?, Bool)> {
        let distance = range.distance(from: range.startIndex, to: range.endIndex) / divisor
        let rangeStride = Swift.stride(from: range.lowerBound, through: range.upperBound, by: distance)
        var indices = Array(rangeStride)
        indices.append(range.upperBound)
        indices.reverse()
        guard var current = indices.popLast(), var next = indices.popLast() else {
            fatalError("Guaranteed count failed in index population")
        }
        var done = false
        return sequence(first: current ... next.advanced(by: -1), next: { _ in
            guard !done else { return nil }
            (current, next) = (next, indices.popLast()!)
            if current == next {
                done = true
                return next ... next
            } else if next == range.upperBound && indices.isEmpty  {
                done = true
                return current ... next
            } else {
                return current ... next.advanced(by: -1)
            }
        })
    }
}

for idx in (0 ... 99) / 5 {
    print(idx) // 0...19, 20...39, 40...59, 60...79, 80...99
}

for idx in (0 ... 100) / 5 {
    print(idx) // 0...19, 20...39, 40...59, 60...79, 80...99, 100...100
}

This isn’t a debugged solution. For example, trying to divide 1 … 1 by 1 or 1 … 2 by 1 will both fail.

Forcing Compiler Errors in Swift

Thanks to SE-0196, Swift 4.2 introduces #warning() and #error() compiler directives. These will allow you to incorporate diagnostic messages and emit errors during compilation. Here are some examples from the proposal, which has already been accepted and implemented:

#warning("this is incomplete")

#if MY_BUILD_CONFIG && MY_OTHER_BUILD_CONFIG
  #error("MY_BUILD_CONFIG and MY_OTHER_BUILD_CONFIG cannot both be set")
#endif

The #error example uses conditional compilation flags (set with a -D option) to check whether conflicting configurations have been established for the build.

I’ve already written extensively about my dislike for screaming snake case (THINGS_LIKE_THIS) in Swift. Inevitably, it seems, devs use screaming snake case for their conditional compilation flags, whether MY_BUILD_CONFIGMY_OTHER_BUILD_CONFIG, or DEBUG. Although an industry standard, it feels like a clash with Swift’s aesthetics.

I’ve also written about my proposal for detecting debug conditions without having to supply an explicit DEBUG condition flag, so I’ll also leave that topic to the side for now. You can click on the link for more.

Back to topic, Swift’s newly adopted #error and #warning directives represents a big step up from current practices, which often rely on run-time rather than compile-time feedback.

#if !DEBUG
fatalError("This code is incomplete. Please fix before release.")
#endif

The unindented style in this snippet is now Swift default, avoiding minor doom pyramids for conditional compilation blocks. Even unindented, surrounding code with those conditional blocks is vertically expansive and subjectively ugly. To counter this, some coders have come up with in-line ways to force compilation (not run-time) errors with minimal condition blocks and a more succinct point-of-use approach.

Here’s an example I discovered from John Estropia. (He, in turn, cribbed it from one of  his co-workers.) He uses conditional compilation to set a TODO or FIXME (or whatever) typealias then uses it in-line at points where a debug build should compile and release builds should error:

#if DEBUG 
internal typealias TODO<T> = T
#endif

print("Remove this later") as TODO

It’s clever. Scoping the TODO typealias to debug builds allows lines annotated with as TODO to throw errors during release builds. This ensures compile-time feedback at all points where a TODO cast is performed:

error: ManyWays.playground:5:31: error: use of undeclared type 'TODO'
print("Remove this later") as TODO

It’s not beautiful but it’s effective. It carries information about the call site location and the message you want to emit. If I were applying this hack, I’d probably build an actual todo function rather than using the casting-gotcha. In the following example, I went with an upper camel case name to make the call look more directive-y and less like a standard global function. However, I drew the line at snake case:

#if DEBUG
internal enum IssueLevel {
    case
    mildImportance,
    moderateImportance,
    highImportance,
    criticalImportance
}

internal func ToDo(_ level: IssueLevel, _ string: String) {}
#endif

// The point of use offers a compilation error,
// a note, and a priority level
ToDo(.highImportance, "Remove this later")

// error: ManyWays.playground:13:1: error: use of unresolved identifier 'ToDo'
// ToDo(.highImportance, "Remove this later")

The nicest bit is that toggling from debug compilation to release is completely automatic and centralized to a single #if check.  It’s a fascinating approach if adopted consistently and ensures that all compilation message notes like this must be resolved and removed before release.

Right now, Swift does not support a #message directive, which performs a similar tasks. As many shops treat warnings as errors, they cannot establish a nuanced distinction between the two. If #warning were a thing, you could use #message to issue exactly this kind of “fix me” feedback. A further refinement, #messageOrDie (or something like that, because naming is hard)  could message for debug builds and error for release, going by whether assert statements would or would not fire.

Dave DeLong offers another approach for structural project semantics. His introduces a Fatal type to provide runtime cues for common development outcomes including notImplemented(), unreachable(), and mustOverride(). Nothing says “you need to remember to implement this” better than a spectacular runtime crash that explains itself with full position and function context. Another cue, brilliantly named silenceXcode() allows you to add methods that you never intend to implement and which should error if ever called.

There’s still space in Swift for expanding this metadevelopment support. I wouldn’t mind seeing both approaches added to the language: one for compile time (like #messageOrDie) and another for run time (like Fatal‘s namespaced static error members).

What do you think of these? And what parts of the metadevelopment process (like macros) are still MIA for you in Swift? Let me know. I’m curious to hear what else could be better established to support your development.

 

Bitpocalypse Now

The bitpocalypse is nigh.

If you can afford to say goodbye, now’s the time to use those 32-bit un-installers while they still run.

Apple writes:

With the recent release of macOS High Sierra 10.13.4, the first time users launch an app that does not support 64-bit they will see an alert that the app is not optimized for their Mac.

As a reminder, new apps submitted to the Mac App Store must support 64-bit, and starting June 2018, app updates and existing apps must support 64-bit. If you distribute your apps outside the Mac App Store, we highly recommend distributing 64-bit binaries to make sure your users can continue to run your apps on future versions of macOS.

You can read more about the macOS 64-bit transition on Apple’s dedicated support page.

To find out what applications will be affected, use file from the command line on “/Applications/*.app/Contents/MacOS/*”. This should give you a good idea indicating which apps are about to die on you.

If you have apps in folders (like Adobe and Microsoft do), you may need to perform several sweeps, adjusting the path to include subfolders and bundled utilities.

My copy of QuickTime Player 7, with its bundled QuickTime Pro features, is on the death list:

QuickTime Player 7.app/Contents/MacOS/QuickTime Player 7: Mach-O executable i386

This makes me immensely sad as I regularly use QTPro to perform video edits from trimming and masking to watermarking and generating image sequences. I do not know of any replacement and would love recommendations.

Microsoft Office 2008 will soon be dead as well. It’s a powerful suite of tools, especially Excel, and I will be bitterly missing its functionality. Numbers and Pages just do not compare to the publishing-standard features offered in the hideously ugly but tremendously functional suite. I do not intend to sign up for a yearly subscription. I may do something with Open Office but I’ll probably stick to Apple and hate doing so.

Microsoft Office 2008/Microsoft Word.app/Contents/MacOS/Microsoft Word: Mach-O universal binary with 2 architectures: [ppc:Mach-O executable ppc] [i386]

Adobe Creative Suite 4 will also die, along with it my most comfortable set of photo editing and vector tools:

Adobe Photoshop CS4/Adobe Photoshop CS4.app/Contents/MacOS/Adobe Photoshop CS4: Mach-O universal binary with 2 architectures: [ppc:Mach-O executable ppc] [i386]

Here too, I have no intention of signing up for a yearly subscription. There are many editing alternatives. They all lack the comfort and familiarity of the tools I know. I don’t want to invest either the money or the time to get up to speed on something else. I may buy a copy of Elements and live with how limited it is. I will probably have to give up Illustrator and Acrobat (the good one, not the reader) entirely.

These three sets of tools (QuickTime Pro, Microsoft Office, and Adobe Creative Suite) all represent a significant investment of know-how and power. None of them were beautiful but they are all part of my current toolbox. I’m floundering around trying to figure out which of my systems I can keep installations on, so I can boot in and use them as needed once the transition happens.

For now, I’m going to get High Sierra running on an external drive and port apps I need but will otherwise lose. It’s not a great solution but it may buy me some time.

What apps you’ll be losing affect you the most and how do you plan to deal with them?

Writing Swift: Adventures in Compiler Mods

Ever since Swift adopted the “implement first, propose second” rule, some contributors to the Swift Evolution process have felt sidelined and dropped out of the community. It’s frustrating having ideas for positive language changes and not being able to implement those changes.

Despite expertise in Swift and Objective-C (not to mention any number of other languages), they like me may not be proficient in C++ and Python, the core tools of Swift language implementation. To date, my code contributions to Swift have been extremely low level.

I think I fixed a comment, added a string, and maybe one or two other tiny tweaks. (I did work on the Changelog file a while back but that is written in Common Mark, and does not involve programming in the slightest.)

I’ve wanted to be able to build what I can dream. And I’ve slowly been diving into the compiler in recent months to see what it takes to build something new. With quite a lot of hand holding from John Holdsworth, I implemented a couple of build directives to test whether (1) asserts can fire and (2) a build is optimized.

What is Debug?

Answering “what does ‘debug’ mean?” was a harder question than I initially thought. Coming as I do from Xcode, where ‘debug’ means a scheme I select from within the IDE, it took a bit of thinking and advice to think about ‘debug’ from a platform independent viewpoint. After going back and forth on the Swift Evolution email list and later the forums, the consensus centered on the two tests I mentioned above: assertions and optimization.

For many projects, a typical debug build is unoptimized where asserts can fire. As projects move into the beta process, that mindset changes. Many in-house and beta builds meant for wider use need optimization.

The state of the art uses the custom conditional compilation flags set with -D . This approach decouples the meaning of ‘debug’ (or for most developers 'DEBUG') from anything in-language that can be decoupled from build settings and persist through source code. Assert configurations have their own flag, -assert-config <#value#>.

Introducing these two tests lets you align your not-for-release code to assert states and/or optimization states:

#if !configuration(optimized)
    // code for un-optimized builds only
#endif

#if configuration(assertsWillFire)
    // code where assertions can fire
#endif

The proof-of-concept implementation coupled with a proposal means I may be able to submit a more substantial and meaningful contribution to the language.

Going Solo

Pushing forward, I wanted to test myself and check whether I could make changes on my own, even if that solo journey was quite small. I started by attempting to build the #exit directive that was being discussed on the forums. This turned out to be a little more complicated than I was ready for.

Among other challenges, #exit may or may not be embedded in an #if build configuration test. Using #exit should allow all material up until that point to be compiled into the build while gracefully excluding all material after. I didn’t know how to check whether the directive was embedded in a condition and how to properly complete the condition (#endif) while discarding remaining text. It was, at my stage of the journey, a step too far.

I put my first attempt to the side and tried something else. I tried to push a scope using with(value) { }, so the material within the scope was native to value. That too proved too difficult without assistance although I am beginning to understand how Swift creates and manages its scope. It was a programming failure but a learning success.

Two projects abandoned, I knew I had to pick something very easy to work with. Although I would have loved to have picked up and run with Dave DeLong’s context pitch (which is discussed here) , I recognized that I needed to bite off something smaller first. So I decided to add a #dogcow token that produces the string value `”????????”` in source. How difficult could that be, right?

About five hours and edits to twenty-one files later, I had it working. Kind of. Because I ran into one of the many frequent headdesk situations that plague Swift compiler development. I had focused on my edits to the most recent repo without rebuilding the supporting tool suite.

Ninja Builds

A ninja build is a quick way to build just the compiler component. But at some point you can’t ninja your way into an entire working toolchain. I couldn’t test my changes until I rebuilt everything, a process that can take many many hours on my Mac:

% ./swift ~/Desktop/test.swift
:0: error: module file was created by an older version of the compiler; rebuild 'Swift' and try again: /Volumes/MusicAndData/BookWriting/LiveGithub/Apple/appleswift/build/Ninja-ReleaseAssert/swift-macosx-x86_64/lib/swift/macosx/x86_64/Swift.swiftmodule

Argh.

Building the compiler is not a quick thing. Even a ninja build is a non-trivial investment of time. So if you want to be completely honest, the total coding and build time a lot longer than just five hours. A lot longer.

Make sure you’ve updated your repo, incorporated the latest changes for swift and all of its support tools, and built them all before working on your branches. It will save you a lot of frustration.

Be aware that in the time it takes to create even a small project, you’ll probably be out of date with master. Hopefully this won’t affect your changes, and the files you think you’re patching are still the right files you should be patching.

Designing My Changes

The best way to add anything to the Swift compiler is to find some construct that has already been contributed and look through pull releases to discover what parts of the language they had touched.

That’s how I got started with my assertions/optimization revision. I looked at the recent canImport()pull, and targeted the seven files that involved. In the end, I only  needed to modify four files in total, excluding tests. It was a fairly “clean” and simple update.

To add #dogcow, again excluding tests, I had to change nearly two dozen files, most of them written in C++, a few using Python and Swift’s own gyb (aka “generate your boilerplate”) macros.

I’ve put up a gist that details my notes as I performed file edits. (I did have to make some further changes once I started testing.) Each group consists of a file name followed by my changes, with some context around them to make it easier to return to those parts of the file.

That’s followed by any relevant grep results that encouraged me to edit the file in question plus error messages from the compiler, of which there were a few, as I made several typos and forgot at times to add semicolons to the ends of lines. (Damn you semicolons! shakes fist) I put ### markers near the errors to make them easier to find in the notes.

As you walk through my notes, you’ll notice that I had to create a token (pound_dogcow), which is a kind of MagicIdentifierLiteralExpr expression. By inserting a simple token without arguments and returning a string, I cut down on my need to parse additional components or produce a complicated return value.

(Sorry Dave! I’ll get there I hope… After all, I know where each of the five components of Context live: file, line, column, function, and dsohandle. I just don’t know how to build and export the struct so that it gets put into place and can be consumed by the Swift user.)

As a string, my #dogcow can be used as a literal, so I conformed it to KnownProtocolKind::ExpressibleByStringLiteral. It needed to be serialized and deserialized, emit its custom string, support code completion, and more. Scrolling down my file, you’ll see the scope of notes including searches, comments, and edits for this one change.

Debugging

One of the most interesting things that happened during this exercise was when I made an actual logic error, not a syntax error, so Swift compiled but my program failed:

Assertion failed: (isString() && "Magic identifier literal has non-string encoding"), function setStringEncoding, file /Users/ericasadun/github/Apple/appleswift/swift/include/swift/AST/Expr.h, line 1052.

For the longest time I was convinced (wrongly) that because I was using Unicode, that I had somehow screwed up the string encoding. This was actually a coding mistake, an actual bug, and had nothing to do with the way my string was built nor the fact that I used emojis. It took a while to track down because my head was in the wrong place.

Notice I have DogCow returning true here. I accidentally swapped the two lines so it was originally returning false, falling into the Line/Column/DSOHandle case.

 bool isString() const {
   switch (getKind()) {
    case File:
    case Function:
    case DogCow: // it's a string!
      return true;
    // it should not have been down here
    case Line:
    case Column:
    case DSOHandle:
      return false;
  }
  llvm_unreachable("bad Kind");

Proof-of-Concept

Once compiled, I used a few simple calls to test my work. Here’s the source code I used. I accidentally added an extra space in the assignment test. You can see in the screenshot as well:

// String interpolation and default argument
func hello(_ greetedParty: String = #dogcow) {
    print("Hello \(greetedParty)")
}

hello()
hello("there")

// Use in assignment
let aDogCow = #dogcow
print("The value is ", aDogCow)

// Use directly in statement
print(#dogcow)

Lessons Learned

Having built a working version of the compiler incorporating my solo changes, no matter how trivial and yes it was extremely trivial, has been a big confidence builder. Exploring the process from consuming tokens to emitting intermediate language representations has enlightening.

  • I learned to update everything and build from scratch before starting my work. Because if you don’t, you’ll end up doing it later and wasting that time.
  • I learned how to track down similar modifications and use them as a template for exploring what parts of the compiler each change touched.
  • I learned that some errors would not be in the compilation but in the testing, as one tends to forget things like “just because it built doesn’t mean it will compile correctly” when one is very very focused on getting things to run and extremely new to the process.

I have now worked on two (technically three) compiler modification projects. Each has  taught me something new. If you’d like, take a peek at some explorations I’ve pushed to my forked repo:

The DogCow changes are clean, in the style of something that I might actually do a pull request for. The optimization checks are not. They retain all my little in-line notes I use for searching through text files to find what I’ve changed.

The early debug checks represent the time before I could get all the compiler tools built on my system. I was basically programming in my head at that point, guessing what would work or not, before the conversation on Swift forums moved me to my current design.

My guesswork was wrong. I focused on using a trio of built-in functions (like _isDebugAssertConfiguration) mentioned on-list. This turned out to be a non-viable solution. I needed to follow the example set by canImport to set my flags.

Finally, a word to the wise: Don’t ./utils/build-script -R in one terminal window and ninja swift in another at the same time. Oops.

Today, playing “No. Yes.”

No.

// So I have a [String: [String]]
// Is there a way to do this without the force unwrapping?
if let _ = result[key] {
    result[key]!.append(decodedValue!)
} else {
    result[key] = [decodedValue!]
}

Yes.

if let decodedValue = decodedValue {
    result[key, default: []].append(decodedValue)
}

Commentary

Dictionaries now support default values (SE-0165 and SE-0154), but even if they didn’t, you could still use nil-coalescing without introducing the check on result’s lookup:

// a [String] type for the rhs is inferred by context
result[key] ?? []

The decodedValue should be handled through with either guard-let for early exit or if-let to unwrap then use. Since this kind of code often appears in loops, if may be the better choice, allowing each optional value to be tested and added.

If you are looping, consider applying a flatMap compactMap to your source array or using for case let, enabling you to unwrap and select your input:

for value in decodedValues.compactMap({ $0 }) { 
    result[key, default: []].append(value) 
}

for case let value? in decodedValues { 
    result[key, default: []].append(value) 
}

The two approaches are not significantly different, although I tend to lean towards the latter. It involves a single pass and I think it’s more elegant.

In which I get hacked: Part 5

Last week, it started again. Numerous people on Twitter from various sites around the world reported my site looked like this:

Nothing on my site itself indicated any changes but something was hacking my Bluehost-sourced content and replacing it with its own. I was unfamiliar (as with all things to do with web hosting) with how this was happening, and (spoilers) I never figured that out.

What I did discover, with the help of Jared Lander from Bluehost, was how to mitigate the problem.  (Thanks Jared!) Forcing the site to exclusively use https links through a WordPress plug-in bypassed the whole “evil hackers will redirect my http content” thing. “All” it took was asking everyone to clear caches, restart web browser, or wait for the changes to propagate over time. It’s been 7 days and I have not had any reports of further issues.

The plug-in in question is Really Simple SSL, which automatically configures websites to use secure links: just get an SSL certificate, install the plugin, and activate it. It’s step 1 that’s a burden in many cases.

This cleared up the non-English main page advertising (for those who saw it) and apparently something that affected my RSS feed. I’m not entirely sure what that was, but there had been some stray BOM characters leaking into that which resolved once the SSL/https problem was addressed. Dave Jones wrote me that my feed, which hadn’t been validating, began working properly after the plug-in.

Much as I hope that my issues are over, I’m assured by a number of people contacting me that it’s not possible to run a secure website anymore without the help of professionals.

I’ll do the best I can. I’ll keep backing up my content. But that’s about all I can do.

Thank you again to everyone who reached out to me with support, feedback, and information.

Cleaning up doc comments for formatted commits

I’m working on a proposal to introduce CountedSet, cousin to NSCountedSet, to Swift. This kind of type involves a massive amount of doc comment content. I decided to adapt the comments from Cocoa Foundation (for NSCountedSet) and Swift Foundation (for Set) as part of my coding and quickly found how ridiculous it was to do this by hand.

At first I tried to write an Xcode “reflow doc comments” extension but as I found in the past, Xcode extensions are a dreadful pain to program and debug. It really wasn’t worth doing this (although it would be my preferred workflow for use) in terms of spending my time well.

Instead, I decided to create a simple playground. I’d paste my Swift file into a known Resources file (in this case, test.swift, although I’m sure you can come up with a better name than that if you use this). I’d process the text with a simple playground implementation and print to stdout.

It was an interesting problem to solve and one that took slightly longer than I anticipated. It’s also one that’s only partially complete. The log jams involved looking ahead at the next line to decide when each blob of text was complete so it could be reflowed, preserving paragraph breaks in the comments, respecting code blocks, and leaving any in-file code intact. Reflowing the words was much easier. I’m sure you’ve written that part of it in any number of algorithms and intro-language classes.

The parts I didn’t tackle were the special formatting required for doc comment keywords, like - Parameter, - Returns, - SeeAlso, and so forth. The associated lines for these items must be reflowed with proper indentation so the Quick Help parser can properly parse them. I leave that for another day because they are relatively minor work compared to reflowing long and complex doc comments as a whole.

I’ve put my code up on Github if you want to offer improvements, fixes, or feedback: