Archive for the ‘WWDC’ Category

Dear Erica: Playground Support Folder

“N” asks: “Hey, is the “shared playground folder” long gone, or does it still exist?”

Still there, still useful.

The big difference for long-time playground users is that it moved into the PlaygroundSupport module from the XCPlayground module. The latter was deprecated in Xcode 7. It’s a tiny module that supports playground-specific features. This constant (playgroundSharedDataDirectory) gives you a well-defined sandboxed folder that’s shared between all playgrounds.

This is, by the way, a terrible symbol name (take note!), as it returns a URL. It used to return a string but the name never got updated:

public let playgroundSharedDataDirectory: URL

I often build playground-specific subfolders so my directory doesn’t get all messy.

Another valuable feature is indefinite execution support (needsIndefiniteExecution) for playground pages that have to perform asynchronous work before completion. You can use this support to build little playground-based utilities instead of writing shell scripts.

I have some pages that work with Imgur, Google search, Wolfram queries, etc. A nice thing about building in playgrounds vs shell is that you can integrate audio and visual elements rather than having to save them to files and open them in helper applications.

If you’re writing API utilities, enable manual execution. Constant reloads can almost immediately deplete, for example, your Gist API query count for the day. Oops.

In Xcode, the shared data folder is available for iOS, macOS, and tvOS playgrounds. The shared data folder is not available on iOS’s Swift Playgrounds. This policy discourages custom local storage and access beyond standard media library locations.

There are some further protocols and types under the PlaygroundSupport umbrella in Swift Playgrounds. These aren’t available for Xcode playgrounds because they’re meant for use in Playground Books.

The extra functionality is part of Playground Book support, which underlies the tech in “Learn to Code”, etc. These additional APIs include items like a key-value data store, message passing between the live view and the primary playground page, and more.

If you want to learn more about Playgrounds, I have a book.  It discusses the features you use in Xcode and an overview of how to use iOS Playgrounds. I quite deliberately did not include much about Playground Book authoring as the topics are somewhat orthogonal.

I’ll probably be revising both Playground Secrets and Power Tips and my Swift Documentation Markup after WWDC. There’s also a three-book bundle available with Swift from Two to Three.

Explorations into the Xcode Source Editor Extensions underbelly: Part 1

Xcode source extensions are wildly exciting, surprisingly limited, and infuriatingly frustrating to work with. So I thought I’d share some experiences so you don’t have to suffer through some of my issues.

There are bunches of posts around the web on how to create extensions and I really don’t want to dupe their effort. In a nutshell, create a new Cocoa app. Then create a new macOS > Application Extension > Xcode Source Editor Extension target in that app. You’re ready to code.

If you watched the WWDC video, you’ll have seen the whole process of “run the target in Xcode and a new version of Xcode pops up with a darkened label and then you can test your extension”. I’m here to tell you that this approach is a huge steaming pile of crap doesn’t work as well as I’d hoped. What you really want to do is this:

In AppDelegate.swift of your extension/app project, add the following method. It lets you launch the app, install the latest extension and get the hell out of Dodge without using the “Xcode with the darkened label”.

func applicationDidFinishLaunching(_ aNotification: Notification) {
    let alert = NSAlert(); alert.messageText = "Extension installed! Click OK to quit."
    alert.beginSheetModal(for: window) { _ in
        exit(0)
    }
}

In your main extension Swift file (the class defined for XCSourceEditorExtensionPrincipalClass), add this. It lets you watch the OS X console and ensure that your extensions loaded the way you expected them to. Notice that I use NSLog and not print. Always use NSLog, so your output and debug info goes to the system console.

func extensionDidFinishLaunching() {
    NSLog("NAMEOFMYTARGET extensions did finish launching")
}

Set up your Info.plist in a separate editor pane or even a separate window. If you intend to create a multi-part extension (and most people will), you’ll want to build that extension incrementally and slowly add items to your XCSourceEditorCommandDefinitions dictionary. A typical entry looks like this:

Screen Shot 2016-07-21 at 10.20.38 AM

Also, don’t do what I did here. I used “sadun.ExtensionTestbed” because I know this is a throwaway. Assume you may write multiple extension groups and reverse namespace them properly, e.g. “org.sadun.ExtensionName.CommandName”.

Each time you want to test do this:

  1. Build your target.
  2. Open Products, right-click the .app file and show in Finder.
  3. Quit Xcode and run the app.
  4. Launch Xcode.
  5. Open any project that isn’t a playground because Xcode hates me and doesn’t want to properly load extensions for playgrounds. Always start with a non-playground project and *then* open a playground if you want to work with playgrounds.
  6. If the extension is greyed out (and if you follow these instructions they usually aren’t because this is the sequence for El Capitan that appears to be super reliable), check the OS X console. Look for assertion failures, XPC connection failures, and invalidated connections. If you see these, quit Xcode and relaunch. What you want to see there is the “did finish launching” log line.

I know this workflow is a super pain, especially given Apple’s promotion about in-Xcode testing without quitting and relaunching (let alone running the standalone app), but it’s transformed my source editor extension duty cycle from impossible to annoying. A huge win.

Notes:

  • Don’t “clean” your Xcode extension project while running the extension. Ooops.
  • Once the extension crashes, you need to re-run the installer app.
  • I can’t seem to grab a single cursor position, there has to be an actual selection, so my extensions for adding and removing “/// ” for doc comment markup mean you have to select a line, not just move the cursor there. Or maybe there’s a bug that’s screwing this up. I haven’t quite figured this out yet.
  • A lot of my code is super redundant: operate on all lines, operate on all selection lines, operate on selected text. I really need to re-architect a common subclassable system because the actual logic for most plugins is only a few lines or a call-out to another class.
  • Any extension that works on all lines rather than individual selections seems to be more stable in my experience.
  • My experiments trying to create interactive UI elements with extensions hasn’t been going very well, they run in the original application and not in Xcode, the window is behind xcode, they can’t be interacted with while Xcode is running, and you inevitably have to force quit something.

Screen Shot 2016-07-21 at 10.39.39 AM

Here are some things I’ve been messing with. Let me know how your extensions are going for you.

Playgrounds 6: Focusing Interaction

Screen Shot 2016-06-24 at 9.37.20 AM

In this screenshot, I’ve circled a bit of directed interaction. There’s a highlight around “0.5” and a pop-up editor that allows the user to change the value. This new feature is made possible by Swift Playground markup.

Like all Swift playgrounds, this page uses live code, some of which you see and other parts of which are hidden, either by placement in Sources or by using playground markup. The code is marked up to disable edits except in this one place. Here’s what the key part of the underlying code looks like:

filter.setValue(/*#-editable-code*/1.0/*#-end-editable-code*/, forKey: "inputAmount")
imageView.image = UIImage(ciImage: filter.outputImage!)

Yes, those are in-line comments adding markup in the  setValue call. It’s kind of ugly and it relies on the old-style C /* */ comment delimiters that allow the markup to be inserted in-place.

From a coding viewpoint, it’s ugly as sin. From a user experience viewpoint, it ensures that users can explore exactly one setting to better understand the Core Image vibrancy filter.

This is no way, at least at this time, to use markup to limit values to legal parameters. For example, Core Image may not like negative numbers or numbers above 1.0. You could work around that by using a separate variable and adding clamping code, e.g.

var inputAmount = make this editable
inputAmount = clamp(inputAmount, 0.0, 1.0)
filter.setValue(inputAmount, forKey: "inputAmount")

But if you do that, you’re drawing attention away from the CI code you’re demonstrating and focusing it on user interaction instead. No one’s going to use this pattern in real code, while they may very well use the pattern in the screenshot. (Which, if you think about it is the real point of this.)

I’ve gone ahead and filed a radar (26996963) about the issue, asking for enhanced markup that allows input qualities for editable playground sections.

I probably should file another one asking for “Auto Run” markup, so these kind of examples will re-execute on each edit, just like they do on OS X.

iOS Playgrounds Part 5: Editing real code

While stuck at the Dr’s office, with my daughter being attended to, I decided to try editing a Swift 2.2 playground in iOS Playgrounds, to see if I could upgrade it to Swift 3.0.

The tl;dr is that the tools, while not fully baked, are just as amazing as they were in the WWDC demos. With a few (notable) exceptions, editing source code, invoking fixits, etc. are beautifully realized to the point that I was able to perform non-trivial edits while sitting in a waiting room and using an iPad:

Screen Shot 2016-06-21 at 11.19.54 AM

I was able to fully update this playground page to run and render.

The Good

Built-in fixits are glorious. As in normal Xcode, they update code by inserting missing labels or replacing code:

Screen Shot 2016-06-21 at 11.24.44 AM

Multiple errors are paged (as you can see with the two dots at the top of the complaint), and there’s amazing cross referencing with related items, such as when a protocol is not properly implemented or partially implemented. You just tap away to the error point for the candidate issue.

Number and color edits are fantastic: I didn’t realize you could use the number tweaker to just “dial” your way up and down numbers by continuing to twirl into the number pad:

Screen Shot 2016-06-21 at 11.21.21 AM    Screen Shot 2016-06-21 at 11.21.29 AM

Pages I didn’t expect to work did work! Like this one:  It’s really short. Give it a try. Screen Shot 2016-06-21 at 11.36.42 AM

The Not So Good

You can’t fixit many errors:

Screen Shot 2016-06-21 at 11.30.49 AM

But you can select things and start to type and Playgrounds picks up the context for you:

Screen Shot 2016-06-21 at 11.31.37 AM

The Ugly

Screenshots are for live views, not playground views. And for some reason, I’m having a lot of difficulty today doing screenshots the old way (hold down sleep/wake, then click home). All of these screenshots are from Xcode’s device window instead.

A lot of confusion comes from inappropriate or missing tool tips. Sometimes it leads you down the garden path:

Screen Shot 2016-06-21 at 11.51.25 AM

Screen Shot 2016-06-21 at 11.53.11 AM

What was actually needed looks more like this:

Screen Shot 2016-06-21 at 12.09.44 PM

Trying to edit “advancedBy” to “advanced” by moving the cursor to the end of By and backspacing was simply a no-go. Playgrounds will remove the entire call at once.

You get a lot of problems with 2.2 items needing to be lowercased for 3, or using old CG constructors like CGRectMake not auto-fixing to CGRect(x:y:width:height). Playgrounds just don’t quite get the “fixit” needed for these simple switches.

Screen Shot 2016-06-21 at 12.16.05 PM

Even when you’re typing things in, Playgrounds doesn’t always offer you the right choices. In this example, all this needed was to lowercase CGAffineTransform to cgAffineTransform. No fixit and no syntax match:

Screen Shot 2016-06-21 at 12.12.39 PM

The really cool “drag parenthesis pairs into place” shown in the demos don’t seem to work in the app yet. Instead all I could drag was the letter “j”. (Update: Drag down and left or down and right and then use a second finger to tap or drag the updated key. It is hard but doable. If you’re not coordinated, use two hands. Thanks Seán Labastille)

Screen Shot 2016-06-21 at 11.44.59 AM

You can’t look up module definitions in-place.

No access to Sources that I can find so I can fix errors:

Screen Shot 2016-06-21 at 11.29.05 AM

The live views are distorted:

Screen Shot 2016-06-21 at 12.18.46 PM

Screen Shot 2016-06-21 at 12.19.23 PM

But other than that, wow, what a cool tool. I’m really enjoying it a lot.

Xcode Woes and Noes

Update: I did finally get stuff working after re-installing Xcode about a dozen times. And then it started acting up again. And then a few installs later it worked, etc. etc. Anyway, the gist at the bottom of this post works and I think it’s much more Swifty.

Screen Shot 2016-06-20 at 1.36.29 PM

So I’m watching 414. And I look at that code and I’m all “I could totally improve on that”. So I start, and once again, Xcode 8 blows up. I get the dreaded “Unable to determine compiler to use – the abstract compiler specification is missing from this Xcode installation” error, just like I do every time I try to create a new extension project or do anything particularly stressing to Xcode 8.

Screen Shot 2016-06-20 at 12.44.00 PM

Reinstalling Xcode 8 fixes everything until I attempt to open or create a project with an extension or I look at it wrong, because this issue isn’t limited to extensions.

You’d think this would be a really easy fix — edit a file somewhere or create a link or something — but I can’t figure out what to fix.  It’s particularly annoying because There’s so much in this original code that ticks me off (and no, that’s not my final code at the top of the list, that was just my first go at it, freehand).

Screen Shot 2016-06-20 at 1.40.47 PM

I can’t specify line numbers because the demonstrator didn’t enable them in Xcode (Preferences > Text Editing > Editing > Show > Line numbers). But here are a few key irritations:

  • Using an integer line index instead of enumerating and using key-value
  • Not guarding a conditional cast versus forced casting
  • Creating an array with () instead of a literal (: [Int] = [])
  • Building an Int-ish array instead of a [XCSourceTextRange] one from the start.
  • The newline update thing surely could have been done better.
  • Rule of Kevin violation with “map”.
  • I hate “in” on the same line in closures.
  • The dictionary is ugly. And it’s declared in every run of the loop.

In any case, it’s unlikely I’ll get to play around with this because Xcode hates me. I thought I’d share and hope someone knows how to fix the dependency issue, and also it’s a kind of fun bad Swift snippet to fix, even if you can’t run the code and test it.

Let me know what you’d fix and if you get it running, share some code!

p.s. Update: Still no idea if this will work, but if you try it do let me know how it goes:

Playgrounds Part III

Screen Shot 2016-06-14 at 2.49.47 PM

I have to call it a day, but I’m really happy with what I’ve learned about authoring Playground Books for iOS today. (I’m less happy learning about how to completely reset Xcode — either reboot your system or hop into terminal, do a ps -ax and grep for Xcode and then kill anything with the name in it.) I thought I’d share a few lessons.

If you are going to author for iOS, start with a new playground created from scratch in Xcode 8 rather than modifying an existing one. If you cannot import PlaygroundSupport, reboot your computer. If it doesn’t show up and you’re still using XCPlayground (now deprecated), you’re doing it wrong.

Understand that the Swift snapshot on iOS 10 is older than the Swift snapshot in Xcode so don’t expect to use really recent evolution implementations:

Screen Shot 2016-06-14 at 2.37.05 PM

I ended up implementing my own sequence functions to make my code work.

The easiest way to get started is just to AirDrop a normal playground.

Screen Shot 2016-06-14 at 3.09.40 PM

It will open automagically in Playgrounds for you. When you transfer books, you have to choose “open in Playgrounds” from the AirDrop menu.

If you want to author a Playgrounds Book before tomorrow’s session, just customize a blank document (download at your own risk). My blank book is a one-pager, and you can tweak the Contents.swift file in Blank.playgroundbook/Contents/Chapters/Document1.playgroundchapter/Pages/Blank.playgroundpage. There are manifests at every point along the way, try not to mess them up otherwise the book won’t open.

When authoring pages, either in a normal playground or playground book, you can annotate them to omit details that would otherwise clutter the interaction space or should not be visible to the reader. In my Spirograph playground (download at your own risk), I use special mix of normal playground markup (I have a couple of books that cover this markup format: my Doc Markup book and my Playgrounds book) and these new keywords:

  • Use #-hidden-code and #-end-hidden-code to exclude code from the reader’s view.
  • Mark editable areas with #-editable-code and #-end-editable-code, and if you like you can add a note to the opening tag.
  • To simplify code completion, you can globally omit code completion #-code-completion(everything, hide) and then choose which selectors to present, as I do in the following sample. You see this in the above screenshot, which offers one-tap access to just three customization points for my spirograph.
/*:
 **Spirograph:** Play around.
 
 - Note: The default values are:
 * minor: -3.0
 * offset: 130.0
 * color: .black()
 
 */
//#-hidden-code
import UIKit
import PlaygroundSupport

public var color = UIColor.black()
public var minor = -2.0
public var offset = 130.0

public func setMinor(_ value: Double) { minor = value }
public func setColor(_ value: UIColor) { color = value }
public func setOffset(_ value: Double) { offset = value }
//#-end-hidden-code
//#-code-completion(everything, hide)
//#-code-completion(identifier, show, setColor(_:), setMinor(_:), setOffset(_:))
//#-editable-code Tap to enter code
setMinor(-3.0)
setColor(.black())
setOffset(130.0)
//#-end-editable-code

//#-hidden-code
let liveView = UIImageView(600, 600, backgroundColor: .white())
PlaygroundPage.current.liveView = liveView
liveView.image = spiroImage(
 size: CGSize(width: 600.0, height: 600.0),
 samples: 4096, minor: minor, offset: offset, color: color)
//#-end-hidden-code

Playgrounds Part II

Apple’s pretty highlight page on Swift Playgrounds is here.

Screen Shot 2016-06-14 at 8.09.21 AM

Jonathan Penn figured out how to send documents between my Mac and iPad: just use AirDrop. With that tip, I was easily able to create content on the iPad and move it Macwards and see the material that Apple provided itself.

Here’s how to go from iPad to Mac:

  1. Open AirDrop on your Mac
  2. Tap “My Playgrounds” (4 squares)
  3. Tap Action (top-left, square with arrow pointing up)
  4. Tap the playground you want to send
  5. Tap an AirDrop destination
  6. Wait for the transfer.

iPad Playgrounds use a new format: “Playground Book”:

Screen Shot 2016-06-14 at 8.14.51 AM

It consists of the playground’s contents and user-created edits (stored as a diffpack), which are stored separately. The contents consist of chapters, which in turn have individual playground pages, shared resources and sources. It’s basically the playground we’ve come to know and love on steroids and hallucinogens.

Want to know more about playgrounds? I have a fairly extensive book on playgrounds that’s available right now, which you may be interested in checking out.

The iPad playgrounds app consumes at least two kinds of playground pages: standard pages and cut scenes. These are included as parts of a playground book, which consists of playground chapters, in which the pages and cut scenes are stored. You can read more about the Playground Book Package Format on Apple’s website. (Thanks to Matt Gallagher for the pointer.)

I’m going to warn you in advance that Swift does not play a huge rule in cut scenes. Instead, the cut scenes were constructed using HTML and Javascript. I’ve pasted a fairly short cut scene here, along with a screen shot of some of the resources to create just this one element:

Screen Shot 2016-06-14 at 8.18.42 AM

<!DOCTYPE html>
<html>
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
 <meta http-equiv="X-UA-Compatible" content="chrome=1,IE=edge" />
 <title>simpleCommands</title>
 <style>
 html {
 height:100%;
 }
 body {
 background-color:#AFE0D9;
 margin:0;
 height:100%;
 }
 </style>
 <!-- copy these lines to your document head: -->

 <meta name="viewport" content="user-scalable=yes, width=1024" />

 <script type="text/javascript" src="simpleCommands.hyperesources/StopPoints.js"></script>
 <script type="text/javascript" src="simpleCommands.hyperesources/HypePagingSupport.js"></script>

 <!-- end copy -->
 </head>
 <body>
 <!-- copy these lines to your document: -->

 <div id="simplecommands_hype_container" style="margin:auto;position:relative;width:1024px;height:768px;overflow:hidden;" aria-live="polite">
 <script type="text/javascript" charset="utf-8" src="simpleCommands.hyperesources/simplecommands_hype_generated_script.js?65524"></script>
 </div>
 <!-- end copy -->
 <!-- text content for search engines: -->
 <div style="display:none">
 <div>Simple Commands</div>
 <div></div>
 <div>Your goal is to figure out which instructions, in which order, will result in something great.</div>
 <div>Writing code allows you to create your own set of instructions for your iPad to carry out.</div>
 <div>Notice the mashed-together words? Code is punctuated and spaced like human languages, but commands have no spaces between words.</div>
 <div>collectGem()</div>
 <div>You need to follow the instructions in the correct order, or you'll end up with something…unexpected.</div>
 <div>For example, you'll tell Byte to move forward:</div>
 <div>Commands always end with parentheses. You'll see why later!</div>
 <div>moveForward()</div>
 <div>You’ll start by writing commands to move a character named Byte around a puzzle world, performing tasks.</div>
 <div>Now it's your turn to write some code!</div>
 <div>Have you ever followed a recipe to bake something delicious?</div>
 <div>Or followed instructions to assemble something cool?</div>
 <div>Learn to Code
</div>
 <div>Or collect a gem:</div>
 </div>
 <!-- end text content: -->
 </body>
</html>

Yeah, I know.

Playground pages include separate Contents.swift and LiveView.swift files, which really surprised me.  The Live Views use a poster asset (simple JPG), and there’s a metric ton of Super Sekrit meta-comment annotation for setting up each page:

/*:
 **Goal:** Find the bugs and fix them.
 
 When you write code, it’s easy to make mistakes. A mistake that keeps your program from running correctly is called a [**bug**](glossary://bug), and finding and fixing bugs is called [**debugging**](glossary://debug).
 
 The code below contains one or more bugs. To debug it, rearrange the commands into the right order to solve the puzzle.
 
 1. steps: Run the code to see where the mistake occurs.
 2. Identify the command that's in the wrong place, then tap it to select it.
 3. Drag the command to the correct location, then run the code again to test it.
*/
//#-hidden-code
playgroundPrologue()
//#-end-hidden-code
//#-code-completion(everything, hide)
//#-code-completion(identifier, show, moveForward(), turnLeft(), collectGem(), toggleSwitch())
//#-editable-code Tap to enter code
moveForward()
turnLeft()
moveForward()
moveForward()
collectGem()
moveForward()
toggleSwitch()
//#-end-editable-code
//#-hidden-code
playgroundEpilogue()
//#-end-hidden-code
//#-hidden-code
//
// Contents.swift
//
// Copyright (c) 2016 Apple Inc. All Rights Reserved.
//
//#-end-hidden-code

As you see, you can control which elements are presented to the reader/user/coder, to allow a subset of coding to be accomplished with simple taps. It’s a pretty limited approach for end-user consumption.

The entire system is run using a new PlaygroundSupport library, which appears to be iPad only for the moment but will replace XCPlayground, presumably soon in Xcode 8 (despite this screen shot).

Screen Shot 2016-06-14 at 9.20.59 AM

According to docs, the PlaygroundSupport module enables you to access a playground page and manage execution, display and dismiss live views, and share and access persistent data, which you use for creating a consistent experience throughout the interaction with your playground book.

A quick check shows that you can refer to: PlaygroundPage (for page state control), PlaygroundValue, PlaygroundQuickLook (should act like the current version), PlaygroundLiveViewable (protocol for live-view suitable types), PlaygroundKeyValueStore (for persistent data storage), PlaygroundRemoteLiveViewProxy,  PlaygroundLiveViewMessageHandler, PlaygroundLiveViewRepresentation  (type enum for items that can act as live views), PlaygroundRemoteLiveViewProxyDelegate.

Carl Brown tracked down some of this in the “Answers” example playground, which uses custom show() and ask() functions built around these new features.

For example, the AnswersLiveViewClient class works as a proxy delegate for interactions in the live view. It’s a subclass of the Apple-supplied PlaygroundRemoteLiveViewProxyDelegate class that acts as a PlaygroundLiveViewMessageHandler for receiving and processing live view requests.

As far as I can tell, there are not yet any tools on the Mac side that build these pieces. So don’t go looking for iPlaygroundAuthor or even an iPlaygroundBookstore quite yet.

Screen Shot 2016-06-14 at 8.35.34 AM

So what is this entire app good for then? Other than some goodwill Apple outreach? Well, it’s cool having interpretable Swift on your iPad (take that original App Store restrictive rules!) and it’s clearly meant for an audience that accepts limited tap-based interaction, with a lot of beautiful highly designed iBooks-Author-esque HTML support.

Without tools to create content (yet) or a storefront to distribute and potentially monetize, it feels like a proof of concept more than an app. We’ll see how things (if you pardon the pun) develop.

iOS Playgrounds

Screen Shot 2016-06-13 at 4.59.35 PM

Really early days:

  • App is baked into iOS 10. Just install a beta to your favorite (newerish) iPad.
  • Houdah’s Type2Phone is the best for working with iOS playgrounds. Really great for copy/pasting.
  • Some (but not all) Emacs keybindings work, which makes editing much easier. However, ^A/^E don’t respect the onscreen folding, so in squish mode, they’ll jump to the start and end of each \n-delineated line, not the folded version.
  • I can’t figure out how to get things onto and off of the iPad: tried emailing playgrounds, playground pages, tried going through iTunes, looked in Xcode, even used PhoneView (Ecamm) and iBrowser (my little hack app) to try to track this stuff down. No luck so far.
  • XCPlayground seems to be PlaygroundSupport — took me forever to figure that out because it’s still XCPlayground on the desktop. And of course, all the new ObjC name import stuff isn’t working as expecting, so you do have to hunt around. Live views are PlaygroundPage.current.liveView assignments.
  • To see both code and live view (or just live view), slide from the right.

Lots still to explore!

 

WWDC Thoughts

Screen Shot 2016-06-13 at 12.42.33 PM

This year, Apple offered a particularly subdued WWDC keynote introducing refreshes along all its product lines with few surprises or innovations. While the audience was politely enthusiastic, I’m not sure the tech press will be wowed.

I love the iOS playgrounds, although I think this is a better opportunity for institutions (say, Stanford) than little tutorial writers, and I got Sherlocked. Xcode introduces source editor extensions and there’s a Siri API using extensions.

Screen Shot 2016-06-13 at 1.06.26 PM

The download and upgrade process for Xcode, OS X (I mean macOS), and iOS has been amazing — just a dream process compared to previous years. Everything is operating smoothly from updating the conference schedule to providing bandwidth for desperate developer downloads. Fingers crossed this continues as well as it has started. I’m amazed at how clean the download site is, how simple the configuration profile approach is working (“Can I temporarily remove some large apps for space and re-install them later?” “Why YES!”). Big applause to whoever made these improvements.

Looking forward to the state of the union later today.

Swift:

Tuesday

  • What’s new in Swift: Tues 11-11:40
  • Getting Started with Swift: Tues 2:40 – 3:40
  • What’s new in Foundation for Swift: Tues 5 – 5:40
  • What’s new in LLVM: Tues 6-6:40

Wednesday

  • Introducing Swift Playgrounds: Weds 12-1
  • Modern Best Practices: Weds 4-4:40
  • Swiftogether: Weds 7:15-8:45

Thursday

  • Using and Extending Xcode’s Source Editor: Thus 6-6:40

Friday

  • Server-side with Swift Open Source: Fri 10-10:40
  • Swift Performance: Fri 12-1
  • Debugging Tips & Tricks: Fri: 2:40 – 3:40
  • Concurrent GCD w/ Swift: Fri 5-5:40
  • Protocol and Value Oriented Programing (in UIKit Apps): 5-5:40

Not Swift:

  • What’s new in Cocoa: Tues 12-1
  • What’s new in Cocoa Touch: Tues 2:40-3:40
  • Typography and Fonts: Weds 10-10:40
  • Speech Recognition API: Weds 5-5 (!?)
  • Intro to SiriKit: Weds 6-7
  • Extending Apps with SiriKit: Thus 2:40-3:40
  • What’s new in GameplayKit: Thus 10 – 10:40
  • Making Apps Adaptive: Thus 12-12:40, Fri 10-10:40
  • Proactive Suggestions: Fri: 2:40-3:40
  • Measurements and Units: Fri 5-5:40

Also: Some nice internationalization and accessibility sessions, a couple of worthy “adopting Metal” sessions, CloudKit best practices

iOS 10 Compatible Devices

I think this is probably the most reliable screenshot:

Screen Shot 2016-06-13 at 1.43.05 PM

There’s also this:

Screen Shot 2016-06-13 at 12.46.07 PM

and this:

Screen Shot 2016-06-13 at 1.46.51 PM

macOS Sierra Compatible Devices

Screen Shot 2016-06-13 at 1.59.25 PM

 

Nostalgia Tuesday: By request, my 2012 Siri Post

Well, if anything does happen on Monday, we can play “How badly did she get it wrong“, right? And to add some icing, here’s a what-if post about Siri controlling your Apple TV and a proof-of-concept Siri-style dictation used in-app.

(There’s a comment on the video that I particularly love: “Scammer watch the mouse across the screen at the end.” I hate to destroy the tinfoil but I was feeding the Apple TV output through EyeTV and recording the output on my Mac. Bless that person’s conspiratorial heart.)

How 3rd Party apps might integrate with Siri

Third-party integration into Siri remains at the top of many of our TUAW wish lists. Imagine being able to say “Play something from Queen on Spotify” or even “I want to hear a local police scanner.” And Siri replying, “OK, you have two apps that have local police scanners. Do you want ScannerPro or Wunder Radio?”

So why doesn’t Siri do that?

Well, first of all, there are no third party APIs. Second, it’s a challenging problem to implement. And third, it could open Siri to a lot of potential exploitation (think of an app that opens every time you say “Wake me up tomorrow at 7:00 AM” instead of deferring to the built-in timer).

That’s why we sat down and brainstormed how Apple might accomplish all of this safely using technologies already in-use. What follows is our thought experiment of how Apple could add these APIs into the iOS ecosystem and really allow Siri to explode with possibility.

Ace Object Schema. For anyone who thinks I just sneezed while typing that, please let me explain. Ace objects are the assistant command requests used by the underlying iOS frameworks to represent user utterances and their semantic meanings. They offer a context for describing what users have said and what the OS needs to do in response.

The APIs for these are private, but they seem to consist of property dictionaries, similar to property lists used throughout all of Apple’s OS X and iOS systems. It wouldn’t be hard to declare support for Ace Object commands in an application Info.plist property lists, just as developers now specify what kinds of file types they open and what kind of URL addresses they respond to.

Practicality. If you think of Siri support as a kind of extended URL scheme with a larger vocabulary and some grammatical elements, developers could tie into standard command structures (with full strings files localizations of course, for international deployment).

Leaving the request style of these commands to Apple would limit the kinds of requests initially rolled out to devs but it would maintain the highly flexible way Siri users can communicate with the technology.

There’s no reason for devs to have to think of a hundred ways to say “Please play” and “I want to hear”. Let Apple handle that — just as it handled the initial multitasking rollout with a limited task set — and let devs tie onto it, with the understanding that these items will grow over time and that devs could eventually supply specific localized phonemes that are critical to their tasks.

Handling. Each kind of command would be delineated by reverse domain notation, e.g. com.spotify.client.play-request. When matched to a user utterance, iOS could then launch the app and include the Ace dictionary as a standard payload. Developers are already well-acquainted in responding to external launches through local and remote notifications, through URL requests, through “Open file in” events, and more. Building onto these lets Siri activations use the same APIs and approaches that devs already handle.

Security. I’d imagine that Apple should treat Siri enhancement requests from apps the same way it currently works with in-app purchases. Developers would submit individual requests for each identified command (again, e.g. com.spotify.client.play-request) along with a description of the feature, the Siri specifications — XML or plist, and so forth. The commands could then be tested directly by review team members or be automated for compliance checks.

In-App Use. What all of this adds up to is an iterative way to grow third party involvement into the standard Siri voice assistant using current technologies. But that’s not the end of the story. The suggestions you just read through leave a big hole in the Siri/Dictation story: in-app use of the technology.

For that, hopefully Apple will allow more flexible tie-ins to dictation features outside of the standard keyboard, with app-specific parsing of any results. Imagine a button with the Siri microphone that developers could add directly, no keyboard involved.

I presented a simple dictation-only demonstration of those possibilities late last year. To do so, I had to hack my way into the commands that started and stopped dictation. It would be incredibly easy for Apple to expand that kind of interaction option so that spoken in-app commands were not limited to text-field and text-view entry, but could be used in place of touch driven interaction as well.