Last June, Izzy inspired me to do something with Bluetooth and playgrounds but honestly, I haven’t had the time and I couldn’t afford a Sphereo. I’ve wrapped up Swift Style. Attempting to write meaningfully about drawing while the Denver Public School system has for reasons I cannot begin to comprehend released my child to my recognizance for two entire weeksseems unlikely. (Another child has half days. Fun.)
To prepare, I purchased one of the cheapest BLE devices I could find, a Mi wristband (Amazon cost under $20 shipped), which has a reverse engineered API that lets you control vibration. A friend of mine just purchased the hugely expensive Buzzies for Autism bands. I’m hoping I can mimic some of that functionality with a playground, a low-rent BLE device, and a full-price child.
Have I mentioned recently how awesome playgrounds are for playing around with and learning about new tech? They really are, especially because you can integrate just one concept at a time, and then test it live before expanding to the next.
I decided to go with Cocoa for my BLE exploration instead of iOS, although the tech is more or less the same on both platforms. When you work in Cocoa, using a macOS playground, the startup speed is phenomenal because you don’t have to work with a simulator.
My first project simply sets up a central manager (CBCentralManager), monitors its state, and lists any devices it finds. I’m pretty happy for this as a first day, not many hours to spend on it, playing around and doing something marginally useful result.
The CoreBluetooth documentation is pretty dire. For example, this is the Swift docs for CBManagerStatePoweredOn. After SE-0005, the constant is actually .poweredOn, as you see in the following sample code, not CBManagerStatePoweredOn. And there’s no documentation in that documentation.
Nonetheless, I persevered and my first child-full day produced a basic helper class. You really need to work in NSObject land for this because of all the delegation. So I set up an objc-friendly class, set it as a manager delegate, and implemented the one required callback method, which follows the manger state.
Try sticking the Bluetooth icon in your system menu bar. (System Preferences > Bluetooth > Show Bluetooth in menu bar.) It’s a lot of fun to toggle it on and off and watch your playground keep tabs on that state.
Next, I added a basic peripheral scan. You need to scan only when the manager achieves poweredOn state.
Apple writes, “Before you call CBCentralManager methods, the state of the central manager object must be powered on, as indicated by the CBCentralManagerStatePoweredOn constant. This state indicates that the central device (your iPhone or iPad, for instance) supports Bluetooth low energy and that Bluetooth is on and available to use.”
That’s why I added the scan to the playground’s “update state” callback. You’ll want to stop scans when the BLE powers off.
Finally, I implemented one more callback, which asynchronously lists discovered peripherals. It picked up nicely on my Apple TV and when I enabled and disabled a hotspot on my iPhone. Great fun.
Here’s the code involved. You can see how very short it is. The struggle wasn’t in the lines of code or complexity, it’s mostly about how very badly documented most everything seems to be.
Anki has been kind enough to let me play with their new Cozmo unit and explore their SDK. Cozmo is a wonderful device, developed by people who understand a lot of core principles about human interaction and engagement.
Cozmo is adorable. When it recognizes your face, it wriggles with happiness. It explores its environment. When it’s bored, it sets up a game to play with you. It can get “upset” and demand attention. It’s one of the most personable and delightful robots I’ve played with.
At its heart is a well-chosen collection of minimal elements. The unit can move around the room, with a 4-wheel/2-tread system. It includes an onboard forklift that can rise and fall, an OLED “face” that expresses emotion, and a camera system that ties into a computer vision system, which I believe is based on PIL, the Python Image Library. (Anki tells me that Cozmo’s vision system “does not use PIL or Python in any way, though the Python SDK interface uses PIL for decoding jpegs, drawing animations, etc.”)
Three lightweight blocks with easily-identified markings complete the Cozmo package, which Cozmo can tap, lift, stack, and roll.
Between its remarkable cuteness and its vision-based API, it’s a perfect system for introducing kids to programming. I was really excited to jump into the SDK and see how far I could push it.
Here is Anki’s “Hello World” code (more or less, I’ve tweaked it a little) from their first developer tutorial:
Make Cozmo say 'Hello Human' in this simple
Cozmo SDK example program.
robot = sdk_conn.wait_for_robot()
if __name__ == '__main__':
except cozmo.ConnectionError as err:
sys.exit("Connection error 😬: %s" % err)
Although simple, this “Hello World” includes quite a lot of implementation details that can scare off young learners. For comparison, here’s the start of Apple’s tutorial on Swift “Learn to Code”:
There’s such a huge difference here. In Apple’s case, everything that Byte (the main character) does is limited to easy-to-understand, simple calls. The entire implementation is abstracted away, and all that’s left are instructions and very directed calls, which the student can put together, re-order, and explore with immediate feedback.
In Anki’s code, you’re presented with material that’s dealing with set-up, exceptions, asynchronous calls, and more. That is a huge amount of information to put in front of a learner, and to then say “ignore all of this”. Cozmo is underserved by this approach. Real life robots are always going to be a lot more fun to work with than on-screen animations. Cozmo deserved as simple a vocabulary as Byte. That difference set me on the road to create a proof of concept.
In this effort, I’ve tried to develop a more engaging system of interaction that better mirrors the way kids learn. By creating high level abstractions, I wanted to support the same kind of learning as “Learn to Code”. Learn to Code begins with procedural calls, and then conditional ones, and moving on to iteration and functional abstraction, and so forth.
My yardstick of success has been, “can my son use these building blocks to express goals and master basic procedural and conditional code?” (I haven’t gotten him up to iteration yet.) So far, so good, actually. Here is what my updated “Hello World” looks like for Cozmo, after creating a more structured entry into robot control functionality:
from Cozmo import *
# run, cozmo, run
'''Specify actions for cozmo to run.'''
# Fetch robot
coz = Cozmo.robot(cozmoLink)
# Say something
Not quite as clean as “Learn to Code” but I think it’s a vast improvement on the original. Calls now go through a central Cozmo class. I’ve chunked together common behavior and I’ve abstracted away most implementation details, which are not of immediate interest to a student learner.
Although I haven’t had the time to really take this as far as I want, my Cozmo system can now talk, drive, turn, and engage (a little) with light cubes. What follows is a slightly more involved example. Cozmo runs several actions in sequence, and then conditionally responds to an interaction:
from Cozmo import *
from Colors import *
# Run, Cozmo, run
'''Specify actions for cozmo to run.'''
# Fetch robot
coz = Cozmo.robot(cozmoLink)
# Say something
# Drive a little
coz.drive(time = 3, direction = Direction.forward)
coz.turn(degrees = 180)
# Drive a little more
coz.drive(time = 3, direction = Direction.forward)
# Light up a cube
cube = coz.cube(0)
# Tap it!
coz.say("You tapped it")
coz.say("Why no tap?")
And here is a video showing Cozmo executing this code:
If you’d like to explore this a little further:
Here is a video showing the SDK feedback during that execution. You can see how the commands translate to base Cozmo directives.
I’ve left a bit of source code over at GitHub if you have a Cozmo or are just interested in my approach.
As you might expect, creating a usable student-focused learning system is time consuming and exhausting. On top of providing controlled functionality, what’s missing here is a lesson plan and a list of skills to master framed into “Let’s learn Python with Cozmo”. What’s here is just a sense of how that functionality might look when directed into more manageable chunks.
Given my time frame, I’ve focused more on “can this device be made student friendly” than producing an actual product. I believe my proof of concept shows that the right kind of engagement can support this kind of learning with this real-world robot.
The thing that appeals most to me about Cozmo from the start has been its rich computer vision capabilities. What I haven’t had a chance to really touch on yet is its high level features like “search for a cube”, “lift it and place it on another cube”, all of which are provided as building blocks in its existing API, and all of which are terrific touch points for a lesson plan.
I can easily see where I’d want to develop some new games with the robot, like lowering reaction time (it gets really hard under about three quarters of a second to tap that darn cube) and creating cube-to-cube sequences of light. I’d also love to discover whether I can extend detection to some leftovers my son brought home from our library’s 3D printer reject bin.
Cozmo does not offer a voice input SDK. It’s only real way to interact is through its cameras (and vision system) and through taps on its cubes. Even so, there’s a pretty rich basis to craft new ways to interact.
As for Anki’s built-ins, they’re quite rich. Cozmo can flip cubes, pull wheelies, and interact in a respectably rich range of physical and (via its face screen) emotional ways.
Even if you’re not programming the system, it’s a delightful toy. Add in the SDK though, and there’s a fantastic basis for learning.
I recently reached out to Anki, creators of the Cozmo robot, to ask if I could explore the device from a developer’s perspective. Shipping with a Python SDK, Cozmo offers some surprisingly sophisticated image processing and recognition features, analogous to Apple’s Core Image.
Before jumping into the programming side of things, let me acknowledge that I am primarily an Apple developer. Therefore I must categorically kick off by evaluating package design.
I didn’t get to start unboxing last night. When my unit arrived during a cold snap, the package window was all fogged up. I opened the top, added a desiccant unit, and let the package dry out and warm out overnight.
This morning, I finally could explore. As you’d expect with high end consumer goods (Cozmo retails for $180), the box used satisfyingly thick cardboard, was easy to open, and presented the product nicely while hiding the user manual, power cord, and accessories.
Unexpectedly, my favorite part of the entire boxing system was Cozmo’s perch. Made of quality yellow plastic, this industrial presentation feature is practically a toy in itself.
Flip it over, and instead of expected wire wraps, there’s an ingenious system to release the Cozmo unit. Pull up the yellow spacer and pinch the two white tabs. It all comes apart, allowing you to remove Cozmo, and start him charging.
Then while he charges, you can spend time putting the pieces back together and taking them apart over and over. It’s practically an extra free toy that ships with the robot. Beautiful design and completely unexpected.
The full complement of parts include the Cozmo robot, a USB charging dock with a separate a wall-adapter, a trio of play blocks (“Interactive Power Cubes”), and a welcome packet.
The instructions say to place him on an open table in a well lit room, with room to move around. Charging from empty to full is specified at 10-12 minutes, with a rated play time of 1-2 hours. (I wouldn’t be surprised if that number drops once you start putting extra demands on his processors.)
That 10-12 minutes subjectively lasts approximately 3-4 months after you first finish unboxing. Child and I got into a heated debate as to whether we’d name him after Cosimo de’ Medici, founder of the Medici political dynasty, or Cosmog, nebula Pokemon and opener of “Ultra Wormholes”.
I don’t want to spoil the instruction booklet for you so let me just say, the writeup is adorable, clever, and simple. The consumer warnings in particular made me laugh out loud. It’s a great example of technical writing and communication, focusing on simplicity and clarity. It’s really well done.
You must install a separate Cozmo application you on an iOS or Android device. It just shipped a 1.1.0 update, so make sure you’re running the latest version. It helps if you watch this video before trying to set-up.
Open Settings and connect to the Cozmo WiFi network. Lift Cozmo’s front arm to display the password, and then use Settings to log in, making sure to type everything using exact upper casing. The password is random mix of upper case letters and numbers, and the iPhone’s keypad doesn’t remember casing when moving between number input and text input.
Anki recommends typing the password into your Notes app, and I endorse this suggestion, especially when I point out how many times I tried to get it to connect. That’s because at first I didn’t realize Cloak VPN was trying to secure the Cozmo WiFi connection. It really really helps if you set it up to trust the Cozmo network. This one detail put a huge crimp into my set-up, causing immeasurable pain and frustration. I finally found this support write-up, which mentions VPN issues at the very bottom.
I haven’t spent much time playing with Cozmo yet but speaking as an iOS dev, there are quite a few things Anki could do to tighten up their app. There’s great content and some terrific games but the app reads as “cross platform with compromise”. (It seems to have Unity under its cover.)
To give just a couple of examples, the cursor controls in a custom text field don’t follow iOS standards. When you use cursor arrows during typing, the active cursor position does not update. That’s iOS 101.
The app would benefit from a HIG/GUI once-over for usability but that’s really the only weak spot in the big package that I’ve encountered.
As for Cozmo himself, he’s a delight. As part of set-up, he learned my face and that of my daughter. Who cannot love a robot that wakes up from charging, sees your face, recognizes you, greets you by name and then wiggles with happiness?!? It’s phenomenally adorable.
Of course, my interest lies primarily in the SDK, and that exploration will have to wait for another write-up. For now, a summary to date:
Great starter games to inspire development possibilities.
There’s an app.
Cozmo is right now exploring the floor of my office, making random offhand comments in robotese about what he’s finding and generally having a ball. I think I’ll stop writing for a short while and join him in having fun.
How do you get your kid to try on all his shirts and pants before school starts? So you can sort them into trash, donate, wear, and “for playtime use only”? Answer: Playgrounds. Thank you playground team!
We used the “deal or no deal” rule. Two spins. You can “stick” or you can re-try, but no peeking at the next roll. Payment for each garment after trying it on, taking it off and folding it. Coins placed into a cauldron (thank you Harry Potter camp) to be counted after we’re done. Any stained items treated with Oxyclean and thrown into the hamper.
For less than a couple of dollars, we got through his entire wardrobe with no complaints except “Don’t I have anything else that needs trying on?”
I never use my Downloads folder. It’s a fusion drive, so it’s precious, fast, and expensive. I don’t need a thousand downloaded copies of Xcode and firmware updates littering its limited space. Instead, I point all my browsers and other apps to download to my secondary data disk.
And before you ask, I use numerical and alphabetic prefixes so everything shows up in the right place and the right order for quick reference and single-letter typing access. Whatever data I can offload from my main drive, I do offload:
However, when it comes to airdropping, it’s generally true that whatever I’m sending back and forth is of immediate interest. In such case, I don’t want it heading into my Downloads folder. I want it on my desktop as soon as it lands. As I’m updating my Playgrounds Book right now, I’m doing a lot more airdropping than I normally would.
I’m not a big user of smart folders and Automator actions. I have a smallish bunch that I occasionally use. Still, they have their place and today was a perfect occasion to bring a new one into the mix.
I just had had it with the Downloads folder and decided to build a bouncer that would automatically throw any item added to ~/Downloads up to the desktop. I thought I’d share how to do this.
Step 1. Create a new Folder Action
Step 2. Choose the Downloads folder.
Step 3. Drag “Move Finder Items” onto “Drag actions or files here to build your workflow”
This creates the following action, with Desktop selected by default. (If it’s not, choose Desktop for the destination.)
Step 4. Then save:
Your new automator action is stored in ~/Library/Workflows/Applications/Folder\ Actions:
Step 5. Test. Drop a file into Downloads and confirm that it moves to the desktop. You should now be ready to airdrop to your desktop.
Note: I’m sure there’s a better way to do this, but I actually wrote an app that quickly opens AirDrop windows on the Mac side of things. I found an appropriate AppleScript online, compiled it to an app, and use Spotlight to launch it. Very handy when I’m more focused on iOS than OS X at the moment.
(There’s a comment on the video that I particularly love: “Scammer watch the mouse across the screen at the end.” I hate to destroy the tinfoil but I was feeding the Apple TV output through EyeTV and recording the output on my Mac. Bless that person’s conspiratorial heart.)
Third-party integration into Siri remains at the top of many of our TUAW wish lists. Imagine being able to say “Play something from Queen on Spotify” or even “I want to hear a local police scanner.” And Siri replying, “OK, you have two apps that have local police scanners. Do you want ScannerPro or Wunder Radio?”
So why doesn’t Siri do that?
Well, first of all, there are no third party APIs. Second, it’s a challenging problem to implement. And third, it could open Siri to a lot of potential exploitation (think of an app that opens every time you say “Wake me up tomorrow at 7:00 AM” instead of deferring to the built-in timer).
That’s why we sat down and brainstormed how Apple might accomplish all of this safely using technologies already in-use. What follows is our thought experiment of how Apple could add these APIs into the iOS ecosystem and really allow Siri to explode with possibility.
Ace Object Schema. For anyone who thinks I just sneezed while typing that, please let me explain. Ace objects are the assistant command requests used by the underlying iOS frameworks to represent user utterances and their semantic meanings. They offer a context for describing what users have said and what the OS needs to do in response.
The APIs for these are private, but they seem to consist of property dictionaries, similar to property lists used throughout all of Apple’s OS X and iOS systems. It wouldn’t be hard to declare support for Ace Object commands in an application Info.plist property lists, just as developers now specify what kinds of file types they open and what kind of URL addresses they respond to.
Practicality. If you think of Siri support as a kind of extended URL scheme with a larger vocabulary and some grammatical elements, developers could tie into standard command structures (with full strings files localizations of course, for international deployment).
Leaving the request style of these commands to Apple would limit the kinds of requests initially rolled out to devs but it would maintain the highly flexible way Siri users can communicate with the technology.
There’s no reason for devs to have to think of a hundred ways to say “Please play” and “I want to hear”. Let Apple handle that — just as it handled the initial multitasking rollout with a limited task set — and let devs tie onto it, with the understanding that these items will grow over time and that devs could eventually supply specific localized phonemes that are critical to their tasks.
Handling. Each kind of command would be delineated by reverse domain notation, e.g. com.spotify.client.play-request. When matched to a user utterance, iOS could then launch the app and include the Ace dictionary as a standard payload. Developers are already well-acquainted in responding to external launches through local and remote notifications, through URL requests, through “Open file in” events, and more. Building onto these lets Siri activations use the same APIs and approaches that devs already handle.
Security. I’d imagine that Apple should treat Siri enhancement requests from apps the same way it currently works with in-app purchases. Developers would submit individual requests for each identified command (again, e.g. com.spotify.client.play-request) along with a description of the feature, the Siri specifications — XML or plist, and so forth. The commands could then be tested directly by review team members or be automated for compliance checks.
In-App Use. What all of this adds up to is an iterative way to grow third party involvement into the standard Siri voice assistant using current technologies. But that’s not the end of the story. The suggestions you just read through leave a big hole in the Siri/Dictation story: in-app use of the technology.
For that, hopefully Apple will allow more flexible tie-ins to dictation features outside of the standard keyboard, with app-specific parsing of any results. Imagine a button with the Siri microphone that developers could add directly, no keyboard involved.
I presented a simple dictation-only demonstration of those possibilities late last year. To do so, I had to hack my way into the commands that started and stopped dictation. It would be incredibly easy for Apple to expand that kind of interaction option so that spoken in-app commands were not limited to text-field and text-view entry, but could be used in place of touch driven interaction as well.
In my most recent post about installing Ubuntu/Swift, I glancingly referred in a screenshot to pulling swift source from a github gist. I thought it was useful enough a tip to pull out to its own post.
Say you have a gist, for example: this one, which is located at gist.github.com/erica/4d31fed94f3668342623. I threw this sample together to help some Linux-ers work with incomplete Foundation and Standard Library implementations on Ubuntu.
If you grab this link’s content from the terminal command-line using curl, you’ll end up with the page’s HTML source, which is pretty much not what you want to compile:
Instead, replace gist.github.com with gist.githubusercontent.com and append /raw to the url:
This morning, I finally broke down and installed Ubuntu 15.10 into Virtual Box. I didn’t take notes, which I am now kicking myself for because it didn’t occur to me until after I had gone through the entire annoyingly tweaky process.
Here are the highlights.
First, I installed Virtual Box. I mostly went with the directions in this Simple Help write-up. The directions are 15.04, but they worked fine for 15.10. Per the write-up, I went with all the defaults. It took an hour or more for the Ubuntu install to get itself set-up. Prepare to head out for a few errands.
Next, I spent time familiarizing myself with Ubuntu, finding the terminal, chsh’ing to tcsh from bash (I had to apt-get tcsh) and setting up my .cshrc. I also installed lns (bless it) and downloaded the latest 15.10 Swift 2.2 snapshot from swift.org.
Then I symbolic linked swift and swiftc into my ~/bin folder, checked that I could run them both from wherever, and then set to working on getting ssh running so I didn’t have to use Virtual Box directly.
To do that, first I had to apt-get install OpenSSH-server, which was no big deal, and then what took the most time was figuring out how to set up networking. Turns out that you need to shut down the machine, set up Adapter 1 to use Bridged Adapter (thanks aciidb) and the find the 10.0.whatever-ip-address using if-config after rebooting. (Mine was 10.0.0.35).
Once I got past that, I was able to ssh in over from my Mac and start using Swift. I began with a simple shell script (thought that was easier) and then moved onto swift and then swiftc compilation.
As Mike Ash pointed out, when encouraging me to use ssh with Virtual Box, you gain all the utility of your normal terminal session including copy, paste, etc without having to deal with mouse capture, etc.
This is my Jeep stroller. It is probably the best stroller ever. It’s also the best shopping cart ever. It has cupholders. It has not just a pop-up glove compartment at the top but extra cargo bags on the left and right for small items.
The basket underneath is enormous. It carries tons of weight. The wheels are rugged and have been able to navigate through just about any terrain, including going to the local market through Colorado winters.
I can load a couple of 12-packs of sodas, carry about 6-8 other bags of groceries. There’s room above, there’s room below, there’s easy-to-tie-to-handles. When I’m at the store, I can stick a basket in it, fill it, surround it, and put coats, gloves, hats, etc below (or use that area for more groceries). It is, in short, the ultimate shopping machine.
Only one problem. My baby is now a few years away from starting to grow facial hair. Try to use this thing and you get side eye, hairy eyeball, every kind of “are you some kind of sick bag-lady with an old shopping cart stroller” look you can imagine.
There’s some kind of unspoken consensus that after our kids have grown, we must transition to granny carts. I used to own a granny cart when I was in college. It was fantastic. (4-wheel variety because two wheels and tilts are a sucker’s game.) I loved that cart for shopping, laundry, and so forth, but after using my Jeep stroller, there is no way I am going back to the granny cart.
I’m throwing this out there to my brain trust. How do I “de-baby” this cart so I can continue using it to lug massive quantities of various haulage without being a social pariah?
It was bad enough a few years ago when I could answer all the “so where’s your…baby…?” questions with “I’m on the way to pick him up from daycare/school/whatever.” But now? I don’t have that excuse.
There is no baby. There is only cart. It doesn’t have to be “cool”. I just don’t want members of the homeowners association to start calling the police about the crazy lady walking around with an empty baby stroller.