Archive for the ‘Hacking’ Category

Programming Cozmo

Anki has been kind enough to let me play with their new Cozmo unit and explore their SDK. Cozmo is a wonderful device, developed by people who understand a lot of core principles about human interaction and engagement.

Cozmo is adorable. When it recognizes your face, it wriggles with happiness. It explores its environment. When it’s bored, it sets up a game to play with you. It can get “upset” and demand attention. It’s one of the most personable and delightful robots I’ve played with.

At its heart is a well-chosen collection of minimal elements. The unit can move around the room, with a 4-wheel/2-tread system. It includes an onboard forklift that can rise and fall, an OLED “face” that expresses emotion, and a camera system that ties into a computer vision system, which I believe is based on PIL, the Python Image Library. (Anki tells me that Cozmo’s vision system “does not use PIL or Python in any way, though the Python SDK interface uses PIL for decoding jpegs, drawing animations, etc.”)

Three lightweight blocks with easily-identified markings complete the Cozmo package, which Cozmo can tap, lift, stack, and roll.

Between its remarkable cuteness and its vision-based API, it’s a perfect system for introducing kids to programming. I was really excited to jump into the SDK and see how far I could push it.

Here is Anki’s “Hello World” code (more or less, I’ve tweaked it a little) from their first developer tutorial:

import sys
import cozmo

'''
Hello Human
Make Cozmo say 'Hello Human' in this simple
Cozmo SDK example program.
'''

def run(sdk_conn):
    robot = sdk_conn.wait_for_robot()
    robot.say_text("Hello Human").wait_for_completed()
    print("Success")

if __name__ == '__main__':
    cozmo.setup_basic_logging()    
    try:
        cozmo.connect(run)
    except cozmo.ConnectionError as err:
        sys.exit("Connection error 😬: %s" % err)

Although simple, this “Hello World” includes quite a lot of implementation details that can scare off young learners. For comparison, here’s the start of Apple’s tutorial on Swift “Learn to Code”:

screen-shot-2016-12-12-at-11-45-24-am

There’s such a huge difference here. In Apple’s case, everything that Byte (the main character) does is limited to easy-to-understand, simple calls. The entire implementation is abstracted away, and all that’s left are instructions and very directed calls, which the student can put together, re-order, and explore with immediate feedback.

In Anki’s code, you’re presented with material that’s dealing with set-up, exceptions, asynchronous calls, and more. That is a huge amount of information to put in front of a learner, and to then say “ignore all of this”. Cozmo is underserved by this approach. Real life robots are always going to be a lot more fun to work with than on-screen animations. Cozmo deserved as simple a vocabulary as Byte. That difference set me on the road to create a proof of concept.

In this effort, I’ve tried to develop a more engaging system of interaction that better mirrors the way kids learn. By creating high level abstractions, I wanted to support the same kind of learning as “Learn to Code”. Learn to Code begins with procedural calls, and then conditional ones, and moving on to iteration and functional abstraction, and so forth.

My yardstick of success has been, “can my son use these building blocks to express goals and master basic procedural and conditional code?” (I haven’t gotten him up to iteration yet.) So far, so good, actually.  Here is what my updated “Hello World” looks like for Cozmo, after creating a more structured entry into robot control functionality:

from Cozmo import *

# run, cozmo, run
def actions(cozmoLink):
    '''Specify actions for cozmo to run.'''
    
    # Fetch robot
    coz = Cozmo.robot(cozmoLink)

    # Say something
    coz.say("Hello Human")

Cozmo.startUp(actions)

Not quite as clean as “Learn to Code” but I think it’s a vast improvement on the original. Calls now go through a central Cozmo class. I’ve chunked together common behavior and I’ve abstracted away most implementation details, which are not of immediate interest to a student learner.

Although I haven’t had the time to really take this as far as I want, my Cozmo system can now talk, drive, turn, and engage (a little) with light cubes. What follows is a slightly more involved example. Cozmo runs several actions in sequence, and then conditionally responds to an interaction:

from Cozmo import *
from Colors import *

# Run, Cozmo, run
def actions(cozmoLink):
    '''Specify actions for cozmo to run.'''
    
    # Fetch robot
    coz = Cozmo.robot(cozmoLink)

    # Say something
    coz.say("Hello")

    # Drive a little
    coz.drive(time = 3, direction = Direction.forward)
    
    # Turn
    coz.turn(degrees = 180)
    
    # Drive a little more
    coz.drive(time = 3, direction = Direction.forward)

    # Light up a cube
    cube = coz.cube(0)
    cube.setColor(colorLime)
    
    # Tap it!
    coz.say("Tap it")    
    if cube.waitForTap():
        coz.say("You tapped it")
    else:
        coz.say("Why no tap?")
    cube.switchOff()

Cozmo.startUp(actions)

And here is a video showing Cozmo executing this code:

If you’d like to explore this a little further:

  • Here is a video showing the SDK feedback during that execution. You can see how the commands translate to base Cozmo directives.
  • I’ve left a bit of source code over at GitHub if you have a Cozmo or are just interested in my approach.

As you might expect, creating a usable student-focused learning system is time consuming and exhausting. On top of providing controlled functionality, what’s missing here is a lesson plan and a list of skills to master framed into “Let’s learn Python with Cozmo”. What’s here is just a sense of how that functionality might look when directed into more manageable chunks.

Given my time frame, I’ve focused more on “can this device be made student friendly” than producing an actual product. I believe my proof of concept shows that the right kind of engagement can support this kind of learning with this real-world robot.

The thing that appeals most to me about Cozmo from the start has been its rich computer vision capabilities. What I haven’t had a chance to really touch on yet is its high level features like “search for a cube”, “lift it and place it on another cube”, all of which are provided as building blocks in its existing API, and all of which are terrific touch points for a lesson plan.

I can easily see where I’d want to develop some new games with the robot, like lowering reaction time (it gets really hard under about three quarters of a second to tap that darn cube) and creating cube-to-cube sequences of light. I’d also love to discover whether I can extend detection to some leftovers my son brought home from our library’s 3D printer reject bin.

Cozmo does not offer a voice input SDK. It’s only real way to interact is through its cameras (and vision system) and through taps on its cubes. Even so, there’s a pretty rich basis to craft new ways to interact.

As for Anki’s built-ins, they’re quite rich. Cozmo can flip cubes, pull wheelies, and interact in a respectably rich range of physical and (via its face screen) emotional ways.

Even if you’re not programming the system, it’s a delightful toy. Add in the SDK though, and there’s a fantastic basis for learning.

Cozmo: The Unboxening

I recently reached out to Anki, creators of the Cozmo robot, to  ask if I could explore the device from a developer’s perspective. Shipping with a Python SDK, Cozmo offers some surprisingly sophisticated image processing and recognition features, analogous to Apple’s Core Image.

Before jumping into the programming side of things, let me acknowledge that I am primarily an Apple developer. Therefore I must categorically kick off by evaluating package design.

I didn’t get to start unboxing last night. When my unit arrived during a cold snap, the package window was all fogged up. I opened the top, added a desiccant unit, and let the package dry out and warm out overnight.

czm6qavweaadneu

This morning, I finally could explore. As you’d expect with high end consumer goods (Cozmo retails for $180), the box used satisfyingly thick cardboard, was easy to open, and presented the product nicely while hiding the user manual, power cord, and accessories.

Unexpectedly, my favorite part of the entire boxing system was Cozmo’s perch. Made of quality yellow plastic, this industrial presentation feature is practically a toy in itself.

screen-shot-2016-12-09-at-9-00-16-am

Flip it over, and instead of expected wire wraps, there’s an ingenious system to release the Cozmo unit. Pull up the yellow spacer and pinch the two white tabs. It all comes apart, allowing you to remove Cozmo, and start him charging.

screen-shot-2016-12-09-at-9-01-55-am

Then while he charges, you can spend time putting the pieces back together and taking them apart over and over. It’s practically an extra free toy that ships with the robot. Beautiful design and completely unexpected.

screen-shot-2016-12-09-at-9-04-28-am

The full complement of parts include the Cozmo robot, a USB charging dock with a separate a wall-adapter, a trio of play blocks (“Interactive Power Cubes”), and a welcome packet.

screen-shot-2016-12-09-at-9-09-51-am

The instructions say to place him on an open table in a well lit room, with room to move around. Charging from empty to full is specified at 10-12 minutes, with a rated play time of 1-2 hours. (I wouldn’t be surprised if that number drops once you start putting extra demands on his processors.)

That 10-12 minutes subjectively lasts approximately 3-4 months after you first finish unboxing. Child and I got into a heated debate as to whether we’d name him after Cosimo de’ Medici, founder of the Medici political dynasty, or Cosmog, nebula Pokemon and opener of “Ultra Wormholes”.

I don’t want to spoil the instruction booklet for you so let me just say, the writeup is adorable, clever, and simple. The consumer warnings in particular made me laugh out loud. It’s a great example of technical writing and communication, focusing on simplicity and clarity. It’s really well done.

You must install a separate Cozmo application you on an iOS or Android device. It just shipped a 1.1.0 update, so make sure you’re running the latest version. It helps if you watch this video before trying to set-up.

Open Settings and connect to the Cozmo WiFi network. Lift Cozmo’s front arm to display the password, and then use Settings to log in, making sure to type everything using exact upper casing. The password is random mix of upper case letters and numbers, and the iPhone’s keypad doesn’t remember casing when moving between number input and text input.

Anki recommends typing the password into your Notes app, and I endorse this suggestion, especially when I point out how many times I tried to get it to connect. That’s because at first I didn’t realize Cloak VPN was trying to secure the Cozmo WiFi connection. It really really helps if you set it up to trust the Cozmo network. This one detail put a huge crimp into my set-up, causing immeasurable pain and frustration. I finally found this support write-up, which mentions VPN issues at the very bottom.

I haven’t spent much time playing with Cozmo yet but speaking as an iOS dev, there are quite a few things Anki could do to tighten up their app. There’s great content and some terrific games but the app reads as “cross platform with compromise”. (It seems to have Unity under its cover.)

To give just a couple of examples, the cursor controls in a custom text field don’t follow iOS standards. When you use cursor arrows during typing, the active cursor position does not update. That’s iOS 101.

In terms of user interface flow, it’s missing iOS’s inherent “deference to the user”. For example, there’s no “try again” button when attempting to connect without having to go to the help screen over and over again. (Which I did, over and over and over, through an hour or so of set-up until I found that VPN advice.) When you’re using games and other features, the ability to quickly switch tasks is somewhat limited.

The app would benefit from a HIG/GUI once-over for usability but that’s really the only weak spot in the big package that I’ve encountered.

As for Cozmo himself, he’s a delight. As part of set-up, he learned my face and that of my daughter. Who cannot love a robot that wakes up from charging, sees your face, recognizes you, greets you by name and then wiggles with happiness?!? It’s phenomenally adorable.

Of course, my interest lies primarily in the SDK, and that exploration will have to wait for another write-up. For now, a summary to date:

  • Adorable robot with amazing humanizing affect display.
  • Top notch Apple-worthy packaging.
  • Great starter games to inspire development possibilities.
  • There’s an app.

Cozmo is right now exploring the floor of my office, making random offhand comments in robotese about what he’s finding and generally having a ball. I think I’ll stop writing for a short while and join him in having fun.

Kid-fu: Pay for play

Screen Shot 2016-08-19 at 7.24.19 PMHow do you get your kid to try on all his shirts and pants before school starts? So you can sort them into trash, donate, wear, and “for playtime use only”? Answer: Playgrounds. Thank you playground team!

We used the “deal or no deal” rule. Two spins. You can “stick” or you can re-try, but no peeking at the next roll. Payment for each garment  after trying it on, taking it off and folding it. Coins placed into a cauldron (thank you Harry Potter camp) to be counted after we’re done. Any stained items treated with Oxyclean and thrown into the hamper.

For less than a couple of dollars, we got through his entire wardrobe with no complaints except “Don’t I have anything else that needs trying on?”

Source code follows:

https://gist.github.com/erica/57dd27ea693b36b79db5581684e6268b

Bouncing AirDrop contents to my desktop

I never use my Downloads folder. It’s a fusion drive, so it’s precious, fast, and expensive. I don’t need a thousand downloaded copies of Xcode and firmware updates littering its limited space. Instead, I point all my browsers and other apps to download to my secondary data disk.

Screen Shot 2016-06-27 at 12.57.18 PM

And before you ask, I use numerical and alphabetic prefixes so everything shows up in the right place and the right order for quick reference and single-letter typing access. Whatever data I can offload from my main drive, I do offload:

Screen Shot 2016-06-27 at 12.59.42 PM

However, when it comes to airdropping, it’s generally true that whatever I’m sending back and forth is of immediate interest. In such case, I don’t want it heading into my Downloads folder. I want it on my desktop as soon as it lands. As  I’m updating my Playgrounds Book right now, I’m doing a lot more airdropping than I normally would.

I’m not a big user of smart folders and Automator actions. I have a smallish bunch that I occasionally use. Still, they have their place and today was a perfect occasion to bring a new one into the mix.

I just had had it with the Downloads folder and decided to build a bouncer that would automatically throw any item added to ~/Downloads up to the desktop. I thought I’d share how to do this.

Step 1. Create a new Folder Action

Screen Shot 2016-06-27 at 1.03.48 PM

Step 2. Choose the Downloads folder.

Screen Shot 2016-06-27 at 1.05.50 PM

Step 3. Drag “Move Finder Items” onto “Drag actions or files here to build your workflow”

Screen Shot 2016-06-27 at 1.06.37 PM

Screen Shot 2016-06-27 at 1.06.48 PM

This creates the following action, with Desktop selected by default. (If it’s not, choose Desktop for the destination.)

Screen Shot 2016-06-27 at 1.07.24 PM

Step 4. Then save:

Screen Shot 2016-06-27 at 1.09.12 PM

Your new automator action is stored in ~/Library/Workflows/Applications/Folder\ Actions:

Screen Shot 2016-06-27 at 1.11.43 PM

Step 5. Test. Drop a file into Downloads and confirm that it moves to the desktop. You should now be ready to airdrop to your desktop.

Note: I’m sure there’s a better way to do this, but I actually wrote an app that quickly opens AirDrop windows on the Mac side of things. I found an appropriate AppleScript online, compiled it to an app, and use Spotlight to launch it. Very handy when I’m more focused on iOS than OS X at the moment.

Nostalgia Tuesday: By request, my 2012 Siri Post

Well, if anything does happen on Monday, we can play “How badly did she get it wrong“, right? And to add some icing, here’s a what-if post about Siri controlling your Apple TV and a proof-of-concept Siri-style dictation used in-app.

(There’s a comment on the video that I particularly love: “Scammer watch the mouse across the screen at the end.” I hate to destroy the tinfoil but I was feeding the Apple TV output through EyeTV and recording the output on my Mac. Bless that person’s conspiratorial heart.)

How 3rd Party apps might integrate with Siri

Third-party integration into Siri remains at the top of many of our TUAW wish lists. Imagine being able to say “Play something from Queen on Spotify” or even “I want to hear a local police scanner.” And Siri replying, “OK, you have two apps that have local police scanners. Do you want ScannerPro or Wunder Radio?”

So why doesn’t Siri do that?

Well, first of all, there are no third party APIs. Second, it’s a challenging problem to implement. And third, it could open Siri to a lot of potential exploitation (think of an app that opens every time you say “Wake me up tomorrow at 7:00 AM” instead of deferring to the built-in timer).

That’s why we sat down and brainstormed how Apple might accomplish all of this safely using technologies already in-use. What follows is our thought experiment of how Apple could add these APIs into the iOS ecosystem and really allow Siri to explode with possibility.

Ace Object Schema. For anyone who thinks I just sneezed while typing that, please let me explain. Ace objects are the assistant command requests used by the underlying iOS frameworks to represent user utterances and their semantic meanings. They offer a context for describing what users have said and what the OS needs to do in response.

The APIs for these are private, but they seem to consist of property dictionaries, similar to property lists used throughout all of Apple’s OS X and iOS systems. It wouldn’t be hard to declare support for Ace Object commands in an application Info.plist property lists, just as developers now specify what kinds of file types they open and what kind of URL addresses they respond to.

Practicality. If you think of Siri support as a kind of extended URL scheme with a larger vocabulary and some grammatical elements, developers could tie into standard command structures (with full strings files localizations of course, for international deployment).

Leaving the request style of these commands to Apple would limit the kinds of requests initially rolled out to devs but it would maintain the highly flexible way Siri users can communicate with the technology.

There’s no reason for devs to have to think of a hundred ways to say “Please play” and “I want to hear”. Let Apple handle that — just as it handled the initial multitasking rollout with a limited task set — and let devs tie onto it, with the understanding that these items will grow over time and that devs could eventually supply specific localized phonemes that are critical to their tasks.

Handling. Each kind of command would be delineated by reverse domain notation, e.g. com.spotify.client.play-request. When matched to a user utterance, iOS could then launch the app and include the Ace dictionary as a standard payload. Developers are already well-acquainted in responding to external launches through local and remote notifications, through URL requests, through “Open file in” events, and more. Building onto these lets Siri activations use the same APIs and approaches that devs already handle.

Security. I’d imagine that Apple should treat Siri enhancement requests from apps the same way it currently works with in-app purchases. Developers would submit individual requests for each identified command (again, e.g. com.spotify.client.play-request) along with a description of the feature, the Siri specifications — XML or plist, and so forth. The commands could then be tested directly by review team members or be automated for compliance checks.

In-App Use. What all of this adds up to is an iterative way to grow third party involvement into the standard Siri voice assistant using current technologies. But that’s not the end of the story. The suggestions you just read through leave a big hole in the Siri/Dictation story: in-app use of the technology.

For that, hopefully Apple will allow more flexible tie-ins to dictation features outside of the standard keyboard, with app-specific parsing of any results. Imagine a button with the Siri microphone that developers could add directly, no keyboard involved.

I presented a simple dictation-only demonstration of those possibilities late last year. To do so, I had to hack my way into the commands that started and stopped dictation. It would be incredibly easy for Apple to expand that kind of interaction option so that spoken in-app commands were not limited to text-field and text-view entry, but could be used in place of touch driven interaction as well.

 

Sing out your Mac

Use tcsh (just enter /bin/tcsh) and then:

  • repeat 22 echo "da" | say -v good
  • repeat 26 echo "da" | say -v cello
  • repeat 22 echo "da" | say -v "bad news"

and of course

  • say -v cello droid

Was reminded of this today by:

How to curl raw content from gists

In my most recent post about installing Ubuntu/Swift, I glancingly referred in a screenshot to pulling swift source from a github gist. I thought it was useful enough a tip to pull out to its own post.

Say you have a gist, for example: this one, which is located at gist.github.com/erica/4d31fed94f3668342623. I threw this sample together to help some Linux-ers work with incomplete Foundation and Standard Library implementations on Ubuntu.

If you grab this link’s content from the terminal command-line using curl, you’ll end up with the page’s HTML source, which is pretty much not what you want to compile:

Screen Shot 2015-12-20 at 11.20.36 AM

Instead, replace gist.github.com with gist.githubusercontent.com and append /raw to the url:

https://gist.githubusercontent.com/erica/4d31fed94f3668342623/raw

This adjusted URL bypasses the HTML page contents and accesses the raw text stored within the gist. Just redirect with > into a Swift file and you’re ready to compile.

Screen Shot 2015-12-20 at 11.24.02 AM

I hope this tip helps minimize the misery of sharing code with Ubuntians.

Living la vida Linux

This morning, I finally broke down and installed Ubuntu 15.10 into Virtual Box. I didn’t take notes, which I am now kicking myself for because it didn’t occur to me until after I had gone through the entire annoyingly tweaky process.

Here are the highlights.

First, I installed Virtual Box. I mostly went with the directions in this Simple Help write-up. The directions are 15.04, but they worked fine for 15.10. Per the write-up, I went with all the defaults. It took an hour or more for the Ubuntu install to get itself set-up. Prepare to head out for a few errands.

Next, I spent time familiarizing myself with Ubuntu, finding the terminal, chsh’ing to tcsh from bash (I had to apt-get tcsh) and setting up my .cshrc. I also installed lns (bless it) and downloaded the latest 15.10 Swift 2.2 snapshot from swift.org.

Then I symbolic linked swift and swiftc into my ~/bin folder, checked that I could run them both from wherever, and then set to working on getting ssh running so I didn’t have to use Virtual Box directly.

To do that, first I had to apt-get install OpenSSH-server, which was no big deal, and then what took the most time was figuring out how to set up networking. Turns out that you need to shut down the machine, set up Adapter 1 to use Bridged Adapter (thanks aciidb) and the find the 10.0.whatever-ip-address using if-config after rebooting. (Mine was 10.0.0.35).

Once I got past that, I was able to ssh in over from my Mac and start using Swift. I began with a simple shell script (thought that was easier) and then moved onto swift and then swiftc compilation.

As Mike Ash pointed out, when encouraging me to use ssh with Virtual Box, you gain all the utility of your normal terminal session including copy, paste, etc without having to deal with mouse capture, etc.

Hardware hacking help solicited: Remake my stroller

Photo 12-10-2015-11.12.49

This is my Jeep stroller. It is probably the best stroller ever. It’s also the best shopping cart ever.  It has cupholders. It has not just a pop-up glove compartment at the top but extra cargo bags on the left and right for small items.

The basket underneath is enormous. It carries tons of weight. The wheels are rugged and have been able to navigate through just about any terrain, including going to the local market through Colorado winters.

I can load a couple of 12-packs of sodas, carry about 6-8 other bags of groceries. There’s room above, there’s room below, there’s easy-to-tie-to-handles. When I’m at the store, I can stick a basket in it, fill it, surround it, and put coats, gloves,  hats, etc below (or use that area for more groceries). It is, in short, the ultimate shopping machine.

Only one problem. My baby is now a few years away from starting to grow facial hair. Try to use this thing and you get side eye, hairy eyeball, every kind of “are you some kind of sick bag-lady with an old shopping cart stroller” look you can imagine.

There’s some kind of unspoken consensus that after our kids have grown, we must transition to granny carts. I used to own a granny cart when I was in college. It was fantastic. (4-wheel variety because two wheels and tilts are a sucker’s game.) I loved that cart for shopping, laundry, and so forth, but after using my Jeep stroller, there is no way I am going back to the granny cart.

I’m throwing this out there to my brain trust. How do I “de-baby” this cart so I can continue using it to lug massive quantities of various haulage without being a social pariah?

It was bad enough a few years ago when I could answer all the “so where’s your…baby…?” questions with “I’m on the way to pick him up from daycare/school/whatever.” But now? I don’t have that excuse.

There is no baby. There is only cart. It doesn’t have to be “cool”. I just don’t want members of the homeowners association to start calling the police about the crazy lady walking around with an empty baby stroller.

Thanks in advance for your help.

Swift: How to find undocumented functions

In response to requests for more clarity. Navigate to your Xcode beta and pop down to Contents/Developer/Platforms. From there go down into any of the Mac platforms, e.g. MacOSX.platform and then to Developer/SDKs and into another platform, e.g. MacOSX10.11.sdk. Next enter System/Library/PrivateFrameworks/Swift.

Read On…