Archive for the ‘Hacking’ Category

Nostalgia Tuesday: By request, my 2012 Siri Post

Well, if anything does happen on Monday, we can play “How badly did she get it wrong“, right? And to add some icing, here’s a what-if post about Siri controlling your Apple TV and a proof-of-concept Siri-style dictation used in-app.

(There’s a comment on the video that I particularly love: “Scammer watch the mouse across the screen at the end.” I hate to destroy the tinfoil but I was feeding the Apple TV output through EyeTV and recording the output on my Mac. Bless that person’s conspiratorial heart.)

How 3rd Party apps might integrate with Siri

Third-party integration into Siri remains at the top of many of our TUAW wish lists. Imagine being able to say “Play something from Queen on Spotify” or even “I want to hear a local police scanner.” And Siri replying, “OK, you have two apps that have local police scanners. Do you want ScannerPro or Wunder Radio?”

So why doesn’t Siri do that?

Well, first of all, there are no third party APIs. Second, it’s a challenging problem to implement. And third, it could open Siri to a lot of potential exploitation (think of an app that opens every time you say “Wake me up tomorrow at 7:00 AM” instead of deferring to the built-in timer).

That’s why we sat down and brainstormed how Apple might accomplish all of this safely using technologies already in-use. What follows is our thought experiment of how Apple could add these APIs into the iOS ecosystem and really allow Siri to explode with possibility.

Ace Object Schema. For anyone who thinks I just sneezed while typing that, please let me explain. Ace objects are the assistant command requests used by the underlying iOS frameworks to represent user utterances and their semantic meanings. They offer a context for describing what users have said and what the OS needs to do in response.

The APIs for these are private, but they seem to consist of property dictionaries, similar to property lists used throughout all of Apple’s OS X and iOS systems. It wouldn’t be hard to declare support for Ace Object commands in an application Info.plist property lists, just as developers now specify what kinds of file types they open and what kind of URL addresses they respond to.

Practicality. If you think of Siri support as a kind of extended URL scheme with a larger vocabulary and some grammatical elements, developers could tie into standard command structures (with full strings files localizations of course, for international deployment).

Leaving the request style of these commands to Apple would limit the kinds of requests initially rolled out to devs but it would maintain the highly flexible way Siri users can communicate with the technology.

There’s no reason for devs to have to think of a hundred ways to say “Please play” and “I want to hear”. Let Apple handle that — just as it handled the initial multitasking rollout with a limited task set — and let devs tie onto it, with the understanding that these items will grow over time and that devs could eventually supply specific localized phonemes that are critical to their tasks.

Handling. Each kind of command would be delineated by reverse domain notation, e.g. com.spotify.client.play-request. When matched to a user utterance, iOS could then launch the app and include the Ace dictionary as a standard payload. Developers are already well-acquainted in responding to external launches through local and remote notifications, through URL requests, through “Open file in” events, and more. Building onto these lets Siri activations use the same APIs and approaches that devs already handle.

Security. I’d imagine that Apple should treat Siri enhancement requests from apps the same way it currently works with in-app purchases. Developers would submit individual requests for each identified command (again, e.g. com.spotify.client.play-request) along with a description of the feature, the Siri specifications — XML or plist, and so forth. The commands could then be tested directly by review team members or be automated for compliance checks.

In-App Use. What all of this adds up to is an iterative way to grow third party involvement into the standard Siri voice assistant using current technologies. But that’s not the end of the story. The suggestions you just read through leave a big hole in the Siri/Dictation story: in-app use of the technology.

For that, hopefully Apple will allow more flexible tie-ins to dictation features outside of the standard keyboard, with app-specific parsing of any results. Imagine a button with the Siri microphone that developers could add directly, no keyboard involved.

I presented a simple dictation-only demonstration of those possibilities late last year. To do so, I had to hack my way into the commands that started and stopped dictation. It would be incredibly easy for Apple to expand that kind of interaction option so that spoken in-app commands were not limited to text-field and text-view entry, but could be used in place of touch driven interaction as well.

 

Sing out your Mac

Use tcsh (just enter /bin/tcsh) and then:

  • repeat 22 echo "da" | say -v good
  • repeat 26 echo "da" | say -v cello
  • repeat 22 echo "da" | say -v "bad news"

and of course

  • say -v cello droid

Was reminded of this today by:

How to curl raw content from gists

In my most recent post about installing Ubuntu/Swift, I glancingly referred in a screenshot to pulling swift source from a github gist. I thought it was useful enough a tip to pull out to its own post.

Say you have a gist, for example: this one, which is located at gist.github.com/erica/4d31fed94f3668342623. I threw this sample together to help some Linux-ers work with incomplete Foundation and Standard Library implementations on Ubuntu.

If you grab this link’s content from the terminal command-line using curl, you’ll end up with the page’s HTML source, which is pretty much not what you want to compile:

Screen Shot 2015-12-20 at 11.20.36 AM

Instead, replace gist.github.com with gist.githubusercontent.com and append /raw to the url:

https://gist.githubusercontent.com/erica/4d31fed94f3668342623/raw

This adjusted URL bypasses the HTML page contents and accesses the raw text stored within the gist. Just redirect with > into a Swift file and you’re ready to compile.

Screen Shot 2015-12-20 at 11.24.02 AM

I hope this tip helps minimize the misery of sharing code with Ubuntians.

Living la vida Linux

This morning, I finally broke down and installed Ubuntu 15.10 into Virtual Box. I didn’t take notes, which I am now kicking myself for because it didn’t occur to me until after I had gone through the entire annoyingly tweaky process.

Here are the highlights.

First, I installed Virtual Box. I mostly went with the directions in this Simple Help write-up. The directions are 15.04, but they worked fine for 15.10. Per the write-up, I went with all the defaults. It took an hour or more for the Ubuntu install to get itself set-up. Prepare to head out for a few errands.

Next, I spent time familiarizing myself with Ubuntu, finding the terminal, chsh’ing to tcsh from bash (I had to apt-get tcsh) and setting up my .cshrc. I also installed lns (bless it) and downloaded the latest 15.10 Swift 2.2 snapshot from swift.org.

Then I symbolic linked swift and swiftc into my ~/bin folder, checked that I could run them both from wherever, and then set to working on getting ssh running so I didn’t have to use Virtual Box directly.

To do that, first I had to apt-get install OpenSSH-server, which was no big deal, and then what took the most time was figuring out how to set up networking. Turns out that you need to shut down the machine, set up Adapter 1 to use Bridged Adapter (thanks aciidb) and the find the 10.0.whatever-ip-address using if-config after rebooting. (Mine was 10.0.0.35).

Once I got past that, I was able to ssh in over from my Mac and start using Swift. I began with a simple shell script (thought that was easier) and then moved onto swift and then swiftc compilation.

As Mike Ash pointed out, when encouraging me to use ssh with Virtual Box, you gain all the utility of your normal terminal session including copy, paste, etc without having to deal with mouse capture, etc.

Hardware hacking help solicited: Remake my stroller

Photo 12-10-2015-11.12.49

This is my Jeep stroller. It is probably the best stroller ever. It’s also the best shopping cart ever.  It has cupholders. It has not just a pop-up glove compartment at the top but extra cargo bags on the left and right for small items.

The basket underneath is enormous. It carries tons of weight. The wheels are rugged and have been able to navigate through just about any terrain, including going to the local market through Colorado winters.

I can load a couple of 12-packs of sodas, carry about 6-8 other bags of groceries. There’s room above, there’s room below, there’s easy-to-tie-to-handles. When I’m at the store, I can stick a basket in it, fill it, surround it, and put coats, gloves,  hats, etc below (or use that area for more groceries). It is, in short, the ultimate shopping machine.

Only one problem. My baby is now a few years away from starting to grow facial hair. Try to use this thing and you get side eye, hairy eyeball, every kind of “are you some kind of sick bag-lady with an old shopping cart stroller” look you can imagine.

There’s some kind of unspoken consensus that after our kids have grown, we must transition to granny carts. I used to own a granny cart when I was in college. It was fantastic. (4-wheel variety because two wheels and tilts are a sucker’s game.) I loved that cart for shopping, laundry, and so forth, but after using my Jeep stroller, there is no way I am going back to the granny cart.

I’m throwing this out there to my brain trust. How do I “de-baby” this cart so I can continue using it to lug massive quantities of various haulage without being a social pariah?

It was bad enough a few years ago when I could answer all the “so where’s your…baby…?” questions with “I’m on the way to pick him up from daycare/school/whatever.” But now? I don’t have that excuse.

There is no baby. There is only cart. It doesn’t have to be “cool”. I just don’t want members of the homeowners association to start calling the police about the crazy lady walking around with an empty baby stroller.

Thanks in advance for your help.

Swift: How to find undocumented functions

In response to requests for more clarity. Navigate to your Xcode beta and pop down to Contents/Developer/Platforms. From there go down into any of the Mac platforms, e.g. MacOSX.platform and then to Developer/SDKs and into another platform, e.g. MacOSX10.11.sdk. Next enter System/Library/PrivateFrameworks/Swift.

Read On…