Last night, I spent a tad more time on SourceKit parsing than any human would find reasonable. After, I looked what I had done, experienced shame, and ripped it all back out. Here is the story of my SourceKit.

SourceKit Calls

I started off here at JP Simard’s discussion of all the cool tools that let you do this at the command line. That post was back in July, and those tools no longer exist.

So after more googlage, I discovered that his SourceKitten repo had been updated to use all sorts of asmname calls like this:

@asmname("sourcekitd_initialize") internal func sourcekitd_initialize() -> Int

To which I responded, “Oh! linkable stuff. I can do that with dlsym.”

void *skptr = NULL;

@implementation SourceKitHelper
+ (void) initIfNeeded
{
    if (skptr == NULL) {
        skptr = dlopen("/Applications/Xcode.app/Contents/\
        Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/\
        sourcekitd.framework/sourcekitd", RTLD_LAZY);
 
        void (* sourcekitd_initialize)() = 
            dlsym(skptr, "sourcekitd_initialize");
        sourcekitd_initialize();
    }
}

+ (void) shutdown
{
    NSInteger (* sourcekitd_shutdown)() = 
        dlsym(skptr, "sourcekitd_shutdown");
    sourcekitd_shutdown();
    dlclose(skptr);
    skptr = NULL;
}

+ (void *) skptr
{
    [self initIfNeeded];
    return skptr;
}
@end

Boom. Instant sourcekit.

Parsing

Next up I needed to parse my files via calls to the SourceKit framework. I prefer to use dlsym as I’m just more familiar with that toolset.

- (NSDictionary *) parse: (NSString *) text 
{
    void *skptr = [SourceKitHelper skptr];
    uint64_t (* sourcekitd_uid_get_from_cstr)(const char *) = 
        dlsym(skptr, "sourcekitd_uid_get_from_cstr");
    xpc_object_t (*sourcekitd_send_request_sync)(xpc_object_t) = 
        dlsym(skptr, "sourcekitd_send_request_sync");
    BOOL (* sourcekitd_response_is_error)(xpc_object_t) = 
        dlsym(skptr, "sourcekitd_response_is_error");
 
    xpc_object_t request = xpc_dictionary_create(NULL, NULL, 0);
    xpc_dictionary_set_uint64(request, "key.request",
        sourcekitd_uid_get_from_cstr("source.request.editor.open"));
    xpc_dictionary_set_string(request, "key.name", "");
    xpc_dictionary_set_string(request, "key.sourcetext", text.UTF8String);
 
    xpc_object_t reply = sourcekitd_send_request_sync(request);
    return sourcekitd_response_is_error(reply) ? nil : 
        [NSDictionary dictionaryWithContentsOfXPCObject:reply];
}

Notice that last dictionaryWithContentsOfXPCObject: call? That’s from Steve Streza’s faboo XPCKit. Drop in the source, add -fno-objc-arc to each file, and boom, you’re ready.

Converting Tokens

The parser produces tokens in the dictionary, not raw strings. You need to convert these to  string representations before you can figure out what role each of your parse tree elements describes:

- (NSString *) stringForSourceKitUID: (NSUInteger) uid
{
    if (uid < 4300000000) return nil;
    void *skptr = [SourceKitHelper skptr];
    char * (*sourcekitd_uid_get_string_ptr)(NSUInteger) = 
        dlsym(skptr, "sourcekitd_uid_get_string_ptr");
    if (!uidStrings) uidStrings = @{}.mutableCopy;
    if (uidStrings[@(uid)]) return uidStrings[@(uid)];
    if (sourcekitd_uid_get_string_ptr(uid))
    {
        NSString *uidString = [NSString stringWithCString:
                sourcekitd_uid_get_string_ptr(uid) 
            encoding:NSUTF8StringEncoding];
    if (uidString) uidStrings[@(uid)] = uidString;
    return uidString;
    }
    return nil;
}

Conclusions

So why my negativity after all this? It turns out, I couldn’t get anything to give me better results than I was already getting with line-by-line lintage without a huge investment of time.

For example,  the sourcekit parse returns values with character offsets instead of lines.  I had to convert positions to line references to emit warnings.

Many of the things I was looking for such as whether items used access modifiers weren’t assisted by this approach. It’s difficult to tell from the returned values (they’re enumerated in this gist) when a method, for example, extends a generic type. It’s not impossible, just difficult.

In the end, I was able to do source code syntax highlighting (fun for two seconds, and there’s already a github repo by Simard) but I couldn’t get the tools to give me the actual functionality I was looking for. Ultimately, I stripped it all out. The rest of the project is in a new github repo. I’ll worry about all this stuff (if at all) after WWDC. Until then, I’ve got my current tools.

If you’d like, give testlint a try and let me know how well it worked for you.