Debugging Tests

How do I run specific tests?

By default, Subliminal will run all the test cases (methods whose names begin with "test", which take no arguments, and return void) of all the SLTest subclasses. You can restrict testing to particular test cases by prefixing their names with "focus_". You can "focus" all test cases of a particular SLTest subclass by prefixing its name with "focus_", too. Don't forget to remove the prefixes before committing your changes!

How can I run log statements when using Subliminal?

When running Instruments, NSLog statements won't appear in the Xcode console. (You can view NSLog statements in the Console app, but this is inconvenient.) To log to Instruments, use SLLog to log from an integration test, and use SLLogAsync to log from the main thread.

How can I use the Debugger while running tests?

You can easily attach the debugger to Instruments, to stop at breakpoints set in Xcode, using the following process:

1. Change the build configuration used to profile the "Integration Tests" scheme from "Release" (the default) to "Debug" (in Xcode, click "Product" -> "Schemes" -> "Manage Schemes…", double-click the "Integration Tests" scheme, select the "Profile" action, and then change the build configuration). Otherwise, debug information will be optimized away during build.

   You need not worry that building in "Debug" will cause your tests to not reflect the actual "Release" state of the app--subliminal-test will override the configuration to use "Release" when building the app from the command line.

2. Launch the tests, then click "Product" -> "Attach to Process" -> (name of the test process, likely "Integration Tests") in Xcode.

   This may be done at any time while the tests are running. If you need to break immediately after launch, you may find it useful to give yourself time to attach the debugger by setting -[SLTestController shouldWaitToStartTesting] to YES.

How can I disable UIAutomation's debug logs?

UIAutomation's debug logs record every user interaction simulated by the Automation instrument. They can be quite noisy. To prevent the logs from appearing in Instruments' GUI, execute this command in Terminal:

defaults write com.apple.dt.Instruments UIAVerboseLogging 4096

To re-enable the logs, execute:

defaults delete com.apple.dt.Instruments UIAVerboseLogging

The subliminal-test script prevents these logs from appearing when tests are run at the command line, unless the --verbose_logging flag is specified.

How can Subliminal tell me where I'm getting "invalid element" and/or "element not tappable" exceptions?

Use the UIAElement macro to log the filename and line number of calls to -[SLUIAElement tap], etc.:

SLButton *button = [SLButton buttonWithAccessibilityLabel:@"foo"];
[UIAElement(button) tap];   // vs. [button tap];

Why does Subliminal say that an SLElement doesn't exist/isn't visible/isn't tappable when I can see/tap it just fine?

There a couple of ways in which your application's structure, as "seen" by Subliminal, might differ from your expectations.

A good way to start debugging these sorts of failures is to use the Accessibility Inspector to examine your application. If you can't focus the Inspector on your element, it's probably not accessible. Apple describes how to fix this here.

If you can focus the Inspector on your element, but your tests still can't manipulate this element, it might be that there are multiple objects in your application's view hierarchy with the same accessibility information--Subliminal might find another match for your element than you intended (and unlike the element you intended, Subliminal's match might be hidden, offscreen, etc.).

Try calling -logElement on your element--does its description match what you had expected? Maybe you find that Subliminal matched a label when you meant to match a button. You can fix this by making your matching criteria more specific, like by using SLButton instead of SLElement. You can apply more complex criteria using +[SLElement elementMatching:]. If you want to restrict matches to visible elements, for instance, you can do:

// at the top of your test class
#import <Subliminal/NSObject+SLVisibility.h>

// in a test case
SLElement *elementToMatch = [SLElement elementMatching:^BOOL(NSObject *obj) {
    // check other criteria first: checking visibility is expensive
    return /* some other criteria */ && [obj slAccessibilityIsVisible];
}];