Monday, December 16, 2013

Testing on Android (Part 1): Unit Tests

I've recently been doing research on Android testing.  I'm quite new to this business; automated testing has (for the longest time) felt unnecessary.  But I've been slowly convinced that the benefits outweigh the time costs - as long as you do it in an efficient manner.  To that end, I set out to evaluate the many, many Android testing frameworks to see which ones seemed to save the most time.

I began by summarizing all the tools/services I could find in a spreadsheet (let me know if I'm missing anything).  Beyond that, I am going to do a series of posts going into more detail about different parts of Android testing:
Let's start with unit testing.

Unit Tests

There's no reason you can't use normal JUnit 4 testing for Android applications... as long as you stay away from anything Android.

Normally you compile against the SDK's android.jar, which contains nothing but stubbed methods that throw exceptions when run.  When you actually upload your APK to a device, it uses the device's implementations of all those stubs.  As a result, when running normal unit tests in your IDE, you get no access to those framework implementations (instead receiving mountains of exceptions).  This is not a big deal if you're testing some simple functionality that doesn't touch Android itself.

Pros:
  • Fast and easy
Cons:
  • Cannot use any Android framework classes
The Android Testing Framework

The Android testing framework is the official method of unit testing on Android.  It loads your application onto a device, then runs JUnit-based test suites.  Since it runs on the actual OS you can use the Android framework as you normally would in your application and can conduct a series of realistic tests that way.

Ostensibly the testing framework is unit testing, but the slowness of having to fully compile and upload your app onto a device before executing any tests makes testing slow.  Plus, you have to make sure you've got a device attached or an emulator running.  As a result, I might consider the testing framework for semi-regular tests (e.g., whenever you push a new commit, or nightly tests) but I would have trouble using them while actively developing.

Pros:
  • Access to the Android framework
Cons:
  • Slow
  • Requires attached device or running emulator
  • Uses JUnit 3 (instead of the newer JUnit 4)
Robolectric

Robolectric is a project that unifies the speed of unit testing with the ability to access the Android framework.  It does this by implementing all those stubs with mocked classes.

Having tried it out, it is lightning fast and works as expected.  I'm also impressed with the amount of active development on it - this is a rapidly improving framework.  However, the active development does take a toll; documentation is a bit lacking, plus some new versions of Robolectric break things in previous versions.  Plus, it can't mock everything - for example, inter-app communication - since it's not on an actual Android OS.  That said, the benefits here far outweigh the negatives when it comes to unit testing Android.

Pros:
  • Fast
  • Can access mocked Android framework
  • Actively developed
Cons:
  • Not the true Android framework
  • Not everything is mocked
  • Lacking documentation
Conclusion

I'm a fan of what Jason Sankey said on StackOverflow about tiered unit testing: Prefer pure unit tests, then Robolectric, then the Android testing framework.  The tests get harder/slower the higher the tier, but sometimes you need it.

4 comments:

  1. What's really the difference between the Android Testing Framework (which is instrumentation testing), and Espresso and Robotium, which as far as I can tell, are just layers on top this to make doing certain things easier? Why wouldn't you want to use Robotium or Espresso to make writing your tests easier? The lines, at least it seems to me, are so blurry between what's a unit test and what's a functional test when it comes to Android.

    ReplyDelete
    Replies
    1. I think the difference is that I can use ADF for non-UI testing (that still required the Android framework). E.g., I've got a series of network calls I want to test, outside the context of a running app. Were I to be doing UI testing then I would only use Espresso or Robotium because they're designed for it.

      Delete
  2. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
    Mobile App Development Company
    Android App Development Company

    ReplyDelete