Simpletest, Drush 8, test-run, and Exit Codes.

If you're using Drush to run your tests, swapping Drush version 6.x out for 8 isn't a simple replacement. First drush test-run has been removed so running Drupal simpletests now requires using the run-tests.sh. This would be workable, however under Drupal 7, scripts/run-tests.sh does not return correct exit codes so if you're relying on failed tests returning a 1 you are out of luck. While this has been fixed for Drupal 8, if you're running Drupal 7 you're going to have to go an alternate route.

To avoid maintaining a patched core and keeping test running scripts out of the web root I copied the run-tests.sh script to bin and applied the exit code patch. This isn't sufficient however because run-tests.sh has assumptions about the Drupal directory structure hard coded. Thus a second change is needed to ensure that it knows where the webroot is and how to call itself again to run the next test. I should have made the fix be an argument that you pass in. Sorry.

I combined the exit code patch for D7 from and the directory fixes into this patch: run-tests.sh.patch

I'm able to run my tests from outside the Drupal root with: php -C bin/run-tests.php --url http://mysite.local MyTests
If you're using CircleCi you will likely need to specify php directly: php -C bin/run-tests.php --php /home/ubuntu/.phpenv/shims/php --url http://mysite.local MyTests

If you're building your site with composer, it's straight forward to apply the exit code patch with your composer file in the Drupal root with something like this:

{
  "require": {
    "cweagans/composer-patches": "~1.0",
    "drupal/drupal": "7.0.*@dev"
  },
  "config": {
    "preferred-install": "source"
  },
  "extra": {
    "patches": {
      "drupal/drupal": {
        "Patch run-scripts.sh to support correct exit codes.": "https://www.drupal.org/files/issues/2189345-39_0.patch"
      }
    }
  }
}