Adventures with PHPUnit, geckodriver and selenium

In my current pet project, I try to not only focus on the features itself, but also in the quality of the software and the development environment. Since I programmed code myself, the professional software development world has evolved quite a lot, and new approaches and tools have emerged and are de facto standards nowadays. While I know them as an observer, I barely used them myself. That I want to change.

Historically, my project’s setup already included a selenium test file, using geckodriver and Facebook’s php-webdriver library. However, the test code itself was a plain php file, executing a series of webdriver instructions and returning failures when some string comparisons did not yield the desired results:

<?php

namespace Facebook\WebDriver;

use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\Remote\RemoteWebDriver;

$host = 'http://localhost:4444';

$driver = RemoteWebDriver::create($host);

$driver->get('https://example.com/foo.php');

$driver->findElement(WebDriverBy::id('title'))
       ->sendKeys('Acceptance test title'); 

...

if (strcmp($driver->findElement(WebDriverBy::id('title'))->getAttribute('value'), 'Acceptance test title')!=0) {
    echo "Just created task cannot be accessed!";
    $driver->quit();
    exit(1);
}

$driver->quit();

This code worked, and helped me to assure that my product didn’t break (to some degree) from a user perspective. However, it was ugly to maintain, didn’t produce any helpful test report and just didn’t feel right.

Failed Approach

After some research I found out about phpunit-selenium, a project combining the powers of phpunit, I guess the dominating unit testing framework in the php space with selenium. Interesting. However, from there I went down a slippery road to trial-and-error-land. I should have probably get cautious, when the only documentation I found was referring to phpunit 3.7 (its now at 9.2), and even the github project was only referring to phpunit 8.x versions. With some back-and-forth, I could identify a version which fitted together with phpunit in composer repos and sucessfully deployed:

{
    "require-dev": {
        "phpunit/phpunit": "9.2",
        "phpunit/phpunit-selenium": "dev-master"
    },
}

Apart from that, I had to install the selenium server, as so far I was directly connecting to the geckodriver. This led to the worst part of issues: For some reasons in 95% of cases, the test cases didnt execute properly, and it was somehow related to firefox crashes. I tried so many things:

• Made sure both firefox and geckodriver had most recent versions (they had), same for selenium server.
• Executed everything as non-root.
• Tweaked permissions.
• Increased log-levels of geckodriver, selenium server and even firefox (the latter didnt even create crash dumps).
• I even started to investigate if my server’s ubuntu has some limiting settings in systemd cgroups, somehow limiting the number of parallel threads and memory. After all, its a browser we are launching here.

All of the above didnt really change anything. It was really frustrating, and the whole chain was quite long to debug in my spare time. After approx. 10 hours of frustrated tests, I gave up and made up my mind.

Solution

Luckily, I found another approach, based on the same tooling I had initially already working, but in a cleaner way. How does this look like? I simply combined phpunit and webdriver calls without any third party addition. Code says more than thousand words:

<?php declare(strict_types=1);

namespace Facebook\WebDriver;

use PHPUnit\Framework\TestCase;
use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Exception\UnknownErrorException;

final class AcceptanceTest extends TestCase
{
    protected static $driver;

    public static function setUpBeforeClass(): void 
    {
        $host = 'http://localhost:4445';

        $capabilities = DesiredCapabilities::firefox();
        $capabilities->setCapability('moz:firefoxOptions', ['args' => ['-headless']]);
        $capabilities->setPlatform(WebDriverPlatform::LINUX);

        self::$driver = RemoteWebDriver::create($host, $capabilities);
    }
    
    public static function tearDownAfterClass(): void
    {
        self::$driver->quit();
    }

    public function testTaskIsSuccessfullyCreatedFromContent()
    {
        self::$driver->findElement(WebDriverBy::id('title')) 
             ->sendKeys('Acceptance test title'); 

...

        $this->assertEquals('Acceptance test title', self::$driver->findElement(WebDriverBy::id('title'))->getAttribute('value'));
    }
}

This gives me the full power of phpunit, with fixtures, asserts, reports and a nice console output:

PHPUnit 9.2.0 by Sebastian Bergmann and contributors.

..                                             2 / 2 (100%)

Time: 00:14.702, Memory: 6.00 MB

OK (2 tests, 3 assertions)