Supercharge Your PHP Code with PEST Mutation Testing

PEST has seen a decent amount of adoption, especially since it has become the de facto testing library for Laravel users, giving the option for Laravel developers to choose between PHPUnit or PEST, running PHPUnit under the hood. More choice can only be a good thing, but the adoption rates also mean a development lifecycle that has seen plenty of new features being shipped.

The rate at which mutation testing caught on was surprising, given that nothing much had happened in the world of testing since Domain-Driven Design and Behavior-Driven Development. I remember being blown away at a demonstration of BDD automatically parsing Gherkin files and spitting out tests from it by Ciaran McNulty at PHP London years back. It’s something I’ve never really done professionally, but once mutation testing started being adopted by PHP developers following the popularity of InfectionPHP, I knew it would be something I’d see high value in.

And so, with mutation testing becoming more popular, Nuno Maduro announced that PEST 3.0 would feature mutation testing. In this article, we’re going to dive into it with an example test using the Vonage Messaging API.

What is Mutation Testing and Why Use It?

Let me start off by saying that any tools you can add to make your code as robust as possible are always going to be a positive. So, you can add it to the list of essentials, such as Static Analysis with the likes of PsalmPHP or PHPStan, linting with something like phpcsfixer, and robust tests with Test-Driven Development.

We are humans, and as such, make mistakes in our tests (I know I have). There’s plenty to go wrong–imagine you’re in the process of writing a test and it doesn’t have any assertions, you just want to test that whatever code has run without throwing an exception.

public function testService()
{
	$service = new \Vonage\ServiceContainer('dev');
	$service->runService();

	// If service fails, exception, so dummy completion assertion
	$this->assertTrue(true);
}

What you’ve got there is a false positive. Sure, you can see the reasoning behind it, but you also have

100% Line Test Coverage

Technically correct, but that’s not a real test. What if the underlying code changes in such a way that the undocumented exception that might be thrown can no longer happen?

What mutation testing does is change the underlying code, one thread at a time (we call these threads “mutants”). If you start modifying the code, technically, all of your tests should fail. We’ve already introduced the human element to it, where we know that might not be the case. The end result we’re looking for is that many, many mutants have been created, but all of them have been “killed,” i.e., the test has failed because of the mutation. If you have mutants that “survive”, your tests are still passing. If they’re still passing, it means that the tests haven’t noticed any changes in your code. That’s not good.

Our Baseline: Vonage PHP SDK Test

Let’s show how we implement a regular test first, by having an imaginary Laravel service that sends an RCS Message using Vonage.

<?php

namespace App\Services;

use Vonage\Client;

use Vonage\Messages\Channel\RCS\RcsText;

class RcsService
{

   public function __construct(
       private Client $vonage
   ) {}

   public function send(string $to, string $message): array
   {
       $response = $this->vonage->messages()->send(
           new RcsText($to, 'VonageApp', $message)
       );

       return $response;
   }
}

For simplicity, we’ll say that Laravel’s service container injects a pre-configured Vonage Client object with credentials into the constructor. Now we need a test for it that doesn’t actually make an API call. We can do this using Mocks.

use App\Services\RcsService;
use Vonage\Client;
use Vonage\Messages\Channel\RCS\RcsText;
use Vonage\Messages\Client as MessagesClient;

beforeEach(function () {
   $this->messagesClient = Mockery::mock(MessagesClient::class);
   $this->vonage = Mockery::mock(Client::class);
   $this->vonage->shouldReceive('messages')->andReturn($this->messagesClient);
});

afterEach(function () {
   Mockery::close();
});

test('Will send message using Vonage SDK to end number', function () {
   $response = ['message_uuid' => 'abc-123', 'to' => '33600000000'];

   $this->messagesClient
       ->shouldReceive('send')
       ->once()
       ->with(Mockery::type(RcsText::class))
       ->andReturn($response);

   $service = new RcsService($this->vonage);
   $result = $service->send('+33600000000', 'Hello world');
   expect($result)->toBeArray();
});

OK, well, the test will pass. Good good. What happens if we mutate the code, though?

Mutation testing is built into PEST 3, meaning we can run mutations on this right now and see what happens. There are plenty of configuration options for PEST, but the simplest way to instruct mutations to happen is using the covers() helper method. In this case, the following line is added to the test file:

use App\Services\RcsService;

covers(RcsService::class);

This tells PEST that when running this test file, go ahead and mutate the code in RcsService. To run PEST mutations with a pass threshold of 100%:

./vendor/bin/pest --mutate --min=100

We get the following result:

 Mutating application files...

  1 Mutations for 1 Files created

   RUN  app/Services/RcsService.php

  ⨯ Line 20: AlwaysReturnEmptyArray

  ---------------------------------------------------------------------------------------------------------------------------  

   UNTESTED  app/Services/RcsService.php  > Line 20: AlwaysReturnEmptyArray - ID: 5befe24d3b8d7f6f

       public function send(string $to, string $message): array

       {

           $response = $this->vonage->messages()->send(new RcsText($to, 'VonageApp', $message));

  -        return $response;

  +        return [];

       }

   }

  

  Mutations: 1 untested, 0 tested

  Score:     0.00%

  Duration:  0.30s

   FAIL  Mutation score below expected: 0.0 %. Minimum: 100.0 %.

Uh-oh, something is wrong. But what? Aha! Behold the awesome power of mutation testing, explained in the following steps:

• PEST modifies the RcsService() to always return an empty array

• PEST runs the test file as a mutant

• We want the mutant to fail, and thus be killed

• It doesn’t, it passes

• This means your test is too weak

So, why is it so weak? The conclusion is: you should be testing the payload coming back from Vonage. Aside from the fact that this Vonage API doesn’t ever actually return a blank array, you should be testing the payload that is returned. Using ->toBeArray() in this case is not enough.

Fixing Your Weakness

The test actually defines the mock output that is given, so it’s safe to assume that what we’re testing for here is that the returned payload (that we provided it) isn’t changed in any way. As we have defined $response, we should change the test to ensure it matches that exact data structure.

expect($result)->toBe($response);

That’s it. One line, just changing that assertion. Running PEST mutations again, we get the following:

Tests:    1 passed (2 assertions)

  Duration: 0.26s

  Mutating application files...

  1 Mutations for 1 Files created

   RUN  app/Services/RcsService.php

  ✓ Line 20: AlwaysReturnEmptyArray

  Mutations: 1 tested

  Score:     100.00%

  Duration:  0.34s

Wow.

Conclusion

I know that, for a starting point, this is a pretty basic example. However, the concept of what mutation testing provides is mainly what we’re covering here. Despite it being just a few lines of code, it demonstrates the sheer power of what mutation testing can do to harden your codebase. Consider using it with something like PHPStan, and PHP developers have never had it any better for shipping battle-hardened code.