Symfony 4 custom deserializer returns empty properties in entity

I have a custom Symfony 4 deserializer

class CardImageDecoder implements EncoderInterface, DecoderInterface
{
    public function encode($data, $format, array $context = [])
    {
        if($format !== 'json') {
            throw new EncodingFormatNotSupportedException(sprintf('Format %s is not supported by encoder %s', $format, __CLASS__));
        }

        $result = json_encode($data);

        if(json_last_error() !== JSON_ERROR_NONE) {
            // don't bother with a custom error message
            throw new \Exception(sprintf('Unable to encode data, got error message: %s', json_last_error_msg()));
        }

        return $result;
    }

    public function supportsEncoding($format)
    {
        return 'json' === $format;
    }

    public function decode($data, $format, array $context = [])
    {
        if($format !== 'array') {
            throw new DecodingFormatNotSupportedException(sprintf('Format %s is not supported by encoder %s', $format, __CLASS__));
        }

        if(!is_array($data)) {
            throw new \UnexpectedValueException(sprintf('Expected array got %s', gettype($data)));
        }

        $cardInstance = new CardImages();
        $cardInstance->setHeight($data['h'] ?? 0);
        $cardInstance->setPath($data['url'] ?? '');
        $cardInstance->setWidth($data['w'] ?? 0);

        return $cardInstance;
    }

    public function supportsDecoding($format)
    {
        return 'array' === $format;
    }
}

The way I'm deserializing is pretty straight forward:

$json = '
{
"url": "some url",
"h": 1004,
"w": 768
}';

$encoders = [new CardImageDecoder()];
$normalizers = [new ObjectNormalizer()];
$serializer = new Serializer($normalizers, $encoders);

$cardImage = $serializer->deserialize(json_decode($json, true), CardImages::class, 'array');

/** @var $cardImage CardImages */
var_dump($cardImage);

However, I get this result returned:

object(App\Entity\CardImages)#158 (5) {
  ["id":"App\Entity\CardImages":private]=>
  NULL
  ["path":"App\Entity\CardImages":private]=>
  NULL
  ["height":"App\Entity\CardImages":private]=>
  NULL
  ["width":"App\Entity\CardImages":private]=>
  NULL
  ["movie":"App\Entity\CardImages":private]=>
  NULL
}

Now, if I were to do a dump, just before the return in the decode part of the decoder, I'd get this:

...
$cardInstance->setWidth($data['w'] ?? 0);

var_dump($cardInstance);

object(App\Entity\CardImages)#153 (5) {
  ["id":"App\Entity\CardImages":private]=>
  NULL
  ["path":"App\Entity\CardImages":private]=>
  string(8) "some url"
  ["height":"App\Entity\CardImages":private]=>
  int(1004)
  ["width":"App\Entity\CardImages":private]=>
  int(768)
  ["movie":"App\Entity\CardImages":private]=>
  NULL
}

Ignoring the not set properties(which I'm fine with), it should work quite nicely, but it doesn't.

For the life of me I can't figure out what's wrong.

Any help is appreciated.