FOSRestBundle's format_listener prevents http_basic challenge

I'm trying to setup my PHP/Symfony4/FOSRestBundle API to accept both JWT and HTTP-Basic authentication but the format_listener appears to be intercepting the UnauthorizedHttpException exception an serializing it thus preventing the auth header from being returned as expected.

I have things configured like so:

security:
    firewalls:
        api:
            pattern:  ^/api/
            stateless: true
            http_basic: ~
            entry_point: app.basic_entry_point
            guard:
                authenticators:
                    - lexik_jwt_authentication.jwt_token_authenticator
    access_control:
        - { path: ^/api/login, roles: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/api/,      roles: IS_AUTHENTICATED_FULLY }

fos_rest:
    unauthorized_challenge: 'Basic realm="%env(AUTH_REALM)%"'
    access_denied_listener:
        json: true
    format_listener:
        enabled: true
        rules:
            - { path: '^/api', priorities: ['json'], fallback_format: json }
            - { path: '^/', stop: true }
services:
    app.basic_entry_point:
        class: App\Security\Http\EntryPoint\BasicAuthenticationEntryPoint
        arguments:
            - '%env(AUTH_REALM)%'

The custom version of BasicAuthenticationEntryPoint extends the stock version to return JSON content along with the auth header.

JWT auth is working fine. If I disable the format_listener, without JWT I get the auth header in the response. What's the trick I'm missing to allow me to enable the format_listener and get that auth header in the response?