How to add prefix to some controllers using annotations?

I have at the moment two controllers.

• GitlabAuthController
• UserController

I need to add the path prefix api to UserController only.

Before I was trying following at annotations.yaml file:

controllers:
    resource: ../../src/Controller/
    type: annotation
    prefix: api

But this adds the prefix to all my controllers.

Is there any way I can add the exception for the GitlabAuthController?