VNC Access permissions on Windows Laptop - Enforce Query Connect

We use VNC Server in multiple situations and normally we connect to a server where we do not want a "request access" prompt.

However, we have a couple of laptops where we want to grant user A access to User B's desktop, but only when User B is there and can give permission.

We have set the Query Connect property to true.

If we place User A in User B's admin group, it allows the login, but bypasses the "Request access" prompt. If we remove User A from the admin group and add User A to the Remote Access group, User A is automatically rejected and the "request access" prompt is never displayed.

Is this likely to be something with the Windows permissions? or VNC Permissions?

How would I go about setting this so that User A can log in, but only if User B is there to grant the permission for each session?

Frank