This article is only relevant if you are unable to connect to VNC Server after installation and licensing.
The Service Mode Server dialog and icon will not appear in an RDP session, so this is why you cannot see them.
Regarding Access Denied, this means that the user account that you are authenticating with does not have permission to access VNC Server. By default, only members of the local Administrators group can access VNC Server.
If you do not have the credentials for a user in the local Administrators group, you will need to set the Permissions parameter. Since you are using RDP, this will need to be done in the Registry. The relevant key is
You will need to create a String value named Permissions. To determine the value for the parameter, we recommend using our VNC Permissions Creator: https://www.realvnc.com/download/file/perm.files/VNC-Permissions-Creator-6.6.0-Windows-32bit.exe
For more information about the Permissions parameter please see: https://www.realvnc.com/en/connect/docs/server-parameter-ref.html#server-permissions