How do I use VNC Server in Virtual Mode on Linux in conjunction with the system Xorg server?

By default under Linux, VNC Server in Virtual Mode uses a version of the Xorg server built-in to Xvnc. This is old, and hard to update. On some platforms, modern desktop environments fail to load (resulting in a gray screen), and modern applications and extensions (particularly those using hardware acceleration) do not work.

From VNC Connect 6.2.0, you can configure VNC Server to utilise the latest version of the Xorg server present on your system instead. More desktop environments, applications and extensions will likely be compatible out-of-the-box, giving a much better user experience, especially on Red Hat-compatible distributions.

Note: If you are using SUSE or Raspberry Pi OS (previously called Raspbian), you should continue to use Xvnc. This means that under SUSE you will still need to switch desktop environment in order to avoid the grey screen. There's no issue under Raspberry Pi OS (previously called Raspbian), however, as the default PIXEL desktop environment works well with Xvnc.

Requirements

Xorg and the Xorg dummy video driver must be installed. For example:

• Red Hat/CentOS 7/8: sudo yum install xorg-x11-drv-dummy
• Ubuntu: sudo apt install xserver-xorg-video-dummy

In addition, VNC Connect 6.2.0+ must be installed and licensed. Download.

Enabling the system Xorg server

To enable the system Xorg server for all users, run vncinitconfig -enable-system-xorg as root. Run without root privileges to enable just for you.

To disable the system Xorg server and use Xvnc again, run vncinitconfig -disable-system-xorg.

Configuring VNC Server

VNC Server parameters

VNC Server parameters can be specified in the following new configuration files:

/etc/vnc/config.d/vncserver-x11-virtual (for all users)
~/.vnc/config.d/vncserver-x11-virtual (just for you)

Note that the vncinitconfig -enable-system-xorg script offers to copy over existing parameters from /etc/vnc/config.d/Xvnc and ~/.vnc/config.d/Xvnc respectively for you.

Note: The RandR VNC Server parameter cannot enable users to switch between available screen resolutions. See Xorg configuration, below.

Xorg options

Xorg server options can continue to be set in the following files:

/etc/vnc/config
/etc/vnc/config.custom
~/.vnc/config (ignored if the -config flag is used, below)

...or in a custom file identified by the vncserver-virtual -config FILE-PATH flag at start-up.

These options are passed directly to the system Xorg server. Consult the Xorg man page for a list of valid options; invalid options cause errors. Note that the following options supported by Xvnc are not supported: -blackpixel, -whitepixel, -linebias, -pixdepths, -pixelformat, -screen.

Xorg configuration

Custom Xorg settings can be specified in the new /etc/X11/vncserver-virtual-dummy.conf file.

Note the VNC Server RandR parameter cannot be used to enable a connected VNC Viewer user to cycle between available screen resolutions during a remote control session. The connected user should use the standard Screen Display or Monitor app instead. Custom resolutions can be specified using Modelines. Alternatively, you can install the RealVNC dummy driver to restore use of RandR.

Xstartup

Xstartup scripts can continue to be used to specify desktop environments and applications for virtual desktops in the same way as for Xvnc.

Installing from a network share

If you're following these instructions to install VNC Server on a central computer, then perform the following operation after step 5 and before step 6:

5a) Run the command vncinitconfig -virtual-xorg-conf as root to generate a /etc/X11/vncserver-virtual-dummy.conf file.

Known issues

1. When using the Xfce or KDE desktop environments, some fonts may appear small, depending on the screen resolution.
   • To fix this for Xfce, navigate to Applications > Settings > Appearance and enable Custom DPI setting on the Fonts tab, making sure this is set to 96.
   • To fix this for KDE, navigate to System Settings > Application Appearance and enable Force fonts DPI on the Fonts tab, making sure this is set to 96.
2. KDE’s keyboard layout management (KDE System Settings -> Input Devices -> Keyboard -> Layouts -> Configure layouts) can cause issues with certain keys/applications. Please disable this setting if you encounter any issues.