Configuring RealVNC Server's Virtual Mode to use SystemXorg mode

By default, RealVNC Server's Virtual Mode uses its own built-in X server, which is based on an old version of Xorg. This can cause modern desktop environments (like GNOME) to fail to load, resulting in a gray or blank screen, and can prevent modern applications from working correctly.

This guide shows you how to configure RealVNC Server to use your computer's own modern Xorg server instead. This provides a much more stable and compatible virtual desktop experience.

Before You Begin: Compatibility

• Supported Systems: This process is highly recommended for modern RedHat-based systems (RHEL, CentOS, Rocky Linux) and Ubuntu.
• Unsupported Systems: You should not follow these instructions on systems running SLES or Raspberry Pi OS. Continue using the default built-in server (Xvnc) for those platforms.
• Requirements:
  • RealVNC Server version 6.2.0 or newer must be installed.
  • You must have root or sudo access to your system.

Step 1: Install required packages

First, ensure the Xorg server and its dummy video driver are installed. Open a terminal and run the command for your Linux distribution:

• For Red Hat / CentOS:

  sudo yum install xorg-x11-drv-dummy

• For Ubuntu:

  sudo apt install xserver-xorg-video-dummy

Step 2: Enable SystemXorg for VNC Server

Next, run the following command in your terminal to tell RealVNC Server to use the system's Xorg for all users:

sudo vncinitconfig -enable-system-xorg

The script will automatically offer to copy your existing VNC Server settings to the new configuration files.

To enable SystemXorg for an individual user, have the user run the command without sudo.

Step 3: Customizing your Virtual Mode session (optional)

After enabling SystemXorg, you can customize various aspects of the virtual session.

Configuring RealVNC Server Parameters

To change RealVNC Server parameters (like Encryption or Authentication), edit the appropriate configuration file.

For system-wide settings, change:

/etc/vnc/config.d/vncserver-x11-virtual

For user-specific settings, change:

$HOME/.vnc/config.d/vncserver-x11-virtual

Customizing Screen Resolutions

The recommended way to use custom screen resolutions is to install the RealVNC dummy driver and set the RealVNC Server RandR parameter, for example RandR=2560x1440,1920x1080, or to enable Dynamic Resolution to automatically resize the session.

Alternatively, to add custom screen resolutions, you must edit the Xorg configuration file used by VNC Server.

• File Location: /etc/X11/vncserver-virtual-dummy.conf
• Instructions: You can enhance this file by either adding individual Modeline entries for new resolutions or by replacing the entire contents with a more comprehensive configuration tuned for performance and multiple displays.

For example, to add support for a 1440p monitor, you would add the following line to the Monitor section of the file:

Modeline "2560x1440" 312.25 2560 2752 3024 3488 1440 1443 1448 1493

Note the RealVNC Server RandR parameter cannot be used to enable a connected RealVNC Viewer user to specify custom screen resolutions for a session. The connected user should use the standard Screen Display or Monitor app instead. Alternatively, you can install the RealVNC dummy driver to restore use of the RandR RealVNC Server parameter.

Defining startup Applications (xstartup)

You can continue to use xstartup scripts to specify which desktop environment and applications launch inside the virtual desktop, in the same way you would with the old Xvnc server.

Step 4: Updated Xorg config settings (optional)

Custom Xorg settings can be specified in the new /etc/X11/vncserver-virtual-dummy.conf file.  By default, RealVNC creates a basic Xorg configuration for its virtual sessions with limited parameters.

Step 5: Setting Xorg command-line options (optional)

For advanced troubleshooting or specific needs, you can pass command-line options directly to the Xorg server when it starts. These options are set as parameters in RealVNC Server's legacy configuration files:

• System-wide: /etc/vnc/config or /etc/vnc/config.custom
• User-specific: ~/.vnc/config
• Custom Settings vncserver-virtual -config FILE-PATH flag at start-up.

Important: The options you add are passed directly to the system's Xorg server. Invalid options will cause errors. Consult the Xorg man page (man xorg) for a list of valid options.

Note: The following options, which were supported by the old Xvnc server, are not supported by the system Xorg server and should not be used:

• -blackpixel
• -whitepixel
• -linebias
• -pixdepths
• -pixelformat
• -screen

For additional information please reference here.

Troubleshooting & Common Issues

Fixing Small Fonts

If fonts appear too small in XFCE or KDE, you may need to force a standard DPI setting.

• For XFCE: Go to Applications > Settings > Appearance > Fonts and enable Custom DPI setting, setting it to 96.
• For KDE: Go to System Settings > Fonts and enable Force fonts DPI, setting it to 96.

KDE Keyboard Layout Issues

If you encounter problems with keyboard input in KDE, disable the layout manager found at System Settings > Input Devices > Keyboard > Layouts.

How to revert changes

If you need to switch back to using the old, built-in X server (Xvnc), simply run the following command:

sudo vncinitconfig -disable-system-xorg