Why does VNC Server in Virtual Mode on Linux appear to hang, show a gray screen or not start at all?

You see a gray screen if the desktop environment of the remote computer requires capabilities that are not available to the X server built-in to VNC Server in Virtual Mode (Xvnc).

This is the case for the default desktop environments used by Ubuntu 13.04+ and CentOS/RHEL 7+.

Using VNC Server 6.2.0 or later?

From VNC Connect 6.2.0 onwards we recommend configuring VNC Server in Virtual Mode to work with the system Xorg server, rather than the outdated version built in to Xvnc. More desktop environments (such as Gnome 3), applications and extensions will be compatible out-of-the-box. See how to set this up.

Using VNC Server 5.x or VNC Server 6.0.x - 6.1.x?

We recommend you install one of the following desktop environments known to work with Xvnc:

Xfce

LXDE

KDE Plasma Desktop 

In addition, older distributions which use Gnome 2 are also known to work well.

Each of the above should be easily installable on most Linux systems using the normal application installer. You can also do this from the command line using the package names mentioned in the table for Debian or Fedora-based systems - for other distributions you may need to search for the exact package name. There are also distributions which use these desktops by default (or example the Xubuntu or Kubuntu "flavours" of Ubuntu).

Custom xstartup files

If the desktop environment you've installed is not the default (for example, you've installed LXDE on a regular Ubuntu system), you will need to modify the xstartup script used by VNC Server to start your chosen desktop environment. Each desktop session should have a corresponding .desktop file in /usr/share/xsessions, which contains the command used to start the environment. You can use the following command to list the startup commands for all the available sessions installed on the system:

grep Exec= /usr/share/xsessions/*.desktop

Next, use a text editor to replace the existing xstartup script with the following template:

#!/bin/sh
DESKTOP_SESSION=name-of-desktop-file
export DESKTOP_SESSION
exec-command-from-desktop-file
vncserver-virtual -kill $DISPLAY

This should be saved to ~/.vnc/xstartup, or /etc/vnc/xstartup.custom for all users on the system, and made executable using chmod +x.

Below are some template xstartup files for common desktop environments.

Xfce

#!/bin/sh
DESKTOP_SESSION=xfce
export DESKTOP_SESSION
startxfce4
vncserver-virtual -kill $DISPLAY

LXDE

#!/bin/sh
DESKTOP_SESSION=LXDE
export DESKTOP_SESSION
startlxde
vncserver-virtual -kill $DISPLAY

KDE Plasma Standard

#!/bin/sh
DESKTOP_SESSION=1-kde-plasma-standard
export DESKTOP_SESSION
startkde
vncserver-virtual -kill $DISPLAY

GNOME

#!/bin/sh
DESKTOP_SESSION=gnome
export DESKTOP_SESSION
gnome-session
vncserver-virtual -kill $DISPLAY

GNOME Classic

#!/bin/sh
DESKTOP_SESSION=gnome-classic
export DESKTOP_SESSION
env GNOME_SHELL_SESSION_MODE=classic gnome-session --session gnome-classic
vncserver-virtual -kill $DISPLAY

xstartup scripts and VNC Server 6.2.0+

Please be aware that VNC Server terminates when the xstartup script terminates. Therefore, it is recommended not to use an ampersand (&) to background processes started in this script if you are running VNC Server 6.2.0+.

Note: This is a change to the behaviour of previous versions of VNC Server which did require use of the ampersand (&). The change was introduced to allow the use of the built-in Xorg to support the Gnome desktop environment. See this article for more information.

So for example, from VNC Server 6.2.0+ use: 

startkde

and not:

startkde &

Known issues

There is a known issue with the RENDER extension in current versions of Xvnc that causes problems with Xfce, resulting in the desktop failing to start. This can be resolved by either enabling System Xorg, or disabling the RENDER extension; to do this, create /etc/vnc/config.custom (if it does not already exist) and add the command:

-extension RENDER