Why do I see a low resolution when connected to VNC Server for Windows on a headless/virtual machine?

When connecting to a headless/virtual Windows computer, you may notice that the list of resolutions in Display Settings is limited to a maximum of 1024 x 768 or 1280 x 1024.

This is a limitation of Windows - since it cannot detect a connected display, a limited selection of "safe" resolutions is offered.

However, it may be possible to force Windows to provide a specific resolution.

PLEASE NOTE: This requires manual editing of the Registry, and should only be attempted on a non-production machine, before rolling out to your production environment.

We recommend exporting a backup of the Registry Key(s) that will be edited.

In this example, we will be setting a resolution of 1920 x 1080.

1. Open Regedit
2. Navigate to HKLM\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration
3. Select the key(s) that start with the name SIMULATED (or NOEDID if SIMULATED does not exist) and then select the child key named 00
4. Change the value PrimSurfSize.cx to the desired horizontal resolution (1920). Be sure to select 'Decimal' when changing these values.
5. Change the value PrimSurfSize.cy to the desired vertical resolution (1080).
6. Select the child key of 00 also named 00.
7. Change the value of ActiveSize.cx to the same horizontal resolution as before (1920).
8. Change the value of ActiveSize.cy to the same vertical resolution as before (1080).

After a reboot, the simulated display should now be at the resolution you set.

(source: https://superuser.com/questions/990398/setting-display-resolution-beyond-1024x768-with-headless-windows-10)