Why does this happen?

When connecting to a Windows computer you may notice that the list of resolutions in Windows' Display Settings is limited to a maximum of 1024x768 or 1280x1024.

This is caused by a limitation of Windows - since it cannot detect a connected display, a limited selection of "safe" resolutions is offered.

Is there a way to avoid this issue?

Connect a monitor to the RealVNC Server device and ensure it is powered on

If you are connecting to RealVNC Server on a physical computer, connect a monitor and ensure it is powered on. If you are connecting to a Virtual Machine, ensure that the Virtual Machine is configured to have an active graphical adapter and monitor device.

Connect a display emulation device to the RealVNC Server device

If you are connecting to RealVNC Server on a physical device, you can connect an EDID emulator to the RealVNC Server device. An example EDID emulator can be seen here.
Note: RealVNC has not tested this particular emulator and it is only provided as an example

Configure Windows to use a different resolution

It may be possible to force Windows to provide a specific resolution. This is usually only possible on desktop computers and not laptops.

PLEASE NOTE: The below option involves editing the Registry, and should only be attempted on a non-production/non-critical machine, before rolling out to your production environment. We recommend backing up the Registry prior to making any changes.

In this example, we will be setting a resolution of 1920x1080.

1. Open Regedit
2. Navigate to HKLM\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration
3. Select the key(s) that start with the name SIMULATED (or NOEDID if SIMULATED does not exist) and then select the child key named 00
4. Change the value PrimSurfSize.cx to the desired horizontal resolution (1920). Be sure to select 'Decimal' when changing these values.
5. Change the value PrimSurfSize.cy to the desired vertical resolution (1080).
6. Select the child key of 00 also named 00.
7. Change the value of ActiveSize.cx to the same horizontal resolution as before (1920).
8. Change the value of ActiveSize.cy to the same vertical resolution as before (1080).

After a reboot, the simulated display should now be at the resolution you set.

This method was sourced from https://superuser.com/questions/990398/setting-display-resolution-beyond-1024x768-with-headless-windows-10).