Why does this happen?

When connecting to a Windows computer, you may notice that the list of resolutions in Display Settings is limited to a maximum of 1024 x 768 or 1280 x 1024.

This is a limitation of Windows - since it cannot detect a connected display, a limited selection of "safe" resolutions is offered.

Is there a way to avoid this issue?

If possible, connect a monitor to the VNC Server computer and ensure it is powered on. If this is not an option and the computer is virtual, or has to be run headless, there are some methods to work around this, listed below.

Connect a display emulation device to the VNC Server computer

If you are connecting to VNC Server on a physical computer, you can connect an EDID emulator to the VNC Server computer. An example EDID emulator can be seen here
Note: RealVNC have not tested this particular emulator and it is only provided as an example

Configure Windows to use a different resolution

It may be possible to force Windows to provide a specific resolution. This is usually only possible on desktop computers.

PLEASE NOTE: The below option involves editing of the Registry, and should only be attempted on a non-production/non-critical machine, before rolling out to your production environment. We recommend backing up the Registry prior to making any changes.

Manual method

In this example, we will be setting a resolution of 1920 x 1080.

1. Open Regedit
2. Navigate to HKLM\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration
3. Select the key(s) that start with the name SIMULATED (or NOEDID if SIMULATED does not exist) and then select the child key named 00
4. Change the value PrimSurfSize.cx to the desired horizontal resolution (1920). Be sure to select 'Decimal' when changing these values.
5. Change the value PrimSurfSize.cy to the desired vertical resolution (1080).
6. Select the child key of 00 also named 00.
7. Change the value of ActiveSize.cx to the same horizontal resolution as before (1920).
8. Change the value of ActiveSize.cy to the same vertical resolution as before (1080).

After a reboot, the simulated display should now be at the resolution you set.

This method was sourced from https://superuser.com/questions/990398/setting-display-resolution-beyond-1024x768-with-headless-windows-10)