This error means the host's computer (the person attempting to host a game) has a resolution higher than its graphics card can encode, and needs to be lowered.
Lowering the resolution
To alter the resolution of the stream, follow our guide on changing the resolution.
Issues with latest NVIDIA driver
There's a recent issue with the encoder on driver version 551.61 (released February 2024) which is listed as an open issue in the NVIDIA forums, affecting some 16 series NVIDIA GPUs. Downgrade to an older version of the driver for now.