This error means the host's computer (the person attempting to host a game) has a resolution higher than its graphics card can encode, and needs to be lowered.
Lowering the resolution
To alter the resolution of the stream, follow our guide on changing the resolution.
Issues with NVIDIA driver
There's an issue with the encoder affecting 16 series NVIDIA GPUs on driver version 551.61 (released February 2024) which has been fixed on version 551.76 (released March 2024). Update your drivers to resolve the issue.