The name of the field where this is selected has changed over time. It has been called "User sessions as managed by" and also "Browser Sessions ". Either way, find the option and select it.
The solution / workaround:when running RM 12.3.1 or earlier: Set the RM session idle timeout to be equal to SSO token expired time + 60 minutes (or some big value like 999999999). In this case the SSO expired dialog will always be raised before RM session will expire