When there is a stable increase in frametimes causing the frametime budget to be exceeded, extra prediction is eventually added to ensure rendered frames are predicted to their actual display time. This probably holds true if the cause is network or encoding/decoding latency as well.

For instance, if a frame takes 30ms to render instead of 10ms, it would need to be predicted an additional 20ms into the future to account for the extra latency.

At a glance SteamLink handles this properly, slowing ramping the Extra Prediction to compensate, as seen in the Advaned Frame Timings graph.

However, if the change in frametime is sudden, the extra prediction amount is not adjusted at all.

This results in a very unresponsive experience when there is a sudden increase in frametimes, and severe overshoot when going the other direction.

The intent was presumably to prevent it from reacting to transient spikes, but if slowly transitioning from 9ms to 50ms adds the extra prediction, maintaining 50ms after a sudden transitioning should probably do the same.

I've graphed out some numbers from IVRServerDriverHost::GetFrameTimings() to better showcase the issue: https://gist.github.com/Nordskog/e169f52064db8fc0ce1235be9186c738#steamlink