Video quality gets worse after some time but not better again

Either my demands on the video quality are too high, or I don't get 4K videos smoothly to run. After some time, the quality is downscaled, and from then on never up again.