To reproduce:
- Open a file with many lines
- Enable display-line-numbers-mode
- Hold down C-n

On my T480s (i5-8350U) in power-saving mode, this results in choppy cursor
movement.

The built-in profiler shows that moving the cursor down through my
1200-row init file results in 25,174 redisplays when
display-line-numbers-mode is enabled, compared to 2,193 when it is
disabled.

I have attached the profiler outputs.