By default ivy/swiper/counsel (not sure which) will visit all results as I scan through a list, e.g. when going through my buffers. I thought I liked this until working on my exwm with multiple monitors and non-emacs buffers are only allowed to appear once, meaning that simply moving down my selection list caused screen to be stolen from my other monitor. Answer Buffer-changes are apparently an effect of using counsel-switch-buffer, which I had bound to C-x b.