I don't think it's a good idea to unfollow people you don't like. You shouldn't stay in an echo chamber all the time.
I only ever remember unfollowing one person, and that's because their posts took up more than half my feed.