I don't think it's a good idea to unfollow people you don't like. You shouldn't stay in an echo chamber all the time.
That's true, mostly I was talking about if you just disagree with them and don't want to listen to them
That's fair. I think your post isn't worded the best as at least for me when I think of people I “don't like” it usually is people who I think are bad people due to their actions and/or how they have treated me specifically. When I think of people I disagree with I usually think of people who just have opposing viewpoints to my own, whether that is politics or something as simple as which browser is best.