I see in my reputation history that I have been removed on January 29th, only three days after my last activity on the site (accepting an answer).
Was this a mistake or are users removed after just a few days of inactivity?
Users get removed for a variety of reasons:
If you want to post the specifics or flag this and provide us the moderators with more information to see if we can even determine which account was deleted, we can look into the specifics.
As you see, the links I have provided are for the main meta, since deletion is about common code and rules for all Stack Exchange sites but in some cases a site moderator has slightly more visibility into specifics on their site.
Since you mentioned reputation, I checked yours at this page:
And there is an entry:
That is what you will see when a user that had given you reputation is removed from the site - taking that reputation back. I too lost reputation on that day (-50 points), but since voting is anonymous and deletion isn't something even moderators can audit or look back to see what happened, it's likely that one user that was here for a while left (causing several people to lose reputation) although it could also be several automatic deletions being handled as a batch and reputation adjusted once to balance the numbers.