As per your comment on bmikes answer (this is too long for a further comment which is how it started, hence now an answer), there is a sort of possible tedious way of making an educated guess in some circumstances, but it relies on the assumption of at least having a shortlist of who to check based on recent activity, and therefore being able to do the checking immediately after the voting happens, unless you want to monitor everyone's activity at the same time on the fly against your Rep etc which simply isn't practical. I got serially upvoted 5 times in 2 mins not long back and took a stab at rounding it down to 3 or 4 people, but nothing was conclusive.
Unless it's sufficiently targeted for the systems algorithms to pick up (no, don't ask us what they are, we don't know) at which point we get alerted automatically, then no-one can ever conclusively say who voted for who and when. And this is a good thing, because even if we had the info, what are we going to do about it? No-one should be forced to explain or justify their participation on an SE site.
I got a downvote on an answer I really liked a while ago, and I'm 100% certain who it was because they left a comment at the same time (be sure, leaving a comment is not a requirement, or even a a polite nicety, but purely optional, just as you can comment without voting); I'd love them to revert it, not because I care about the rep, but because they simply read the answer wrongly and misunderstood it and therefore said it didn't work, which could lead others to think it's a wrong answer too. All I could do was explain as much in a further comment and leave it at that.
You've got close to 4000 rep, therefore you clearly have a lot of posts that someone could downvote. These 6 points are like not even 0.2% of your Rep. Each of the answers in question had received at least 1 downvote already, and 1 has had a further on since, so it's reasonable to assume that it's not necessarily a surprise to get them - I suggest you just let it be, and let them go.
My last comment is merely this - if someone downvotes you (the Royal you, i.e. this applies to everyone), please do not comment asking for an explanation, or asking for the voter to own up. That's not fair, and I would remove any such comments as inappropriate. Take the rough (there isn't much) with the smooth.