Why is Bastard a slur?

I recently have been reading a book where the main character is called "bastard" a lot. It got me thinking. I've always understood "bastard" to be a slur for an illegitimate male child... as if being unluckily born into a non-traditional family is something to be ashamed of.

What I don't understand is why there is no slur for someone who fathers an illegitimate child. He's the one who probably has something to be ashamed of.