Its pretty ambigous, there is evidence both ways
1. In the arkham asylum/hospital, he finds records stating he was adopted, that his adoptive mother let her boyfriend abuse him, maybe even damage his brain. The butler (Alfred) at Wayne Manor earlier also tells him that his mother was fired for being delusional, and she was never pregnant. This supports the theory that his mother is nuts, lied and abused him, and he is not Thomas Wayne's son.
2. simultaneously, at the end of the file he steals, he finds a photo of his mom signed "Love your smile, TW". We assume TW is Thomas Wayne. Why would he send that kind of note to his mom, if he didn't have an affair with her? Did TW or his parents frame her to get rid of a problem bastard child?
3. Joker is an extremely unreliable narrator, entire segments of the movie we find out are completely false. His relationship with his neighbor is the obvious big one, but his whole comedy bit (he imagines a laughing audience, we see its empty), his time with DeNiro, etc. More subtly, we see a scene where his clown friend gives him a gun and sorta treats him like a son (weird), but later he's told by the manager that the guy claims Joker asked to buy a gun. The fat clown comes off as a scumbag though. Another subtle one was the garbage strike, we keep hearing on the radio that garbage is piling up everywhere (real and metaphor) but when Joker walks the streets its actually not piled up everywhere. Did he imagine that ("this city is dirty") or not?
So it really just depends on you as the viewer on whether you think Thomas Wayne is a mostly honest rich guy who is caught up by a crazy lady, or if he is a scumbag rich guy who got a maid pregnant and then fired her, made her appear nuts and had her committed, and maybe even abused his own son in that process.
Mrs. Haus
and where as it's dark as hell, and sad, it's also brilliant. 
