season 4 episodes 15 and 16 pandora/linchpin is the episode where we meet the female CIA agent that Castle Followed for a year or so ~10 years before.
in the 2nd episode(linchpin) the Agent says -essentially i do not have the exact quote- the only reason why he(Castle) was allowed access to her(CIA Agent-who ended up being a double agent/the bad guy) was because of who his father was. castle goes, you know who my father is, and she just looks at him, says something like naturally. i will review that episode tonight to double check my recall.