No one likes to feel like they're losing themself.

And Madison probably already felt a little lost and swallowed up by things by being a seer. I'm not surprised he's stubbornly clinging onto the identity he has when it feels like he already had a lot taken from him.