They don't. What gets in the way is how shoddy our memories are from being drained for so long.

I'm the only one to remember my name and body, that's a recent thing. [He pushes away from the wall, coming over to stand near by but not too close. He hasn't seen any sort of reason to be wary but it never hurts. He vaguely remembers a lot of shouting back when he first died.]