The mob tags to the raid. FTE has some oddities to it that don't match wow from 10+ years ago.
1. You cannot do anything at all to the mob that is not encounter locked to you. The name is grey and attempting to attack it or cast on it gives you a red "your target is in an encounter you're not a part of".
2. You can still heal and buff people who are locked to encounters.
3. You can /y off the mob to 'break' the encounter and allow it to become FFA.
4. If you get to the top of the mobs aggro list, it will still attack you. Once this happens, the encounter breaks the same as if you /y the mob.
The mobs also have some reset mechanics, where they reset after dropping combat. This can happen when you mez/blur an add or charm a pet. This is the source of the current pet bug where if you hold a pet longer than it takes for it to drop the encounter, it kind of goes into a weird state where you don't see the charm break message and the window doesn't go away.
The main issue is that there are binary choices being made at each step of the 'encounter locking' logic, and right now this is the least bad setup they could think of. What cupcaek was saying is that if mobs didn't interact with anyone outside of the encounter, then you could do things like keep all of your healers outside of the raid and have them healing without any aggro, etc.
