I haven't looked in to it with BG III, but for tabletop 5e, Mage Armor counts as armor and Unarmored Defense only works if the Barbarian isn't wearing any armor.
Mage Armor: The target's base AC becomes 13 + its Dexterity modifier.
Unarmored Defense: While you are not wearing any armor, your Armor Class equals 10 + your Dexterity modifier + your Constitution modifier.
Normally, when you have two competing armor calculations the player chooses which to take. Obviously, everyone is going to choose the best option and I assume that's what's happening in BG III.