So I figured it was time to start a list of campsites that you can't switch loadouts in since it looks like there are several.
1. Rothe Valley main camp. 2. Rothe Valley Drow Slave Camp. 3. Master Spellplague. 4. Whispering Caverns Tunnels. 5. Ebon Downs: Ancient Barrow 6. MSP Just before 2nd boss