I know this isn't an easy thing, but I'd like to put some ideas out there with regards to how players interact with NPC enemy critters such as mobs and bosses.
I'll start by examining the way things are now.
What we currently have seems to be very simple scripts that mobs and bosses follow, there also seems to be some simple path finding critters follow to get and avoid combat advantage while engaging with players. Critters generally have a very small set of combat abilities and they rotate through them with minimal intelligence if any at all. Everything is very predictable all the way down to boss phases and group positioning. Generally encountering a new enemy, boss or otherwise, is a matter of figuring out the "patterns" and reacting accordingly. While this has been the modus operandi of video games since the arcades, I think we can do better.
It would be really cool to be surprised from time to time with how critters behave. How can we achieve this? What one could do is start by giving critters a larger tool kit (more abilities), and group them into sets of "tactics", they could have abilities that are offensive, defensive, evasive, summoning, buffing (offensive, defensive), and some confusion type of abilities (mirror image, CC). Each type of critter would ofc have ability sets tailored to their particular personalities, and some would have more abilities of one tactic than another, or even lacking entirely one or more tactics while favoring one or two. The critter would use some simple AI or heuristics to determine which tactic to take depending on the situation and that would restrict it's skill set while in that tactic. A critter could "choose" to switch tactics at any time, but cooldowns would have to transfer over into the new tactic's set of abilities (think of slots having the cd instead of the actual ability). I'm not an AI expert but I've messed with it a bit, and I know there are some well known algorithms that can have emergent properties such that enemies will start to behave as though they are cooperating, when in fact they aren't, if that's achievable I think that would be great, one critter could choose a particular tactic such as cc/confusion, while another chooses defensive, and another offensive, and they should appear to coordinate. I think some emergent tactics in Solo boss fights could appear in a such a way that a Boss takes advantage of switching tactics despite CD transitions, in order to say CC and Buff, or Buff and then become Offensive.
With regards to Phases in Boss fights, this could also incorporate a change in the abilities that make up their "tactic sets", and some interesting things could be done with that. The key is to use an algorithm or heuristic that allows the critter to change tactics based on the situation, and this would also be kinda cool say if we had a "Hard Core Mode" that had an option to add random spawns throughout a instance, like in the Barovia hunts, that would change the dynamics of some bosses tactics on the fly, suddenly they would have minions to "cooperate" tactics with.
Anywho, this was just kind of a stream of thought,
@cwhitesidedev#9752 maybe you can get some ideas from this, either way, Thanks for reading