You can't use custom costumed NPCs in the actual Grant Mission Dialog box.
However, you can (and many authors, myself included, do) add a storyboard dialog tree in between the Grant Mission Dialog and the Map Transition Dialog to the first map. This dialog tree is shown immediately after you accept the mission in the Grant Mission Dialog, and you can build it like any other storyboard dialog tree, including using custom costumed NPCs in it. I presume this is where you actually saw the custom NPCs used.
What we usually do is this: Grant dialogue is a boff that says, Captain so and so is calling. This is followed by a popup dialogue in the storyboard prior to the first map transition. So, you can have a long interaction, but it is introduced quickly by a non-customized npc.
Comments
However, you can (and many authors, myself included, do) add a storyboard dialog tree in between the Grant Mission Dialog and the Map Transition Dialog to the first map. This dialog tree is shown immediately after you accept the mission in the Grant Mission Dialog, and you can build it like any other storyboard dialog tree, including using custom costumed NPCs in it. I presume this is where you actually saw the custom NPCs used.
My Foundry missions | My STO Wiki page | My Twitter home page