Production Developer Blog: Feedback Prioritization

nitocris83

Sometimes Dev responses on the forums become so lengthy that they become blogs! A common question we often get is how is community feedback processed (or if it even is processed) so Producer @terramak put together this overview that we hope helps provide some insight into Production and Feedback. Please note that it is not meant to be the full and final process, we are continuously adapting to new needs or factors, and is written from the perspective of the Production aspect of Development.

http://www.arcgames.com/en/games/neverwinter/news/detail/10470163

thefabricant

It would be interesting to know, as a follow up to this, how player feedback could be improved upon to better facilitate bug fixes etc. For example:

1) Is it more helpful if we post bug reports individually or do you prefer threads with a large number of bugs posted at once.

2) Is it helpful if we speculate on the cause of bugs, or would you rather we just post a, "how to replicate," and leave it at that.

3) If a bug cannot be reliably replicated, what is the best way to report it?

Also, some things from the devs side that might help is clearer tooltips and explanations of mechanics. In the example given in the article, fair enough, you used a commonly used abbreviation for aspect of the pack, however, in the case of a bug report I made recently, there was a misunderstanding between the dev who responded and myself purely because that particular mechanic isn't explained anywhere in the game and so we as players had to invent our own terminology for it. If the mechanic was explained elsewhere or the devs provided some form of terminology it would not be an issue.

terramak

Regarding your last paragraph: That's totally legit, and though I doubt we'll ever be able to clear up all potential communication hurdles, having fewer unexplained mechanics and clearer tooltips on our end would certainly improve the situation.

The player feedback I've seen is, in general, pretty good and clear, and the answers to your questions are kind of case-by-case, but I'll do my best to answer them regardless. Note that these are all from my perspective, and how my brain and reading style work. As individuals sometimes process things differently, your mileage may vary.

Note: "Repro" in my personal jargon means "Steps to reproduce an issue (cause it to reliably happen)," and "Repro case" means "A case in which an issue reliably reproduces."


1) Report bugs individually or grouped in threads?

Individual bug report threads are good for the following situations:

• When the details of a bug are not easily summarized, so it must be broken out into several lines / paragraphs.
  
• When inviting discussion on a bug itself.
  
• When speculating on the potential causes of bugs, and relationships of bugs with other previously-identified ones.

The main drawback to this type of report is that it's easy to lose / miss some of the discussion involved, and if someone hits the nail on the head midway through a 4-page discussion, it's less likely for a person investigating the issue to notice.

If I analyze my own behavior correctly, and am correctly projecting it onto others: The first few posts and the final few posts on the thread at the time of viewing are the most likely to be seen and read more thoroughly.

Threads with a large number of bugs are good for the following situations:

• When summarizing a set of issues that follow a similar theme. A recent example of a very useful case was the sets of bug reports for the Devoted Cleric changes.
  
• When summarizing issues reported in other threads that one feels should get more attention. (Links to those threads make it much easier to dig in / drill down when necessary.)

I end up making these sorts of threads more frequently with the Patch Notes threads for given releases on our preview shards. Generally, people want to bring my attention to issues I hadn't yet acknowledged in previous threads - In that case, one-line "In X situation, when I do Y, Z happens" reports are super helpful.

It gets a little unwieldy once there are more than 3 pages (That's a guideline, not a rule!) - and I will admit, once a thread gets into heated debates on whether a class is overpowered or whether X bug should be prioritized over Y, it becomes much more difficult to piece together the information on what is actually broken.

Summary:
Individual report threads are best for digging in and figuring out a given issue, and keeping the discussion focused.
Bug report compilation threads are most useful for broadly assessing the status of a feature, a class, etc., and for generating a set of fixes. These most benefit from being dense in directly usable data, with concise summaries that may even include repro.



2) Speculation on cause of bugs?
This can help or hurt, depending on who's reading it.

The benefit of speculation is that it can encourage lateral thinking, and get someone reading it to consider potential vectors of reproducing / fixing an issue that they wouldn't have thought of otherwise. I find the most useful speculation to be, "This feels similar to a bug with that happened before, I think it was fixed mid-2015," as a savvy developer can potentially dig through historical data to figure out how a bug was identified and fixed.

The drawback of speculation is that it can easily become a red herring. For example, with the Great Weapon Fighter's Unstoppable bug, there was speculation that it was related to the Temporary HP that was gained when activating Unstoppable. This turned out to be half-right; the bug was absolutely introduced in the same build (even probably the same change) that we added Temp HP to the power, but it wasn't directly related, and as a result we ended up spending a bit more time than we should have following up on the Temp HP angle. It wasn't a bad guess - but we ended up following the wrong track for a bit.

Summary:
Speculation seems to be most helpful when noticeably similar to other issues. But I don't have enough data to really back that up, so speculate away. (Conciseness and clarity are still the best aspect of a bug report, though.)



3) How to report a bug that has unreliable repro?
This is a tough one, and I honestly don't have a solid answer - I struggle with it myself when writing up reports internally.
Ultimately, I think it comes down to frequency and severity, and ability to narrow down the circumstances surrounding it.

I'm gonna use the GWF Unstoppable bug as an example again, because it stumped us for so long. The sort of reports that helped, or would have helped:

• (Indicator of severity) This issue prevents me from reaching acceptable damage output during boss encounters, and is extremely frustrating when it occurs.
  
• (Indicator of frequency) This issue happens all the time when fighting the five boss dragons in Well of Dragons.
  
• (Potential elimination of repro cases) This issue happens more frequently in extended encounters, and less frequently when I'm not actively attacking.
  
• (Indicator of workaround) This issue seems to fix itself when I drop out of combat, or when I stop attacking for about five to ten seconds mid-combat.

It's a bit of a sticky issue, since that's also when speculation helps us reach good hypotheses, but the signal to noise ratio gets worse as a necessity. It's extremely difficult to tell, without knowing how all the pieces fit together, whether a given piece of information will be helpful or unrelated.

Summary:
When a bug has unreliable info, knowing how bad it is (severity and frequency) will help us determine whether pinning it down should be a priority, and knowing the circumstances in which it happens (and doesn't happen, or happens and fixes itself!) helps us develop better hypotheses we can test against for repro cases. Severity and frequency can also help determine the cause of an issue.



While I go on about conciseness, I need to practice it myself! Hope this was all useful, though.

Thank you very much for asking!


(Edit 3:44 PDT: Typo fix + addition of thanks)

thefabricant

Oh, @terramak thanks for responding, I didn't initially see this response but I really appreciate it

Unknown

This content has been removed.

terramak

@thefabricant - And thank you! Sorry I forgot to @ you.
@dragonwizard - You're right! In my role, I tend to focus more on bugs as active, provable detractors to a poor gameplay experience, but feedback is a necessity for us to take into account as well.

I don't have as good an answer for you on feedback, unfortunately. I know the best time to give feedback on a given feature is when that feature is being introduced on the NeverwinterPreview shard as part of a module release, and generally has its own feedback thread, but for features that have been in the game a while, it's a slightly different style of prioritization (though it has echoes of the bug prioritization).

Some tips I can give for giving feedback in general, though:

Clarity in purpose and subject
Example: "I feel like I have to spend too long in River District to make any headway on restoring a Mirage artifact weapon."

Being clear about what aspect of the game is being referenced, and what the problem is, can help frame your case for a change in the game.


Specifics, perhaps with precedent
Example (with made up values): "Arcanic Focus drops extremely rarely. With Sea of Moving Ice, I got restoration materials after about three hours of gameplay, and the drop rate seemed consistent. In River District, it takes me about twelve hours for the equivalent value, and the drop rate seems much less consistent. I did twenty dig sites before I even got my first!"

Having that comparison point gives us some data to look at, so that we can directly compare how one thing was set up versus another, and what tradeoffs we made in the updated gameplay experience. Not everything is expected to be equivalent or have the same experience, but it's important to see the impact of changes that were made to previous experiences.


Suggested changes (optional)
Example: "I could see a couple potential changes here. I'd recommend increasing the drop rates so that they drop on average one every four or five runs. If they're already fine on average, then maybe you could give some consolation reward that lets me still make progress if I get unlucky with drops. Alternatively, having another guaranteed one from daily quests would let me try for more, but still keep the guaranteed progression slower if that's the intent."

Suggestions are a little weird, because it's hard to see the big picture of a given feature. So while the suggestion may work for a specific case, it might not work for the game as a whole. Still, seeing the types of things a person suggests can give a developer insight into what types of solutions the player's hoping for, or what they'd find palatable. It can also give insight into where that line is between "rewarding replaying" and "encouraging a boring grind" in a case like the example above.


Other Stuff
I also find it very useful, both when giving feedback and receiving, to make it clear that I'm giving candid feedback on the feature and not passing judgment on the people involved. It's totally okay to give visceral emotional reactions, as long as they're tempered with constructive criticism.

Using an entirely fictional example:
"I didn't have fun playing through Terramak's Terrace of Terror. I liked the ambience at the start, it had a strong feeling of creeping dread. I think the designer was going for a frantic feeling about five minutes in, when sending wave after wave of wererats, but when playing solo I couldn't reasonably kill them in time. They basically swarmed and killed me without any chance to react. It was even worse when I released - The respawn point was right in the middle of the pack, so they killed me instantly again. I was so frustrated at that point that I ragequit."

If I were the developer of that fictional quest, this is the kind of feedback I'd like to see. From this, I understand that the player giving the feedback was trying to appreciate the content, and that some of the stuff I was doing worked - The ambience of the first five minutes of gameplay, at least. It's clear that the wererat swarm in my content isn't working, and that the respawn point is in a bad spot for the gameplay. And it's clear that the intended mood is broken by the frustration generated by the above issues.

"Your content is bad and you should feel bad!" is a bit of a silly example, but it's more difficult to take feedback for what it is when a personal judgment is made.


And I just wrote up another much longer post than I intended to. Sorry! For your final question, I think QoL feedback works best in categories. "VIP improvements" in one, "UI and HUD element transparency requests" in another, "UI default size" in a potential third thread.

Thanks again!