NOTE Apr 23: This thread was originally posted in the beta forums in mid-March, before I joined the Foundry beta. It should not be taken as gospel -- I wanted to preserve it mainly as a reference for all the COH Mission Architect data (which still holds true today). I will probably make an updated post after Open Beta Headstart to reflect the actual state of the Foundry.
Subtitle: "Why a good Search/Browse interface is more important than being able to (do a particular thing) in the Foundry"
Sub-subtitle: "Sheesh. Can this guy write any
more text? I'm already bored, and
I'm ****ing dead." (see page 4 of
http://www.rpgwatch.com/files/Files/00-0208/Torment_Vision_Statement_1997.pdf if you didn't recognize this
)
Hi there, fellow Foundry enthusiasts!
This really really long series of posts (mostly shamelessly cut-and-pasted from my earlier posts in Foundry-related threads) attempts to start discussion about various aspects of the Foundry that are
not related to its actual capabilities about creating adventures, but are (I think) just as important. If I wanted to TL;DR the entire post, I'd condense my points like so:
- relying on star ratings does not work in the long run
- relying on player reviews / hand-picked dev selections does not scale well at all to hundreds of thousands of UGC
- farming is a legitimate playstyle; trying to fight and ostracize farmers is only going to backfire, badly
- some suggestions about how to improve the EXISTING system (without having to rewrite the entire thing, which I realize is unrealistic)
BIG OL' DISCLAIMER
Obviously almost everything in these posts pertaining to NW is speculation, since the Foundry isn't live yet -- what we have seen in beta weekends was a work-in-progress. However, there is quite a bit of specific MMO UGC experience behind it. I tried to identify the issues that were some of the most problematic in City of Heroes' Mission Architect based on my 3.5 years of experience with the MA as an author/player as well as its surrounding ingame and out-of-game communities in City of Heroes (COH). The MA was Cryptic's first (and so far longest-lived) UGC system in a MMO that had close to 600000 (that's six hundred thousand) player-created campaigns published by the time COH shut down in Dec 2012. The Search/Browse interface also works
very similarly to NW's (current) implementation, hence this thread. If anyone wants my MA 'credentials' (not that it's particularly relevant here), shoot me a PM.
Now of course, the search/browse interface doesn't seem like it could be a source of many problems. It's there and it works, that's good enough... right? IMO -- not so much. It is the only Foundry interface towards the playerbase, and needs to provide them with tools that allow them to reliably find
new, good-quality content that they are probably going to like, while keeping in mind that everyone has different tastes. IMO paying attention to how the relevant parts of the interface (and its underlying systems, such as rating) work is absolutely crucial, and will decide what happens to NW UGC after the first year or so. Will the system become a sustainable content machine making NW into a unique gem in the MMO market, or will UGC in NW collapse into a subsystem that 99% of the players either ignore or only use for farming with a tiny niche of players trying to tell and play through stories, just as it was in COH (and based on input from my STO-playing friends, in STO)? And yes, I watched the GDC video everyone seems to link around here. The presenter said that the stories that are worthwhile to play will eventually "bubble up to the top" and get 5-star averages, and that's it. Quote: "it works really really well". Only this isn't what actually happens, especially after a year or two. They even admit the search system itself is barely used (47:00-ish into the talk), and people just play whatever is in front. IMO, improving the search system is a much better solution than the small-scale spotlight / featuring processes they've proposed during the talk.
This is a VERY LONG post that I tried to split into multiple parts dealing with the issues with a summary (TLDR) at the start to make it a bit easier to digest. Unfortunately, neither the problems nor the proposed solutions are simple enough to summarize them all in a single paragraph, so... yeah. I've always been really passionate about this (back on the COH forums too), so it's physically impossible for me to write short posts on the subject. It's like a geas or something.
I'm also pretty sure nothing in here is 100% new. Indeed, some of it was posted in the general and beta forums before (and then forgotten). I just tried to underscore some (probably well-known) issues with actual experiences - and sometimes data - about what worked and didn't work in a similar-scale UGC setting.
Problems:
- Four-star hell: why a completely rating-driven browsing system fails in symmetrical UGC systems
- Old vs new: how incentives can backfire, and why being first is (almost) everything
- Farming: why farmers are not the enemy
Solutions that work and what look like they'd work, but in fact don't work:
- Scaling: why player reviews, third-party sites and dev picks shouldn't be counted on as the ultimate solution
- Suggestions: how to mitigate these problems
Comments
TLDR: It may look like a system driven (almost) completely by user rating works fine. And it can, as long as there are very few pieces of player-generated content created by a small handful of players compared to the pool of content consumers (see also: the original Neverwinter games - or any mod community, really). However, when EVERYONE starts to create content... things tend to fall apart a bit, and not just because of Sturgeon's Law. The end result is that unless you somehow arrange to get a large number of 5-star reviews as a new author right off the bat, you may as well not bother.
Symmetrical UGC woes
The star rating system works in many asymmetrical content provider / consumer relationships: mod communities, book/movie reviews, etc etc. There is a comparatively small number of people pumping out content, there's a high barrier of entry to creating content, and a large community consuming and judging content (or in case of professional reviews, a large community following the lead of a few well-known reviewers, but that's not really relevant here). Still, there is a grey area even there - if you are looking for a player home mod in Skyrim, do you look beyond page 10 (12 hits per page) and how often do you select a new-and-untested 4-star mod when there's a tried-and-true 5-star alternative? Still, reviewers actually help a lot in mod communities like that (more on why they aren't really a solution for MMO-scale UGC later).
A MMO with a low barrier of entry toolset that's built into the game (COH, STO, NW) is different. Every player is a potential author, and many of them will at least want to try out the Foundry once, especially since in NW it's marketed as one of the most important features of the game (not the case in COH and STO). There'll be a massive number of quests and dungeons in the system, and many of them will be abandoned / broken / unmaintained / never intended for mass consumption / just plain unenjoyable; I'm not even talking about purposefully broken quests, spam, and offensive content (which I assume will be reported and removed eventually). Sturgeon's Law is in full effect! This will - other than making the "review queue" function probably unusable - suddenly make that 2-page browser jump to 200 pages (or probably even more) overnight. Some of this content will be good and get played by people, giving them 5-star averages, and lifting them above the mass of uncertainty. They'll also be pushed to the forefront, giving them even more exposure, more plays, etc etc; I'll write more about the 'early bird' problem later. What is important here is that there's suddenly 20 pages of 5-star average quests, followed by 50 pages of 4-star averages, finally an unending mass of 3-star and below content that nobody will ever touch. Now let's say you make a new quest, have 2 of your buddies playtest it, release it, you get those 2 aforementioned buddies to play it so it gets the requisite 5-stars to put you on the map, which gets 2 other people to play it in short order. They both vote 4 (or one votes 5 and the other 3) because while the quest is good, they had some problem with it -- probably something subjective like "there was a reference to Drizzt, and I hate Drizzt". You end up with the quest's rating at 4.5 stars, meaning your quest now shows up at the start of the 4-star pages*. Welcome to 4-star hell, aka "nobody will ever find your quest again". (I could also include some math mumbo jumbo here about just how hard it is to keep your average in the 4.5-5 zone if there's any player at all who gives you a lower-than-4-star rating, but you get the idea)
All authors go to 4-star hell
The term 'four-star hell' in COH was very real, and it got worse with time. The situation described above is actually really favorable compared to how it worked in practice, and is only likely to happen near launch. It is also something you can sort-of do something about: round up a few friends to play and 5-star the arc, and bam it's back in the 5-star zone! Then 1-2 people 4-star it and it's back in four-star hell again. Add a few years and a couple of hundred ratings on those 5-star quests in the first 10 pages, and even having a 5-star average with less than 40 plays (basically impossible unless it is a super-popular farm or you have a massive group of friends play it) will relegate you to page 50+, aka "5-star hell". This was the case in CoH after a year or two. It got so bad that authors actually made a pact to 5-star each others' arcs in-game regardless of actual quality, and use a separate scoring system for reviews.
As for the "people will just play random quests to get those 4-star and 5-star quests the exposure they deserve" argument - they won't. I've played Random Arc Roulette in COH, and 10 times out of 10 I ended up playing broken / unfinished / just plain bad arcs that I couldn't in good conscience rate higher than 2 (and I'm a VERY lenient rater, mostly due to the 4-star hell problem; in fact I made it a point to not rate arcs ingame if I'd give them less than a 5-star). Others posted on the forum about the same experience.
* It's true that a 4.5 rating would still keep your average at 5 in COH... but I didn't want to find an example where the rating dips to 4.4 or somesuch.
Getting good content to players in the Foundry - challenges and solutions
Handle: @zaphtastic
TLDR: When it comes to getting people to play your missions, momentum isn't everything... but it's very close. The authors that are there first and create decent-quality missions will likely become household names, while those who arrive several months (or years) later will have to fight a severe uphill battle to get anyone to even try their work -- even if it's much better than that of the well-established veterans'! Substantial DM incentives have the potential to make the divide much worse. In the long run, this makes the Foundry a stale thing for players, and a frustrating mess for new or unrecognized authors.
I was here first!
Quests and missions that have been there since the get-go (whether that 'get-go' is alpha, beta, or launch depends on when quests are wiped) will enjoy a significant head-start and amass a VERY sizable number of plays. Basically, if you're there at the launch rush (EVERY MMO has a launch rush with a sharp drop-off in players thereafter, NW will be no different), you'll get far more exposure than someone who comes in 3 years later. In Dec 2012, the only story-focused arcs with more than 100 plays were either the ones hand-picked by the devs, or ones that were made in the first ~2 months of the Mission Architect (as a reminder, the MA was implemented in April 2009). There were maybe a handful of exceptions of heavily marketed and super-popular story arcs that broke this trend, and that's not an exaggeration. Someone on the MA forums kept track of the Hall of Fame (story arcs with 999+ plays and a 5-star average) and there was precisely one arc that was created more than 2 WEEKS after the system went live; it was a great arc, it was very heavily marketed everywhere, and it was still created in the first 2 months. I personally had a storyarc that I considered 'decent' made in Aug 2009 (by that time a lot of people have stopped using the MA to search for stories, but it was still 'a thing', sort of the tail end of the Golden Age), and it picked up about 70 ratings by the end of 2012. A (IMO much better) storyarc I made in Oct 2011 had 18 plays by Dec 2012, and I can account for 15 of those as guildies / friends / arc reviewers I specifically asked to play the arc. Both of these arcs had a 5-star average, btw.
What does this mean? If you're a new author, it's going to be rough to 'break in', which can severely hamper enthusiasm and the influx of new blood. I've played many, many story arcs in COH that had 0-5 plays and were languishing in the middle of nowhere on the list -- and they were GREAT arcs, better than some of the ones with 5-star averages and 200+ plays! How disheartened would you feel if you spent several weeks putting an awesome quest together and nobody played it because you're a newcomer and your quest was played by 1 person who voted 4 (if it was even played in the first place)? Cults of personality are nice and all, but arc rating should not devolve into a simple popularity and 'who was here first' contest.
It really is all about momentum. If you have something decent on page 1 with a 5-star average, it'll continue getting plays and 5-star ratings... and chances are, people will probably call it the best thing ever, because who cares about random noname quest #345346 on page 3749?
Voting cliques
Voting cliques are bad, period -- whether it's 50 people upvoting a quest made by their friend (just because he's a friend) or 10 people deliberately 1-starring quests made by people they dislike on the forums (or just random people they want to grief). When the number of plays-per-mission is low (as it is expected to be here as well), getting even 2-3 people to one-star a quest basically guarantees that the quest will drop off the face of the earth. I don't think more details are needed here. This was a problem with COH, but was overshadowed by other unfortunate shortcomings of the system. Part of the reason it happened in COH was actually due to an incentivization technique...
A new element in NW is the completely F2P nature of the game. Anyone will be able to create multiple free accounts and have their own voting clique if they want. I assume there are safeguards to protect against this -- and if not, there should be.
The downsides of incentivization
COH had really weak incentivization (you got a tiny amount of tickets based on the stars you got from your play -- but you could farm up tickets equivalent to hundreds of 5-star plays in 30 minutes), and it still drove some authors to e.g. push their farming arcs to page 1 and maybe even mass-downvote story arcs that were in the way.
Now, I've been hearing about incentivizing authors in NW, and it worries me. I personally think the Astral Diamond tip thing is a really bad idea, since it further enforces and incentivizes the 'household names' to keep creating content that'll immediately get played (because people already follow them, notice they have a shiny name they got from being a level 20 DM, etc) and get them even more ADs. I'm actually OK with authors getting rewarded -- but I'm NOT OK with this working to stifle new authors even further. It's a simple win-more system that has the potential to split the Foundry community into haves and have-nots, and that's just bad all around. That's even if I'm being generous and ignoring the griefers that will no doubt spring up. DM rewards (as said in the GDC talk) are another form of this incentivization.
I seriously hope that content types that can be created will not be tied in any way to author GM levels. Not only is a new author at a massive disadvantage from the get-go simply because of arriving late, but if they have to make do with inferior tools to tell their stories, it'll just ensure nobody will play them (or worse yet, play them and give them a bad rating because they don't have some of the expected features). COH actually did this right, you could unlock almost every map and enemy type by playing through the equivalent of 2-3 Foundry missions -- you certainly didn't need to create your own feature-limited mission and get people to play it! Based on the GDC talk, this shouldn't be a problem, but it was never explicitly said one way or another, so I'm putting it in here.
COH had its own way of rewarding the authors of really popular storyarcs. If an arc got 999+ plays while maintaining a 5-star average, it was automatically put in the "Hall of Fame", giving the same perks as a "Dev's Choice" mission (see the scaling post below for the details of that system), basically allowing it to drop normal loot instead of tickets, which was probably better for profit if you were going to run them over and over. I'll let you guess what happened by the end of the game's life:
Marketing your own missions is another grey aspect of incentivization. In COH you didn't really have any incentives for having people play your stuff, and people still heavily marketed their arcs -- some authors even paying people (just ingame money, thankfully) to play their new arcs sometimes. I'm too much of a purist to do something like that, but I don't blame them... later in the MA's life you couldn't get people outside the author community / your own pool-of-friends-and-guildies to play an unknown storyarc unless you literally bribed them. Of course it's also possible to create an awesome out-of-game marketing campaign with bells, whistles, trailer videos, comic book covers, etc etc... which is nice and all, but does nothing for the 99.9% of authors that don't have access to talent or effort like that.
Here today, gone tomorrow
Unlike a typical single-player game (especially one that's no longer getting patches), NW is a MMO that's constantly being updated. These updates WILL affect the Foundry in many ways; they will add new features, fix bugs, possibly break things, rebalance events... in general, 'do stuff that affects Foundry content and needs to be dealt with manually'. To keep up, authors need to update their content and re-run them every patch to make sure everything is OK. No 3rd-party patches here!
The problem is, people come and go. Maybe they don't necessarily leave the game (realistically though, a lot of them will; churn is a fact of MMO life), but a lot of people may just get tired of the Foundry and the intense time investment it requires, or they take a break for RL stuff / new MMO coming out / any other reason. On one hand, this leaves a lot of potentially broken content in the system. On the OTHER hand, if this happens to a visible quest, it'll either get reported (and subsequently removed) or 1-starred enough times so it falls into the 4-star abyss. Now, if the author decides to return, he'll find a once-great quest lost and forgotten... and probably won't be able to restore it to its former glory (that's assuming it wasn't reported and removed permanently for being broken).
Of course, the biggest problem is if you have permanently highlighted quests (Dev Choice / Hall of Fame in COH) that are prevented from leaving the front page completely. In COH's case, this meant that the COH Mission Architect dev actually had to make sure himself that all the old Dev's Choice and Hall of Fame arcs were still operational after a patch, and apply necessary changes if they weren't. Neverwinter doesn't seem to have "permanent highlights" like this, but this is an issue to consider if it's added in the future.
Getting good content to players in the Foundry - challenges and solutions
Handle: @zaphtastic
TLDR: you should expect most of the Foundry content to consist of low-risk high-reward 'farming' quests that will have FAR more plays and 5-star votes than any story-focused quest. Farmers are NOT the enemy, and they should not be considered as such -- even if you personally dislike farming (I do, I don't plan on playing any such quests myself, and my own storyarcs in COH were heavily story-focused).
Farms, farms everywhere!
Throughout the life of the Mission Architect, the COH MA in-game interface for viewing player-made content was almost completely dominated by missions (or 'arcs' for 'story arcs') designed to get as much reward with as little risk as possible; especially a year or two into the MA's life after the 'shiny' has worn off from the MA, and many people stopped using it to play story-focused arcs. There were plenty of story-focused arcs and challenge-based arcs in the system, but the search tools (and in part, player attitude) were just not up to the task when someone was trying to find them. You'd see pages upon pages upon pages of similar content against the same enemies that were all rated 5* (people were incentivized to create new ones even if there was a farm focusing on a certain enemy type because there was some kickback to the author of the arc if people played/rated them), but finding any but the most aggressively-marketed story arcs (this meant using every bit of your online / forum presence to advertise it, or even holding contests / paying people to play the arcs) was a lost cause because even a five-star story arc was completely buried under hundreds of pages of farm arcs.
In Soviet Russia, Farm Nerfs You
The predictable 'farming nerf' tug-of-war only made things worse. An enterprising group of farmers discovered a particular enemy / map combination that allowed for extremely high return when playing a certain character build (e.g. fire farms where 100% of the incoming damage was fire, and a character with the fire aura defense powerset could destroy any number of them easily without being in danger). In concept, these farms weren't different than ones already existing in the world (the Television mission with eleventy billion enemies, for example), but were much more efficient. In response, the devs selectively nerfed features that actually ended up hurting legitimate authors. Just as an example, the Hydra enemies were farmed so heavily that the devs decided to halve their xp. I was using Hydra enemies (along with some other enemy types) in one of my story arcs since they actually fit the story -- now everyone playing my arc was only getting half the rewards, yay! This was also touched on in that GDC talk -- fixing an exploit can have a negative effect on legitimate players and authors. And that's bad.
We're All In This Together
And here's the thing -- the farmers are not the enemy. Heck, I'll even say there's no such thing as a "pure farmer", there are people who may decide that they want to play something quick and easy to wind down, or be at least moderately entertained while grinding currency_du_jour_01. Keep in mind that there'll ALWAYS be people who try to maximize rewards while minimizing risk, and I'm 99.99% sure they'll be better at this than the devs. If there's a particular map/enemy/whatever combination, they WILL find it. Trying to fight this is completely pointless, and will only alienate players. Similarly, attempting to ostracize farmers on forums and such will only result in grudges and instigate sniping / griefing wars between various "factions" of the community (which is a bit silly in my eye to begin with -- we're all players!).
In the end, it comes down to playstyle. There are people who want Foundry content so they can beat stuff up and get phat loot (be honest: a LOT more people than those interested in playing story content made by amateurs, regardless of what the vocal minority of the official forums is saying). There are going to be a lot of people who want to get lots of achievements with the Foundry. There are people who want Foundry content for RP and guild events. There are people who want Foundry content to challenge themselves and see how strong their character is. There are people making silly content, serious content, "how to use the Foundry" content, and probably a lot more types of content catering to playstyles not listed here. As story creators, we should be making content alongside them, not in competition of them.
Getting good content to players in the Foundry - challenges and solutions
Handle: @zaphtastic
TLDR: Player reviews (on forums, third-party sites, podcasts, etc) are absolutely great. So is focused dev attention on UGC / official contests that end up with some lucky quests getting their name engraved in gold somewhere and everyone playing them for a week. However, these solutions don't scale well at all. If a community highlights 200 (generous number) campaigns (I use this number because the UGC talk mentioned that there were 200 "good" missions out of the entire pool of STO UGC), what happens to the other 599800? Sure, most of them are probably not very good... but after you played through the 200 'said to be really good by someone who may have different tastes from me' missions everyone else did, how can you tell which ones ARE?
Reviews: Drop in the ocean
Since the City of Heroes forums are no more, I unfortunately can't check how many reviews there were in the end. I do have some earlier data from the Rotten CoHmatoes project -- which was a massive effort to unify ALL story arc reviews in a simple searchable database. It was near the end of the golden era of the COH MA -- the system was still fresh (4 months old + beta), with a lot of active reviewers and a lot of activity in the forums, though it was starting to get a lot less lively than it was at launch. Most of these reviewers became inactive during 2009, and in 2012 there were just a tiny handful still reviewing arcs (admittedly the pool of story arcs didn't grow that fast either after the initial rush -- there were ~600000 in the end). So this should show the best possible picture, right? Well, let's take a look at the numbers.
Amusingly, the 4-hour Foundry cap within STO (which will be there in NW too, I take it) that was mentioned in the GDC talk is going to limit the number of reviews and reviewers; why spend your 4 hours playing potentially <font color="orange">HAMSTER</font> content when you could be playing the "tried-and-true good stuff" instead, unless you're REALLY dedicated to reviews?
So, the entire review community across multiple sites and forums (possibly a podcast too, I'm not sure if AEntertainment Tonight was a thing back then) right at the tail end of MA's Golden Age managed to cover... 0.17% of all story arcs in the game at the time. Again, this isn't a slight against the reviewer community at all -- it was an awesome initiative that was successful at what it did. But for better or worse, it was still scratching the surface, looking at story arcs made by the tiny minority of forum-dwellers. Which brings us to...
Communities: I am the 1%!
I forgot which MMO developer quipped about this, but it's obvious -- unless you have a tiny indie game, only a really, really, really small minority of players will even bother to read the forums. 1% is probably not the right number, but it's about the right proportion. If you limit your search to active forum posters, this number drops even further to a really insignificant value. What this means in practice is that forums and fansites cannot fix the "problem" of finding content -- because most of the playerbase will probably not even be aware of their existence! And if you're looking at who is posting in or even reading the Foundry forum, you'll probably find a high population of authors and not many players at all. If there is a solution, it has to be within the client.
Note, though, that this doesn't mean forums and community sites are useless. On the contrary: they're great for exchanging information about the Foundry's workings, getting reviews and feedback from fellow authors, and, well, building a good community. Ultimately though, all these small communities will just explore a tiny slice of the Foundry and their influence on the general public will be quite limited.
Dev choice: Even smaller drop in the ocean
COH had hand-picked "Dev's Choice" story arcs that dropped normal loot instead of tickets. An additional advantage of having your arc put in this category (along with Hall of Fame, see previous post) was that they were in the first 2 pages of the "browse" interface, so they got FAR more exposure than anything else. Also, if an author got one of their arcs honored this way, it no longer counted against their story arc limit (which was up to 8 arcs total in 2012 if you bought the extra slots).
There were (sort of) yearly official Mission Architect competitions in COH where the main MA developer decided on a theme and some other restrictions (such as "a level 1-10 mission about someone losing their superpower"), authors submitted their arcs, the dev(s?) played through the arcs, and finally came up with the winners. Now, this was enough to highlight maybe 5 arcs per year (along with some minor attention on the others due to forum activity) out of a pool of 600000. Plus, even though the MA developer had the best of intentions, it was still only one person (or maybe a few colleagues) that may not necessarily find the same sort of content good as the majority of the playerbase. Indeed, in later contests there was some bitterness between MA authors about some authors "pandering to the judges"... I'm sure you can see where I'm going with this.
In NW, featured / spotlighted quests are going to be the equivalent of this (see the 'Feature the Good Content' part of the GDC talk). Obviously having a 'dev choice' every week instead of every year is better, but it's still a really tiny part of the Foundry based on the live team's subjective take (which will likely be influenced by the community, denying people who e.g. don't post on the forums of ever being noticed like this). It'll also drive people to play the spotlighted content and be leery of going beyond that, because if people are doing the job of filtering high-quality quests for you, why bother looking beyond that?
Manual QC: Only Human
Vetting processes with a small number of "quality control" players work when the pool of content creators is static and controlled. Right now there's a really small number of missions in the beta, so at first glance it looks like the review queue (48:00-ish in the GDC talk) would solve a lot of problems. However, if NW is popular at all (and I think we all want it to be), there are going to be over 100000 quests made in the first few weeks with the pace keeping up for several months at least -- and that's a very conservative estimate. You can't hire enough people (or have enough reliable community volunteers) to deal with that, which means 'vetting' players will fall back to just looking at the star rating, which inevitably leads to people playing the already-popular missions in the first 2-3 pages and ignoring everything else. Also, the perils of having a small/voluntary community basically have full power over deciding whether a quest will sink or swim sounds brutal to a new author.
Getting good content to players in the Foundry - challenges and solutions
Handle: @zaphtastic
TLDR: Tagging missions with playstyles (such as 'challenge', 'farm' or 'story') to narrow down searches to the type of content the player wants. Adding personalized (and sharable) favorites lists of missions, and allowing players to browse the Foundry based on their friends'/guildmates' preferences. Implementing a (highly customizable) suggestion system that suggests little-played but high-quality missions to players that are similar to content they have played and enjoyed in the past. Improving the rating system with checks and balances (based on proven real-life solutions such as Amazon) to make it more fair.
This is probably the single most important post in this thread. I posted this earlier in a 'things you'd like to see in the Foundry' thread, but well... it went the way General Discussion threads usually go. Anyway, this list is probably very incomplete -- and I'm sure others can come up with solutions, too! I really tried to avoid unrealistic solutions here; saying something like "oh, just do what Google does and have some smart personalized pagerank system used for sorting quests" is probably way out there, as is "implement an expert system that can unerringly advise each player about which quest to play next". Similarly, I only collected solutions that assume that the design of the search/browse/rating systems as they are is pretty much set in stone (e.g. can't change to a like-based system from the current star rating system).
Playstyle tags
Suggestion: Have fixed tags for playstyle that people can apply to quests (or search quests through them), and have them be a quick one-click filter in the quest browser. Some example playstyles are 'farm', 'story', 'friends', 'event', 'challenge'. These would be in addition to the tags already supported by the Foundry (as well as custom tags). See this thread by optimates for some more suggestions related this, such as multi-level tags.
Example: Alice likes to test her characters' power by playing challenge-oriented quests: ones that have lots of tough monsters and traps, or are otherwise challenging to defeat. Bob likes playing story-oriented quests: ones with lots of text, lore, details, and probably not optimized for loot/xp. Chris likes playing farm-oriented quests: ones focused on getting as much loot/xp as possible within the limits of the Foundry (without exploiting). They all want to find quests they want, but they end up having to wade through hundreds of other quests in the list that don't apply to their playstyle -- they tend to rate these quests poorly and then move on.
Enter the tag system. Just by selecting the tag, Alice will only see challenge-focused quests, Bob will only see story quests, Chris will only see farm quests, and they'll all be happy. Similarly, quest makers will make sure to select the appropriate tag for their quest so that they can reach the appropriate audience. The same thing applies to solo-friendly quests, outdoors quests, quests where you have to fight drow, and any Boolean combination of these factors (story AND NOT combat-heavy AND drow).
Potential exploits: None. You could tag your quest with all possible playstyles, but chances are, it won't be good at all things for all people, and thus your rating will suffer for it.
Favorites lists
Suggestion: Implement a personalized "favorites list" for each player (no limit to the number of favorites). If a player really likes a particular quest, they can mark it as a favorite; there'd be an interface element to remove quests from the favorites list as well. If someone looks for quests in the Foundry, they get a parallel window next to the "usual" one that shows the quests their friends and guildmates like the most (in order of "# of favorites", and a mouseover shows which friend(s) like a particular quest), with the option to remove friends/guildies from this filter. Maybe make it possible to send favorites lists to people directly (as a mail attachment) with the standard anti-spam protection measures in place. Undecided whether it would be a good idea to allow people to favorite their own creations (probably not a good idea).
Example: Jane wants to try out a good quest from the Foundry, but she's already played the top 10 ones (which are the only ones everyone on the forum talks about), and there are hundreds of thousands of barely-played quests in the Foundry. Luckily, her friends have a lot of quests they like -- both popular and obscure ones. Jane looks at her friends' favorite lists and plays the quests from them that she hasn't tried yet. While doing so, her "recommendation list" (see next suggestion) gets populated as well.
Potential exploit: Authors making alt accounts and mass-favoriting their own stories. However, these alt accounts would not get on friends lists, so the effect would be really limited. Spam advertising stories through favorites lists is possible, but would probably backfire... still, it's probably a good idea to keep the scope of this to friends and guildmates.
Auto-generated quest recommendations
Suggestion: Implement a "You might like..." / "You might want to try..." function in the Foundry quest browser. This function would contain a separate list of (10/25/50) quests the player has NOT yet played that fulfill either of the following requisites:
- Highly-rated quest that was liked by players who liked the same quests the player has previously played and liked
- Unknown quest that shares many keywords and/or tags with quest(s) that were highly rated by the player before, and has at least a 3.0 rating average (if it is rated at all)
This list would be recalculated every time the player finished a quest for the first time and 'liked' it.
Exact details (how many people is 'many', whether 'liked' means '4-5 stars', '5 stars', 'favorite', 'donated to author', whether 'unknown' means '0 plays', '0-5 plays' or '0-25 plays') could be customizable per-player. I'm sure others can come up with improvements to the algorithm.
Example: Bob completes "The Endless Hunger", a story-heavy quest involving vampires. He really likes it and gives it 5 stars. His "recommendation" list changes to include several story-heavy quests involving vampires that don't have many plays, but have a generally good rating, as well as several highly-rated quests that were played and enjoyed by people who also played and enjoyed "The Endless Hunger". In the end, authors whose stories aren't in the limelight get (potential) recognition, and Bob can find quests that he is more likely to enjoy. Win-win.
Potential exploit: Not really. Maybe someone could go to a lot of trouble to get their quest to show up in the "related" field for popular quests? It'd probably be downvoted in short order and disappear from the lists.
Improved rating system
Suggestion: Add functions to the Foundry that protect against people gaming the rating system, or help people find ratings that are more relevant for them. Keep in mind that griefers and rating abusers will be there, and the focus should be on either minimizing their impact, or making it easier to notice (and ignore) them. Some possibilities based on real-life systems like Amazon:
- friendly rating: Give players an option to view ratings by their friends, guildies, and possibly 'people who match their taste' (this is probably not really feasible to implement) and sort Foundry content based on that.
- rating weighting: All players have a (hidden?) 'reliability' stat that weights all the ratings they make. This stat starts out at 100% and increases or decreases based on how many people agree with the player's rating -- this "meta-rating" would also be weighted, of course.
- helpfulness: All comments have a "was this helpful? yes/no" button next to them. If a comment is deemed helpful, it increases the rating weight of the comment and the commenter; if it was deemed unhelpful, it decreases it. Apply diminishing returns as needed.
- mandatory comment: Ratings below 4 are not accepted unless the player leaves a comment (even just a single character). This would show the player's @name to the story author, and possibly to everyone else(depending on whether it's possible to set comments as private), which gives a degree of accountability, and makes it really easy to locate griefer accounts for reporting. Of course this requires the implementation of a "profile" page for each player, showing their past votes / ratings. If rating adjustments are used, this would expose the voter to upvotes/downvotes as well.
Example: This'd be way too long to describe here... and not very exciting either.
Potential exploits: It's supposed to protect against them in the first place. Obviously there are some things like an author making hundreds of alt accounts to upvote 5stars and downvote 1stars on his quest; these'd need some system-level protection / anomaly detection techniques to deal with. The actual techniques would need a bit more work to be functional in the Foundry context, too.
And here's the most important part that I'm posting this for... time to bust out the size 5 font:
If you have any ideas about how to improve the rating, search, or browse systems, speak up!
Even though this isn't something related to "basic" Foundry functionality, the rating and browse system is something that absolutely has to be improved before the game launches... and the more suggestions we have, the better.
Getting good content to players in the Foundry - challenges and solutions
Handle: @zaphtastic
I noticed during the second beta weekend, that they had a list of featured foundry modules. I think they should rotate these featured foundry content on a weekly basis such that anything within the last week or two will not be featured again unless if there is nothing new/ or newly popular hitting the live servers with/without the dev's seal of awesomesauce.
I also feel it will be important to make content available and to avoid the issues a system that strictly rewards ratings and early authors. Nothing will kill The Foundry faster than authors not wanting to make content because only a small percentage of authors actually get their content played.
Anyway, I really hope Cryptic has already considered the issues and, if not, considers this thread.
I plan on throwing myself into the fountain of Foundry authors as soon as I get the opportunity and I strongly support the argument zaphtastic made. I had the mindset that the early authors were going to get over-exposed and cut me out of any real player interaction. So I was going to make missions specifically for the enjoyment of my friends and guildies, and see how much I can exploit the mechanics for something different than a linear dungeon crawl. If the recommendation system that zaphtastic described were implemented I have no doubt that anything I make would have a far greater chance of getting played.
Whilst the Foundry may never become a true replacement to the NW toolsets, it may become a home for lost module builders who sustained the NW community for many years and have since wandered, waiting for a successor.
If a subtler system of voting provides more oxygen to quality work suffocating beneath incomplete, thoughtless, rushed or simply ordinary content, sign me up. It may be I personally produce no quality work, but would surely appreciate an audience on the off chance that I do, especially if it arrives late to the party.
As many others, I've watched the UGC video a number of times and have tried to understand in my head how things will work. Unfortunately for me, I've never used the NWN tool, COH or STO Foundry so have been unable to base my thoughts on anything substantial.
What your post has done is given me more food for thought in relation to those areas that I had already flagged from the video;
* The voting system
* Dev's choice highlighting 'favourite' content
* How new authors break into an established list
I only hope that Cyptic are able to learn from their experiences on other games and address many of those points in NW. I understand there are quite a few people who could be turned away from this game completely if the Foundry doesn't come up to expectations.
I wonder what the solution will be?
From the Foundry Videos, there seems to be author pages, whereas you can link all your content and make blog entries.
If this works out well, people can subscribe to your page. Maybe this is the work around for the search engine shortcomings and inflated star ratings?
Open Beta is launch. There are no more character wipes.
Here is list of questions/answers called Frequently Asked Questions (FAQ).
http://nw.perfectworld.com/about/faq
http://nw-forum.perfectworld.com/showthread.php?18401-Neverwinter-Extended-FAQ
If not sure if a thread is posted, try the SEARCH function below:
http://nw-forum.perfectworld.com/search.php
I know it's just an implied and obvious truth of the way the system works as outlined in your post, but I think it bears repeating that the fact that first-past-the-gate adventures are most likely to end up rated high early and end up pinned to the top forever not only disincentivizes new content creators after the fact, but encourages those from day 1 to rush the creation of their adventures.
Polish rooms with lots of detail to make the world look lived in? Edit dialogue to a professional standard? Nah, we've got to get this **** out there IMMEDIATELY so it can actually be played by someone ever, before it sinks to the bottom. That's really what I'm more concerned about--this system discourages quality on a basic level.
Someone (and realistically, LOTS of someones) still has to find you and be convinced of your quality work in the first place, which is the crux of this thread, I think.
The last day of open beta #2 I spent about 4 hours just going through some of the lower non-rated quests in the foundry. Way down at the bottom somewhere was a wicked quest-line that I really enjoyed, and promoted as such. At the same time there are a lot of duds on there as well. Some broken, yes I am aware of some of the reasons for this. Broken I am speaking of Player being spawned half way in the floor, Traps that have been set to far away from player access to disarm it, the list could go on... Well I could go on and on...
Anyways, I digress. My point is, players will have to dig to find the quests in the foundry that will suit them. Some of the provided recommendations by the OP may serve to help improve that.
Website Dedicated to Foundry Tutorials: www.NWUGC.com Twitter: @NWUGC
Gold or Bones!!
I agree. Getting them to try your quest from the 1000s on the list is the problem.
I think if they do one of your quests, and really like it, they can look you up and subscribe. This will lead to people having lists of 'favorite' authors.
I also liked the idea of classifying what kind of quest it is, challenge, farm, RP etc. This would allow you to sort the five stars to your taste.
Open Beta is launch. There are no more character wipes.
Here is list of questions/answers called Frequently Asked Questions (FAQ).
http://nw.perfectworld.com/about/faq
http://nw-forum.perfectworld.com/showthread.php?18401-Neverwinter-Extended-FAQ
If not sure if a thread is posted, try the SEARCH function below:
http://nw-forum.perfectworld.com/search.php
I hope this gets in front of the eyes of the developers!
I agree with everything said, it is a point of consideration because user generated content will be a part of leveling content and endgame content. The reason the foundry is in this is not just for the creative use of the players, its because it helps keep players playing the game yes? It makes the game better?
Well then why not make this as accessible as possible? Have Authors be required to "Tag" their content and use those tags in a search engine. I would even go so far as suggesting an auction house-like search for the foundry. Where you can search for missions with say Drow; or with Goblin story arc. I would also ask that there be a second difficulty applicable for endgame content. A "heroic" difficulty that when checked by an author means that the mission is invariably harder than the normal scaling. Also something else that could be searched for.
Please consider the OP's Long post, I would really appreciate it if some of his Ideas where incorporated into Neverwinter.
Not that this solves the problems you talk about. It just lessens them.
1. An automatic "new" tag for new foundry content, based on creation date and number of times played. This could get newer content viewed.
2. Ratings by category. These could include such things as farm, story, challenge. But also polish, complexity, etc.
We feel that this way we can better review the content ourselves with very well defined criteria for the review, which will be publicly posted on the site, as well as our own thoughts and reasoning for why we rated the missions as we rated them. We also like to think we nicely balance one another out. My wife is very big on just pure enjoy-ability/fun factor of the game, along with lore qualities (she absolutely loved Dweomer Keep and the Crypt of Carnage giving them both 5 stars), while I tend to be harder on things from an originality/story/aesthetics angle (likewise I rated both high for their aesthetics and creativity in layout, but dinged them on story).
But anyways that's the long and short of what we hope to do to help others out. I really wish Cryptic had done some sort of linking between the Foundry and their website so that people could go out and give longer and more detailed reviews for others to read instead of the limited space/characters in the review box. And I know not everyone will necessarily agree or appreciate our reviews, but it is just something we want to do.
Yes Charlie, keep clicking, yup, keep clicking, see how high score rises?
I am stunned people do this. It's not legitimate and I don't feel the least bit elitist in saying so. It's just XP without any skill, effort or risk. These same people will then whine there's "nothing to do" when do his max.
I am glad Cryptic is serious about nerfing exploits, including obvious farming. This is correct and it has led to better success than UCG that allowed it without any control or recourse.
Putting that aside, I do think tagging modules is key. And I'm confident the current search for the Foundry will have to be improved. It's not there yet. It is fine for initial release, but it will need to be augmented.
I'd like the ability to "subscribe" to authors (like clicking "like" on FB or something), so I can see when they have updates. A column of the top 5 from my favorite authors would be ideal as well, including the intelligence that I've completed some of them, sorting newer/unplayed modules to the top.
It will get there, it has a ways to go yet, but I am confident it will get there.
Check out SHADOW on YouTube!
Quite a read indeed! Some good points raised and discussed, but I would just like to throw another idea on the table... why not make content that YOU and YOUR FRIENDS would like to play and let the masses hustle in the pit of madness?
Not everyone wants to be a high profile module superstar (admittedly many do.) I think the motivations of the many should not overshadow the niche designers and developers who just want to make really meaningful and fun content. If farming content is fun for you, then make some, play it and enjoy. If you want in-depth lore based puzzles with obscure references in the Salvatore books, then craft the experience to your liking and pit players against your diabolical (HardCoreRuleset-styled) mind-benders.
Let us remember why we are playing the game, whether it is for fun, for immersion, for lulz or for trollz - play it the way you want to.
Don't be put off by the naysayers and if you want some help, reach out to the community. There is already a useful foundry guide for building stories using freely available online tools. If you already have l33t skillz from the Aurora Toolset days, then dust off your narratives, throw it through the new foundry when you get a chance and enjoy.
Building things in teams can add a whole new perspective and keep the dream alive too, so if you hit a stumbling block, ask a friend or post in the forums and rally people to your banner. As long as you can verbalise or describe what you are aiming for, I am almost certain that help will arrive in some form.
Don't fear the 4-star mosh-pit, if you make something awesome then I am sure it will be recognised and given due merit by your peers and the community.
- Jasc
Some comments of my own: Well, I assume they're going to rotate the featured list. The problem with the list is twofold: first, it only highlights content that matches the live team's tastes; second, the live team has to select featured items somehow... and I'm fairly sure they'll either pick from already-popular stories, or stories that have a lot of buzz around them in the forums. I've also read about a "submit your story to the devs to be considered for highlighting!" queue, but that idea is DOA, sorry. It'll be swamped with thousands of modules in the first week, and at that point there's no way a small team of reviewers can hope to keep up with it.
---- Yep, this is another ugly side of the entire thing. The other is, of course, that a new author (even if s/he is greatly talented and has a lot of pen-and-paper DM experience) is likely to be inexperienced with how the Foundry works and will probably create a quest that would only rate as a 4-star (or even a 3-star). This'll lead to the content being lost forever even faster, and since nobody will be playing it, nobody will be giving the author any feedback! This can be SOMEWHAT mitigated by reviewers offering their services on the forums, but the time investment problem comes up again (plus authors that don't read/participate in the forums, though I admit the author community is more likely to do so).
This is not a bad idea... basically it'd be a tag system you could rate. So you'd select the "story" tag in the browser and get a sorted list of quests with the "story" tag depending on the rating of the tag itself. Obviously the UI would need to accomodate people who only vote on one particular aspect of a quest or even just the main rating (such as greying out all tags the user hasn't explicitly selected a star rating for). So it'd be "3 challenge, 5 story" instead of "3 challenge, 5 story, 1 farm, 1 event, 1 soloable". Further, tags would probably morph into various rating aspects (e.g. "combat-heavy" would be "level of combat", "solo-friendly" would be "soloability"), but I think that is fine.
---- Difficulty should definitely be shown on the UI... and I think this is one of those things that's best when generated by the Foundry automatically. For example, see how COH handled it:
In the image above, everything except for "Keywords", "Morality", "Description", "Story Status" has been automatically generated. An overview like this allows everyone to see what to expect -- the story arc above is probably a 45-60 minute affair (4 missions, mostly small maps with no kill-all-enemies objectives), can be challenging (it has elite bosses), and there are custom enemies (enemies with custom power selections, and 'civilians'). The enemy groups are shown too, so players who really don't like fighting Crey (note the comment in the description!) can avoid playing it.
Now take all this and put it in the searchable UI, and we can have the beginning of a beautiful friendship-- err, I mean, something good!
---- THIS is why I still have hope. Unlike COH and STO, the Foundry is a centerpiece here. This helps in mitigating the loss of the "shiny status" (as it was in COH after a few months), but honestly, I think these problems can cause the loss of "shiny status" by themselves. Only, in Neverwinter's case they can result in the disillusioned players stopping playing altogether instead of just ignoring the Foundry!
---- [1] is probably a given. We have something like this in the 'review queue' (where new content is placed), and it is necessary... otoh, by itself it won't do much. Depending on where you set the threshold, either the majority of the Foundry's content will be tagged "new" forever, or the "new" tag will go away during the first few plays by the author's friends/guildies, and then we're back where we started. Still, having a new tag can't hurt -- and its usefulness can be enhanced by other features.
[2] is a great idea -- see selenti's comment above. I think adding a secondary rating system like this could work, but I'm a bit skeptical about getting the average player to vote on them.
---- Of course, review sites like this are crucial too for authors to improve their work (see the first response to selenti above)... but unfortunately they don't really help with the mass appeal thing, especially considering how much effort it takes to review Foundry content and just how much of it is out there. Realistically these sites are visited by a tiny minority of players; they may be more frequented by authors, but most of the playerbase doesn't even touch the official forums, never mind a third-party site.
Again, don't feel disheartened -- initiatives like this are great! They just cannot fix the biggest underlying problems of the system.
---- I don't disagree about farming being boring and mind-numbing. I hate doing farming myself, but here are some counterpoints about why this isn't a problem (but it DOES become a problem if vilified).
- First things first: exploits are NOT farming. Exploits are exploits, and should be fixed. If someone can get 500 gold of drops in an hour of Foundry missions or rocket from level 1 to level 60 in 2 hours, that's an exploit. If someone can get the same drops and xp (maybe even 50% more xp) that they'd get by running an official dungeon over and over, that's just an optimized farming mission. There was farming before the MA, and the payout of those (dev-created!) missions wasn't that much worse than an average farm's.
- Farmers aren't hurting anyone (except for the AH, maybe, but they'd be doing that even without farming Foundry missions). Yes, they may be 'playing the game wrong' in your opinion. In their opinion, WE are 'playing the game wrong' by being enthusiastic about content done in a restrictive toolset by amateurs! If someone levels to 60 in 3 days (instead of 4 days if you do the regular questing routine), so what? It's their playtime. They probably didn't do it just to have the character stand around and look pretty -- maybe they dislike the levelling process and want to get into the endgame, pvp, what have you.
- There are a lot of people who 'farm' in games, and I'm not talking about RMT'ers. Some may do it as a regular activity, but most probably do it as a way to wind down or make grinding a bit more palatable (and we can admit that f2p games are heavy into the grind, even if it's disguised behind things like pvp or daily quests or... Foundry content). In fact, I'd say they make up a very sizable part of any game. They just aren't represented on the forums.
- Trying to wage a holy war against farmers always backfires. It backfires very badly. There are a lot of 'farmers' out there, and you don't want them to flood the Foundry with eleventy billion farming quests that look like normal story quests on the surface (because you keep reporting farms), downvoting story quests because they're "in the way of farming", and forum sniping / community splintering because they were ostracized by the (comparatively tiny) author community.
Yes, the subscription functionality is already there, I think. Of course this only reinforces known authors (and/or friends), but at least it's something. And yeah, the top 5 / favorites list could be extended to authors as well -- it's not a bad idea. I also agree that played/unplayed content should be displayed appropriately, and it should be possible to tick a box to only display unplayed content. (in the COH MA screenshot above, the checkmark next to the name of the arc indicates that it has been played)---- Of course, that's a completely valid stance to take. It's just that in my experience most authors want their content to be played. I personally didn't want to become a superstar (even though I did when I won the official contest with one of my story arcs and got it the coveted 'Dev's Choice' status) as much as wanted as many people to play and enjoy the content as possible... while providing feedback. I LOVE feedback - especially from random people playing my arcs -, and it saddens me that I got so little of it outside the one Dev Choice arc that I had.
Ultimately, a 'latecomer' or less-known author can have niche appeal. I know a few in the COH MA who got reliably high ratings from the strictest reviewers, were well-liked within the author community... but still struggled to get more than a single-digit number of plays on their stories. This may be enough of a motivator for some people -- but I think most want, if not recognition, at least the knowledge that there are players out there playing and enjoying their creations.
Getting good content to players in the Foundry - challenges and solutions
Handle: @zaphtastic
And speaking for the masses, if better missions are coming out, but there is no way to find them (and no, I'm not going to swim through thousands, and there WILL be thousands, of missions looking for the diamond in the ore), then I and other players are missing out on something of real value that a lot of people will have put hard work into. I WILL go beyond the recommended list, but I'm just one set of eyes in what will be a big world.
"There are two things that drive technology forward in a huge leap in this world, one is war and one is business..."-- Timothy Wade 2012
"If you wish to join a guild, that will be running a guild foundry RP campaign then come take a look at [BLOOD]"
We already have a swag of suggestions that will promote the good content. The 4-star pit of doom is not the be-all-and-end-all of finding good content and the community has already identified that fansite reviews are going to be forthcoming. The community will find ways to promote the best content which may or may not influence PWE and Cryptic to change their model. The discussions here will already be providing Cryptic with ideas to better utilise their system and the quality of the feedback here could be paralleled by the OPs original intent - how will the useful and meaningful posts in the forums make their way to the "top" so that they can be considered for improving the overall systems in Neverwinter?
A lot of people have invested a lot of time (the developers, publishers and the community) to creating a successful game and user-generated content is an integral part of that success. If the 4-5 star system was the only way of finding good content then we would have a problem, but with all the ideas that have been offered here, I have no fear whatsoever that the good stuff will be lost because it is our (the community) passion that will provide the sanity check in ensuring everyone else knows what the best stuff is, through tags, fansite reviews, forums posts, blogs, fanzines, email and all the other ideas!
Keep the ideas coming though; be they convergent, divergent or disruptive.
Final word - Community!
- Jasc
"New" content is a good idea.
"Promising" content filter sounds good.
Subscribing is good as well, I don't mind the rating system, but the star system frankly stinks, because players are low-rating good modules because of bugs with the engine.
This system does need "tags". If they allow "farming" quests, so be it. Just make a farm tag.