test content
What is the Arc Client?
Install Arc

I can filter Spam better than you, so let me do it instead, please

2»

Comments

  • micky1p00micky1p00 Member, NW M9 Playtest Posts: 3,594 Arc User
    edited February 2016


    @micky1p00 I am not sure if this would work, but would hiring a group to do something similar to the bot group, only to have their list of messages all fed into a spam filter and then used for filtering chat solve the majority of the problem? As you pointed out, the moment you create the need for them to rapidly change the message, they will, but is it possible to create a filter that rather then being reactive, is preventative and blocks messages before they create them?

    It's never 100%, but you don't actually need to hire anyone, you have a very large group already, and it's the player base.
    Each on it's own we have a problem against an automated system, but as a whole we have an advantage, the trick is to utilize it, as I wrote in that big post, simple idea is to add the message to the black list once it's reported as spam, precautions should be made for time windows, thresholds and white lists to not block trades or lfg. And at the end you want only one person going through the new blacklisted items (daily for example) and verifying the system.

    This is a reactive system, but it combines much more 'human power'. Now for example to make it more preventative we can check string distance, a simplified example will be to 'spell check', we check each message vs our dictionary (the already blacklisted messages) if for example we found a variation of less than 5 chars, we filter it out (in practice it's more complex to account for place changes - but at the end we are looking for some sort of similarity rating). We can also in this case (with restrictions) add this new variation to a suspected list and add it to the blacklist with lower thresholds.

    Like everything else it's not perfect and has downsides, but I think it's a better start..

    You can also make it interesting, give 100 AD for each unique message the reported, for each account that has a char over 60. Penalize false reports. And you just hired your anti-spam army.
    Post edited by micky1p00 on
  • l0th4ri0l0th4ri0 Member Posts: 589 Arc User
    Guys... the spammers are not going to rapidly change messages, even after they implement the clientside filter. Lazy copypasting is all they can manage. These are not disciplined professional criminals like Ocean's 11, these are guys who couldn't make a living any other way but to follow orders from some vermin who scams credit cards in Southeast Asia.
  • thefabricantthefabricant Member, NW M9 Playtest Posts: 5,248 Arc User
    l0th4ri0 said:

    Guys... the spammers are not going to rapidly change messages, even after they implement the clientside filter. Lazy copypasting is all they can manage. These are not disciplined professional criminals like Ocean's 11, these are guys who couldn't make a living any other way but to follow orders from some vermin who scams credit cards in Southeast Asia.

    You are assuming lazy copy pasting is all they can manage, unless you can prove your opinion is a fact, its only an opinion. TBH, I think a client side chat filter would work, but for a different reason. I think they wouldn't bother to combat such a change simply because anyone filtering them out, isn't going to buy from them anyhow, simply put, its not like they losing revenue. The people who are buying from them, will also not be filtering them out so such a change wouldn't impact their revenue in any way.

    With that being said, Janne knows more about this then I do and I am simply using logic and common sense rather then actual background knowledge of the field, so I would properly trust Janne's opinion over mine.
  • edited February 2016
    This content has been removed.
  • mrgiggles651mrgiggles651 Member Posts: 790 Arc User

    They use every means possible to prevent being shut down.

    Obvious implication is obvious, implication.
    I wasted five million AD promoting the Foundry.
  • micky1p00micky1p00 Member, NW M9 Playtest Posts: 3,594 Arc User
    edited February 2016


    TBH, I think a client side chat filter would work, but for a different reason. I think they wouldn't bother to combat such a change simply because anyone filtering them out, isn't going to buy from them anyhow, simply put, its not like they losing revenue. The people who are buying from them, will also not be filtering them out so such a change wouldn't impact their revenue in any way.

    This is actually an interesting point, I jumped into the technical side, and didn't think of this.
    l0th4ri0 said:

    Guys... the spammers are not going to rapidly change messages, even after they implement the clientside filter. Lazy copypasting is all they can manage. These are not disciplined professional criminals like Ocean's 11, these are guys who couldn't make a living any other way but to follow orders from some vermin who scams credit cards in Southeast Asia.

    Perhaps.
    Maybe they copy paste and probably are, the spammers can be fully automated for all we care.

    And yet those that made the bots, those bots that can attack, run, interact, and so on.. Admit that it's a very sophisticated piece of software.
    I'm sure those people are more than capable of making a basic script that will change some letters, and give a list of modified messages.
    We need only one 'person' that is capable, the minions will get their "message to post now" and will keep copy pasting it, they don't need to know anything, just copy from one window, to another.


    Profit is generated in 2 ways: You need something to sell, and you need people willing to buy it. Simply because there will always be people to buy their goods as no publisher can afford to drop their cash shop low enough to keep up with bot prices especially with a currency converter in place, the only way to hit them is by removing their supply, which is RP.
    Once you make anything worth selling in the open game maps and lairs BtA, they have nowhere to go. You can't script dungeons like LostMeal or EtoS because the bosses react to the way they are attacked and there's no way to detect red areas fast enough to script it or they'd farm heralds like we do. They are stuck on the easy stuff: skill nodes and mob drops. Quick rinse and repeats. And once they lose the ability to script, they lose their advantage and with that, their profits.

    I tried SkyForge, and there is no trade option there. At all. So nothing to bot, nothing to buy or sell. So I thought "Wow they just solved the whole issue, shame there is no player economy but well, definitely no botting"
    To my total surprise after some time the chat there wen full spam, they offered to grind for you, to level for you and etc..
    Can't get rid of them, it's an arms race, only minimize.


    Anyway, looks like our problem is solved, at least for now.
  • edited February 2016
    This content has been removed.
  • thefabricantthefabricant Member, NW M9 Playtest Posts: 5,248 Arc User
    micky1p00 said:


    TBH, I think a client side chat filter would work, but for a different reason. I think they wouldn't bother to combat such a change simply because anyone filtering them out, isn't going to buy from them anyhow, simply put, its not like they losing revenue. The people who are buying from them, will also not be filtering them out so such a change wouldn't impact their revenue in any way.

    This is actually an interesting point, I jumped into the technical side, and didn't think of this.
    l0th4ri0 said:

    Guys... the spammers are not going to rapidly change messages, even after they implement the clientside filter. Lazy copypasting is all they can manage. These are not disciplined professional criminals like Ocean's 11, these are guys who couldn't make a living any other way but to follow orders from some vermin who scams credit cards in Southeast Asia.

    Perhaps.
    Maybe they copy paste and probably are, the spammers can be fully automated for all we care.

    And yet those that made the bots, those bots that can attack, run, interact, and so on.. Admit that it's a very sophisticated piece of software.
    I'm sure those people are more than capable of making a basic script that will change some letters, and give a list of modified messages.
    We need only one 'person' that is capable, the minions will get their "message to post now" and will keep copy pasting it, they don't need to know anything, copy from one window, to another.


    Profit is generated in 2 ways: You need something to sell, and you need people willing to buy it. Simply because there will always be people to buy their goods as no publisher can afford to drop their cash shop low enough to keep up with bot prices especially with a currency converter in place, the only way to hit them is by removing their supply, which is RP.
    Once you make anything worth selling in the open game maps and lairs BtA, they have nowhere to go. You can't script dungeons like LostMeal or EtoS because the bosses react to the way they are attacked and there's no way to detect red areas fast enough to script it or they'd farm heralds like we do. They are stuck on the easy stuff: skill nodes and mob drops. Quick rinse and repeats. And once they lose the ability to script, they lose their advantage and with that, their profits.

    I tried SkyForge, and there is no trade option there. At all. So nothing to bot, nothing to buy or sell. So I thought "Wow they just solved the whole issue, shame there is no player economy but well, definitely no botting"
    To my total surprise after some time the chat there wen full spam, they offered to grind for you, to level for you and etc..
    Can't get rid of them, it's an arms race, only minimize.
    Yeah Janne, the major difference between a player created chat filter and a dev enforced chat filter is that the one has no link to their income, writing a proper, usable chat filter is time consuming and finnicky, not something people are going to do unless they have absolutely no interest in buying from those sites and only see those sites as an irritation in chat. In contrast, a dev enforced chat filter does directly effect their income, because they do not know if the people in game would have bought their goods or not, quite simply, its blanket filtering these messages for everyone, rather then selectively filtering it out for those that aren't going to buy it anyhow.

    As a side effect of this, I am sure the devs could monitor who runs a chat filter and who does not, then pay special attention to the accounts that do not as they are more likely to buy from 3rd party sites then players who do run a chat filter. It isn't particularly ethical, but it helps punish offenders and reduces the irritation that those of us who are not interested in 3rd party goods have to endure if we do not turn off our lfg/zone/trade.
  • micky1p00micky1p00 Member, NW M9 Playtest Posts: 3,594 Arc User
    edited February 2016


    That will be the last resort yes. And that's something Cryptic can solve easily by tuning EE XP and speed. Let's face it, nobody here needs leveling apart from that little snag. Leveling is actually fun in this game. Perhaps even more so than the end-game. Plus there's a big difference from the buyer's point of view. Handing over and allowing a stranger on your account -which can be logged and dealt with by Cryptic. That's how we ended up banning the buyers in my older games and interest went down rather quickly after that- to bot for you is a LOT more risky than just buying something from the AH. Losing AD.. fine. Paying is relatively safe and chargebacks are possible. But risking your entire account for the sake of some RP or boons is not only stupid, but a big threshold for potential clients.
    Either way, it would be a start and it would stop a lot of bot traffic on the server, which isn't a bad thing either.

    Yeah, I agree, just was surprised that they even offered, just shows how huge of an industry it is.


    Yeah Janne, the major difference between a player created chat filter and a dev enforced chat filter is that the one has no link to their income, writing a proper, usable chat filter is time consuming and finnicky, not something people are going to do unless they have absolutely no interest in buying from those sites and only see those sites as an irritation in chat. In contrast, a dev enforced chat filter does directly effect their income, because they do not know if the people in game would have bought their goods or not, quite simply, its blanket filtering these messages for everyone, rather then selectively filtering it out for those that aren't going to buy it anyhow.

    As a side effect of this, I am sure the devs could monitor who runs a chat filter and who does not, then pay special attention to the accounts that do not as they are more likely to buy from 3rd party sites then players who do run a chat filter. It isn't particularly ethical, but it helps punish offenders and reduces the irritation that those of us who are not interested in 3rd party goods have to endure if we do not turn off our lfg/zone/trade.

    I thought of it when you mentioned it, and I think that this type of client side solution will have very low usage rate.
    From users that technically will don't want to tinker with this, to people (like me) who don't see it at all (the spam).

    I don't think tracking who uses it or not is helpful, more so, at least for me, even if I was buying something from a company, I don't appreciate being spammed by it, happens in RL with many shops and mailing lists. So as an indicator it's not a good one, people who buy can block after visiting and saving (or registering there directly or something) and those that do not want to see it can still not use the filter.

    Additionally there is a chicken and an egg problem here, I don't want to use zone / trade / wtt because it no longer has any interesting chat for me. But it wont have any interesting chat until the bots are not filtered. Client solution only (meaning partial to the player base) will not help here.

    Side note:
    Because of this thread I did found how to log the chat.
  • thefabricantthefabricant Member, NW M9 Playtest Posts: 5,248 Arc User
    Tbh, I think the most effective way to prevent chat spam is to move all RP to epic dungeons that are made difficult, so that it cannot be farmed by bots but can be farmed and sold by players. Some players might complain that this just means that players with good gear and the capability to farm will be unfairly rewarded, but honestly, what it really boils down to is either a few players will get rich, or a bot company will get rich and you just have to pick and choose which one you would rather have. Although, to be honest, this will never happen as a complex and skill requiring system doesn't cater to casual players and this game is pretty much aimed at the lowest common denominator.
  • edited February 2016
    This content has been removed.
  • l0th4ri0l0th4ri0 Member Posts: 589 Arc User
    edited February 2016


    Actually.. no.
    These are highly effective criminal organizations and not some backdoor-amateurs like NeverwinterGod. I've had dealings with them in the past where I was an in-game mod/GM for other games.
    These guys are everywhere, in every MMO where they can make a profit of scripted activities. And they aren't stupid either. They use every means possible to prevent being shut down. Farming accounts go over proxies so that if 1 goes down, the rest stays alive. Spam accounts are not the accounts where the AD is stored.

    You see, that just doesn't strike me as impressive... using proxies and so on is just common sense if you are one of the guys setting up the bot network. They cycle accounts so that if one of the spambots gets squelched they still have the message going out. To put it succinctly, if a bit cheekily: DUH. :smile:

    My whole point that I've been trying to make and some of us (ahem) are resistant to hearing for whatever reason, is that attacking the spambot is pointless (thus the results of every dev attempt to stop the spam = fail ever since they started trying). The bad guys are just gonna make more bots (this game is F2P). But if you attack the message itself, and the botters don't have any feedback about how successful they are, then they won't even know what it is that is getting blocked in the first place, or if any blocking is actually happening. Remember the Fog of War. They're not gonna know anything except that maybe some ppl are blocking their spam advertisements in some way.

    They aren't going to go through extraordinary measures to try to defeat our various unknowable locally-stored unique spam filters. They won't even try. Just trust me: blocking the dollar sign $ may be enough to last for months before you see any more spam.
    Post edited by l0th4ri0 on
  • vantalkvantalk Member Posts: 2 New User
    Just because I like this game I'm going to share some ideas on fixing this. The tech team should be in on this:
    1. Use a flexible language (like Ruby or something better) to strip the chat log for symbols. Use a regex pattern. Also Ruby has a gem called damerau-levenshtein which will be able to tell that "sp@m", "s2am" etc actually mean "spam" etc. If you know a better language use that. This point will help with basic filtering.
    2. Put in a player based reputation system. If more than 10 ppl report a person (in a given timeframe if you like) for spamming for ex, then silence that player on different chat channels (for a period of time). Make a notify to the person that they have been silenced. They should probably have a button that immediately sends the admin a report of the last conversations or the game should send them automatically to be held if needed for inspection. To prevent abuse from other people making fake reports, decrease those peoples reputation if they are found lying (use a similar logging system). Players that have been silenced too many times can be banned.

    That is part of my proposed solution and I do not state that it's the best, I just don't yet know a better one. I'm giving free ideas from my own game so make some use of it if it helps.
  • scathiasscathias Member Posts: 1,174 Arc User
    Vantalk, that second point you suggest is already in the game actually
    Guild - The Imaginary Friends
    We are searching for slave labor, will pay with food from our farm!
  • l0th4ri0l0th4ri0 Member Posts: 589 Arc User
    scathias said:

    Vantalk, that second point you suggest is already in the game actually

    Yeah, they implemented that over a year ago, I think.
  • micky1p00micky1p00 Member, NW M9 Playtest Posts: 3,594 Arc User
    And damerau-levenshtein is exactly a string distance algorithm like I mentioned several times before..
    Actually if it interests someone:
    http://stackoverflow.com/questions/9842188/compare-similarity-algorithms
    http://www.joyofdata.de/blog/comparison-of-string-distance-algorithms/
  • l0th4ri0l0th4ri0 Member Posts: 589 Arc User
    Another thing I'd like to bring up while we're here: Adblockers.

    All the ones I've ever seen run off of a blacklist (cough, cough) controlled by the end user, clientside. I run Firefox and never have ads, for example.

    Sure, if the ad people wanted to, they could go through extraordinary measures to try to get past the adblocker blacklists that are freely posted all over app sites and so on... but they don't. They don't even bother. It's obviously not worth it for them to keep having to fight that battle against the filters who are able to easily defeat them with a few keystrokes.

    So far, most or all the naysayers on this thread have pointed out the supposed cleverness, tenacity, and indefatigability of the spammers, but I am just shy of 3576% sure that they have none of these qualities when faced with a system that actually has a chance to defeat the ability to "just create another account and repeat same activity".

    The devs may not wish to admit it, but the F2P model has hamstrung them in the fight vs. spam. They have given a powerful weapon to the enemy: the ability to respawn endlessly and come back for more. Ban one account, two more come back in its place. There is nothing they can do for us server side against spam. Their fight is with the people who cheat buying AD and the accounts that bot, hack, swamp the AH, or otherwise are shady as hell. They at least have a chance to do something about those. But the spam... it's time to admit defeat and hand me my blacklist filter. They did their best but it's F2P, man.
  • micky1p00micky1p00 Member, NW M9 Playtest Posts: 3,594 Arc User
    edited February 2016
    l0th4ri0 said:

    Another thing I'd like to bring up while we're here: Adblockers.
    All the ones I've ever seen run off of a blacklist (cough, cough) controlled by the end user, clientside. I run Firefox and never have ads, for example.

    Sure, if the ad people wanted to, they could go through extraordinary measures to try to get past the adblocker blacklists that are freely posted all over app sites and so on... but they don't. They don't even bother. It's obviously not worth it for them to keep having to fight that battle against the filters who are able to easily defeat them with a few keystrokes.

    Ad Blockers are totally unrelated, ads are added by the site owner, and are adding revenue to the owner.
    Large sites that live on ad revenue either will block access to certain parts or will ask you politely to remove the ad blocker, they have to balance this very gently because they do want you to keep using the site (They can't just block you entirely) and at the same time they need you to white-list their site.

    Companies invest man hours and money to solve this problem, like: http://marketingland.com/yahoo-acquires-anti-ad-blocker-startup-clarityray-96242

    20 mil dollar investment for company that is struggling is not "They don't even bother".

    The fitting analogy for this case is email spammers. And server side multi account learning / adaptive filters always better than a single account local rules. But hey... if you want to fight.... why not...

    BTW, you looked at PE zone/trade/lfg post last patch ? Very clean
    And it's "USD" already, not $
    Post edited by micky1p00 on
  • vantalkvantalk Member Posts: 2 New User
    > @l0th4ri0 said:
    > Vantalk, that second point you suggest is already in the game actually
    >
    >
    >
    > Yeah, they implemented that over a year ago, I think.
    Cool. I haven't played for that long huh? :)

    > @micky1p00 said:
    > And damerau-levenshtein is exactly a string distance algorithm like I mentioned several times before..

    Nice article. Yep. In this case make sure you don't go overboard on banning ppl for saying "cool" instead of "gold". Can't raise that acceptable distance too high. Also probably make use of a blacklist & whitelist of words.
    But if this is quite a cancer, I would rather charge 1$ for purchasing the game so I can get an authentication of who the person is and if he actually is a real person. You get 1$ in exchange for the trouble each person causes.. or any other authentification method. In return Neverwinter should probably give some stuff to the players and/or maybe remove/add some feature(s) that bug/appease ppl, so it feels more of a tradeoff.
  • l0th4ri0l0th4ri0 Member Posts: 589 Arc User
    edited February 2016
    micky1p00 said:


    And it's "USD" already, not $

    Well those guys have used USD for a long while now, I kinda forgot about that. But everyone else still has the dollar sign. It's not like the spammers read this thread and reacted to us talking about it.

    And about Yahoo trying to circumvent Adblockers: you just proved my point. Look at what a Gigantic Internet Presence has to do to try (TRY) to get around a simple clientside filter. Good luck getting past mine. They are doomed to fail, I promise you. They might as well have built a bonfire with that 20 million and had a weenie roast. :smile:

    PS I checked PE just now. Spam is definitely WAAAY down. I would love to be cautiously optimistic, but the last time I got my hopes up, the spamblocking lasted less than a day. In the meantime I'm gonna enjoy this respite from the ads, and not worry how long before it is spammed again. See you guys ingame!
  • l0th4ri0l0th4ri0 Member Posts: 589 Arc User
    ... And just as promised the spam is back, saturating zonechat with the same messages repeated over and over and over again across different cycled accounts. Golly... sure would be nice to be able to block the dollar sign with a clientside filter. Maybe add c0m to that as well. Hey! Where did all the spam go? :wink:
  • This content has been removed.
  • mrgiggles651mrgiggles651 Member Posts: 790 Arc User


    PS I checked PE just now. Spam is definitely WAAAY down.

    Coincidentally it's Chinese New Year, basically everyone in China is on vacation.
    I wasted five million AD promoting the Foundry.
  • wylonuswylonus Member, NW M9 Playtest Posts: 2,376 Arc User
    geez, i hadnt been in PE for few weeks since winterfest to get away from PE's spams, so, last night...
    fastest spams i ever seen, and they used multiple chat formats with "LFG", "Trade", "Local", and auction channel, all within 15 seconds cycling.
    also i saw 5-6 spammers, and i put them on ignore, and only to get more new names start spamming as soon i check Boward weekly quests, fuel merchant, and seal vendor.

  • l0th4ri0l0th4ri0 Member Posts: 589 Arc User
    wylonus said:

    geez, i hadnt been in PE for few weeks since winterfest to get away from PE's spams, so, last night...
    fastest spams i ever seen, and they used multiple chat formats with "LFG", "Trade", "Local", and auction channel, all within 15 seconds cycling.
    also i saw 5-6 spammers, and i put them on ignore, and only to get more new names start spamming as soon i check Boward weekly quests, fuel merchant, and seal vendor.

    If you had been able to block the dollar sign and "c0m" then you would have seen none of that. The spammers could make as many accounts and cycle through as much as they want and you would have seen nothing.

    Sure would be nice...
  • regenerderegenerde Member Posts: 3,050 Arc User
    Add some "cheap", "fast", "stock" and "USD" to that filter, and it would really start causing problems for those AD sellers...
    I do believe in killing the messenger...
    Want to know why?
    Because it sends a message!
Sign In or Register to comment.