Lag Trek Online

gradii

is there any ETA on when the game will become playable again? Currently playing the "Rubberband across the entire SFA campus repeatedly" minigame. excellent content.

questerius

Sadly it is a bit of a tradition to have rubberbanding around events, QWW included.
Best guess is that it'll become a bit better once QWW has ended.

markhawkman

questerius wrote: »

Sadly it is a bit of a tradition to have rubberbanding around events, QWW included.
Best guess is that it'll become a bit better once QWW has ended.

I'm not sure it's the event. It's like the entire internet is overloaded right now.

questerius

markhawkman wrote: »

questerius wrote: »

Sadly it is a bit of a tradition to have rubberbanding around events, QWW included.
Best guess is that it'll become a bit better once QWW has ended.

I'm not sure it's the event. It's like the entire internet is overloaded right now.

We should add an algorithm to the internet that whoever watches adult content (you know the 18+ type) is automatically kicked off the net.
Imagine the amount of bandwidth that would open up.

ussvaliant

Nothing to do with the winter event this has been ongoing for nearly a year. Tracerts all point to Akamai Technologies where the connection times spike.

1 <1 ms <1 ms <1 ms ttrouter [192.168.1.1]
2 * * * Request timed out.
3 9 ms 9 ms 9 ms ae54-ner002.csm.as13285.net [78.144.1.19]
4 12 ms 11 ms 12 ms ae54-scr101.loh.as13285.net [78.144.1.18]
5 13 ms 13 ms 13 ms akamai.prolexic.com [195.66.224.31]
6 13 ms 13 ms 12 ms po110.bs-a.sech-lon2.netarch.akamai.com [72.52.60.192]
7 14 ms 13 ms 13 ms a72-52-1-189.deploy.static.akamaitechnologies.com [72.52.1.189]
8 13 ms 13 ms 13 ms ae120.access-a.sech-lon2.netarch.akamai.com [72.52.60.197]
9 85 ms 85 ms 85 ms 93.191.173.11
10 91 ms 91 ms 91 ms a209-200-147-156.deploy.static.akamaitechnologies.com [209.200.147.156]
11 98 ms 93 ms 93 ms 198.49.243.237
12 92 ms 91 ms 91 ms patchserver.crypticstudios.com [208.95.184.200]

This Lag/Latency/Rubberbanding issue is going nowhere until Cryptic addresses this.

rattler2

Cryptic could also invest a bit on buffing the server and maybe getting a better ISP.

I give them some slack because they're not as big as Blizzard or even Square Enix who have Trip-A titles like WoW or FF14, but you gotta admit they did a pretty decent job with what they had.
While I haven't played yet today, I do get a bit of rubber banding here and there on STO. Don't really get any on FF14. But the difference is probably due to size and, ultimately, money.

jonsills

Cryptic can't "address this", other than complaining to their ISP. Akamai Technologies is a third-party company, whose software is designed to blunt the impact of DDoS attacks by slowing internet traffic to give the affected servers time to respond. Unfortunately, while the programmers were techbros, they were clearly not gamers, as their software appears to treat user spikes on MMOs as if they were a DDoS attack from a botnet. This was first noted, so far as I can tell, on Elder Scrolls Online; Bethesda, a much larger company than Cryptic, has yet to get any changes made to Akamai's software, so it seems doubtful that Cryptic will get much further.

sennahcherib

yes, currently the lag and the rubberbanding are awful, some tfos are unplayable. Today in qonos tfo, impossible to use my weapon (greyed icons) or my skills; each time that I tried to use a skill nothing happened. same thing in space, I use a skill, nothing happen, i have to push on the skill icon at least 3 times, or sometimes nothing happens but the cd is launched.

At this point, it is not serious at all. I don't understand why cryptic is not worried by these problems and use something else instead of akamai. after all, it is all about the fame of the game

sennahcherib

I know but a contract can be broken if 1 of the part doesn't do its job, or doesn't follow the contract. i don't think that cryptic has signed a contract where these lag and rubberbanding were a part of the contract.

please stop to want making people look like fools, after a moment it's too visible and lacks subtlety.

jonsills

Cryptic isn't using Akamai. Their ISP is using Akamai. And given the monopolies handed out to many broadband companies, Cryptic might not even have any options about using the ISP they're on; around here, for example, Rainier Connect is the major competition in the ISP market for Comcast, but their hardware would be simply incapable of handling the amount of incoming data even the smallest MMO gets. (I've been in their offices - had a temp job reorganizing their filing system when they came to understand that their Internet services were at least as big a business as their cable TV services.)

I mean, if Akamai were employed by the end user, do you honestly believe Bethesda would have permitted them to throttle Elder Scrolls Online for longer than it took to isolate the issue? And yet as of the most recent postings in their forums, it would appear lag is still an issue there.

ussvaliant

You would think with the evolution of the internet and the speeds ISP's are now offering, lag/rubberbanding and latency issues would be reducing not increasing. I guess the infrastructure to support those speeds isn't in place yet ?

jonsills

ussvaliant#6064 wrote: »

You would think with the evolution of the internet and the speeds ISP's are now offering, lag/rubberbanding and latency issues would be reducing not increasing. I guess the infrastructure to support those speeds isn't in place yet ?

As noted, it's not about the ability of the Net to handle the traffic, it's about Akamai's interpretation of what might constitute a Directed Denial of Service (DDoS) attack. A DDoS attack overwhelms the servers by sending multiple, rapid-fire connection requests; at first glance, the data transfer associated with an MMO, especially during special events when traffic loads increase, can look a lot like that. Accordingly, Akamai's software throttles the speed of the data requests, in order to prevent them from overloading the servers. Unfortunately, what that means on our end is that the client is unable to communicate with the server in anything like real-time, resulting in lag (waiting for information) and rubberbanding (correcting from what the client thought your position in-game was to what the server holds it to be).

What can we do about it? Keep complaining, mostly. If you have any sort of "in" with Akamai Technologies, complaining to them might work. Just try not to get angry with Cryptic about it; that makes as much sense as getting angry with Amtrak when the Seattle-Portland train is delayed by a mudslide across the tracks in Tacoma.

foppotee

The lag has increased ever sense Cryptic switched to that company that begins with an "A" earlier this year. Cryptic kept stating they were monitoring & trying to reduce the lag, but I think that was just jib-jab in an attempt to placate the masses. However, lag always increases during events due to the increase players logging in & out & playing. I've noticed increased lag too over the last couple of weeks, but roughly 75%+ of my experience has still been playable without frustration.

leemwatson

jonsills wrote: »

ussvaliant#6064 wrote: »

You would think with the evolution of the internet and the speeds ISP's are now offering, lag/rubberbanding and latency issues would be reducing not increasing. I guess the infrastructure to support those speeds isn't in place yet ?

As noted, it's not about the ability of the Net to handle the traffic, it's about Akamai's interpretation of what might constitute a Directed Denial of Service (DDoS) attack. A DDoS attack overwhelms the servers by sending multiple, rapid-fire connection requests; at first glance, the data transfer associated with an MMO, especially during special events when traffic loads increase, can look a lot like that. Accordingly, Akamai's software throttles the speed of the data requests, in order to prevent them from overloading the servers. Unfortunately, what that means on our end is that the client is unable to communicate with the server in anything like real-time, resulting in lag (waiting for information) and rubberbanding (correcting from what the client thought your position in-game was to what the server holds it to be).

What can we do about it? Keep complaining, mostly. If you have any sort of "in" with Akamai Technologies, complaining to them might work. Just try not to get angry with Cryptic about it; that makes as much sense as getting angry with Amtrak when the Seattle-Portland train is delayed by a mudslide across the tracks in Tacoma.

It makes you wonder how game streaming companies are going to cope doesn't it.

lexusk19

Lag has been horrible for like the last year. They have done nothing about it so far. Dont have high hopes.

rattler2

@jonsills You in the Pacific Northwest too?

pottsey5g

jonsills wrote: »

ussvaliant#6064 wrote: »

You would think with the evolution of the internet and the speeds ISP's are now offering, lag/rubberbanding and latency issues would be reducing not increasing. I guess the infrastructure to support those speeds isn't in place yet ?

“What can we do about it? Keep complaining, mostly. If you have any sort of "in" with Akamai Technologies, complaining to them might work. Just try not to get angry with Cryptic about it; that makes as much sense as getting angry with Amtrak when the Seattle-Portland train is delayed by a mudslide across the tracks in Tacoma.”

The lag started when cryptic switched companies. They could you know switch back. It’s very clear the new ISP is not fit for purpose. If they cannot change contracts then its still Cryptic fault for writing such a bad contract where they cannot leave after a year of problems from the company that should be in breach of any decently written contract.

There are lots Cryptic could do at the very least they could clearly publicly acknowledge the problem in a news post, not as a tiny point hidden at the bottom of an unrelated blog. If it really is unfixable at least explain what is going off instead of leaving players in the dark and expreinceing poor quality game play. Plus it’s not all Akamai's and the ISP some of it’s the server which is directly Cryptic's fault and problem. The problem with the ground maps seems to be Cryptic's server at fault which is amplified by the other IPS problem.

leemwatson

pottsey5g wrote: »

jonsills wrote: »

ussvaliant#6064 wrote: »

You would think with the evolution of the internet and the speeds ISP's are now offering, lag/rubberbanding and latency issues would be reducing not increasing. I guess the infrastructure to support those speeds isn't in place yet ?

“What can we do about it? Keep complaining, mostly. If you have any sort of "in" with Akamai Technologies, complaining to them might work. Just try not to get angry with Cryptic about it; that makes as much sense as getting angry with Amtrak when the Seattle-Portland train is delayed by a mudslide across the tracks in Tacoma.”

The lag started when cryptic switched companies. They could you know switch back. It’s very clear the new ISP is not fit for purpose. If they cannot change contracts then its still Cryptic fault for writing such a bad contract where they cannot leave after a year of problems from the company that should be in breach of any decently written contract.

There are lots Cryptic could do at the very least they could clearly publicly acknowledge the problem in a news post, not as a tiny point hidden at the bottom of an unrelated blog. If it really is unfixable at least explain what is going off instead of leaving players in the dark and expreinceing poor quality game play. Plus it’s not all Akamai's and the ISP some of it’s the server which is directly Cryptic's fault and problem. The problem with the ground maps seems to be Cryptic's server at fault which is amplified by the other IPS problem.

If Cryptic openly blame Akamai, then it's opening them to litigation. You don't go throwing about blame without absolute and undeniable facts. But I will say, once again, that lag appears to be regional, because I don't experience it often at all.

darkbladejk

People need to understand that diagnosing lag isn't as simple as people think it is as there's a thousand and one different factors that can effect lag. Some of those include how powerful of a computer you have, how good of servers Cryptic has, what kind of connection you're getting through your ISP and data as just a few examples. Computers built in 2019 are going to have much easier time with the game than a rig someone bought back in 2014. The internet functions sort of as a digital version of our snail mail systems. When you send a command to the servers to move your toon to the left and fire their weapon, that command doesn't make a straight jump to the server as though you have a dedicated line, it gets routed through several data centers first, much like snail mail letters go through sorting centers. Your command goes through several hops to get there and once the command is received, the server sends back an acknowledgement along that same path. If there is any kind of issue along that path, especially in the physical cable lines, this will manifest typically as lag. If it occurs between your rig and their servers, especially along the lines, the only thing you can really do is try to re-route your connection and come at it from another way if you know how to do that. Otherwise the lag you experience is completely out of Cryptic's control. This assumes that your rig and their servers are working perfectly and there is nothing at either end, such as a hardware issue, that could be contributing to the issue as well. If more than one issue as at play, what causes your lag from one moment to the next may not be the same.

In the case of Akamai, all data I have seen so far points to them being the most likely culprit or at the very least there being a correlation. Akamai is also a security company meant to protect in part against DDoS attacks. The problem with that sort of thing is it can often times be hard to distinguish between legitimate MMORPG traffic, and an actual DDoS attack due to the nature of what a DDoS attack is. A DDoS is designed to basically ping a server so many times in a short period that it can't keep up with the volume of requests and it essentially fails, much like clogging up a fuel line in a card will choke the life from an engine. When a company like that of Akamai detects a potential DDoS it slows down the amount of data going towards the server which is meant to give the folks who own the server time to mount a defense and stop the potential attack.

The big problem, and thus the issue some folks experience, is how do you tell the difference between a full blown DDoS and legitimate MMORPG traffic? How do you allow for the normal traffic a game like STO, WoW, SWTOR, or similar type games will get without leaving them extremely vulnerable to DDoS attacks? That's a question Akamai and every security company has to ask themselves. I had submitted a batch of data to Kael awhile ago. Far as to what the higher-ups on the food chain have decided to do with it I have no idea.

In regards to rubber-banding, there's a couple of things in that regard folks need to understand as well about computers. When computers are given too much data to process or more than they can handle, they start to make assumptions. If hypothetically the most amount of DPS the servers could process is 500k, and you are doing 650k, that extra 150k will not be reflected in what the computer shows you and the maximum result of 500k is what's occurring. If you were moving forward but gave the command to move left when a spike hits, the computer will assume you kept moving forward and ignore the command leading to a data overload, or simply move you back to where you were before the data overload. If there is de-sync between your rig and the server it can display as rubber-banding as well. Your rig says you're in one place but the server says you're in another, so the 2 will typically agree to place you back where they both last agreed you were at.

I wish it were as simple as flipping switches with this stuff, but it's often not.

ussvaliant

I doubt in dyson battlezone where the zone has a population of less than 4 and I simply walk from the beam in area to Commander Arnold and rubberband back is a lack of processing power on the part of my PC.

Windows Home 64bit
2.9GHz quad-core Intel Core i5 processor (Turbo Boost up to 3.6GHz) with 6MB L3 cach
8GB (two 4GB) of 1600MHz DDR3 memory
1TB (5400-rpm) hard drive
NVIDIA GeForce GT 750M graphics processor with 1GB of GDDR5 memory


The issues for me def started when Cryptic/PWE finished those huge behind the scenes updates that saw the game down for half days. Before that unless there genuinely was a connection problem somewhere along the line on either my or there end I rarely got rubberbanding or lag.

In fact ground powers misfiring was a rarity to the point you could count it on 1 hand and still have fingers spare for the rest of the year. Now every effing play session. And space well lets not get started there.

Now yes there are factors outside Cryptics control and we should show some understanding in that regards, But we are connecting and using their product and thus it is their responsibility if they have any concerns over play retention or market image/brand to look into this and resolve it asap. Sure Kael asked for info to pass on but how long does it take to investigate its been many many months.

darkbladejk

ussvaliant#6064 wrote: »

I doubt in dyson battlezone where the zone has a population of less than 4 and I simply walk from the beam in area to Commander Arnold and rubberband back is a lack of processing power on the part of my PC.

Depending on the cause, lag can happen anywhere at any time and without warning.

ussvaliant#6064 wrote: »

Windows Home 64bit
2.9GHz quad-core Intel Core i5 processor (Turbo Boost up to 3.6GHz) with 6MB L3 cach
8GB (two 4GB) of 1600MHz DDR3 memory
1TB (5400-rpm) hard drive
NVIDIA GeForce GT 750M graphics processor with 1GB of GDDR5 memory

To be perfectly blunt dude after looking at those specs they may not be the chief cause of your issues, but they're definitely contributing to it, especially depending on your graphics settings. That hard drive is doing you no favors at that low of a spin speed. Also that graphics card is from 2013 and most of that rig is ancient by today's standards. You're not really going to be running the game at peak performance. Now that's not to say it's incapable, but you're definitely getting bottle-necked hard by that hard drive, and by the graphics card. Also Windows Home doesn't tell us what version of windows it is, windows 7, 8.1, 10 etc, also makes a huge difference. You don't have the power you think you do in that rig. At the very least first chance you get I would pick up a SSD for Windows and STO. The rest of the stuff will function for now but you definitely need to get a newer hard drive with the way the game is today.

ussvaliant#6064 wrote: »

The issues for me def started when Cryptic/PWE finished those huge behind the scenes updates that saw the game down for half days. Before that unless there genuinely was a connection problem somewhere along the line on either my or there end I rarely got rubberbanding or lag.

In fact ground powers misfiring was a rarity to the point you could count it on 1 hand and still have fingers spare for the rest of the year. Now every effing play session. And space well lets not get started there.

This kind of stuff is why they were asking for data to pinpoint commonalities. I've had some folks say it started with the behind the scenes stuff, and others say it was going on before that. With as many different variations as there can be in a computer system it's not going to effect every rig the same. Again there is a thousand and one factors that can contribute to lag, simply because you didn't have it before doesn't preclude you from having it in the future.

ussvaliant#6064 wrote: »

Now yes there are factors outside Cryptics control and we should show some understanding in that regards, But we are connecting and using their product and thus it is their responsibility if they have any concerns over play retention or market image/brand to look into this and resolve it asap. Sure Kael asked for info to pass on but how long does it take to investigate its been many many months.

The only thing Cryptic is responsible for is their own servers. Once they've taken care of their own servers, they have zero control over the health of your rig, or what happens between your rig and their servers. As to how long it takes to fix something, again I'm going to be blunt, do you want it fixed quickly and have to do a ton of minor patches, or do you want it fixed correctly the first time? You can't have both when dealing with this sort of thing. Knowing what's broken is only half the battle as you then need to formulate a fix. Using Kemocite as an example, they knew what was broken for awhile, but actually fixing it took much longer to accomplish.

If it is Akamai as alot of folks are saying, they must have their ducks in a row before going to them and saying "hey you're slowing our stuff down even though it's legit." Then there's the whole bat guano storm of potential contracts if any of those exist and so on. I know this isn't what you want to hear and I wish I didn't have to say it, but when dealing with things outside their servers they generally have zero control over it, and what bits they can address take time. I can tell you they're not ignoring the data being sent, but when dealing with something like this, potentially involving security (Akamai is a security company( you need some kind of protection. I already explained above why it's difficult for not only Cryptic but Akamai as well. A DDoS and legitimate traffic can appear to be the same thing due to how they work. Being able to tell the difference is tricky thing. Once myself or my cohorts know something, you guys will know something.

sci321

The part I've learned from this topic that I find most surprising is that DDoS attacks have enough in common with legitimate MMO internet traffic for the later to be mistaken for the former. Sure, an MMO's servers need to process a lot of pings between it and the numerous clients that connect to it, and a DDoS attack works by flooding a server with so many pings that it can't keep up, but the difference is really SO SMALL that, at a quick glance, it's easy to mistake one for the other?

markhawkman

azrael605 wrote: »

> @sci321 said:
> The part I've learned from this topic that I find most surprising is that DDoS attacks have enough in common with legitimate MMO internet traffic for the later to be mistaken for the former. Sure, an MMO's servers need to process a lot of pings between it and the numerous clients that connect to it, and a DDoS attack works by flooding a server with so many pings that it can't keep up, but the difference is really SO SMALL that, at a quick glance, it's easy to mistake one for the other?

Yes.

Especially since a DDOS setup could use multiple STO game clients.

hallek420staymad

questerius wrote: »

We should add an algorithm to the internet that whoever watches adult content (you know the 18+ type) is automatically kicked off the net.
Imagine the amount of bandwidth that would open up.

Pretty swell idea

sennahcherib wrote: »

it is all about the fame of the game

more like infamy at this point

jslyn

questerius wrote: »

We should add an algorithm to the internet that whoever watches adult content (you know the 18+ type) is automatically kicked off the net.
Imagine the amount of bandwidth that would open up.

I have it on good authority it is those greedy First Responders that keep hogging the internet for their "planning" and "deployment."

protoneous

darkbladejk wrote: »

ussvaliant#6064 wrote: »

I doubt in dyson battlezone where the zone has a population of less than 4 and I simply walk from the beam in area to Commander Arnold and rubberband back is a lack of processing power on the part of my PC.

Depending on the cause, lag can happen anywhere at any time and without warning.

ussvaliant#6064 wrote: »

Windows Home 64bit
2.9GHz quad-core Intel Core i5 processor (Turbo Boost up to 3.6GHz) with 6MB L3 cach
8GB (two 4GB) of 1600MHz DDR3 memory
1TB (5400-rpm) hard drive
NVIDIA GeForce GT 750M graphics processor with 1GB of GDDR5 memory

To be perfectly blunt dude after looking at those specs they may not be the chief cause of your issues, but they're definitely contributing to it, especially depending on your graphics settings. That hard drive is doing you no favors at that low of a spin speed. Also that graphics card is from 2013 and most of that rig is ancient by today's standards. You're not really going to be running the game at peak performance. Now that's not to say it's incapable, but you're definitely getting bottle-necked hard by that hard drive, and by the graphics card. Also Windows Home doesn't tell us what version of windows it is, windows 7, 8.1, 10 etc, also makes a huge difference. You don't have the power you think you do in that rig. At the very least first chance you get I would pick up a SSD for Windows and STO. The rest of the stuff will function for now but you definitely need to get a newer hard drive with the way the game is today.

ussvaliant#6064 wrote: »

The issues for me def started when Cryptic/PWE finished those huge behind the scenes updates that saw the game down for half days. Before that unless there genuinely was a connection problem somewhere along the line on either my or there end I rarely got rubberbanding or lag.

In fact ground powers misfiring was a rarity to the point you could count it on 1 hand and still have fingers spare for the rest of the year. Now every effing play session. And space well lets not get started there.

This kind of stuff is why they were asking for data to pinpoint commonalities. I've had some folks say it started with the behind the scenes stuff, and others say it was going on before that. With as many different variations as there can be in a computer system it's not going to effect every rig the same. Again there is a thousand and one factors that can contribute to lag, simply because you didn't have it before doesn't preclude you from having it in the future.

ussvaliant#6064 wrote: »

Now yes there are factors outside Cryptics control and we should show some understanding in that regards, But we are connecting and using their product and thus it is their responsibility if they have any concerns over play retention or market image/brand to look into this and resolve it asap. Sure Kael asked for info to pass on but how long does it take to investigate its been many many months.

The only thing Cryptic is responsible for is their own servers. Once they've taken care of their own servers, they have zero control over the health of your rig, or what happens between your rig and their servers. As to how long it takes to fix something, again I'm going to be blunt, do you want it fixed quickly and have to do a ton of minor patches, or do you want it fixed correctly the first time? You can't have both when dealing with this sort of thing. Knowing what's broken is only half the battle as you then need to formulate a fix. Using Kemocite as an example, they knew what was broken for awhile, but actually fixing it took much longer to accomplish.

If it is Akamai as alot of folks are saying, they must have their ducks in a row before going to them and saying "hey you're slowing our stuff down even though it's legit." Then there's the whole bat guano storm of potential contracts if any of those exist and so on. I know this isn't what you want to hear and I wish I didn't have to say it, but when dealing with things outside their servers they generally have zero control over it, and what bits they can address take time. I can tell you they're not ignoring the data being sent, but when dealing with something like this, potentially involving security (Akamai is a security company( you need some kind of protection. I already explained above why it's difficult for not only Cryptic but Akamai as well. A DDoS and legitimate traffic can appear to be the same thing due to how they work. Being able to tell the difference is tricky thing. Once myself or my cohorts know something, you guys will know something.

It's good that you go into so much detail as that is the hallmark of a good troubleshooter. However, my laptop is at least several generations newer than ussvaliant's and our results are often the same. Let's not confuse lag with stuttering, frame rates, thermal throttling, bottle-necking, etc.

It's not that other things shouldn't be taken into consideration and discussed, but in this particular case I'm pretty confident that a seasoned player can tell the difference between lag and the rest of the stuff.

I was part of a thread here that (some time ago) debated whether or not there was a measurable increase in game performance from a mechanical hard drive versus a 4x lane PCIe connected SSD. From what I can recall the differences were very negligible. People got their stop watches out to ensure accurate responses.

What version of windows being run making a huge difference is another question mark. I can't tell the difference between Win 7 and 10. Both are 64 bit. Why would one work better than the other?

If I were to ask a general question about trace routes and their interpretation (which as I have read and have observed from previous lag threads can be difficult at best to interpret ) is about where is the comfort zone for total round trip response numbers? From what I've been told anything around 100 ms or less is considered quite good.

If the traceroute is good and the PC is good, are there other aspects from Cryptic's end that could be considered? Terms such as loading, distribution, capacity, or even time to compute or computational limits come to mind but are of course just part of a larger picture, as you've pointed out.