Mass disconnection thread

Options
dblazen1
dblazen1 Posts: 1,177 Arc User
edited September 2016 in General Discussion
Update on the thread (i'll just include thenamesdomino and orinj posts in the main posts so people will see it right away:
Going to jump in here, we are aware of all the disconnections that are occurring. It's not limited to PWI, all of our Bejing titles have been affected. We believe that this is a Networking issue at our West Coast datacenter. Unfortunately I don't have more updates than that, but we are working on it. Have been for the past 3 days.
orinj wrote: »
Going to jump in here, we are aware of all the disconnections that are occurring. It's not limited to PWI, all of our Bejing titles have been affected. We believe that this is a Networking issue at our West Coast datacenter. Unfortunately I don't have more updates than that, but we are working on it. Have been for the past 3 days.

Research continues. Network failures + hardware failures now. We'll be swapping out some switches soon. Thanks for your patience.


Considering the explosion of threads the last few days regarding this topic:
This thread is for you guys.

Instead of making a lot of single threads, let's just make a collective one, where each and everyone one of you (if you want) should post a traceroute from your end to the server when the problem occurs.

So what should you do? (instead of reading this thread, because that has a lot of technical detail about why and what, I am shamelessly copy posting very few items from that thread)
Do the following:
Step 1: Get the address of the server you play on:
Twilight Temple = pwigc2.perfectworld.com
Etherblade = pwiwest4.perfectworld.com
Tideswell = pwieast2.perfectworld.com
Dawn Glory = pwieu3.en.perfectworld.eu

Step 2: open a command prompt; click your windows start button, then write cmd, press enter.

Step 3: write the following: tracert <your server address>

Step 4: after it's completely done, copy paste the result here, in a code block
(in case you have trouble copy pasting: after it's done, rightclick in the command prompt, choose mark, then mark all the text, then rightclick again, it will now be copied)
(to be nice: tidy up your paste data in notepad)

example traceroute to twilight temple:
C:\Users\admin>tracert pwigc2.perfectworld.com

Tracing route to pwigc2.perfectworld.com [66.151.133.238]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2     1 ms    <1 ms    <1 ms  92-111-36-73.static.chello.nl [92.111.36.73]
  3    10 ms    10 ms     7 ms  10.119.224.1
  4    10 ms     6 ms     7 ms  212.142.53.225
  5    11 ms     7 ms     9 ms  212.142.11.105
  6     *        *        *     Request timed out.
  7    21 ms    11 ms    12 ms  nl-ams04a-ri2-ae6-0.aorta.net [84.116.135.182]
  8    12 ms    13 ms    12 ms  ae-6.r25.amstnl02.nl.bb.gin.ntt.net [80.249.208.36]
  9   102 ms   112 ms   100 ms  ae-5.r23.asbnva02.us.bb.gin.ntt.net [129.250.6.162]
 10   170 ms   170 ms   171 ms  ae-10.r22.snjsca04.us.bb.gin.ntt.net [129.250.6.237]
 11   172 ms   171 ms   169 ms  ae-19.r01.snjsca04.us.bb.gin.ntt.net [129.250.3.27]
 12   168 ms   169 ms   169 ms  ae-0.internap.snjsca04.us.bb.gin.ntt.net [157.238.64.138]
 13   250 ms   231 ms   296 ms  border1.t8-1-bbnet2.sje004.pnap.net [66.151.144.79]
 14   172 ms   171 ms   172 ms  perfectworldent-5.border1.sje004.pnap.net [66.151.145.166]
 15     *        *        *     Request timed out.
 16   173 ms   173 ms   177 ms  66.151.133.238

in this specific trace, hop 6 and 15 are ok, this is because those nodes don't react to ICMP packets.
If it were to actually drop packets, the increase in MS would be a lot higher than it is in this trace.
This trace tells me that I can be connected to the server properly, however, it would be laggy.
hop 13 has a pretty big increase, and if i'm correct, that's where PWE's end starts. (correct me if i'm wrong there)

@orinj, @thenamesdomino, @arspaulina#4310 can any of you 3 confirm if there's any issues going on at your end, since the issues from traceroutes i have seen in the last few days (at least the limited amount i've seen) all seem to indicate somethings wrong on PWE's part in this case.

Now go forth and post those traceroutes when you are experience DC's!​​
Jws3dXe.gif
The only fitting image for this forum.

Post edited by dblazen1 on

Comments

  • heerohex#3018
    heerohex#3018 Posts: 4,807 Community Moderator
    edited August 2016
    Options
    Their are now 4 threads open on this subject.

    The info is so scattered it going to be hard to sort it out.

    I have also posted various info on this subject as results of info others have posted.

    It would seam to either be a node failing or the internal server balance. Either way Hopefully a restart should fix some of the presented problems. Constantly pining the servers will cause more issues with bandwidth. So if you must do so keep it to a minimum until we have something back from PWE.

    Thanks.

    Edit make that about 5 threads.​​
  • greenfire312
    greenfire312 Posts: 269 Arc User
    Options
    Let me summarize my observations and my personal theory from the other thread. @orinj @thenamesdomino this might be helpful to pass along to your server folks.

    1. It occurs in waves of mass DCs with periods of relative calm in between.
    2. You are more likely to DC if you're in an instance. I have observed the behavior in DFSP and in Eden so far.
    3. Whether or not you're using Arc has nothing to do with it (none of my computers use Arc, or Steam for that matter, to connect)
    4. Most importantly, only certain characters/accounts are affected.

    To explain #4... my main account contains four active characters, and my alt account contains one. Of these five, only the alt account character suffers from the DC problem. Any character on my main account remained on the server, even in DFSP during a mass DC incident. These conditions persist across many logins over multiple days, including periods of being offline which should be more than long enough for any relevant session data to expire. And yet, the issue continues to occur to the same character(s), while others on the same computer (which has no problem pinging the game server) are entirely immune.

    My current theory - just an educated guess - is that multiple servers or sub-servers exist behind each of the publicly-visible ones (e.g. Etherblade, Tideswell), and the issue lies with one particular server in this group. These other servers would either be part of a specific game server (e.g. Etherblade would have a certain number of subservers to help manage its load), or they'd be separate from any of the game servers and are specifically used to handle logins, sessions/connections, etc.

    Come to think of it, the latter setup seems more and more likely to me - because if it were just one of Etherblade's subservers, for instance, only Etherblade would be having the problem. Many games have separate login servers, and I wouldn't be surprised if PWE has multiple ones which you have to go through before you ever get to the game servers themselves. If even one of these is having issues, every character/account tracked through that server would suffer the result. Immediately after DCing, you can load up the server list and find that your game server is not pinging within the client - whereas if you were to ping it directly from the command prompt, you'd get a normal response (and again, in my particular case, only one of my clients - the one specifically running my alt account - is affected).

    What's strange to me is how these tracking relationships persist across logins - I would think it randomizes which login server you connect to each time, but that doesn't seem to be the case.

    What I would ask @orinj and company right now is this: Could you confirm if there are issues with any server(s) on your end? And if so, could we be enlightened as to what server(s) aside from the game server are involved in our connection to your side? And do any of them have external names/addresses we can ping, or are they all internal to your network?
    Mains: Miugre / LigerKing
    Etherblade server
  • arspaulina#4310
    arspaulina#4310 Posts: 347 Community Moderator
    Options
    I'm slowly in the process of closing out all the other threads to try and get it all rerouted here. I'm not sure how much information I can get out of them regarding server infrastructure, but I can certainly try. (Also I'm a tad bit behind in delivering news for other things such as TW rewards so I've got to hop to that too)
    ⋆Have a question? Feel free to shoot me an ask!⋆
  • greenfire312
    greenfire312 Posts: 269 Arc User
    Options
    No doubt there's a limit to how much they can reveal. And assuming there are sub-servers, many if not most of them will be internal only, meaning we couldn't ping them anyway. But if PWI has login servers at all, at least one of them should be pingable - if I understand correctly, anything that responds directly to external requests (i.e. our logins) should be pingable just like the game servers are. Having the names of all servers that handle external requests would be extremely helpful in giving us an idea of whether an issue lies with PWE or not. Some games even have unofficial server-status websites where you can check if one of the login servers, etc. isn't responding at the time.
    Mains: Miugre / LigerKing
    Etherblade server
  • xskysoldier
    xskysoldier Posts: 128 Arc User
    Options
    What's going on with this game? Are you guys slowly shutting down? I can hardly do anything with these mass dc's.


  • mymax64
    mymax64 Posts: 4 Arc User
    Options
    Everyone is assuming that PWE/Arc cares about their customers experience. There is no indication that they do or they would officially address the issue here in the forums where their customers are complaining. PWI is the evil stepchild that PWE does not really want.

    Everyone just has to get used to the DCs like it is part of the game.
  • arspaulina#4310
    arspaulina#4310 Posts: 347 Community Moderator
    Options
    Im pretty sure they still care if Orinj takes time out of his schedule to answer things. Maybe they don't have a solid answer they can give and rather than string you along with Soon(tm), they decide to remain silent until they have a better beat on whats going on. I'm doing my best to ping them, but as I mentioned in another thread, they themselves were having a few issues at their offices so it might be more widespread than we thought (or could be unrelated who knows).

    Just because they don't instantly answer within a few hours/days doesn't mean they don't care.
    ⋆Have a question? Feel free to shoot me an ask!⋆
  • mymax64
    mymax64 Posts: 4 Arc User
    Options
    I have been seeing issues and no responses for most of the 8 years I have been playing the game. The presence of GMs in PWI is like seeing a unicorn in real life. The current issue of DCs started on Friday.
  • greenfire312
    greenfire312 Posts: 269 Arc User
    Options
    Yes, the issues started on Friday... specifically Friday evening, during NW as I recall. Start of the weekend, when they'd probably have a skeleton crew at best. They've had a couple of days since then, true, and we really do need an update (even if it's just acknowledgement of the issue and what their next step will be), but concluding that they don't care isn't going to accomplish anything.

    Besides, if you really have been playing for 8 years, you would know perfectly well that the actual GMs have nothing to do with this (unless said GMs double as the sysadmins). I've been playing 8 years too, and I've personally watched the stupidest possible decisions out of PWE and the most opaque of CMs try to cover for them over the years. All that's going on here is that PWE is having hardware problems and orinj and domino are dragging their feet on giving us an update for longer than we'd probably like. This is nowhere near the nadir of PWE customer relations over the years, so it'd be good if we could avoid busting out the "PWE hates their customers" card just yet.
    Mains: Miugre / LigerKing
    Etherblade server
  • hypereccentrik
    hypereccentrik Posts: 529 Arc User
    Options
    I believe that these dc'so are happening too often and to too many people for it to be badluck on their end.
    Definitely makes me not want to cash shop.
  • thenamesdomino
    thenamesdomino Posts: 545 Perfect World Employee
    Options
    Going to jump in here, we are aware of all the disconnections that are occurring. It's not limited to PWI, all of our Bejing titles have been affected. We believe that this is a Networking issue at our West Coast datacenter. Unfortunately I don't have more updates than that, but we are working on it. Have been for the past 3 days.
  • mydarksun
    mydarksun Posts: 11 Arc User
    edited August 2016
    Options
    Going to jump in here, we are aware of all the disconnections that are occurring. It's not limited to PWI, all of our Bejing titles have been affected. We believe that this is a Networking issue at our West Coast datacenter. Unfortunately I don't have more updates than that, but we are working on it. Have been for the past 3 days.

    Thank you for the update. It is nice to know you are aware of the problem and trying to fix it. It is exactly the type of response that most of us have been wanting. I do understand that network issues can be complicated. I am a retired network administrator.
  • greenfire312
    greenfire312 Posts: 269 Arc User
    edited August 2016
    Options
    mydarksun wrote: »
    Going to jump in here, we are aware of all the disconnections that are occurring. It's not limited to PWI, all of our Bejing titles have been affected. We believe that this is a Networking issue at our West Coast datacenter. Unfortunately I don't have more updates than that, but we are working on it. Have been for the past 3 days.

    Thank you for the update. It is nice to know you are aware of the problem and trying to fix it. It is exactly the type of response that most of us have been wanting. I do understand that network issues can be complicated. I am a retired network administrator.
    Seconded - updates are always, always appreciated. Even if it's "we haven't made any progress yet but we tried X today."

    I'd be very much interested in your take on this situation, as I'm sure your guesses would be far more educated than mine. What could cause this kind of chronic problem and yet consistently limit it to specific characters/accounts while leaving others immune?

    Also, if it's just the west coast datacenter, is Tideswell unaffected? Can anyone confirm? I've seen the thread on TT server and personally experienced it on Etherblade, so that sounds about right...
    Mains: Miugre / LigerKing
    Etherblade server
  • heerohex#3018
    heerohex#3018 Posts: 4,807 Community Moderator
    Options
    Going to jump in here, we are aware of all the disconnections that are occurring. It's not limited to PWI, all of our Bejing titles have been affected. We believe that this is a Networking issue at our West Coast datacenter. Unfortunately I don't have more updates than that, but we are working on it. Have been for the past 3 days.

    Thank you Domino.​​
  • mymax64
    mymax64 Posts: 4 Arc User
    edited August 2016
    Options
    Thank you for the update Domino. Keeping us informed, even if you feel you do not have all the answers, will always help. It lets us know that you are aware of what we are going through and are working on it.
  • orinj
    orinj Posts: 460 admin
    Options
    Going to jump in here, we are aware of all the disconnections that are occurring. It's not limited to PWI, all of our Bejing titles have been affected. We believe that this is a Networking issue at our West Coast datacenter. Unfortunately I don't have more updates than that, but we are working on it. Have been for the past 3 days.

    Research continues. Network failures + hardware failures now. We'll be swapping out some switches soon. Thanks for your patience.
    F.E.A.R. has two meanings: Forget Everything And Run - OR - Face Everything And Rise. The choice is yours, my friend.
  • mymax64
    mymax64 Posts: 4 Arc User
    Options
    Thank you for the update Orinj
  • mydarksun
    mydarksun Posts: 11 Arc User
    Options
    Thank You for the update. :)