test content
What is the Arc Client?
Install Arc

Why not patch on shard maintenance day?

Has there ever been an explanation why they don´t do it?
Instead of harassing us with two downtimes?

Comments

  • greywyndgreywynd Member, NW M9 Playtest Posts: 7,083 Arc User
    Maintenance is usually patch, server restart, or server upgrade/repair.
    I'm not looking for forgiveness, and I'm way past asking permission. Earth just lost her best defender, so we're here to fight. And if you want to stand in our way, we'll fight you too.
  • plasticbatplasticbat Member, NW M9 Playtest Posts: 12,184 Arc User

    Has there ever been an explanation why they don´t do it?
    Instead of harassing us with two downtimes?

    If they did not patch it yesterday, they would not need a patch today.
    *** The game can read your mind. If you want it, you won't get it. If you don't expect to get it, you will. ***
  • At least their patches are not lasting days because of patching to fix patches to fix the update to fix the patch. It's not like this game is EQ, I don't mind a few hours compared to days worth.
  • sandukutupusandukutupu Member Posts: 2,285 Arc User
    A very good friend of mine works for a database engineering team in Ypsilanti, Michigan. They do work on banks, department stores, etc. dealing with real world money. He explained to me with the follow analogy about the difference between maintenance back ups and patches. It is my sincere hope this helps others understand the process better as well.

    Imagine you write a book but during the week you make changes to several pages of your book on a copy called the Preview shard. When you patch you only need to replace the changes and nothing else. It is much quicker, than making another copy of the book called a back up.

    Most banks do nightly maintenance back ups at 2 am or 3 am, that process can last several hours. Online video games cannot afford this luxury as they do not close for any reason other than maintenance.

    Now back to the analogy about the book. The models and textures for this game are on your PC in libraries. They are not called by a name like dinosaur or waitress. They are numbered in the order they appear in the library, such as 2093874. What happens sometimes, the numbers get mixed up and we get dinosaurs instead of waitress. Like replacing the wrong page numbers in the patch of your book. Oops!

    The Cryptic database tells the clients on our computers to display model number 2093784 instead of 2093874 and we get Klingons instead of orcs. Actually I think that would be a stretch, because the STO libraries exist in a separate folder. Basically that analogy, has gone a long way to help me comprehend how a database sends the wrong reference number and my client displays the wrong models.
    wb-cenders.gif
  • valka#6506 valka Member Posts: 14 Arc User
    > @sandukutupu said:
    > A very good friend of mine works for a database engineering team in Ypsilanti, Michigan. They do work on banks, department stores, etc. dealing with real world money. He explained to me with the follow analogy about the difference between maintenance back ups and patches. It is my sincere hope this helps others understand the process better as well.
    >
    > Imagine you write a book but during the week you make changes to several pages of your book on a copy called the Preview shard. When you patch you only need to replace the changes and nothing else. It is much quicker, than making another copy of the book called a back up.
    >
    > Most banks do nightly maintenance back ups at 2 am or 3 am, that process can last several hours. Online video games cannot afford this luxury as they do not close for any reason other than maintenance.
    >
    <snip>
    If this was going to cripple ATM/Banks nationwide I heartily agree but it is not. Given that ARC developers don't even know when Thanksgiving is I'm guessing your 2am is their 2pm and they are deploying on their schedule? Also happy Diwali
  • greywyndgreywynd Member, NW M9 Playtest Posts: 7,083 Arc User


    If this was going to cripple ATM/Banks nationwide I heartily agree but it is not. Given that ARC developers don't even know when Thanksgiving is I'm guessing your 2am is their 2pm and they are deploying on their schedule? Also happy Diwali

    Of course they do it on their schedule. Why should they disrupt their work schedule because it is inconvenient somewhere else in the world?

    And no matter when they do it, it will be inconvenient somewhere in the world.
    I'm not looking for forgiveness, and I'm way past asking permission. Earth just lost her best defender, so we're here to fight. And if you want to stand in our way, we'll fight you too.
  • valka#6506 valka Member Posts: 14 Arc User
    > @greywynd said:
    > (Quote)
    > Of course they do it on their schedule. Why should they disrupt their work schedule because it is inconvenient somewhere else in the world?
    >
    > And no matter when they do it, it will be inconvenient somewhere in the world.

    As a professional developer I am going to say no. We do it when it is convenient for our target audience not when it is personally convenient for us.

    इंडोनेशिया में अभी 4 लोगों के माफी मांगने के साथ ही आपको लक्ष्मी का आशीर्वाद भी
  • greywyndgreywynd Member, NW M9 Playtest Posts: 7,083 Arc User
    Which part of the world is their target audience when their players are all over the world?
    I'm not looking for forgiveness, and I'm way past asking permission. Earth just lost her best defender, so we're here to fight. And if you want to stand in our way, we'll fight you too.
  • sandukutupusandukutupu Member Posts: 2,285 Arc User

    > @greywynd said:

    > (Quote)

    > Of course they do it on their schedule. Why should they disrupt their work schedule because it is inconvenient somewhere else in the world?

    >

    > And no matter when they do it, it will be inconvenient somewhere in the world.



    As a professional developer I am going to say no. We do it when it is convenient for our target audience not when it is personally convenient for us.



    इंडोनेशिया में अभी 4 लोगों के माफी मांगने के साथ ही आपको लक्ष्मी का आशीर्वाद भी

    Just how do you propose they do that? If you are talking about the North American audience then they cannot know or guess when people will be online playing the game. The continent is 4 time zones wide. Since I have been online at 2 am or 4am in the morning there is always someone playing this game. As a result, someone has to not play for 2 hours so they can patch the game. A shopping center, a bank, a mall, are businesses that can and do shut down nightly, they can spend all night doing back ups. Online games cannot simply hang out a closed for the weekend sign or gone fishing.
    wb-cenders.gif
  • greywyndgreywynd Member, NW M9 Playtest Posts: 7,083 Arc User
    Well, they can, but then we'd be wondering what broke that required that much downtime.
    I'm not looking for forgiveness, and I'm way past asking permission. Earth just lost her best defender, so we're here to fight. And if you want to stand in our way, we'll fight you too.
  • kreatyvekreatyve Member, Neverwinter Moderator, NW M9 Playtest Posts: 10,545 Community Moderator
    Due to the fact that it's always going to be inconvenient for someone in the world, they do it at a time that's best convenient for them, I'm sure. They don't want to do it in the middle of the night because sleepy devs make more mistakes.
    My opinions are my own. I do not work for PWE or Cryptic. - Forum Rules - Protector's Enclave Discord - I play on Xbox
    Any of my comments not posted in orange are based on my own personal opinion and not official.
    Any messages written in orange are official moderation messages. Signature images are now fixed!
    kuI2v8l.png
  • valka#6506 valka Member Posts: 14 Arc User
    > @sandukutupu said:
    > (Quote)
    > Just how do you propose they do that? If you are talking about the North American audience then they cannot know or guess when people will be online playing the game. The continent is 4 time zones wide. Since I have been online at 2 am or 4am in the morning there is always someone playing this game. As a result, someone has to not play for 2 hours so they can patch the game. A shopping center, a bank, a mall, are businesses that can and do shut down nightly, they can spend all night doing back ups. Online games cannot simply hang out a closed for the weekend sign or gone fishing.

    The best way would be a "hot spare" of the production environment. You can then deploy updates in (for the consumer) minutes not hours. It may still take 2 hours (but it won't. a 2 hour deploy is a failed deploy and rollback) but they won't ever see it.

    If you don't have the budget for that it is trivial for developers to see how many people are on at what hour and pick a time of low activity, just like it is trivial to patch things like graphical assets ahead of time.

    Patching during the closing hours of an event is always the wrong answer unless your server farm is literally on fire (which may indeed require the full 2 hours)
  • sandukutupusandukutupu Member Posts: 2,285 Arc User
    LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?
    wb-cenders.gif
  • valka#6506 valka Member Posts: 14 Arc User
    > @sandukutupu said:
    > LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?

    Good to know. How long have you worked there?
  • micky1p00micky1p00 Member, NW M9 Playtest Posts: 3,594 Arc User
    edited November 2019

    LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?

    You "LOL" at them, when they actually understand more, and more correct than you.

    Every number of instances are spun on a virtual or physical server, with their own IP. A coordinator server redirects a player on zone load.

    The authentication server is separate, the AH is a separate server, the patchserver, you guessed it right, a separate server, inventory database, with data synced to instance server on client load, separate database server.
    And so on, not counting the preview servers, and all the internal networking. No matter how many of those virtualized, share same VM or hardware as a software, and how many, exactly, multiple hardware servers it all takes, it is a cluster, or a farm if perhaps small one compared to some other companies.


    Post edited by micky1p00 on
  • putzboy78putzboy78 Member, NW M9 Playtest Posts: 1,950 Arc User
    kreatyve said:

    Due to the fact that it's always going to be inconvenient for someone in the world, they do it at a time that's best convenient for them, I'm sure. They don't want to do it in the middle of the night because sleepy devs make more mistakes.

    At current activity levels, there are probably more support people logged into the server than gamers. So yeah, doing it when its most convenient to cryptic totally makes sense
  • sandukutupusandukutupu Member Posts: 2,285 Arc User
    micky1p00 said:

    LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?

    You "LOL" at them, when they actually understand more, and more correct than you.

    Every number of instances are spun on a virtual or physical server, with their own IP. A coordinator server redirects a player on zone load.

    The authentication server is separate, the AH is a separate server, the patchserver, you guessed it right, a separate server, inventory database, with data synced to instance server on client load, separate database server.
    And so on, not counting the preview servers, and all the internal networking. No matter how many of those virtualized, share same VM or hardware as a software, and how many, exactly, multiple hardware servers it all takes, it is a cluster, or a farm if perhaps small one compared to some other companies.

    I laughed at you too Micky, just so you don't feel left out. <3 Virtual or otherwise, this is not a server farm. Frankly I am amazed at what Cryptic can accomplish with their limitations. I am not poking fun at anyone's knowledge or ignorance. I already stated the facts above, I needed an expert to explain things to me in the most simple terms to understand. I assume you believe they should be "hot swapping" the server, so there is no down time as well? My IT personnel express there is a considerable risk to the hardware. Now the question is, would you be willing to lose everything on your account because it wasn't 100% sufficient? I can wait 2 or even 6 hours a week for the server maintenance. I don't want them to rush and break all the eggs.
    wb-cenders.gif
  • putzboy78putzboy78 Member, NW M9 Playtest Posts: 1,950 Arc User
    server is an over used term. often times when people say "server" they are talking about services and instances. Effectively what you have is a multitiered architectured system with varies services running on it. As an example you will have network services which include authentication, you will have database to store the information, you will have application to process information, depending on how its constructed you can have a separate database for mail than inventory. This can all sit on one physical and/or virtual server or the services can be divided up to balance loads... ultimately none of this matters because the ones who knows if and how the systems are divided up are not the people playing this game... its cryptic.

    And none of that is relevant. Cryptic plans on a 2 hour weekly update schedule which includes server maintenance. Sometimes its faster than others because the maintenance has less requirements. Sometimes they schedule additional time if they know they are going to need it.

    Is it possible to pre load an application update on a separate instance and move the pointers to the new instance to reduce update time, absolutely. And there is no way for anyone playing the game to know if they are or not doing that.

    And @sandukutupu it could be possible (depending on the multitiered design) to preload an application change without impacting the data assuming the database is separate. The point being is we don't know how cryptic divides its architecture and therefore can not assume that preloading is possible without sacrificing data integrity. It's a fair question to ask, but I'm 100% sure cryptic will answer with silence and think its nunya
  • micky1p00micky1p00 Member, NW M9 Playtest Posts: 3,594 Arc User
    edited November 2019

    micky1p00 said:

    LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?

    You "LOL" at them, when they actually understand more, and more correct than you.

    Every number of instances are spun on a virtual or physical server, with their own IP. A coordinator server redirects a player on zone load.

    The authentication server is separate, the AH is a separate server, the patchserver, you guessed it right, a separate server, inventory database, with data synced to instance server on client load, separate database server.
    And so on, not counting the preview servers, and all the internal networking. No matter how many of those virtualized, share same VM or hardware as a software, and how many, exactly, multiple hardware servers it all takes, it is a cluster, or a farm if perhaps small one compared to some other companies.

    I laughed at you too Micky, just so you don't feel left out. <3 Virtual or otherwise, this is not a server farm. Frankly I am amazed at what Cryptic can accomplish with their limitations. I am not poking fun at anyone's knowledge or ignorance. I already stated the facts above, I needed an expert to explain things to me in the most simple terms to understand. I assume you believe they should be "hot swapping" the server, so there is no down time as well? My IT personnel express there is a considerable risk to the hardware. Now the question is, would you be willing to lose everything on your account because it wasn't 100% sufficient? I can wait 2 or even 6 hours a week for the server maintenance. I don't want them to rush and break all the eggs.</p>
    I just can't even.....

    Hot swapping the server in this case has nothing to do with hardware. You have no clue what you talk about, miscommunicate it to whomever understands and miscomunicate it back here. Which is not the issue on its own, but then talk about ignorance...

    The only single action that done to "hot swap" the game version is to change the ip number of the server coordinator and authentication server the client has, from old version set A to new version set B. And request the client to re-log to the new set.

    And no I don't care either way, live back up requires capability the custom made Crpytic database system may not have, so it needs to be done on a higher level, OS or VM, which will probably impact they ability to look up old chars and roll back.
    offline backup just saves them the complexity, and costs of maintaining the system, as offline backups / cloning are much simpler and easier to do.

    But that doesn't negate the fact that you have no clue what you write about, while talk about other people ignorance.
  • valka#6506 valka Member Posts: 14 Arc User
    > @putzboy78 said:

    > And @sandukutupu it could be possible (depending on the multitiered design) to preload an application change without impacting the data assuming the database is separate. The point being is we don't know how cryptic divides its architecture and therefore can not assume that preloading is possible without sacrificing data integrity. It's a fair question to ask, but I'm 100% sure cryptic will answer with silence and think its nunya

    Thank you voice of sanity. Your user data is in a database. If a deploy fails the worst case scenario is that it gets restored from database backups. It isn't all getting shredded because a developer is sleepy.

    I suspect it's a conventional database but I guess Cryptic could have re-implemented this using Redis or MongoDB or some other in-memory system. In theory it could even be a flat file sitting on your PC but since it's no longer 1994 and those are trivial to hack...probably not.
  • kreatyvekreatyve Member, Neverwinter Moderator, NW M9 Playtest Posts: 10,545 Community Moderator
    And I do believe that at this point, this thread has ran its course.
    My opinions are my own. I do not work for PWE or Cryptic. - Forum Rules - Protector's Enclave Discord - I play on Xbox
    Any of my comments not posted in orange are based on my own personal opinion and not official.
    Any messages written in orange are official moderation messages. Signature images are now fixed!
    kuI2v8l.png
This discussion has been closed.