There was a patch maintenance day....today's patch is to correct yesterdays patch....
Halvulv the Barbie
5
greywyndMember, NW M9 PlaytestPosts: 7,150Arc User
Maintenance is usually patch, server restart, or server upgrade/repair.
I'm not looking for forgiveness, and I'm way past asking permission. Earth just lost her best defender, so we're here to fight. And if you want to stand in our way, we'll fight you too.
0
plasticbatMember, NW M9 PlaytestPosts: 12,404Arc User
At least their patches are not lasting days because of patching to fix patches to fix the update to fix the patch. It's not like this game is EQ, I don't mind a few hours compared to days worth.
A very good friend of mine works for a database engineering team in Ypsilanti, Michigan. They do work on banks, department stores, etc. dealing with real world money. He explained to me with the follow analogy about the difference between maintenance back ups and patches. It is my sincere hope this helps others understand the process better as well.
Imagine you write a book but during the week you make changes to several pages of your book on a copy called the Preview shard. When you patch you only need to replace the changes and nothing else. It is much quicker, than making another copy of the book called a back up.
Most banks do nightly maintenance back ups at 2 am or 3 am, that process can last several hours. Online video games cannot afford this luxury as they do not close for any reason other than maintenance.
Now back to the analogy about the book. The models and textures for this game are on your PC in libraries. They are not called by a name like dinosaur or waitress. They are numbered in the order they appear in the library, such as 2093874. What happens sometimes, the numbers get mixed up and we get dinosaurs instead of waitress. Like replacing the wrong page numbers in the patch of your book. Oops!
The Cryptic database tells the clients on our computers to display model number 2093784 instead of 2093874 and we get Klingons instead of orcs. Actually I think that would be a stretch, because the STO libraries exist in a separate folder. Basically that analogy, has gone a long way to help me comprehend how a database sends the wrong reference number and my client displays the wrong models.
> @sandukutupu said: > A very good friend of mine works for a database engineering team in Ypsilanti, Michigan. They do work on banks, department stores, etc. dealing with real world money. He explained to me with the follow analogy about the difference between maintenance back ups and patches. It is my sincere hope this helps others understand the process better as well. > > Imagine you write a book but during the week you make changes to several pages of your book on a copy called the Preview shard. When you patch you only need to replace the changes and nothing else. It is much quicker, than making another copy of the book called a back up. > > Most banks do nightly maintenance back ups at 2 am or 3 am, that process can last several hours. Online video games cannot afford this luxury as they do not close for any reason other than maintenance. > <snip> If this was going to cripple ATM/Banks nationwide I heartily agree but it is not. Given that ARC developers don't even know when Thanksgiving is I'm guessing your 2am is their 2pm and they are deploying on their schedule? Also happy Diwali
0
greywyndMember, NW M9 PlaytestPosts: 7,150Arc User
If this was going to cripple ATM/Banks nationwide I heartily agree but it is not. Given that ARC developers don't even know when Thanksgiving is I'm guessing your 2am is their 2pm and they are deploying on their schedule? Also happy Diwali
Of course they do it on their schedule. Why should they disrupt their work schedule because it is inconvenient somewhere else in the world?
And no matter when they do it, it will be inconvenient somewhere in the world.
I'm not looking for forgiveness, and I'm way past asking permission. Earth just lost her best defender, so we're here to fight. And if you want to stand in our way, we'll fight you too.
> @greywynd said: > (Quote) > Of course they do it on their schedule. Why should they disrupt their work schedule because it is inconvenient somewhere else in the world? > > And no matter when they do it, it will be inconvenient somewhere in the world.
As a professional developer I am going to say no. We do it when it is convenient for our target audience not when it is personally convenient for us.
इंडोनेशिया में अभी 4 लोगों के माफी मांगने के साथ ही आपको लक्ष्मी का आशीर्वाद भी
3
greywyndMember, NW M9 PlaytestPosts: 7,150Arc User
Which part of the world is their target audience when their players are all over the world?
I'm not looking for forgiveness, and I'm way past asking permission. Earth just lost her best defender, so we're here to fight. And if you want to stand in our way, we'll fight you too.
> Of course they do it on their schedule. Why should they disrupt their work schedule because it is inconvenient somewhere else in the world?
>
> And no matter when they do it, it will be inconvenient somewhere in the world.
As a professional developer I am going to say no. We do it when it is convenient for our target audience not when it is personally convenient for us.
इंडोनेशिया में अभी 4 लोगों के माफी मांगने के साथ ही आपको लक्ष्मी का आशीर्वाद भी
Just how do you propose they do that? If you are talking about the North American audience then they cannot know or guess when people will be online playing the game. The continent is 4 time zones wide. Since I have been online at 2 am or 4am in the morning there is always someone playing this game. As a result, someone has to not play for 2 hours so they can patch the game. A shopping center, a bank, a mall, are businesses that can and do shut down nightly, they can spend all night doing back ups. Online games cannot simply hang out a closed for the weekend sign or gone fishing.
0
greywyndMember, NW M9 PlaytestPosts: 7,150Arc User
Well, they can, but then we'd be wondering what broke that required that much downtime.
I'm not looking for forgiveness, and I'm way past asking permission. Earth just lost her best defender, so we're here to fight. And if you want to stand in our way, we'll fight you too.
2
kreatyveMember, Neverwinter Moderator, NW M9 PlaytestPosts: 10,545Community Moderator
Due to the fact that it's always going to be inconvenient for someone in the world, they do it at a time that's best convenient for them, I'm sure. They don't want to do it in the middle of the night because sleepy devs make more mistakes.
My opinions are my own. I do not work for PWE or Cryptic. - Forum Rules - Protector's Enclave Discord - I play on Xbox Any of my comments not posted in orange are based on my own personal opinion and not official. Any messages written in orange are official moderation messages. Signature images are now fixed!
> @sandukutupu said: > (Quote) > Just how do you propose they do that? If you are talking about the North American audience then they cannot know or guess when people will be online playing the game. The continent is 4 time zones wide. Since I have been online at 2 am or 4am in the morning there is always someone playing this game. As a result, someone has to not play for 2 hours so they can patch the game. A shopping center, a bank, a mall, are businesses that can and do shut down nightly, they can spend all night doing back ups. Online games cannot simply hang out a closed for the weekend sign or gone fishing.
The best way would be a "hot spare" of the production environment. You can then deploy updates in (for the consumer) minutes not hours. It may still take 2 hours (but it won't. a 2 hour deploy is a failed deploy and rollback) but they won't ever see it.
If you don't have the budget for that it is trivial for developers to see how many people are on at what hour and pick a time of low activity, just like it is trivial to patch things like graphical assets ahead of time.
Patching during the closing hours of an event is always the wrong answer unless your server farm is literally on fire (which may indeed require the full 2 hours)
LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?
> @sandukutupu said: > LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?
Good to know. How long have you worked there?
0
micky1p00Member, NW M9 PlaytestPosts: 3,594Arc User
LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?
You "LOL" at them, when they actually understand more, and more correct than you.
Every number of instances are spun on a virtual or physical server, with their own IP. A coordinator server redirects a player on zone load.
The authentication server is separate, the AH is a separate server, the patchserver, you guessed it right, a separate server, inventory database, with data synced to instance server on client load, separate database server. And so on, not counting the preview servers, and all the internal networking. No matter how many of those virtualized, share same VM or hardware as a software, and how many, exactly, multiple hardware servers it all takes, it is a cluster, or a farm if perhaps small one compared to some other companies.
Post edited by micky1p00 on
3
putzboy78Member, NW M9 PlaytestPosts: 1,950Arc User
Due to the fact that it's always going to be inconvenient for someone in the world, they do it at a time that's best convenient for them, I'm sure. They don't want to do it in the middle of the night because sleepy devs make more mistakes.
At current activity levels, there are probably more support people logged into the server than gamers. So yeah, doing it when its most convenient to cryptic totally makes sense
LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?
You "LOL" at them, when they actually understand more, and more correct than you.
Every number of instances are spun on a virtual or physical server, with their own IP. A coordinator server redirects a player on zone load.
The authentication server is separate, the AH is a separate server, the patchserver, you guessed it right, a separate server, inventory database, with data synced to instance server on client load, separate database server. And so on, not counting the preview servers, and all the internal networking. No matter how many of those virtualized, share same VM or hardware as a software, and how many, exactly, multiple hardware servers it all takes, it is a cluster, or a farm if perhaps small one compared to some other companies.
I laughed at you too Micky, just so you don't feel left out. Virtual or otherwise, this is not a server farm. Frankly I am amazed at what Cryptic can accomplish with their limitations. I am not poking fun at anyone's knowledge or ignorance. I already stated the facts above, I needed an expert to explain things to me in the most simple terms to understand. I assume you believe they should be "hot swapping" the server, so there is no down time as well? My IT personnel express there is a considerable risk to the hardware. Now the question is, would you be willing to lose everything on your account because it wasn't 100% sufficient? I can wait 2 or even 6 hours a week for the server maintenance. I don't want them to rush and break all the eggs.
0
putzboy78Member, NW M9 PlaytestPosts: 1,950Arc User
server is an over used term. often times when people say "server" they are talking about services and instances. Effectively what you have is a multitiered architectured system with varies services running on it. As an example you will have network services which include authentication, you will have database to store the information, you will have application to process information, depending on how its constructed you can have a separate database for mail than inventory. This can all sit on one physical and/or virtual server or the services can be divided up to balance loads... ultimately none of this matters because the ones who knows if and how the systems are divided up are not the people playing this game... its cryptic.
And none of that is relevant. Cryptic plans on a 2 hour weekly update schedule which includes server maintenance. Sometimes its faster than others because the maintenance has less requirements. Sometimes they schedule additional time if they know they are going to need it.
Is it possible to pre load an application update on a separate instance and move the pointers to the new instance to reduce update time, absolutely. And there is no way for anyone playing the game to know if they are or not doing that.
And @sandukutupu it could be possible (depending on the multitiered design) to preload an application change without impacting the data assuming the database is separate. The point being is we don't know how cryptic divides its architecture and therefore can not assume that preloading is possible without sacrificing data integrity. It's a fair question to ask, but I'm 100% sure cryptic will answer with silence and think its nunya
0
micky1p00Member, NW M9 PlaytestPosts: 3,594Arc User
LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?
You "LOL" at them, when they actually understand more, and more correct than you.
Every number of instances are spun on a virtual or physical server, with their own IP. A coordinator server redirects a player on zone load.
The authentication server is separate, the AH is a separate server, the patchserver, you guessed it right, a separate server, inventory database, with data synced to instance server on client load, separate database server. And so on, not counting the preview servers, and all the internal networking. No matter how many of those virtualized, share same VM or hardware as a software, and how many, exactly, multiple hardware servers it all takes, it is a cluster, or a farm if perhaps small one compared to some other companies.
I laughed at you too Micky, just so you don't feel left out. Virtual or otherwise, this is not a server farm. Frankly I am amazed at what Cryptic can accomplish with their limitations. I am not poking fun at anyone's knowledge or ignorance. I already stated the facts above, I needed an expert to explain things to me in the most simple terms to understand. I assume you believe they should be "hot swapping" the server, so there is no down time as well? My IT personnel express there is a considerable risk to the hardware. Now the question is, would you be willing to lose everything on your account because it wasn't 100% sufficient? I can wait 2 or even 6 hours a week for the server maintenance. I don't want them to rush and break all the eggs.</p>
I just can't even.....
Hot swapping the server in this case has nothing to do with hardware. You have no clue what you talk about, miscommunicate it to whomever understands and miscomunicate it back here. Which is not the issue on its own, but then talk about ignorance...
The only single action that done to "hot swap" the game version is to change the ip number of the server coordinator and authentication server the client has, from old version set A to new version set B. And request the client to re-log to the new set.
And no I don't care either way, live back up requires capability the custom made Crpytic database system may not have, so it needs to be done on a higher level, OS or VM, which will probably impact they ability to look up old chars and roll back. offline backup just saves them the complexity, and costs of maintaining the system, as offline backups / cloning are much simpler and easier to do.
But that doesn't negate the fact that you have no clue what you write about, while talk about other people ignorance.
> And @sandukutupu it could be possible (depending on the multitiered design) to preload an application change without impacting the data assuming the database is separate. The point being is we don't know how cryptic divides its architecture and therefore can not assume that preloading is possible without sacrificing data integrity. It's a fair question to ask, but I'm 100% sure cryptic will answer with silence and think its nunya
Thank you voice of sanity. Your user data is in a database. If a deploy fails the worst case scenario is that it gets restored from database backups. It isn't all getting shredded because a developer is sleepy.
I suspect it's a conventional database but I guess Cryptic could have re-implemented this using Redis or MongoDB or some other in-memory system. In theory it could even be a flat file sitting on your PC but since it's no longer 1994 and those are trivial to hack...probably not.
0
kreatyveMember, Neverwinter Moderator, NW M9 PlaytestPosts: 10,545Community Moderator
And I do believe that at this point, this thread has ran its course.
My opinions are my own. I do not work for PWE or Cryptic. - Forum Rules - Protector's Enclave Discord - I play on Xbox Any of my comments not posted in orange are based on my own personal opinion and not official. Any messages written in orange are official moderation messages. Signature images are now fixed!
Comments
Imagine you write a book but during the week you make changes to several pages of your book on a copy called the Preview shard. When you patch you only need to replace the changes and nothing else. It is much quicker, than making another copy of the book called a back up.
Most banks do nightly maintenance back ups at 2 am or 3 am, that process can last several hours. Online video games cannot afford this luxury as they do not close for any reason other than maintenance.
Now back to the analogy about the book. The models and textures for this game are on your PC in libraries. They are not called by a name like dinosaur or waitress. They are numbered in the order they appear in the library, such as 2093874. What happens sometimes, the numbers get mixed up and we get dinosaurs instead of waitress. Like replacing the wrong page numbers in the patch of your book. Oops!
The Cryptic database tells the clients on our computers to display model number 2093784 instead of 2093874 and we get Klingons instead of orcs. Actually I think that would be a stretch, because the STO libraries exist in a separate folder. Basically that analogy, has gone a long way to help me comprehend how a database sends the wrong reference number and my client displays the wrong models.
> A very good friend of mine works for a database engineering team in Ypsilanti, Michigan. They do work on banks, department stores, etc. dealing with real world money. He explained to me with the follow analogy about the difference between maintenance back ups and patches. It is my sincere hope this helps others understand the process better as well.
>
> Imagine you write a book but during the week you make changes to several pages of your book on a copy called the Preview shard. When you patch you only need to replace the changes and nothing else. It is much quicker, than making another copy of the book called a back up.
>
> Most banks do nightly maintenance back ups at 2 am or 3 am, that process can last several hours. Online video games cannot afford this luxury as they do not close for any reason other than maintenance.
>
<snip>
If this was going to cripple ATM/Banks nationwide I heartily agree but it is not. Given that ARC developers don't even know when Thanksgiving is I'm guessing your 2am is their 2pm and they are deploying on their schedule? Also happy Diwali
And no matter when they do it, it will be inconvenient somewhere in the world.
> (Quote)
> Of course they do it on their schedule. Why should they disrupt their work schedule because it is inconvenient somewhere else in the world?
>
> And no matter when they do it, it will be inconvenient somewhere in the world.
As a professional developer I am going to say no. We do it when it is convenient for our target audience not when it is personally convenient for us.
इंडोनेशिया में अभी 4 लोगों के माफी मांगने के साथ ही आपको लक्ष्मी का आशीर्वाद भी
Any of my comments not posted in orange are based on my own personal opinion and not official.
Any messages written in orange are official moderation messages. Signature images are now fixed!
> (Quote)
> Just how do you propose they do that? If you are talking about the North American audience then they cannot know or guess when people will be online playing the game. The continent is 4 time zones wide. Since I have been online at 2 am or 4am in the morning there is always someone playing this game. As a result, someone has to not play for 2 hours so they can patch the game. A shopping center, a bank, a mall, are businesses that can and do shut down nightly, they can spend all night doing back ups. Online games cannot simply hang out a closed for the weekend sign or gone fishing.
The best way would be a "hot spare" of the production environment. You can then deploy updates in (for the consumer) minutes not hours. It may still take 2 hours (but it won't. a 2 hour deploy is a failed deploy and rollback) but they won't ever see it.
If you don't have the budget for that it is trivial for developers to see how many people are on at what hour and pick a time of low activity, just like it is trivial to patch things like graphical assets ahead of time.
Patching during the closing hours of an event is always the wrong answer unless your server farm is literally on fire (which may indeed require the full 2 hours)
> LOL did you just say "server farm"?? It is just one server at Cryptic for all 3 games (soon to be 4), did you think this is run by Dice or some other game company?
Good to know. How long have you worked there?
Every number of instances are spun on a virtual or physical server, with their own IP. A coordinator server redirects a player on zone load.
The authentication server is separate, the AH is a separate server, the patchserver, you guessed it right, a separate server, inventory database, with data synced to instance server on client load, separate database server.
And so on, not counting the preview servers, and all the internal networking. No matter how many of those virtualized, share same VM or hardware as a software, and how many, exactly, multiple hardware servers it all takes, it is a cluster, or a farm if perhaps small one compared to some other companies.
And none of that is relevant. Cryptic plans on a 2 hour weekly update schedule which includes server maintenance. Sometimes its faster than others because the maintenance has less requirements. Sometimes they schedule additional time if they know they are going to need it.
Is it possible to pre load an application update on a separate instance and move the pointers to the new instance to reduce update time, absolutely. And there is no way for anyone playing the game to know if they are or not doing that.
And @sandukutupu it could be possible (depending on the multitiered design) to preload an application change without impacting the data assuming the database is separate. The point being is we don't know how cryptic divides its architecture and therefore can not assume that preloading is possible without sacrificing data integrity. It's a fair question to ask, but I'm 100% sure cryptic will answer with silence and think its nunya
Hot swapping the server in this case has nothing to do with hardware. You have no clue what you talk about, miscommunicate it to whomever understands and miscomunicate it back here. Which is not the issue on its own, but then talk about ignorance...
The only single action that done to "hot swap" the game version is to change the ip number of the server coordinator and authentication server the client has, from old version set A to new version set B. And request the client to re-log to the new set.
And no I don't care either way, live back up requires capability the custom made Crpytic database system may not have, so it needs to be done on a higher level, OS or VM, which will probably impact they ability to look up old chars and roll back.
offline backup just saves them the complexity, and costs of maintaining the system, as offline backups / cloning are much simpler and easier to do.
But that doesn't negate the fact that you have no clue what you write about, while talk about other people ignorance.
> And @sandukutupu it could be possible (depending on the multitiered design) to preload an application change without impacting the data assuming the database is separate. The point being is we don't know how cryptic divides its architecture and therefore can not assume that preloading is possible without sacrificing data integrity. It's a fair question to ask, but I'm 100% sure cryptic will answer with silence and think its nunya
Thank you voice of sanity. Your user data is in a database. If a deploy fails the worst case scenario is that it gets restored from database backups. It isn't all getting shredded because a developer is sleepy.
I suspect it's a conventional database but I guess Cryptic could have re-implemented this using Redis or MongoDB or some other in-memory system. In theory it could even be a flat file sitting on your PC but since it's no longer 1994 and those are trivial to hack...probably not.
Any of my comments not posted in orange are based on my own personal opinion and not official.
Any messages written in orange are official moderation messages. Signature images are now fixed!