To those that have lost levels out there, be patient. There are probably hundreds is not more that they have to identify that this actually happened to in the first place. They can't simply magically parse the data logs and pull the all out at once. And I am sure they are looking to make sure that it is only people that were demoted to RA39 as well. I suggest if you really want to play the game until they sort it out, that you start a new toon or do something else in the meantime. It really is not worth getting upset over.
Also, wish to thank Coderanger for the detailed report to the playerbase.
45K a year? LOL In my business a good experienced DBA can cost over 100K easy. Guess you don't have much experience as a DBA. Any experience. Heh, I think our starting salary for a junior programmer right out of school is better then 45K. Of course we expect competance.
Yes they messed up. But this isn't some little joke desktop app who's idea of an advanced database is MySQL. It most likely has dozens of server machines to handle thousands of clients.
Someone compared them to SoE. That's an untterly unfair comparison. SoE would NEVER had told the players exactly what the problem was and ADMITTED they screwed up. Never. They'd have just piled on the BS.
These downtimes are unfortunate but unavoidable. Yes it's particularly bad that it happened on a weekend, but that's the kind of thing which happens when they're constantly trying to improve the game. Improving the game means making changes. Anyone with any experience in IT should understand that with every change comes a risk. Sometimes these changes backfire. No risk == no change. Ever. (actually, even with no change there's some risk, just less risk).
If they made no effort to improve the game this wouldn't have happened. But the game would never get any better. I for one would prefer they take some risk and work on improving the game, and when the risk pans out and it goes into meltdown I'll deal with it. Yes they should take precautions. Yes they should learn from this so this particular problem never happens again. But they have to take some risk. I hope they aren't so burned by this they avoid changes and the risk they entail.
Thanks for the update Coderanger. Straightforward and honest, just what I've come to appreciate from you (and most of the other Devs in the last few weeks).
The explanation is all I felt I was owed for yesterday's downtime. I ultimately enjoyed having an excuse to go out and get some fresh air. Sorry you guys didn't get to do the same.
To those that have lost levels out there, be patient. There are probably hundreds is not more that they have to identify that this actually happened to in the first place. They can't simply magically parse the data logs and pull the all out at once. And I am sure they are looking to make sure that it is only people that were demoted to RA39 as well. I suggest if you really want to play the game until they sort it out, that you start a new toon or do something else in the meantime. It really is not worth getting upset over.
Also, wish to thank Coderanger for the detailed report to the playerbase.
I'm captain -29, so I don't know what that has to do with RA39, that makes me wonder if you have a firm grasp of the problem.
It really is worth getting upset over, some of us don't like playing more than one character.
heh, I got hit with the RA -39 bug yesterday. I was so bored I managed to level up a bit. Now I am RA -34 and its possible to do Tier 1 PVP with my Tier 5 ship lol
I'm captain -29, so I don't know what that has to do with RA39, that makes me wonder if you have a firm grasp of the problem.
It really is worth getting upset over, some of us don't like playing more than one character.
Well bubba, I am RA45. The "grasp of the problem" is that some individuals (unknown how many) that played certain PvP maps were demoted. It was unclear as to actually who was effected so it will take some time to go through the data logs to sort out. It does no good to just set there with a character you can't play. So, my suggestion, do something else. OR, like so many have done, they can come to the forums to moan and groan which is not going to make it happen any faster.
Now, how is that for having a "grasp of the problem"?
My BG -39 has been -39 for 27 Hours now (which means I'm going on missing dailies for 2 days). Any ETA on getting these guys fixed? More importantly, will we need to do anything?
Back in the early SWG days, when this happened; people at to line up at the Star Port to be individually sorted out, will this be like that?
Are you talking about hand jamming in fixes for hundreds of people? Or will this be automated? Can we get an update on when we can expect to see a reversion if it is automated?
My BG -39 has been -39 for 27 Hours now (which means I'm going on missing dailies for 2 days). Any ETA on getting these guys fixed? More importantly, will we need to do anything?
Back in the early SWG days, when this happened; people at to line up at the Star Port to be individually sorted out, will this be like that?
Are you talking about hand jamming in fixes for hundreds of people? Or will this be automated? Can we get an update on when we can expect to see a reversion if it is automated?
Thanks
And are they 100% sure it can be done without a rollback now? So far we get words that still leave it a possibility like "we plan" and such.
Since I complained so loudly yesterday about being mushroomed I have to say thank you coderanger for the honest explanation...I do appreciate the honesty.
human problems are easy to understand and therefore easier to forgive and forget.. good work STO
thanks for the info, glad its up and running again.
realise you guys have issues that can be avoided but werent, and would be nice to see future avoidable issues avoided , but still happy to see the server running again
I'm captain -29, so I don't know what that has to do with RA39, that makes me wonder if you have a firm grasp of the problem.
It really is worth getting upset over, some of us don't like playing more than one character.
Agreed. And I've been patient, but a broken character for more than 24 hours is getting a bit ridiculous. And it would be great if those unaffected who are getting to play stopped telling those of us who can't that we're "just not being patient enough".
Agreed. And I've been patient, but a broken character for more than 24 hours is getting a bit ridiculous. And it would be great if those unaffected who are getting to play stopped telling those of us who can't that we're "just not being patient enough".
:mad:Agreed, they wouldnt be saying that if it had happene to them:mad:
I'm captain -29, so I don't know what that has to do with RA39, that makes me wonder if you have a firm grasp of the problem.
It really is worth getting upset over, some of us don't like playing more than one character.
Joebane, you are on the list of affected players that I have pulled from the server logs, so you will be fixed. The mention of "RA-39" was just an example. We do understand that people other than Admirals were affected by the issue and are going to fix everyone..
Considering my toon has been - 39 for almost 24 hours now, you comment is irrelevant.
Why should we have to accept incompetence, its not like this is a new game engine for CP lol
Like most other I have been asleep, went to the pub and did some shopping this morning.
Guess what, still not fixed ROFl
you shouldn't - why not go play World of Warcraft (Which, BTW at 60 days out STILL had way more server issues, and character batabase issues then STO has had to date). BTW WoW recently did a TESTED patch that had their servers lagged for nearly two weeks before the Blizzard Dev team got things sorted - and that's on an MMO that's been online 5+ YEARS.
Oh, and BTW - if you ever DO have an account issue with your WoW character (ie it' TRIBBLE, or something corrupted) - GOOD LUCK in getting Blizzard to fix it in 24 or so hours. 72 hours to 1 week or more has been my friends experience in that regard - and agin with WoW were talking the 800lbs MMO goirilla with a nearly unlimited development budget, etc.
The fact is - Cryptic DID TRIBBLE up - someone fast tracked a fix, skipped some in place review procedures, and as a result, we had the PvP issues. the database issue was something completely unexpected that took the Net Ops folks a while to understand (and as someone who works in IT and has done an 'all weekend' mission critical server restore/fix on occassion); and I'm amazed (but happy) to see they got it figured out and resolved; and futher that they learned something new that should improve DB performance in the long run from this point forward. Given the size of the codebase they're working with, not to mentuion the sheer amount of data involved, they DID resolve it in a relativel short timeframe; and on that level the Net Ops and Dev teams should be commended. (But I also hope that you consider giving a couple extra days subscription time to the month to month crowd - I'm an LTS so LTS+2 is meaningless to me - but hey, I already have some nice perks ).
Also, the fact that they:
1) kept the entire playerbase in the loop duuring the server repatch
and
2) provided an honest, up front/no spin post mortem publically.
Really scores a lot of points in my book, and I DO hope they can keep mthe Dev and Net Ops folks STO currently has (even the ones who made the mistake - which I hope they learned from0. If there's one thing mosyt server admins and support people learn over time is that you NEVER really know what a small/simple change will do until you apply it; and anytime you fasttrack and skip testing in place testing procedures (which CAN be needed if you're combating a back hack/virus attack situation - which was not the case here); you never know what will happen until you see it's behavior in production.
So, while I agree, it never should have happened in the first place, I will also applaud the response we got to it from the STO team across ALL LEVELS. I have never seen this level of response from SOE, Mythic, Funcom, CCP, or any other major MMO developer; and it was both interesting and refreshing to see a 'no spin' response.
I will also say that (to keep the Cryptic Team's sanity on both games you currently run); you might want to consider shifting you Dev schedule, so that new 'update' patches comes of Tuesdays instead of Thursdays. That way, you have move time to avoid weekend issues on the Holodeck server, and have less weekend emergency OT on the books.
Joebane, you are on the list of affected players that I have pulled from the server logs, so you will be fixed. The mention of "RA-39" was just an example. We do understand that people other than Admirals were affected by the issue and are going to fix everyone..
I love this idea of 'just roll it back', like the database is analogous to your desktop, with Windows holding your hand and making constant backup points for when you nail it downloading virused TRIBBLE. There isn't anyone else out there to care about who would lose the result of their day playing even if there was some magic fairy dust capable of restoring the server, like some sort of real world quicksave.
This post if for Coderangers OP, as I have not read anyone elses.
I think its great that you guys in-detail, told us what the situation is, for that I thank you. I hope you all can keep a open dialog with the community when able, I believe this is how a true community is suppose to work.
It did suck not being able to play much yesterday/last night, but I understand things like this happen. I appreciate you guys getting the situation under control as soon as you could, which was late night/early morning.
I love this idea of 'just roll it back', like the database is analogous to your desktop, with Windows holding your hand and making constant backup points for when you nail it downloading virused TRIBBLE. There isn't anyone else out there to care about who would lose the result of their day playing even if there was some magic fairy dust capable of restoring the server, like some sort of real world quicksave.
I take it you didn't get hit with the -39 bug. I think it would suck for people to lose a day of play but right now anyone sitting at -39 is close to a full character wipe. In some ways worse. I'm all for a rollback if it's the only way to fix this.
Comments
Yes, both skill points and bridge officer skill points will be restored.
Also, wish to thank Coderanger for the detailed report to the playerbase.
Yes they messed up. But this isn't some little joke desktop app who's idea of an advanced database is MySQL. It most likely has dozens of server machines to handle thousands of clients.
Someone compared them to SoE. That's an untterly unfair comparison. SoE would NEVER had told the players exactly what the problem was and ADMITTED they screwed up. Never. They'd have just piled on the BS.
These downtimes are unfortunate but unavoidable. Yes it's particularly bad that it happened on a weekend, but that's the kind of thing which happens when they're constantly trying to improve the game. Improving the game means making changes. Anyone with any experience in IT should understand that with every change comes a risk. Sometimes these changes backfire. No risk == no change. Ever. (actually, even with no change there's some risk, just less risk).
If they made no effort to improve the game this wouldn't have happened. But the game would never get any better. I for one would prefer they take some risk and work on improving the game, and when the risk pans out and it goes into meltdown I'll deal with it. Yes they should take precautions. Yes they should learn from this so this particular problem never happens again. But they have to take some risk. I hope they aren't so burned by this they avoid changes and the risk they entail.
Crashes to desktop showing my Klingon on the character select each time.
Even after your so called fixes last night.:mad:
The explanation is all I felt I was owed for yesterday's downtime. I ultimately enjoyed having an excuse to go out and get some fresh air. Sorry you guys didn't get to do the same.
I'm captain -29, so I don't know what that has to do with RA39, that makes me wonder if you have a firm grasp of the problem.
It really is worth getting upset over, some of us don't like playing more than one character.
Thanks for communicating with us. Over communication can solve a LOT of problems if done correctly.
Well bubba, I am RA45. The "grasp of the problem" is that some individuals (unknown how many) that played certain PvP maps were demoted. It was unclear as to actually who was effected so it will take some time to go through the data logs to sort out. It does no good to just set there with a character you can't play. So, my suggestion, do something else. OR, like so many have done, they can come to the forums to moan and groan which is not going to make it happen any faster.
Now, how is that for having a "grasp of the problem"?
Back in the early SWG days, when this happened; people at to line up at the Star Port to be individually sorted out, will this be like that?
Are you talking about hand jamming in fixes for hundreds of people? Or will this be automated? Can we get an update on when we can expect to see a reversion if it is automated?
Thanks
And are they 100% sure it can be done without a rollback now? So far we get words that still leave it a possibility like "we plan" and such.
human problems are easy to understand and therefore easier to forgive and forget.. good work STO
thanks for the info, glad its up and running again.
realise you guys have issues that can be avoided but werent, and would be nice to see future avoidable issues avoided
Agreed. And I've been patient, but a broken character for more than 24 hours is getting a bit ridiculous. And it would be great if those unaffected who are getting to play stopped telling those of us who can't that we're "just not being patient enough".
:mad:Agreed, they wouldnt be saying that if it had happene to them:mad:
no dilies
no infected
no cure
no pvp
we get aggro = we are dead (on the ground)
we get aggro = we are almost sure dead (in space)
my record was being hit by the explosion of a cube getting 183000 dmg
183000 dmg
http://bit.ly/ra-39
Given the choice I'd toss you to the back of the line to get fixed too.
Joebane, you are on the list of affected players that I have pulled from the server logs, so you will be fixed. The mention of "RA-39" was just an example. We do understand that people other than Admirals were affected by the issue and are going to fix everyone..
you shouldn't - why not go play World of Warcraft (Which, BTW at 60 days out STILL had way more server issues, and character batabase issues then STO has had to date). BTW WoW recently did a TESTED patch that had their servers lagged for nearly two weeks before the Blizzard Dev team got things sorted - and that's on an MMO that's been online 5+ YEARS.
Oh, and BTW - if you ever DO have an account issue with your WoW character (ie it' TRIBBLE, or something corrupted) - GOOD LUCK in getting Blizzard to fix it in 24 or so hours. 72 hours to 1 week or more has been my friends experience in that regard - and agin with WoW were talking the 800lbs MMO goirilla with a nearly unlimited development budget, etc.
The fact is - Cryptic DID TRIBBLE up - someone fast tracked a fix, skipped some in place review procedures, and as a result, we had the PvP issues. the database issue was something completely unexpected that took the Net Ops folks a while to understand (and as someone who works in IT and has done an 'all weekend' mission critical server restore/fix on occassion); and I'm amazed (but happy) to see they got it figured out and resolved; and futher that they learned something new that should improve DB performance in the long run from this point forward. Given the size of the codebase they're working with, not to mentuion the sheer amount of data involved, they DID resolve it in a relativel short timeframe; and on that level the Net Ops and Dev teams should be commended. (But I also hope that you consider giving a couple extra days subscription time to the month to month crowd - I'm an LTS so LTS+2 is meaningless to me - but hey, I already have some nice perks
Also, the fact that they:
1) kept the entire playerbase in the loop duuring the server repatch
and
2) provided an honest, up front/no spin post mortem publically.
Really scores a lot of points in my book, and I DO hope they can keep mthe Dev and Net Ops folks STO currently has (even the ones who made the mistake - which I hope they learned from0. If there's one thing mosyt server admins and support people learn over time is that you NEVER really know what a small/simple change will do until you apply it; and anytime you fasttrack and skip testing in place testing procedures (which CAN be needed if you're combating a back hack/virus attack situation - which was not the case here); you never know what will happen until you see it's behavior in production.
So, while I agree, it never should have happened in the first place, I will also applaud the response we got to it from the STO team across ALL LEVELS. I have never seen this level of response from SOE, Mythic, Funcom, CCP, or any other major MMO developer; and it was both interesting and refreshing to see a 'no spin' response.
I will also say that (to keep the Cryptic Team's sanity on both games you currently run); you might want to consider shifting you Dev schedule, so that new 'update' patches comes of Tuesdays instead of Thursdays. That way, you have move time to avoid weekend issues on the Holodeck server, and have less weekend emergency OT on the books.
When, Guys, WHEN?
I think its great that you guys in-detail, told us what the situation is, for that I thank you. I hope you all can keep a open dialog with the community when able, I believe this is how a true community is suppose to work.
It did suck not being able to play much yesterday/last night, but I understand things like this happen. I appreciate you guys getting the situation under control as soon as you could, which was late night/early morning.
I take it you didn't get hit with the -39 bug. I think it would suck for people to lose a day of play but right now anyone sitting at -39 is close to a full character wipe. In some ways worse. I'm all for a rollback if it's the only way to fix this.