test content
What is the Arc Client?
Install Arc

Unexpected Downtime (March 11, 2012)

SystemSystem Member, NoReporting Posts: 178,019 Arc User
edited March 2012 in Galactic News Network [PC]
The shard is currently offline due to technical issues.

UPDATE: Our current ETA to have the shard back up is about 2 hours ~ 9.50 PM PST.

=====

EDIT from PWE_BranFlakes: Incase you missed this the other day from STO Lead Programmer Rehpic:
Rehpic wrote:
The free to play launch of STO was supported by a substantial upgrade to the hardware used to run the live game, and that hardware is more than adequate to support our current user load, which is substantially higher than before f2p launch.

We have had 6 database crashes, which have brought the game down, over the last month. In 3 of those cases we deployed an already scheduled patch and were able to cancel a scheduled downtime.

The crashes are occurring because of a software bug which is corrupting a data structure used to keep track of large chunks of memory that are being kept around for future use (the term of art among programmers is "Heap Corruption"). These types of bugs are often the most subtle and difficult to track down in large software systems. The crash only occurs after 3-4 days of running under heavy load, and despite our best efforts we have been unable to reproduce it in the lab.

We have had a dedicated team working on this problem since the first crash, including most of our infrastructure programming team, myself, Stephen D. (back in the CTO chair!) and other programmers we pull in as needed. We have been trying to reproduce the crash, studying all code changes from the last couple of months, spending days examining crash dumps, and adding new code to help diagnose the problem in every patch.

With every crash we gather more information and get closer to a solution. The reason for the extended downtime today was that we decided to debug the live database when it crashed, rather than examining a crash dump after the fact. This allowed us to get more information and test some theories in ways we could not do when examining a dump after the fact, so it contributed to getting us to a solution faster.

I just wanted to let you all know that we are working hard on the problem (it is almost midnight here!) and are making progress towards a solution. We have the resources we need, namely the people who know the code the best.

Cheers,

Brandon =/\=

=====
Post edited by Unknown User on
«13456745

Comments

  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    This post has been edited to remove content which violates the Perfect World Entertainment Community Rules and Policies. ~Alecto
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    The server is done for me. Not sure what to do now..

    * Goes off to troll the ME3 forums *
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    At least I finished that mission with the Defari.
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    my ex wife was more reliable than these servers...:(
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    KABOOM! *sound of debris falling from the heavens*
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Are they trying really hard to kill this game? I have to ask because it looks that way....
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    It wasn't me this time, honest!
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Again... >_<
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Put in the fix to the supposed Jem'Hadar shields... So the whiners can stop crying.
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Sure does crash a lot...
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    One crash a day keeps the players away. Or something.
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Getting ****ing old PWE, fix your issues. Sad that I could call your so called Unexpected Downtime before you even had a chance to post in the forums.
    Get your act together before you start losing all of your paying customers.
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    They forget to set clocks ahead 1 hour?
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Gold members need some kind of reimbursement. This is getting ridiculousness. You maid enough money off the lock boxes to pay people and buy hardware to keep the game up. You're just going to drive people away from the game if you keep this up.

    Gold members should get a month rebate or 800 stipend for 1 month.
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Free2crash and burn baby!
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Cryptic, a lot of us are professional nerds and probably make more money than you do for doing the same kind of coding. What in the Sam Hill is the issue? Is it server capacity or are you oblivious to the idea of alpha and beta testing? Why do you keep adding new content when the old content is crashing the server multiple times per week? I frickin love the game and that is saying a lot because I absolutely hate RPG's but Star Trek is frickin' awesome and I like blowing stuff up with my assault cruiser. I'm sure there's at least a single digit percentage of the player base that would be competent and happy to look at your code and see what is so unstable about it. I desperately want this game to be awesome and successful.
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    LMFAO. As soon as I saw retrieving list of char and then disconnect I knew it. MAJOR FREAKING FAIL ACROSS THE BOARD.
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    You have got yo be kidding me....... again
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    This has been happening with alarming frequency of late. I hope there are steps being taken on the back end to remedy the underlying cause.
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Seriously? Come on. This is really getting old. Can we kick the F2P accounts till we get this fixed?
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    It amuses me that when people complained about the almost daily maintenance alittle while ago the butt kissers told us that we should be grateful that we only have to put up with schedlued maintenance because other games crash all the time! ;)
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Hey, I like micro machines... I had all the star trek ones when I was a kid.....
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Then: Pay to Play
    Now: Pray to Play
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Maybe Cryptic hasnt paid their monthly dues to PW yet... lol
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    wtf not again -.- is that now every day lol
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Incase you missed this the other day from STO Lead Programmer Rehpic:
    Rehpic wrote:
    The free to play launch of STO was supported by a substantial upgrade to the hardware used to run the live game, and that hardware is more than adequate to support our current user load, which is substantially higher than before f2p launch.

    We have had 6 database crashes, which have brought the game down, over the last month. In 3 of those cases we deployed an already scheduled patch and were able to cancel a scheduled downtime.

    The crashes are occurring because of a software bug which is corrupting a data structure used to keep track of large chunks of memory that are being kept around for future use (the term of art among programmers is "Heap Corruption"). These types of bugs are often the most subtle and difficult to track down in large software systems. The crash only occurs after 3-4 days of running under heavy load, and despite our best efforts we have been unable to reproduce it in the lab.

    We have had a dedicated team working on this problem since the first crash, including most of our infrastructure programming team, myself, Stephen D. (back in the CTO chair!) and other programmers we pull in as needed. We have been trying to reproduce the crash, studying all code changes from the last couple of months, spending days examining crash dumps, and adding new code to help diagnose the problem in every patch.

    With every crash we gather more information and get closer to a solution. The reason for the extended downtime today was that we decided to debug the live database when it crashed, rather than examining a crash dump after the fact. This allowed us to get more information and test some theories in ways we could not do when examining a dump after the fact, so it contributed to getting us to a solution faster.

    I just wanted to let you all know that we are working hard on the problem (it is almost midnight here!) and are making progress towards a solution. We have the resources we need, namely the people who know the code the best.

    Cheers,

    Brandon =/\=
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Once again, the server crashes. this is the second time this week. Oh well, not much we the consumer can do about it. Personally i love the game, one of the better ones i have played. And worth the wait. Just wish it would not happen so often.
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    I am soooo glad i don't pay for this I would be ****ed! I guess i'll go play superhero at dcuo they only go down when they give you fair warning maybe i'll try again later lol.......you guys have fun waiting
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Whats everyone complaining about...I'M STILL PLAYING :P
  • Archived PostArchived Post Member Posts: 2,264,498 Arc User
    edited March 2012
    Well, I WAS on DS9 and NOTHING was working. Couldn't even move stuff in my inventory. Then I stupidly beamed up to...the U.S.S. Loading Screen. I must be stuck in the transport buffer.:rolleyes:
This discussion has been closed.