APRIL 29 UPDATE - 9:30AM PT
Hey everyone – After implementing the rollback on April 3, this specific issue was resolved across all impacted datacenters. Our team has continued monitoring the servers over the past month to make sure problem didn't arise again, and we're happy to say that it hasn't popped back up.
As a result, we'll be closing out this specific investigation thread. If you encounter issues online moving forward, please report them in this dedicated support thread.
Thank you again for helping us troubleshoot this issue!
APRIL 4 UPDATE - 4:25PM PT
We've got even more good news today as well. After bringing the West US datacenter back online and monitoring it closely, we can confirm it is no longer seeing the issue. This means that our NorthCentral US, SouthCentral US, and West US databases are back online and functioning as expected.
While we're considering this issue resolved, we will continue to monitor our all of our datacenters to ensure this does not impact any other locations or players. If you see any remaining cases of warping or hitching, please share a video of it happening and a include a link to the match here on Waypoint.
Thank you again for your patience as we worked through this over the past two weeks. It was a tough one to track down, solve, and apply in the live environment, so we truly appreciate you hanging in there with us.
-
APRIL 3 UPDATE - 7PM PT
Today comes with good news! Let's jump straight into it by location so you can see how things have progressed.
-
APRIL 2 UPDATE - 8PM PT
The rollback continued today and progressed enough for us to turn our NorthCentral US datacenter back on. Unfortunately, it has not completed in our other datacenters. Here's a status of each of our impacted locations:
Please know that we're working hard to bring everything back to normal and will provide daily updates until we're able to do so.
We can't thank you enough for your patience and understanding as we work though this with multiple external teams.
-
APRIL 1 UPDATE - 5:30PM PT
We worked with partner teams today to ensure the rollback continued throughout the day. Our new estimate is that it will be completed within the next 24 hours. While this has taken much longer than we initially anticipated on Friday, we're hoping that it will bring a resolution to these issues.
We'll continue to monitor the impacted datacenters and provide another update tomorrow on how the rollback has progressed.
-
APRIL 1 UPDATE - 12PM PT
Thanks for the continued reports over the weekend everyone. We noticed that our rollback has taken longer than anticipated and has not reached all of the impacted datacenters.
Our team has directed traffic from our West US datacenter, which was seeing some of these hitches, to other locations in order to help improve your experience until the rollback has completed.
We plan on updating this thread again before the end of the day with our progress. Please stay tuned for more information.
-
Please check this post to see all updates from March. Stay tuned for the most recent updates here.
-
Hey everyone – After implementing the rollback on April 3, this specific issue was resolved across all impacted datacenters. Our team has continued monitoring the servers over the past month to make sure problem didn't arise again, and we're happy to say that it hasn't popped back up.
As a result, we'll be closing out this specific investigation thread. If you encounter issues online moving forward, please report them in this dedicated support thread.
Thank you again for helping us troubleshoot this issue!
APRIL 4 UPDATE - 4:25PM PT
We've got even more good news today as well. After bringing the West US datacenter back online and monitoring it closely, we can confirm it is no longer seeing the issue. This means that our NorthCentral US, SouthCentral US, and West US databases are back online and functioning as expected.
While we're considering this issue resolved, we will continue to monitor our all of our datacenters to ensure this does not impact any other locations or players. If you see any remaining cases of warping or hitching, please share a video of it happening and a include a link to the match here on Waypoint.
Thank you again for your patience as we worked through this over the past two weeks. It was a tough one to track down, solve, and apply in the live environment, so we truly appreciate you hanging in there with us.
-
APRIL 3 UPDATE - 7PM PT
Today comes with good news! Let's jump straight into it by location so you can see how things have progressed.
- NorthCentral US: Rollback completed yesterday. We're seeing servers continue to perform exceptionally at this location.
- SouthCentral US: Rollback completed today. While we anticipated this to take until Saturday, our partners found and expedited an alternative solution which has resolved the issue at this datacenter.
- West US: Rollback completed today. This finished as planned today, but because it completed right before peak hours we opted to bring this location online tomorrow morning. In doing so, we ensure West US players can jump in without any issues tonight and allow the datacenter time for a more smooth return (and scaling) tomorrow.
-
APRIL 2 UPDATE - 8PM PT
The rollback continued today and progressed enough for us to turn our NorthCentral US datacenter back on. Unfortunately, it has not completed in our other datacenters. Here's a status of each of our impacted locations:
- NorthCentral US: Rollback completed. The location has been brought back online and we're already seeing servers perform well.
- SouthCentral US: Rollback in progress. Our partners anticipate that this datacenter will finish rolling back on Saturday. While only a fraction of the matches hosted in this location were encountering the issue, you may still experience lingering funkiness until everything is complete.
- West US: Rollback in progress. Our partners anticipate that this datacenter will finish rolling back tomorrow. Once it is completed, we'll bring it back online and monitor the performance of its matches.
- NorthCentral US: Actively monitor the matches hosted here and ensure it continues to perform well.
- SouthCentral US: Our focus will be to monitor the rollback with partner teams and ensure we're both prepared for it to complete by Saturday.
- West US: Validate that the rollback completes on the WestUS datacenter, bring it back online, and monitor it closely (as mentioned above)
Please know that we're working hard to bring everything back to normal and will provide daily updates until we're able to do so.
We can't thank you enough for your patience and understanding as we work though this with multiple external teams.
-
APRIL 1 UPDATE - 5:30PM PT
We worked with partner teams today to ensure the rollback continued throughout the day. Our new estimate is that it will be completed within the next 24 hours. While this has taken much longer than we initially anticipated on Friday, we're hoping that it will bring a resolution to these issues.
We'll continue to monitor the impacted datacenters and provide another update tomorrow on how the rollback has progressed.
-
APRIL 1 UPDATE - 12PM PT
Thanks for the continued reports over the weekend everyone. We noticed that our rollback has taken longer than anticipated and has not reached all of the impacted datacenters.
Our team has directed traffic from our West US datacenter, which was seeing some of these hitches, to other locations in order to help improve your experience until the rollback has completed.
We plan on updating this thread again before the end of the day with our progress. Please stay tuned for more information.
-
Please check this post to see all updates from March. Stay tuned for the most recent updates here.
-