We have been monitoring and responding to specific performance issues throughout the afternoon, but the main issue is now resolved. The root cause of this related to differences in how load-balancing works in our new environment compared to before; this meant the database server received more connection requests than it was designed to handle. We offer our apologies to those affected by today's issues.
Posted Mar 25, 2024 - 21:29 GMT
Monitoring
Following a configuration change and an application restart, PICS should now have retuned to normal operation. If you continue to experience issues, please provide details to the support team via the usual channels. We will continue to monitor things for a while.
Posted Mar 25, 2024 - 10:57 GMT
Update
We are currently restarting the web application - it will go offline for a few seconds.
Posted Mar 25, 2024 - 09:59 GMT
Update
Our engineering teams continue to investigate these issues. In addition to HTTP 500 errors, some page may return the message "The underlying provider failed to Open". We will update you as soon as we know more. Apologies for the disruption caused so far this morning.
Posted Mar 25, 2024 - 09:46 GMT
Investigating
Some customers are reporting that PICS is running slow this morning, or that they are getting HTTP 500 errors on the Home screen. We are working with the engineering team to investigate further.