We would like to inform you about the incident that occurred on the 14th of October 2024, which may have impacted our payment platform and affected your services. Below, we provide an overview of the incident, including its root cause, duration, and current status.
Incident Overview: On the 14th of October 2024, our payment platform experienced a service disruption affecting transaction processing. The issue primarily impacted e-commerce transactions and caused an outage on our Datatrans gateway.
Root Cause: The outage was due to an application crash on one of our primary servers. Initial investigations indicate that a rare race condition within the persistence framework (EclipseLink version 2.7.15) might have triggered a memory leak. This memory leak led to server resource exhaustion, preventing transaction processing and ultimately causing the crash. Our engineering teams are still investigating the exact cause.
Duration: - Start of impact: 14th October 2024, 19:55 CEST - End of impact: 14th October 2024, 20:48 CEST The total duration of the service disruption was approximately 53 minutes.
Current Status: The issue has been resolved by switching the platform to a backup server, and services are now fully restored. However, we are continuing to investigate the root cause to ensure such issues are prevented in the future.
Preventive Measures - We are implementing the following measures: - Reviewing and optimizing the monitoring and alert system to ensure faster escalation of critical incidents. - Exploring improvements to the failover mechanism to minimize future downtime. - Conducting a detailed review of the memory management and thread handling processes in the affected system components.
Next Steps: We are working diligently to implement the necessary safeguards to prevent this from happening again.
We apologize for the inconvenience this may have caused and appreciate your understanding as we work towards enhancing the stability and resilience of our platform. Should you have any further questions, please reach out to our support team at support@pci-proxy.com.