Partial eCom Slowness EU1

Incident Report for Lightspeed - Status Page

Postmortem

On Friday afternoon we identified a slow down in our eCom applications concerning region EU1, affecting most of the EU webshops. A timeline of events is listed below:

Slow down was identified
[14:51 CEST] - Updated our statuspages
[15:06 CEST] - The cause of the slow down was found to be external shipping party, PostNL. PostNL was showing increased response times slowing down the entire platform. This impacted the store front, back offices, API and webhooks.
[15:47 CEST] - Traffic to the webshops was restored. Connectivity issues remained in back offices.
[16:12 CEST] - All traffic to all applications was restored to normal response times.
[16:39 CEST] - Incident closed.

The root cause of this incident is related to an external service, PostNL. PostNL showed raised API response times and this had cascading affects across the various applications that use the PostNL integration.

After the incident we made changes to our infrastructure to minimize the impact when this reoccurs.

We understand that these issues have occurred at a critical time for your business and we have made every effort to ensure that you don't experience any further system interruptions.

Posted Apr 19, 2019 - 11:04 EDT

Resolved

The incident has been resolved.

If you see missing order numbers in your overview, we recommend you to take a look at the Invoices and Shipment pages. Those orders are listed in that overview. No data has been lost, our systems are catching up in processing the orders.

Posted Apr 19, 2019 - 10:39 EDT

Monitoring

All our applications are running and accessible. If you see missing order numbers in your overview, we recommend you to take a look at the Invoices and Shipment pages. Those orders are listed in that overview. No data has been lost, our systems are catching up in processing the orders. We remain to monitor the incident.

Posted Apr 19, 2019 - 10:12 EDT

Update

The merchant websites have been restored in traffic. Consumers are able to access the websites again. We continue to investigate the availability of the back offices.

Posted Apr 19, 2019 - 09:47 EDT

Identified

The degraded performance have been identified in an external shipping service. A dedicated team is working on investigating potential solutions.

Posted Apr 19, 2019 - 09:06 EDT

Update

The slowdown is detected in both the API, Webhooks, Frontend and Backend. As a result you can be prompted with 50x errors and non-loading pages.

Posted Apr 19, 2019 - 08:56 EDT

Investigating

We have identified a potential slowdown of our eCom platform. This is only affecting the EU1 region. Updates will follow as we are learning.

Posted Apr 19, 2019 - 08:54 EDT

This incident affected: Lightspeed eCom (C-Series) (Lightspeed eCom (Europe)).