Partial eCom Slowness EU1
Incident Report for Lightspeed - Status Page
Postmortem

On Friday afternoon we identified a slow down in our eCom applications concerning region EU1, affecting most of the EU webshops. A timeline of events is listed below:

  • Slow down was identified
    [14:51 CEST] - Updated our statuspages
    [15:06 CEST] - The cause of the slow down was found to be external shipping party, PostNL. PostNL was showing increased response times slowing down the entire platform. This impacted the store front, back offices, API and webhooks.
    [15:47 CEST] - Traffic to the webshops was restored. Connectivity issues remained in back offices.
    [16:12 CEST] - All traffic to all applications was restored to normal response times.
    [16:39 CEST] - Incident closed.

The root cause of this incident is related to an external service, PostNL. PostNL showed raised API response times and this had cascading affects across the various applications that use the PostNL integration.

After the incident we made changes to our infrastructure to minimize the impact when this reoccurs.

We understand that these issues have occurred at a critical time for your business and we have made every effort to ensure that you don't experience any further system interruptions.

Posted 4 months ago. Apr 19, 2019 - 11:04 EDT

Resolved
The incident has been resolved.

If you see missing order numbers in your overview, we recommend you to take a look at the Invoices and Shipment pages. Those orders are listed in that overview. No data has been lost, our systems are catching up in processing the orders.
Posted 4 months ago. Apr 19, 2019 - 10:39 EDT
Monitoring
All our applications are running and accessible. If you see missing order numbers in your overview, we recommend you to take a look at the Invoices and Shipment pages. Those orders are listed in that overview. No data has been lost, our systems are catching up in processing the orders. We remain to monitor the incident.
Posted 4 months ago. Apr 19, 2019 - 10:12 EDT
Update
The merchant websites have been restored in traffic. Consumers are able to access the websites again. We continue to investigate the availability of the back offices.
Posted 4 months ago. Apr 19, 2019 - 09:47 EDT
Identified
The degraded performance have been identified in an external shipping service. A dedicated team is working on investigating potential solutions.
Posted 4 months ago. Apr 19, 2019 - 09:06 EDT
Update
The slowdown is detected in both the API, Webhooks, Frontend and Backend. As a result you can be prompted with 50x errors and non-loading pages.
Posted 4 months ago. Apr 19, 2019 - 08:56 EDT
Investigating
We have identified a potential slowdown of our eCom platform. This is only affecting the EU1 region. Updates will follow as we are learning.
Posted 4 months ago. Apr 19, 2019 - 08:54 EDT
This incident affected: Lightspeed eCom (Lightspeed eCom (Europe)).