Online bookings take too many time to confirm, Formitable app booking updates are blinking
Resolved
Dec 30 at 08:48pm CET
Timing
19:45 to 20:48 CET
Who was impacted
All customers using the Formitable booking widget during the timeframe above
Restaurants using the Formitable platform during the same timeframe
What happened (Symptoms)
During this period, the platform remained online and accessible, but bookings and updates were not confirmed in real time.
• Back-office (Formitable platform): The timeline remained accessible, but actions such as creating or updating bookings were very slow to save.
Some bookings appeared to “blink” or remain in a “processing” state for longer than expected.
• Widget (online bookings): The widget stayed available, but some bookings did not confirm successfully.
In some cases, a prepayment step was triggered but the booking never reached confirmation.
Why it happened (Root cause)
Formitable processes bookings through a background system that validates and confirms actions in real time. During the incident, an unexpected automated task generated a large number of background actions at once, which temporarily slowed down the confirmation of new bookings and updates.
During the incident window, an automated task that runs once per year unexpectedly triggered a high volume of events. This created a backlog in the processing queue, which meant critical events (like booking creation and confirmation) were delayed or blocked.
As a result:
booking confirmations were delayed or failed,
modifications took much longer than usual,
some bookings stayed in “processing” mode (blinking),
and in some cases, the prepayment flow couldn’t complete because confirmation events were stuck behind the backlog.
What we did (Immediate resolution)
• We identified and cleared the event processing backlog.
• We restored normal event processing and booking confirmations.
• We replayed all delayed booking creation and update events to ensure no booking information was lost.
Pending online bookings submitted during the incident window have now been processed.
What we’re doing to prevent this from happening again
We are permanently disabling and removing the event type responsible for the backlog.
We are adding safeguards to ensure that unexpected spikes of background events cannot block critical booking flows.
We are increasing processing capacity and improving monitoring so that we detect and mitigate any backlog faster in the future.
Affected services
Created
Dec 30 at 08:15pm CET
Online bookings take too many time to confirm, Formitable app booking updates are blinking
Affected services