Stabilising a Fragile Booking Platform and Restoring Operational Confidence

An organisation depended on a custom-built booking platform to manage its daily operations.

Over time, the system had become fragile, unpredictable, and difficult to maintain. Reliability issues were increasing, and internal confidence in the platform was declining.

Anorthic Labs was engaged to stabilise the system, improve reliability, and restore trust in the software.

The platform had evolved over several years without consistent engineering oversight. Several critical issues had emerged:

• Jobs were accumulating faster than they could be processed
• System performance was degrading under normal operational load
• Failures were difficult to diagnose due to lack of internal safeguards
• The architecture allowed duplication of background processing tasks
• Internal visibility into system health was limited

At its peak, over 80,000 background jobs had accumulated in the queue, creating a growing backlog and increasing operational risk.

Although the system was still functioning, its reliability was deteriorating. This created uncertainty and risk for the organisation.

The first priority was to understand the system's behaviour under real conditions. This involved:

• Analysing job processing logic
• Reviewing background worker configuration
• Identifying architectural weaknesses
• Mapping how data flowed through the system

Several key structural issues were identified. Background jobs could be dispatched repeatedly without restriction. The scheduler allowed overlapping execution, creating duplicate processing. Queue workers were insufficiently provisioned for the workload.

These factors combined to create exponential job growth. The solution required architectural correction, not temporary fixes.

Anorthic Labs implemented a series of engineering improvements to stabilise the system. These included:

Introducing uniqueness constraints on background jobs to prevent duplication.

Implementing scheduler protection to prevent overlapping execution.

Reconfiguring queue workers to process jobs more efficiently and in parallel.

Removing tens of thousands of stale and invalid jobs from the system.

Improving internal structure to ensure the system could sustain future growth.

These changes addressed the root causes of instability – not just the symptoms.

System stability was immediately restored. Background job processing returned to normal levels. The backlog was eliminated.

The system was able to operate reliably under real-world conditions. Internal confidence in the platform was restored.

Most importantly, the platform returned to being an asset rather than a source of operational risk.