The Power of Simplicity: How a random iterator saved the day
…and my database.
How many times has something so simple saved the day? Sometimes, the simplest solutions can have a significant impact on a problematic situation. In this blog post, we'll explore one such scenario where a small algorithm change led to substantial performance improvements in an existing system.
The issue at hand involved a component in a system using a database table as a makeshift queue for processing updated rows. Multiple processor instances read from the same table, with Redis locking in place to prevent concurrent processing of the same row. However, the processors working in the same order led to numerous collisions and timeouts, causing the system to slow down.
As illustrated in the image, the processors were interfering with one another; while they ultimately completed the work, excessive time was consumed in fruitless attempts to lock records, which consequently increased the strain on the database.
The solution: A Random Iterator
A random iterator is an algorithm that iterates over the elements of a collection in random order. Here is a C# extension method for such an iterator over IList<T>.
By implementing a random iterator in the component, each processor would grab a full copy of the table and process it randomly, significantly reducing the chances of locking collisions. The table's size allowed it to fit entirely into memory, and idempotence ensured that duplicated attempts were properly handled.
The result was a drastic performance improvement, allowing the number of processor instances to be reduced, and the overall solution went from a system barely performing to over-performing.
A simple change can sometimes make a world of difference in a system's performance. It's essential to think outside the box and question existing architectural decisions. Interestingly, some of these simple solutions can be traced back to our college days, reminding us that even the most basic ideas and algorithms from our early education can still be powerful tools in our professional lives.
While it may not always be possible to rewrite an entire system, a bit of ingenuity can often lead to significant improvements. Remember that good architecture is a roadmap, not a final state.