Man it feels good to come in a middle of a project and fix something!
There was a high priority project that came in last quarter and members from various teams were pulled in to create a
tiger team. Project was completed in time and it was pretty good for how little time was given.
But recently it started to fall hard under heavy load. Memory leaks and it was just god awful slow. Sometimes we got away with just throwing more memory at it and re-running. However there has been some interest in it lately since it was generating a good amount of income for our company. As a result our team was put on task of fixing it.
The project is written in Scala, however none of the code followed any good FP principles so it was very difficult to read and reason through. To be fair everyone who touched that code had less than 6 months of Scala experience.
What annoyed me was that my boss and team members were a little hesitant in letting me re-write that portion of the service.
I recognized it as the classic producer consumer problem, so I said screw it and started to rewrite it using a thread pool for producer and 1 thread for reader. Spent the whole day finishing it up followed by benchmarks and profiling the memory. Deployed just recently and now it's running in production with no memory leaks and high throughput
Prior to this, it was taking 6 hours to run this service with the average heap size at 10 gigs, now it is expected to finish in 1.5 hours with less than 5 gigs of heap used.