Description:

Recently at chi.mp we converted the back-end feed processing engine that consumes tens-of-thousands of data feeds from Python to JRuby. With native threading we were able to greatly reduce the number of machines required to run the processing engine while increasing the throughput. But all is not unicorns and rainbows. In the process we discovered issues both with Ruby libraries that were not thread-safe as well as limitations due to blocking IO. With JRuby we are able to solve both of these issues and have fun doing it. In this talk I will go through some of the details of what we've learned and how we've used the awesome Java integration layer in JRuby to wrap thread-safe Java libraries and take advantage of Java's non-blocking IO capabilities through the Apache HTTPCore NIO library.