An update to my previous post. I was able to improive the average lag by disabling the serial monitor, passing a reference of the controller to the polling logic (eliminating the need to loop over the current state and previous state to determine if buttons should be pressed), and adding a 1ms delay between loops (should have realized that the board need some down time between calls). I've added the code since I think 8ms is a perfectly good lag result for a diy project.