Almost all stuttering during scroll has been removed!
For those who want to know the technical reasons, all the caching was happening on the same thread that was handling the scrolling, but moving it to a background thread makes it feel much better!