This happens literally all the time for me both personally and professionally. I see mostly low effort attempts across various ports or things like sweeps of common username/password attempts on ssh or common management endpoints on http.
This is why it's important to keep all publicly accessible servers and services updated and follow standard security guidelines. Things like only using public key auth for ssh for instance.
At work we get hit occasionally in large bursts and have to ban ips for a bit to get them to go away.
If it were me and I was intending to automate this I would probably do the following. Set up each test distro as a VirtualBox image and take a snapshot so I could easily roll back. Then I would write a script for each distro that downloaded the package, installed and launched the app. I would then probably query the window system to make sure the gui showed up, wait a period of time if I had to and take a screenshot.
This can probably all be done as a set of bash scripts.