Your site probably gets more traffic than you think. Search engines and some spambots that load the pages will also trigger the WordPress CRON actions.
Otherwise, the only way to make sure that the WordPress faux CRON functions actually get triggered at regular intervals is to disable the built-in WordPress CRON triggers, and instead run the WordPress CRON from a real server level CRON function that you set up to call the WordPress CRON function.
Takes a bit of server knowledge, but Tom McFarlin spells out the issues and how to do it properly here:
https://tommcfarlin.com/wordpress-cron-jobs/
Since my plugin already schedules its own functions (for reminders), you can skip steps 1 & 2 in the second section on “Setup your Event code”.