Due to a little blip two weeks ago and impeccable timing, our background task for processing monthly subscription renewals and cancellations haven’t been processed since. We just caught this issue when noticing subscriptions were lagging behind and have put the system at work to catch up, which has just completed.
If you’re interested in the technical background…
Our subscriptions are renewed by a Scheduler task that runs a couple of times a day, which checks if there are any subscriptions due or cancelled that need to be processed. The task runs, and starts off by rescheduling itself to run again at a certain time to make sure it happens continuously.
Two weeks ago I was shuffling around some dependencies in an effort to simplify site deployments, which caused the site to be down for roughly a minute (if that) due to a path change that needed to be adjusted. It seems that in that exact moment Scheduler was triggered and tried to run the Subscriptions task. As Scheduler couldn’t find the task, it simply failed and it was not rescheduled like it would normally.
Other tasks (including backups, creating invoices, sending weekly stats into our team Slack etc) were not affected, making it harder to spot something was amiss until subscriptions were overdue.
Rather than a single self-repeating task, I’ve now scheduled two independent self-repeating tasks that run at opposite intervals. While this means it will trigger quite a bit more often than necessary at our current subscription levels, it does make it more resilient against Mr. Murphy deciding to break the flow at the one point in time where it’ll fail to reschedule.