A migration where every scheduled job silently stopped

The site was quicker, the pages were correct, and nothing in the logs looked wrong. Product imports had not run once since the move, and neither had the background queue or the monthly sales report.

The cause

The command line tool those jobs depend on was installed at one path on the new server and the cron entries called it at another. Every job failed instantly with a file-not-found error, and every one of them ended by redirecting its output to the null device, which is the convention everybody uses to stop cron emailing them.

So the failures produced no email, no log line and no alert. A one-line fix, hidden behind a habit that is otherwise sensible. Redirecting output to nowhere turns a failing job into a job that appears not to exist.

The rest of the move

• Object caching over a Unix socket, which needs the socket permissions and the group membership set correctly, and a full process restart rather than a reload before it takes effect.
• Product imports switched from being triggered by fetching a URL to running directly through the command line, with lock files so a long import cannot overlap the next scheduled run.
• Page caching moved to CDN-only, because two cache layers disagreeing is worse than one.
• Authenticated origin pulls, so the server only answers requests that arrived through the CDN.
• Offsite backups over SFTP to separate storage.
• Two fatal errors on the newer PHP version tracked down and resolved before they reached a customer.

One trap worth knowing

The hosting panel on that server treats the percent character in a cron command as a line break, so anything containing a date format string is silently truncated into nonsense. Those commands have to be wrapped in a small shell script instead. It is the kind of detail that is obvious once, and costs an afternoon the first time.

What I changed afterwards

Scheduled jobs now write to a log rather than to nowhere, and the important ones record when they last completed. A job that fails loudly is a minor annoyance. A job that fails silently is discovered weeks later by a customer.