A cron job and a well-tested script
We are regularly asked to "move the pipelines onto a proper orchestrator". Sometimes that is the right call. More often the existing setup is a handful of cron entries that run once a night, and the real problem is that the scripts they call are not safe to run twice.
Before reaching for new infrastructure, we check four things.
1. Can two copies run at once?
If yesterday's run is slow and today's starts on top of it, what happens? Usually something bad. A lock costs one line:
10 2 * * * flock -n /run/lock/import-a.lock /opt/jobs/import-a2. Is a rerun safe?
The job should produce the same end state whether it runs once or three times for the same day. That usually means loading into a staging table and swapping, or deleting the day's partition before inserting it - never blindly appending.
3. Does the exit code mean something?
Cron, systemd timers and every CI system understand exit codes. A script that catches every exception and exits 0 has opted out of all of that tooling.
4. Can a new person rerun it?
If the answer to "how do I rerun Tuesday" is "ask Sam", the job is not finished. One paragraph in a runbook fixes this.
Once those four hold, a nightly cron job is often the most reliable component in the stack. When they do not, moving it onto an orchestrator mostly moves the problem somewhere with a nicer UI.