What you will learn Not a chapter to read but one to do. Build a three-stage pipeline, then break the middle stage on purpose and see what the later stages do.
If you have never set one up, read scheduled runs first.
Keep that last line. Put a send or a post in your first pipeline and step 4, where you break it deliberately, becomes a real incident.
Draw it on paper first. The key is writing down what each stage leaves behind.
Every stage must leave a file. A stage that holds its result only in its head cannot be re-run from the middle — this is what "stages must be re-runnable" in pipeline design actually looks like.
Do not build all three at once. Stand one up and verify it.
Do not skip the fourth. A file existing and a file being right are different things. Skip it and, when stage 3 comes out wrong, you cannot tell where it went wrong.
Build each later stage so that it reads the previous stage's output.
Do not put stage 2 at 09:05 because stage 1 usually takes three minutes. There will be slow days. Leave at least three times the usual duration, or write the stage so it stops when the previous output is missing.
This is the point of the exercise. Rename or delete stage 1's output file, then run stages 2 and 3.
| What stage 2 did | Verdict |
|---|---|
| Left "no previous result" and stopped | ✓ properly designed |
| Produced an empty table and completed | stage 3 now summarises that empty table |
| Found and used an older file | the worst — today's report goes out on yesterday's data |
That third row is the classic production incident. No error is raised, so nobody notices. The numbers just quietly slip by a day.
If it did not propagate, fix the stage 2 instruction and do step 4 again. One stage's failure being laundered into success by the next is the most common way a pipeline fails.
Do count the first one. Every stage carries cost, and running daily makes it accumulate. Three stages every day is ninety runs a month.
1. Why must every stage leave a file?
So you can re-run from the middle. Without the result on disk you cannot re-run stage 2 alone; you have to redo everything from the start.
2. Why is "found and used an older file" the most dangerous outcome?
No error is raised, so nobody notices. The pipeline reports success and today's report goes out built on yesterday's data.
3. Why leave generous gaps between stages?
There will always be slow days. If the next stage runs before the previous finished, it proceeds on a missing or stale result. Widen the gap, or write the stage to stop when there is no result.
Next, turning the repeated procedure itself into a reusable unit → Capturing procedures as skills
HyperTeams□ 60 minutes□ Schedules working□ Auto-start enabled — schedules do not fire if it is down□ One working directory that is easy to restore□ Do not include a stage that goes outside this time[1 collect] -> what it leaves:[2 shape] -> what it leaves:[3 summarise] -> what it leaves:Example: 1 collect every day at 09:00, write yesterday's changed files to a file 2 shape read that list, group by type, write a table 3 summarise read the table, write a three-line summary□ You created the stage 1 schedule□ You ran it once by hand□ The output file actually appeared□ You opened that file and checked the contents□ Stage 2 reads stage 1's output file□ Stage 3 reads stage 2's output file□ Each stage runs later than the one before <- check the gapsStage 1 09:00 -> Stage 2 09:20 -> Stage 3 09:40The line to put in stages 2 and 3: "If there is no file to read, produce nothing and leave only 'no previous result'."□ You opened stage 3's output□ Did "none" propagate into it, or did you get a plausible summary?□ You counted how many times a day each of the three runs□ You turned off the practice schedules□ For anything you kept, the failure behaviour is in the instruction□ Every stage leaves a file□ You stood the stages up one at a time and verified each□ Gaps are at least three times the usual duration□ You broke the middle and checked the later stages <- skip this and you did not do it□ "No previous result" propagated all the way to stage 3