Course outline for Reliable Workflow Execution

Module 1: Own execution dependencies

• Coordination: Assign stage ownership and durable completion records so dependent work proceeds only after its required artifacts exist.
• Scheduling: Order ready stages by dependency and execution constraints, separating eligibility to run from successful stage completion.
• Parallelism: Identify independent report stages that can run together without conflicting writes or violating required artifact dependencies.
• DAGs: Represent acyclic stage dependencies with typed artifacts, making prerequisite completion and safe parallel execution explicit.

Practical outcome: Define a report execution graph with typed stage artifacts, safe parallel work, and a durable completion record.

Module 2: Bound failure handling

• Queues: Track pending execution attempts separately from durable stage state so repeated delivery does not imply repeated publication.
• Idempotency: Use effect keys to recognize repeated publication attempts and verify one committed mock effect despite lost acknowledgements.
• Retries: Bound attempts for transient failures and separate harmless recomputation from effects that require deduplication or reconciliation.
• Timeouts: Treat an expired wait as uncertain effect status, investigating successful actions whose acknowledgements were lost before retrying.

Practical outcome: Implement retry limits and effect keys, then investigate a lost acknowledgement after a successful mock publication.

Module 3: Recover with explicit ownership

• Compensation: Map failed execution steps to possible corrective actions, distinguishing reversible effects from irreversible outcomes requiring manual reconciliation.
• Checkpointing: Persist completed stages and artifact references so interruption recovery resumes from recorded progress rather than restarting all work.
• Recovery: Resume an interrupted approved run without repeating completed stages, checking pending effects and recording unresolved failures.
• Distributed execution: Protect shared execution state with version-checked ownership, rejecting stale worker updates while acknowledging partial-failure risks.

Practical outcome: Resume from a checkpoint, reject a stale worker update, and define a compensation table for reversible and irreversible effects.

Practical exercise

• Implement a local report runner with a state store, approval gate, mock publication sink, and failure-injection report.
• Complete a valid approved run; reject an unapproved publication and a stale worker's state update.
• Inject a transient failure, interruption, and lost acknowledgement; verify bounded attempts and one committed publication for the fixture's effect key.
• Resume without repeating completed stages, and document which failed effects need compensation or manual reconciliation.