Skip to content

Add test implementation of jobs/stages/tasks

Jonathan Mace requested to merge mace_countdownlatch into master

This test illustrates an execution pattern that I've seen before in batch processing frameworks (specifically, Hadoop and Spark).

Each job is conceptually a separate request and we want to trace all of the activity in each job. A job maps to multiple stages. A stage maps to multiple tasks. Tasks within a stage can execute concurrently. Stages must execute in a specific order.

Ideally, the beginning of a stage N should receive all of the contexts from the tasks of stage N-1. Although the "simple" instrumentation approach might have stage N receive just the context of the final task of stage N-1.

Merge request reports