Retries and durability

An OuterProduct Run forms a computation graph. The top-level client.run() creates a new Run and its root Call. Inside that Call, the scoped client can create child Calls within the same Run. This hierarchy allows OuterProduct to track each Call independently. For example, in the following code, the pipeline call orchestrates a 2-step workflow (first, prepare_data , then train_model ). Every function receives an OuterProduct client as its first argument. A function can ignore it, as _client does in the example below, or use it to create more Calls. Breaking an application into Calls gives OuterProduct durable boundaries where work can be retried, tracked, and checkpointed.

import outerproduct as op

image: op.Image = op.Image.from_registry("python:3.12-slim")


def prepare_data(_client, records: list[str]) -> list[str]:
    # ...
    return prepared_records


def train_model(_client, records: list[str]) -> object:
    # ...
    return trained_model


def pipeline(client, records: list[str]) -> object:
    prepared: list[str] = client.run(
        fn=prepare_data,
        args=(records,),
        image=image,
    ).wait()

    return client.run(
        fn=train_model,
        args=(prepared,),
        image=image,
    ).wait()


if __name__ == "__main__":
    client = op.init()

    run: op.Run[object] | op.Call[object] = client.run(
        fn=pipeline,
        args=(["To be", "or not to be"],),
        image=image,
    )

    model: object = run.wait()

Retrying a Call

Use RetryStrategy to specify the maximum number of Attempts for a Call:

run: op.Run[object] | op.Call[object] = client.run(
    fn=train_model,
    args=(training_data,),
    image=training_image,
    gpu_type="L4",
    retry_strategy=op.RetryStrategy(max_attempts=4),
)

max_attempts includes the first Attempt. In this example, OuterProduct can try the Call up to three times. If no retry strategy is provided, the Call receives three Attempt by default. Pass RetryStrategy(max_attempts=1) to disable retries. Each Call defines its own retry strategy; a parent and its children can therefore use different limits.

Durable execution

OuterProduct checkpoints work at Call boundaries. Once a Call succeeds, its result is stored as part of the Run. If a parent Call fails and is retried, it begins executing again. When it reaches a child Call that already succeeded, OuterProduct returns the stored result instead of executing that child again. A long workflow can therefore recover from a failure without repeating all of its successful work. For example:

Attempt 1
├── prepare_data   ✓ completed
├── train_model    ✓ completed
└── publish_model  ✗ failed

Attempt 2
├── prepare_data   ↪ reuse result
├── train_model    ↪ reuse result
└── publish_model  ✓ completed

If a Call fails halfway through its function, its next Attempt starts that function again from the beginning. Split expensive or important stages into separate Calls when you want them checkpointed independently.

Getting started

Execution

Governance

Retrying a Call

Durable execution

​Retrying a Call

​Durable execution

Retrying a Call

Durable execution