Error Handling and Retries in Workflows

Understanding the failure taxonomy is prerequisite to designing appropriate recovery logic. Transient failures are temporary and resolve on retry without any change to the request: API rate limits, model service congestion, network timeouts, and temporary webhook delivery failures all fall into this category. A retry after a short delay (5–15 seconds for rate limits, 30–60 seconds for model congestion) resolves most transient failures within two to three attempts.

Deterministic failures will not resolve on retry with the same inputs: content policy rejections, invalid parameter combinations, references to non-existent model IDs, and malformed input images all produce the same error regardless of how many times the node is retried. Retrying a content policy rejection wastes credits and delays the workflow without any chance of success. The correct response to a deterministic failure is immediate escalation — stop retrying, log the failure reason, and route to a human review queue or a fallback branch with an alternative prompt or model.

Structural failures arise from errors in the workflow definition itself: circular dependencies, missing input connections, incompatible data types between connected nodes. These should be caught at workflow validation time (before execution begins) rather than during execution. In Floniks, the editor surfaces structural failures as validation warnings when you attempt to save or run a workflow, making them the easiest category to address — fix the graph structure before the first run.

For each AI generation node in a workflow, configure a retry policy directly on the node: maximum retry count (2–3 for most use cases), retry delay (10 seconds for the first retry, exponentially increasing for subsequent retries — 10s, 30s, 90s), and the set of error codes that qualify for automatic retry. In Floniks, the node retry configuration panel accepts error code patterns: "429" for rate limits, "503" for service unavailable, "timeout" for request timeouts. Errors matching these patterns trigger automatic retry; all other errors are treated as deterministic and bypass the retry logic entirely.

Exponential backoff is important because retrying too quickly under rate limit conditions will immediately trigger another rate limit error. The exponential delay gives the upstream API time to reset its rate limit window. Cap the maximum delay at 90 seconds — delays beyond this point are more likely to indicate a service outage than a transient overload, and the appropriate response shifts from retry to fallback-model escalation. Log every retry attempt with its error code, delay duration, and attempt number in the workflow task record so post-run analysis can identify which node families are most retry-prone and whether the current retry configuration is sufficient.

When a node exhausts its retry budget or receives a deterministic error code, the workflow must branch to a predefined fallback path rather than halting. The fallback branch typically takes one of three forms depending on the failure type.

Content policy fallback: the primary generation node's prompt triggered a content filter. The fallback branch connects to a Prompt Sanitizer node that applies conservative rewriting (remove specific descriptors, generalize sensitive language) and then routes to a secondary generation node with the sanitized prompt. Configure the Prompt Sanitizer to log both the original and sanitized prompts in the task record for human review — automatic rewriting should be transparent, not silent.

Model unavailability fallback: the primary model returned a 503 or "model unavailable" error after retry exhaustion. The fallback branch routes to an equivalent model from a different provider. For image generation, a FAL.ai model that is unavailable can fall back to an APImart-hosted equivalent. The fallback model may produce slightly different style outputs; document this in the workflow template description so users are not surprised by style variation in fallback-generated outputs.

Quality fallback: the primary node's output failed a downstream quality check (consistency validator, realism validator). The fallback branch adjusts model parameters — higher guidance scale, different sampler settings — and regenerates. Quality fallbacks are distinct from retry because they modify the request parameters, not just re-submit the same request.

When a workflow run ends in partial or complete failure after retry and fallback branches are exhausted, the failure must be reported in a structured, actionable format — not just a generic "workflow failed" status. Implement a Failure Aggregator node at the terminal stage of every failure branch in the workflow. This node collects: the ID of the failed node, the error code and message, the input parameters that were active at failure time, the retry count before escalation, and whether the failure was transient or deterministic. It writes this structured record to the workflow task document's 'execution_logs' field and, optionally, triggers a notification to a configured webhook endpoint (Slack, email, internal alerting system).

The structured failure report allows an operator to diagnose the root cause in seconds. A report showing "node: background-replacement, error: 429, retries: 3, input: image_url=..., prompt='...'" immediately tells the operator that the background replacement model is rate-limited and either the request volume is too high or a rate limit upgrade is needed. Compare this to a generic "pipeline failed" alert that requires manually re-running the pipeline in debug mode to reproduce. Structured failure reporting is as important as the retry logic itself — it is what allows your team to continuously improve the workflow's robustness over time.

A critical but often skipped step is deliberately testing every error branch before deploying a workflow to production. Floniks supports a test mode per node where you can configure the node to return a specific simulated error code — 429, 503, content policy rejection — regardless of what the actual API returns. Use this to walk every retry and fallback path in isolation: set node three to simulate three consecutive 429 errors and verify that the retry logic fires correctly and then eventually routes to the model fallback. Set a generation node to simulate a content policy rejection and verify that the Prompt Sanitizer fires and the sanitized prompt is logged.

Document the expected behavior of each error branch in the workflow template's description field so any team member who inherits or troubleshoots the workflow can reference it without reverse-engineering the graph. Include test prompts that reliably trigger each error type — for example, prompts known to trigger the content filter can be used to verify the content policy fallback without relying on real API behavior. This investment in testing error paths before production deployment prevents the most common category of production incident: a workflow that worked fine during development with clean inputs but fails silently in production when encountering real-world error conditions.