How It Works

biocomposer decomposes the reasoning required to run a bioinformatics workflow into a stack of abstraction levels, so that at each level only one kind of decision has to be made. Composing tools normally forces the user to reason about everything at once: which tools to chain, how their formats and arguments line up, how to invoke each one, and how to provision its environment. biocomposer separates these concerns into layers, and each layer resolves its own class of decision before delegating the rest downward. The result is that the burden of reasoning about data flow moves smoothly from the user’s intent down to a concrete execution, with no single level holding all of the complexity.

Concretely, a pipeline is expressed against a small Python API; the API operates on a graph of typed nodes; and the nodes are backed by a layer of specializations (connector generation, per-item processing, feedback loops via decision nodes). Each level is a thin surface over the one beneath it, the user calls add_node(), which resolves a typed tool from the registry, which in turn pulls an image and triggers a container run.

Pipeline Dataflow

A pipeline is a directed acyclic graph (DAG): tools are nodes and data hand-offs are directed edges.

        flowchart LR
    A(["inputs"]) --> B["align"]
    B --> C["trim"]
    C --> D["fold"]
    D --> E(["result"])

The DAG representation gives the system three properties. Dependencies are explicit: an edge A → B declares that B consumes A’s output, and so run order never has to be specified by hand. The model generalises past linear chains: fan-out, fan-in, branching, and result sharing are all instances of the same node/edge structure. And it localises data conversion: mapping one tool’s output to another’s input is handled per edge, rather than as global script logic.

The API surface

The top layer is a small graph-construction API: add_input_node() for user-supplied values, add_node() for tools, add_edge() to wire them, set_output_node() to mark the result, and execute() to run. Construction is declarative: it records the graph, and no tool runs until execute() is called. These calls are deliberately thin. add_node triggers manifest resolution and image setup beneath it, and execute drives the evaluation described below, so the user never reasons about either.

The graph and node layer

Each node carries the typed input/output schema of its tool’s manifest. The following node types are specified:

  • Input nodes hold user files and parameters.

  • Tool nodes wrap a single registry tool and run once.

  • Gather nodes run a one-input tool once per item of an upstream collection, then gather the results.

  • Decision nodes re-run a node with adjusted inputs until a scored condition holds, a feedback loop contained within the node, so the graph as a whole stays acyclic. The score and modifier are each a Python function or a registry tool.

  • Subgraphs expose a one-in/one-out pipeline as a single node.

Edges are generated rather than written: each edge is realised as a connector that maps one tool’s output onto the next tool’s input, handling format conversion and item selection. When two edges supply the same input key, resolution is deterministic and governed by edge order (Input merging and key clashes).

The tool and container layer

At the base, each tool is a container image paired with a typed manifest. add_node("clustalo") resolves the manifest and pulls the image from DockerHub/Biocontainers/etc. Executing each tool in its own container pins its exact software and versions, making runs reproducible and letting tools with incompatible dependencies coexist in one graph. This layer is itself a wrapper: the tool library sits over external image sources, purpose-built images, BioContainers, or Docker Hub, so adding a tool means writing a manifest over an existing image rather than repackaging it (Tools and the registry, Adding your own tool).

How biocomposer runs tools

Constructing a graph defines a pipeline without executing it; computation begins only when execute() is called. At that point, biocomposer runs each tool in an isolated container, transfers data between consecutive steps, and returns the results. Tool installation and environment management are handled automatically, and the same pipeline runs on local or cloud hardware without modification.

Local vs. cloud

The same pipeline script runs in two places (see Installation):

  • Locally, using Docker on your machine, fine for lightweight tools.

  • On the cloud (Modal), which provides remote GPU machines for heavy tools like RFdiffusion or ColabFold. Results are stored on a persistent cloud volume you can download from.

Checkpoints

execute() returns the final step’s output as a dictionary of named results, typically paths to the files each tool produced. Every intermediate step’s output is also saved on disk under results/, so nothing is lost between steps.

Note

Tools are cited and used faithfully: each is run as its authors originally presented it, without modifying the original architecture or design choices.