biocomposer: graph-based orchestration of bioinformatics tools
Machine Learning @ Berkeley (ML@B)
Aditya Lakshminarasimhan · Arshia Nayebnazar · Avyukth Harish
· Lucas Gu · Tvisha Londhe · Tejas Prabhune
Overview
========
biocomposer is a graph-based orchestration framework in python for abstracting data flow
between bioinformatics tools in executable pipelines. You describe a pipeline as a graph
of tools; biocomposer runs each tool in its own container, generates the code that converts
one tool's output into the next tool's input, and derives the execution order for you.
Why it exists
-------------
Composing bioinformatics tools or orchestrating them together, normally forces the
user to reason about which tools to chain, how their formats and arguments line up,
how to invoke each one, and how to install dependencies. Ultimately, there is a lot
of complex python coding knowledge and syntax, along with many hours or weeks
of debugging, just to build a modular bioinformatics pipeline. Additionally, some
tools may be highly specialized and tailored to one specific task on very specific
inputs, which limits their ability to interoperate with other tools and models.
For this reason, we have developed biocomposer as an open-source effort to standardize
various bioinformatics tools, and modularize them so that they can be discovered by
other scientists, and be used in pipelines with other scientific tools to investigate
various hypotheses.
- **Inconsistent I/O.** One tool emits ``.pdb`` where the next expects ``.cif``;
argument names and file layouts differ between every pair of tools.
- **Idiosyncratic invocation.** Each tool has its own flags, entrypoints, and
environment, and many ship as under-documented repositories rather than packages.
- **No orchestration layer.** There is no common substrate for wiring tools
together, so even semantically compatible models (say, structure generation and
docking) are tedious to combine.
biocomposer removes the integration burden so effort goes into *workflow design*
rather than format conversion and command-line plumbing. It handles compatibility
between steps, performs LLM-guided data transformation across them, and runs the
result reproducibly, locally or on the cloud.
An Example Pipeline:
--------------------
This script aligns a FASTA file with Clustal Omega:
.. code-block:: python
from biocomposer import Graph
g = Graph()
seqs = g.add_input_node(sequences="/vol/inputs/family.fasta")
align = g.add_node("clustalo")
g.add_edge((seqs, align))
g.set_output_node(align)
result = g.execute()
The edge ``(seqs, align)`` says only *what connects to what*. The hand-off itself,
turning the input file into the exact arguments and format Clustal Omega expects, is
not written by hand: biocomposer reads both tools' manifests, inspects the data at
runtime, and generates the connector for that edge.
Package Specs
------------------
- **An explicit graph and generated glue, together.** The pipeline is a graph you
define in code, version, and re-run; the data conversion on each edge is generated
per edge from the tools' typed schemas and the data seen at runtime. The
structure is highly modular for the scientist, and enables quick tool-swapping and
controlled orchestration.
- **Generated connectors are ordinary, inspectable code.** Each edge's converter is
written out as a Python function and cached on disk, so the transformation between
two tools is a concrete artifact you can read and reuse, not a hidden runtime step.
- **The language model only writes the glue.** It is used to generate the connector
for an edge, from the two tools' schemas and the runtime data. It does not choose
the tools, run them, or interpret their results; those remain the user's graph and
the tools' own execution.
- **Tools are standardized behind a registry.** Each tool is a container image plus
a typed TOML manifest, resolved from the open `bv registry