Python SDK and Headless API (Huzaifa)

This page covers the interfaces Huzaifa built so experiments can run from Python, for example in Google Colab, without the 3D view. It extends the headless control described in Internal API (Ahmed) and adds a Python package, the RODENT SDK (rodent_sdk), that works with the same projects, experiments and saved runs as the website.

For researchers, the step-by-step guide is Python and Google Colab. This page is the developer reference.

The moving agent is still the statistical demonstration placeholder. Running it from Python does not make its behaviour biologically meaningful.

Why it exists

The client wanted to run experiments from an external Python environment without watching the rendering, and to have every run stored on the website for review and replay. It is also the path for Sprint 4, where trained models will control the rat.

We compared four designs:

Option	Decision
Python runs headless Godot itself; the website stores the results	Chosen. Fast, needs no server of our own, and suits model training.
Python remote-controls the Godot running in a browser tab	Rejected. Needs a browser open and rendering, which the client wanted to avoid.
The website's server runs Godot for each experiment	Rejected. The hosted site is Supabase plus static pages; there is no server to run Godot.
Rewrite the simulation in Python	Rejected. Two copies of the logic would drift apart.

How the pieces connect

Colab notebook
      |
      v
rodent_sdk  --TCP, JSON lines-->  headless Godot + rodent.pck
      |
      | HTTPS, signed in as the researcher
      v
Supabase (Auth, REST, row-level security)  <---  website Review

Three rules hold the design together:

1. One simulation. Python runs the same Godot code as the website, exported as one file (rodent.pck), so the same experiment and seed give the same run everywhere.
2. One source of truth. Projects, experiments and saved runs live in Supabase. Python is another way of working with them.
3. One set of permissions. The SDK uses the same Supabase REST API as the website, so row-level security and the scope trigger decide every change.

Headless API additions

The headless server (tools/headless_simulation.gd, protocol rodent-headless-v1) gained five commands. The command logic moved into src/core/headless_session.gd (HeadlessSession), so it can be tested without a network connection; the server is now a thin wrapper around it.

Command and fields	What it does
info	Engine version, format versions and the list of commands.
load_config {configuration}	Loads a full experiment object, validated first. If it is refused, problems lists why and the current experiment is kept.
set_door {id, open}	Opens or closes a door during a run. Recorded as a door_toggled intervention.
set_stimulus {id, active}	Switches a stimulus on or off during a run. Recorded as stimulus_toggled.
results {include_recording}	Run metadata and the per-step recording, in the same format as a replay saved by the website.

Every reply includes "api": "rodent-headless-v1", and failures carry error and problems. A live change is recorded only when the state actually changes, so closing a door that is already closed records nothing.

Muhammed's stimulus-effects work later added fields to info (observation_schema_version, metrics_schema_version, model_adapter_protocol), a metrics summary to results, and per-step stimulus fields to the recording.

act is unchanged: it moves the rat directly and does not respect walls. It suits a researcher's intervention, not model control. Sprint 4 needs movement that goes through the same collision rules as the agent.

Running from an exported pack

Colab does not have the repository, so the simulation is exported as a single pack with the RODENT Headless Linux preset (no export templates needed, about 800 KB):

godot --headless --path . --export-pack "RODENT Headless Linux" build/rodent.pck
godot --headless --main-pack build/rodent.pck --script res://tools/headless_simulation.gd -- --port=19091

The pack alone, in an empty folder, reproduces exactly the same run as the project folder.

Python client additions

python/rodent_client.py gained main_pack= (run from a pack instead of a repository folder), methods for each new command, and RodentCommandError, whose problems attribute holds validation messages.

from python.rodent_client import RodentClient

with RodentClient("godot", main_pack="build/rodent.pck") as sim:
    sim.load_config(configuration)
    sim.set_door("bridge_entry_gate", False)
    sim.step(5000)
    replay = sim.results()

The SDK (rodent_sdk)

The package lives in python/rodent_sdk/. Release builds contain the simulation pack, and Godot is downloaded automatically on first use and cached. It uses only the Python standard library; pandas and matplotlib are optional and already present in Colab.

import rodent_sdk as rodent

# Offline
run = rodent.run("complex_habitat", steps=5000, seed=42)

# With the website
client = rodent.login()                         # reads the login from Colab Secrets
exp = client.project("Anxiety study").experiment("Baseline")
exp.doors["bridge_entry_gate"].open = False
exp.save()                                      # checked against the researcher's modules
run = exp.run(steps=5000, changes={2500: {"doors": {"bridge_entry_gate": True}}})
run.summary()
run.save()                                      # appears under Review on the website

Main calls

Call	What it does
rodent.templates()	The six arena names.
rodent.experiment(template, seed, name)	An editable offline experiment.
rodent.run(template, steps, seed, changes)	A one-line offline run.
rodent.login()	Signs in: arguments, then environment variables, then Colab Secrets, then a prompt.
client.projects(), client.project(name_or_id)	Projects the researcher belongs to. An ambiguous name is refused with a list of ids.
client.create_project(name, template)	A new project with the caller as lead.
project.experiment(name_or_id), project.create_experiment(name, template)	Experiments. Creating from a template is for project leads and instructors.
exp.doors[id].open, exp.stimuli[id].intensity, exp.treatment.condition	Readable editing of the experiment JSON. Setting a door updates every field the website uses.
exp.save(), exp.save_as_new(name), exp.publish()	Store edits, copy into a new draft, or freeze a version.
exp.run(steps, changes)	Runs headlessly, with optional live door and stimulus changes at chosen steps.
run.summary(), run.to_dataframe(), run.plot()	Time, entries and first entry per region; every step as a table; the trajectory.
run.save()	Saves the run to the website's Review.

The full list, with every field, is in docs/PYTHON_SDK.md in the application repository.

Settings

Variable	Purpose
RODENT_EMAIL, RODENT_PASSWORD	Website login. In Colab, store them in Secrets so they never appear in a notebook.
RODENT_SUPABASE_URL, RODENT_SUPABASE_KEY	Website address and publishable key. Release builds already contain them.
RODENT_GODOT, RODENT_PACK	Use an existing Godot or a different pack.
RODENT_CACHE	Where downloads are kept.

How a run reaches Review

run.save() inserts a row into replays with the replay object, the seed and the simulated duration, in exactly the format the website's replay player reads. It also tags the replay with the SDK and Godot versions that produced it.

The website replays a run on the experiment's saved arena. If the researcher changed the experiment in Python without saving, run.save() refuses, because the replay would show the rat moving through the wrong walls.

Safety rules the SDK relies on

The SDK never bypasses the database. What it can do is decided by the rules described in Database Schema and Deployment:

• A change outside the researcher's modules is rejected by the enforce_experiment_scope trigger; the SDK names the modules they may change.
• Creating an experiment from a template is limited to project leads and instructors. Everyone else copies one with clone_experiment, which checks modules.
• Saves carry the revision number, so if someone saved first, the save is refused instead of overwriting their work.
• Published versions are frozen; researchers continue from them with save_as_new.

Releases

tools/build_sdk.py builds a release:

python tools/build_sdk.py --godot <godot> --supabase-url <url> --supabase-key <publishable key>

It exports the pack, puts it inside the package with BUILD_INFO.json (SDK version, Godot version, git commit, pack checksum and website address), builds dist/rodent_sdk-<version>-py3-none-any.whl, and checks the pack inside against the export. It refuses secret or service_role keys.

Releases are uploaded to the public rodent-sdk Supabase Storage bucket, and notebooks install one with pip install <URL>. Two rules keep results reproducible:

• Never overwrite or delete a version. A notebook pinned to a version keeps getting the same simulation.
• Bump the version when behaviour changes. Version 0.3.0 followed Muhammed's stimulus-effects merge, because the same seed now produces a different run.

Evidence

Check	Result
Headless protocol test and end-to-end socket test	5,000 steps in about 1.2 s on Linux and 2.6 s on Windows.
The exported pack run from an empty folder	Same run as the project folder.
Colab	Installed from the bucket, downloaded Godot, and produced the same per-region times as local runs to one decimal place.
SDK unit tests (python/tests)	43 tests with Godot and the network faked.
SDK against real PostgREST on a copy of the live schema	Over 30 checks across lead, editor, light-only researcher, viewer and outsider accounts.
Real website	A run saved from Colab replayed correctly under Review.

See Automated testing and CI Pipeline for where each check runs.

Known limits

• Runs of about 20,000 steps produce replays over 20 MB, which the website may reject. Moving large runs to Supabase Storage (replays.storage_path) is planned.
• Colab clears its machine between sessions, so Godot (about 70 MB) is downloaded again in each new session.
• Only doors and stimuli can change during a run.
• act ignores walls (see above).