A lab server: many cores, large sites, batches

Up to 16.5 the app was built for one machine, a public server with 2 cores and 3 GB. Version 17.0 runs the same code on a private lab server, for example 22 cores and 128 GB on the lab's network:

• requests in parallel: one per core;
• larger sites and molecules;
• batch surveys of every ligand in many structures.

One setting chooses which: the runtime profile.

#Two profiles

--profile auto picks lab on a machine with at least 8 cores and 32 GB, else vps. Any value can be set on the command line, and a flag beats the profile (python3 server.py --help).

#Why worker processes

Python runs one thread of Python code at a time: the global interpreter lock. Nearly all of the orbital layer is Python. So one server process uses about one core, however many threads it has, and twenty visitors at once would each get a twentieth of a core.

On the lab profile the HTTP server (the front process) only reads requests and sends answers. /api/site, /api/model, /api/pi-model, the SVG render and batches go to worker processes (compute_pool.py), each with its own caches:

• Which worker. A structure is identified by its SHA-256 and ranks the workers in a fixed order (rendezvous hashing, Thaler & Ravishankar 1998). It goes to its first or second choice when one is free, else to any free worker. A SMILES does the same for /api/model. So a structure is usually parsed once and stays parsed where its requests go.
• The file travels only when needed. The front sends a structure's bytes only to a worker that does not hold it yet.
• The answer comes back finished. The worker returns the answer already compressed, so the front only moves bytes.
• A worker that dies (out of memory, a crash in a library) is replaced at once. Its request gets HTTP 500 worker_restarted; it is not retried, since it would likely kill the next worker too. A job that runs longer than 15 minutes is stopped the same way (HTTP 504 worker_timeout).
• When every queue is full the server answers 503 busy with Retry-After: 5.

On Linux the workers are forked from a fork server that has imported the app (and RDKit) once, so they share those pages.

#Install

As on the public server (DEPLOYMENT.md): Python 3.10 or newer, a virtual environment with requirements.txt (RDKit, NumPy; py2opsin and Java for names), and the browser libraries.

cd /opt
sudo mkdir quantum-orbitals && sudo chown $USER quantum-orbitals
unzip -q ~/quantum_orbitals_integrated_server_v17_0.zip -d quantum-orbitals
cd quantum-orbitals
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
./install_3dmoljs.sh && ./install_rdkitjs.sh && ./install_molstar.sh && ./install_swagger_ui.sh && ./install_fonts.sh
.venv/bin/python -m unittest discover -s tests -t .        # must end with OK and no skips

The PDB's component dictionary, locally (optional, recommended). With it, every ligand's dictionary route works without asking data.rcsb.org, also in batches of thousands:

sudo mkdir -p /srv/ccd && sudo chown $USER /srv/ccd
curl -o /srv/ccd/components.cif.gz https://files.wwpdb.org/pub/pdb/refdata/chem_comp/components.cif.gz

The first start unpacks it next to the .gz (about 400 MB) and writes an index (components.cif.qo-index.json), which takes a few seconds; later starts read the index. The workers map the same file, so it sits in memory once. The wwPDB updates the dictionary weekly; fetch it again when new ligands matter, and restart.

#Start

Try it by hand first:

.venv/bin/python server.py --profile lab --host 0.0.0.0 --port 8000 --ccd-file /srv/ccd/components.cif.gz

The log says what it chose (your counts and times will differ):

Local component dictionary: <n> components from /srv/ccd/components.cif.gz (<t> s)
Worker pool: 20 processes (forkserver), started in <t> s
Profile: lab · machine 22 cores, <m> GB · workers 20 · batch on
Atom limit: 250 (SMILES) · 12000 with hydrogens (protein sites) · structure files up to 256 MB · pockets up to 12 Å

Then as a service: copy quantum-orbitals-lab.service.example to /etc/systemd/system/quantum-orbitals-lab.service, adjust the paths and the user, and

sudo systemctl daemon-reload && sudo systemctl enable --now quantum-orbitals-lab
journalctl -u quantum-orbitals-lab -n 20 --no-pager

#On the lab's network only

The lab profile has no login and no rate limit: everyone who reaches the port can start a 16 s computation, or a batch of 2 000. Keep it off the internet:

• bind it to the lab's interface (--host 192.168.1.20) or put it behind the firewall:

  bash

  sudo ufw allow from 192.168.1.0/24 to any port 8000 proto tcp
  sudo ufw deny 8000/tcp

• do not forward the port, and do not put the lab server behind the public nginx.

/api/render-svg still needs its bearer token (QUANTUM_ORBITALS_API_TOKEN), as on the public server.

#Check it

curl -s localhost:8000/api/site-health | python3 -m json.tool | grep -E 'profile|workers|batch|ccd_local'
curl -s 'localhost:8000/api/server-status?ping=1' | python3 -m json.tool | less     # every worker: memory, caches, jobs

#Batches

From the page. /sites shows "Survey every ligand in this file" under the ligand list. In "Find structures by ligand", each component's binding sites get a button "Survey these N binding sites". The survey lists each ligand: which route gave its SMILES, whether the others agree, its links, metals and pocket, and with "with π orbitals" its π systems. A summary per component comes first. A row opens its ligand; the whole survey downloads as CSV or JSON.

From a script, POST /api/batch:

curl -s localhost:8000/api/batch -H 'Content-Type: application/json' -d '{
  "jobs": [{"example": "19hc"}, {"gl1x": "0xd58b01f6c18086e5202cdc5d7ad3e41790360102/7", "ligands": ["HEM:A:301"]}],
  "orbitals": true, "pocket_radius": 6}' | python3 -m json.tool | less

A job names a structure as /api/site does: structure_ref, gl1x, example, or the file itself. It may also name the ligands (ligands); without them every ligand, sugar, additive and ion is surveyed. The answer has rows (one per ligand), consensus (one per component) and errors (jobs whose structure could not be read).

A folder of files, without the server, on every core:

.venv/bin/python tools/lab/batch_ligands.py /data/pdb_subset --ccd /srv/ccd/components.cif.gz --orbitals --out survey
# → survey.jsonl, survey.csv, survey_consensus.json

Molecules, not strings. The summary counts distinct molecules: the same heavy atoms with the same number of hydrogens, wherever those sit. A haem read with its two N–H on different nitrogens is one molecule, as in InChI's mobile-H layer.

#Sending a file once

Every /api/site answer carries the file's SHA-256 (source.sha256). Later requests may send "structure_ref": "<sha256>" instead of the file; the page does this by itself. The front process keeps recent files: 96 MB on vps, 3 % of the memory on lab (3.9 GB). When it no longer holds one it answers 400 structure_unknown, and the file is sent once more.

#Measure it

The numbers in the experiment come from a 2-core sandbox. Four things can only be measured on the lab machine itself: 20 requests at once, memory with RDKit, a batch on 20 workers, and the real dictionary. With the service stopped (or on other ports):

.venv/bin/python server.py --profile lab --port 8100 --ccd-file /srv/ccd/components.cif.gz &
.venv/bin/python server.py --profile lab --workers 0 --port 8101 &
.venv/bin/python tools/lab/bench.py --url http://127.0.0.1:8100 --compare http://127.0.0.1:8101 \
        --k 1,2,4,8,16,20 --files /data/pdb_subset --out bench_lab.json
.venv/bin/python tools/lab/scale_site.py --json scale_lab.json
kill %1 %2

bench_lab.json and scale_lab.json hold everything the forecast is scored on. If 20 requests at once take more than twice the time of one, the forecast says what to do: run several plain server processes behind nginx instead of the pool.

#Tuning

The cache sizes follow from the memory (runtime_profile.py): a quarter of it for parsed structures and an eighth for computed sites, split over the workers, and 3 % for raw files in the front process. On 128 GB that is 50 GB when every cache is full.

#What does not change

• The public server keeps the vps profile, and nothing about it changes. Its limits, its two compute slots and its single process are as in 16.5.1.
• The chemistry is the same: a worker returns the same answer as the single process, apart from the timing fields.
• A site of 12 000 atoms is a lot for a browser to draw. The page opens it on its quick view (the donor pairs, the π HOMO); "every orbital of the fragment" may be more than Mol* will take, and the page then draws in 3Dmol.js only.