*** UNDER CONSTRUCTION *** — please excuse the dust while we wire up more CV goodies

pyewacket

Computer vision for robotics · VIO + VPS · MAVLink to PX4 / ArduPilot · Phase 0 scaffold online

You are visitor number 000042 | Last updated: November 7, 2026

Welcome. This page explains the research stack in pyewacket: computer vision for robotics and drone control — one navigation product. High-rate visual-inertial odometry (VIO) flies the vehicle. Low-rate visual position sensing (VPS) only recenters that pose onto a site map. The autopilot receives one fused ODOMETRY stream. Language models search and explain; they do not fly. Advisory agents forecast degeneracy, prior VPS retrieval, curate survey dossiers, gate magnetics, and narrate forensics — without a second pose stream.

Docs: vision plan · VPS · agents · coordinate frames · README

1. What this is
A computer-vision research stack for robotics and drone control — visual pose, map-frame fixes, and companion fusion into the autopilot. Complementary layers, different drift characters:

Layer	Role	Status
VIO	High-rate 6-DOF pose + body velocity. Drifts over distance/time.	Running (FakeVIO in SITL)
VPS	~1 Hz map-frame absolute fix. Pulls drifting VIO back.	Running (FakeVPS + 4-DoF fusion)
Magnetic	Yaw aiding + anomaly relocalization near ferrous structures.	Later (Phase 2)
VLM / agents	Advisory: degeneracy forecast, place prior, survey dossier, mag gate, forensics. Agents advise, geometry flies.	Scaffold + FakePerception (Phase 3.0)

2. One pose — who is allowed to say what
The flight controller is not told there are two estimators. It sees one `ODOMETRY` on UDP `14540`, component id 197, frame `MAV_FRAME_LOCAL_FRD`. Two vision streams make the EKF fight itself.

Quantity	Owner	Rate	Why
Position x, y, z (local FRD)	Fusion	30–50 Hz	`p_map = Rz(yaw) * p_vio + t`. Between fixes, VIO motion runs through a held alignment.
Body velocity vx, vy, vz	VIO only	30–50 Hz	A 1 Hz match has no trustworthy velocity.
Roll and pitch	VIO only	30–50 Hz	IMU gravity beats a sparse visual match.
Yaw	VIO by default	30–50 Hz	`yaw_fusion` is off until SITL yaw tests pass. A bad yaw match toilet-bowls the vehicle.
`ODOMETRY.quality`	VIO health	with pose	0–100, or −1 if VIO failed. Stale VPS must not set −1. The FCU uses this (`EKF2_EV_QMIN`).
Position covariance	Fusion	with pose	Grows ~1% of distance since the last accepted fix. ×4 when VPS is stale.
`reset_counter`	VIO + jumps	event	A step over ~1 m or ~10° increments once so the EKF sees a declared reset, not a silent teleport.

3. Alignment (4-DoF, not 6)

State is (tx, ty, tz, yaw) only: p_published = Rz(yaw) * p_vio + (tx, ty, tz). Translation slides the VIO origin onto the map. Yaw would rotate the horizontal plane; it stays off for now. Roll and pitch are not in the alignment — a visual match that disagrees with the IMU on tilt is usually a bad match.

VIO is a tape measure that slowly stretches. VPS does not replace the tape. About once a second it says where you actually are, and the filter moves the tape's zero. Between those words, the tape still measures.

Each 50 Hz tick, and the publisher never waits for a match:

Queue depth 1. A newer fix replaces an unprocessed one.
Mahalanobis gate 3. An implausible fix is rejected. FakeVPS outlier must fail this.
Small corrections slew: 0.3 m/s horizontal, 0.2 m/s vertical. The EKF sees a pull, not a teleport.
A jump over ~1 m or ~10° increments reset_counter and applies immediately.
No accepted fix for 2 s: status stale, covariance ×4, quality unchanged. The vehicle keeps flying on VIO.
A ~10 s dropout must not spam reset_counter. Recovery is another gated update.
VIO quality = -1 stops the pose. A map match cannot resurrect dead odometry.

Code: src/pyewacket_vps/fusion.py, src/pyewacket_vps/alignment.py.

4. Navigation state (derived, not chosen)
The pilot does not pick a VIO mode or a VPS mode. The dashboard should lead with this word, plus VIO quality, VPS age, and last residual.

State	Condition	Meaning
Fused	Accepted fix, age under 2 s	On the site map. Tight covariance.
VIO-only	No map loaded, or VPS off	Relative hover. Drift will grow.
VPS-stale	Map loaded, no accepted fix for >2 s	Still flying. Do not trust the map pin.
VPS-rejected	Fix arrived and failed the gate	Matcher ignored. Last good alignment held.
Degraded	VIO quality low or −1	Land. Vision pose stops.

5. Observability UI (browser + TUI)
Not a GCS. QGroundControl arms and flies. Pyewacket only watches the companion stack at 10 Hz and stresses FakeVIO/FakeVPS in SITL. Browser and curses TUI are co-equal until field trials pick a default.

Surface	Start	Role
Browser	`python -m pyewacket_ui` `http://127.0.0.1:8080`	Canvas XY trails, mode dropdowns, WebSocket telemetry
TUI	`python -m pyewacket_tui`	ASCII plot, frame-check wizard, screens 1–6 (`q` quit)
UDP	`:9101` snap (web), `:9103` (TUI), `:9102` modes, `:9104` agent hints	Same `TelemetrySnapshot` JSON everywhere (+ optional `advisory`)

Hero label (every screen)

Lead with derived Navigation state from section 4 — not FakeVIO/FakeVPS mode names. Subtitle: odom_hz, VIO quality, vps.age_ms, last residual. SITL mode pickers stay below the hero (debug).

Plan: docs/pyewacket-vio-vision-plan.md (UI chapter). Hero wired in app.js, pyewacket_tui/render.py, pyewacket_bridge/navigation.py.

Screen	Browser	TUI keys
Overview + XY	Shipped	`1` / F1
Frame check wizard	Shipped (tab + API)	`2`
FakeVIO / FakeVPS modes	Shipped	`3`
VPS detail	Pills	`4`
Mag stub / Agents	Advisory strip	`5` / `6`

6. Agents advise, geometry flies
Language searches and explains. The 50 Hz fusion path never waits on a model. Agents publish hints on UDP `:9104` at ~2 Hz; offline CLI tools write dossiers and forensics briefs. Spec: docs/agents.md.

Agent	Problem	Rate	Output (advisory)
DF	VIO dies before `quality` drops	~2 Hz	`cov_scale`, forecast seconds — never writes quality
SVP	Repeating towers confuse retrieval	~2 Hz	Candidate keyframe ids + alias risk — geometry still verifies
SDA	0.5 m loop gate is numeric only	Offline	`dossier.md` / `dossier.json` — does not arm
MAB	Outdoor mag is noise except near steel	~2 Hz	`mag.gated` fail-closed — never a pose
NFC	Hard to reconstruct nav-state timelines	Offline	`brief.json` via same `derive_navigation_state`

7. Agent hints (not a pose)
Port `:9104`. Age over 2 s → identity (no inflation, empty prior, mag off). `ODOMETRY.quality` stays VIO health. No sixth navigation state.

Field	Range	Fusion effect	Forbidden
`cov_scale`	1–4	Multiply position variance only	Attitude, velocity, quality
`alias_risk`	0–1; >0.6 special	Skip cov tighten on accept	Forcing reject
`candidate_keyframe_ids`	0–8 ids	Narrow retrieval; empty = full map	Inventing a VpsFix from names
`mag.gated`	bool, default false	Telemetry until Phase 2 mag filter	Publishing mag as vision pose

8. Degeneracy forecast
Late `quality` is the bug: open sky, sun, or blur can kill VIO seconds before the number drops. The forecaster sets `score_now`, `forecast_sec` (≤30), and `cov_scale` (cap 4). Acceptance: warn at least ~8 s before a scripted quality drop; healthy / `perfect` mode must not inflate above 1.2.

9. Place prior on a site graph
Utility corridors repeat steel structures. `maps/<site>/vN/site_graph.json` holds nodes and edges. Agents may only emit node ids that exist in that file. Geometry verifies. If the prior fails, one full-map fallback runs and the UI can show “geometry overrode place prior.” Inliers win.

10. Survey dossier
Beside the map: `survey_report.json` (numeric 0.5 m loop gate), `dossier.md`, `dossier.json`. The residual is copied, never invented by the LLM. CLI still exits 0 if a narrative check fails. The flight stack does not read the dossier to arm. `python -m pyewacket_agents dossier --map maps/demo/v1`

11. Magnetic gate
Outdoor mag is useless in an open field. `gated` is fail-closed false. Classes: lattice tower, conductor, bridge, none. Phase 2 may fuse mag only when gate, confidence ≥0.5, and near-site-node agree. A hallucinated “ferrous” in an open field must not enable mag.

12. Forensics copilot
Replay telemetry JSONL through the same `derive_navigation_state` function as the live UI. `brief.json` is deterministic without an API key. Optional LLM prose must quote those state labels. No param writes. `python -m pyewacket_agents brief --jsonl … --out flights/…`

13. FakePerception (SITL)
No GPU required. Quality drops are scripted in FakeVIO or replay, not by the agent. `./scripts/agents.sh degeneracy_soon`

Mode	Behavior
`off`	No publisher
`perfect`	Healthy DF; correct top-2 keyframe ids; mag only near ferrous node
`delayed`	Hints lagged ~500 ms
`degeneracy_soon`	After 5 s: cov×2 and forecast 8 s (quality still 100 until scripted drop)
`wrong_place`	Prior points at neighbor tower
`dropout`	Stop publishing ~10 s
`hallucinate`	Confident wrong place; tests must show geometry still wins

14. What agents must not do
No arm / disarm / mode / param writes No second MAVLink vision stream No writes to `ODOMETRY.quality` No Mahalanobis override (inliers win) No camera-rate LLM on the vehicle No cloud agent on the 50 Hz path Prompt packs: utility corridor (lead), campus, canopy — `config/agents/*.yaml`.

15. Survey, localize, fly
Three jobs of one product, not three products. QGroundControl still arms and changes flight mode.

Job	What it does
Survey	Slow flight with the same camera used later. Keyframe poses come from that pass's VIO, never from `LOCAL_POSITION_NED` (that is the EKF you are feeding). Accept the map only if a short loop closes within 0.5 m.
Localize	About 1 Hz against the map. Output is a `VpsFix` (time, xyz, variance, inliers, status), not a MAVLink message. Off the mapped area: status stale, and no invented fixes.
Fly	50 Hz fusion into one `ODOMETRY`. No map means state VIO-only.

Map folder (customer-owned)

maps/<site_id>/vN/
  origin.json          local FRD at survey start
  keyframes/           pose + descriptors
  survey_report.json   loop residual, accepted, camera, light note

C1 session: survey this morning, fly this afternoon, same camera, similar light, 0.5 m gate. C2 persistent: the customer keeps the folder and must re-test after season or lighting change; if inliers collapse, fail closed to VIO-only. C3 cloud: not v1. Same file format if it ever exists.

16. When it fails

What happened	State	Aircraft
Textureless sky, VIO quality −1	Degraded	Stop the vision pose. Pilot lands.
Flew off the mapped pad	VPS-stale	Keep the last alignment. Covariance grows. Position hold gets softer.
False match (repeated window)	VPS-rejected	Ignored. No reset.
Survey loop residual 1.2 m	No map	Do not arm a mapped mission. Fly VIO-only or re-survey.
First good fix after a long drift, error ~3 m	Jump + `reset_counter`	EKF takes a declared vision reset. Expect a correction, not a flyaway, if frames are right.

17. GPS (flight-controller source, not a third pose)
Pyewacket still publishes one vision odometry. GPS is not fused inside `VioVpsFusion`. Which profile is the headline is still an open decision; the page records the options.

Profile	Params	Story
A1 denied	`EKF2_GPS_CTRL=0`	Under a bridge, indoors, or jammed. The map is the only absolute frame. This is the demo.
A2 blended	GPS outside, then switch to A1 on site	Enterprise default: GPS in transit, this stack inside the site. Simultaneous fusion is later, after a handoff log. GPS and vision must not both yank horizontal position.
A3 origin only	GPS sets a global origin once	Weak denied story. Not the headline.

18. What this is not

Not a separate VIO product and a separate VPS product. Map use is optional; the pose stream is one.
Not two MAVLink vision streams into the autopilot.
Not a second GCS. QGroundControl arms. This page's stack only estimates and displays.
Not a cloud map service. The customer stores the map folder.
Not yaw-from-VPS until a SITL matrix (identity, +90°, slew, jump) stays stable in Position mode.
Not a VLM override. If a language-model landmark and a geometric match disagree, geometric inliers win.

19. Path to the autopilot

Gazebo ground truth (SITL) or a future camera feed → FakeVIO / VIO → optional 4-DoF VPS alignment → one ODOMETRY message at 30–50 Hz → pyewacket_bridge → PX4 EKF2 or ArduPilot EKF3.

QGroundControl arms and flies. The browser UI (:8080) and curses TUI only watch a 10 Hz telemetry snapshot (UDP :9101 web, :9103 TUI). FakeVIO / FakeVPS mode commands go to UDP :9102.

Flight control stays in QGC. Observability surfaces never arm or publish ODOMETRY.

20. Visual-Inertial Odometry (VIO)

Primary relative pose: high-rate position, attitude, and body-FRD velocity. Roll and pitch always come from VIO. The published field ODOMETRY.quality means VIO health (0–100; −1 = failed).

Phase 0 uses FakeVIO: synthetic odometry from Gazebo ground truth (never from LOCAL_POSITION_NED), so you can stress the EKF without a camera.

FakeVIO mode	Behavior
`perfect`	Clean GT pose and velocity
`delayed`	Injected latency
`noisy`	Sensor-like noise on pose / velocity
`degraded`	Quality drop / reliability stress
`wrong_frame`	Frame-axis regression (must fail the axis walk)

21. Visual Position Sensing (VPS)

Low-rate absolute position fix in a local map frame. Content: map xyz, optional yaw. No velocity. Companion fusion keeps publishing one aligned ODOMETRY stream; fusion never blocks on a match.

Alignment is 4-DoF (map-from-VIO): p_map = Rz(yaw) * p_vio + t. Roll and pitch stay from VIO. Map origin is local only (SITL: Gazebo world → FRD; hardware: survey start). No GPS origin, no GPS_INPUT.

Stale VPS inflates pos_covariance and sets status stale. It does not set quality = -1 (that flag is reserved for VIO health).

Full contract: docs/vps.md

FakeVPS mode	Behavior
`perfect`	1 Hz GT + centimeter noise
`delayed`	200–500 ms latency
`outlier`	Multi-meter false match (must be rejected)
`dropout`	Fixes stop ~10 s, then recover without `reset_counter` spam

22. Coordinate frames
MAVLink `ODOMETRY` uses `MAV_FRAME_LOCAL_FRD`. Velocities in the message are body FRD, not world NED.

Frame	Axes
NED / local FRD	X north, Y east, Z down
Body FRD	X forward, Y right, Z down
ROS (later)	ENU world, FLU `base_link` — converted by `pyewacket_frames`

Axis walk (body aligned to world): forward → x up; right → y up; lift → z decreases. See docs/coordinate-frames.md and pyewacket-frames check.

23. PX4, ArduPilot, and MAVLink

Both autopilots accept the same companion message: ODOMETRY at 30–50 Hz with FRD body frame, position/attitude covariance, quality, and reset_counter.

PX4 EKF2 — params in config/px4_ekf2_vio.params (EKF2_EV_CTRL 15, EKF2_GPS_CTRL 0, height ref = vision).
ArduPilot EKF3 — draft for Phase 4 in config/ardupilot_ekf3_vio.params (VISO_TYPE, EK3_SRC1_*).
SITL — simulation/px4_sitl/README.md; launch helpers in scripts/README.md.

24. ROS 2 (planned)

Not in the running packages yet. There are no launch files or ROS nodes in this Phase 0 scaffold.

The design plan’s intent is PX4 SITL plus ROS 2 for rapid iteration, while this Python MAVLink bridge stays autopilot-agnostic (MAVROS2 on the ArduPilot side). Frame helpers already speak both worlds: ENU/FLU ↔ NED/FRD via pyewacket_frames.

See the ROS / autopilot notes in docs/pyewacket-vio-vision-plan.md.

25. Packages you can run

Install and test: pip install -e ".[dev]" then pytest tests/. Stack helper: ./scripts/run-stack.sh noisy perfect (FakeVIO + UI + TUI). Agents: ./scripts/agents.sh perfect. Or by hand:

python -m pyewacket_fakevio --mode perfect --vps perfect
python -m pyewacket_agents run --fake perfect
python -m pyewacket_ui → http://127.0.0.1:8080
python -m pyewacket_tui

Module	Role
`pyewacket_frames`	ENU/FLU ↔ NED/FRD transforms
`pyewacket_bridge`	MAVLink `ODOMETRY` publisher + telemetry fan-out
`pyewacket_fakevio`	Synthetic VIO from Gazebo GT
`pyewacket_vps`	Absolute map-frame fixes + 4-DoF fusion into one `ODOMETRY`
`pyewacket_perception`	FakePerception + VLM protocol (advisory)
`pyewacket_agents`	AgentHint bus, DF/SVP/MAB workers, dossier + forensics CLI
`pyewacket_ui`	Local FastAPI + WebSocket dashboard
`pyewacket_tui`	Full-screen curses console (subscriber; no arm / no `ODOMETRY`)

Phase 0 scaffold · local research stack · flight control stays in QGroundControl
Back to portfolio