Built MFG · System overview
Turning a single photo into a millimetre-accurate garment inspection
Built MFG measures and quality-checks flat soft goods — shirts, labels, cuffs, collars — from one ordinary camera photo. The trick is a printed ChArUco board laid on the table next to the garment. Because the board's real-world geometry is known down to the millimetre, the software can turn every pixel in the photo back into a real position on the table. A text-prompted AI model outlines each garment part; those outlines become real-world shapes; and each shape is compared against an approved golden reference to produce a grade an operator can act on immediately.
The core idea
Why a checkerboard solves the measurement problem
A photo on its own carries no scale. The same shirt looks bigger up close and smaller far away, and a tilted camera skews every distance. You cannot measure from raw pixels.
The ChArUco board fixes all of this at once. It is a printed grid of 5 × 7 squares at a known 30 mm pitch, with ArUco markers tucked inside the white squares. The markers carry unique IDs so the software knows exactly which corner is which, and the checkerboard intersections can be located to sub-pixel precision. From the way that known grid appears distorted in the photo, the software recovers the camera's position and angle — and from there, a mapping that flattens the whole image onto the table plane. Anything lying flat on that plane can now be measured in real millimetres, regardless of how the photo was taken.
How it's built
Two packages, one pipeline
the engine
charuco_table
The pure computer-vision library: board detection, camera calibration, pose, rectification, polygon comparison and the quality classifier. Built on OpenCV, NumPy and Shapely — deliberately no PyTorch, so it stays fast and portable. It talks to the AI model only over the network.
the product
garment_guard
The web app that wraps the engine: a FastAPI backend with a Postgres database and a phone-friendly browser UI. It handles users, camera profiles, garment definitions, production runs, and the live capture-and-grade inspection flow.
Two supporting pieces round it out: sam3_service, a self-hosted AI segmentation server for offline/dev use (the only part that needs a GPU and PyTorch), and the production path that calls the same model hosted on fal.ai.
The measurement pipeline
From printed board to graded result, step by step
Every inspection runs the same eight stages. The first three are one-time setup; the rest run on each photo.
SETUP · board.py
Print the board
Generate and print the reference target — a 5 × 7 ChArUco board (OpenCV's DICT_6X6_250 markers) with 30 mm squares. The square size is the real-world ruler; it must match the printed sheet exactly. Tape it flat to the inspection table.
SETUP · detect.py
Detect the board
In any photo, OpenCV's detector finds the ArUco markers and checkerboard corners, then matches each detected corner to its known 3D position on the board. The result is a set of 2D-pixel ↔ 3D-board point pairs — the raw material for everything geometric that follows. At least four matched corners are required, or the frame is rejected.
SETUP · calibrate.py
Calibrate the camera
Every camera has its own focal length and lens distortion. Feeding the system 10+ board photos from varied angles and distances lets it solve for the camera's intrinsics — the focal lengths and optical centre (the K matrix) plus five lens-distortion coefficients. This is a per-camera, one-time step, saved as a reusable camera profile.
PER PHOTO · pose.py
Find where the board sits
With the camera calibrated, a single photo is enough to solve the board's full 3D position and orientation (a Perspective-n-Point solve, producing a rotation and translation). A by-product is the camera's tilt relative to straight-down — the app uses this to insist on a roughly overhead shot before it lets the operator capture, which keeps measurements repeatable.
PER PHOTO · segment.py · SAM3
Trace the garment with AI
Each garment part is described in plain English — "left sleeve", "black woven label", "size tag". The SAM3 segmentation model takes the photo and a text prompt and returns a pixel mask of that part. The mask is traced to its outline and simplified to a clean polygon (Douglas–Peucker, ~0.5 px tolerance) so only meaningful vertices remain.
PER PHOTO · rectify.py
Convert pixels to millimetres
This is where pixels become measurements. Each polygon is first undistorted (removing lens curvature using the calibration), then projected through a homography derived from the board pose, landing every vertex on the flat board plane in real metres. The outcome: a garment outline you can take a tape measure to — area, perimeter, edge lengths, all in true units, independent of how the photo was framed.
PER PHOTO · compare.py
Compare to the golden shape
The measured outline is matched against the approved golden polygon. Two families of metrics are computed. Shape-only metrics — area, perimeter, diameter — don't care where the part sits. Fit metrics first rigidly align the shapes, then measure agreement: IoU (overlap), Hausdorff (worst-case edge gap, 95th percentile) and ASSD (average edge gap). A tolerance envelope around the golden allows intentional placement slack.
PER PHOTO · quality.py
Grade it
Finally those metrics are rolled into one of six operator-facing grades — not detected, wrong garment, low, misplaced, medium, high — by an ordered set of rules where the first match wins. An inspection of several parts takes the worst grade among them.
See exactly how each grade is computedHow grading works
The six grades, in detail
The classifier walks an ordered list of rules and the first one that matches wins — so the order below is the order checks actually run in, not best-to-worst. Each rule reads a handful of geometry signals from the comparison step. Thresholds live in the rules themselves (charuco_table/quality.py); this is a mirror of them.
Nothing usable came back — the segmenter errored or returned no polygon for the piece. An inspection with no segments at all also lands here. Distinct from wrong garment, where a shape was produced.
error is set OR post_rectification_polygon is missingA polygon came back, but its shape doesn't match the golden at all — the wrong item (or nothing garment-like) was on the table.
< 0.50best aligned IoU = max(
iou_free, iou)The catch-all. The shape clears the wrong-garment floor but isn't a clean enough free-fit to be merely "misplaced" and doesn't pass the envelope test. A recognizable but poorly-fitting piece.
Right shape, wrong position. The piece fits its golden well by its own rigid alignment (it is the right part), but its boundary leaves the allowed placement envelope.
iou_free ≥ 0.80 AND• with an envelope:
edge_fit_frac < 0.95• no envelope:
iou < 0.50The boundary stays inside the placement envelope, but the shape isn't tight enough to clear the strict distance bars that high requires. A passing piece with looser edges.
edge_fit_frac ≥ 0.95Inside the envelope and a tight boundary match — both worst-case and average edge deviation within tolerance. Right shape, right place, made to spec.
edge_fit_frac ≥ 0.95 AND hausdorff_mm < 10 AND assd_mm < 10How a piece is graded
Rules run top to bottom; the first match assigns the grade.
Errored or no polygon? → stop here.
Best aligned IoU below 0.50? → stop here.
Strong free-fit (≥ 0.80) but boundary leaves the envelope? → stop here.
Inside envelope and Hausdorff & ASSD both under 10 mm? → stop here.
Inside the envelope (edge fit ≥ 0.95) but not strict enough for high? → stop here.
Nothing above matched → low (fallthrough).
The signals each rule reads
iou
Placement IoU in the section frame — overlap with the golden where the section actually landed. The placement signal for top-level prompts.
iou_free
Free-fit IoU — the piece aligned to its golden by its own rigid transform, ignoring where it landed. Pure shape agreement. Carried only by child parts; the grader prefers it when available.
edge_fit_frac
Fraction of the boundary that stays inside the placement envelope — the deliberate tolerance ribbon around the golden. The 0.95 bar separates "inside the envelope" from "misplaced".
hausdorff_mm
Worst-case boundary deviation in millimetres — the single farthest gap between the piece edge and the golden edge. Under 10 mm required for high.
assd_mm
Average symmetric surface distance — the typical edge-to-edge gap averaged around the boundary. Under 10 mm required for high.
The inspection product
What the app does with all this
garment_guard turns the pipeline into a day-to-day QC tool for a factory floor. Its job is to catch problems early — a part sewn to the wrong size, a label placed off-position, a missing tag, dimensions drifting out of spec — before a unit moves down the line.
Two kinds of people use it. Admins define what "correct" means for each garment. Operators (the sewists at the table) scan real units against those definitions and get an instant verdict. Everyone signs in with their company ERP account.
Admin workflow
Onboarding a garment
Before a garment can be inspected, an admin teaches the system its ideal form on the /onboard page:
1Create the garment — a name plus any SKUs. Variants that share one measurement spec share one definition.
2Add stages — the inspection checkpoints (e.g. "Front Collar", "Cuff Assembly"). A production run inspects each stage in turn.
3Add prompts & parts — the plain-English descriptions SAM3 will look for. Prompts are top-level regions; parts are sub-features nested inside them.
4Draw the golden polygons — the approved shape for each prompt and part, in real board-plane coordinates. Traced on an exemplar, or auto-proposed from past production.
5Set tolerance envelopes (optional) — inner/outer ribbons around the golden that allow acceptable variation without failing the part.
6Upload an exemplar — the "ideal shot". It only saves if it grades high, and operators view it before capturing to know what good looks like.
Operator workflow
Running an inspection
First, each operator calibrates their phone once (a guided live-capture flow that walks the board around the frame), or adopts a shared high-quality profile for the same phone model. Then inspections run against a production run — one operator, one garment, with a target count per stage.
On the mobile scan screen (/m/scan):
1Pick a run and a stage, then open the full-screen camera.
2The shutter stays locked until the board is fully detected and the phone is roughly overhead — guaranteeing a measurable frame.
3Capture → the photo runs the whole pipeline → a grade comes back in seconds.
4Green (high) continue · amber (medium / misplaced) operator's choice to keep or retake · red (low / wrong / not detected) forces a retake.
5Repeat until the stage's target count is met; history keeps every inspection and its metrics.
The desktop flow (/garment) mirrors this with more room for detailed per-part metric cards and golden-overlay views. A run moves through open → pending_review → succeeded / failed as an admin signs it off.
Glossary
Key concepts
ChArUco board
The printed reference target. Its known geometry is what makes real-world measurement possible.
Camera profile
A camera's calibrated intrinsics + lens distortion. Per-user, reusable, shareable across same-model phones.
Stage
One inspection checkpoint within a garment (e.g. "Collar"). A run inspects every stage.
Prompt / Part
A plain-English description SAM3 segments. Prompts are top-level regions; parts nest inside them.
Golden polygon
The approved ideal shape, in real board-plane coordinates, that a measured part is compared against.
Envelope
Inner/outer tolerance ribbons around the golden that allow intentional placement and shape slack.
Exemplar
A reference "ideal shot" (must grade high) that operators view before capturing their own.
Run / Stage run
A production batch tied to one operator + garment; a stage run tracks scanned vs. target count per stage.
Under the hood
Tech stack
Vision & geometry
App & platform
The frontend is deliberately plain — vanilla JavaScript, no framework or build step for page logic, with Tailwind compiled ahead of time into a single stylesheet. Authentication is always a real ERP-issued token. Production runs on Railway with Postgres, Supabase blob storage, and SAM3 on fal.ai; there is no GPU in production.
Built MFG · charuco_table + garment_guard