diff --git a/README.md b/README.md index fe18076..f7f65eb 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,79 @@ # CodeNavigator -A single binary that serves the WebGL frontend and the -indexing API (jobs, SSE progress, snapshots, search, source) on one origin. -No dependencies beyond the Go standard library. +**See your code as a city.** CodeNavigator indexes a GitHub repository (or, if you allow it, a local +folder) and turns it into a place you can explore: a zoomable map, a 3D landscape, and a walkable +night-time city where every file is a building and every folder is a street. + +![CodeNavigator city view](docs/images/city.png) + +It's a single Go binary with no dependencies beyond the standard library. It serves both the +indexing API and the browser frontend (plain ES modules and WebGL2, no build step). + +## What you can do + +| | | +|---|---| +| ![2D map](docs/images/map-2d.png) | **2D map**: a squarified treemap of every file, sized by lines or references, with source code rendered right on the tiles as you zoom in. | +| ![3D landscape](docs/images/landscape-3d.png) | **3D landscape**: the same map extruded, orbit, pan and dolly around it. | +| ![Routes in the city](docs/images/city-routes.png) | **City**: folders become districts and blocks, files become buildings (footprint from lines, height from complexity, lit windows from definitions). Select a file to see its imports as cyan routes and its dependents as pulsing magenta routes along the streets. | +| ![Walking the city](docs/images/city-walk.png) | **Walk and fly** at street level with signs on every wall, folder street signs on the corners, source code on the facade in front of you, wandering residents and a city wall covered in posters. | + +Also: + +- **Search** every file and symbol, with results raised as light beacons in the city. +- **Private GitHub repos** via a GitHub App sign-in, scoped to the visitor's own access. +- **Security alerts**: with the right GitHub permissions, files with open code scanning or + Dependabot alerts smoke, burn (critical/high) and get wrapped in hazard tape. +- **Multi-user**: every browser gets its own private sessions, snapshots and sign-in. +- **Guided tour** of the biggest districts, and **shareable links** to an exact spot. +- **Touch controls**: joystick, look and pinch. + +## Quick start ```bash -go run . # http://127.0.0.1:4177 -go test -race ./... -node --test webtest/*.test.mjs # frontend maths: camera, city layout, collision, signs, links -go test -run x -bench Index # indexes $CODENAV_BENCH_DIR (default: Go's stdlib source) -./build.sh [tag] # Docker image, tests run during the build +go run . # http://127.0.0.1:4177 +``` + +Open the page, click **Open a codebase**, and paste a public GitHub URL such as +`https://github.com/prometheus/prometheus`. `git` must be on your `PATH` for cloning. + +With Docker: + +```bash +./build.sh # builds codenavigator:latest, running the tests inside docker run --rm -p 4177:4177 codenavigator:latest ``` -Configuration: +## Configuration -- `PORT` (default `4177`, or `--port`), `CORS_ORIGIN` (default `*`) -- `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`: a GitHub App's credentials; enables sign-in for private repositories -- `GITHUB_APP_SLUG`: the app's URL name, for the "grant repository access" link +| Variable | Default | Purpose | +|---|---|---| +| `PORT` | `4177` | Listening port (`--port` overrides) | +| `CORS_ORIGIN` | `*` | `Access-Control-Allow-Origin` for the API | +| `SHOW_LOCAL` | unset | `true` enables uploading local folders; otherwise only GitHub repositories | +| `GITHUB_CLIENT_ID` | unset | GitHub App client ID; enables sign-in for private repos and security alerts | +| `GITHUB_CLIENT_SECRET` | unset | GitHub App client secret | +| `GITHUB_APP_SLUG` | unset | The app's URL name, for the "grant repository access" link | -`git` must be on `PATH` to load GitHub repositories (the image includes it). +## Keyboard -## Views +| Key | Where | Action | +|---|---|---| +| `Ctrl`+`O` | anywhere | Open a codebase | +| `/` | anywhere | Search | +| `F` | map views | Fit everything in view | +| `[` | anywhere | Hide or show the sidebar | +| `1` `2` `3` | city | Helicopter, walk, fly | +| `W` `A` `S` `D`, `Shift` | walk/fly | Move, run | +| `Space` / `C` | fly | Up / down | +| `E` | walk/fly | Inspect the building in the crosshair | +| `T` | city | Guided tour | -- **2D map**: squarified treemap of files, sized by lines or references. -- **3D landscape**: the same map extruded, orbit with drag, pan with Shift+drag, scroll to dolly. -- **City**: folders become districts and blocks separated by roads, files become buildings - (footprint from lines, height from complexity, lit windows from definitions). - - `1` Helicopter, `2` Walk, `3` Fly. In walk/fly: click for mouse-look, WASD, Shift to run, - Space/C up and down when flying, `E` inspects the building in the crosshair. - - Double-click a street to drop in; the minimap shows your heading and teleports on click. - - Selecting a file draws import trails (cyan known, amber inferred); search hits raise beacons. - - `T` or **Tour** flies over the largest districts. **Copy link** shares the repository, view, - camera and selected file as a URL fragment (GitHub repositories only). - - Touch: left thumb joystick, right thumb look, pinch to zoom. - - Selecting a file also lays glowing routes along the streets: cyan/amber chevrons out to each - file it imports, and pulsing magenta dashes in from each file that imports it, in separate lanes. - - Residents wander the streets (toggle in Settings). A wall with posters encloses the city. - - Signed in with GitHub, files with open code scanning or Dependabot alerts smoke (and burn - for critical/high) and are wrapped in hazard tape. The GitHub App needs read access to - code scanning alerts and Dependabot alerts. +## Documentation -Frontend modules (plain ES modules, no build step): `camera.js` (matrices and orbit pose), -`city.js` (layout, collision, picking, sign placement, tour and links), `labels.js` (text atlas), -`renderer.js` (WebGL2 passes) and `app.js` (UI and input). - -## Multiple users - -Every browser gets an HttpOnly session cookie. Jobs, snapshots and GitHub sign-ins belong to -that session and are invisible (404) to others. Each session keeps its latest snapshot; snapshots -idle for an hour are deleted, at most 8 are held (least recently used evicted), and two indexes -run at once while the rest queue. State is in memory, so a restart signs everyone out. - -## Private GitHub repositories - -Uses a GitHub App's user authorisation. When an anonymous clone fails, the frontend offers -**Sign in with GitHub**; the callback (`/api/github/callback`) stores an 8-hour user token in -memory for the session. Clones send it as an HTTP header via environment git config, never in -the URL or arguments, and host git config and credential helpers are ignored. A signed-in user -can only open repositories where the app is installed and they have access, otherwise the -frontend links to the app's installation page. - -GitHub App settings: callback URL `https:///api/github/callback`, setup URL `https:///` -with "Redirect on update", token expiry on, no webhook, repository permission Contents: read-only. - -The browser frontend in `web/` is plain JavaScript (it has to run in the browser) and -is embedded into the binary; `web/config.js` points it at the same origin. - -Notes: -- `GET /` serves the frontend instead of a JSON service description. -- Jobs and snapshots are private to the browser session that created them (see above). -- Search queries are percent-decoded properly. -- `.gitignore` support covers standard globs but not global excludes or `.git/info/exclude`. +- [City guide](docs/CITY.md): what everything in the city means and how to get around +- [Architecture](docs/ARCHITECTURE.md): indexer, server, sessions, rendering pipeline +- [HTTP API](docs/API.md): every endpoint, with request and response shapes +- [GitHub App setup](docs/GITHUB_APP.md): private repos and security alerts +- [Deployment](docs/DEPLOYMENT.md): Docker, systemd, nginx, operating limits +- [Development](docs/DEVELOPMENT.md): running, testing, benchmarking, project layout diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..e837927 --- /dev/null +++ b/docs/API.md @@ -0,0 +1,156 @@ +# HTTP API + +All endpoints are under `/api/` and return JSON unless noted. Everything else is the static frontend. +Every response carries `Cache-Control: no-store`, `X-Content-Type-Options: nosniff` and CORS +headers (`Access-Control-Allow-Origin` from `CORS_ORIGIN`). `OPTIONS` on any path returns `204`. +Errors look like `{"error": "message"}`. + +## Sessions and ownership + +The first `POST /api/jobs` (or GitHub sign-in) sets an `HttpOnly`, `SameSite=Lax` cookie called +`codenav_session` (`Secure` when served over HTTPS or behind a proxy sending +`X-Forwarded-Proto: https`). Jobs, snapshots and GitHub sign-ins belong to that session. Any other +session gets `404` for them, exactly as if they didn't exist. IDs are random (`crypto/rand`). + +## Health + +### `GET /api/health` + +```json +{ "ok": true, "version": "0.1.0", "localFolders": false } +``` + +`localFolders` is `true` only when the server runs with `SHOW_LOCAL=true`. + +## Jobs (indexing) + +### `POST /api/jobs` + +```json +{ "kind": "github", "url": "https://github.com/owner/repo", "name": "repo" } +{ "kind": "local", "name": "my-folder" } +``` + +| Status | Meaning | +|---|---| +| `201` | `{"jobId": "job-…"}`; GitHub jobs start cloning immediately | +| `400` | Invalid body, unknown kind, missing or invalid GitHub URL (`https://github.com/owner/repo` only) | +| `403` | `kind: local` while local folders are disabled | + +GitHub clones are shallow (`--depth=1`), time out after 10 minutes, ignore the host's git config and +credential helpers, and authenticate with the visitor's GitHub token when they're signed in. + +### `POST /api/jobs/{id}/files?path=relative/path` + +Local jobs only. The request body is the raw file content (at most 64 MB). `path` must be relative +with no `..` segments. Returns `{"ok": true}`, `400` for unsafe paths, `404` for unknown or non-local jobs. + +### `POST /api/jobs/{id}/commit` + +Local jobs only. Starts indexing the uploaded files. Returns `202 {"ok": true}`. + +### `GET /api/jobs/{id}` + +```json +{ + "id": "job-…", "sourceKind": "github", "name": "prometheus", + "phase": "parsing", "completed": 420, "total": 1676, "message": "Parsing 421 of 1676", + "snapshotId": null, "error": null, "warnings": [], + "authRequired": "signin" +} +``` + +`phase` is `receiving`, `cloning`, `queued`, `scanning`, `parsing`, `linking`, `ready` or `error`. +`authRequired` only appears on failed clones that signing in (`signin`) or granting the GitHub App +access (`install`) could fix. + +### `GET /api/jobs/{id}/events` + +Server-sent events. Each change to the job is sent as: + +``` +event: progress +data: {…same shape as GET /api/jobs/{id}…} +``` + +A `: keep-alive` comment is sent every 15 seconds. The stream ends when the job has a `snapshotId` +or an `error`, or when the client disconnects. At most two jobs index at once; others report +`queued`. + +## Snapshots + +### `GET /api/snapshots/{id}` + +```json +{ "id": "snapshot-…", "name": "prometheus", "source": "github", "fileCount": 1676, + "totalLines": 568224, "definitions": 13717, "references": 790, "edges": 790, "warnings": [] } +``` + +### `GET /api/snapshots/{id}/scene` + +Everything the frontend needs to lay out the map and city: + +```json +{ + "id": "snapshot-…", "name": "prometheus", "source": "github", + "files": [{ "id": 1, "path": "cmd/prometheus/main.go", "name": "main.go", "directory": "cmd/prometheus", + "extension": "go", "language": "Go", "layer": "application", "lines": 2296, + "bytes": 91234, "complexity": 90, "preview": "// first 60 lines…", "symbolCount": 55 }], + "edges": [{ "from": 1, "to": 2, "kind": "import", "confidence": "known" }], + "totalLines": 568224, "definitions": 13717, "references": 790 +} +``` + +After a scene is delivered the server drops its copy of the previews. `layer` is one of +`application`, `library`, `test`, `generated`, `vendor`, `platform`, `docs`. `confidence` is `known` +(resolved path) or `inferred` (matched by module name). + +### `GET /api/snapshots/{id}/search?q=text` + +Case-insensitive substring match on file paths and symbol names (at least 2 characters, at most 180 hits): + +```json +[{ "entityId": 1, "path": "cmd/prometheus/main.go", "name": "main", "line": 120, "kind": "definition", "preview": "func main() {" }] +``` + +### `GET /api/snapshots/{id}/entities/{fileId}` + +The full file record, including `symbols: [{name, kind, line, signature}]`. + +### `GET /api/snapshots/{id}/entities/{fileId}/source?start=0&limit=600` + +```json +{ "start": 0, "lines": ["package main", "…"], "totalLines": 2296 } +``` + +`limit` is clamped to 1–4096. Once every line of a file has been delivered the server deletes it +from disk, and the workspace is removed once all files are delivered. + +### `GET /api/snapshots/{id}/alerts` + +Open security alerts for a GitHub snapshot, fetched with the viewer's own token: + +```json +{ + "available": true, + "codeScanning": { "status": "ok", "alerts": [ + { "path": "web/app.js", "title": "Cross-site scripting", "severity": "critical", "url": "https://github.com/…", "line": 7 } ] }, + "dependabot": { "status": "forbidden", "alerts": [] } +} +``` + +When alerts can't be fetched at all: `{"available": false, "reason": "not-github" | "signin"}`. +Per-set `status` is `ok`, `forbidden` (permission not granted or feature disabled) or +`unavailable`. Severities are normalised to `critical`, `high`, `medium` or `low`. Dependabot alerts +point at the manifest that declares the vulnerable package. + +## GitHub sign-in + +| Endpoint | Purpose | +|---|---| +| `GET /api/github/status` | `{"configured", "connected", "login", "installUrl"}` for the current session | +| `GET /api/github/login` | Redirects to GitHub with a single-use `state` bound to the session (10 minutes) | +| `GET /api/github/callback` | Exchanges the code, stores the user token in memory, redirects to `/?github=connected` (or `denied`/`error`) | +| `POST /api/github/logout` | Forgets the session's token | + +Login, callback and logout return `404` when no GitHub App is configured. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..cb39c7e --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,92 @@ +# Architecture + +``` +browser ──HTTP/SSE──▶ Go server (main.go, github.go) ──▶ indexer (indexer.go) ──▶ temp workspace + │ │ + │ └── GitHub (OAuth token exchange, alerts API, git clone) + └── WebGL2 frontend (web/): app.js · renderer.js · city.js · camera.js · labels.js +``` + +## Server + +One Go binary using only the standard library. The frontend is embedded with `go:embed` and served +from the same origin as the API. + +### Indexing (`indexer.go`) + +1. **Walk** the checkout in lexical order. Skip `.git`, `node_modules`, `target`, `dist`, `build` + and `.next`, honour `.ignore` files everywhere and `.gitignore` inside git checkouts. Only regular + files count (no symlinks). +2. **Parse** files in parallel (one worker per CPU). Binary files (a NUL byte in the first 4 KB) and + files over 8 MB are skipped. For each file it records lines, bytes, a complexity score (branching + tokens), the first 60 lines as a preview, definitions (class/struct/function patterns) and import + targets (from/import/require/#include/use/using patterns). Each regex has a cheap literal + prefilter, which made indexing about 3× faster than regex alone. +3. **Link** imports to files: relative paths first (`known`), then module-style names resolved by + path suffix or file stem (`inferred`). Ambiguous names are never guessed. +4. Classify each file's **language** (by extension) and **layer** (test, vendor, generated, platform, + library, docs or application) by path. + +### Sessions, jobs and snapshots (`main.go`) + +- **Sessions:** a random cookie identifies each browser, and jobs, snapshots and GitHub tokens are + owned by it. +- **Jobs:** a job is a GitHub clone or a local upload, reporting progress over SSE. +- **Snapshots:** each session keeps only its latest snapshot. Snapshots idle for an hour are expired, + at most 8 are held (least recently used evicted), and at most 2 indexes run at once. +- **Delivery:** once the scene is sent, previews are released, and each source file is deleted after + every line of it has been fetched. +- **Workspace cleanup:** only paths inside the workspace directory are ever deleted. + +### GitHub (`github.go`) + +- **Sign-in:** the GitHub App user-authorisation flow. A single-use `state` is bound to the + session, and the resulting user token is held in memory only (8-hour expiry). +- **Clones** pass the token as an HTTP header through `GIT_CONFIG_*` environment variables, never in + arguments or URLs, with host git config and credential helpers disabled. +- **Security alerts** are fetched per request with the viewer's token, following pagination only on + the API host. + +## Frontend (`web/`) + +Plain ES modules, no framework, no build step. + +| Module | Responsibility | +|---|---| +| `app.js` | UI, input, data loading, 2D overlay canvas, city orchestration (signs, routes, residents, tour, links, alerts) | +| `renderer.js` | All WebGL2 drawing: instanced boxes, sign quads, route ribbons, particles, sprites | +| `camera.js` | Matrix maths: perspective, lookAt, inverse, orbit pose | +| `city.js` | Pure city logic, unit tested: layout, collision, navigation grid and routing, sign placement, tape, walls, posters, residents, place links | +| `labels.js` | Text atlas for signs (shelf-packed canvas) and the poster atlas | +| `api.mjs` | `fetch` wrapper and SSE-over-fetch parser | + +### Views + +- **2D:** squarified treemap in a 1000×680 world, orthographic. Source text is painted into an + overview canvas when zoomed out, and into per-tile textures (LRU, 192 MB budget) when zoomed in. +- **3D:** the same instances extruded, rendered through a perspective orbit camera. Roofs sample the + overview canvas as a GPU texture, and only nearby roofs get sharp per-file textures. +- **City:** `layoutCity` lays the directory tree out in metres. Folders shrink by half a road on + each side (18 m avenues between districts, narrower streets deeper down). Files become buildings, + with footprint ≈ 1.1 m² per line and height 6 m + 10 m × log₂(1 + complexity). + +### City rendering passes (in order) + +1. **Floor:** ground and folder pavements in nesting order, depth test off, top faces only (so + nested pavements a few centimetres apart can't z-fight). +2. **Buildings and wall:** instanced boxes. The city shader adds procedural windows (lit share from + definition count), neon edges, a concrete wall and exponential fog. +3. **Effects:** additive and never occluding. Search beacons, fire and smoke particles (stateless, + animated in the vertex shader), residents (procedural stick-figure sprites), street routes + (mitred ribbons with chevrons or pulsing dashes, kept in right-hand lanes) and import arcs. +4. **Signs:** textured quads from the label atlas, depth-tested but not depth-writing. A shader test + discards any quad face that would read mirrored, which is how two-faced blade signs work. + +### Routing + +- **Grid:** a 3 m occupancy grid blocks buildings (grown by 0.5 m) and the city wall, and makes + pavements slightly dearer than roads. +- **Search:** one Dijkstra run from the selected building's doors reaches every import and + dependent (paths cost the same both ways). It uses float64 distances and a settled set. +- **Shaping:** routes are straightened by line-of-sight, extended into both buildings, and boxed-in + targets fall back to the nearest reachable street. diff --git a/docs/CITY.md b/docs/CITY.md new file mode 100644 index 0000000..268d09c --- /dev/null +++ b/docs/CITY.md @@ -0,0 +1,66 @@ +# City guide + +Switch to **City** in the toolbar. The city is laid out from the repository's folder tree at human +scale, so a large repository is a large city. + +## Reading the city + +| You see | It means | +|---|---| +| A **district** (big block, name floating above its tallest tower) | A top-level folder | +| A **block** inside it, separated by streets | A nested folder; deeper folders have narrower streets | +| A **building** | A file | +| Building **footprint** | Lines of code | +| Building **height** | Complexity (branching) | +| **Lit windows** | Definitions: more functions and classes, more lights on | +| Building **colour** | Semantic layer (application, tests, vendor, …), switchable palette | +| Name on the **roof** and **wall** | The file name | +| **Green street sign** sticking out at a corner | The folder that block belongs to; each corner of an intersection names its own folder | +| **Smoke** | The file has open medium/low security alerts | +| **Fire and smoke** | The file has an open critical/high security alert | +| **Yellow tape** rising across the walls | Dependabot alert (on the manifest file) | +| **Red tape** falling across the walls | Code scanning alert | +| **Light beam** into the sky | A search result | +| **Cyan chevrons** on the road | Route to a file the selected file imports (**amber** if the import was inferred) | +| **Pulsing magenta dashes** on the road | Route from a file that imports the selected file | +| **Arcs** in the sky | The same import relationships, as the crow flies | +| **Green figures** | Residents. They don't mean anything; they just live there | +| **Wall with posters** | The city limits, with a few facts about the codebase pasted on | + +## Getting around + +**Helicopter (`1`)**: drag to orbit, Shift+drag (or right-drag) to pan, scroll to zoom. +Double-click a building to fly to it, or double-click a street to drop down and walk. + +**Walk (`2`)**: click the map to capture the mouse and look around, `WASD` to move, `Shift` to run, +`Esc` to release the mouse. The crosshair names the building ahead. Press `E` (or click) to open +it in the sidebar. Walk right up to a building and its source code appears on the wall. You can't +walk through buildings or out past the city wall. + +**Fly (`3`)**: like walking, but `Space` and `C` climb and descend, and moving follows where you +look. + +**Touch**: left thumb is a joystick, right thumb looks around, pinch zooms (or climbs when flying). + +**Minimap**: click it to jump there. It shows your heading, the city wall, security alerts and +the selected file's routes. + +## Following dependencies + +Select a file (click it, search for it, or pick it from the sidebar). Routes appear on the streets +to everything it imports and from everything that imports it. With a file selected, switching to +**Walk** puts you at its door facing along the first route: follow the arrows. + +## Tour and links + +- **Tour (`T`)**: flies over the six largest districts with captions, and stops as soon as you + touch anything. +- **Copy link**: copies a URL with the repository, view, camera position and selected file. + Opening it loads the repository and puts you in the same spot. Links work for GitHub + repositories only. + +## Settings + +The gear opens settings: landscape palette, source text on tiles, **city residents** (they keep +the city animating, so switch them off on slow machines) and a frame-rate monitor. The sidebar +folds away with the tab on its left edge, or `[`. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md new file mode 100644 index 0000000..c483300 --- /dev/null +++ b/docs/DEPLOYMENT.md @@ -0,0 +1,114 @@ +# Deployment + +## Docker + +```bash +./build.sh [tag] # default codenavigator:latest; runs go test inside the build +docker run -d --name codenavigator -p 127.0.0.1:4177:4177 \ + --env-file /etc/codenavigator/app.env \ + --memory 2g --cpus 4 --pids-limit 512 \ + codenavigator:latest +``` + +The image is a static binary on Alpine with `git` and CA certificates, running as a non-root user, +with a `HEALTHCHECK` on `/api/health`. Workspaces live in the container's `/tmp` and disappear with +it. + +`app.env` (keep it `chmod 600`): + +```bash +GITHUB_CLIENT_ID= +GITHUB_CLIENT_SECRET= +GITHUB_APP_SLUG= +# SHOW_LOCAL=true # only if visitors should be able to upload local folders +``` + +## systemd + +`/etc/systemd/system/codenavigator.service`: + +```ini +[Unit] +Description=CodeNavigator (docker container) +After=docker.service network-online.target +Requires=docker.service +Wants=network-online.target + +[Service] +Restart=always +RestartSec=5 +TimeoutStartSec=0 +ExecStartPre=-/usr/bin/docker rm -f codenavigator +ExecStart=/usr/bin/docker run --rm --name codenavigator \ + --env-file /etc/codenavigator/app.env \ + -p 127.0.0.1:4177:4177 \ + --memory 2g --cpus 4 --pids-limit 512 \ + codenavigator:latest +ExecStop=/usr/bin/docker stop -t 30 codenavigator + +[Install] +WantedBy=multi-user.target +``` + +```bash +sudo systemctl daemon-reload && sudo systemctl enable --now codenavigator +``` + +Redeploy: `./build.sh && sudo systemctl restart codenavigator`. + +## nginx + +```nginx +server { + listen 80; + server_name codenav.example.com; + return 301 https://$host$request_uri; +} + +server { + listen 443 ssl http2; + server_name codenav.example.com; + + ssl_certificate /etc/letsencrypt/live/codenav.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/codenav.example.com/privkey.pem; + + # Local folder uploads send one file per request; the frontend skips files over 8 MB. + client_max_body_size 10m; + + location / { + proxy_pass http://127.0.0.1:4177; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + # Indexing progress is a server-sent event stream: don't buffer it. + proxy_buffering off; + proxy_read_timeout 300s; + } +} +``` + +`X-Forwarded-Proto: https` makes the session cookie `Secure`. Certificates: `certbot certonly --nginx -d codenav.example.com`. + +## Operating limits + +| Limit | Value | +|---|---| +| Concurrent indexing | 2 jobs; others queue | +| Snapshots held | 8 in total, 1 per session, least recently used evicted | +| Idle expiry | 1 hour for snapshots, finished jobs and abandoned uploads | +| Clone timeout | 10 minutes | +| File size | files over 8 MB are skipped; uploads capped at 64 MB | +| GitHub tokens | in memory, 8 hours | + +All state is in memory. A restart clears snapshots and signs everyone out. + +## Security notes + +- **Local folders** are off unless `SHOW_LOCAL=true`. With them on, anyone who can reach the site + can upload files to the server's temporary workspace. +- **Anyone who can reach the site can clone public repositories** through it. Put it behind + authentication if that matters to you. +- **Visitors' GitHub tokens** never reach the browser, git arguments or logs. Host git config and + credential helpers are ignored, so the server's own credentials can't be used by a visitor. diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md new file mode 100644 index 0000000..9e0aa2a --- /dev/null +++ b/docs/DEVELOPMENT.md @@ -0,0 +1,75 @@ +# Development + +Requirements: Go 1.26+, git, and Node 20+ (only to run the frontend tests). + +```bash +go run . # http://127.0.0.1:4177, frontend served from web/ via go:embed +SHOW_LOCAL=true go run . # also allow local folder uploads +``` + +The frontend is embedded at build time, so **restart `go run` after changing files in `web/`**. + +## Tests + +```bash +go test -race ./... # server, indexer, GitHub flows (fake GitHub) +node --test webtest/*.test.mjs # frontend maths: camera, city, routing, signs, posters +go test -run x -bench Index # indexing benchmark on $CODENAV_BENCH_DIR (default: Go's stdlib) +``` + +What the suites cover: + +- **Go:** + - Indexing: ignore rules, parsing and linking. + - Jobs: the local job lifecycle end to end (upload → SSE → scene → source release), session + isolation, expiry and eviction, the concurrency queue, SSE keep-alives, and the `SHOW_LOCAL` gate. + - GitHub: the sign-in flow including forged and reused state, clone token handling and timeouts, + security alert pagination and permissions. +- **Node:** + - Camera: parity with the original landscape projection. + - City: layout invariants (no overlaps, avenue widths), collision and movement, routing (detours, + enclosed buildings, a large-city time budget), ribbon lanes and mitred corners. + - Signs and walls: sign readability and placement, hazard tape, walls and posters, text fitting. + - Other: residents staying walkable, and place links rejecting untrusted input. + +## Project layout + +``` +main.go HTTP server, sessions, jobs, snapshots, expiry +github.go GitHub App sign-in, clone command, security alerts +indexer.go Walking, parsing, linking, classification +*_test.go Go tests +web/ + index.html Markup + styles.css All styling (see the design notes below) + app.js UI, input, overlays, city orchestration + renderer.js WebGL2 passes and shaders + city.js Pure city logic (layout, routing, signs, residents…) + camera.js Matrix maths + labels.js Sign atlas, poster atlas, text fitting + api.mjs fetch + SSE helpers + fonts/ Self-hosted Archivo Black, Atkinson Hyperlegible, Space Mono +webtest/ node:test suites for web/ modules +docs/ This documentation and screenshots +Dockerfile, build.sh +``` + +## Conventions + +- **No dependencies:** the Go side uses only the standard library, and the frontend is plain ES + modules with no bundler. +- **Keep logic testable:** anything that can be pure maths or data goes in `city.js`, `camera.js` + or `labels.js`, with a `node:test` test. `app.js` wires it to the DOM and `renderer.js` draws it. +- **Commits** are small and describe why, not just what. + +## Design notes + +The interface is deliberately hand-made rather than polished: + +- **Colour:** flat warm ink surfaces, hard 2px black edges, offset shadows instead of blur, and a + clashing lime/pink/sky/tangerine palette. +- **Type:** chunky Archivo Black headings, Atkinson Hyperlegible body text and Space Mono labels. +- **Personality:** small tilts, stickers and wiggles give it some energy. Everything animated + respects `prefers-reduced-motion`. +- **Tokens:** colours and fonts are CSS variables at the top of `styles.css`, and the city's + palette lives in the shaders in `renderer.js`. diff --git a/docs/GITHUB_APP.md b/docs/GITHUB_APP.md new file mode 100644 index 0000000..0e90777 --- /dev/null +++ b/docs/GITHUB_APP.md @@ -0,0 +1,65 @@ +# GitHub App setup + +Public repositories work without any setup. A GitHub App adds two things: + +- **Private repositories**: visitors sign in and can open repositories the app is installed on and + they have access to. +- **Security alerts**: files with open code scanning or Dependabot alerts burn in the city. + +CodeNavigator uses the app's *user authorisation* only: no private key, no webhooks, and tokens act +with the visitor's own permissions. + +## 1. Register the app + +Go to **https://github.com/settings/apps/new** (or an organisation's Developer settings) and fill in: + +| Field | Value | +|---|---| +| GitHub App name | anything; it becomes the slug in `github.com/apps/` | +| Homepage URL | `https://your-host` | +| Callback URL | `https://your-host/api/github/callback` | +| Expire user authorization tokens | ✅ | +| Request user authorization (OAuth) during installation | ☐ off | +| Enable Device Flow | ☐ off | +| Setup URL | `https://your-host/`, with **Redirect on update** ✅ | +| Webhook → Active | ☐ off | +| Where can this GitHub App be installed? | **Any account** to let others use it | + +**Repository permissions** (everything else "No access"): + +| Permission | Access | Needed for | +|---|---|---| +| Contents | Read-only | Cloning private repositories | +| Metadata | Read-only | Added automatically | +| Code scanning alerts | Read-only | Red tape and fire | +| Dependabot alerts | Read-only | Yellow tape and fire | + +Create the app, copy the **Client ID**, and click **Generate a new client secret**. + +## 2. Configure the server + +```bash +GITHUB_CLIENT_ID=Iv23li... +GITHUB_CLIENT_SECRET=... +GITHUB_APP_SLUG=your-app-slug +``` + +Restart the server. `GET /api/github/status` should now report `"configured": true`. + +## 3. Install it + +Visit `https://github.com/apps/`, click **Install**, and choose the repositories it may read. +Each user (or organisation admin) installs it for their own repositories. When a signed-in visitor +opens a repository the app can't see, CodeNavigator offers a **Grant repository access** button +linking to that page. + +## How sign-in behaves + +- A private repo that fails to clone shows **Sign in with GitHub**. After approving on GitHub, + visitors land back on the site with the Open codebase dialog already showing the result and the + repository filled in. +- Tokens live only in server memory for 8 hours and are never sent to the browser. Restarting the + server signs everyone out. +- **Adding permissions later** (e.g. alerts): existing installations must accept the new permissions + on GitHub, and visitors need to sign out and back in to get a token with them. Until then the + sidebar reports which alert kinds it has no access to. diff --git a/docs/images/city-routes.png b/docs/images/city-routes.png new file mode 100644 index 0000000..fa2bceb Binary files /dev/null and b/docs/images/city-routes.png differ diff --git a/docs/images/city-walk.png b/docs/images/city-walk.png new file mode 100644 index 0000000..fe2b12d Binary files /dev/null and b/docs/images/city-walk.png differ diff --git a/docs/images/city.png b/docs/images/city.png new file mode 100644 index 0000000..756c0fa Binary files /dev/null and b/docs/images/city.png differ diff --git a/docs/images/landscape-3d.png b/docs/images/landscape-3d.png new file mode 100644 index 0000000..b021f7c Binary files /dev/null and b/docs/images/landscape-3d.png differ diff --git a/docs/images/map-2d.png b/docs/images/map-2d.png new file mode 100644 index 0000000..79bf472 Binary files /dev/null and b/docs/images/map-2d.png differ diff --git a/docs/images/welcome.png b/docs/images/welcome.png new file mode 100644 index 0000000..2bd6344 Binary files /dev/null and b/docs/images/welcome.png differ