# AGENTS.md

> vuln-digest is one document, `README.md`: instructions that AI assistants follow to set up a scheduled digest of new vulnerabilities for their user. This repository holds that document, a small helper script, and the build that publishes them at https://vuln-digest.secmy.app/. This file is for agents that change vuln-digest itself.

If you were asked to set up a vulnerability digest for a user, this file is not for you. Read `README.md` and follow it

## Project layout

- `README.md` - the document. It is the only source of the published text
- `tools/vuln_feed.py` - the helper script that the document tells agents to download. Python standard library only
- `tools/tests/` - offline tests of the helper script, with fixtures of each source
- `site/build.sh` - builds the site into `_site/`
- `site/check.sh` - checks the built site
- `site/landing.html`, `site/landing.css` - the landing page for people at `/`. Hand-written, no external scripts or fonts
- `site/404.html` - the page for unknown addresses
- `site/style.css`, `site/footer.html`, `site/toc.lua` - the look of the instructions page at `/docs/`, its footer, and a filter that puts the title above the table of contents
- Links between pages are relative, so the site works on its own domain and under a path such as `/vuln-digest/` on github.io. Only `site/404.html` uses absolute paths: the build prefixes them with `BASE_PATH`, set in the workflow
- `site/head.html` - description, canonical address, icons and social preview tags for the instructions page. Paths in it start with `../`, because the page lives at `/docs/`
- `site/assets/` - the icon and the social preview image. Each PNG is rendered from the SVG next to it
- `site/llms.txt`, `site/robots.txt`, `site/ai.txt`, `site/sitemap.xml` - published as they are
- `.github/workflows/pages.yml` - runs the tests, the build, the check, the linters and the secret scan on every pull request, and publishes from `main` only when all of them pass
- `.github/ci-requirements.txt` - versions and hashes of ruff and zizmor for the `lint` job
- `.github/dependabot.yml` - weekly updates of the actions and of `ci-requirements.txt`, with a 7 day cooldown
- `.github/CODEOWNERS` - the owner of the pipeline and the build scripts
- `ruff.toml` - the security rules (S) that ruff checks in `tools/`
- `SECURITY.md` - how to report a vulnerability

## Build and check

You need Python 3.9 or later, and Docker with a running daemon. Run the commands from the repository root:

```sh
python3 -I -m unittest discover -s tools/tests
sh site/build.sh
sh site/check.sh
```

All must exit with code 0. `site/check.sh` prints `all checks passed`. The same commands run in CI, in the `build` job

CI also runs two more jobs. Both must pass before the site is published:

- `lint` - actionlint and zizmor check `.github/`, ruff checks `tools/` with the rules in `ruff.toml`
- `secrets` - Betterleaks scans the whole git history for secrets

## Published files

All of these are generated by `site/build.sh`. Do not edit anything in `_site/`

| URL | Source |
|---|---|
| `/` | exact copy of `site/landing.html`, with `/landing.css` |
| `/docs/` | `README.md` converted to HTML, plus `site/footer.html`, with `../style.css` |
| `/404.html` | exact copy of `site/404.html` |
| `/README.md`, `/index.md`, `/llms-full.txt` | exact copies of `README.md` |
| `/AGENTS.md` | exact copy of this file |
| `/llms.txt`, `/robots.txt`, `/ai.txt`, `/sitemap.xml` | exact copies from `site/` |
| `/vuln_feed.py` | exact copy of `tools/vuln_feed.py` |
| `/favicon.svg`, `/favicon-32.png`, `/apple-touch-icon.png`, `/social-preview.png` | exact copies from `site/assets/` |

If you add a published file, add it to `site/build.sh`, to `site/check.sh` and to this table. If it is a document for readers, add it to `site/sitemap.xml` and to `site/llms.txt` too

If you change an SVG in `site/assets/`, render its PNG again at the same size (social preview 1200x630, touch icon 180x180, favicon 32x32) and commit both. The build does not render images. If you change the title or the summary of the document, update `site/head.html`, `site/landing.html`, `site/llms.txt` and the social preview image too

The landing page shows the start prompt twice. `site/check.sh` holds the same text and checks it. If you change the prompt, change all three, and keep the address of `README.md` in it

The landing page describes what the instructions do. When you change the behaviour in `README.md` (questions, sources, the digest format), check that the landing page still says the same

The crawler policy is: everything is allowed, for search, for fetching on a user's behalf and for training. `site/robots.txt` and `site/ai.txt` must agree with it

## Conventions for the helper script

- standard library only. An agent must be able to read the whole script before it runs it
- every change to fetching, deduplication or the state format comes with a test in `tools/tests/` that runs offline from fixtures
- the state format has a version. When you change it, migrate the previous version in `load_state` and add a test
- never mark an item as delivered in `fetch`. Only `mark` writes delivered items, and only from the snapshot
- a failing source goes to `errors` and does not stop the other sources
- if you change a command or an option, update the "Helper script" section of `README.md`

## Conventions for README.md

- plain English and short sentences. The text speaks to the agent as "you"
- use the hyphen `-`, not a long dash. Do not put a period at the end of a paragraph or a list item
- do not write versions, image digests or commit SHAs of tools in `README.md`. The agent that reads it finds them
- name a tool only after you checked it in its official source. Say whether it is the author's choice or a starting point
- when you add, remove or move a step, update every reference to step numbers: the stage description in "How to work", cross references inside the steps and the run steps R1-R8
- the example profile, the job prompt and the digest templates are examples. Keep them short and keep them consistent with the rules

## Conventions for the workflow and scripts

- pin every action by its full commit SHA, with the version in a comment
- pin every image by digest, and keep the tag next to the digest
- use versions that are at least 7 days old. Read SHAs and digests from GitHub and from the registry, not from memory
- never write a `${{ ... }}` expression that holds data from a pull request inside `run:`
- the workflow token stays read-only. Only the `deploy` job has write permissions
- tools that the workflow downloads are pinned by version and checksum. Python tools go to `.github/ci-requirements.txt` with `--hash`
- do not make a check non-blocking, and do not add `noqa`, a zizmor ignore or a Betterleaks allowlist entry without a comment that explains why
- Dependabot does not update image digests in `site/build.sh` and in the workflow, or the actionlint checksum. Update them by hand

## How to change this repository

- create a branch from `main` and open a pull request. One topic per pull request
- the `main` branch does not accept direct pushes, unsigned commits or force pushes
- sign your commits with the key that is set up in git. Do not use `--no-gpg-sign`
- do not add AI attribution lines to commit messages or pull request text
- wait until the `build` check passes. Merge a pull request only when the user asks you to

## Do not

- do not rewrite git history, force push, or delete branches and tags
- do not change repository settings, rulesets, Pages settings or DNS
- do not keep a second copy of the document text in the repository. The published copies come from `README.md` at build time
- do not weaken or remove a check in `site/check.sh` to make the build pass
