# Data sources

> Sync an external JSON API into Galerne docs on a schedule or a webhook, with a declarative mapping, field ownership, overrides that survive syncs, review mode, and the source tools.

A data source brings items from an external API into docs of one type: products, events, repositories, team members. The synced data lives in normal docs, so you query it, preview it, branch it and let agents enrich it like any other content.

The source owns the fields it maps. When a person or an agent edits one of them, the edit wins: the field becomes an override and later syncs leave it alone. **reset to source** gives the field back to the source.

## How a run works

1. Galerne fetches every page of the API. Any error stops the run before a single write.
2. An empty list, an answer that is not JSON, or a missing items list also stops the run. The previous data stays as it is.
3. Each item is mapped to a doc with an id built from the item, like `repo-galerne`. Running twice never creates duplicates.
4. Items whose mapped values did not change since the last run are skipped (a hash per item). A run with no changes writes nothing.
5. New and changed items are written and published. Overridden fields are never written.
6. Docs whose item left the API are unpublished and marked removed. They are never deleted, and they come back if the item does.

Each run is logged with its counts (created, updated, unchanged, removed, failed), the errors per item and its duration. A run that changed something emits the event `source.synced`; a run that failed or stopped emits `source.failed`.

## Create a source

In the studio, open **Settings › Data sources**, pick the type to fill, give the URL and map the fields. **test** fetches the API and shows the first docs it would write, with the item from the API next to them, and writes nothing. Agents do the same with `test_source` and `create_source`.

```json
{
  "name": "github",
  "definition": {
    "title": "GitHub repositories",
    "type": "repo",
    "fetch": {
      "url": "https://api.github.com/orgs/velours/repos?per_page=100",
      "headers": { "Authorization": "Bearer {{secret.TOKEN}}" },
      "pagination": { "type": "link" }
    },
    "key": "id",
    "id": "repo-{{name}}",
    "fields": {
      "name": "name",
      "slug": "name",
      "description": "description",
      "stars": "stargazers_count",
      "topics": "topics",
      "homepage": "homepage",
      "owner": { "path": "owner.login", "id": "person-{{value}}" },
      "seo.title": { "template": "{{name}} by {{owner.login}}" }
    },
    "schedule": { "every": 60 },
    "target": "main"
  },
  "secrets": { "TOKEN": "github_pat_…" }
}
```

| Setting | Meaning |
|---|---|
| `type` | The doc type the items fill. Singletons are not supported. |
| `fetch.url` | An `https` URL. Templates: `{{lang}}`, `{{page}}`, `{{cursor}}`, `{{secret.NAME}}`. |
| `fetch.method`, `fetch.body` | `GET` (default) or `POST` with a body, for GraphQL. The body takes the same templates. |
| `fetch.headers` | Request headers. Literal values of auth headers are moved to the secrets when you save. |
| `fetch.items` | JSON path of the list in the answer, like `data.items`. Empty when the answer is the list. |
| `fetch.pagination` | `none`, `page` (`param`, `start`, `sizeParam`, `size`), `cursor` (`param`, `path` of the next cursor) or `link` (the `Link` header). `maxPages` stops the reading, 20 by default. |
| `key` | JSON path of the item's id in the API, `id` by default. |
| `id` | Doc id template. Default `<source>-{{key}}`, plus `-{{lang}}` for doc-level i18n. Ids keep letters, digits, `.`, `_` and `-`. |
| `fields` | Target field path → mapping. These are the fields the source owns. |
| `languages` | Doc-level i18n: `{ "mode": "request" }` fetches once per language with `{{lang}}`; `{ "mode": "field", "path": "locale" }` reads the language of each item. |
| `schedule.every` | Minutes between runs, 15 at least. Without it, the source runs on its webhook or by hand. |
| `target` | `main` publishes each run. `review` writes the run on a branch and requests a review. |
| `webhook` | `true` accepts `POST /p/<project>/v1/sources/<name>/hook`. |

Requests go through the same guard as webhooks: public addresses only, 30 seconds and 10 MB per page, 10,000 items per run. A `Link` header that points to another host stops the run, so secret headers never leave the API's host.

## Mapping

A mapping is a JSON path into the item, or an object:

| Key | Meaning |
|---|---|
| `path` | JSON path: `name`, `owner.login`, `images[0].url`, `tags[*].name` (a list) |
| `template` | Text with placeholders: `"{{first}} {{last}}"` |
| `value` | A constant |
| `transform` | How the value is converted (below). By default it follows the field type. |
| `default` | Used when the item has no value |
| `id` | `reference`: the referenced doc id, with `{{value}}`, like `person-{{value}}` |
| `alt` | `image`: JSON path of the alt text |

| Transform | Result |
|---|---|
| `string`, `number`, `integer`, `boolean` | Converted values (`"12.6"` → `13` for integers, `"yes"` → `true`) |
| `date`, `datetime` | `2026-10-18` and ISO date times; numbers read as Unix time |
| `slug` | `Été à Paris` → `ete-a-paris` |
| `markdown`, `html` | Rich text |
| `image` | The image is downloaded once per URL and added to the media library |
| `reference` | `{ "_ref": "<id>" }`, the id built with the `id` template |
| `language` | `fr-FR` → `fr` when `fr` is a project language |
| `json` | The value as it is |

Target paths reach object fields (`seo.title`) and one language of a localized field (`tagline.fr`). Lists map item by item. A mapped field with no value in the item is cleared.

## Overrides

Docs a source fills carry a `_source` value: the source, the item id, the hash of the last mapped values, the owned paths, the overrides, and `removed` when the item left the API.

- Any edit to an owned field marks it overridden: in the studio while typing, with `patch_doc`, by an import or a restore. Syncs never write it again.
- `_source` itself cannot be written by tools; patches on it are refused.
- **reset to source** in the doc editor, or `reset_override`, restores the value from the last run at once. Publish the doc to put it online.
- `set_override` keeps the current value of an owned field without editing it.
- The studio shows **synced from …** on synced docs, and **synced** or **overridden** on each owned field.

A doc a person deleted is not recreated by later runs.

## Review changes

With `"target": "review"`, a run with changes writes them on a branch named `source/<name>/<date>-<time>` and requests a review (Pro plans and above). Later runs add to the same branch while it is open, so reviews do not pile up. Promote the branch with **publish** to put the changes online. Removals that reach `main` through a promotion are unpublished on the next run.

## Webhook trigger

With `"webhook": true`, the answer to `create_source` holds a hook secret once (`ghk_…`). Make a new one with `update_source` and `rotateHookSecret`. Call the hook with any of:

```sh
curl -X POST https://api.galerne.cloud/p/<project>/v1/sources/<name>/hook \
  -H "Authorization: Bearer ghk_…"
```

- `x-galerne-signature: t=<unix>,v1=<hex>`, the HMAC-SHA256 of `<t>.<body>`, signed like [Galerne webhooks](/docs/webhooks);
- `x-hub-signature-256: sha256=<hex>`, the HMAC-SHA256 of the body, as GitHub sends it.

The hook answers `202` and the run starts within 15 seconds. A disabled source answers `409`.

## Sources in galerne.config.ts

Sources can live in the schema and be deployed with it:

```ts
export default defineConfig({
  types: { repo: { fields: { name: { type: 'string', required: true }, stars: { type: 'number' } } } },
  sources: {
    github: {
      type: 'repo',
      fetch: { url: 'https://api.github.com/orgs/velours/repos', headers: { Authorization: 'Bearer {{secret.TOKEN}}' } },
      fields: { name: 'name', stars: 'stargazers_count' },
      schedule: { every: 60 },
    },
  },
})
```

`galerne schema deploy` checks them with the rest of the schema. Secret values never go in the config: set them in the studio or with `update_source`.

Precedence: names are unique in a project, and the config wins. A source declared in the config follows each deploy: added, changed, and removed when it leaves the config (the docs it filled stay). Its mapping is read only in the studio and in `update_source`; its secrets, hook secret and enabled switch are set there. A studio source with the same name as a config source becomes a config source on deploy and keeps its runs. It keeps its secrets when the config fetches from the same host (scheme and host of `fetch.url`). When the host differs, or a later deploy moves a config source to another host, Galerne removes the saved secrets and logs a failed run that says so: secrets given for one API never go to another. Set them again with `update_source`.

## Complex sources

The mapping is declarative on purpose: Galerne does not run your code. When an API needs real code (joins, several calls per item, custom logic), run it on your side and push the result through the [import API](/docs/migrate) with stable ids.

## Tools

| Tool | Right | What it does |
|---|---|---|
| `list_sources` | read | Sources with their schedule, target and last run |
| `get_source` | read | One source, its mapping, secret names and recent runs |
| `create_source` | admin | Adds a source |
| `update_source` | admin | Changes the mapping, secrets, enabled switch, or makes a new hook secret |
| `delete_source` | admin | Removes a studio source; its docs stay |
| `test_source` | admin | Dry run on the first items |
| `sync_source` | import and publish on `main` | Runs now (`wait: false` queues the run) |
| `source_runs` | read | The run log |
| `set_override`, `reset_override` | write on the doc | Keep a value, or give it back to the source |

Runs respect plan limits: on the Free plan, docs past the cap are not created (updates go on), and paused writes skip the run.
