docs menu

Data sources

A data source brings items from an external API into docs of one type: products, events, repositories, team members. The synced data lives in normal docs, so you query it, preview it, branch it and let agents enrich it like any other content.

The source owns the fields it maps. When a person or an agent edits one of them, the edit wins: the field becomes an override and later syncs leave it alone. reset to source gives the field back to the source.

How a run works#

  1. Galerne fetches every page of the API. Any error stops the run before a single write.
  2. An empty list, an answer that is not JSON, or a missing items list also stops the run. The previous data stays as it is.
  3. Each item is mapped to a doc with an id built from the item, like repo-galerne. Running twice never creates duplicates.
  4. Items whose mapped values did not change since the last run are skipped (a hash per item). A run with no changes writes nothing.
  5. New and changed items are written and published. Overridden fields are never written.
  6. Docs whose item left the API are unpublished and marked removed. They are never deleted, and they come back if the item does.

Each run is logged with its counts (created, updated, unchanged, removed, failed), the errors per item and its duration. A run that changed something emits the event source.synced; a run that failed or stopped emits source.failed.

Create a source#

In the studio, open Settings › Data sources, pick the type to fill, give the URL and map the fields. test fetches the API and shows the first docs it would write, with the item from the API next to them, and writes nothing. Agents do the same with test_source and create_source.

json
{
  "name": "github",
  "definition": {
    "title": "GitHub repositories",
    "type": "repo",
    "fetch": {
      "url": "https://api.github.com/orgs/velours/repos?per_page=100",
      "headers": { "Authorization": "Bearer {{secret.TOKEN}}" },
      "pagination": { "type": "link" }
    },
    "key": "id",
    "id": "repo-{{name}}",
    "fields": {
      "name": "name",
      "slug": "name",
      "description": "description",
      "stars": "stargazers_count",
      "topics": "topics",
      "homepage": "homepage",
      "owner": { "path": "owner.login", "id": "person-{{value}}" },
      "seo.title": { "template": "{{name}} by {{owner.login}}" }
    },
    "schedule": { "every": 60 },
    "target": "main"
  },
  "secrets": { "TOKEN": "github_pat_…" }
}
SettingMeaning
typeThe doc type the items fill. Singletons are not supported.
fetch.urlAn https URL. Templates: {{lang}}, {{page}}, {{cursor}}, {{secret.NAME}}.
fetch.method, fetch.bodyGET (default) or POST with a body, for GraphQL. The body takes the same templates.
fetch.headersRequest headers. Literal values of auth headers are moved to the secrets when you save.
fetch.itemsJSON path of the list in the answer, like data.items. Empty when the answer is the list.
fetch.paginationnone, page (param, start, sizeParam, size), cursor (param, path of the next cursor) or link (the Link header). maxPages stops the reading, 20 by default.
keyJSON path of the item's id in the API, id by default.
idDoc id template. Default <source>-{{key}}, plus -{{lang}} for doc-level i18n. Ids keep letters, digits, ., _ and -.
fieldsTarget field path → mapping. These are the fields the source owns.
languagesDoc-level i18n: { "mode": "request" } fetches once per language with {{lang}}; { "mode": "field", "path": "locale" } reads the language of each item.
schedule.everyMinutes between runs, 15 at least. Without it, the source runs on its webhook or by hand.
targetmain publishes each run. review writes the run on a branch and requests a review.
webhooktrue accepts POST /p/<project>/v1/sources/<name>/hook.

Requests go through the same guard as webhooks: public addresses only, 30 seconds and 10 MB per page, 10,000 items per run. A Link header that points to another host stops the run, so secret headers never leave the API's host.

Mapping#

A mapping is a JSON path into the item, or an object:

KeyMeaning
pathJSON path: name, owner.login, images[0].url, tags[*].name (a list)
templateText with placeholders: "{{first}} {{last}}"
valueA constant
transformHow the value is converted (below). By default it follows the field type.
defaultUsed when the item has no value
idreference: the referenced doc id, with {{value}}, like person-{{value}}
altimage: JSON path of the alt text
TransformResult
string, number, integer, booleanConverted values ("12.6" → 13 for integers, "yes" → true)
date, datetime2026-10-18 and ISO date times; numbers read as Unix time
slugÉté à Paris → ete-a-paris
markdown, htmlRich text
imageThe image is downloaded once per URL and added to the media library
reference{ "_ref": "<id>" }, the id built with the id template
languagefr-FR → fr when fr is a project language
jsonThe value as it is

Target paths reach object fields (seo.title) and one language of a localized field (tagline.fr). Lists map item by item. A mapped field with no value in the item is cleared.

Overrides#

Docs a source fills carry a _source value: the source, the item id, the hash of the last mapped values, the owned paths, the overrides, and removed when the item left the API.

  • Any edit to an owned field marks it overridden: in the studio while typing, with patch_doc, by an import or a restore. Syncs never write it again.
  • _source itself cannot be written by tools; patches on it are refused.
  • reset to source in the doc editor, or reset_override, restores the value from the last run at once. Publish the doc to put it online.
  • set_override keeps the current value of an owned field without editing it.
  • The studio shows synced from … on synced docs, and synced or overridden on each owned field.

A doc a person deleted is not recreated by later runs.

Review changes#

With "target": "review", a run with changes writes them on a branch named source/<name>/<date>-<time> and requests a review (Pro plans and above). Later runs add to the same branch while it is open, so reviews do not pile up. Promote the branch with publish to put the changes online. Removals that reach main through a promotion are unpublished on the next run.

Webhook trigger#

With "webhook": true, the answer to create_source holds a hook secret once (ghk_…). Make a new one with update_source and rotateHookSecret. Call the hook with any of:

sh
curl -X POST https://api.galerne.cloud/p/<project>/v1/sources/<name>/hook \
  -H "Authorization: Bearer ghk_…"
  • x-galerne-signature: t=<unix>,v1=<hex>, the HMAC-SHA256 of <t>.<body>, signed like Galerne webhooks;
  • x-hub-signature-256: sha256=<hex>, the HMAC-SHA256 of the body, as GitHub sends it.

The hook answers 202 and the run starts within 15 seconds. A disabled source answers 409.

Sources in galerne.config.ts#

Sources can live in the schema and be deployed with it:

ts
export default defineConfig({
  types: { repo: { fields: { name: { type: 'string', required: true }, stars: { type: 'number' } } } },
  sources: {
    github: {
      type: 'repo',
      fetch: { url: 'https://api.github.com/orgs/velours/repos', headers: { Authorization: 'Bearer {{secret.TOKEN}}' } },
      fields: { name: 'name', stars: 'stargazers_count' },
      schedule: { every: 60 },
    },
  },
})

galerne schema deploy checks them with the rest of the schema. Secret values never go in the config: set them in the studio or with update_source.

Precedence: names are unique in a project, and the config wins. A source declared in the config follows each deploy: added, changed, and removed when it leaves the config (the docs it filled stay). Its mapping is read only in the studio and in update_source; its secrets, hook secret and enabled switch are set there. A studio source with the same name as a config source becomes a config source on deploy and keeps its runs. It keeps its secrets when the config fetches from the same host (scheme and host of fetch.url). When the host differs, or a later deploy moves a config source to another host, Galerne removes the saved secrets and logs a failed run that says so: secrets given for one API never go to another. Set them again with update_source.

Complex sources#

The mapping is declarative on purpose: Galerne does not run your code. When an API needs real code (joins, several calls per item, custom logic), run it on your side and push the result through the import API with stable ids.

Tools#

ToolRightWhat it does
list_sourcesreadSources with their schedule, target and last run
get_sourcereadOne source, its mapping, secret names and recent runs
create_sourceadminAdds a source
update_sourceadminChanges the mapping, secrets, enabled switch, or makes a new hook secret
delete_sourceadminRemoves a studio source; its docs stay
test_sourceadminDry run on the first items
sync_sourceimport and publish on mainRuns now (wait: false queues the run)
source_runsreadThe run log
set_override, reset_overridewrite on the docKeep a value, or give it back to the source

Runs respect plan limits: on the Free plan, docs past the cap are not created (updates go on), and paused writes skip the run.