erdatools

Utilities for listing folders and reading image EXIF metadata from the Electronic Research Data Archive (ERDA) at Aarhus University. Supports both SFTP access via SSH ControlMaster and public anonymous HTTP URLs.

Includes a REST API (plumber) for programmatic indexing and database integration: AMI sourceimages/sessions writes (replacing the ami-indexer workflow) and index-only ingest of CamAlien on-board images into the camalien database.

Installation

The erdatools package is publically available via r-universe.dev.

# Enable the ldalby universe
options(repos = c(
    ldalby = 'https://ldalby.r-universe.dev',
    CRAN = 'https://cloud.r-project.org'))

# Then simply install the package
install.packages('erdatools')

Pipeline scripts

Entry-point scripts live in pipelines/ (not part of the package build):

Script Purpose
download_imgs.R SFTP download + EXIF extraction -> per-folder parquet files
read_exif_http.R HTTP-based EXIF extraction (no local storage) -> per-folder parquet files
read_exif_parquet.R Combine parquets, parse UserComment JSON into detections
index_erda.R BFS directory indexer (full or update mode) -> parquet index
run_deployment.R Full deployment pipeline (standalone + IoT): index -> filter -> group -> write to the AMI database
camalien_write.R Index-only CamAlien on-board ingest: read index -> build -> write to the camalien database

Database administration helpers (bulk-insert deployments/partners, post-deploy integrity checks, project-year deploy loop) are in admin/.

REST API

The API provides endpoints for remote ERDA indexing and sourceimages database writes, with async job tracking for long-running operations.

Quick start

# Dev mode (from package root)
Rscript -e "pkgload::load_all(); et_api_run()"

# Docker
docker build -t erdatools-api .
docker run -p 8000:8000 \
  -e ERDA_SHARELINK_AMI=your_token \
  erdatools-api

Endpoints

Method Path Description
GET /health Status, version, uptime
POST /index/full Start BFS indexing (async)
POST /index/update Diff-update index (async)
GET /index/status/{job_id} Poll job status
GET /index/jobs List all jobs
POST /index/filter Filter index by extensions
POST /sourceimages/build Preview sourceimages transform
POST /sourceimages/write Build + write to DB/parquet (async)
POST /sourceimages/diff Preview new records not yet in DB
POST /pipeline/run Group + build + write for one image type (async)
POST /pipeline/deploy Full pipeline from index to DB for a deployment (async)
GET /deployments List deployment IDs filtered by projectid and/or year
POST /camalien/write Write CamAlien on-board images from a pre-built index to the camalien DB (async)
POST /camalien/deploy Fresh SFTP index + CamAlien write in one job (async)

Authentication

Set the API_KEY env var to enable API key auth via the X-API-KEY header. The API refuses to start when neither API_KEY nor API_ALLOW_NO_AUTH is set, so a misconfigured container fails loudly in docker logs instead of serving open endpoints. For local development without auth, set API_ALLOW_NO_AUTH=true. The /health endpoint is always unauthenticated.

Behind an API Gateway there are two independent auth layers: the gateway's Lambda authorizer reads Authorization, plumber reads X-API-KEY. Clients going through the gateway must send both headers -- set API_GW_KEY alongside API_KEY and the admin scripts add both. See deploy/README.md.

Base URLs

The same calls work against all three instances; only the base URL and headers differ.

Instance Base URL Headers
Local (dev) http://localhost:8000 none with API_ALLOW_NO_AUTH=true, else X-API-KEY
Test (next) https://next.api.erdatools.ecoscience.dk Authorization (gateway) + X-API-KEY
Production https://latest.api.erdatools.ecoscience.dk Authorization (gateway) + X-API-KEY 
# Local
curl -s http://localhost:8000/health | jq

# Test
curl -s https://next.api.erdatools.ecoscience.dk/health \
  -H "Authorization: $API_GW_KEY" \
  -H "X-API-KEY: $API_KEY" | jq

# Production
curl -s https://latest.api.erdatools.ecoscience.dk/health \
  -H "Authorization: $API_GW_KEY" \
  -H "X-API-KEY: $API_KEY" | jq

A deployment run against each instance (the examples in the workflows below use localhost; swap in the base URL and add the two headers):

# Local
curl -s -X POST http://localhost:8000/pipeline/deploy \
  -H "Content-Type: application/json" \
  -d '{"deployment_id": "IT1#2023"}' | jq

# Test
curl -s -X POST https://next.api.erdatools.ecoscience.dk/pipeline/deploy \
  -H "Authorization: $API_GW_KEY" -H "X-API-KEY: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"deployment_id": "IT1#2023"}' | jq

# Production
curl -s -X POST https://latest.api.erdatools.ecoscience.dk/pipeline/deploy \
  -H "Authorization: $API_GW_KEY" -H "X-API-KEY: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"deployment_id": "IT1#2023"}' | jq

Poll and list with the same headers:

curl -s https://next.api.erdatools.ecoscience.dk/index/status/$JOB_ID \
  -H "Authorization: $API_GW_KEY" -H "X-API-KEY: $API_KEY" | jq

curl -s 'https://latest.api.erdatools.ecoscience.dk/deployments?projectid=ias&year=2025' \
  -H "Authorization: $API_GW_KEY" -H "X-API-KEY: $API_KEY" | jq

While a job runs, its status shows which step it is in and when it last made progress:

{
  "status": "running",
  "stage": "edge_detections",
  "progress": { "files_done": 114, "files_total": 139, "detections_written": 713029, "detection_tracks_written": 246930 },
  "last_progress_at": "2026-10-01T12:11:10",
  "worker_pid": 412
}

If last_progress_at stops advancing while status is running, the job is probably stalled. updated_at only changes when the job finishes. A worker that dies mid-job (e.g. out of memory) is reported as failed. Worker output (stage changes, progress bars) goes to docker logs.

The loop helper takes the same settings via env vars:

API_BASE=https://next.api.erdatools.ecoscience.dk \
API_KEY=... API_GW_KEY=... \
PROJECT_ID=ias YEAR=2025 \
  Rscript admin/deploy_project_year.R

/index/filter, /sourceimages/build and /sourceimages/diff are not usable through the gateway (30 s limit; see Deployment).

Workflow: Index to sourceimages parquet

A typical workflow indexes the remote ERDA directory, filters the index to JPGs, previews the sourceimages transform, then writes the result. All examples below assume the API is running locally without auth.

1. Start the API (dev mode)

Rscript -e "pkgload::load_all(); et_api_run()"

2. Check health

curl -s http://localhost:8000/health | jq

{
  "status": "ok",
  "version": "0.13.0",
  "uptime": 12.4
}

3. Start full indexing

This triggers an async BFS traversal of the remote ERDA tree. Requires the ERDA_SHARELINK_AMI env var to be set when the API starts. The tree to walk comes from the request: pass remote_path (relative to the sharelink root), or pass deployment_id and let the API look the path up in the AMI database.

curl -s -X POST http://localhost:8000/index/full \
  -H "Content-Type: application/json" \
  -d '{"remote_path": "/storage/onestop/2025/"}' | jq

{
  "job_id": "index_full_20260315_120000_a1b2c3",
  "status": "running"
}

4. Poll job status

curl -s http://localhost:8000/index/status/index_full_20260315_120000_a1b2c3 | jq

While running:

{
  "id": "index_full_20260315_120000_a1b2c3",
  "type": "index_full",
  "status": "running",
  "result": null,
  "error": null,
  "created_at": "2026-03-15T12:00:00",
  "updated_at": "2026-03-15T12:00:00"
}

When completed:

{
  "id": "index_full_20260315_120000_a1b2c3",
  "type": "index_full",
  "status": "completed",
  "result": {
    "rows": 1117699,
    "files": 1100000,
    "path": "./erda_index.parquet"
  },
  "error": null,
  "created_at": "2026-03-15T12:00:00",
  "updated_at": "2026-03-15T12:41:00"
}

On failure:

{
  "id": "index_full_20260315_120000_a1b2c3",
  "type": "index_full",
  "status": "failed",
  "result": null,
  "error": "SSH connection timed out",
  "created_at": "2026-03-15T12:00:00",
  "updated_at": "2026-03-15T12:05:00"
}

5. Filter the index to JPGs

curl -s -X POST http://localhost:8000/index/filter \
  -H "Content-Type: application/json" \
  -d '{"extensions": ["jpg", "jpeg"]}' | jq '.[0:2]'

[
  {
    "dir": "/storage/onestop/2025/portugal/2025_07_10-11_10",
    "name": "CT_1720612200_GT_5678_lon_12.34_lat_56.78_alt_120.5_vel_0.0_expt_0.01_gain_1.4.jpg",
    "is_dir": false,
    "size": 102400,
    "mtime": "Jul 10 11:10"
  },
  {
    "dir": "/storage/onestop/2025/portugal/2025_07_10-11_10",
    "name": "CT_1720612260_GT_5678_lon_12.34_lat_56.78_alt_120.5_vel_0.0_expt_0.01_gain_1.4.jpg",
    "is_dir": false,
    "size": 98304,
    "mtime": "Jul 10 11:11"
  }
]

6. Preview the sourceimages transform

curl -s -X POST http://localhost:8000/sourceimages/build \
  -H "Content-Type: application/json" \
  -d '{"deploymentid": 42}' | jq '.[0:1]'

[
  {
    "index_ts": "2026-03-15T12:45:00",
    "filename": "CT_1720612200_GT_5678_lon_12.34_lat_56.78_alt_120.5_vel_0.0_expt_0.01_gain_1.4.jpg",
    "deploymentid": 42,
    "url": "https://anon.erda.au.dk/share_redirect/TOKEN/storage/onestop/2025/portugal/2025_07_10-11_10/CT_1720612200_GT_5678_lon_12.34_lat_56.78_alt_120.5_vel_0.0_expt_0.01_gain_1.4.jpg",
    "timestamp": "1720612200",
    "issnapshot": true,
    "processed": false,
    "queued": false
  }
]

7. Write sourceimages to parquet

curl -s -X POST http://localhost:8000/sourceimages/write \
  -H "Content-Type: application/json" \
  -d '{
    "deploymentid": 42,
    "output": {"db": false, "parquet": true, "parquet_path": "./out/"}
  }' | jq

{
  "job_id": "sourceimages_write_20260315_124500_d4e5f6",
  "status": "running"
}

Poll with /index/status/{job_id} until completed. The result will include:

{
  "rows": 12835,
  "parquet_path": "./out/sourceimages.parquet"
}

Workflow: Write to database

To write sourceimages directly to a PostgreSQL database, set the AMI_* env vars and enable DB output:

curl -s -X POST http://localhost:8000/sourceimages/write \
  -H "Content-Type: application/json" \
  -d '{
    "deploymentid": 42,
    "output": {"db": true, "parquet": false}
  }' | jq

If you don't know the deploymentid, provide the deployment lookup fields instead:

curl -s -X POST http://localhost:8000/sourceimages/write \
  -H "Content-Type: application/json" \
  -d '{
    "projectid": "onestop",
    "partnerid": 1,
    "year": 2025,
    "trap_code": "PT001",
    "output": {"db": true, "parquet": true, "parquet_path": "./out/"}
  }' | jq

Workflow: Full deployment pipeline

The /pipeline/deploy endpoint runs the complete pipeline for a deployment: SFTP index, filter, group motion + snapshot images, build sourceimages and sessions, diff, write to DB, and assign sessions. Only requires a deployment_id.

curl -s -X POST http://localhost:8000/pipeline/deploy \
  -H "Content-Type: application/json" \
  -d '{"deployment_id": "IT1#2023"}' | jq

{
  "job_id": "pipeline_deploy_20260409_120000_a1b2c3",
  "status": "running"
}

Poll with /index/status/{job_id} until completed:

{
  "status": "completed",
  "result": {
    "motion_images": 1234,
    "snapshot_images": 567,
    "motion_sessions": 42,
    "snapshot_sessions": 10,
    "sourceimages_new": 1801,
    "sourceimages_total": 1801,
    "sessions_total": 52
  }
}

For processing a single image type with a pre-existing index, use /pipeline/run instead:

curl -s -X POST http://localhost:8000/pipeline/run \
  -H "Content-Type: application/json" \
  -d '{"deployment_id": "IT1#2023", "type": "motion"}' | jq

Workflow: Loop deploy over a project + year

To run /pipeline/deploy for every deployment in a given project and year, use the /deployments endpoint to discover IDs and the admin/deploy_project_year.R helper to loop sequentially. Sequential is intentional — the ERDA sharelink has a 16-session concurrent limit and parallel deploys risk stalling.

curl -s 'http://localhost:8000/deployments?projectid=ias&year=2025' | jq

{
  "count": 12,
  "deployment_ids": ["BG1#2025", "IT1#2025", "..."]
}

PROJECT_ID=ias YEAR=2025 API_BASE=http://localhost:8000 \
  Rscript admin/deploy_project_year.R

The script polls /index/status/{job_id} until each deploy reaches completed or failed, prints a per-deployment summary, and exits non-zero if any failed.

Workflow: CamAlien on-board images

CamAlien on-board images are ingested into the camalien database data2 model from the ERDA index alone — every image attribute (position, timestamps, exposure, gain, ...) is parsed from the filename. No EXIF is read and no detections are written; new images get the initial status (pending) and the backend pipeline advances them from there. Partner and session rows are looked up, never created: partners match by name by default (partner_by also accepts partnerkey or storagepath), scoped to project_slug (default "onestop").

Requires the CAMALIEN_* env vars (see below). With a pre-built parquet index:

curl -s -X POST http://localhost:8000/camalien/write \
  -H "Content-Type: application/json" \
  -d '{
    "index_path": "./erda_index.parquet",
    "partner": "Aarhus University",
    "session_name": "denmark",
    "year": 2025
  }' | jq

Or run a fresh SFTP index and write in a single job (requires ERDA_SHARELINK_CAMALIEN):

curl -s -X POST http://localhost:8000/camalien/deploy \
  -H "Content-Type: application/json" \
  -d '{
    "remote_path": "/storage/onestop/2025/denmark",
    "partner": "Aarhus University",
    "session_name": "denmark",
    "year": 2025
  }' | jq

Both return a job_id; poll /index/status/{job_id} until completed:

{
  "status": "completed",
  "result": {
    "files_indexed": 5230,
    "images": 5230,
    "images_new": 4100,
    "images_written": 4100
  }
}

The same ingest is available as a script, pipelines/camalien_write.R, driven entirely by env vars (CAMALIEN_PARTNER, CAMALIEN_SESSION_NAME, CAMALIEN_YEAR, INDEX_PATH, ...) — see the script header for the full list.

Environment variables

Variable Description Required for Default
ERDA_SHARELINK_AMI Sharelink for the AMI ERDA workgroup /index/*, /sourceimages/*, /pipeline/* --
ERDA_SHARELINK_CAMALIEN Sharelink for the CamAlien ERDA workgroup /camalien/deploy --
INDEX_PATH Output parquet for index All endpoints reading the index ./erda_index.parquet
API_KEY API key for auth via the X-API-KEY header Always, unless API_ALLOW_NO_AUTH=true --
API_ALLOW_NO_AUTH Set to "true" to run without auth in local dev. The API refuses to start when neither this nor API_KEY is set Local dev only --
API_DAEMONS Concurrent async job workers (mirai daemons), 1-8 Async endpoints 2
API_GW_KEY Client-side only. Sent as the Authorization header when calling through the API Gateway; separate from X-API-KEY Clients calling through the gateway --
AMI_HOST PostgreSQL host AMI DB write --
AMI_USER PostgreSQL user AMI DB write --
AMI_PASSWORD PostgreSQL password AMI DB write --
AMI_DB PostgreSQL database name AMI DB write ami_ias
CAMALIEN_HOST camalien PostgreSQL host CamAlien ingest (/camalien/*) --
CAMALIEN_DB camalien database name CamAlien ingest camalien
CAMALIEN_USER camalien PostgreSQL user CamAlien ingest --
CAMALIEN_PASSWORD camalien PostgreSQL password CamAlien ingest --
CAMALIEN_SSLROOTCERT Root CA cert path for the camalien connection CamAlien ingest --

Deployment

The API is packaged as a Docker image and runs on the ECOS EC2 estate. CI builds registry.gitlab.au.dk/ecos/tools/r-pkgs/erdatools and pushes :next from the dev branch (test host, Cloud Map namespace ecos.test) and :$TAG plus :latest from a v* git tag (production host, ecos.prod).

The full operator runbook -- AWS Secrets Manager entries, EC2 host setup, and the optional API Gateway front end -- is in deploy/README.md, together with the versioned deploy/run_api.sh that goes on the host (plus an optional api.env; see deploy/api.env.example).

Two things to know before pointing a client at a deployed instance:

Performance

Benchmarks for the heavy-I/O pipeline scripts, which cover different trade-offs between speed, disk usage, and ERDA load.

Indexing (index_erda.R)

SFTP download (download_imgs.R)

HTTP EXIF (read_exif_http.R)