tools::SIGKILL is not
defined, so the r-universe Windows check passes again./pipeline/run and /pipeline/deploy no longer ingest IoT edge data by default. They
take a new ingest_edge flag (default false), so IoT deployments get only their
sourceimages (and sessions) written and the cloud pipeline can process the images. Send
ingest_edge: true to also write the edge tracks, detections and classifications as
before; the job result reports edge_ingested. admin/deploy_project_year.R passes the
flag through via INGEST_EDGE=true. Mixing edge ingest with the cloud pipeline on the
same deployment can duplicate or overwrite results (#7).Long-running API jobs are observable (#5). /index/status/{id} returns stage,
progress, last_progress_at and worker_pid, and /index/jobs adds stage and
last_progress_at, so a working job can be told apart from a hung one. A worker that
dies mid-job (e.g. OOM kill) is reported as failed with a readable error instead of
completed. Worker messages and progress bars now reach the container logs.
Edge CSV downloads check the exit status and the local file, retry once, then abort
rather than hanging or skipping a file silently. The SSH ControlMaster sends keepalives,
and et_sftp_batch() gains a timeout argument.
et_db_deployment_path() no longer puts a partner level in IoT paths. IoT traps upload
to /{project}/{code}/ (e.g. /ias/BIO25/), so every IoT deployment with a partner
folder resolved to a non-existent directory, and /pipeline/deploy finished as
completed with 0 images. Re-run any IoT deployment deployed before this fix.
et_index_dir() aborts when the root remote_path does not exist instead of returning
an empty index, so a wrong path fails the job rather than reporting success.
The API now refuses to start when API_KEY is unset. Previously it forwarded every
request unauthenticated in that case, so a container started without a key served the
database-writing endpoints to anyone who could reach the port. Set API_KEY, or set
API_ALLOW_NO_AUTH=true to opt in to the old open behaviour for local development.
IoT edge ingest writes the tracker identity as tracy / luke instead of kbe-tracker /
v1.0. This shipped unversioned after 0.12.0 and invalidates existing
uq_tracks_session_edge_algorithm rows.
The API no longer reads ERDA_SHARELINK. A deployment that set it must set
ERDA_SHARELINK_AMI and/or ERDA_SHARELINK_CAMALIEN instead; see New features
below for why there is no fallback. The variable is unchanged for the pipelines/
and analysis/ scripts.
The API no longer reads any ERDA remote path from the environment.
ERDA_SFTP_REMOTE_DIR was the default remote root for /index/full and
/index/update, so a request naming no path indexed whatever the host was configured
for; those endpoints now answer 400 unless the request carries remote_path or a
deployment_id to look it up from. The remote path is request data — it says what to
index on this call — and a host default made the same request return different trees on
different hosts. Worse, it silently outranked the deployment_id lookup: with the
variable set, posting only a deployment_id indexed the configured root instead of that
deployment, with no error. POST /camalien/deploy, which has no deployment lookup, now
requires remote_path. The variable is unchanged for the pipelines/ and analysis/
scripts, where the invocation is the request.
POST /camalien/deploy takes remote_path instead of remote_dir, lining the
parameter up with the argument of et_index_dir() it is passed to.
POST /sourceimages/build, /sourceimages/write and /sourceimages/diff no longer
accept a remote_dir body field. They resolved it and then never used it:
et_build_sourceimages() has no such argument and builds each URL from http_base, the
sharelink, and the directory recorded in the index. The field was documented in the
OpenAPI spec and silently ignored, so removing it changes no response.
All AMI database connections now use sslmode = "verify-full". Five API call sites passed
sslmode = "disable" while the rest used the secure default, so the same RDS instance was
reached with and without TLS depending on which endpoint was called.
API key comparison is constant-time.
The container image runs as a non-root user, ships a HEALTHCHECK, bundles the AWS RDS
eu-north-1 CA certificate, and no longer leaves the package source at the working
directory — which previously left the API one dependency away from silently starting in
pkgload::load_all() development mode.
Added .dockerignore. The build context previously included the 285 MB output/ directory
and .git.
INDEX_PATH defaults to /data/erda_index.parquet in the image so the index survives a
container restart when /data is a volume.
Added .gitlab-ci.yml: runs the test suite on push, and builds and pushes the image to
registry.gitlab.au.dk (:next from dev, :$TAG and :latest from v* tags).
Added deploy/ with an EC2 run script, an example env file, and an operator runbook covering
AWS Secrets Manager, host setup, and API Gateway.
deploy/run_api.sh now takes the environment as its first argument (next or latest)
and the image tag as an optional second one, so test and production read separate Secrets
Manager entries. Previously the secret ids were hard-coded and identical for both hosts, which
would have pointed a test deployment at the production databases. It also refuses any secret id
that does not end in the host's environment, reads each ERDA sharelink from its own
SFTP_USER/SFTP_PASS secret rather than from keys on the API secret, and accepts per-host id
overrides in deploy.conf for a rename that has to roll out host by host.
Fixed every API endpoint except /health returning 500 (could not find function) in the
container. The image's erdatools::et_api_run() loads the namespace without attaching it,
and plumber.R skipped library(erdatools) whenever the namespace was already loaded. It
now attaches the namespace in that case. CI's smoke test also checks /index/jobs, since
/health calls no erdatools function and could not catch this.
The image now installs libuv1. The fs binary from Posit Package Manager links it, so
without it every SFTP-backed job (/index/*, /pipeline/*, /camalien/deploy) failed
with libuv.so.1: cannot open shared object file. The build now loads every runtime
package rather than only checking it is installed, so a missing system library fails the
build instead of the first job.
ERDA sharelinks are now resolved per workgroup. The AMI and CamAlien corpora
live in different ERDA workgroups and so need different sharelinks, but every
endpoint previously read a single process-wide ERDA_SHARELINK, meaning one
container could only ever reach one workgroup. Endpoints now resolve
ERDA_SHARELINK_AMI or ERDA_SHARELINK_CAMALIEN for their own scope. There
is deliberately no fallback to the unscoped ERDA_SHARELINK: an unconfigured
scope returns 400 rather than authenticating against the other workgroup and
indexing the wrong tree. The sharelink is now the only ERDA configuration the
API holds — it is a credential, and it fixes the root that each request's
remote_path resolves against. The unscoped variable is still what the
pipelines/ and analysis/ scripts use; only the API requires the scoped
names.
API_DAEMONS sets the number of concurrent async job workers (default 2, previously
hard-coded).
admin/deploy_project_year.R and analysis/cleanup_mambo_snapshots.R send an Authorization
header from API_GW_KEY when set, for use behind an API Gateway Lambda authorizer. Both
headers remain optional, so direct and tunnelled access are unchanged.