Package 'ecospgr'

Description

source_url is stored in meta.dataset_version.source_meta, which warehouse_reader can read, so any query string is stripped here — a download URL that carries an API key must not become catalog metadata.

Usage

epg_build_source_meta(source_url, metadata_url = NULL, extra = list())

Arguments

Value

A named list of source metadata.

Description

Upstream services silently cap responses (WFS at count, ArcGIS at maxRecordCount), so a load can succeed while holding only part of the data. Call this with the count the source reported before publishing.

Usage

epg_check_feature_count(n_loaded, n_expected, tolerance = 0, what = "dataset")

Arguments

Value

TRUE invisibly when the count is acceptable.

Description

Reads optional JSON from a file path and/or an inline JSON string and merges them (inline takes precedence over file).

Usage

epg_collect_source_meta_overrides(json_file_path = NULL, json_inline = NULL)

Arguments

Value

A named list of override fields (may be empty).

Description

Uses glue_sql() for safe SQL interpolation. Identifier components are validated to contain only safe characters (alphanumeric + underscore).

Usage

epg_create_partition(con, dataset, ver_id, geom_type = "MultiPolygon")

Arguments

Description

Looks for schemaLocation hints in the GML header, then falls back to filename-based matching and sibling XSD files.

Usage

epg_detect_xsd_for_gml(gml_path)

Arguments

Value

Path to the XSD file, or NA_character_ if none found.

Description

Returns a named list of credential query parameters for Datafordeler URLs. Prefers API-key auth (⁠apikey=⁠) when DFD_API_KEY is set; falls back to legacy username/password (⁠username=…&password=…⁠). Errors if neither is available.

Usage

epg_dfd_auth_params(
  api_key = Sys.getenv("DFD_API_KEY", ""),
  username = Sys.getenv("DFD_USERNAME", ""),
  password = Sys.getenv("DFD_PASSWORD", "")
)

Arguments

Value

Named list with either apikey OR username + password.

Description

Returns a URL-encoded query-string fragment for the credentials returned by epg_dfd_auth_params(), e.g. "apikey=abc123" or "username=u&password=p". Does not include a leading ⁠?⁠ or &.

Usage

epg_dfd_auth_qs(...)

Arguments

Value

Character scalar.

Description

Drops the version's partition (⁠vector.<dataset>_v<id>⁠), any staging table a failed run left behind, and its meta.dataset_version row, all in one transaction. Use it to take a bad load — truncated, duplicated, loaded with a stale loader — out of the warehouse before re-loading the same tag.

Usage

epg_drop_version(
  con,
  topic,
  dataset,
  version_tag,
  on_latest = c("error", "previous", "clear")
)

Arguments

Details

If rows for the version survive anywhere in ⁠vector.<dataset>⁠ (a partition not named by the pipeline's convention, say), the foreign key from the data to meta.dataset_version refuses the delete and the whole transaction rolls back, so a version is never removed from the catalog while its rows remain.

Removing the version that ⁠vector.<dataset>_latest⁠ serves would leave that view empty, so on_latest must say what happens instead.

COGs in raster.cog_asset are kept: that table records its source version without a foreign key, so a COG stays a valid historical artefact. They are reported, so you know to rebuild them.

Value

A list with version_id, rows (rows removed), was_latest, and latest (the tag the latest flag was moved to, or NA if it was not moved), invisibly.

Description

Errors if topic does not exist, rather than silently inserting nothing and letting a later step fail with an opaque message.

Usage

epg_ensure_dataset(con, topic, dataset)

Arguments

Description

Ensure a topic row exists in meta.topic

Usage

epg_ensure_topic(con, topic, description = NULL)

Arguments

Description

Finalize a dataset version: mark latest and drop staging

Usage

epg_finalize_version(
  con,
  topic,
  dataset,
  version_tag,
  stage_tbl,
  set_latest = TRUE
)

Arguments

Description

Applies iconv() only to strings that are not already valid UTF-8. Converting unconditionally double-encodes correct UTF-8 — ø (⁠C3 B8⁠) becomes ⁠ø⁠ — and because Latin-1 has no invalid bytes, iconv() never errors to warn you. It just corrupts.

Usage

epg_fix_encoding(x, from = "latin1")

Arguments

Value

x with invalid entries re-decoded.

Description

Insert normalized geometries from staging into a vector partition

Usage

epg_insert_normalized(
  con,
  dataset,
  ver_id,
  stage_tbl,
  geom_transform =
    "ST_Multi(ST_CollectionExtract(ST_MakeValid(s.geom), 3))::geometry(MultiPolygon,25832)",
  delete_before_insert = FALSE
)

Arguments

Value

The number of rows inserted (invisible).

Description

Returns the value following flag. A flag that is absent yields default. A flag that is present but malformed — repeated, or given without a value — is an error rather than a silent fallback to default, because these loaders run unattended and a typo must not quietly load with defaults.

Usage

epg_parse_arg(flag, default = NULL, args = commandArgs(trailingOnly = TRUE))

Arguments

Value

The value following the flag, or default when the flag is absent.

Description

If value starts with @, the remainder is treated as a file path whose contents are read and parsed as JSON.

Usage

epg_parse_json_arg(value, label)

Arguments

Value

Parsed JSON as an R list, or NULL if value is empty/NULL.

Description

Parse a JSON string

Usage

epg_parse_json_payload(payload, label)

Arguments

Value

Parsed JSON as an R list.

Description

Persist source metadata on a dataset version

Usage

epg_persist_source_meta(con, ver_id, source_meta)

Arguments

Description

Reads PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD from the environment. PGDATABASE, PGUSER and PGPASSWORD are required: this package is shared across warehouses, so guessing a database or falling back to a default superuser password risks connecting somewhere unintended.

Usage

epg_pg_connect(application_name = NULL)

Arguments

Value

A PqConnection object.

Description

Year-vintage datasets have several compose services sharing one dataset name and differing only by version tag (2024, 2025, 2026). Loading an older vintage must not leave it holding is_latest.

Usage

epg_promote_latest(con, topic, dataset, year = format(Sys.Date(), "%Y"))

Arguments

Details

The rule is "the newest eligible vintage wins". Crucially, when no eligible version exists — which is the situation every 1 January, before the new year's data is published — the current flag is left alone. Clearing it instead would leave ⁠vector.<dataset>_latest⁠ returning zero rows, silently, until someone published a new vintage.

The flag itself is moved by meta.set_latest(), so the database keeps a single implementation of what "latest" means.

Value

The version tag now flagged latest, or NA_character_ if the flag was left as it was.

Description

Read and parse a JSON file

Usage

epg_read_json_file(path, label)

Arguments

Value

Parsed JSON as an R list, or NULL if path is empty/NULL.

Description

Replaces the value of any known credential query parameter with REDACTED, and strips userinfo from the authority. Use this on every URL that reaches a log message, an error, or meta.dataset_version.

Usage

epg_redact_url(url)

Arguments

Details

Unparseable input is reduced to its path, never returned verbatim, so a malformed URL cannot leak a secret through this function.

Value

url with credential values replaced by REDACTED.

Description

Register a new dataset version

Usage

epg_register_version(
  con,
  topic,
  dataset,
  version_tag,
  on_conflict = c("error", "reuse")
)

Arguments

Value

The dataset_version.id (integer).

Description

Reads S3_ENDPOINT, S3_REGION, S3_BUCKET, S3_ACCESS_KEY and S3_SECRET_KEY, the ECOS secret key names, and RAW_PREFIX for the key prefix of raw files.

Usage

epg_resolve_s3_config(default_prefix = "")

Arguments

Details

The older ⁠SCW_*⁠ and ⁠RAW_*⁠ storage names are rejected rather than read, so a stale env file fails instead of being half-applied.

Value

A named list with components endpoint, region, bucket, prefix, access_key, secret_key, and configured (logical).

Description

Convenience wrapper that calls epg_ensure_topic(), epg_ensure_dataset(), epg_register_version(), epg_set_raw_file_url(), epg_persist_source_meta(), epg_create_partition(), epg_write_staging(), epg_insert_normalized(), and epg_finalize_version() inside dbWithTransaction().

Usage

epg_run_pipeline(
  con,
  sf_obj,
  topic,
  dataset,
  version_tag,
  topic_description = NULL,
  source_meta = NULL,
  raw_url = NA_character_,
  geom_type = "MultiPolygon",
  geom_transform =
    "ST_Multi(ST_CollectionExtract(ST_MakeValid(s.geom), 3))::geometry(MultiPolygon,25832)",
  delete_before_insert = FALSE,
  set_latest = TRUE,
  on_conflict_version = "error",
  expected_features = NA_integer_,
  count_tolerance = 0,
  allow_empty = FALSE
)

Arguments

Details

Everything that touches the database happens inside the transaction, so a failure at any step leaves no registered version, no partition and no half-loaded data. Do the download before calling this.

Value

The dataset_version.id (invisible).

Description

Lowercases, replaces non-alphanumeric characters with underscores, and strips leading/trailing underscores.

Usage

epg_sanitize_name(x)

Arguments

Value

A sanitized character string.

Description

Credentials are stripped before the URL is stored: meta is readable by warehouse_reader, so a query string must never reach this column.

Usage

epg_set_raw_file_url(con, ver_id, raw_url)

Arguments

Description

Danish sources ship shapefiles both as Latin-1 and as UTF-8. Forcing ENCODING=ISO-8859-1 overrides a .cpg sidecar that says otherwise, and silently mojibakes every æ, ø and å into the attrs jsonb — where nothing downstream can tell it from real data.

Usage

epg_shapefile_encoding_opts(shp_path, fallback = "ISO-8859-1")

Arguments

Details

So: trust an explicit .cpg declaration when there is one, and only assume Latin-1 when the shapefile declares nothing.

Value

A character vector of GDAL open options for sf::st_read().

Description

Upload raw files to S3-compatible object storage

Usage

epg_upload_raw_files(files, s3_config, on_error = c("stop", "warn"))

Arguments

Value

The S3 URI(s) of uploaded files (character vector), or NA_character_ for files that failed when on_error = "warn".

Description

Returns the scheme, authority and path only. Use for the source_url recorded in meta.dataset_version.source_meta when the query string carries credentials.

Usage

epg_url_without_query(url)

Arguments

Value

url without its query string or fragment.

Description

Creates the stage schema if needed, writes the sf object via st_write, verifies creation, normalizes the geometry column name to geom, and returns a row count.

Usage

epg_write_staging(con, sf_obj, dataset, ver_id)

Arguments

Value

A list with stage_tbl (qualified name like "stage.foo_s_42") and n_rows (count of rows with non-NULL geometry).