Internals

disclaimer()

Print the OBIS interpretation disclaimer.

OBIS_DISCLAIMER

The interpretation disclaimer from the OBIS data policy, reproduced verbatim.

Printed by disclaimer.

RECORD_SELECTIONS

The three values absence, dropped and event accept.

• :exclude — leave them out. The API default.
• :include — return them alongside ordinary records.
• :only — return only these records.

Used by the absence, dropped and event keywords. The distinction matters: :include and :only answer different questions, and a boolean keyword could not express both.

FieldSpec(name, type, kind)

One column of a canonical schema.

• name: column name, matching the API's field name so that documentation transfers.
• type: concrete element type. The column's element type is Union{Missing,type} unless kind supplies a non-missing default.
• kind: how the raw JSON value is converted. See coerce_field.

OCCURRENCE_SCHEMA

Core columns of an occurrence table, in column order.

Includes every field OBIS's quality pipeline adds or interprets, plus the record-level Darwin Core terms that carry the identity, taxonomy, position and time of an observation. Verbatim provider fields outside this list — including year, month and day, whose interpreted counterpart is date_year — are kept in the extra column.

TAXON_SCHEMA

Core columns of a taxon or checklist table.

records is populated by /checklist and absent from /taxon, so it is missing for the latter rather than a column that appears and disappears between the two.

DATASET_SCHEMA

Core columns of a dataset table.

license and license_url are not API fields. OBIS reports the licence as free prose in intellectualrights, so the package derives a normalized identifier and a canonical URL from it and keeps the original text alongside. See normalize_license.

NODE_SCHEMA

Core columns of a node table.

INSTITUTE_SCHEMA

Core columns of an institute table. id is the OceanExpert identifier.

AREA_SCHEMA

Core columns of an area table.

COUNTRY_SCHEMA

Core columns of a country table.

coerce_field(spec, value)

Convert one raw JSON value into the type its FieldSpec declares.

Returns missing when the value is absent or cannot be represented, except for :flagset and list kinds, which return an empty collection. An empty flag set means "no quality flags were raised", which is a fact about the record rather than a gap in it, so flags is never missing and set membership tests need no missing-handling.

Convert to Float64, accepting the integers JSON produces for whole numbers.

Parse a JSON array of strings; a bare string becomes a one-element vector.

Pull one key out of each object in a JSON array of objects.

column_type(spec) -> Type

Element type of the column a FieldSpec describes.

Collection-valued columns are never missing: an empty collection already says "nothing here", and adding missing would force every caller to handle two kinds of emptiness.

schema_names(schema) -> Vector{Symbol}

Column names of a schema, in order.

coerce_epoch_ms(v) -> Union{Missing,DateTime}

Convert a Unix timestamp in milliseconds to a DateTime.

The API's date_start, date_mid and date_end are milliseconds, not seconds, so unix2datetime applied directly would place every record about fifty thousand years into the future.

coerce_iso_datetime(v) -> Union{Missing,DateTime}

Parse the timestamp forms OBIS metadata uses.

Dataset timestamps are ISO 8601 with milliseconds and a Z suffix, while occurrence modified values use a space separator and no zone, so both are accepted.

coerce_flagset(v) -> Set{String}

Parse quality flags into a set.

The API sends a JSON array; the CSV downloads use a comma-separated string. Both are accepted so that flag handling is the same whichever surface the values came from. A set is the honest representation: flags are unordered and each is present at most once.

build_table(records, schema, meta) -> OBISTable

Turn decoded JSON records into a typed table with a fixed column set.

Every core column is materialized whether or not any record carried the field, which is what makes the schema stable across queries. Remaining fields are collected per row into extra.

json_to_julia(v)

Convert a JSON value into plain Julia containers.

Conversion is deliberate rather than storing the JSON3 views directly: those views reference the parsed buffer, so keeping them would pin the whole response in memory for as long as any row survives.

Objects become Dict{String,Any}, matching the field names the API documentation uses. The top level of a row's extra is keyed by Symbol instead, because those names are column names.

empty_table(schema, meta) -> OBISTable

An empty table with the full schema.

A query that matches nothing still returns every column, so downstream code that selects columns behaves the same whether or not there were records.

take_rows(t, n) -> OBISTable

The first n rows of a table, keeping the schema and provenance.

concat_tables(tables) -> OBISTable

Concatenate tables that share a schema into one.

Used to assemble a complete result from paged responses. All inputs must have identical column names, which they do by construction because the schema is fixed per endpoint.

api_get(endpoint; cfg, params...) -> JSON3 payload

Fetch and decode one API response, consulting the cache when one is configured.

This is the single point every endpoint goes through, so caching, courtesy pacing and error interpretation apply uniformly.

raw_request(endpoint, params; cfg) -> (body::String, headers::Dict)

Perform one request with retries and backoff, returning the raw response body.

Raises OBISConnectionError when the retry budget is exhausted and OBISAPIError for a permanent failure. Response content is not inspected here; that is check_response's job.

build_url(endpoint, params; base_url) -> String

Join an endpoint path and query parameters into a request URL.

request_headers(cfg) -> Vector{Pair{String,String}}

Headers sent on every request.

The User-Agent names the package, its version and its repository so OBIS can tell which client is generating traffic and reach the maintainers if it misbehaves.

retryable(status, body) -> Bool

Whether a failed response is worth retrying.

429 and 5xx are transient in general, but the API also answers permanent client mistakes with 5xx and an explanatory error field in the body, and retrying those only wastes the server's time.

backoff_delay(attempt, cfg, retry_after) -> Float64

Delay before retry number attempt, honouring a Retry-After header when the server sends one and otherwise doubling from backoff_base up to backoff_max.

A small deterministic jitter spreads retries from concurrent sessions.

throttle!(cfg)

Sleep, if needed, so consecutive requests are at least request_gap seconds apart.

check_response(payload, endpoint, params) -> payload

Inspect a decoded response body for failures the HTTP status did not report.

The API returns NAME_NOT_FOUND inside an HTTP 200 for an unmatched scientific name, and returns other errors in the same error field. Both are raised here so that an empty result set always means "no records matched", never "the query was wrong".

results_of(payload) -> Vector

Extract the results array from a standard {total, results} envelope.

Several endpoints answer with a bare object or a bare array instead, so this returns an empty vector rather than raising when the key is absent, and callers that expect another shape read the payload directly.

total_of(payload, fallback) -> Int

Read the total field of a response envelope, falling back when it is absent.

Transport

Function that performs one HTTP GET and returns (status, headers, body).

Swappable so the test suite can serve recorded fixtures without reaching the network.

default_transport(url, headers, timeout) -> (status, headers, body)

Perform one HTTP GET with HTTP.jl.

status_exception = false because the API uses 4xx and 5xx bodies to explain failures, and those bodies are more informative than the status code.

DOWNLOADER

Function that writes the body at url to the file at path, given request headers.

The second seam beside TRANSPORT, and separate from it because a bulk export is a file rather than a response: it goes to disk as it arrives instead of being held in memory, so it cannot share the transport's (status, headers, body) shape. Swappable for the same reason — the export route has to be testable without reaching AWS.

default_downloader(url, path, headers) -> String

Stream url into path with HTTP.jl.

update_period = Inf silences HTTP.jl's own progress logging: the package reports progress itself, and two indicators disagreeing is worse than neither.

http_error_advice(status, body) -> String

Turn an HTTP status into a sentence the caller can act on.

Timestamp of the last request, used to honour request_gap.

build_params(; kwargs...) -> QueryParams

Validate the filter keywords shared by the OBIS endpoints and render them for the wire.

Keywords map onto API parameters, with three deliberate differences:

• aphiaid is accepted as an alias for taxonid, because the identifier is called an AphiaID everywhere except in the query string.
• absence, dropped and event take :exclude, :include or :only rather than a boolean, matching the three behaviours the API actually offers.
• flags and exclude accept any case and any iterable, and are normalized to the upper-case forms the API matches on.

QueryParams

An ordered, validated set of query parameters.

Order is kept stable so that the cache key derived from a query is reproducible across sessions and Julia versions.

selection_value(name, sel) -> Union{Nothing,String}

Translate one of RECORD_SELECTIONS into the string the API expects, or nothing when the parameter should be omitted entirely.

validate_geometry(wkt) -> String

Check that wkt looks like well-formed WKT before sending it.

The API answers invalid WKT with HTTP 200 and an empty result set and no error field, so an unchecked typo silently reads as "no records in this area". This performs a structural check — a recognized geometry keyword and balanced parentheses — not a full parse.

format_date(name, value) -> String

Render a date as the YYYY-MM-DD string the API requires.

Accepts Date, DateTime, or a string already in that form. A malformed date is one of the few inputs the API does report, but with HTTP 500 and a body that still looks like a valid empty result, so it is checked here instead.

validate_depth(name, value) -> Int

Check a depth bound. Depths are metres below the surface and increase downwards.

validate_uuid(name, value) -> String

Check that an identifier looks like the UUID the API expects.

Dataset and node identifiers are UUIDs. A malformed one matches nothing and the API reports no error, so the shape is checked before sending.

validate_size(value) -> Int

Check a page size against the API maximum.

validate_fields(fields) -> String

Check a fields allow-list and render it for the wire.

Two behaviours make this worth checking locally: the API silently discards unknown field names, so a typo yields a missing column rather than an error; and flags cannot be selected through fields at all, even though it is present in every unrestricted record.

sorted_pairs(q::QueryParams) -> Vector{Pair{String,String}}

Parameters in a canonical order, for hashing a query into a cache key.

CitationEntry

One dataset's contribution to a result, with everything a citation needs.

citation_entries(t) -> Vector{CitationEntry}

Gather per-dataset citation metadata for the datasets present in a result.

Dataset metadata is fetched with the result's own query, so this costs one request regardless of how many datasets are involved.

format_citation(e::CitationEntry) -> String

Render one dataset citation in the format the OBIS data policy specifies.

Follows the policy template: the provider's citation, then the OBIS availability statement and the access date. Where the provider supplied no citation, one is assembled from the title and publication year so that the dataset is still identifiable — a missing citation string is not a reason to leave data uncredited.

bibtex_escape(s) -> String

Escape the characters BibTeX treats specially, so a provider's citation text cannot break the generated file.

dataset_counts(t) -> Dict{String,Int}

How many records in a result came from each dataset.

dataset_info(params; ids) -> Dict{String,DatasetInfo}

Build a dataset_id to rights mapping covering ids.

Strategy is chosen by cost. When the occurrence query carries filters, one /dataset call with those same filters covers every dataset the query can touch. When it carries none — which would otherwise pull metadata for all of OBIS — the datasets actually present in the result are fetched individually instead.

dataset_filters(params) -> QueryParams

Keep only the parameters /dataset filters on.

Pagination and occurrence-only parameters are dropped. The result may match more datasets than the occurrence query strictly touches, which is harmless for a lookup table.

attach_licenses(t, params) -> OBISTable

Fill the license, license_url and dataset_citation columns of an occurrence table.

Called automatically by occurrence; pass licenses = false there to skip it. The columns exist either way, so the table's shape does not depend on the choice.

finalize_dataset_table(t) -> OBISTable

Fill the derived license and license_url columns from the published rights text.

fetch_dataset_records(meta, ids) -> Dict{String,Any}

Retrieve dataset metadata covering ids, reusing the result's query where possible.

DatasetInfo

Rights and citation metadata for one dataset, as attached to occurrence records.

guard_query_size(params)

Refuse to page through a query larger than config().api_record_limit.

The refusal is deliberate. OBIS asks that large volumes be taken from the bulk export rather than the API, and the two routes do not cover the same records — absence and dropped records are reachable only through the API. Switching routes silently would therefore change the answer, so the choice is left to the caller, with the alternatives spelled out.

stats_filters(params) -> QueryParams

Drop pagination and projection parameters, which change what a page contains but not how many records match.

parse_licenses_tsv(text) -> OBISTable

Parse the export licence table.

The format is tab-separated with a header, which is why this does not need a CSV dependency. Citation strings are sometimes quoted, so a record is treated as complete only once it has the expected number of fields and an even number of quote characters; a citation containing a line break is therefore reassembled rather than truncated.

raw_request_absolute(url) -> (body, headers)

Fetch an absolute URL with the package's retry, backoff and identification rules.

Used for the export bucket, which is not under the API base URL.

download_file(url, path) -> String

Download to a temporary file and move it into place, so an interrupted transfer cannot leave a truncated file that looks complete.

EXPORT_COLUMN_SOURCES

Where each occurrence column comes from in a GeoParquet export, for the columns that are not simply interpreted.<name>.

The export nests the provider's own terms under source and the quality pipeline's output under interpreted. interpreted is what the API serves, so it is the default source for every canonical column. The exceptions are the four that sit at the top level of the file, the two whose name differs there, the three filled from the licence table, and the two the export does not carry at all. A nothing means the column cannot be read from the file.

interpreted.license exists in the schema but was empty in every export examined, so the licence is taken from export_licenses rather than from the row.

export_select(spec) -> String

One entry of the SELECT list that reads a canonical column out of an export file.

Built from OCCURRENCE_SCHEMA rather than written out, so a column added to the schema is read from the export without a second list to remember to update.

cache_key(endpoint, params; base_url) -> String

Hash a query into a stable cache key.

Parameters are sorted before hashing so that keyword order does not change the key, which is what lets the same query written two different ways hit the same entry.

cached_body(cache, endpoint, params) -> Union{Nothing,String}

Return a stored response body, or nothing when the query is not cached.

store_body!(cache, endpoint, params, body, headers)

Write a response body and its provenance metadata into the cache.

The OBIS API base URL. The version is part of the path; there is no other server.

Public HTTPS mirror of the OBIS Open Data bucket on AWS.

Repository URL, sent in the User-Agent header so OBIS can identify the client.

Largest page the API accepts for /occurrence.

format_count(n)

Render an integer with thousands separators, for error and log messages.