OpenAQ Transform

A Typescript library for transforming air quality data sources into a single standardized output.

Warning

OpenAQ Transform is a work in progress and may contain breaking changes until reaching a 1.0.0 version

Overview

OpenAQ transform provides a declarative configuration layer to solve common tasks for transforming, normalizing and reshaping air quality measurement data.

Core Concepts

Client

The Client class is the main entry point for defining how Transform fetches, parses, and processes data from an external source. To connect a new data source, you extend one of the platform-specific subclasses such as NodeClient and configure it by setting properties on the class.

At minimum a client needs a provider name and a resource to fetch from:

export class Client extends NodeClient {
  provider = 'example';
  resource = new Resource({ url: 'https://api.example.com/data' });
}

Calling client.load() on an instance will fetch the resource, parse the response, and return the processed data in a format ready for ingestion.

For sources that expose separate endpoints for locations and measurements, resource can be an indexed object. Each key maps to a named resource, and the results are accumulated and made available to subsequent resource reads via DataContext:

export class Client extends NodeClient {
  provider = 'example';
  resource = {
    locations: new Resource({ url: 'https://api.example.com/locations' }),
    measurements: new Resource({ url: 'https://api.example.com/measurements' }),
  };
}

The reader and parser properties control how each resource is fetched and transformed. The values default to "api" and "json" respectively, which covers most REST APIs returning JSON. Custom readers and parsers can be provided as functions, or as indexed objects to use different strategies per resource.

Secrets

Client (and exported subclasses like NodeClient) accept and optional secrets object, for storing API keys, tokens or credentials. Client is generic over the shape of secrets, so each client can declare its own type.

Basic Usage

export class ExampleClient extends NodeClient<{ apiKey: string }> {
  provider = 'example';
  resource = {
    locations: new Resource({
      url: 'https://api.example.com/locations',
      auth: {
        type: 'APIKey',
        position: 'query',
        key: 'token',
        value: () => this.secrets.apiKey,
      },
    }),
  };
}

Note

this.secrets isn't populated until after class field initializers run, so any resource config that reads from it must access it through a function like () => this.secrets.apiKey rather than reading it directly. This defers the lookup until the value is actually needed, by which point secrets is guaranteed to be set (and reads the current value, if secrets are ever updated later). Fields that support this pattern (like auth.value) accept either a plain value or a function for exactly this reason.

Resource

The Resource class defines an external data source, either a remote URL or an uploaded file, and provides it to a reader along with any configuration needed to fetch it.

new Resource({
  url: 'https://api.example.com/data'
})

File resources

A Resource can also wrap an uploaded File object. Note that parameters and body cannot be used with file resources.

new Resource({ file: uploadedFile })

Parameters

The parameters option generates one URL (and optional body) per parameter object. In its simplest form it is a static array:

new Resource({
  url: 'https://api.example.com/data?page=:page',
  parameters: [{ page: 1 }, { page: 2 }, { page: 3 }]
})

Parameters can also be a function that receives the accumulated DataContext, the combined result of all previously loaded resources, and returns an array of parameter objects. This allows later resources to be parameterised using values fetched by earlier ones:

new Resource({
  url: 'https://api.example.com/locations/:id/measurements',
  parameters: (d) => d.locations.map(location => ({ id: location.id }))
})

Parameters can also be a JMESPath expression evaluated against the same DataContext:

new Resource({
  url: 'https://api.example.com/locations/:id/measurements',
  parameters: jmespath('locations[*].{id: id}')
})

Note

The jmespath() helper function is the preferred method to define a jmespath query. A literal ParseExpression object can also be provided. The example below is functionally identical to that above.

new Resource({
  url: 'https://api.example.com/locations/:id/measurements',
  parameters: { type: 'jmespath', value: 'locations[*].{id: id}' }
})

Context

context merges additional fields onto every row returned by a resourcem useful for attaching data that isn't present in the raw response, such as tagging rows with the parameters used to fetch them.

context accepts either a static object, applied to every URL:

ts new Resource({ url: 'https://api.example.com/data', context: { provider: 'example' } })

Or a function of the resolved parameters (and DataContext), letting context vary per-request:

new Resource({
  url: 'https://api.example.com/locations/:id/measurements',
  parameters: [{ id: 'A' }, { id: 'B' }],
  context: (params) => ({ locationId: params.id })
})

Fields already present on a row take precedence over context, it only fills in values that are not already there.

Body

For HTTP POST requests, body accepts a string, URLSearchParams, or FormData. Template variables in the body are substituted using the same:key syntax as the URL:

new Resource({
  url: 'https://api.example.com/data',
  body: JSON.stringify({ station: ':station' }),
  parameters: [{ station: 'ABC' }, { station: 'DEF' }]
})

Properties

Property	Type	Default	Description
`url`	`string`		URL template with optional `:key` placeholders
`file`	`File`		Uploaded file object. Mutually exclusive with `url`
`parameters`	`Parameters[] \| function \| PathExpression`		Generates one request per parameter object
`body`	`string \| URLSearchParams \| FormData`		Request body for POST requests. URL resources only
`output`	`"array" \| "object"`		How to combine responses from multiple URLs. See API reader
`readAs`	`"json" \| "text" \| "blob"`		Overrides content-type detection
`context`	`Context \| (params, data) => Context`		Fields merged onto rows from this resource, a static object or a function of the resolved parameters
`strict`	`boolean`	`false`	If `true`, throws on first error. If `false`, errors are passed to `errorHandler`

Authentication

Authentication is configured at the Resource level via the auth parameter. Transform resolves auth at request time. Auth headers are merged into the request headers and query parameter-based auth keys are appended to the generated URL.

Because auth lives on the resource rather than the client, a client with an indexed resource object can authenticate each endpoint differently.

new Resource({
  url: 'https://api.example.com/data',
  auth: {
    type: 'Bearer',
    token: 'abc123',
  },
});

Types

`type`	Required fields	Where it goes
`"APIKey"`	`position`, `key`, `value`	Header, cookie, or query string, depending on `position`
`"Bearer"`	`token`	`Authorization: Bearer <token>` header
`"Basic"`	`username`, `password`	`Authorization: Basic <base64>` header

APIKey

APIKey covers the common case of a provider-issued token passed as a named key/value pair. position decides where that pair is placed:

`position`	Behavior
`"header"`	Sets a request header named `key` with the value `value`
`"query"`	Sets a query string parameter named `key` on every generated URL
`"cookie"`	Sets the `Cookie` header to `value`

####### Examples

API Key Header

new Resource({
  url: 'https://api.example.com/data',
  auth: {
    type: 'APIKey',
    position: 'header',
    key: 'X-API-Key',
    value: () => this.secrets.apiKey,
  },
});

Would result in an equivalent cURL command like:

curl 'https://api.example.com/data' -H 'X-API-Key: abc123'

API Key Query parameter

new Resource({
  url: 'https://api.example.com/data',
  auth: {
    type: 'APIKey',
    position: 'query',
    key: 'token',
    value: () => this.secrets.apiKey,
  },
});

Would result in an equivalent cURL command like:

curl 'https://api.example.com/data?token=abc123'

Note

Query keys are set with URLSearchParams.set, so an API key placed in the query string will overwrite a parameter of the same name produced by parameters on the resource.

Bearer Auth Header

Example

new Resource({
  url: 'https://api.example.com/data',
  auth: {
    type: 'Bearer',
    token: 'abc123',
  },
});

curl 'https://api.example.com/data' -H 'Authorization: Bearer eyJhbGciOi...'

Basic Auth

username and password are joined and base64-encoded into a standard Authorization: Basic header.

Example

new Resource({
  url: 'https://api.example.com/data',
  auth: {
    type: 'Basic',
    username: 'admin',
    password: 'admin',
  },
});

curl 'https://api.example.com/data' -H 'Authorization: Base YWRtaW46YWRtaW4='

Properties

Property	Type	Description
`type`	`"APIKey" \| "Bearer" \| "Basic"`	Authentication strategy
`position`	`"header" \| "query" \| "cookie"`	`APIKey` only. Where the credential is placed
`key`	`string \| () => string`	`APIKey` only. Header or query parameter name. Unused when `position` is `"cookie"`
`value`	`string \| () => string`	`APIKey` only. The credential
`token`	`string`	`Bearer` only. Passed through as-is
`username`	`string \| () => string`	`Basic` only
`password`	`string \| () => string`	`Basic` only

Readers

Transform provides built-in readers to handle common methods of fetching data including HTTP calls using the fetch API and file-based interfaces for Node.js using fs.readFile or file uploads in the browser through the File API.

Readers are used in the context of a Client subclass, where the reader property controls which reader is used when client.load() is called. The reader property accepts:

a string (e.g. "api") — resolved to a built-in reader
a function — a custom reader used for all resources
an object of key → string or function — for indexed resources, one reader per resource key

API reader

The api built-in reader fetches data from one or more URLs defined on a Resource, with support for pagination, content-type detection, and flexible output strategies.

Parameters

Parameter	Type	Default	Description
`resource`	`Resource`	required	Resource instance with `urls`, `output`, `readAs`, and `strict`
`options`	`RequestInit`	`{ method: "GET" }`	HTTP fetch options passed to `fetch()`
`concurrency`	`number`	`3`	Number of URLs fetched in parallel
`errorHandler`	`function`	`undefined`	`(error, strict) => void`; if omitted, errors are logged and thrown when `strict` is set

Output

The resource.output property controls how responses from multiple URLs are combined:

`output`	Behavior
`undefined` (default)	Returns response as-is; single URL → value, multiple URLs → array
`"array"`	Array responses are flattened; object responses are collected; always returns an array
`"object"`	All responses are merged by concatenating nested arrays; always returns an object

Content-type detection

Unless resource.readAs is set explicitly, the reader auto-detects the read format from the Content-Type response header.

Custom reader

A custom reader is a function assigned to the reader property of a Client subclass. It must be an arrow function to correctly the Client this context such as this.readers.

async ({ resource, options }, parser, data) => {
  // ...
}

Custom readers can call built-in readers via this.readers, which is useful for unwrapping API responses before they reach the field mapping stage:

export class Client extends NodeClient {
  provider = 'example';
  resource = {
    measurements: new Resource({ url: 'https://api.example.com/v1/measurements' }),
    locations: new Resource({ url: 'https://api.example.com/v1/locations' }),
  };
  parser = 'json';
  reader = {
    locations: async ({ resource, options }, parser, data) => {
      const res = await this.readers.api({ resource, options }, parser, data);
      return res.results;
    },
    measurements: async ({ resource, options }, parser, data) => {
      const res = await this.readers.api({ resource, options }, parser, data);
      return res.results;
    }
  };
}

In this example, the API returns an object like { results: [...] }. Because resource.output is not set to "array", apiReader returns the object as-is, and the custom reader unwraps .results before returning it for processing.

The data parameter contains the accumulated resource data from previously loaded resources and can be forwarded to this.readers.api when later resources depend on earlier ones. When no dependency exists, it can be passed as {} or omitted.

Parsers

Parsers are used to parse string data as returned from a reader and parse into JavaScript objects for transformation. Parsers are provided to Readers as a dependency to allow the reader to return deserialized data objects.

OpenAQ transform provides pre-built parsers for common serialized data formats such as JSON, csv, and tsv. You can write a custom parser to handle other one-off cases as needed, but the core provided parsers are intended to handle most cases.

Field mappings

After data are read and parsed the transform Client can map fields from the original form to create the standardized output. Data field lookups can be defined in three different ways:

Key lookups from a string e.g. 'locationId', 'datetime'
A path expression using a DSL such a JMESpath to look up values e.g. .coordinates.latitude
A function for dynamically joining, reshaping or otherwise manipulating the field values e.g. (d) => `${d.dateString}T${d.time}Z

For certain fields that accept boolean or number, a literal value matching the appropriate type can be passed as constant in the case that a dynamic lookup or mapping is not applicable.

Geohash derived IDs

Some sources have no stable site identifier. Setting useGeohash = true derives siteId from each row's coordinates instead of looking up locationId:

export class Client extends NodeClient {
  provider = 'example';
  useGeohash = true;
  xGeometry = 'lon';
  yGeometry = 'lat';
}

The site id is a geohash prefixed with gh_, e.g. example/gh_c207nr1ugz. Rows in the same cell resolve to the same location. The hash is always computed from WGS84 coordinates, reprojecting first if geometryProjection names something else.

geohashPrecision (default 10) sets the cell size, and so how far apart two coordinates can be and still count as one location:

Note

Coordinates must be present on every record, including measurement rows. Sources that supply them only in a separate locations resource will throw a MissingAttributeError.

Datetime handling

Transform turns each row's raw datetime value into a normalized, timezone-aware timestamp. This is controlled by four related properties: datetime, datetimeType,datetimeFormat, timezone, and timeEnding.

datetime is a field lookup (key, path expression, or function) that identifies where the timestamp lives in the source row, same as any other field mapping:

datetime = 'observed_at'

Parsing datetimes

datetimeType tells transform how to interpret the value derived from datetime:

`datetimeType`	Description
"string" (default)	A formatted date/time string, parsed using `datetimeFormat` and `timezone`.
"seconds"	A Unix epoch value in seconds (number or numeric string).
"milliseconds"	A Unix epoch value in milliseconds (number or numeric string).

For string timestamps, datetimeFormat defaults to ISO_UTC, which handles ISO-8601, 2026-06-30T22:00:00Z, with or without milliseconds, and with either a Z designator or a numeric offset.

For other shapes, set datetimeFormat to a Luxon format string.

Transform includes some named constants which provide common formats in a convenient variable:

Constant	Matches
`ISO_UTC` (default)	`2026-06-30T22:00:00Z`, 2026-06-30T22:00:00+00:00
`SQL_UTC`	`2026-06-30 22:00:00Z` space separator, Zulu designator
`SQL_NAIVE`	2026-06-30 22:00:00 — no zone info; requires timezone

import { SQL_UTC } from '@openaq/transform/core';

datetime = 'date_added';
datetimeFormat = SQL_UTC;

Important

Set timezone only when the source string carries no time zone information. If the value already ends in 'Z' or an offset (e.g. '-05:00'), supplying timezone throws a TypeError.

datetime = 'observed_at';
datetimeType = 'string'; // default, can be omitted
datetimeFormat = "ISO_NAIVE";
timezone = 'America/Los_Angeles';

For Unix timestamps, set datetimeTypeto seconds or milliseconds and omit datetimeFormat:

datetime = 'observed_at';
datetimeType = 'seconds'; // or 'milliseconds'

Note

datetimeFormat is only used when datetimeType is "string". Setting both datetimeFormat and a datetimeType as "seconds" or "milliseconds" will throw a ConfigError.

Time-beginning vs. time-ending intervals

Timestamps can represent the start of an averaging interval (e.g. a reading at 03:00 represents the average from 03:00–04:00), or thrather than the end. Transform's canonical output is always time-ending. If the source data is time-beginning, set timeEnding = false and transform will shift the parsed timestamp forward by the sensor's averagingIntervalSeconds to produce a time-ending value

timeEnding = false;
averagingInterval = 3600; // required in this case, in seconds

timeEnding	Behavior
true (default)	The parsed datetime value is used as-is.
false	The parsed datetime value is treated as time-beginning and adjusted by `averagingIntervalSeconds` to produce a time-ending.

Parameter and unit mappings

OpenAQ transform provide built-in definitions for common air quality and meteorological parameters, including PM, ozone, NO_x, SO₂, CO, temperature, relative humidity, and pressure. Each definition specifies a canonical parameter name, a standard output unit, and a set of converters that normalize provider data into that unit automatically.

To use a parameter, you supply a parameter mapping that tells transform how to find and interpret values in your source data. Each mapping has three fields:

parameter — The canonical parameter name used by transform (e.g. "pm25", "o3", "temperature").
unit — The unit your source data uses (e.g. "ug/m3", "ppb", "f"). transform will convert this to the standard output unit automatically.
key — The column name (wide format) or the value of the parameter name field (long format, set via parameterNameKey) that identifies this parameter in the source data.

e.g.

{ parameter: 'pm25', unit: 'ug/m3', key: 'pm25' }

Number formats

Global sources may not represent numeric strings the same way: 1,234.5, 1.234,5, 1 234,5 and 1'234.5 all mean the same thing. numberFormat tells transform how to interpret numeric strings before conversion, so they normalize to a plain JavaScript number rather than failing or truncating.

Without this comma digit group markers would evaluate to NaN e.g.

Number('1,234.00')
// NaN

a numeric string that uses a dot for digit group marker would evaluate incorrectly as a decimal value 1.234, when the local representation really means 1234 e.g.

Number('1.234');
// 1.234

numberFormat accepts a DecimalDigitGroup object with two fields:

decimal: the character used as the decimal separator.
digitGroup: the character used to group thousands. Optional; omit it when the source has no grouping separators.

Only certain pairings are valid, reflecting real-world conventions:

`decimal`	Allowed `digitGroup`	Example
`"point"` (default)	`"comma"`, `"space"`, `"apostrophe"`	`1,234.5`
`"comma"`	`"dot"`, `"space"`, `"apostrophe"`	`1.234,5`
`"arabic"`	`"comma"`, `"space"`	`١٬٢٣٤٫٥`
`"interpunct"`	`"comma"`	`1,234·5`

The default is { decimal: 'point' }, a decimal point with no grouping separator.

Client-level

Set numberFormat on the client to apply it to every numeric field parsed from that source, including coordinates and interval values:

export class Client extends NodeClient {
  provider = 'example';
  numberFormat = { decimal: 'comma', digitGroup: 'dot' };
}

Per-parameter

A parameter mapping can override the client default when a single field uses a different convention from the rest of the response:

parameters = [
  { parameter: 'pm25', unit: 'ug/m3', key: 'pm25' },
  {
    parameter: 'temperature',
    unit: 'c',
    key: 'temp',
    numberFormat: { decimal: 'comma', digitGroup: 'space' },
  },
];

Guides

Long format data

Data in long format is where each variable is a column and each observation is a row. To handle data in the format openaq-transform needs information on the column

import { NodeClient } from 'openaq-transform/node';
import { Resource, constant } from 'openaq-transform/core';


export class Client extends NodeClient {
  provider = 'example'
  resource = {
      locations: new Resource({ url: 'https://api.example.com/locations' }),
      measurements: new Resource({ url: 'https://api.example.com/measurements' })
  };
  parser = 'json';
  reader = 'api';
  averagingInterval = constant(3600);
  isMobile = false;
  longFormat = true
  locationId = 'locationId'
  datetime = 'datetime'
  locationLabel = 'name'
  xGeometry = 'lon'
  yGeometry = 'lat'
  parameterName = 'parameter'
  parameterValue = 'value'
  parameters = [
    { parameter: 'pm25', unit: 'ug/m3', key: 'pm25' },
  ]
}

TransformData Output

TransformData is the main output of the transform client, returned by client.load(). It contains everything the ingestor needs to process a batch of air quality data.

Structure

{
  "meta": {
    "schema": "v0.1",
    "sourceName": "provider-name",
    "ingestMatchingMethod": "ingest-id" | "source-spatial",
    "startedOn": "2024-01-01T00:00:00+00:00",
    "finishedOn": "2024-01-01T00:00:05+00:00",
    "exportedOn": "2024-01-01T00:00:05+00:00",
    "fetchSummary": { ... }
  },
  "measurements": [ ... ],
  "locations": [ ... ]
}

`meta`

Field	Type	Description
`schema`	`string`	Schema version for the output format.
`sourceName`	`string`	The provider name, used to identify the data source.
`ingestMatchingMethod`	`"ingest-id"` \| `"source-spatial"`	How the ingestor should match incoming data to existing records. `"ingest-id"` matches on the sensor key; `"source-spatial"` matches on coordinates.
`startedOn`	`string \| undefined`	Timestamp when `load()` began.
`finishedOn`	`string \| undefined`	Timestamp when `load()` completed.
`exportedOn`	`string \| undefined`	Timestamp when the output was serialized.
`fetchSummary`	`Summary`	Counts of locations, systems, sensors, flags, measurements, datetime range, bounding box, and error totals. Useful for logging and debugging.

`measurements`

An array of MeasurementJSON objects. Each represents a single sensor reading:

Field	Type	Description
`key`	`string`	The sensor key this measurement belongs to. Composed of provider, site, system, and metric info.
`timestamp`	`string`	ISO 8601 timestamp of the reading.
`value`	`number \| null`	The measured value after unit conversion/validation. `null` if the value was flagged.
`flags`	`string[] \| undefined`	Optional flags applied during value processing (e.g., out-of-range).
`coordinates`	`object \| undefined`	Optional per-measurement coordinates, only present for mobile sensors.

`locations`

An array of LocationJSON objects. Each represents a monitoring site and its full sensor hierarchy:

Field	Type	Description
`key`	`string`	Unique location key (`{provider}/{siteId}`).
`site_id`	`string`	The provider's identifier for this site.
`site_name`	`string`	Human-readable name for the site.
`coordinates`	`object`	Longitude and latitude of the site.
`ismobile`	`boolean`	Whether the station is mobile.
`systems`	`SystemJSON[]`	Nested array of sensor systems (manufacturer/model groupings), each containing an array of sensors with their metric, intervals, status, and any flags.

Key relationships

Location (site)
  └── System (manufacturer + model)
        └── Sensor (metric + version + instance)
              └── Measurements (timestamped values)

Measurements reference sensors by key. The ingestor uses this hierarchy to upsert location/system/sensor metadata and then append measurements.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
src		src
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
biome.json		biome.json
knip.json		knip.json
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.json		tsconfig.json
tsdown.config.ts		tsdown.config.ts
vitest.config.ts		vitest.config.ts

Latest commit

History

Folders and files

Repository files navigation

OpenAQ Transform

Overview

Core Concepts

Client

Secrets

Basic Usage

Resource

File resources

Parameters

Context

Body

Properties

Authentication

Types

APIKey

Bearer Auth Header

Example

Basic Auth

Example

Properties

Readers

API reader

Parameters

Output

Content-type detection

Custom reader

Parsers

Field mappings

Geohash derived IDs

Datetime handling

Parsing datetimes

Time-beginning vs. time-ending intervals

Parameter and unit mappings

Number formats

Client-level

Per-parameter

Guides

Long format data

TransformData Output

Structure

meta

measurements

locations

Key relationships

About

Resources

Code of conduct

Contributing

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

`meta`

`measurements`

`locations`