zarr_metadata.v3.definition

zarr_metadata.v3.definition

Metadata fields read against their definitions: the public door.

Every Zarr v3 extension point -- codecs, data types, chunk grids, chunk key encodings, storage transformers -- is a metadata field: a name, and a configuration whose JSON the extension defines. A definition says what that JSON is and what is allowed in it. It is a value, not a class to subclass:

• name, the name the metadata carries;
• configuration, the TypedDict the configuration's JSON is, which is the one declaration of it: the type checker is compiled from it, and a checked configuration has it as its static type. It reads as the typing spec defines a TypedDict -- total, Required, NotRequired, closed and extra_items mean what they mean to a type checker, whether or not its module postpones annotations;
• rules, a function over that TypedDict yielding what the spec disallows -- a bound, members read together -- located in the configuration.

What kind of metadata a definition defines is its type: CodecDefinition (with the codec's kind, and its size: whether the size of what it gives out is fixed by the size of what it is handed, "static", or depends on the values, "dynamic"), DataTypeDefinition, ChunkGridDefinition, ChunkKeyEncodingDefinition, StorageTransformerDefinition.

Reading JSON. Three steps, each feeding the next, and each usable on its own by a caller that holds nothing but JSON:

1. check(value, SomeTypedDict), from zarr_metadata.typed_json and here too, type-checks JSON against a TypedDict and needs nothing else: a value of the TypedDict or None, and every problem, each located. The value holds what the TypedDict admits and nothing else. A member typed with a field alias is checked as the JSON a metadata field is.
2. definition.judge(configuration) is the check, each nested field's envelope judged -- a stray member, a must_understand of false -- and then the rules, for one configuration: GZIP_CODEC.judge({"level": 12}).

3. resolve(field, CodecDefinition, CORE_AND_EXTENSIONS) reads a whole field in a scope: its envelope judged, its name related to a definition, its configuration judged, and each nested field read the same way. It returns Resolved -- the field's JSON, its resolution, the definition and the checked configuration -- and every problem. A name nothing in scope claims is out_of_scope: left unjudged, which is what keeps the format open. A field is read as one of the five kinds, with or without type arguments; resolve(field, Definition, scope) is a TypeError, since nothing is filed under it. configuration_of(resolved, GZIP_CODEC) is the configuration typed as that definition's TypedDict, when it read it.

   from zarr_metadata.v3.codec.gzip import GZIP_CODEC from zarr_metadata.v3.definition import ( CORE_AND_EXTENSIONS, CodecDefinition, configuration_of, resolve, )

   resolved, problems = resolve({"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_AND_EXTENSIONS) resolved.resolution # 'invalid' problems[0].loc # ('configuration', 'level')

   resolved, problems = resolve({"name": "gzip", "configuration": {"level": 5}}, CodecDefinition, CORE_AND_EXTENSIONS) configuration_of(resolved, GZIP_CODEC) # {'level': 5}, a GzipCodecConfiguration

Problems are values, not exceptions: ValidationProblem(loc, message, kind), with kind one of invalid_type, invalid_value, missing_key, unknown_key and invalid_json. A field with an unknown key is still read -- the key reported, the configuration judged without it -- so a consumer that tolerates one filters by kind and uses what was read; the field is valid only when there is no problem at all.

Writing an extension. A TypedDict, a function for its rules, and a definition; then a scope that holds it. The TypedDict is a typing_extensions.TypedDict: closed and extra_items are PEP 728's, which typing.TypedDict does not take on the versions this package supports. The rules are handed the configuration and the fields it holds as the scope read them: a field that is read keeps what it read inside it as Resolved.nested, a Nested mapping by where each sits, so a struct's rules reach its field types. judge, which reads in no scope, hands them none.

from collections.abc import Iterator

from typing_extensions import TypedDict

from zarr_metadata.v3.definition import (
    CORE_AND_EXTENSIONS,
    CodecDefinition,
    Nested,
    ValidationProblem,
)


class AcmeLz4Configuration(TypedDict, closed=True):
    acceleration: int


def acme_lz4_rules(
    configuration: AcmeLz4Configuration, nested: Nested
) -> Iterator[ValidationProblem]:
    if configuration["acceleration"] < 1:
        yield ValidationProblem(("acceleration",), "expected an integer >= 1", "invalid_value")


ACME_LZ4 = CodecDefinition(
    name="acme.lz4",
    configuration=AcmeLz4Configuration,
    kind="bytes_bytes",
    size="dynamic",
    rules=acme_lz4_rules,
)
SCOPE = CORE_AND_EXTENSIONS.extended_with(ACME_LZ4)

A scope reads whole documents as well as fields: validate_array_metadata_v3(document, context=SCOPE), from zarr_metadata.model, reads each extension point of a v3 array document through the definitions in SCOPE, and so do the model's from_json and from_key_value: a fill value is judged against the data type it names, by that data type's definition, the chunk grid against the shape, by the grid's definition, and the codecs as a pipeline, each by its definition against the chunk it is handed.

The TypedDict says what a key it does not declare is: with closed=True, a problem, as above; with extra_items=, a key holding that type; with closed=False, anything at all. One that says none of these is open by default, and would take a misspelled key without a word, so a definition refuses it. Its members are the shapes JSON takes: int, float, bool, str, None, JSONValue, a Literal, tuple[T, ...] and tuple[T1, T2], a union, a TypedDict, Mapping[str, V], a NewType and a type alias.

A member holding another metadata field is annotated with the field alias of its kind -- a shard's codecs: tuple[CodecField, ...] -- and read in the scope its field is read in. A member that takes codecs of static size only is annotated StaticCodecField -- a shard's index_codecs, since a reader finds the index by a size it knows before reading it -- and a codec of dynamic size there is a problem at its place, where the field is read in a scope; a name nothing claims is left unjudged, its size unknown with the rest of it. ZarrV3MetadataFieldJSON is the same JSON, but checks as JSON and nothing more, so a definition refuses a member typed with it. An extension with nothing to configure takes EmptyConfiguration, and is written as its bare name.

A data type also says what its fill value is: fill_value, the JSON shape of one as an annotation the checker reads -- Int8FillValue -- and fill_value_rules, a function yielding what the spec disallows in a fill value of that shape: an integer out of range, a hex string of another width. The rules are handed the configuration, the fields it holds as the scope read them, and the typed fill value, so a struct judges each field's fill value by that field's own type. fill_value_problems(data_type, value) judges a fill value against a data type field the scope read; one nothing in scope claims leaves it unjudged. A data type that says nothing of its fill value takes any JSON.

A data type says how its values are stored, too: storage, a function of its configuration and the fields it holds, giving a StorageClass -- in single bytes, in several bytes at a time, or each in as many as it needs. A struct's is its fields'. storage_of(data_type) asks it of a data type field the scope read: the bytes codec takes an endian for numbers of several bytes, and a struct refuses a field whose values vary in size. A data type that says nothing of it leaves it unknown.

A chunk grid says which arrays it fits: shape_rules, a function yielding what the spec disallows in a grid of its configuration over an array of a given shape -- a dimension with no chunk length, chunks that fall short of one -- located in the configuration. A grid that says nothing of the shape fits every one. It also says the lengths its chunks take along each axis of an array it fits, chunk_lengths: a set per axis, since a rectilinear grid's chunks differ. chunk_grid_lengths(grid, shape) gives both of a chunk grid field the scope read: an entry for each dimension of the shape, None where nothing says the lengths.

A codec is judged against what it is handed. The array hands its first codec a Chunk: the lengths of its grid's chunks along each of the array's dimensions, and its data type field, with None for what nothing says. A codec handed an array says what the spec disallows in it handed a chunk: chunk_rules, located in its configuration -- a transpose whose order has another number of axes. An array -> array codec says what it hands the next, whatever its chunk rules found: transition -- transpose permutes the axes. read_pipeline(codecs, chunk) reads codec fields the scope read as a pipeline: their order -- array -> array codecs, one array -> bytes codec, bytes -> bytes codecs -- and then each against the chunk it is handed, giving each codec's Stage with that chunk. Nothing is guessed: the codec after one the scope did not read, or after one that says nothing of what it hands on, is handed a chunk nothing is known of, Chunk(), which is refused nothing; a codec after that hands on only what it says of its own accord.

Raw bits are the one data type whose name carries its configuration: a document writes r and the size in bits, and r16 reads as r*, as the specification's table writes raw bits, with {"bits": 16}. A reader that reads raw bits its own way defines r*; a data type named r16 is refused, since that name reads as r*. r* itself is notation, and a document that writes it names nothing in any scope.

The simplest spelling. canonicalize(field, kind, scope) gives a field without problems in its simplest equivalent spelling: each nested field in its own simplest spelling, then the definition's canonical -- blosc drops a typesize that noshuffle ignores, a rectilinear grid run-length encodes its chunk shapes -- and the envelope in the fewest words; raw bits write their size back into the name, in decimal, so r008 is r8. A field with any problem, an unknown key included, has none: a simpler spelling of it would erase what its author wrote. What canonical gives is judged again: one that does not hold is a ValueError, a fault in the definition.

A definition checks itself when it is built, and each of these is a TypeError saying what is wrong: a configuration that is not a TypedDict, says nothing of the keys it does not declare, or has a member no checker reads, named down to the TypedDict that holds it; a name that is not a string; a member declared as a function -- rules, canonical, fill_value_rules, storage, shape_rules, chunk_lengths, chunk_rules, transition -- that is not one; a data type's fill_value no checker reads; a codec kind that is not one of the three, or a size that is not "static" or "dynamic"; chunk rules of a bytes -> bytes codec, which is handed bytes, or a transition of a codec that hands on bytes; a data type named as raw bits of one size are written. A scope refuses a definition of no kind. Nothing happens at class creation.

CORE module-attribute

CORE: Final = Context.of(*_CORE)

Only what the Zarr v3 specification defines.

CORE_AND_EXTENSIONS module-attribute

CORE_AND_EXTENSIONS: Final = Context.of(
    *_CORE, *_EXTENSIONS
)

What the specification defines, plus what zarr-extensions registers.

ChunkGridField module-attribute

ChunkGridField = TypeAliasType(
    "ChunkGridField", ZarrV3MetadataFieldJSON
)

A configuration member holding a chunk grid.

ChunkKeyEncodingField module-attribute

ChunkKeyEncodingField = TypeAliasType(
    "ChunkKeyEncodingField", ZarrV3MetadataFieldJSON
)

A configuration member holding a chunk key encoding.

CodecField module-attribute

CodecField = TypeAliasType(
    "CodecField", ZarrV3MetadataFieldJSON
)

A configuration member holding a codec: a shard's codecs is tuple[CodecField, ...].

CodecKind module-attribute

CodecKind = Literal[
    "array_array", "array_bytes", "bytes_bytes"
]

What a codec does to what it is handed: the three positions a pipeline orders.

CodecSize module-attribute

CodecSize = Literal['static', 'dynamic']

Whether the size of what a codec gives out is fixed by the size of what it is handed.

static: it is -- bytes writes each element in its width, crc32c adds four bytes. dynamic: it depends on the values -- every compressor.

DataTypeField module-attribute

DataTypeField = TypeAliasType(
    "DataTypeField", ZarrV3MetadataFieldJSON
)

A configuration member holding a data type, read in the scope the member's field is read in.

JSONValue module-attribute

JSONValue = TypeAliasType(
    "JSONValue",
    int
    | float
    | bool
    | str
    | list["JSONValue"]
    | tuple["JSONValue", ...]
    | Mapping[str, "JSONValue"]
    | None,
)

A recursive type alias for JSON-encodable values.

Defined via TypeAliasType (rather than a plain TypeAlias) so the self-reference is a named recursion point that pydantic can resolve when building a TypeAdapter; a bare recursive TypeAlias raises PydanticUserError/RecursionError at validation time.

Lengths module-attribute

Lengths: TypeAlias = tuple[frozenset[int] | None, ...]

Per axis, every length chunks take along it -- a set, since a rectilinear grid's differ -- or None where unknown.

Loc module-attribute

Loc: TypeAlias = tuple[str | int, ...]

Where in a document a value sits: the keys and indices down to it.

Nested module-attribute

Nested: TypeAlias = Mapping[Loc, Resolved[Any]]

The fields a configuration holds, each as the scope read it, by where it sits in the configuration.

ProblemKind module-attribute

ProblemKind = Literal[
    "missing_key",
    "invalid_type",
    "invalid_value",
    "invalid_json",
    "unknown_key",
]

Machine-readable classification of a ValidationProblem.

• missing_key: a required key (document key or store key) is absent.
• invalid_type: a value has the wrong structural type (e.g. a string where a mapping is required, a non-JSON-serializable object).
• invalid_value: a value has an acceptable type but an invalid content (e.g. zarr_format: 2 in a v3 document, order: "Q").
• invalid_json: bytes that do not decode as JSON.
• unknown_key: a key an object's type does not declare, where the type says it is closed, as a closed TypedDict does. Whether a Zarr configuration is closed is rarely said (zarr-developers/zarr-specs#270 has been open since 2023), and many readers refuse such a key. It gets a kind of its own so that a caller who tolerates it can tell it from a wrong value, and so that it never masks the other findings about the same object.

Resolution module-attribute

Resolution = Literal['read'] | Unread

What a scope made of a field: read by the definition that claims it, or unread, and why.

StaticCodecField module-attribute

StaticCodecField = TypeAliasType(
    "StaticCodecField", ZarrV3MetadataFieldJSON
)

A configuration member holding a codec of static size: a shard's index_codecs is one, since a reader finds the index by a size it knows before reading it.

StorageClass module-attribute

StorageClass = Literal[
    "single_byte", "multi_byte", "variable_length"
]

How a data type's values are stored: in single bytes, in several bytes at a time, or each in as many as it needs.

A number of several bytes is stored in a byte order, which the bytes codec's endian says. A value made of single bytes -- a uint8, or a struct of int8 fields -- has no byte order, and a value whose size varies takes a codec of its own.

StorageTransformerField module-attribute

StorageTransformerField = TypeAliasType(
    "StorageTransformerField", ZarrV3MetadataFieldJSON
)

A configuration member holding a storage transformer.

Unread module-attribute

Unread = Literal['out_of_scope', 'invalid']

A field no definition read: nothing in scope claims its name, or it could not be read.

ZarrV3MetadataFieldJSON module-attribute

ZarrV3MetadataFieldJSON = str | ZarrV3NamedConfigJSON

The JSON shape of any v3 metadata extension-point entry: either a bare short-hand name string or a {name, configuration, must_understand} envelope.

Used for data_type, chunk_grid, chunk_key_encoding, individual codec entries, and storage_transformers in v3 array metadata, and for the inner codecs / index_codecs lists of the sharding_indexed codec.

Chunk dataclass

What a codec is handed: chunks of some lengths along each axis, of a data type.

What nothing says is None: the lengths along an axis the grid does not say, and every part of the chunk handed on by a codec that says nothing of what it hands on. A data type field the scope did not read is held as written, and says nothing of the values either. A codec's chunk rules judge what is known and leave the rest, so a chunk nothing is known of, Chunk(), is refused nothing.

Source code in src/zarr_metadata/v3/_definition.py

@dataclass(frozen=True, slots=True)
class Chunk:
    """What a codec is handed: chunks of some lengths along each axis, of a data type.

    What nothing says is None: the lengths along an axis the grid does not
    say, and every part of the chunk handed on by a codec that says nothing
    of what it hands on. A data type field the scope did not read is held
    as written, and says nothing of the values either. A codec's chunk
    rules judge what is known and leave the rest, so a chunk nothing is
    known of, `Chunk()`, is refused nothing.
    """

    lengths: Lengths | None = None
    """Per axis, the lengths the chunks take along it; None when not even the number of axes is known."""
    data_type: Resolved[DataTypeDefinition[Any]] | None = None
    """The data type field of the values, as a scope read it; None when no field says what they are."""

    def __post_init__(self) -> None:
        lengths = cast("object", self.lengths)
        if lengths is not None and not _is_lengths(lengths):
            msg = f"a chunk's lengths are a frozenset of integers or None per axis, got {lengths!r}"
            raise TypeError(msg)
        data_type = cast("object", self.data_type)
        if data_type is not None and not _is_data_type_field(data_type):
            msg = f"a chunk's data type is a data type field a scope read, got {data_type!r}"
            raise TypeError(msg)

    @property
    def rank(self) -> int | None:
        """The number of axes; None when unknown."""
        return None if self.lengths is None else len(self.lengths)

data_type class-attribute instance-attribute

data_type: Resolved[DataTypeDefinition[Any]] | None = None

The data type field of the values, as a scope read it; None when no field says what they are.

lengths class-attribute instance-attribute

lengths: Lengths | None = None

Per axis, the lengths the chunks take along it; None when not even the number of axes is known.

rank property

rank: int | None

The number of axes; None when unknown.

ChunkGridDefinition dataclass

Bases: Definition[C]

A chunk grid, and the arrays it fits.

shape_rules is what the spec disallows in a grid of this configuration over an array of a given shape: a dimension with no chunk length, chunks that fall short of one. It is handed the configuration, the fields it holds as the scope read them, and the shape, and locates its problems in the configuration. A grid that says nothing of the shape fits every one.

chunk_lengths is what the first codec of the array's pipeline is handed: the lengths the grid's chunks take along each axis of an array of a shape it fits -- one for each axis of a regular grid, every length a rectilinear grid lists. It is asked only of a grid its shape rules accept. A grid that says nothing of it leaves the lengths along every axis unknown.

Source code in src/zarr_metadata/v3/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True)
class ChunkGridDefinition(Definition[C]):
    """A chunk grid, and the arrays it fits.

    `shape_rules` is what the spec disallows in a grid of this
    configuration over an array of a given shape: a dimension with no
    chunk length, chunks that fall short of one. It is handed the
    configuration, the fields it holds as the scope read them, and the
    shape, and locates its problems in the configuration. A grid that
    says nothing of the shape fits every one.

    `chunk_lengths` is what the first codec of the array's pipeline is
    handed: the lengths the grid's chunks take along each axis of an
    array of a shape it fits -- one for each axis of a regular grid, every
    length a rectilinear grid lists. It is asked only of a grid its shape
    rules accept. A grid that says nothing of it leaves the lengths along
    every axis unknown.
    """

    shape_rules: Callable[[C, Nested, tuple[int, ...]], Iterable[ValidationProblem]] = no_rules
    """What the spec disallows in this grid over an array of a shape, located in the configuration."""
    chunk_lengths: Callable[[C, Nested, tuple[int, ...]], Lengths] = unknown_lengths
    """The lengths its chunks take along each axis of an array of a shape it fits, None where unknown."""

chunk_lengths class-attribute instance-attribute

chunk_lengths: Callable[
    [C, Nested, tuple[int, ...]], Lengths
] = unknown_lengths

The lengths its chunks take along each axis of an array of a shape it fits, None where unknown.

shape_rules class-attribute instance-attribute

shape_rules: Callable[
    [C, Nested, tuple[int, ...]],
    Iterable[ValidationProblem],
] = no_rules

What the spec disallows in this grid over an array of a shape, located in the configuration.

ChunkKeyEncodingDefinition dataclass

Bases: Definition[C]

A chunk key encoding.

Source code in src/zarr_metadata/v3/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True)
class ChunkKeyEncodingDefinition(Definition[C]):
    """A chunk key encoding."""

CodecDefinition dataclass

Bases: Definition[C]

A codec: what it does to what it is handed, and whether the size of what it gives out is static.

A codec handed an array -- array -> array, array -> bytes -- says what the spec disallows in it handed a Chunk: chunk_rules, handed the configuration, the fields it holds as the scope read them, and the chunk, and locating its problems in the configuration -- a bytes codec without endian, handed a multi-byte data type. An array -> array codec also says what it hands on: transition, the chunk the next codec is handed, given the one it is handed -- transpose permutes the axes, cast_value changes the data type. The two are the spec's pair: a codec computes what it gives from the shape and data type it is handed, and "If the decoded_representation_type is not supported, this algorithm must fail with an error" (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L987-L994). The transition is asked of every chunk the codec is handed, whatever its chunk rules found, so it gives only what holds either way: a transpose whose order has another number of axes hands on lengths nothing is known of. A codec that says nothing of what it hands on hands the next a chunk nothing is known of.

Source code in src/zarr_metadata/v3/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True)
class CodecDefinition(Definition[C]):
    """A codec: what it does to what it is handed, and whether the size of what it gives out is static.

    A codec handed an array -- array -> array, array -> bytes -- says what
    the spec disallows in it handed a `Chunk`: `chunk_rules`, handed the
    configuration, the fields it holds as the scope read them, and the
    chunk, and locating its problems in the configuration -- a `bytes`
    codec without `endian`, handed a multi-byte data type. An array ->
    array codec also says what it hands on: `transition`, the chunk the
    next codec is handed, given the one it is handed -- `transpose`
    permutes the axes, `cast_value` changes the data type. The two are the
    spec's pair: a codec computes what it gives from the shape and data
    type it is handed, and "If the decoded_representation_type is not
    supported, this algorithm must fail with an error"
    (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L987-L994).
    The transition is asked of every chunk the codec is handed, whatever
    its chunk rules found, so it gives only what holds either way: a
    `transpose` whose `order` has another number of axes hands on lengths
    nothing is known of. A codec that says nothing of what it hands on
    hands the next a chunk nothing is known of.
    """

    kind: CodecKind
    size: CodecSize
    chunk_rules: Callable[[C, Nested, Chunk], Iterable[ValidationProblem]] = no_rules
    """What the spec disallows in this codec handed a chunk, located in the configuration."""
    transition: Callable[[C, Nested, Chunk], Chunk] = unknown_chunk
    """The chunk the next codec is handed, given the one this array -> array codec is handed."""

    def _refusal(self) -> str | None:
        kind: object = self.kind
        if kind not in get_args(CodecKind):
            return f"{self.name!r}: kind is one of {get_args(CodecKind)!r}, got {kind!r}"
        size: object = self.size
        if size not in get_args(CodecSize):
            return f"{self.name!r}: size is one of {get_args(CodecSize)!r}, got {size!r}"
        if kind == "bytes_bytes" and self.chunk_rules is not no_rules:
            return f"{self.name!r}: chunk_rules, of a bytes -> bytes codec, which is handed bytes"
        if kind != "array_array" and self.transition is not unknown_chunk:
            return f"{self.name!r}: a transition, of a codec that hands on bytes"
        return None

chunk_rules class-attribute instance-attribute

chunk_rules: Callable[
    [C, Nested, Chunk], Iterable[ValidationProblem]
] = no_rules

What the spec disallows in this codec handed a chunk, located in the configuration.

transition class-attribute instance-attribute

transition: Callable[[C, Nested, Chunk], Chunk] = (
    unknown_chunk
)

The chunk the next codec is handed, given the one this array -> array codec is handed.

Context dataclass

The definitions in scope while metadata is read.

A value with no reading of its own: resolve reads a field in it, and claimant is the one question it answers, which definition a name belongs to. Built from definitions with Context.of, extended with more by extended_with.

Source code in src/zarr_metadata/v3/_registry.py

@dataclass(frozen=True, slots=True)
class Context:
    """The definitions in scope while metadata is read.

    A value with no reading of its own: `resolve` reads a field in it,
    and `claimant` is the one question it answers, which definition a
    name belongs to. Built from definitions with `Context.of`, extended
    with more by `extended_with`.
    """

    tables: Tables

    @classmethod
    def of(cls, *definitions: Definition[Any]) -> Context:
        """A scope of exactly these definitions; a later one takes a name over from an earlier.

        `TypeError` for a definition of no kind, which no position in a
        document could hold.
        """
        tables: dict[type[Definition[Any]], dict[str, Definition[Any]]] = {
            kind: {} for kind in KINDS
        }
        for definition in definitions:
            kind = kind_of(definition)
            if kind is None:
                msg = (
                    f"{definition.name!r} is a definition of no kind; build it as a "
                    "CodecDefinition, DataTypeDefinition, ChunkGridDefinition, "
                    "ChunkKeyEncodingDefinition or StorageTransformerDefinition"
                )
                raise TypeError(msg)
            tables[kind][definition.name] = definition
        return cls(
            MappingProxyType({kind: MappingProxyType(table) for kind, table in tables.items()})
        )

    def extended_with(self, *definitions: Definition[Any]) -> Context:
        """This scope, plus definitions of your own.

        A name already filed under the same kind is taken over by what is
        passed here, which is how a reader substitutes its own reading of a
        codec the package already defines -- or of raw bits, by defining
        `r*`.
        """
        return Context.of(*self.definitions(), *definitions)

    def definitions(self) -> tuple[Definition[Any], ...]:
        """Every definition in scope, kind by kind."""
        return tuple(entry for table in self.tables.values() for entry in table.values())

    def claimant(self, kind: type[D], name: str) -> D | None:
        """The definition of `kind` in scope that reads `name`, a name a document writes; None if none does.

        The one filed under the name `spelled` reads it as: itself, but
        for raw bits, `r16` read by the definition of `r*`.
        """
        asked = as_kind(kind)
        filed, _ = spelled(asked, name)
        if filed is None:
            return None
        return cast("D | None", self.tables.get(asked, {}).get(filed))

claimant

claimant(kind: type[D], name: str) -> D | None

The definition of kind in scope that reads name, a name a document writes; None if none does.

The one filed under the name spelled reads it as: itself, but for raw bits, r16 read by the definition of r*.

Source code in src/zarr_metadata/v3/_registry.py

def claimant(self, kind: type[D], name: str) -> D | None:
    """The definition of `kind` in scope that reads `name`, a name a document writes; None if none does.

    The one filed under the name `spelled` reads it as: itself, but
    for raw bits, `r16` read by the definition of `r*`.
    """
    asked = as_kind(kind)
    filed, _ = spelled(asked, name)
    if filed is None:
        return None
    return cast("D | None", self.tables.get(asked, {}).get(filed))

definitions

definitions() -> tuple[Definition[Any], ...]

Every definition in scope, kind by kind.

Source code in src/zarr_metadata/v3/_registry.py

def definitions(self) -> tuple[Definition[Any], ...]:
    """Every definition in scope, kind by kind."""
    return tuple(entry for table in self.tables.values() for entry in table.values())

extended_with

extended_with(*definitions: Definition[Any]) -> Context

This scope, plus definitions of your own.

A name already filed under the same kind is taken over by what is passed here, which is how a reader substitutes its own reading of a codec the package already defines -- or of raw bits, by defining r*.

Source code in src/zarr_metadata/v3/_registry.py

def extended_with(self, *definitions: Definition[Any]) -> Context:
    """This scope, plus definitions of your own.

    A name already filed under the same kind is taken over by what is
    passed here, which is how a reader substitutes its own reading of a
    codec the package already defines -- or of raw bits, by defining
    `r*`.
    """
    return Context.of(*self.definitions(), *definitions)

of classmethod

of(*definitions: Definition[Any]) -> Context

A scope of exactly these definitions; a later one takes a name over from an earlier.

TypeError for a definition of no kind, which no position in a document could hold.

Source code in src/zarr_metadata/v3/_registry.py

@classmethod
def of(cls, *definitions: Definition[Any]) -> Context:
    """A scope of exactly these definitions; a later one takes a name over from an earlier.

    `TypeError` for a definition of no kind, which no position in a
    document could hold.
    """
    tables: dict[type[Definition[Any]], dict[str, Definition[Any]]] = {
        kind: {} for kind in KINDS
    }
    for definition in definitions:
        kind = kind_of(definition)
        if kind is None:
            msg = (
                f"{definition.name!r} is a definition of no kind; build it as a "
                "CodecDefinition, DataTypeDefinition, ChunkGridDefinition, "
                "ChunkKeyEncodingDefinition or StorageTransformerDefinition"
            )
            raise TypeError(msg)
        tables[kind][definition.name] = definition
    return cls(
        MappingProxyType({kind: MappingProxyType(table) for kind, table in tables.items()})
    )

DataTypeDefinition dataclass

Bases: Definition[C]

A data type, and the fill value an array of it takes.

fill_value is the JSON shape of a fill value -- Int8FillValue, an annotation the checker reads as it reads a configuration's members -- and fill_value_rules is what the spec disallows in a fill value of that shape: an integer out of range, a hex string of another width. The rules are handed the configuration, the fields it holds as the scope read them (a struct's field types), and the typed fill value. A data type that says nothing of its fill value takes any JSON.

storage says how its values are stored -- in single bytes, in several bytes at a time, or each in as many as it needs -- which is what the bytes codec asks of the data type it is handed: an endian, for numbers of several bytes. A struct's is its fields', so it is handed the fields the configuration holds as the scope read them. A data type that says nothing of it leaves it unknown.

One named as a document writes raw bits of one size -- r16 -- is refused: that name reads as r*, so nothing would ever read it with this definition.

Source code in src/zarr_metadata/v3/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True)
class DataTypeDefinition(Definition[C]):
    """A data type, and the fill value an array of it takes.

    `fill_value` is the JSON shape of a fill value -- `Int8FillValue`, an
    annotation the checker reads as it reads a configuration's members --
    and `fill_value_rules` is what the spec disallows in a fill value of
    that shape: an integer out of range, a hex string of another width.
    The rules are handed the configuration, the fields it holds as the
    scope read them (a struct's field types), and the typed fill value. A
    data type that says nothing of its fill value takes any JSON.

    `storage` says how its values are stored -- in single bytes, in
    several bytes at a time, or each in as many as it needs -- which is
    what the `bytes` codec asks of the data type it is handed: an
    `endian`, for numbers of several bytes. A struct's is its fields', so
    it is handed the fields the configuration holds as the scope read
    them. A data type that says nothing of it leaves it unknown.

    One named as a document writes raw bits of one size -- `r16` -- is
    refused: that name reads as `r*`, so nothing would ever read it with
    this definition.
    """

    fill_value: object = JSONValue
    """The JSON shape of a fill value, as an annotation: `Int8FillValue`."""
    fill_value_rules: Callable[[C, Nested, Any], Iterable[ValidationProblem]] = no_rules
    """What the spec disallows in a fill value of that shape, located in it."""
    storage: Callable[[C, Nested], StorageClass | None] = unknown_storage
    """How its values are stored, given the configuration and the fields it holds; None when unknown."""

    def _refusal(self) -> str | None:
        if RAW_BYTES_NAME_PATTERN.fullmatch(self.name) is not None:
            return (
                f"{self.name!r} is how a document writes raw bits of one size, which read as "
                f"{RAW_BYTES_NAME!r}; to read raw bits your own way, define {RAW_BYTES_NAME!r}"
            )
        try:
            _fill_value_parser(self.fill_value)
        except TypeError as error:
            return f"{self.name!r}: fill_value: {error}"
        return None

fill_value class-attribute instance-attribute

fill_value: object = JSONValue

The JSON shape of a fill value, as an annotation: Int8FillValue.

fill_value_rules class-attribute instance-attribute

fill_value_rules: Callable[
    [C, Nested, Any], Iterable[ValidationProblem]
] = no_rules

What the spec disallows in a fill value of that shape, located in it.

storage class-attribute instance-attribute

storage: Callable[[C, Nested], StorageClass | None] = (
    unknown_storage
)

How its values are stored, given the configuration and the fields it holds; None when unknown.

Definition dataclass

Bases: Generic[C]

One extension's metadata, as JSON: its name, the TypedDict its configuration is, its rules.

configuration is the TypedDict, and so the one declaration of the JSON: the checker is compiled from it, the static type of a checked configuration is it, and a document's author writes to it. It reads as the typing spec defines it -- total, Required, NotRequired, closed and extra_items mean what they mean to a type checker -- and it says what a key it does not declare is: with closed=True, a problem, reported and left out; with extra_items=, a key of that type; with closed=False, anything. rules yields what the spec disallows in a configuration of that type, as it finds each; it is handed only a configuration that has passed the check, holding what the TypedDict admits and nothing else, and the fields it holds as the scope read them -- a struct's field types -- which is nothing when no scope read it. judge is the two, for a caller holding JSON.

canonical is where two spellings of the configuration that mean the same thing are made one.

Built by hand, a definition refuses what it could not read with: a configuration that is not a TypedDict, says nothing of the keys it does not declare, or has a member no checker reads, which is named; and a name or rules that are not what they say.

Source code in src/zarr_metadata/v3/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True)
class Definition(Generic[C]):
    """One extension's metadata, as JSON: its name, the TypedDict its configuration is, its rules.

    `configuration` is the TypedDict, and so the one declaration of the
    JSON: the checker is compiled from it, the static type of a checked
    configuration is it, and a document's author writes to it. It reads
    as the typing spec defines it -- `total`, `Required`, `NotRequired`,
    `closed` and `extra_items` mean what they mean to a type checker --
    and it says what a key it does not declare is: with `closed=True`, a
    problem, reported and left out; with `extra_items=`, a key of that
    type; with `closed=False`, anything. `rules` yields what the spec
    disallows in a configuration of that type, as it finds each; it is
    handed only a configuration that has passed the check, holding what
    the TypedDict admits and nothing else, and the fields it holds as the
    scope read them -- a struct's field types -- which is nothing when no
    scope read it. `judge` is the two, for a caller holding JSON.

    `canonical` is where two spellings of the configuration that mean the
    same thing are made one.

    Built by hand, a definition refuses what it could not read with: a
    `configuration` that is not a TypedDict, says nothing of the keys it
    does not declare, or has a member no checker reads, which is named;
    and a `name` or rules that are not what they say.
    """

    name: str
    """The name the metadata carries, which a scope files the definition under."""
    configuration: type[C]
    """The TypedDict the configuration is."""
    rules: Callable[[C, Nested], Iterable[ValidationProblem]] = no_rules
    """What the spec disallows in a well-typed configuration and the fields it holds, located in it."""
    canonical: Callable[[C], C] = unchanged
    """A well-typed, allowed configuration in its simplest equivalent spelling.

    Only the definition's own members: a nested field is put in its own
    canonical form by `canonicalize`, which knows where each one sits.
    """

    def __post_init__(self) -> None:
        refusal = _malformed(self) or self._refusal()
        if refusal is not None:
            raise TypeError(refusal)
        try:
            _vet(self.configuration)
        except TypeError as error:
            msg = f"{self.name!r}: {error}"
            raise TypeError(msg) from error

    def _refusal(self) -> str | None:
        """What is wrong with the members a kind adds; None when nothing is, or it adds none."""
        return None

    @property
    def requires_configuration(self) -> bool:
        """Whether a document must write a configuration: whether the TypedDict has a required key."""
        return len(typeddict_keys(self.configuration).required) != 0

    def check(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
        """`value` type-checked as this definition's configuration, each nested field's envelope judged.

        `zarr_metadata.typed_json.check` is the type check alone; this also
        judges the envelope of each metadata field a member holds.
        """
        return _configuration_checked(value, self.configuration, loc)

    def judge(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
        """`value` type-checked, then judged by the rules: the configuration if it holds, and every problem.

        The rules are asked only of a configuration that type-checked and
        whose nested fields are well formed, holding what its TypedDict
        admits and nothing else, so a caller holding JSON never reaches a
        rule with a member of the wrong type, or one the type says cannot
        be there. No scope reads the fields it holds, so the rules see
        none of them read, and a rule about one -- a struct's field of a
        type whose values vary in size -- finds nothing to judge: `resolve`
        reads the field in a scope, and asks every rule.
        """
        configuration, problems = self.check(value, loc)
        if configuration is None:
            return None, problems
        refused = ruled(self, lambda: self.rules(configuration, _nothing_nested()), loc)
        return (configuration if len(refused) == 0 else None), (*problems, *refused)

canonical class-attribute instance-attribute

canonical: Callable[[C], C] = unchanged

A well-typed, allowed configuration in its simplest equivalent spelling.

Only the definition's own members: a nested field is put in its own canonical form by canonicalize, which knows where each one sits.

configuration instance-attribute

configuration: type[C]

The TypedDict the configuration is.

name instance-attribute

name: str

The name the metadata carries, which a scope files the definition under.

requires_configuration property

requires_configuration: bool

Whether a document must write a configuration: whether the TypedDict has a required key.

rules class-attribute instance-attribute

rules: Callable[
    [C, Nested], Iterable[ValidationProblem]
] = no_rules

What the spec disallows in a well-typed configuration and the fields it holds, located in it.

check

check(
    value: object, loc: Loc = ()
) -> tuple[C | None, Problems]

value type-checked as this definition's configuration, each nested field's envelope judged.

zarr_metadata.typed_json.check is the type check alone; this also judges the envelope of each metadata field a member holds.

Source code in src/zarr_metadata/v3/_definition.py

def check(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
    """`value` type-checked as this definition's configuration, each nested field's envelope judged.

    `zarr_metadata.typed_json.check` is the type check alone; this also
    judges the envelope of each metadata field a member holds.
    """
    return _configuration_checked(value, self.configuration, loc)

judge

judge(
    value: object, loc: Loc = ()
) -> tuple[C | None, Problems]

value type-checked, then judged by the rules: the configuration if it holds, and every problem.

The rules are asked only of a configuration that type-checked and whose nested fields are well formed, holding what its TypedDict admits and nothing else, so a caller holding JSON never reaches a rule with a member of the wrong type, or one the type says cannot be there. No scope reads the fields it holds, so the rules see none of them read, and a rule about one -- a struct's field of a type whose values vary in size -- finds nothing to judge: resolve reads the field in a scope, and asks every rule.

Source code in src/zarr_metadata/v3/_definition.py

def judge(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
    """`value` type-checked, then judged by the rules: the configuration if it holds, and every problem.

    The rules are asked only of a configuration that type-checked and
    whose nested fields are well formed, holding what its TypedDict
    admits and nothing else, so a caller holding JSON never reaches a
    rule with a member of the wrong type, or one the type says cannot
    be there. No scope reads the fields it holds, so the rules see
    none of them read, and a rule about one -- a struct's field of a
    type whose values vary in size -- finds nothing to judge: `resolve`
    reads the field in a scope, and asks every rule.
    """
    configuration, problems = self.check(value, loc)
    if configuration is None:
        return None, problems
    refused = ruled(self, lambda: self.rules(configuration, _nothing_nested()), loc)
    return (configuration if len(refused) == 0 else None), (*problems, *refused)

EmptyConfiguration

Bases: TypedDict

The configuration of a definition with nothing to configure, written as its bare name.

Source code in src/zarr_metadata/v3/_definition.py

class EmptyConfiguration(TypedDict, closed=True):
    """The configuration of a definition with nothing to configure, written as its bare name."""

MetadataValidationError

Bases: ValueError

Raised when a value fails validation, by the entry points that raise rather than report.

Carries every problem found (not just the first) in .problems, as an immutable tuple: a raised error is a finished report, and a caller inspecting it must not be able to edit the record.

Source code in src/zarr_metadata/_json.py

class MetadataValidationError(ValueError):
    """Raised when a value fails validation, by the entry points that raise rather than report.

    Carries every problem found (not just the first) in `.problems`, as an
    immutable tuple: a raised error is a finished report, and a caller
    inspecting it must not be able to edit the record.
    """

    problems: tuple[ValidationProblem, ...]

    def __init__(self, problems: Sequence[ValidationProblem]) -> None:
        self.problems = tuple(problems)
        for entry in cast("tuple[object, ...]", self.problems):
            # The runtime half of the annotation: a caller that is not
            # type-checked, and hands over anything else, fails here
            # rather than far away, where a `loc` is read off it.
            if not isinstance(entry, ValidationProblem):
                msg = (
                    "MetadataValidationError takes ValidationProblem values, "
                    f"got {type(entry).__name__}"
                )
                raise TypeError(msg)
        super().__init__("\n".join(str(problem) for problem in self.problems))

    def __reduce__(
        self,
    ) -> tuple[
        type[MetadataValidationError], tuple[tuple[ValidationProblem, ...]], dict[str, object]
    ]:
        # An exception pickles and copies as its class called with its
        # `args`, which here are the message; it is built from its problems,
        # and the rest of its state -- its notes among it -- follows.
        return (type(self), (self.problems,), self.__dict__)

Resolved dataclass

Bases: Generic[D]

One metadata field, as read in a scope: its JSON, and what the scope made of it.

resolution is what became of the configuration: read by the definition that claims the name, claimed by nothing, or not readable. A problem with the envelope around it -- a stray member, a must_understand of false -- is reported with the field, and leaves the resolution as it is.

Source code in src/zarr_metadata/v3/_definition.py

@dataclass(frozen=True, slots=True)
class Resolved(Generic[D]):
    """One metadata field, as read in a scope: its JSON, and what the scope made of it.

    `resolution` is what became of the configuration: read by the
    definition that claims the name, claimed by nothing, or not readable.
    A problem with the envelope around it -- a stray member, a
    `must_understand` of `false` -- is reported with the field, and leaves
    the resolution as it is.
    """

    json: JSONValue
    """The field as written, refined: arrays as tuples. `None` for a value that was not JSON, as for `null`."""
    resolution: Resolution
    definition: D | None
    """The definition that claims the field's name; None when nothing in scope does, or it names none."""
    configuration: Mapping[str, JSONValue] | None
    """The configuration, type-checked and allowed by the rules, when the field was read; None otherwise."""
    nested: Nested = dataclasses.field(default_factory=_nothing_nested)
    """The fields the configuration holds, each as the scope read it, by where it sits in the configuration.

    A struct's field types at `("fields", 0, "data_type")`, a shard's
    codecs at `("codecs", 0)`: what a definition's functions consult about
    the fields inside its own. Empty unless the field was read.
    """

configuration instance-attribute

configuration: Mapping[str, JSONValue] | None

The configuration, type-checked and allowed by the rules, when the field was read; None otherwise.

definition instance-attribute

definition: D | None

The definition that claims the field's name; None when nothing in scope does, or it names none.

json instance-attribute

json: JSONValue

The field as written, refined: arrays as tuples. None for a value that was not JSON, as for null.

nested class-attribute instance-attribute

nested: Nested = dataclasses.field(
    default_factory=_nothing_nested
)

The fields the configuration holds, each as the scope read it, by where it sits in the configuration.

A struct's field types at ("fields", 0, "data_type"), a shard's codecs at ("codecs", 0): what a definition's functions consult about the fields inside its own. Empty unless the field was read.

Stage dataclass

One codec of a pipeline, and the chunk it is handed.

Source code in src/zarr_metadata/v3/_pipeline.py

@dataclass(frozen=True, slots=True)
class Stage:
    """One codec of a pipeline, and the chunk it is handed."""

    codec: Resolved[CodecDefinition[Any]]
    """The codec, as the scope read it."""
    incoming: Chunk | None
    """The chunk it is handed.

    None for a codec handed bytes -- a bytes -> bytes codec, or any codec
    after the array -> bytes codec -- and for a codec nothing in scope
    claims when what it is handed is not known to be an array.
    """

codec instance-attribute

codec: Resolved[CodecDefinition[Any]]

The codec, as the scope read it.

incoming instance-attribute

incoming: Chunk | None

The chunk it is handed.

None for a codec handed bytes -- a bytes -> bytes codec, or any codec after the array -> bytes codec -- and for a codec nothing in scope claims when what it is handed is not known to be an array.

StorageTransformerDefinition dataclass

Bases: Definition[C]

A storage transformer.

Source code in src/zarr_metadata/v3/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True)
class StorageTransformerDefinition(Definition[C]):
    """A storage transformer."""

ValidationProblem dataclass

A single problem found in a value: where it is, what is wrong, and what kind of wrong.

loc is the path from the root of what was judged to the offending value, e.g. ("codecs", 0, "name") in a document, and an empty loc refers to that root. kind classifies the failure mode for programmatic dispatch; message is the human-readable description.

Source code in src/zarr_metadata/_json.py

@dataclass(frozen=True, slots=True)
class ValidationProblem:
    """A single problem found in a value: where it is, what is wrong, and what kind of wrong.

    `loc` is the path from the root of what was judged to the offending
    value, e.g. `("codecs", 0, "name")` in a document, and an empty `loc`
    refers to that root.
    `kind` classifies the failure mode for programmatic dispatch; `message`
    is the human-readable description.
    """

    loc: tuple[str | int, ...]
    message: str
    kind: ProblemKind

    def __post_init__(self) -> None:
        # The runtime half of the annotations: a rule written without a type
        # checker, as an extension's may be, fails where it builds a problem
        # rather than reporting one at a location that is not one.
        loc = cast("object", self.loc)
        if not isinstance(loc, tuple) or not all(
            isinstance(part, str) or (isinstance(part, int) and not isinstance(part, bool))
            for part in cast("tuple[object, ...]", loc)
        ):
            msg = f"a ValidationProblem's loc is a tuple of keys and indices, got {loc!r}"
            raise TypeError(msg)
        message = cast("object", self.message)
        if not isinstance(message, str):
            msg = f"a ValidationProblem's message is a string, got {message!r}"
            raise TypeError(msg)
        kind = cast("object", self.kind)
        if not isinstance(kind, str) or kind not in get_args(ProblemKind):
            msg = f"a ValidationProblem's kind is one of {get_args(ProblemKind)!r}, got {kind!r}"
            raise TypeError(msg)

    def __str__(self) -> str:
        location = ".".join(str(part) for part in self.loc) if self.loc else "<root>"
        return f"{location}: {self.message}"

canonicalize

canonicalize(
    data: object,
    kind: type[D],
    context: Context,
    loc: Loc = (),
) -> tuple[JSONValue | None, Problems]

data, one metadata field, in its simplest equivalent spelling, and every problem.

Only a field without problems has one. A simplest spelling says what the author wrote in fewer words, and a key the TypedDict does not declare, a stray envelope member or a must_understand of false is something the author wrote that it would erase, so a field with any problem comes back None, with its problems. Otherwise each nested field goes in its own simplest spelling, and then the definition's canonical has the rest -- judged again, so a canonical that gives a configuration that does not hold is a ValueError, a fault in the definition rather than the field. The envelope takes the fewest words: the bare name when nothing is configured, and no must_understand, since true is what absence means. A name nothing in scope claims comes back as written, since what it simplifies to is its own definition's call.

Source code in src/zarr_metadata/v3/_definition.py

def canonicalize(
    data: object, kind: type[D], context: Context, loc: Loc = ()
) -> tuple[JSONValue | None, Problems]:
    """`data`, one metadata field, in its simplest equivalent spelling, and every problem.

    Only a field without problems has one. A simplest spelling says what
    the author wrote in fewer words, and a key the TypedDict does not
    declare, a stray envelope member or a `must_understand` of `false` is
    something the author wrote that it would erase, so a field with any
    problem comes back None, with its problems. Otherwise each nested
    field goes in its own simplest spelling, and then the definition's
    `canonical` has the rest -- judged again, so a `canonical` that gives
    a configuration that does not hold is a `ValueError`, a fault in the
    definition rather than the field. The envelope takes the fewest
    words: the bare name when nothing is configured, and no
    `must_understand`, since `true` is what absence means. A name nothing
    in scope claims comes back as written, since what it simplifies to is
    its own definition's call.
    """
    resolved, problems = resolve(data, kind, context, loc)
    if len(problems) != 0:
        return None, problems
    if resolved.definition is None:
        return resolved.json, ()
    return _canonical_field(resolved.definition, resolved), ()

check

check(
    value: object, shape: type[T], loc: Loc = ()
) -> tuple[T | None, tuple[ValidationProblem, ...]]

value type-checked as shape, a TypedDict: a value of it or None, and every problem.

value is refined to JSON first -- arrays as tuples, string keys, finite floats -- and then checked member by member, each problem located under loc. What comes back holds what shape admits and nothing else: a key a closed TypedDict does not declare is reported, as unknown_key, and left out, and the value still comes back. Anything else wrong and it does not. TypeError for a shape that is not a TypedDict, or holds something no parser reads.

Source code in src/zarr_metadata/_typed_json.py

def check(
    value: object, shape: type[T], loc: Loc = ()
) -> tuple[T | None, tuple[ValidationProblem, ...]]:
    """`value` type-checked as `shape`, a TypedDict: a value of it or None, and every problem.

    `value` is refined to JSON first -- arrays as tuples, string keys,
    finite floats -- and then checked member by member, each problem
    located under `loc`. What comes back holds what `shape` admits and
    nothing else: a key a closed TypedDict does not declare is reported,
    as `unknown_key`, and left out, and the value still comes back.
    Anything else wrong and it does not. `TypeError` for a `shape` that is
    not a TypedDict, or holds something no parser reads.
    """
    if not is_typeddict(shape):
        msg = f"{shape!r} is not a TypedDict"
        raise TypeError(msg)
    refined, problems = refine_json(value, loc)
    if len(problems) != 0:
        return None, problems
    typed, found = _checker(shape)(refined, loc)
    readable = all(problem.kind == "unknown_key" for problem in found)
    return (cast("T", typed) if readable else None), found

chunk_grid_lengths

chunk_grid_lengths(
    chunk_grid: Resolved[ChunkGridDefinition[Any]],
    shape: tuple[int, ...],
    loc: Loc = (),
) -> tuple[Lengths, Problems]

The lengths the chunks of chunk_grid, a chunk grid field a scope read, take along each axis of an array of shape, and what is wrong with the grid over it.

A chunk has an extent "for each dimension of the array" (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L277-L281), so the lengths have an entry for each dimension of shape, None where nothing says them. The grid's shape rules judge it first, and locate their problems in the configuration, under loc, where the field sits: a regular grid with a chunk length for each of two dimensions, over an array of three. Only a grid that fits the shape says its lengths; a grid the scope did not read, out of scope or invalid, is left unjudged and says none. What a grid's chunk_lengths gives is checked: lengths of another type are a TypeError, and of another number of axes than shape has a ValueError, each a fault in the definition, not the field.

Source code in src/zarr_metadata/v3/_definition.py

def chunk_grid_lengths(
    chunk_grid: Resolved[ChunkGridDefinition[Any]], shape: tuple[int, ...], loc: Loc = ()
) -> tuple[Lengths, Problems]:
    """The lengths the chunks of `chunk_grid`, a chunk grid field a scope read, take along each axis of an array of `shape`, and what is wrong with the grid over it.

    A chunk has an extent "for each dimension of the array"
    (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L277-L281),
    so the lengths have an entry for each dimension of `shape`, None where
    nothing says them. The grid's shape rules judge it first, and locate
    their problems in the configuration, under `loc`, where the field
    sits: a regular grid with a chunk length for each of two dimensions,
    over an array of three. Only a grid that fits the shape says its
    lengths; a grid the scope did not read, out of scope or invalid, is
    left unjudged and says none. What a grid's `chunk_lengths` gives is
    checked: lengths of another type are a `TypeError`, and of another
    number of axes than `shape` has a `ValueError`, each a fault in the
    definition, not the field.
    """
    unknown: Lengths = (None,) * len(shape)
    definition = chunk_grid.definition
    configuration = chunk_grid.configuration
    if definition is None or configuration is None:
        return unknown, ()
    at = (*loc, "configuration")
    problems = ruled(
        definition, lambda: definition.shape_rules(configuration, chunk_grid.nested, shape), at
    )
    if len(problems) != 0:
        return unknown, problems
    lengths = asked(
        definition,
        "chunk lengths",
        lambda: cast("object", definition.chunk_lengths(configuration, chunk_grid.nested, shape)),
        at,
    )
    if not _is_lengths(lengths):
        msg = (
            f"{definition.name!r}: its chunk_lengths give a frozenset of integers or None "
            f"per axis, got {lengths!r}"
        )
        raise TypeError(msg)
    if len(lengths) != len(shape):
        msg = (
            f"{definition.name!r}: its chunk_lengths gave {len(lengths)} axes, "
            f"for a shape of {len(shape)}"
        )
        raise ValueError(msg)
    return lengths, ()

configuration_of

configuration_of(
    resolved: Resolved[Any], definition: Definition[C]
) -> C | None

The configuration resolved holds, typed as definition declares it, if definition read it.

Resolved holds a configuration as the mapping every one is; asked with the definition that read the field, this is the same mapping, as its TypedDict. None when another definition read it, or none did.

Source code in src/zarr_metadata/v3/_definition.py

def configuration_of(resolved: Resolved[Any], definition: Definition[C]) -> C | None:
    """The configuration `resolved` holds, typed as `definition` declares it, if `definition` read it.

    `Resolved` holds a configuration as the mapping every one is; asked
    with the definition that read the field, this is the same mapping, as
    its TypedDict. None when another definition read it, or none did.
    """
    if resolved.definition is not definition or resolved.configuration is None:
        return None
    return cast("C", resolved.configuration)

fill_value_problems

fill_value_problems(
    data_type: Resolved[DataTypeDefinition[Any]],
    value: object,
    loc: Loc = (),
) -> Problems

What is wrong with value as a fill value of data_type, a data type field a scope read.

value is refined to JSON first: not JSON is the first verdict, whatever the data type. It is then checked against the JSON shape the data type's definition declares, and judged by its fill value rules, as judge judges a configuration: a key the shape does not declare is reported and left out, and the rules still judge the rest. The rules see the fields the configuration holds as the scope read them: a struct judges each field's fill value by that field's own type. A data type the scope did not read, out of scope or invalid, leaves a JSON fill value unjudged. loc prefixes every problem.

Source code in src/zarr_metadata/v3/_definition.py

def fill_value_problems(
    data_type: Resolved[DataTypeDefinition[Any]], value: object, loc: Loc = ()
) -> Problems:
    """What is wrong with `value` as a fill value of `data_type`, a data type field a scope read.

    `value` is refined to JSON first: not JSON is the first verdict,
    whatever the data type. It is then checked against the JSON shape the
    data type's definition declares, and judged by its fill value rules, as
    `judge` judges a configuration: a key the shape does not declare is
    reported and left out, and the rules still judge the rest. The rules
    see the fields the configuration holds as the scope read them: a
    struct judges each field's fill value by that field's own type. A data
    type the scope did not read, out of scope or invalid, leaves a JSON fill
    value unjudged. `loc` prefixes every problem.
    """
    refined, problems = refine_json(value, loc)
    definition = data_type.definition
    configuration = data_type.configuration
    if len(problems) != 0 or definition is None or configuration is None:
        return problems
    typed, problems = _fill_value_parser(definition.fill_value)(refined, loc)
    if not _usable(problems):
        return problems
    refused = ruled(
        definition,
        lambda: definition.fill_value_rules(configuration, data_type.nested, typed),
        loc,
    )
    return (*problems, *refused)

read_pipeline

read_pipeline(
    codecs: Sequence[Resolved[CodecDefinition[Any]]],
    chunk: Chunk,
    loc: Loc = (),
) -> tuple[tuple[Stage, ...], Problems]

Each of codecs, codec fields a scope read, as a pipeline handed chunk, with the chunk it is handed, and what is wrong with them.

The order first: array -> array codecs, then one array -> bytes codec, then bytes -> bytes codecs. Then each codec in turn, handed the chunk the one before it handed on: judged by its chunk rules, which locate their problems in its configuration, and, an array -> array codec, asked what it hands on. Problems are located under loc, where the codecs sit, each codec at its index.

Source code in src/zarr_metadata/v3/_pipeline.py

def read_pipeline(
    codecs: Sequence[Resolved[CodecDefinition[Any]]], chunk: Chunk, loc: Loc = ()
) -> tuple[tuple[Stage, ...], Problems]:
    """Each of `codecs`, codec fields a scope read, as a pipeline handed `chunk`, with the chunk it is handed, and what is wrong with them.

    The order first: array -> array codecs, then one array -> bytes codec,
    then bytes -> bytes codecs. Then each codec in turn, handed the chunk
    the one before it handed on: judged by its chunk rules, which locate
    their problems in its configuration, and, an array -> array codec,
    asked what it hands on. Problems are located under `loc`, where the
    codecs sit, each codec at its index.
    """
    problems = list(_order_problems(codecs, loc))
    stages: list[Stage] = []
    # What the next codec is handed: None once that is not known to be an
    # array -- past the array -> bytes codec, or a codec of unknown kind.
    handed: Chunk | None = chunk
    for index, codec in enumerate(codecs):
        definition, configuration = codec.definition, codec.configuration
        if definition is None:
            stages.append(Stage(codec, handed))
            handed = None
            continue
        if definition.kind == "bytes_bytes":
            stages.append(Stage(codec, None))
            handed = None
            continue
        incoming = Chunk() if handed is None else handed
        stages.append(Stage(codec, incoming))
        at = (*loc, index, "configuration")
        if configuration is not None:
            problems.extend(_chunk_problems(definition, configuration, codec.nested, incoming, at))
        if definition.kind == "array_bytes":
            handed = None
        elif configuration is None:
            handed = Chunk()
        else:
            handed = _handed_on(definition, configuration, codec.nested, incoming, at)
    return tuple(stages), tuple(problems)

resolve

resolve(
    data: object,
    kind: type[D],
    context: Context,
    loc: Loc = (),
) -> tuple[Resolved[D], Problems]

data, one metadata field, read as a kind in context: what the scope made of it, and every problem.

All three steps for one field. data is refined to JSON and its envelope judged -- an extra member, a configuration that is not an object, a must_understand that is not a boolean or is false, each a problem. The name is related to a definition in context; the configuration is checked against its TypedDict and judged by its rules; each nested field the check met is read the same way, in the same scope. A name nothing claims is out_of_scope: an unmodelled extension, left unjudged, which is what keeps the format open. loc prefixes every problem. kind is one of KINDS, with or without type arguments; anything else is a TypeError.

Source code in src/zarr_metadata/v3/_definition.py

def resolve(
    data: object, kind: type[D], context: Context, loc: Loc = ()
) -> tuple[Resolved[D], Problems]:
    """`data`, one metadata field, read as a `kind` in `context`: what the scope made of it, and every problem.

    All three steps for one field. `data` is refined to JSON and its
    envelope judged -- an extra member, a `configuration` that is not an
    object, a `must_understand` that is not a boolean or is `false`, each
    a problem. The name is related to a definition in `context`; the
    configuration is checked against its TypedDict and judged by its
    rules; each nested field the check met is read the same way, in the
    same scope. A name nothing claims is `out_of_scope`: an unmodelled
    extension, left unjudged, which is what keeps the format open. `loc`
    prefixes every problem. `kind` is one of `KINDS`, with or without
    type arguments; anything else is a `TypeError`.
    """
    asked = as_kind(kind)
    refined, problems = refine_json(data, loc)
    if len(problems) != 0:
        return Resolved(None, "invalid", None, None), problems
    resolved, found = _resolve_field(refined, asked, context, loc)
    return cast("Resolved[D]", resolved), found

storage_of

storage_of(
    data_type: Resolved[DataTypeDefinition[Any]],
) -> StorageClass | None

How the values of data_type, a data type field a scope read, are stored; None when unknown.

Unknown when the scope did not read it, or its definition does not say. Its storage is the extension author's code: what it gives is checked to be a storage class, and an error it raises says which data type's storage raised it.

Source code in src/zarr_metadata/v3/_definition.py

def storage_of(data_type: Resolved[DataTypeDefinition[Any]]) -> StorageClass | None:
    """How the values of `data_type`, a data type field a scope read, are stored; None when unknown.

    Unknown when the scope did not read it, or its definition does not
    say. Its `storage` is the extension author's code: what it gives is
    checked to be a storage class, and an error it raises says which data
    type's storage raised it.
    """
    definition = data_type.definition
    configuration = data_type.configuration
    if definition is None or configuration is None:
        return None
    found = asked(
        definition,
        "storage",
        lambda: cast("object", definition.storage(configuration, data_type.nested)),
    )
    if found is not None and found not in get_args(StorageClass):
        msg = (
            f"{definition.name!r}: its storage gives one of {get_args(StorageClass)!r} or None, "
            f"got {found!r}"
        )
        raise TypeError(msg)
    return cast("StorageClass | None", found)