zabgrab.adapters.formats.hdf5¶

HDF5 (.h5) and MATLAB v7.3 (.mat) files.

Both use the same layout:

/fetch_0001                 one group per fetch (one MATLAB variable)
    meta                    JSON metadata
    t_from, t_to            the fetched range, POSIX seconds (UTC)
    items/
        item_12345/         (item_12345_2, ... if two servers share an ID)
            itemid, hostid, value_type
            host, host_name, name, key, units, server, server_name
            time, value, ...    the series' columns (N x 1)
/fetch_0002 ...

.h5 files are plain HDF5: text is UTF-8, and vectors are 1-D. .mat files add what MATLAB’s load needs: the 512-byte v7.3 header, MATLAB_class attributes, text as UTF-16 char arrays, text columns as cell arrays (in /#refs#, as MATLAB itself writes them), and vectors stored so that MATLAB sees N x 1.

zabgrab.adapters.formats.hdf5.GROUP_PATTERN¶

Automatic group names (fetch_NNNN).

Type:: re.Pattern

zabgrab.adapters.formats.hdf5.IDENTIFIER¶

Valid group names (MATLAB variable names).

Type:: re.Pattern

zabgrab.adapters.formats.hdf5.USERBLOCK¶

The size of the MATLAB header before the HDF5 data.

Type:: int

zabgrab.adapters.formats.hdf5.REFS¶

The group MATLAB keeps cell array contents in.

Type:: str

zabgrab.adapters.formats.hdf5.LARGE_TEXT_ACCEPTED¶

The file attribute recording that the user accepted a lot of text in a .mat file, so updates don’t ask again.

Type:: str

zabgrab.adapters.formats.hdf5.ITEM_FIELDS¶

The scalar fields of an item group (everything else is a column).

Type:: tuple of str

zabgrab.adapters.formats.hdf5.INTEGER_COLUMNS¶

Columns stored as integers (besides unsigned values).

Type:: frozenset of str

Functions

`is_mat`(path)	Check whether a file starts with a MATLAB v7.3 header.
`is_valid_group_name`(name)	Check whether `name` can be a group (MATLAB variable) name.
`large_text_accepted`(path)	Check whether a lot of text in `path` was accepted before (see write_fetch).
`next_group_name`(existing)	Get the next unused `fetch_NNNN` name after `existing` groups.
`read_fetches`(path)	Read every fetch group in a file written by zabgrab.
`read_metadata`(path)	Read the metadata of every fetch group in a file, checking it can be read.
`read_ranges`(path)	Read every fetch group's metadata and time range, but not its data.
`text_entries`(path)	Count the strings stored as cell array entries in a `.mat` file.
`write_fetch`(path, result, metadata, *, mat, ...)	Write a fetch as a new group, creating the file or appending to it.

Classes

`FetchGroup`(name, metadata, t_from, t_to, series)	One fetch read back from a file.
`GroupRange`(name, metadata, t_from, t_to)	A fetch group's metadata and time range, without its data.

class zabgrab.adapters.formats.hdf5.FetchGroup(name, metadata, t_from, t_to, series)[source]¶

Bases: object

One fetch read back from a file.

Parameters:

name (str)
metadata (dict[str, Any])
t_from (float)
t_to (float)
series (list[Series])

name¶

The group’s name.

Type:: str

metadata¶

The fetch’s metadata (see zabgrab.services.metadata).

Type:: dict

t_from, t_to

The fetched time range, in POSIX seconds.

Type:: float

series¶

The data.

Type:: list of Series

class zabgrab.adapters.formats.hdf5.GroupRange(name, metadata, t_from, t_to)[source]¶

Bases: object

A fetch group’s metadata and time range, without its data.

Parameters:

name (str)
metadata (dict[str, Any])
t_from (float)
t_to (float)

name¶

The group’s name.

Type:: str

metadata¶

The fetch’s metadata (see zabgrab.services.metadata).

Type:: dict

t_from, t_to

The fetched time range, in POSIX seconds.

Type:: float

zabgrab.adapters.formats.hdf5.is_valid_group_name(name)[source]¶

Check whether name can be a group (MATLAB variable) name.

Parameters:: name (str)
Return type:: bool

zabgrab.adapters.formats.hdf5.next_group_name(existing)[source]¶

Get the next unused fetch_NNNN name after existing groups.

Parameters:: existing (Sequence[str])
Return type:: str

zabgrab.adapters.formats.hdf5.write_fetch(path, result, metadata, *, mat, t_from, t_to, append=False, group=None, in_place=False, large_text_accepted=False, created=None)[source]¶

Write a fetch as a new group, creating the file or appending to it.

Writing goes via a temporary file that then replaces path (so a crash can’t corrupt it), unless appending in_place.

Parameters:

path (Path) – The file.
result (FetchResult) – The data.
metadata (mapping) – The fetch’s metadata, stored as JSON.
mat (bool) – Write a MATLAB v7.3 file rather than plain HDF5.
t_from (float) – The fetched time range, in POSIX seconds.
t_to (float) – The fetched time range, in POSIX seconds.
append (bool, optional) – Add to the file if it exists (else it’s replaced).
group (str, optional) – The group’s name (default: the next fetch_NNNN).
in_place (bool, optional) – Append to the file directly, without the safety copy.
large_text_accepted (bool, optional) – Record that the user accepted a lot of text in a .mat file.
created (datetime, optional) – The creation time for a new .mat file’s header (default: now).

Returns:

str – The group’s name.

Raises:

ZabgrabError – If the group name is invalid or taken, or when appending, the file has groups this version can’t read (see read_metadata()), or isn’t a zabgrab MATLAB file when appending to it as one. Also if it can’t be written.

Return type:

str

zabgrab.adapters.formats.hdf5.read_fetches(path)[source]¶

Read every fetch group in a file written by zabgrab.

Parameters:: path (Path) – The .h5 or .mat file.
Returns:: list of FetchGroup – The groups, in the order they were written.
Raises:: ZabgrabError – If the file doesn’t exist or isn’t HDF5, or has groups this version can’t read (see read_metadata()).
Return type:: list[FetchGroup]

zabgrab.adapters.formats.hdf5.read_metadata(path)[source]¶

Read the metadata of every fetch group in a file, checking it can be read.

Each group records the version of the file layout it was written with (FORMAT_VERSION). Groups written by a newer version of zabgrab, or without a version, are refused rather than misread.

Parameters:: path (Path) – The .h5 or .mat file.
Returns:: list of tuple of str and dict – Each group’s name and metadata, in the order they were written.
Raises:: ZabgrabError – If the file doesn’t exist or isn’t HDF5, a group wasn’t written by zabgrab, or its layout is newer than this version reads.
Return type:: list[tuple[str, dict[str, Any]]]

zabgrab.adapters.formats.hdf5.read_ranges(path)[source]¶

Read every fetch group’s metadata and time range, but not its data.

Parameters:: path (Path) – The .h5 or .mat file.
Returns:: list of GroupRange – The groups, in the order they were written.
Raises:: ZabgrabError – As for read_metadata().
Return type:: list[GroupRange]

zabgrab.adapters.formats.hdf5.is_mat(path)[source]¶

Check whether a file starts with a MATLAB v7.3 header.

Parameters:: path (Path)
Return type:: bool

zabgrab.adapters.formats.hdf5.large_text_accepted(path)[source]¶

Check whether a lot of text in path was accepted before (see write_fetch).

Parameters:: path (Path)
Return type:: bool

zabgrab.adapters.formats.hdf5.text_entries(path)[source]¶

Count the strings stored as cell array entries in a .mat file.

Parameters:: path (Path)
Return type:: int

zabgrab

Navigation

Related Topics

zabgrab.adapters.formats.hdf5¶