Loaders, savers and providers¶

A loader reads batches on an executor, and a saver writes them. Each is given a provider, which turns a key into an open reader or writer. The functions that assemble them with the defaults most programs want come first, and the classes they put together follow.

Assembling¶

vitrio.loader(workers=None, cache=None, formats=None, executor=None, scratch=None)¶

Assemble an image loader over a thread pool.

Wires the four objects a loader stands on: the formats, a reader provider, an executor and the loader itself. A caller reaches read_batch_async and read_patches_async without naming them.

Each loader built this way owns its executor. Two of them are two thread pools, each sized to the machine. Pass one executor to both to avoid that.

Parameters:

workers – How many threads to run reads on. Defaults to what the machine reports. Ignored when executor is given.
cache – How many open readers to keep between reads. Defaults to opening a file every time it is asked for.
formats – The formats to recognize files with. Defaults to the bundled ones.
executor – Where reads run. Defaults to a thread pool of its own.
scratch – A scratch to serve reads through, as reader_provider takes one. Defaults to reading the files alone.

Returns:

The assembled loader.

Return type:

ExecutorImageLoader

vitrio.saver(writers, workers=None, executor=None)¶

Assemble an image saver over a thread pool.

The provider is taken and not assembled, since whoever writes keeps it to declare the files and to close them.

Each saver built this way owns its executor, as each loader built by loader does.

Parameters:

writers – Where a key becomes an open writer.
workers – How many threads to run writes on. Defaults to what the machine reports. Ignored when executor is given.
executor – Where writes run. Defaults to a thread pool of its own.

Returns:

The assembled saver.

Return type:

ExecutorImageSaver

vitrio.reader_provider(cache=None, formats=None, scratch=None)¶

Assemble a reader provider that opens files.

Parameters:

cache – How many open readers to keep between reads. Defaults to opening a file every time it is asked for, which suits reads that do not revisit a file.
formats – The formats to recognize files with. Defaults to the bundled ones.
scratch – A scratch to serve reads through: what it holds is read from it, and what it does not yet hold is read from the file and taken in. Defaults to reading the files alone.

Returns:

The assembled provider. Its keys are paths.

Return type:

ImageReaderProvider

vitrio.writer_provider(formats=None)¶

Assemble a writer provider that creates files.

A file has to be declared on it, with the descriptor it is created as, before anything can be written to it. It is closed once it is finished.

Parameters:: formats – The formats to create files with. Defaults to the bundled ones.
Returns:: The assembled provider, with no file declared. Its keys are paths.
Return type:: FileImageWriterProvider

Loaders and savers¶

class vitrio.ImageLoader¶

Reads batches of images and patches asynchronously.

It is what read_batch_async and read_patches_async dispatch their reads to. vitrio.loader assembles one.

class vitrio.ExecutorImageLoader(readers: vitrio._binding.ImageReaderProvider, executor: vitrio._binding.Executor)¶

A loader that runs its reads on an executor.

Make a loader.

Parameters:

readers – The ImageReaderProvider that opens the files.
executor – The Executor the reads run on.

class vitrio.ImageSaver¶

Writes batches of images asynchronously.

It is what write_batch_async dispatches its writes to. vitrio.saver assembles one.

class vitrio.ExecutorImageSaver(writers: vitrio._binding.ImageWriterProvider, executor: vitrio._binding.Executor)¶

A saver that runs its writes on an executor.

Make a saver.

Parameters:

writers – The ImageWriterProvider where a key becomes a writer.
executor – The Executor the writes run on.

Providers¶

class vitrio.ImageReaderProvider¶

Turns a key into an open reader of images.

What a key means is up to the provider. The providers of files take paths, and CachingImageReaderProvider and ScratchImageReaderProvider stand in front of another provider. vitrio.reader_provider assembles them.

class vitrio.FileImageReaderProvider(formats: vitrio._binding.ImageFileReadFormatSelector)¶

Opens the file at a path with the format that suits it.

Make a provider.

Parameters:: formats – The ImageFileReadFormatSelector that opens the files.

class vitrio.CachingImageReaderProvider(backing: vitrio._binding.ImageReaderProvider, capacity: int)¶

Keeps the readers last asked for open, in front of a provider.

Going back to a file then does not open it again. When it holds more readers than its capacity, the one used least recently is closed.

Put a cache in front of a provider.

Parameters:

backing – The ImageReaderProvider that opens what is not cached.
capacity – How many readers to keep open at once.

property capacity¶: How many readers it may keep open at once.

property reader_count¶: How many readers it keeps open now.

class vitrio.ScratchImageReaderProvider(backing: vitrio._binding.ImageReaderProvider, scratch: vitrio._binding.ImageScratch)¶

Serves reads from a scratch, in front of another provider.

What the scratch holds is read from it, and what it does not hold yet is read from the provider behind and taken in.

Put a scratch in front of a provider.

Parameters:

backing – The ImageReaderProvider that reads what is not held.
scratch – The ImageScratch to serve reads from.

class vitrio.ImageWriterProvider¶

Turns a key into an open writer of images.

FileImageWriterProvider, whose keys are paths, is the one that comes with vitrio.

class vitrio.FileImageWriterProvider(formats: vitrio._binding.ImageFileWriteFormatSelector)¶

Writes files, which are declared and closed explicitly.

Creating a file replaces what was there, so a writer that was dropped and reopened would truncate what had been written. Each file therefore has one lifecycle: declare it, write to it as often as needed, and close it. A file is created the first time it is written to, and closing it flushes it and gives its file descriptor back.

Make a provider with no file declared.

Parameters:: formats – The ImageFileWriteFormatSelector that creates the files.

close(self, key: str | os.PathLike) → None¶

Finish a file.

It is flushed and its writer dropped. A closed file can be declared again.

Parameters:: key – The path of the file.
Raises:: IndexError – If the file is not declared.

declare(self, key: str | os.PathLike, descriptor: vitrio._binding.ImageDescriptor) → None¶

Make a file writable.

Nothing is created until the file is first written to.

Parameters:

key – The path of the file.
descriptor – The ImageDescriptor the file is created with.

Raises:

RuntimeError – If the file is still declared. Close it first.

property file_count¶: How many files are declared.

flush(self) → None¶

Make what has been written to the open files reach the storage.

The files stay declared and writable. The GIL is released meanwhile.

Formats¶

class vitrio.ImageFileReadFormatSelector¶

Holds the formats a file may be read with.

It opens a file with the one that suits it.

A file is recognized by its contents where its format allows it, and by its extension otherwise.

get_shared() → vitrio._binding.ImageFileReadFormatSelector¶

Get the selector that holds the formats that come with vitrio.

Returns:: MRC, TIFF and EER (when built with TIFF) and DM.
Return type:: ImageFileReadFormatSelector

class vitrio.ImageFileWriteFormatSelector¶

Holds the formats a file may be written with.

It creates a file with the one its path names.

get_shared() → vitrio._binding.ImageFileWriteFormatSelector¶

Get the selector that holds the formats that come with vitrio.

Returns:: MRC and, when built with TIFF, TIFF.
Return type:: ImageFileWriteFormatSelector

Executors¶

class vitrio.Executor¶

Where the work of a loader or a saver runs.

Its subclasses are SynchronousExecutor, which runs it in the calling thread, and ThreadPoolExecutor.

class vitrio.SynchronousExecutor¶

Runs the work in the thread that submits it, before the call returns.

Reads and writes through it are asynchronous in name only. It suits tests and debugging.

Make an executor.

class vitrio.ThreadPoolExecutor(worker_count: int)¶

Runs the work on a fixed number of threads.

Destroying it waits for its workers, which finish what they have been given first. So keep it alive for as long as work is submitted to it, and for the completions of that work.

Start the threads.

Parameters:: worker_count – How many threads to run.

property worker_count¶: How many threads run the work.