deeptrack.sources.base Module#

Utility classes and functions for dynamic data sources in DeepTrack2.

This module provides core abstractions for representing and manipulating collections of data in a modular and composable way. It defines the structure and behavior of dynamic sources that can be indexed, filtered, combined, and tracked through DeepTrack’s computational graph.

These tools are primarily used in scenarios where data needs to be dynamically manipulated, filtered, or combined for feature generation in machine learning pipelines.

Key Features#

Dynamic Data Access

Sources return dictionary-like items (SourceItem) that activate custom callbacks when accessed, enabling dynamic behavior such as dependency tracking and delayed evaluation.
Composable Data Structures

Includes tools like Product, Subset, and Sources to manipulate and combine data sources for flexible pipeline construction.
Hierarchical Node System

The SourceDeepTrackNode class extends DeepTrackNode to support hierarchical field access and automatic dependency propagation.
Random Splitting Utilities

Provides utilities such as random_split() for reproducible partitioning of sources into disjoint subsets for training/validation/test workflows.

Module Structure#

Classes:

Source: Represents one or more named sequences of data.

Provides access to items as SourceItem and integrates with features for graph-based computation.
SourceItem: A dict-like object that triggers callbacks when called.

Wraps data fields from a Source and activates dependency updates on use.
SourceDeepTrackNode: A DeepTrack node that supports attribute access.

Automatically creates child nodes when dictionary-like attributes are accessed (e.g., source.a.b).
Product: Cartesian product of a Source with additional fields.

Allows combining items with new fields, either with or without a base source.
Subset: Represents a filtered view of a Source via explicit indices.

Provides indexed access to a restricted set of items.
Sources: Joins multiple Source objects into one dynamic access point.

Enables field sharing and flexible evaluation across datasets.
Join: Alias for Sources.

Functions:

random_split(source, lengths, generator) -> list[Subset]

Randomly splits a Source into multiple non-overlapping subsets.

Examples#

import deeptrack as dt

Trigger callbacks when a source item is accessed

>>> from deeptrack.sources import Source
>>>
>>> source = Source(a=[1, 2], b=[3, 4])
>>>
>>> @source.on_activate
... def callback(item):
...     print("Activated:", item)

>>> source[0]();
Activated: SourceItem({'a': 1, 'b': 3}, 2 callback(s))

Access nested dictionary-like data with dynamic nodes

>>> from deeptrack.sources.base import SourceDeepTrackNode
>>>
>>> node = SourceDeepTrackNode(lambda: {"a": 1, "b": {"x": 42}})
>>> node.a()
1

>>> node.b.x()
42

Use shared features across multiple sources

>>> from deeptrack.sources import Source, Sources
>>>
>>> train = Source(a=[1, 2], b=[3, 4])
>>> val = Source(a=[5, 6], b=[7, 8])
>>> joined = Sources(train, val)
>>> feature = dt.Value(joined.a) + dt.Value(joined.b)
>>> feature(train[0])
4

>>> feature(val[0])
12

Create a Cartesian product of fields

>>> from deeptrack.sources import Source
>>>
>>> source = Source(a=[1, 2])
>>> product = source.product(b=[10, 20])
>>> list(product)
[SourceItem({'b': 10, 'a': 1}, 1 callback(s)),
 SourceItem({'b': 20, 'a': 1}, 1 callback(s)),
 SourceItem({'b': 10, 'a': 2}, 1 callback(s)),
 SourceItem({'b': 20, 'a': 2}, 1 callback(s))]

Extract a subset of selected indices

>>> from deeptrack.sources import Source, Subset
>>>
>>> source = Source(a=[1, 2, 3], b=[10, 20, 30])
>>> subset = Subset(source, [0, 2])
>>> list(subset)
[SourceItem({'a': 1, 'b': 10}, 1 callback(s)),
 SourceItem({'a': 3, 'b': 30}, 1 callback(s))]

Split a source randomly into multiple parts

>>> from deeptrack.sources import random_split, Source
>>>
>>> source = Source(
...     a=list(range(10)),
...     b=list(range(10, 20)),
... )
>>> train, val, test = random_split(source, [0.5, 0.3, 0.2])
>>> len(train), len(val), len(test)
(5, 3, 2)

Functions#

random_split(source, lengths[, generator])

Randomly split a source into non-overlapping subsets of specified sizes.

Classes#

`Source`(**kwargs)	A class that represents one or more sources of data.
`SourceItem`(callbacks, **kwargs)	A dictionary-like object that triggers a list of callbacks when called.
`Product`([_Product__source])	Cartesian product of a source with one or more additional fields.
`Subset`(source, indices)	A subset of a source defined by a list of indices.
`Sources`(*sources)	Join multiple sources into a single dynamic access point.
`Join`	alias of `Sources`