Skip to main content

Tensor and Sample

The runtime data model has two primary types:

  • Tensor: typed numeric payload (shape, dtype, layout, storage, device).
  • Sample: envelope around tensor payload with media/runtime metadata.

Tensor​

Use Tensor when you only need numeric data and shape/type semantics.

Typical metadata in tensor:

  • shape / dtype / layout
  • storage and mapping behavior
  • a semantic field carrying typed per-domain sub-metas (image, detection, preprocess, etc.) — see Typed semantic sub-metas below

Related references:

Typed semantic sub-metas​

Every Tensor carries a semantic field — a discriminated union of per-domain sub-metas describing what the bytes mean. Each sub-meta is an std::optional (C++) / Optional (Python); only the ones relevant to the tensor are set.

Sub-metaSet when
semantic.imagePixel data (RGB / NV12 / etc.).
semantic.audioAudio samples.
semantic.tokensNLP token stream.
semantic.textUTF-8 text.
semantic.byte_streamOpaque raw bytes.
semantic.encodedEncoded video frame (H.264, etc.).
semantic.quantQuantized integer tensor (scale / zero-point).
semantic.tessMLA tile-blocked layout.
semantic.detectionDetection-decoder output (BBOX today; masks / keypoints later).
semantic.preprocessInverse-transform breadcrumb left by the preprocess stage.

Consumers query the optional that matters to them:

if (bbox.semantic.detection.has_value()) {
    assert(bbox.semantic.detection->format == "BBOX");
}

Detection outputs (BBOX)​

When Model.run returns the output of a detection model — for example YOLOv8 with decode_type = BoxDecodeType.YoloV8 — the output is a packed BBOX tensor. Decode the whole run output with decode_bbox, a TensorList → TensorList transform (positional 1:1): each BBOX tensor becomes a float32 tensor of shape [num_detections, 6] with columns (x1, y1, x2, y2, score, class_id). decode_bbox recognizes a BBOX tensor by its semantic.detection.format tag when present, and otherwise by its canonical rank-1 UInt8 wire shape; a tensor that is neither raises a typed error rather than being silently skipped.

outputs = model.run([image_tensor])   # TensorList
decoded = neat.decode_bbox(outputs)    # TensorList, same length
boxes   = decoded[0].to_numpy()        # ndarray [num_detections, 6]
note
A note on tess.format

Historically the BBOX wire format was tagged via semantic.tess.format. That slot is for MLA tile geometry — BBOX is not tessellated — so the overload is deprecated. New code should read and write BBOX-format via semantic.detection. The legacy path still works for one release as a back-compat fallback inside decode_bbox_tensor / read_detection_format.

NumPy and PyTorch interop​

For Python developers familiar with NumPy and PyTorch, Tensor supports DLPack-based interop:

  • Tensor.from_numpy(...)
  • Tensor.to_numpy(...)
  • Tensor.from_torch(...)
  • Tensor.to_torch(...)
  • Tensor.from_dlpack(...)
  • Tensor.__dlpack__()

This keeps interop paths explicit and enables zero-copy where backend data layout permits it.

import numpy as np
import pyneat as neat

# HWC uint8 image-like tensor
arr = np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8)

t = neat.Tensor.from_numpy(arr, copy=False, image_format=neat.PixelFormat.RGB)
arr_back = t.to_numpy(copy=False)

# If you already have a model, NumPy can be passed directly.
# out = model.run(arr, timeout_ms=2000)

Sample​

Use Sample when you need pipeline metadata in addition to tensor bytes.

Typical sample fields:

  • caps_string, media_type, payload_tag
  • pts_ns, dts_ns, duration_ns
  • stream_id, frame_id, port_name
  • fields for bundle outputs

Related references:

#include "neat/runtime.h"
#include "neat/nodes.h"

simaai::neat::Graph graph;
graph.add(simaai::neat::nodes::Input({}));
graph.add(simaai::neat::nodes::Output({}));

auto run = graph.build(simaai::neat::Sample{}, simaai::neat::RunMode::Async);

simaai::neat::Sample in;
in.kind = simaai::neat::SampleKind::Tensor;
in.stream_id = "cam-0";
in.frame_id = 42;
// in.tensor = ...;

run.push(in);
auto out = run.pull(1000);
if (out) {
  std::cout << "stream=" << out->stream_id
            << " frame=" << out->frame_id
            << " media_type=" << out->media_type << "\n";
}

Runtime handle​

Both types flow through Run (push, pull, push_and_pull, run).

See also​

Tutorials​