Group Intel GPU OpenCL interoperability#

group Intel GPU OpenCL interoperability

Set of C++ classes and properties to work with Remote API for Intel GPU OpenCL plugin.

Typedefs

using gpu_handle_param = void*#

Shortcut for defining a handle parameter.

Enums

enum class ContextType#

Enum to define the type of the shared context.

Values:

enumerator OCL#

Pure OpenCL context.

enumerator VA_SHARED#

Context shared with a video decoding device.

enumerator ZE#

Pure Level Zero context.

enum class SharedMemType#

Enum to define the type of the shared memory buffer.

Values:

enumerator OCL_BUFFER#

Shared OpenCL buffer blob.

enumerator OCL_IMAGE2D#

Shared OpenCL 2D image blob.

enumerator USM_USER_BUFFER#

Shared USM pointer allocated by user.

enumerator USM_HOST_BUFFER#

Shared USM pointer type with host allocation type allocated by plugin.

enumerator USM_DEVICE_BUFFER#

Shared USM pointer type with device allocation type allocated by plugin.

enumerator VA_SURFACE#

Shared video decoder surface or D3D 2D texture blob.

enumerator DX_BUFFER#

Shared D3D buffer blob.

enumerator BUFFER_FROM_HANDLE#

OS-level external memory handle (e.g. DX12 NT handle on Windows, DMA-BUF fd on Linux) imported by the plugin into a cl_mem

enumerator CPU_VA#

Shared mmap-backed/aligned allocated host pointer mapped by plugin.

enumerator MMAPED_FILE#

Memory-mapped file buffer read and wrapped by the plugin.

enum class MemType

Enum to define memory type for pointer-based tensor sharing API.

Values:

enumerator SHARED_BUF

Shared OpenCL buffer handle passed as void* or int.

enumerator CPU_VA

CPU Virtual Address buffer.

enum class AccessMode#

Enum to define access mode of memory shared with the plugin (host virtual-address buffers, memory-mapped files, etc.)

Values:

enumerator READ#

Tensor data is only read.

enumerator READ_WRITE#

Tensor data is also written back; requires writable memory (or a writable file)

Variables

static constexpr Property<ContextType> context_type = {"CONTEXT_TYPE"}#

Shared device context type: can be either pure OpenCL (OCL) or shared video decoder (VA_SHARED) context.

static constexpr Property<gpu_handle_param> ocl_context = {"OCL_CONTEXT"}#

This key identifies OpenCL context handle in a shared context or shared memory blob parameter map.

static constexpr Property<int> ocl_context_device_id = {"OCL_CONTEXT_DEVICE_ID"}#

This key identifies ID of device in OpenCL context if multiple devices are present in the context.

static constexpr Property<int> tile_id = {"TILE_ID"}#

In case of multi-tile system, this key identifies tile within given context.

static constexpr Property<gpu_handle_param> ocl_queue = {"OCL_QUEUE"}#

This key identifies OpenCL queue handle in a shared context.

static constexpr Property<gpu_handle_param> va_device = {"VA_DEVICE"}#

This key identifies video acceleration device/display handle in a shared context or shared memory blob parameter map.

static constexpr Property<SharedMemType> shared_mem_type = {"SHARED_MEM_TYPE"}#

This key identifies type of internal shared memory in a shared memory blob parameter map.

static constexpr Property<gpu_handle_param> mem_handle = {"MEM_HANDLE"}

This key identifies OpenCL memory handle in a shared memory blob parameter map.

static constexpr Property<os_handle_param> os_handle = {"OS_HANDLE"}#

This key identifies system memory handle (fd on Linux, NT handle on Windows)

static constexpr Property<void*> cpu_va = {"CPU_VA"}#

This key identifies cpu pointer.

static constexpr Property<int64_t> cpu_va_size = {"CPU_VA_SIZE"}#

This key identifies size of allocated memory of cpu pointer.

static constexpr Property<AccessMode> cpu_va_access = {"CPU_VA_ACCESS"}#

This key identifies access mode (read or read-write) of the memory pointed to by cpu_va.

static constexpr Property<gpu_handle_param> dev_object_handle = {"DEV_OBJECT_HANDLE"}#

This key identifies video decoder surface handle in a shared memory blob parameter map.

static constexpr Property<uint32_t> va_plane = {"VA_PLANE"}#

This key identifies video decoder surface plane in a shared memory blob parameter map.

static constexpr Property<FileDescriptor> file_descriptor = {"FILE_DESCRIPTOR"}

This key identifies the file descriptor in a memory-mapped tensor parameter map.

class D3DBufferTensor : public ov::intel_gpu::ocl::ClBufferTensor#
#include <dx.hpp>

This class represents an abstraction for GPU plugin remote tensor which is shared with Direct3D 11 buffer. The plugin object derived from this class can be obtained with D3DContext::create_tensor() call.

Note

User can also obtain OpenCL buffer handle from this class.

Public Functions

inline operator ID3D11Buffer*()#

ID3D11Buffer conversion operator for the D3DContext object.

Returns:

Pointer to underlying ID3D11Buffer interface

Public Static Functions

static inline void type_check(const Tensor &tensor)#

Checks that type defined runtime parameters are presented in remote object.

Parameters:

tensor – a tensor to check

class D3DSurface2DTensor : public ov::intel_gpu::ocl::ClImage2DTensor#
#include <dx.hpp>

This class represents an abstraction for GPU plugin remote tensor which is shared with Direct3D 11 2D texture. The plugin object derived from this class can be obtained with D3DContext::create_tensor() call.

Note

User can also obtain OpenCL 2D image handle from this class.

Public Functions

inline operator ID3D11Texture2D*()#

ID3D11Texture2D conversion operator for the D3DContext object.

Returns:

Pointer to underlying ID3D11Texture2D interface

inline uint32_t plane()#

Returns plane ID of underlying video decoder surface, or 0 if no video surface was shared.

Returns:

Plane ID

Public Static Functions

static inline void type_check(const Tensor &remote_tensor)#

Checks that type defined runtime parameters are presented in remote object.

Parameters:

remote_tensor – remote tensor to check

class D3DContext : public ov::intel_gpu::ocl::ClContext#
#include <dx.hpp>

This class represents an abstraction for GPU plugin remote context which is shared with Direct3D 11 device. The plugin object derived from this class can be obtained either with CompiledModel::get_context() or Core::create_context() calls.

Note

User can also obtain OpenCL context handle from this class.

Public Functions

inline operator ID3D11Device*()#

ID3D11Device conversion operator for the D3DContext object.

Returns:

Pointer to underlying ID3D11Device interface

inline D3DContext(Core &core, ID3D11Device *device, int target_tile_id = -1)#

Constructs D3DContext remote context object from ID3D11Device.

Parameters:

  • core – OpenVINO Runtime Core object instance

  • device – A pointer to ID3D11Device to be used to create a remote context

  • target_tile_id – Desired tile id within given context for multi-tile system. Default value (-1) means that root device should be used

inline std::pair<D3DSurface2DTensor, D3DSurface2DTensor> create_tensor_nv12(const size_t height, const size_t width, ID3D11Texture2D *nv12_surf)#

This function is used to obtain a NV12 tensor from NV12 DXGI video decoder output. The resulting tensor contains two remote tensors for Y and UV planes of the surface.

Parameters:

  • height – Height of Y plane

  • width – Width of Y plane

  • nv12_surf – A ID3D11Texture2D instance to create NV12 tensor from

Returns:

A pair of remote tensors for each plane

inline D3DBufferTensor create_tensor(const element::Type type, const Shape &shape, ID3D11Buffer *buffer)#

This function is used to obtain remote tensor object from ID3D11Buffer.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • buffer – A pointer to ID3D11Buffer instance to create remote tensor based on

Returns:

A remote tensor instance

inline D3DSurface2DTensor create_tensor(const element::Type type, const Shape &shape, ID3D11Texture2D *surface, uint32_t plane = 0)#

This function is used to obtain remote tensor object from ID3D11Texture2D.

Note

The underlying ID3D11Texture2D can also be a plane of output surface of DXGI video decoder

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • surface – Pointer to ID3D11Texture2D interface of the objects that owns NV12 texture

  • plane – ID of the plane to be shared (0 or 1)

Returns:

D3DSurface2DTensor tensor

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, const cl_mem buffer)#

This function is used to obtain remote tensor object from user-supplied cl_mem object.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • buffer – A cl_mem object that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, const cl::Buffer &buffer)#

This function is used to obtain remote tensor object from user-supplied cl::Buffer object.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • buffer – A cl::Buffer object that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline ClImage2DTensor create_tensor(const element::Type type, const Shape &shape, const cl::Image2D &image)#

This function is used to obtain remote tensor object from user-supplied cl::Image2D object.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • image – A cl::Image2D object that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, SharedBufferHandle handle)#

This function is used to obtain a remote tensor object from a user-supplied external memory handle.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • SharedBufferHandle – External memory handle from another API (DX12 shared NT handle on Windows passed as void*, DMA-BUF fd on Linux passed as int)

Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, VirtualAddressMemory buff)#

This function is used to obtain a remote tensor object from a user-supplied VirtualAddressMemory.

Parameters:
Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, const FileDescriptor &file_descriptor)#

This function is used to obtain a remote tensor object from a file. The plugin memory-maps the file and keeps the mapping alive for the whole tensor lifetime, so the file must not be modified until the returned tensor is destroyed.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • file_descriptor – Descriptor with the path, offset and access mode of the file containing tensor data. The offset must be a multiple of the system memory mapping alignment: the page size on Linux (typically 4 KiB) and the allocation granularity on Windows (typically 64 KiB). AccessMode::READ_WRITE additionally requires the file to be writable by the calling process and makes the tensor writes visible in the file.

Returns:

A remote tensor instance

inline USMTensor create_tensor(const element::Type type, const Shape &shape, void *usm_ptr)#

This function is used to obtain remote tensor object from user-supplied USM pointer.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • usm_ptr – A USM pointer that should be wrapped by a remote tensor

Returns:

A remote tensor instance

Public Static Functions

static inline void type_check(const RemoteContext &remote_context)#

Checks that type defined runtime parameters are presented in remote object.

Parameters:

remote_context – A remote context to check

class ClBufferTensor : public ov::RemoteTensor#
#include <ocl.hpp>

This class represents an abstraction for GPU plugin remote tensor which can be shared with user-supplied OpenCL buffer. The plugin object derived from this class can be obtained with ClContext::create_tensor() call.

Note

User can obtain OpenCL buffer handle from this class.

Subclassed by ov::intel_gpu::ocl::D3DBufferTensor

Public Functions

inline cl_mem get()#

Returns the underlying OpenCL memory object handle.

Returns:

underlying OpenCL memory object handle

inline operator cl_mem()#

OpenCL memory handle conversion operator.

Returns:

cl_mem

inline operator cl::Buffer()#

Standard Khronos cl::Buffer wrapper conversion operator.

Returns:

cl::Buffer object

Public Static Functions

static inline void type_check(const Tensor &tensor)#

Checks that type defined runtime parameters are presented in remote object.

Parameters:

tensor – a tensor to check

class ClImage2DTensor : public ov::RemoteTensor#
#include <ocl.hpp>

This class represents an abstraction for GPU plugin remote tensor which can be shared with user-supplied OpenCL 2D Image. The plugin object derived from this class can be obtained with ClContext::create_tensor() call.

Note

User can obtain OpenCL image handle from this class.

Subclassed by ov::intel_gpu::ocl::D3DSurface2DTensor, ov::intel_gpu::ocl::VASurfaceTensor

Public Functions

inline cl_mem get()#

Returns the underlying OpenCL memory object handle.

Returns:

underlying OpenCL memory object handle

inline operator cl_mem()#

OpenCL memory handle conversion operator.

Returns:

cl_mem

inline operator cl::Image2D()#

Standard Khronos cl::Image2D wrapper conversion operator for the ClContext object.

Returns:

cl::Image2D object

Public Static Functions

static inline void type_check(const Tensor &tensor)#

Checks that type defined runtime parameters are presented in remote object.

Parameters:

tensor – a tensor to check

class USMTensor : public ov::RemoteTensor#
#include <ocl.hpp>

This class represents an abstraction for GPU plugin remote tensor which can be shared with user-supplied USM device pointer. The plugin object derived from this class can be obtained with ClContext::create_tensor() call.

Note

User can obtain USM pointer from this class.

Public Functions

inline void *get()#

Returns the underlying USM pointer.

Returns:

underlying USM pointer

Public Static Functions

static inline void type_check(const Tensor &tensor)#

Checks that type defined runtime parameters are presented in remote object.

Parameters:

tensor – a tensor to check

class ClContext : public ov::RemoteContext#
#include <ocl.hpp>

This class represents an abstraction for GPU plugin remote context which is shared with OpenCL context object. The plugin object derived from this class can be obtained either with CompiledModel::get_context() or Core::create_context() calls.

Subclassed by ov::intel_gpu::ocl::D3DContext, ov::intel_gpu::ocl::VAContext

Public Functions

inline ClContext(Core &core, cl_context ctx, int ctx_device_id = 0)#

Constructs context object from user-supplied OpenCL context handle.

Parameters:

  • core – A reference to OpenVINO Runtime Core object

  • ctx – A OpenCL context to be used to create shared remote context

  • ctx_device_id – An ID of device to be used from ctx

inline ClContext(Core &core, cl_command_queue queue)#

Constructs context object from user-supplied OpenCL context handle.

Note

Only latency mode is supported for such context sharing case.

Parameters:

  • core – A reference to OpenVINO Runtime Core object

  • queue – An OpenCL queue to be used to create shared remote context. Queue will be reused inside the plugin.

inline cl_context get()#

Returns the underlying OpenCL context handle.

Returns:

cl_context

inline operator cl_context()#

OpenCL context handle conversion operator for the ClContext object.

Returns:

cl_context

inline operator cl::Context()#

Standard Khronos cl::Context wrapper conversion operator for the ClContext object.

Returns:

cl::Context object

inline std::pair<ClImage2DTensor, ClImage2DTensor> create_tensor_nv12(const cl::Image2D &nv12_image_plane_y, const cl::Image2D &nv12_image_plane_uv)#

This function is used to construct a NV12 compound tensor object from two cl::Image2D wrapper objects. The resulting compound contains two remote tensors for Y and UV planes of the surface.

Parameters:

  • nv12_image_plane_y – cl::Image2D object containing Y plane data.

  • nv12_image_plane_uv – cl::Image2D object containing UV plane data.

Returns:

A pair of remote tensors for each plane

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, const cl_mem buffer)#

This function is used to obtain remote tensor object from user-supplied cl_mem object.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • buffer – A cl_mem object that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, const cl::Buffer &buffer)#

This function is used to obtain remote tensor object from user-supplied cl::Buffer object.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • buffer – A cl::Buffer object that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline ClImage2DTensor create_tensor(const element::Type type, const Shape &shape, const cl::Image2D &image)#

This function is used to obtain remote tensor object from user-supplied cl::Image2D object.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • image – A cl::Image2D object that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, SharedBufferHandle handle)#

This function is used to obtain a remote tensor object from a user-supplied external memory handle.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • SharedBufferHandle – External memory handle from another API (DX12 shared NT handle on Windows passed as void*, DMA-BUF fd on Linux passed as int)

Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, VirtualAddressMemory buff)#

This function is used to obtain a remote tensor object from a user-supplied VirtualAddressMemory.

Parameters:
Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, const FileDescriptor &file_descriptor)#

This function is used to obtain a remote tensor object from a file. The plugin memory-maps the file and keeps the mapping alive for the whole tensor lifetime, so the file must not be modified until the returned tensor is destroyed.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • file_descriptor – Descriptor with the path, offset and access mode of the file containing tensor data. The offset must be a multiple of the system memory mapping alignment: the page size on Linux (typically 4 KiB) and the allocation granularity on Windows (typically 64 KiB). AccessMode::READ_WRITE additionally requires the file to be writable by the calling process and makes the tensor writes visible in the file.

Returns:

A remote tensor instance

inline USMTensor create_tensor(const element::Type type, const Shape &shape, void *usm_ptr)#

This function is used to obtain remote tensor object from user-supplied USM pointer.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • usm_ptr – A USM pointer that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline USMTensor create_usm_host_tensor(const element::Type type, const Shape &shape)#

This function is used to allocate USM tensor with host allocation type.

Parameters:
Returns:

A remote tensor instance

inline USMTensor create_usm_device_tensor(const element::Type type, const Shape &shape)#

This function is used to allocate USM tensor with device allocation type.

Parameters:
Returns:

A remote tensor instance

RemoteTensor create_tensor(const element::Type &type, const Shape &shape, const AnyMap &params = {})#

Allocates memory tensor in device memory or wraps user-supplied memory handle using the specified tensor description and low-level device-specific parameters. Returns a pointer to the object that implements the RemoteTensor interface.

Parameters:

  • type – Defines the element type of the tensor.

  • shape – Defines the shape of the tensor.

  • params – Map of the low-level tensor object parameters.

Returns:

Pointer to a plugin object that implements the RemoteTensor interface.

Public Static Functions

static inline void type_check(const RemoteContext &remote_context)#

Checks that type defined runtime parameters are presented in remote object.

Parameters:

remote_context – A remote context to check

class VASurfaceTensor : public ov::intel_gpu::ocl::ClImage2DTensor#
#include <va.hpp>

This class represents an abstraction for GPU plugin remote tensor which is shared with VA output surface. The plugin object derived from this class can be obtained with VAContext::create_tensor() call.

Note

User can also obtain OpenCL 2D image handle from this class.

Public Functions

inline operator VASurfaceID()#

VASurfaceID conversion operator for the VASurfaceTensor object.

Returns:

VASurfaceID handle

inline uint32_t plane()#

Returns plane ID of underlying video decoder surface.

Returns:

Plane ID

Public Static Functions

static inline void type_check(const Tensor &tensor)#

Checks that type defined runtime parameters are presented in remote object.

Parameters:

tensor – a tensor to check

class VAContext : public ov::intel_gpu::ocl::ClContext#
#include <va.hpp>

This class represents an abstraction for GPU plugin remote context which is shared with VA display object. The plugin object derived from this class can be obtained either with CompiledModel::get_context() or Core::create_context() calls.

Note

User can also obtain OpenCL context handle from this class.

Public Functions

inline operator VADisplay()#

VADisplay conversion operator for the VAContext object.

Returns:

Underlying VADisplay object handle

inline VAContext(Core &core, VADisplay device, int target_tile_id = -1)#

Constructs remote context object from VA display handle.

Parameters:

  • core – OpenVINO Runtime Core object

  • device – A VADisplay to create remote context from

  • target_tile_id – Desired tile id within given context for multi-tile system. Default value (-1) means that root device should be used

inline std::pair<VASurfaceTensor, VASurfaceTensor> create_tensor_nv12(const size_t height, const size_t width, const VASurfaceID nv12_surf)#

This function is used to obtain a NV12 tensor from NV12 VA decoder output. The resulting tensor contains two remote tensors for Y and UV planes of the surface.

Note

The application must keep the VADisplay and the exact VA surface allocation identified by nv12_surf valid until both returned tensors and all their copies are destroyed. For every infer request to which either tensor was passed, also keep the surface valid until the request is destroyed or an inference using a replacement tensor has completed. If either tensor is passed to the AUTO or MULTI plugin, keep the surface valid until the compiled model is destroyed because its internal requests are not exposed to the application. Destroying the surface or recycling its ID earlier results in undefined behavior.

Parameters:

  • height – A height of Y plane

  • width – A width of Y plane

  • nv12_surf – NV12 VASurfaceID to create NV12 from

Returns:

A pair of remote tensors for each plane

inline VASurfaceTensor create_tensor(const element::Type type, const Shape &shape, const VASurfaceID surface, const uint32_t plane = 0)#

This function is used to create remote tensor from VA surface handle.

Note

The application must keep the VADisplay and the exact VA surface allocation identified by surface valid until the returned tensor and all its copies are destroyed. For every infer request to which the tensor was passed, also keep the surface valid until the request is destroyed or an inference using a replacement tensor has completed. If the tensor is passed to the AUTO or MULTI plugin, keep the surface valid until the compiled model is destroyed because its internal requests are not exposed to the application. Destroying the surface or recycling its ID earlier results in undefined behavior.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • surface – A VASurfaceID to create remote tensor from

  • plane – An index of a plane inside VASurfaceID to create tensor from

Returns:

A remote tensor wrapping VASurfaceID

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, const cl_mem buffer)#

This function is used to obtain remote tensor object from user-supplied cl_mem object.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • buffer – A cl_mem object that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, const cl::Buffer &buffer)#

This function is used to obtain remote tensor object from user-supplied cl::Buffer object.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • buffer – A cl::Buffer object that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline ClImage2DTensor create_tensor(const element::Type type, const Shape &shape, const cl::Image2D &image)#

This function is used to obtain remote tensor object from user-supplied cl::Image2D object.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • image – A cl::Image2D object that should be wrapped by a remote tensor

Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, SharedBufferHandle handle)#

This function is used to obtain a remote tensor object from a user-supplied external memory handle.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • SharedBufferHandle – External memory handle from another API (DX12 shared NT handle on Windows passed as void*, DMA-BUF fd on Linux passed as int)

Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, VirtualAddressMemory buff)#

This function is used to obtain a remote tensor object from a user-supplied VirtualAddressMemory.

Parameters:
Returns:

A remote tensor instance

inline ClBufferTensor create_tensor(const element::Type type, const Shape &shape, const FileDescriptor &file_descriptor)#

This function is used to obtain a remote tensor object from a file. The plugin memory-maps the file and keeps the mapping alive for the whole tensor lifetime, so the file must not be modified until the returned tensor is destroyed.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • file_descriptor – Descriptor with the path, offset and access mode of the file containing tensor data. The offset must be a multiple of the system memory mapping alignment: the page size on Linux (typically 4 KiB) and the allocation granularity on Windows (typically 64 KiB). AccessMode::READ_WRITE additionally requires the file to be writable by the calling process and makes the tensor writes visible in the file.

Returns:

A remote tensor instance

inline USMTensor create_tensor(const element::Type type, const Shape &shape, void *usm_ptr)#

This function is used to obtain remote tensor object from user-supplied USM pointer.

Parameters:

  • type – Tensor element type

  • shape – Tensor shape

  • usm_ptr – A USM pointer that should be wrapped by a remote tensor

Returns:

A remote tensor instance

RemoteTensor create_tensor(const element::Type &type, const Shape &shape, const AnyMap &params = {})#

Allocates memory tensor in device memory or wraps user-supplied memory handle using the specified tensor description and low-level device-specific parameters. Returns a pointer to the object that implements the RemoteTensor interface.

Parameters:

  • type – Defines the element type of the tensor.

  • shape – Defines the shape of the tensor.

  • params – Map of the low-level tensor object parameters.

Returns:

Pointer to a plugin object that implements the RemoteTensor interface.

Public Static Functions

static inline void type_check(const RemoteContext &remote_context)#

Checks that type defined runtime parameters are presented in remote object.

Parameters:

remote_context – A remote context to check

struct SharedBufferHandle#
#include <remote_properties.hpp>

Platform OS memory handle for importing externally allocated memory into GPU plugin tensors. On Linux this is a DMA-BUF file descriptor (int). On Windows this is a DX12 shared NT HANDLE (void*).

Public Types

using value_type = void*#

DX12 shared NT HANDLE.

struct VirtualAddressMemory#
#include <remote_properties.hpp>

Host (CPU) memory descriptor for wrapping mmap-backed or aligned host buffers as GPU plugin tensors without copying.

Public Members

int64_t size = -1#

Buffer size in bytes; -1 means “derive from tensor shape”.

AccessMode access = AccessMode::READ_WRITE#

Whether the plugin may only read this memory or also write to it.

struct FileDescriptor#
#include <remote_properties.hpp>

File descriptor for wrapping tensor data memory-mapped from a file as a GPU plugin tensor. The plugin memory-maps the file and keeps the mapping alive for the whole tensor lifetime, so the file must not be modified until the returned tensor is destroyed.

Public Members

std::filesystem::path path#

File path.

std::size_t offset = 0#

Offset in bytes to read from the file.

AccessMode access = AccessMode::READ#

Access mode of the mapping.