tvm.runtime¶

TVM runtime namespace.

class tvm.runtime.DataType(type_str)¶: TVM datatype structure

class tvm.runtime.DataTypeCode¶: DataType code in DLTensor.

class tvm.runtime.Device(device_type, device_id)¶

TVM device strucure.

Typically constructed using convenience function tvm.runtime.device().

Exposes uniform interface to device-specific APIs such as CUDA or OpenCL. Some properties may return None depending on whether an API exposes that particular property.

NOTE! The integer values in MASK2STR and STR2MASK must correspond to the values provided by the DLDeviceType and TVMDeviceExtType enums.

property api_version¶

Returns version number of the SDK used to compile TVM.

For example, CUDA_VERSION for cuda or VK_HEADER_VERSION for Vulkan.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: version – The version of the SDK
Return type:: int or None

property compute_version¶

Get compute version number as string.

Returns maximum API version (e.g. CUDA/OpenCL/Vulkan) supported by the device.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: version – The version string in major.minor format.
Return type:: str or None

create_raw_stream()¶

Create a new runtime stream at the context.

User should free the stream after use.

Returns:: stream – The created runtime stream.
Return type:: TVMStreamHandle

property device_name¶

Return the vendor-specific name of device.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: device_name – The name of the device.
Return type:: str or None

property driver_version¶

Returns version number of the driver

Returns driver vendor’s internal version number. (e.g. “450.408.256” for nvidia-driver-450)

Returns device value for opencl and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: version – The version string in major.minor.patch format.
Return type:: str or None

property exist¶

Whether this device exists.

Returns True if TVM has support for the device, if the physical device is present, and the device is accessible through appropriate drivers (e.g. cuda/vulkan).

Returns:: exist – True if the device exists
Return type:: bool

free_raw_stream(stream)¶

Free a created stream handle.

Parameters:: stream (TVMStreamHandle) – The stream which should to be released.

property l2_cache_size_bytes¶

Return the size of the device L2 cache in bytes

Supported devices include CUDA/ROCM/OpenCL.

Returns:: l2_cache_size_bytes – The size of the device L2 cache in bytes returned by device runtime API. Return None if the device does not support this feature.
Return type:: int or None

Note

The value returned by opencl’s API is smaller than actual device L2 cache size.

property max_clock_rate¶

Return the max clock frequency of device (kHz).

Returns device value for cuda, rocm, and opencl. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: max_clock_rate – The maximum clock frequency of the device (kHz)
Return type:: int or None

property max_shared_memory_per_block¶

Total amount of shared memory per block in bytes.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: max_shared_memory_per_block – Total amount of shared memory per block in bytes
Return type:: int or None

property max_thread_dimensions¶

Return the maximum size of each thread axis

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: dims – The maximum length of threadIdx.x, threadIdx.y, threadIdx.z
Return type:: List of int, or None

property max_threads_per_block¶

Maximum number of threads on each block.

Returns device value for cuda, metal, rocm, opencl, and vulkan devices. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: max_threads_per_block – The number of threads on each block
Return type:: int or None

property multi_processor_count¶

Return the number of compute units in the device.

Returns device value for cuda, rocm, and opencl. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: multi_processor_count – Thee number of compute units in the device
Return type:: int or None

set_raw_stream(stream)¶

Set a created stream handle.

Parameters:: stream (TVMStreamHandle) – The stream which should to be set to the device.

sync(stream=None)¶

Synchronize until jobs finished at the context.

Parameters:: stream (TVMStreamHandle) – Jobs in this stream should be finished.

texture_spatial_limit()¶

Returns limits for textures by spatial dimensions

Returns:: limit – Maximum size of the texture by spatial dimensions
Return type:: int or None

property total_global_memory¶

Return size of the total global memory.

Supported devices include CUDA/ROCm/Metal/OpenCL.

Returns:: total_global_memory – Return the global memory available on device in bytes. Return None if the device does not support this feature.
Return type:: int or None

property warp_size¶

Number of threads that execute concurrently.

Returns device value for cuda, rocm, and vulkan. Returns 1 for metal and opencl devices, regardless of the physical device. Returns remote device value for RPC devices. Returns None for all other devices.

Returns:: warp_size – Number of threads that execute concurrently
Return type:: int or None

class tvm.runtime.Module(handle)¶

Runtime Module.

clear_imports()¶: Remove all imports of the module.

property entry_func¶

Get the entry function

Returns:: f – The entry function if exist
Return type:: tvm.runtime.PackedFunc

export_library(file_name, *, fcompile=None, fpack_imports=None, addons=None, workspace_dir=None, **kwargs)¶

Export the module and all imported modules into a single device library.

This function only works on host LLVM modules, other runtime::Module subclasses will work with this API but they must support implement the save and load mechanisms of modules completely including saving from streams and files. This will pack your non-shared library module into a single shared library which can later be loaded by TVM.

Parameters:

file_name (str) – The name of the shared library.
fcompile (function(target, file_list, kwargs), optional) –
The compilation function to use create the final library object during export.

For example, when fcompile=_cc.create_shared, or when it is not supplied but module is “llvm,” this is used to link all produced artifacts into a final dynamic library.

This behavior is controlled by the type of object exported. If fcompile has attribute object_format, will compile host library to that format. Otherwise, will use default format “o”.
fpack_imports (function(mod: runtime.Module, is_system_lib: bool, symbol_prefix: str,) –
workspace_dir: str) -> str Function used to pack imported modules from mod into a file suitable for passing to fcompile as an input file. The result can be a C source, or an .o object file, or any other file that the fcompile function can handle. The function returns the name of the created file.

If not provided, the imported modules will be serialized either via packing to an LLVM module, or to a C source file.
workspace_dir (str, optional) – The path of the directory used to create the intermediate artifacts when exporting the module. If this is not provided a temporary dir will be created.
kwargs (dict, optional) – Additional arguments passed to fcompile

Returns:

result of fcompile() – If the compilation function returns an artifact it would be returned via export_library, if any.

Return type:

unknown, optional

property format¶: Get the format of the module.

get_function(name, query_imports=False)¶

Get function from the module.

Parameters:

name (str) – The name of the function
query_imports (bool) – Whether also query modules imported by this module.

Returns:

f – The result function.

Return type:

tvm.runtime.PackedFunc

get_property_mask()¶

Get the runtime module property mask. The mapping is stated in ModulePropertyMask.

Returns:: mask – Bitmask of runtime module property
Return type:: int

get_source(fmt='')¶

Get source code from module, if available.

Parameters:: fmt (str, optional) – The specified format.
Returns:: source – The result source code.
Return type:: str

implements_function(name, query_imports=False)¶

Returns True if the module has a definition for the global function with name. Note that has_function(name) does not imply get_function(name) is non-null since the module may be, eg, a CSourceModule which cannot supply a packed-func implementation of the function without further compilation. However, get_function(name) non null should always imply has_function(name).

Parameters:

name (str) – The name of the function
query_imports (bool) – Whether to also query modules imported by this module.

Returns:

b – True if module (or one of its imports) has a definition for name.

Return type:

Bool

import_module(module)¶

Add module to the import list of current one.

Parameters:: module (tvm.runtime.Module) – The other module.

property imported_modules¶

Get imported modules

Returns:: modules – The module
Return type:: list of Module

property is_binary_serializable¶

Returns true if module is ‘binary serializable’, ie can be serialzed into binary: stream and loaded back to the runtime module.

Returns:: b – True if the module is binary serializable.
Return type:: Bool

property is_dso_exportable¶

Returns true if module is ‘DSO exportable’, ie can be included in result of export_library by the external compiler directly.

Returns:: b – True if the module is DSO exportable.
Return type:: Bool

property is_runnable¶

Returns true if module is ‘runnable’. ie can be executed without any extra compilation/linking steps.

Returns:: b – True if the module is runnable.
Return type:: Bool

save(file_name, fmt='')¶

Save the module to file.

This do not save the dependent device modules. See also export_shared

Parameters:

file_name (str) – The name of the file.
fmt (str) – The format of the file.

See also

tvm.register_func: How to register global function.
tvm.get_global_func: How to get global function.

class tvm.runtime.Report(calls: Sequence[Dict[str, Object]], device_metrics: Dict[str, Dict[str, Object]], configuration: Dict[str, Object])¶

A container for information gathered during a profiling run.

calls¶

Per-call profiling metrics (function name, runtime, device, …).

Type:: Array[Dict[str, Object]]

device_metrics¶

Per-device metrics collected over the entire run.

Type:: Dict[Device, Dict[str, Object]]

csv()¶

Convert this profiling report into CSV format.

This only includes calls and not overall metrics.

Returns:: csv – calls in CSV format.
Return type:: str

classmethod from_json(s)¶

Deserialize a report from JSON.

Parameters:: s (str) – Report serialize via json().
Returns:: report – The deserialized report.
Return type:: Report

json()¶

Convert this profiling report into JSON format.

Example output:

Returns:: json – Formatted JSON
Return type:: str

table(sort=True, aggregate=True, col_sums=True)¶

Generate a human-readable table

Parameters:

sort (bool) – If aggregate is true, whether to sort call frames by descending duration. If aggregate is False, whether to sort frames by order of appearancei n the program.
aggregate (bool) – Whether to join multiple calls to the same op into a single line.
col_sums (bool) – Whether to include the sum of each column.

Returns:

table – A human-readable table

Return type:

str

class tvm.runtime.Scriptable¶

A base class that enables the script() and show() method.

script(*, name: str | None = None, show_meta: bool = False, ir_prefix: str = 'I', tir_prefix: str = 'T', relax_prefix: str = 'R', module_alias: str = 'cls', buffer_dtype: str = 'float32', int_dtype: str = 'int32', float_dtype: str = 'void', verbose_expr: bool = False, indent_spaces: int = 4, print_line_numbers: bool = False, num_context_lines: int = -1, syntax_sugar: bool = True, show_object_address: bool = False, path_to_underline: List[ObjectPath] | None = None, path_to_annotate: Dict[ObjectPath, str] | None = None, obj_to_underline: List[Object] | None = None, obj_to_annotate: Dict[Object, str] | None = None) → str¶

Print TVM IR into TVMScript text format

Parameters:

name (Optional[str] = None) – The name of the object
show_meta (bool = False) – Whether to print the meta data of the object
ir_prefix (str = "I") – The prefix of AST nodes from tvm.ir
tir_prefix (str = "T") – The prefix of AST nodes from tvm.tir
relax_prefix (str = "R") – The prefix of AST nodes from tvm.relax
module_alias (str = "cls") – The alias of the current module at cross-function call, Directly use module name if it’s empty.
buffer_dtype (str = "float32") – The default data type of buffer
int_dtype (str = "int32") – The default data type of integer
float_dtype (str = "void") – The default data type of float
verbose_expr (bool = False) – Whether to print the detailed definition of each variable in the expression
indent_spaces (int = 4) – The number of spaces for indentation
print_line_numbers (bool = False) – Whether to print line numbers
num_context_lines (int = -1) – The number of lines of context to print before and after the line to underline.
syntax_sugar (bool = True) – Whether to output with syntax sugar, set false for complete printing.
show_object_address (bool = False) – Whether to include the object’s address as part of the TVMScript name
path_to_underline (Optional[List[ObjectPath]] = None) – Object path to be underlined
path_to_annotate (Optional[Dict[ObjectPath, str]] = None) – Object path to be annotated
obj_to_underline (Optional[List[Object]] = None) – Object to be underlined
obj_to_annotate (Optional[Dict[Object, str]] = None) – Object to be annotated

Returns:

script – The TVM Script of the given TVM IR

Return type:

str

show(style: str | None = None, black_format: bool | None = None, *, name: str | None = None, show_meta: bool = False, ir_prefix: str = 'I', tir_prefix: str = 'T', relax_prefix: str = 'R', module_alias: str = 'cls', buffer_dtype: str = 'float32', int_dtype: str = 'int32', float_dtype: str = 'void', verbose_expr: bool = False, indent_spaces: int = 4, print_line_numbers: bool = False, num_context_lines: int = -1, syntax_sugar: bool = True, show_object_address: bool = False, path_to_underline: List[ObjectPath] | None = None, path_to_annotate: Dict[ObjectPath, str] | None = None, obj_to_underline: List[Object] | None = None, obj_to_annotate: Dict[Object, str] | None = None) → None¶

A sugar for print highlighted TVM script.

Parameters:

style (str, optional) – Pygmentize printing style, auto-detected if None. See tvm.script.highlight.cprint for more details.
black_format (Optional[bool]) –
If true, use the formatter Black to format the TVMScript. If false, do not apply the auto-formatter.

If None (default), determine the behavior based on the environment variable “TVM_BLACK_FORMAT”. If this environment variable is unset, set to the empty string, or set to the integer zero, black auto-formatting will be disabled. If the environment variable is set to a non-zero integer, black auto-formatting will be enabled.

Note that the “TVM_BLACK_FORMAT” environment variable only applies to the .show() method, and not the underlying .script() method. The .show() method is intended for human-readable output based on individual user preferences, while the .script() method is intended to provided a consistent output regardless of environment.
name (Optional[str] = None) – The name of the object
show_meta (bool = False) – Whether to print the meta data of the object
ir_prefix (str = "I") – The prefix of AST nodes from tvm.ir
tir_prefix (str = "T") – The prefix of AST nodes from tvm.tir
relax_prefix (str = "R") – The prefix of AST nodes from tvm.relax
module_alias (str = "cls") – The alias of the current module at cross-function call, Directly use module name if it’s empty.
buffer_dtype (str = "float32") – The default data type of buffer
int_dtype (str = "int32") – The default data type of integer
float_dtype (str = "void") – The default data type of float
verbose_expr (bool = False) – Whether to print the detailed definition of each variable in the expression
indent_spaces (int = 4) – The number of spaces for indentation
print_line_numbers (bool = False) – Whether to print line numbers
num_context_lines (int = -1) – The number of lines of context to print before and after the line to underline.
syntax_sugar (bool = True) – Whether to output with syntax sugar, set false for complete printing.
show_object_address (bool = False) – Whether to include the object’s address as part of the TVMScript name
path_to_underline (Optional[List[ObjectPath]] = None) – Object path to be underlined
path_to_annotate (Optional[Dict[ObjectPath, str]] = None) – Object path to be annotated
obj_to_underline (Optional[List[Object]] = None) – Object to be underlined
obj_to_annotate (Optional[Dict[Object, str]] = None) – Object to be annotated

class tvm.runtime.ShapeTuple(shape)¶: TVM runtime ShapeTuple object. :param shape: The shape list used to construct the object. :type shape: list[int]

class tvm.runtime.String(content)¶

TVM runtime.String object, represented as a python str.

Parameters:: content (str) – The content string used to construct the object.

tvm.runtime.cl(dev_id=0)¶

Construct a OpenCL device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

tvm.runtime.const(value, dtype=None, span=None)¶

construct a constant

Parameters:

value (number) – The content of the constant number.
dtype (str or None, optional) – The data type.
span (Optional[Span]) – The location of the constant value in the source.

Returns:

const_val – The result expression.

Return type:

tvm.Expr

tvm.runtime.convert(value, span=None)¶

Convert value to TVM object or function.

Parameters:

value (python value) –
span (Optional[Span]) – The location of this statement in the source code.

Returns:

tvm_val – Converted value in TVM

Return type:

Object or Function

Note

This function is redirected to convert_to_object as it is widely used in the codebase. We can choose one to keep and discard the other one later.

tvm.runtime.convert_to_object(value, span=None)¶

Convert a Python value to corresponding object type.

Parameters:

value (str) – The value to be inspected.
span (Optional[Span]) – The location of this itervar in the source code.

Returns:

obj – The corresponding object value.

Return type:

Object

tvm.runtime.cpu(dev_id=0)¶

Construct a CPU device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

tvm.runtime.cuda(dev_id=0)¶

Construct a CUDA GPU device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

tvm.runtime.device(dev_type, dev_id=0)¶

Construct a TVM device with given device type and id.

Parameters:

dev_type (int or str) – The device type mask or name of the device.
dev_id (int, optional) – The integer device id

Returns:

dev – The corresponding device.

Return type:

tvm.runtime.Device

Examples

Device can be used to create reflection of device by string representation of the device type.

assert tvm.device("cpu", 1) == tvm.cpu(1)
assert tvm.device("cuda", 0) == tvm.cuda(0)

tvm.runtime.enabled(target)¶

Whether module runtime is enabled for target

Parameters:: target (str) – The target device type.
Returns:: enabled – Whether runtime is enabled.
Return type:: bool

Examples

The following code checks if gpu is enabled.

>>> tvm.runtime.enabled("gpu")

tvm.runtime.ext_dev(dev_id=0)¶

Construct a extension device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

Note

This API is reserved for quick testing of new device by plugin device API as ext_dev.

tvm.runtime.gpu(dev_id=0)¶

Construct a CUDA GPU device

deprecated:: 0.9.0 Use tvm.cuda() instead.

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

tvm.runtime.load_module(path, fmt='')¶

Load module from file.

Parameters:

path (str) – The path to the module file.
fmt (str, optional) – The format of the file, if not specified it will be inferred from suffix of the file.

Returns:

module – The loaded module

Return type:

runtime.Module

Note

This function will automatically call cc.create_shared if the path is in format .o or .tar

tvm.runtime.load_param_dict(param_bytes)¶

Load parameter dictionary from binary bytes.

Parameters:: param_bytes (bytearray) – Serialized parameters.
Returns:: params – The parameter dictionary.
Return type:: dict of str to NDArray

tvm.runtime.load_param_dict_from_file(path)¶

Load parameter dictionary from file.

Parameters:: path (str) – The path to the parameter file to load from.
Returns:: params – The parameter dictionary.
Return type:: dict of str to NDArray

tvm.runtime.load_static_library(path, func_names)¶: Load the .o library at path which implements functions with func_names. Unlike the generic load_module the result will remain as a static_library and will not be relinked on-the-fly into a .so library.

tvm.runtime.metal(dev_id=0)¶

Construct a metal device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

tvm.runtime.mtl(dev_id=0)¶

Construct a metal device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

tvm.runtime.num_threads() → int¶

Get the number of threads in use by the TVM runtime.

Returns:: Number of threads in use.
Return type:: int

tvm.runtime.opencl(dev_id=0)¶

Construct a OpenCL device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

tvm.runtime.rocm(dev_id=0)¶

Construct a ROCM device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

tvm.runtime.save_param_dict(params)¶

Save parameter dictionary to binary bytes.

The result binary bytes can be loaded by the GraphModule with API “load_params”.

Parameters:: params (dict of str to NDArray) – The parameter dictionary.
Returns:: param_bytes – Serialized parameters.
Return type:: bytearray

Examples

# set up the parameter dict
params = {"param0": arr0, "param1": arr1}
# save the parameters as byte array
param_bytes = tvm.runtime.save_param_dict(params)
# We can serialize the param_bytes and load it back later.
# Pass in byte array to module to directly set parameters
tvm.runtime.load_param_dict(param_bytes)

tvm.runtime.save_param_dict_to_file(params, path)¶

Save parameter dictionary to file.

Parameters:

params (dict of str to NDArray) – The parameter dictionary.
path (str) – The path to the parameter file.

tvm.runtime.system_lib(symbol_prefix='')¶

Get system-wide library module singleton.

System lib is a global module that contains self register functions in startup. Unlike normal dso modules which need to be loaded explicitly. It is useful in environments where dynamic loading api like dlopen is banned.

To build system lib function, simply specify target option `llvm --system-lib` The system lib will be available as long as the result code is linked by the program.

The system lib is intended to be linked and loaded during the entire life-cyle of the program. If you want dynamic loading features, use dso modules instead.

Parameters:: symbol_prefix (Optional[str]) – Optional symbol prefix that can be used for search. When we lookup a symbol symbol_prefix + name will first be searched, then the name without symbol_prefix.
Returns:: module – The system-wide library module.
Return type:: runtime.Module

tvm.runtime.vpi(dev_id=0)¶

Construct a VPI simulated device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device

tvm.runtime.vulkan(dev_id=0)¶

Construct a Vulkan device

Parameters:: dev_id (int, optional) – The integer device id
Returns:: dev – The created device
Return type:: Device