LensVM

Introduction

LensVM is a bi-directional data transformation engine originally developed for DefraDB, now available as a standalone tool. It enables transforming data both forwards and in reverse directions using user-defined modules called Lenses, which are compiled to WebAssembly (WASM). Each Lens runs inside a secure, sandboxed WASM environment, enabling safe and modular pipelines, even when composed from multiple sources.

This guide provides the foundational steps for writing and composing Lenses using the LensVM framework. It includes examples for Rust-based Lenses using the official SDK, as well as lower-level implementations without the SDK in other languages.

Before you begin

Before getting started, ensure the following are installed:

• Golang (required to run the Lens engine)
• WASM-compatible compiler: Choose a compiler that targets the wasm32-unknown architecture, based on your preferred programming language.

Note: The LensVM Engine executes Lenses in isolated WASM environments. It manages data flow, memory allocation, and function calls between the host application and the Lens module.

Writing lenses

Lenses can be authored in any language that compiles to valid WebAssembly. To interface with the LensVM engine, each Lens must implement a specific set of exported and imported functions.

Required and optional functions

Each Lens must implement the following interface:

Function	Type	Required	Description
alloc(unsigned64)	Exported	Yes	Allocates a memory block of the given size. Called by the LensVM engine.
next() -> unsigned8	Imported	Yes	Called by the Lens to retrieve a pointer to the next input data item from the engine.
set_param(unsigned8) -> unsigned8	Exported	No	Accepts static configuration data at initialization. Receives a pointer to the config and returns a pointer to an OK or error response. Called once before any input is processed.
transform() -> unsigned8	Exported	Yes	Core transformation logic. Pulls zero or more inputs using next(), applies transformation, and returns a pointer to a single output item. Supports stateful transformations.
inverse() -> unsigned8	Exported	No	Optional reverse transformation logic, same interface as transform().

WASM data format

LensVM communicates with Lenses using a binary format across the WASM boundary. The format is as follows:

[TypeId][Length][Payload]

• TypeId: A signed 8-byte integer
• Length: An optional unsigned 32-byte integer, depending on the TypeId
• Payload: Raw binary or serialized data (e.g., JSON)

TypeId Values

TypeId	Meaning	Notes
-1	Error	May include an error message in the Payload.
0	Nil	No Length or Payload.
1	JSON	Payload contains a JSON-serialized object.
127	End of Stream	Signals that there are no more items to process.

Developing with the Rust SDK

To simplify development, LensVM provides a Rust SDK. It abstracts much of the boilerplate required to build Lenses, allowing you to focus on transformation logic.

The SDK:

• Implements the required interface automatically
• Handles safe memory and data exchange across the WASM boundary
• Provides helpful macros and utilities for Lens definition

You can find it on crates.io and in the official GitHub repository.

Example Lenses

Example Lenses written in:

• Rust
• AssemblyScript

can be found in this repository and in DefraDB.

Basic Lens Example

The easiest way to get started writing a Lens is by using Rust, thanks to the lens_sdk crate, which provides helpful macros and utilities for Lens development.

A minimal example is shown in the define! macro documentation. This example demonstrates a simple forward transformation that iterates through input documents and increments the age field by 1.

Writing a Lens using the SDK

Writing a Lens with the Rust SDK is straightforward and well-documented. The examples provided in the lens_sdk documentation build progressively:

• Example 1: A minimal forward transformation using the define! macro.
• Example 2: Adds parameters and an inverse function to demonstrate bi-directional transformations.

For more advanced examples, refer to the following repositories:

• Lens test modules
• DefraDB lens tests

These cover schema-aware transformations, reversible pipelines, and other real-world use cases.

Writing a Lens without the SDK

Creating a Lens without the Rust SDK is intended for advanced use cases—such as developing in non-Rust languages or needing fine-grained control over serialization and transformation behavior.

Currently, the only working non-Rust example is written in AssemblyScript:

• AssemblyScript Lens Example

This approach requires:

• Manual implementation of memory allocation and serialization
• A deep understanding of the LensVM protocol
• Proficiency in AssemblyScript (or your chosen language)

Recommendation: For most users, we strongly recommend using the Rust SDK, even partially. It can significantly reduce development time and complexity. You can start with full SDK support and incrementally replace parts with custom logic as needed.

Composing Lenses

Lenses can be composed into pipelines using the Go config sub-package:

• Go Config Package

Pipeline composition is handled via the model.Lens type:

• model.Lens Definition

You can compose pipelines either by:

• Supplying a model.Lens object directly
• Referencing a JSON configuration file (from local storage or a URL) that conforms to the model.Lens schema

Note: Composing Lenses does not execute them. Instead, it builds an enumerable pipeline object, which you can then iterate over to apply transformations.

You can extend this enumerable pipeline by:

• Adding additional Lenses through the config package
• Chaining in native Go-based enumerables for advanced customization

Composition Examples

For practical examples of pipeline composition, explore the following:

• Go engine tests
• CLI integration tests

These examples demonstrate how to build and extend Lens pipelines declaratively for various environments and workflows.