HybridTM - Semantic Translation Memory Engine

HybridTM logo

HybridTM is a semantic translation memory engine that stores bilingual content in LanceDB and scores matches by combining semantic embeddings (Hugging Face Transformers.js) with the built-in MatchQuality fuzzy metric.

The source code is available on GitHub under the Eclipse Public License v1.0. Developers can clone, adapt, and ship the library under the terms of that license, or contact Maxprograms for commercial arrangements.

Highlights

  • Imports XLIFF 2.x, TMX 1.4b, and SDLTM files, preserving metadata, notes, and custom properties
  • Generates semantic vectors with any Hugging Face Transformers.js-compatible text model — ships three ready-to-use presets (compact, standard, large) and also accepts any other Hugging Face feature-extraction model id; default is HybridTM.LARGE_MODEL (gte-multilingual-base)
  • Provides semanticTranslationSearch, semanticSearch, and concordanceSearch APIs with metadata-aware filtering
  • Streams data into LanceDB through a JSONL-based batch importer to keep memory usage predictable
  • Prevents duplicate segments by rewriting entries with deterministic IDs (fileId:unitId:segmentIndex:lang)
  • Backs up any instance to a single format-agnostic XML file and restores it into an existing or brand-new instance — optionally under a different embedding model — making it easy to move data to another computer, switch models, or simply keep a backup
  • Ships a hybridtm command-line tool for creating instances, importing files, backing up and restoring data, and enriching XLIFF files with TM match candidates from the shell

Models download automatically the first time you initialize an instance and are cached in the standard Hugging Face directory.

The three embedding presets trade off footprint against accuracy differently depending on language: standard is a good default for Western European languages and matches or beats large on accuracy while being faster; large is worth its extra cost for morphologically rich languages such as Finnish; compact has the smallest footprint but should be treated as a deliberate trade-off rather than a safe default.

Requirements

  • Node.js 24 LTS or later
  • npm 11+
  • Disk space for both the LanceDB directory you choose and the embedding model cache