lightonai/embeddings-fine-tuning-multilingual-unfiltered

Overview

This dataset provides multilingual and code retrieval data for fine-tuning text embedding models. It is composed of high quality data sources with mined documents annotated with bi-encoder scores. For each query, the 2048 closest documents are mined with snowflake-arctic-embed-l-v2.0 for MIRACL and MLDR and with gte-modernbert-base for CodeEditSearchTrain, and annotated with their bi-encoder similarity score. No false-negative filtering or cross-encoder annotation is applied, enabling custom negative selection and filtering strategies such as the NV-Retriever setup: mine the closest documents to each query as negatives, and filter out false negatives if their bi-encoder similarity is higher than a percentage of the query-positive similarity score. The mined datasets are MIRACL, MLDR and CodeEditSearchTrain, each sample containing the query, the positive and the 2048 mined documents, covering 7 natural languages (Arabic, English, French, German, Italian, Portuguese, Spanish) and 13 programming languages.

For more information, please read our multilingual models blog post, our English models blog post and our paper.

How to use

The mined documents are annotated with their bi-encoder similarity score but are not filtered, so using the data as contrastive data in either sentence-transformers or PyLate first requires selecting the negatives among them and mapping them to the contrastive format. The code below applies the NV-Retriever filtering (a mined document is kept as a negative only if its similarity score is below nv_threshold times the query-positive score) that keeps the num_negatives hardest remaining documents, drops the samples left with fewer valid negatives, and maps everything to the (query, positive, negative_0, negative_1, ..., negative_n) contrastive format. Our filtered datasets, such as the English fine-tuning, are available on our collection and provide annotations from mxbai-rerank-large-v2 cross-encoder scores that can be used as teacher scores for knowledge distillation.

Python code to cast to contrastive format

import datasets
import numpy


class NVRetrieverToContrastive:
    """Selects the negatives of a split with NV-Retriever filtering and maps it to the contrastive format.

    Parameters
    ----------
    queries
        Queries subset of the split.
    documents
        Documents subset of the split, looked up by document_id.
    num_negatives
        Number of hard negatives to keep per query.
    nv_threshold
        A mined document is a negative only if its score is below nv_threshold * positive score.
    """

    def __init__(
        self,
        queries: datasets.Dataset,
        documents: datasets.Dataset,
        num_negatives: int = 10,
        nv_threshold: float = 0.95,
    ) -> None:
        self.queries = dict(zip(queries["query_id"], queries["query"]))
        self.documents = documents
        self.document_ids = documents.with_format("numpy")["document_id"]
        self.num_negatives = num_negatives
        self.nv_threshold = nv_threshold

    def document(self, document_id: int) -> str:
        # document_ids are sorted but not contiguous, so the row holding the text is found by binary search
        row = int(numpy.searchsorted(self.document_ids, document_id))
        return self.documents[row]["document"]

    def negative_indices(self, example) -> list[int]:
        """Mined documents scored clearly below the positive and with a non-empty text."""
        threshold = self.nv_threshold * example["scores"][0]
        negatives = []
        for candidate in range(1, len(example["document_ids"])):
            if len(negatives) == self.num_negatives:
                break
            if example["scores"][candidate] < threshold and self.document(example["document_ids"][candidate]).strip():
                negatives.append(candidate)
        return negatives

    def has_enough_negatives(self, example) -> bool:
        return bool(
            example["document_ids"]
            and self.queries.get(example["query_id"], "").strip()
            and self.document(example["document_ids"][0]).strip()
            and len(self.negative_indices(example)) == self.num_negatives
        )

    def map_to_query_positive_negatives(self, example) -> dict:
        negatives = self.negative_indices(example)
        return {
            "query": self.queries[example["query_id"]],
            "positive": self.document(example["document_ids"][0]),
            **{
                f"negative_{negative}": self.document(example["document_ids"][candidate])
                for negative, candidate in enumerate(negatives)
            },
        }


def load_train_datasets(num_negatives: int = 10, nv_threshold: float = 0.95) -> datasets.DatasetDict:
    """Load every split as a (query, positive, negative_0, ..., negative_n) dataset."""
    repo = "lightonai/embeddings-fine-tuning-multilingual-unfiltered"
    splits = [
        "miracl_ar", "miracl_en", "miracl_es", "miracl_fr", "mldr_en", "mldr_ar", "mldr_de",
        "mldr_es", "mldr_fr", "mldr_it", "mldr_pt", "CodeEditSearch_c", "CodeEditSearch_cpp",
        "CodeEditSearch_go", "CodeEditSearch_java", "CodeEditSearch_javascript",
        "CodeEditSearch_php", "CodeEditSearch_python", "CodeEditSearch_ruby",
        "CodeEditSearch_rust", "CodeEditSearch_scala", "CodeEditSearch_shell",
        "CodeEditSearch_swift", "CodeEditSearch_typescript",
    ]

    train_dataset = datasets.DatasetDict()
    for split in splits:
        # data_files restricts the download to the split being processed, hence skipping the checks on the other splits
        load = lambda config: datasets.load_dataset(
            repo,
            name=config,
            data_files=f"{config}/{split}-*",
            split="train",
            verification_mode="no_checks",
        )
        scores = load("scores")
        processor = NVRetrieverToContrastive(
            queries=load("queries"),
            documents=load("documents"),
            num_negatives=num_negatives,
            nv_threshold=nv_threshold,
        )
        train_dataset[split] = scores.filter(
            processor.has_enough_negatives,
            desc=f"Filtering the queries with less than {num_negatives} negatives ({split})",
        ).map(
            processor.map_to_query_positive_negatives,
            remove_columns=scores.column_names,
            desc=f"Creating the contrastive dataset ({split})",
        )
    return train_dataset


train_dataset = load_train_datasets()
print(train_dataset)

Dataset structure

The dataset is composed of 3 high quality datasets (MIRACL, MLDR and CodeEditSearchTrain) across 24 language splits, defined by the splits parameters. Each split contains 3 subsets, one containing the queries, one containing the documents and one joining tables also containing the corresponding pairwise query-documents scores.

Documents

Column	Type	Description
document_id	int64	Unique identifier of the document within the split.
document	string	Raw text of the document/passage.

Split	Rows
miracl_ar	2.01M
miracl_en	32.00M
miracl_es	10.03M
miracl_fr	13.93M
mldr_en	207k
mldr_ar	8.8k
mldr_de	11k
mldr_es	11k
mldr_fr	10.7k
mldr_it	11.2k
mldr_pt	7.9k
CodeEditSearch_c	4.3k
CodeEditSearch_cpp	1.7k
CodeEditSearch_go	2.9k
CodeEditSearch_java	9.6k
CodeEditSearch_javascript	48.7k
CodeEditSearch_php	16.9k
CodeEditSearch_python	37.9k
CodeEditSearch_ruby	57k
CodeEditSearch_rust	525
CodeEditSearch_scala	1.9k
CodeEditSearch_shell	22.3k
CodeEditSearch_swift	1.9k
CodeEditSearch_typescript	3.4k
Total	58.44M

Queries

Column	Type	Description
query_id	int64	Unique identifier of the query within the split.
query	string	Raw text of the query.

Split	Rows
miracl_ar	3.5k
miracl_en	2.9k
miracl_es	2.2k
miracl_fr	1.1k
mldr_en	10k
mldr_ar	1.8k
mldr_de	1.8k
mldr_es	2.3k
mldr_fr	1.6k
mldr_it	2.2k
mldr_pt	1.8k
CodeEditSearch_c	4.4k
CodeEditSearch_cpp	1.7k
CodeEditSearch_go	2.9k
CodeEditSearch_java	9.8k
CodeEditSearch_javascript	49.3k
CodeEditSearch_php	17.2k
CodeEditSearch_python	38.7k
CodeEditSearch_ruby	58.2k
CodeEditSearch_rust	533
CodeEditSearch_scala	2.0k
CodeEditSearch_shell	23k
CodeEditSearch_swift	1.9k
CodeEditSearch_typescript	3.4k
Total	244k

Scores

Column	Type	Description
query_id	int64	Identifier joining back to the corresponding row in queries.
document_ids	list[int64]	List of document IDs (joining back to documents). The first element is the positive document, followed by the 2048 closest documents mined with the bi-encoder, sorted by decreasing score (fewer when the split corpus is smaller).
scores	list[float]	Bi-encoder relevance scores for each document w.r.t the query, in the same order as document_ids. Can be used for negative filtering and knowledge distillation.

Split	Rows
miracl_ar	6.2k
miracl_en	7.9k
miracl_es	10k
miracl_fr	2.3k
mldr_en	10k
mldr_ar	1.8k
mldr_de	1.8k
mldr_es	2.3k
mldr_fr	1.6k
mldr_it	2.2k
mldr_pt	1.8k
CodeEditSearch_c	4.4k
CodeEditSearch_cpp	1.7k
CodeEditSearch_go	2.9k
CodeEditSearch_java	9.8k
CodeEditSearch_javascript	49.3k
CodeEditSearch_php	17.2k
CodeEditSearch_python	38.7k
CodeEditSearch_ruby	58.2k
CodeEditSearch_rust	533
CodeEditSearch_scala	2.0k
CodeEditSearch_shell	23k
CodeEditSearch_swift	1.9k
CodeEditSearch_typescript	3.4k
Total	261k

Citation

If you are using this dataset, please consider citing our work

@misc{sourty2026denseonlateonfullyopen,
  title         = {DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search},
  author        = {Raphaël Sourty and Antoine Chaffin and Paulo Roberto Moura Junior and Amélie Chatelain},
  year          = {2026},
  eprint        = {2607.27178},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2607.27178},
}```