About
Desbordante is a high-performance data profiler that is capable of discovering and validating many different patterns in data using various algorithms.
Tool Positioning
Why Desbordante? Why not just use an LLM or other black-box machine learning approaches?
Unlike these, Desbordante takes a symbolic, explainable approach to data analysis, producing explicit rules or rule-shaped artifacts that are verifiable, such as data dependencies, association rules, or other patterns. This is a distinct paradigm for data analysis that complements black-box learning. In contrast to free-form model outputs, discovered patterns have precisely defined semantics and can be algorithmically validated against the underlying data.
These patterns add data-grounded and rule-based structure, which provides useful properties for systems built around machine learning and large language models: consistency, provenance, explainability, and auditability. Further, the patterns can serve as inputs to downstream data-quality checks, validation procedures, integrity constraints, training-data analysis, and other rule-based safeguards. They can also be used to choose a direction of an ablation study, to generate examples that violate rules for use in machine learning model training and testing, to assist fact completion, and to facilitate acceptance checks and guardrails for pipelines involving retrieval-augmented generation. This is roughly the motivation used for the work in [1], but it applies more broadly to the patterns Desbordante works with.
[1] François Amat. Mining Rules on Tabular Data. Institut Polytechnique de Paris, 2025.
From other angles, the discovered patterns can be used in many ways:
-
For scientific data, especially those obtained experimentally, an interesting pattern allows to formulate a hypothesis that could lead to a scientific discovery. In some cases it even allows one to draw conclusions immediately, if there is enough data. At the very least, the found pattern can be used to provide a direction for further study.
-
For business data, it is also possible to obtain a hypothesis based on found patterns. However, there are more down-to-earth and more in-demand applications in this case: clearing errors in data, finding and removing inexact duplicates, performing schema matching, and many more.
-
For database data, found patterns can help with recovering/defining primary and foreign keys, checking/setting up all kinds of integrity constraints.
General
Desbordante supports three types of tasks.
The Discovery task is designed to identify all instances of a specified pattern type of a given dataset.
The Validation task is different: it is designed to check whether a specified pattern instance is present in a given dataset. This task not only returns True or False, but it also explains why the instance does not hold (e.g. it can list table rows with conflicting values).
For some patterns Desbordante supports a dynamic task variant. The distinguishing feature of dynamic algorithms compared to classic (static) algorithms is that after a result is obtained, the table can be changed and a dynamic algorithm will update the result based just on those changes instead of processing the whole table again. As a result, they can be up to several orders of magnitude faster than classic (static) ones in some situations.
Desbordante support four data types:
- Tabular
- Graph
- Transactional
- Event sequences
以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗
