Fixes #31227: stop the fuzzy ranking stage widening recall on identifier lookups by harshach · Pull Request #31228 · open-metadata/OpenMetadata

harshach · 2026-08-08T17:09:45Z

Describe your changes:

I made the fuzzyName ranking stage bow out when it cannot actually be fuzzy, because ranking stages are combined under a should with minimum_should_match: 1 — so every stage widens recall, not just score — and getFuzziness() already disables fuzziness past 2 sub-tokens, leaving an OR multi_match at 70% token coverage that admits every document sharing 70% of the query's tokens (on an FQN, exactly its siblings under the same parent). That is why ColumnSearchIndexIT.testColumnFqnSearchIsPrecise saw 21 hits for a one-column FQN, with all 20 extras matching via ranking:fuzzyName alone, and why it only ever passed by winning a race against indexing. I also de-flaked RdfGlossaryGraphIT.glossaryIdFilterScopesGraphToRequestedGlossary, which awaited term nodes in the unscoped graph (capped at limit=500 and shared with the whole lane) and then did a single un-retried scoped fetch, even though scoping filters on the term → glossary membership edge that projects separately from the node.

Type of change:

Bug fix

High-level design:

N/A — small change (5 files).

The one design note worth flagging: this is a real relevance change, not just a test fix. The alternative was to weaken the assertion to match current behavior, which would have hidden a gap in a fix that landed a day earlier — #31106 claims "an FQN search now matches the one column precisely (count == results)", and that currently only holds on the aggregation path, not the ranked search path. The gate is deliberately narrow: the stage is dropped only when fuzziness is off and the query is a single whitespace term. Multi-word searches keep the stage (partial token coverage across terms is its purpose) and short searches keep real typo tolerance; two of the three new unit tests exist to pin exactly that.

Tests:

Use cases covered

Searching a column's full FQN in the Explore Columns tab returns that one column, not its table siblings — deterministically, regardless of what else is indexed
A multi-word Explore search (sample_data table) still gets the fuzzy stage's partial token coverage
A short misspelled search (custmer) still gets typo tolerance
The glossary graph scoped by glossaryId contains the requested glossary's terms and excludes another glossary's, under concurrent writes

Unit tests

Added unit tests for the changed logic.
Files updated: openmetadata-service/src/test/java/org/openmetadata/service/search/SearchSourceBuilderFactoryTest.java (+3 tests)
These build against the shipped searchSettings.json rather than the file's hand-built fixture, which mocks settings with no ranking configuration and so silently exercises the unranked legacy path — assertions on ranking stages against that fixture would have passed vacuously.
RED check: reverting isFuzzyStageUseful makes testFqnQueryDropsTheRecallWideningFuzzyStage fail, so the fix is load-bearing.
Coverage on changed classes (jacoco, from SearchSourceBuilderFactoryTest + SearchUtilsTest + SearchRankingHelperTest only — the full suite is higher): ElasticSearchSourceBuilderFactory 83.8%, OpenSearchSourceBuilderFactory 83.8%, search/SearchUtils 75.0%.

Backend integration tests

Not applicable — no API change. Two existing ITs are the subject of the fix: ColumnSearchIndexIT (unchanged; the product fix is what makes it deterministic) and RdfGlossaryGraphIT (rewritten to await the scoped graphs).

Ingestion integration tests

Not applicable (no ingestion changes).

Playwright (UI) tests

Not applicable (no UI changes).

Manual testing performed

mvn -pl openmetadata-spec,openmetadata-service jacoco:prepare-agent test jacoco:report -Dtest='SearchSourceBuilderFactoryTest,SearchUtilsTest,SearchRankingHelperTest' → 180 tests, 0 failures
Ran all 25 org.openmetadata.service.search.* unit test classes → all green
Temporarily reverted isFuzzyStageUseful and re-ran → testFqnQueryDropsTheRecallWideningFuzzyStage fails as expected; restored
mvn -pl openmetadata-integration-tests test-compile → BUILD SUCCESS
mvn -pl openmetadata-service,openmetadata-integration-tests spotless:check → clean

Not run locally: the two ITs themselves. They need testcontainers (MySQL + Elasticsearch) and Docker was unavailable in my environment. The column fix follows directly from the captured CI response — the 20 unwanted hits matched via ranking:fuzzyName alone, so dropping that stage leaves exactly the target — but that is inference from the failure payload, not a local green run. Worth watching this PR's own IT lanes.

UI screen recording / screenshots:

Not applicable.

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes <issue-number>: <short explanation>
My PR is linked to a GitHub issue via Fixes #<issue-number> above.
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: not applicable, no schema changes.
For UI changes: not applicable.
I have added tests (unit / integration / Playwright as applicable) and listed them above.

I have added a test that covers the exact scenario we are fixing (testFqnQueryDropsTheRecallWideningFuzzyStage, verified to fail without the fix; issue FQN column search returns sibling columns; ColumnSearchIndexIT and RdfGlossaryGraphIT flaky in CI #31227 referenced in the test comment).

🤖 Generated with Claude Code

…ntifier lookups Ranking stages are combined under a `should` with minimum_should_match:1, so every stage widens recall, not just the score. getFuzziness() already turns fuzziness off past 2 sub-tokens, which leaves the fuzzy stage as an OR multi_match at 70% token coverage: it can no longer correct a typo, it only admits every document sharing 70% of the query's tokens. On a single-term identifier such as a fully-qualified name those are exactly its siblings under the same parent. ColumnSearchIndexIT.testColumnFqnSearchIsPrecise (added by #31106) asserts an FQN search returns the one column. In CI it returned 21 -- every column of every table the sibling test methods had created, plus the two other columns of its own table -- and all 20 extra hits matched via `ranking:fuzzyName` alone. The test only passed by winning a race against indexing, so it was flaky from birth rather than wrong. Skip the fuzzy stage when fuzziness is disabled and the query is a single whitespace term, letting exact/phrase/tokenCoverage decide recall. Multi-word searches and short typo-tolerant searches are untouched; two of the three new unit tests pin that gate. Also de-flake RdfGlossaryGraphIT.glossaryIdFilterScopesGraphToRequestedGlossary: it awaited term nodes in the unscoped graph, then did a single un-retried scoped fetch. Scoping filters on the term -> glossary membership edge, which projects separately from the term node, and the unscoped graph is capped at limit=500 and shared with every other test writing glossary terms in the lane. Await the scoped graphs instead, glossaryB first so the exclusion assertions mean "filtered out" not "not written yet". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-08-08T17:09:56Z

✅ PR checks passed

The linked issue has a description and all required Shipping project fields set. Thanks!

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR tightens ranked search behavior for identifier/FQN lookups by preventing the fuzzyName ranking stage from widening recall when fuzziness is effectively disabled, improving determinism/precision for exact identifier searches. It also de-flakes an RDF glossary graph integration test by awaiting the glossary-scoped projections that the assertions actually depend on.

Changes:

Add SearchUtils.isFuzzyStageUseful(query) and use it in both ES/OS ranked query builders to skip the fuzzy ranking stage when it cannot provide typo tolerance (single-term + fuzziness disabled).
Add unit tests that validate the fuzzy stage is dropped for single-term FQNs but retained for multi-term searches and short typo-tolerant queries, using shipped searchSettings.json.
Rework RdfGlossaryGraphIT.glossaryIdFilterScopesGraphToRequestedGlossary to await scoped graphs and make exclusion assertions deterministic under concurrency.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
openmetadata-service/src/test/java/org/openmetadata/service/search/SearchSourceBuilderFactoryTest.java	Adds tests to ensure ranked queries drop/keep the fuzzy stage appropriately; loads shipped search settings for realistic stage coverage.
openmetadata-service/src/main/java/org/openmetadata/service/search/SearchUtils.java	Introduces `isFuzzyStageUseful` and constants to centralize fuzziness decisions and gate fuzzy-stage inclusion.
openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OpenSearchSourceBuilderFactory.java	Uses `isFuzzyStageUseful` to skip building the fuzzy ranking stage when it would only widen recall.
openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchSourceBuilderFactory.java	Mirrors the OpenSearch behavior for Elasticsearch ranked query generation.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/RdfGlossaryGraphIT.java	Deflakes glossary scoping test by polling scoped projections (edge + node) rather than relying on unscoped graph presence.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

harshach · 2026-08-08T18:43:13Z

+  private static SearchSettings shippedSearchSettings() throws IOException {
+    List<String> jsonDataFiles =
+        EntityUtil.getJsonDataResources(".*json/data/settings/searchSettings.json$");
+    String json =
+        CommonUtil.getResourceAsStream(
+            EntityRepository.class.getClassLoader(), jsonDataFiles.getFirst());
+    return JsonUtils.readValue(json, SearchSettings.class);
+  }
+
+  private static String rankedOpenSearchQuery(String query) throws IOException {
+    OpenSearchSourceBuilderFactory factory =
+        new OpenSearchSourceBuilderFactory(shippedSearchSettings());
+    return serializeOpenSearchRequest(
+        factory.getSearchSourceBuilderV2(Entity.TABLE_COLUMN, query, 0, 15));
+  }
+
+  private static String rankedElasticSearchQuery(String query) throws IOException {
+    ElasticSearchSourceBuilderFactory factory =
+        new ElasticSearchSourceBuilderFactory(shippedSearchSettings());
+    return factory.getSearchSourceBuilderV2(Entity.TABLE_COLUMN, query, 0, 15).query().toString();
+  }


Good catch — fixed in 1c26514.

The shipped searchSettings.json now loads once into a static in @BeforeAll, and each test builds its OpenSearch/Elasticsearch query once into a local instead of re-invoking the helper per assertion. That takes it from 8 classpath scans across the three tests down to 1.

Two side benefits: the assertions carry the serialized query as the failure message again, and the tests no longer need throws IOException.

Re-verified after the refactor — 180 tests green, and the RED check still holds (reverting isFuzzyStageUseful fails testFqnQueryDropsTheRecallWideningFuzzyStage), so the caching didn't neuter the assertion.

Sharing one SearchSettings instance across tests is safe here: both source-builder factories only read it.

github-actions · 2026-08-08T18:03:11Z

✅ Playwright Results — workflow succeeded

Validated commit 88ffec15a62be8ee525394c45418d193f363920a in Playwright run 31298193256, attempt 1.

✅ 1025 passed · ❌ 0 failed · 🟡 1 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 52m 23s

⏱️ Max setup 3m 3s · max shard execution 19m 12s · max shard-job elapsed before upload 22m 53s · reporting 5s

🌐 186.53 requests/attempt · 2.21 app boots/UI scenario · 13.28% common-shard skew

Optimization targets still in progress:

Application boot ratio was 2.21 per UI scenario (2309 boots / 1043 scenarios; convergence target: at most 1).

Shard	Passed	Flaky
✅ Shard chromium-01	158	0
✅ Shard chromium-02	161	0
🟡 Shard chromium-03	169	1
✅ Shard chromium-04	179	0
✅ Shard chromium-05	172	0
✅ Shard data-asset-rules-01	61	0
✅ Shard domain-isolation-01	14	0
✅ Shard global-state-01	34	0
✅ Shard import-export-01	7	0
✅ Shard ingestion-01	2	0
✅ Shard reindex-01	28	0
✅ Shard search-01	11	0
✅ Shard search-rbac-01	29	0

🟡 1 flaky test(s) (passed on retry)

Pages/ExplorePageRightPanel.spec.ts › Should allow Data Consumer to view all tabs for searchIndex (shard chromium-03, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

Each ranked-query helper call resolved it via a full classpath scan (EntityUtil.getJsonDataResources), so the three tests triggered eight scans. Load it once in @BeforeAll and build each query once per test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

gitar-bot · 2026-08-09T06:08:05Z

Code Review ✅ Approved

Stops the fuzzy ranking stage from widening recall on exact identifier lookups to prevent unwanted sibling matches. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (1)

openmetadata-service/src/test/java/org/openmetadata/service/search/SearchSourceBuilderFactoryTest.java:441

The test comment says this case keeps a non-zero fuzziness (real typo tolerance), but the assertions only check that the ranking:fuzzyName stage is present. This test would still pass if the fuzzy stage were emitted with fuzziness disabled (or not emitted with any fuzziness parameter at all), so it isn’t actually pinning the behavior it describes.

    String osQuery = rankedOpenSearchQuery("custmer");
    String esQuery = rankedElasticSearchQuery("custmer");

    assertTrue(osQuery.contains(FUZZY_STAGE_QUERY_NAME), osQuery);
    assertTrue(esQuery.contains(FUZZY_STAGE_QUERY_NAME), esQuery);

harshach requested a review from a team as a code owner August 8, 2026 17:09

Copilot AI lite review requested due to automatic review settings August 8, 2026 17:09

github-actions Bot added backend safe to test Add this label to run secure Github workflows on PRs labels Aug 8, 2026

test: reference issue #31227 in the fuzzy-stage regression test

a624698

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot started reviewing on behalf of harshach August 8, 2026 17:10 View session

Copilot AI reviewed Aug 8, 2026

View reviewed changes

harshach temporarily deployed to test August 8, 2026 17:37 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings August 8, 2026 18:43

mohityadav766 approved these changes Aug 8, 2026

View reviewed changes

Copilot started reviewing on behalf of harshach August 8, 2026 18:43 View session

Copilot AI reviewed Aug 8, 2026

View reviewed changes

harshach temporarily deployed to test August 8, 2026 18:54 — with GitHub Actions Inactive

Merge branch 'main' into harshach/fix-failing-ci-test

88ffec1

Copilot AI review requested due to automatic review settings August 9, 2026 06:07

Copilot started reviewing on behalf of mohityadav766 August 9, 2026 06:08 View session

Copilot AI reviewed Aug 9, 2026

View reviewed changes

mohityadav766 temporarily deployed to test August 9, 2026 06:35 — with GitHub Actions Inactive

Navigation Menu

Uh oh!

Fixes #31227: stop the fuzzy ranking stage widening recall on identifier lookups - #31228

Fixes #31227: stop the fuzzy ranking stage widening recall on identifier lookups#31228
harshach wants to merge 4 commits into
mainfrom
harshach/fix-failing-ci-test

harshach commented Aug 8, 2026

Uh oh!

github-actions Bot commented Aug 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

harshach Aug 8, 2026

Uh oh!

github-actions Bot commented Aug 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

gitar-bot Bot commented Aug 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

harshach commented Aug 8, 2026

Describe your changes:

Type of change:

High-level design:

Tests:

Use cases covered

Unit tests

Backend integration tests

Ingestion integration tests

Playwright (UI) tests

Manual testing performed

UI screen recording / screenshots:

Checklist:

Uh oh!

github-actions Bot commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ PR checks passed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

harshach Aug 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Playwright Results — workflow succeeded

Performance

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

gitar-bot Bot commented Aug 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Aug 8, 2026 •

edited

Loading

github-actions Bot commented Aug 8, 2026 •

edited

Loading