Metadata properties hidden from annotators can still be used as search filters and sorts, leaking their values through result membership

[AUTHENSOR]

Metadata properties hidden from annotators can still be used as search filters and sorts, leaking their values through result membership

Affected: 5338519 (argilla-server)

Summary

A metadata property configured with visible_for_annotators: false is
withheld from annotators when record metadata values are returned: the search
handlers call _filter_record_metadata_for_user, which keeps only properties
the actor may read under RecordPolicy.get_metadata
(api/policies/v1/record_policy.py:78, backed by allowed_roles).

However, the search request validator never checks that flag.
SearchRecordsQueryValidator._validate_metadata_filter_scope
(src/argilla_server/contexts/search.py:72-79) only verifies the property
exists in the dataset. The search endpoints authorized for plain workspace
members (DatasetPolicy.search_records,
POST /api/v1/me/datasets/{dataset_id}/records/search) therefore accept a
terms filter or a sort on a hidden metadata property, and the filter is
translated into an Elasticsearch/OpenSearch terms query on metadata.<name>
(search_engine/commons.py:540-556).

An annotator can submit one search per candidate value and observe which
records come back. The hidden value itself is stripped from the response, but
record membership in the result set reveals it exactly: a filter
metadata.hidden_gold in ["A"] that returns record R proves R's hidden value
is A. A sort on the hidden property adds an ordering oracle. Both work with
the annotator's own API key.

Impact

Metadata visibility is commonly used to keep gold answers, quality flags, or
payment/review metadata out of annotator view while still measuring them. With
this gap, any workspace member can fully recover those hidden values, record
by record, and align their submissions to hidden gold answers. The protection
is enforced on the projection (returned values) but not on the query plane
(filters and sorts), so the confidentiality of the feature does not hold
against the very role it targets.

Reproduction

The attached poc/poc_f2.py runs the real code at the commit above with no
Elasticsearch server required:

1. It builds an in-memory sqlite database with the real models: a workspace,
   an owner, an admin, an annotator (workspace member), a dataset, a text
   field, a metadata property hidden_gold with
   allowed_roles=[admin, owner] (so visible_for_annotators is false), a
   visible property lang, and two records with hidden_gold values A and B.
2. It calls the real validate_search_records_query with a terms filter on
   hidden_gold: no error is raised (existence is the only check).
3. It calls the real _to_search_engine_filter and the real
   BaseElasticAndOpenSearchEngine._scope_to_elasticsearch_field /
   _map_filter_to_es_filter for the annotator, producing
   {"terms": {"metadata.hidden_gold": ["A"]}}.
4. It calls the real handler search_current_user_dataset_records directly
   with a stub search engine that applies the produced ES query to documents
   built by the real _map_record_metadata_to_es. Filter value A returns only
   record r1, filter value B returns only record r2, and the returned records
   contain only lang in their metadata (the value is stripped, the filter
   still applies).
5. Contrast arm: is_authorized(annotator, RecordPolicy.get_metadata(r1, "hidden_gold")) is false, proving the read policy would deny the value
   itself.

Expected runtime is a few seconds; see poc/run_all.sh for the interpreter,
dependency install, and double-run commands. Evidence files are written to
poc/out/f2/.

A manual API-level check on a deployed instance:

1. Create a dataset with a terms metadata property, setting
   visibleForAnnotators to false in the settings request.
2. Log records with that metadata set to distinct values, as admin.
3. As a user with the annotator role in that workspace, POST to
   /api/v1/me/datasets/{dataset_id}/records/search with body
   {"filters":{"and":[{"type":"terms","scope":{"entity":"metadata","metadata_property":"<hidden>"},"values":["A"]}]}}.
4. The response returns exactly the records whose hidden value is A, while the
   records' metadata field omits the property.

Expected behavior

Filters and sorts whose scope is a metadata property that the requesting user
is not allowed to read should be rejected (422) or ignored, mirroring what
_filter_record_metadata_for_user already enforces for returned values.

Actual behavior

The validator only checks existence, the ES terms query is built on the hidden
property for the annotator, and the result set membership discloses the hidden
value per record.

Suggested fix

In SearchRecordsQueryValidator (or in the handlers right after authorize,
where the actor is known), resolve each MetadataFilterScope to its
MetadataProperty and require MetadataPropertyPolicy.get (or
visible_for_annotators for annotator actors) for every filter and sort
scope, not just for record output. The metadata properties are already loaded
by the search handlers via selectinload(Dataset.metadata_properties), so the
check is cheap. Consider also excluding hidden metadata from any
metrics/distribution aggregation responses reachable by annotators.