PEP 825: assorted updates - #5086
Open
mgorny wants to merge 26 commits into
Open
Conversation
Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com> Co-authored-by: konsti-openai <konsti@openai.com>
We do not really have a good use case for these, I have been leaning backwards to even provide example, the justification is limited to "completeness" and they're just adding complexity. I think it's better to remove them before we put more effort into adding consistency requirements. Signed-off-by: Michał Górny <mgorny@quansight.com>
Expand the schema versioning to strictly follow semantic versioning. That is, major versions represent breaking changes and require tools to reject incompatible versions, whereas minor versions represent backwards compatible changes and permit tools to accept them. Signed-off-by: Michał Górny <mgorny@quansight.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
Use the current Discourse thread slug in the header URLs; the old pep-817 slug still redirects, but no longer names the PEP.
- The removed key was `default-priorities.property`, not `default-priorities.value`; the latter never existed. The error came from the publishing section, which had the same mistake. - List the four remaining changes made in this round: the "Implementation requirements" section, the per-key metadata scopes, the per-key consistency requirements, and the new rejected idea. - Move the 11-May-2026 entry into place; the list is otherwise in reverse chronological order.
The prose introduced the subtable as `[packages.variants-json]`, while the proposed specification text and the example used `[packages.variant_json]`. Use the spelling with a dash throughout, that is consistent with PEP 751.
Four of the schema's constraints disagreed with the normative text:
- The variant label pattern still carried a 16-character cap
(`^[a-z0-9_.]{1,16}$`), left over from before the label length limit
was removed on 17-Feb-2026. The specification requires only
`^[0-9a-z_.]+$`. The cap rejected this PEP's own example label,
`x86_64_v3_openblas` (18 characters).
- The namespace and feature name patterns permitted `.`, which the
specification does not: both MUST match `^[a-z0-9_]+$`. The feature
*value* pattern correctly keeps `.` and is unchanged.
- The schema required `default-priorities.namespace` to be non-empty
(`minItems: 1`), but the prose only required the list to contain all
namespaces used in variant properties. State the requirement in the
text as well: a package version providing variant wheels MUST use at
least one variant namespace.
- The subschema for a namespace's value declared no type, so an array
there satisfied the schema; `patternProperties` and
`additionalProperties` only constrain objects. The specification
requires a mapping of feature names, so declare `"type": "object"`.
Also drop the `uniqueItems` keywords whose instance is an object;
`uniqueItems` only constrains arrays, so those never applied. The two
on arrays are kept.
These are bugs in the schema relative to the prose, so it is corrected
in place rather than versioned: 0.x is reserved for drafts and the PEP
states that tools must not make compatibility assumptions across them.
The abstract was largely unchanged from PEP 817, so update it to make it clearer what the scope of this PEP is. Avoid introducing unnecessary vocabulary while at it. Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
Decouple the variant ordering algorithm from index, as requested on DPO. While most of the time it will be done against wheels from an index, the algorithm itself is generic, so replace the references to "index-level metadata" with more generic "combined variant metadata", indicating that it can either be obtained from an index or from wheels. Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
The additions to Specification are as discussed on DPO. The Rationale additions aim to answer all the concerns and misunderstandings we've seen in discussions of the design of variant environment markers. Signed-off-by: Michał Górny <mgorny@quansight.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
Two changes to "Removing ordering information from wheel files": - Avoid a link to a specific post (this was my suggestion, changed my mind - apologies) - Reduce the length of that section; acknowledge there is a risk. We will address this by the overall argument for metadata consistency that is still to come.
Paul asked for this new section, and it's actually helpful to have a section titled "Metadata consistency" I think. It shows there aren't that many requirements, and explains it's auto-fulfilled when building from a single source. Also one small fix in the "Index-level metadata" section. Co-authored-by: Michał Górny <mgorny@gentoo.org>
Follow-up to PR 72, which describes the index-level metadata file as an optimization. Several sections still assumed it was the authoritative source. Variant ordering no longer requires the index-level file. The combined metadata may be sourced from it or from the wheels; both are required to agree, and tools SHOULD prefer the file where available because it is cheaper. This also removes the need to say what "installing from an index" means, which was never defined and which the Rationale straddles by counting a webserver directory listing as an index. The suggested implementation logic gave two answers for a missing file: step 5 treated the variant wheels as incompatible, while the paragraph below it said to read the metadata from the wheels. Keep the latter, and make it clear that the cost is not as high as one might think at first (because previous filtering). Also make declining an option. The section is non-normative, and a tool that finds the remaining cost unacceptable may treat the variant wheels as incompatible instead. The case the Rejected Ideas argument rests on is installing from a local directory, where the wheels are already present and the fallback costs nothing. Publishing no longer restates the merge rules (the description had drifted). The multi-source section justified leaving behavior undefined on the grounds that the metadata is scoped at index level - and we decoupled that. Finally, split the response to invalid data by the role of the tool. Requiring consumers to refuse it meant an installer skipping a version outright, so one malformed wheel could act as a de facto yank for every user. Indexes should reject at upload; installers should degrade.
- The ``variants`` dictionary of an individual wheel is now required to contain exactly one entry rather than merely to contain the wheel's own label. Both the JSON example and the installation example already described it that way, so this settles the two in favour of the narrower reading. - Soften the claim that variant metadata "follows the same compatibility rules as those for Core Metadata". It does not: the versioning rules stated a few paragraphs earlier are semantic versioning, which is not Core Metadata's scheme. Say the versioning is similar in spirit.
Three sections said different things about what a tool owes when the variant metadata is not consistent. Settle on one position: tools are not required to verify conformance; where they have established that the data does not conform, the response in "Implementation requirements" applies, split by role as before.
The pylock.toml section said nothing about markers, leaving a reader to work out whether a marker scoped to a selected wheel can be expressed in a lock file at all (discussed on DPO). The two places that have no such context follow from the existing rule that variant markers may only appear in dependency specifiers, so they are noted as a consequence. Also tighten the allowance to drop metadata from the inlined table - it was a bit too vague before.
Filtering the marker values against what the target system supports assumes the selected wheel is one the system supports. Selection overrides could break that assumption: a tool offering "install this variant regardless of my hardware" would select a wheel whose properties are unsupported, the filter would reduce `variant_properties` to the empty set, and every dependency gated on those properties would disappear. The install would then be missing exactly the dependencies the chosen wheel needs, silently. Note that this is not a matter of ordering the pipeline differently. Filtering is an intersection with what the system supports, so it comes out empty whenever the override runs. Constrain the overrides instead: they act on the wheels already found compatible, and may reorder or narrow that set but not reach past it. Installing for a different system than the one being installed to is a question of what counts as supported, which is already deferred to a later PEP, so point there for that case. Also correct step 7. of the installation example, which described the markers as evaluated against the properties stored in the `variants` dictionary, without the filtering that the marker section requires.
The Rationale said only that without narrowing a wheel "would pull in the dependencies of every value a variant feature lists", which states the mechanism without giving a reader anything to picture, and the installation example illustrated a case where narrowing changes nothing observable. Between them the step looked like complexity with no purpose. Address that by: - Explicitly naming the main use case it exists for. - Note also when the alternative (the dependency also publishing variants) applies. There may be other reasons, like "my dependency has a bug for specific hardware". Those reasons also occur for other environment markers. However, that would make the text even longer - one example should suffice. In the end, the ability to treat GPU hardware like CPU families (selection-wise) is what matters.
Signed-off-by: Michał Górny <mgorny@quansight.com>
Documentation build overview
3 files changed± pep-0764/index.html± pep-0825/index.html± pep-0825/appendix-variant-json-schema/index.html |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PEP 123: Summary of changes)A large batch of updates based on the DPO feedback and multiple passes of rereading the PEP. This includes both clarification and actual specification changes. To quote from the change history:
default-priorities.featureanddefault-priorities.property.variant_propertiesmarker use variant properties compatible with the system rather than all the properties specified in the metadata.The changes are explained in greater detail in the commit messages.