Use version-sorting for all sorting

[joshtriplett]

Add a description of a version-sorting algorithm. (This algorithm does
not precisely match strverscmp; it's intentionally simpler in its
handling of leading zeroes, and produces a result easier for humans to
easily understand and do by hand.)

Change all references to sorting to use version-sorting.

Change all references to "ASCIIbetically" to instead say "sort
non-lowercase before lowercase".

[rustbot]

r? @compiler-errors

(rustbot has picked a reviewer for you, use r? to override)

[rustbot]

Some changes occurred in src/doc/style-guide

cc @rust-lang/style

[RalfJung]

Is there a deep reason for this? I think I would have expected the one with more leading zeroes to come later, similar to how with usual lexicographic ordering the longer string (in that case with more trailing characters) comes later.

[joshtriplett]

Roughly, consistency with numeric sorting. Suppose you have 000, 001, 002, ..., 010, 011, ..., 100, 101. We sort those in that order, not for any reason involving the leading zeroes, but because of their numeric value. But the net effect is also that numbers with three leading zeroes come first, then two leading zeroes, then one leading zero, then no leading zeroes. So it seemed consistent, to me, to put "more leading zeroes" first in this case too.

[calebcartwright]

I think I follow (roughly), with a couple inline questions for clarity.

I don't have any strong opinions one way or another about the resultant sort order, but I do think it would be worth expanding on the set of examples (if not in the guide itself, then at least as for discussion with t-style members to help everyone visualize the differences with current state)

[calebcartwright]

Do I presume correctly that the usage of the term "strings" here does not bound the following prescriptions just to literal strings? I.e. we want the same algorithm to also apply in all other sorting contexts (e.g. idents in imports)

[joshtriplett]

Right, the intention of "strings" in "compare two strings for version-sorting" here is in the sense of it being a string to the tool parsing it, not that it's a literal string in the code being parsed.

Suggestions for clearer, more unambiguous wording welcome.

[compiler-errors]

@joshtriplett: Would you like to start an FCP on this, or do you want to have synchronous discussion on this? If you want sync discussion, please re-nominate this, otherwise just kick off the FCP 😺.

[compiler-errors]

cc rust-lang/style-team#143 as well

[joshtriplett]

@rfcbot merge

[rfcbot]

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @calebcartwright
• @compiler-errors
• @joshtriplett
• @yaahc

Concerns:

• discuss edge-cases raised by ijackson resolved by Use version-sorting for all sorting #115046 (comment)
• require all checkboxes resolved by Use version-sorting for all sorting #115046 (comment)

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[calebcartwright]

@rfcbot concern require all checkboxes

(we require all members to provide their check due to small team size)

[calebcartwright]

@rfcbot resolve require all checkboxes

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[calebcartwright]

For example consider "p00b" vs "p0a". Debian's algorithm does: "p" vs "p", same; "00" vs "0" number, same; "b" vs "g"

Should that last bit be "b" vs "a" or have I gotten confused somewhere 🤔

[ijackson]

Should that last bit be "b" vs "a" or have I gotten confused somewhere 🤔

Oops, yes, sorry. The keys are ... err ... like right next to each other? Edited to fix it, thanks.

[joshtriplett]

We discussed this at length in today's @rust-lang/style meeting.

For context on my original writeup of this algorithm, I have no attachment to being anywhere close to the strverscmp algorithm (and I do think strverscmp itself has some historical issues and bugs). I started out wanting to solve the problem of sorting x8 < x16 < x32 < x64 < x128 in that order, and similar, and knew version-sorting would get there, so I looked at strverscmp. Then I found its quirks (from the manpage) as well as historical bugs (notably, musl's version history has some notes on glibc implementation issues that musl reworked to be bug-compatible with), so I reworked this algorithm description to make more sense because we have no need to be bug-compatible with strverscmp.

What we discussed and agreed on in the style meeting was that we shouldn't be path-dependent and end up with "strverscmp with some tweaks/fixes". We should decide on the properties we want, come up with an algorithm that provides those properties, and not worry about how closely that aligns with strverscmp or any other version-sorting algorithm.

We're also mindful of the fact that this is mostly a sort for Rust identifiers (and possibly future Rust style needs like Cargo.toml but not yet), not a general sorting algorithm, and that sorting algorithms are context dependent. We want to prioritize making the mental model simple, but we're more flexible about the implementation being somewhat more complicated.

We do need to cover more corner cases in the document (it included some, but not all the ones raised here), by listing a series of identifiers and how they sort. (We don't think it's critical to document every difference between this and other version-sorting algorithms, but we can give a high-level difference like "This differs from other version-sorting algorithms, notably in the treatment of underscores and leading zeroes.".)

Properties we want to achieve, as agreed in the style meeting:

• We want to sort numeric values without regard for leading zeroes, so x0a < x00b < x0c.
• We have to provide a total order, so we have to have a well-defined order between any non-equal strings.
• If identifiers are identical all the way to the end, other than the number of leading zeroes in one of its numeric components (e.g. x005a09b and x5a009b), we should sort them based on the first such difference.
  • (We didn't try to figure out in the meeting whether we had consensus on which direction to sort in that case. Speaking for myself only, I have a mild preference for putting more leading zeroes first, but if there's a concrete use case that argues the other direction I'd prioritize the concrete use case.)
• We agreed that _ makes sense to special-case, since it acts like a space between words. (We noted that for a fully general sort algorithm, we'd need to specify things like how _ and sort relative to each other, but for characters allowed in Rust identifiers, we don't.)

I'm going to revise this proposal to support these properties.

[joshtriplett]

@ijackson wrote:

It is more valuable for the algorithm to be simple than for it to do something precisely optimal with mixed length digit strings. So IMO we should have the following order: "a0" "a00" "p0a" "p00a" "p0b" "p00b"

As far as I can tell, we're going to need an explicit check at the end of identical-other-than-leading-zeroes strings based on remembering the first leading-zero difference. We shouldn't just do "if they're identical other than leading zeroes, compare lexically", because AFAICT that would produce this order: x0 x00 x000 x001 x01 x1 x002 x02 x2. (NUL sorts lexically before 0, 0 sorts lexically before 1.)

I think we should do one of the following two orders, for consistency:

• x000 x00 x0 x001 x01 x1 x002 x02 x2
• x0 x00 x000 x1 x01 x001 x2 x02 x002

Of the two, I would favor the former over the latter, because other than the case of numeric substrings that are all zero but have a different number of zeroes (which seems far more unlikely), it's consistent with lexical ordering ('0' < '1'), so it's more likely to be what people expect.

[ijackson]

We want to sort numeric values without regard for leading zeroes, so x0a < x00b < x0c.

Is that because it's like hex? The problem with this is that it requires a fallback rescan, which makes the algorithm more difficult to explain. I don't think we actually care very much about the precise ordering of strings which differ only in number of leading zeroes in numerical components.

So I think we could have an algorithm which doesn't have a fallback rescan, and just has a rule which treats numerical pieces with the same numerical vlaue but different lengths, as unequal right away, rather than treating them as equal now and looking at them again later if the rest of the string is equal too.

But, the way it looks like hex got me thinking. Can we make it so that identifiers containing hex values come out in the right order? Identifiers with hex in aren't rare. Ideally we don't want an algorithm which guesses the base, so it falls out of the rest of the algorithm, but I'm not sure that's possible.

Ie for hex I think we want: x0 x12 xa12 xa1f and x000 x012 xa12 xa1f and 0x0 0x12 0xa12.

This is hard because xa12 vs xa1f looks like xa, 12 vs xa, 1, f. I think this shows that we can't do hex sensibly without base-guessing?

If we had a very simple base-guessing heuristic perhaps people woudl be able to structure their identifiers to trigger it. "Previous character was x" maybe.

[ijackson]

If we had a very simple base-guessing heuristic perhaps people woudl be able to structure their identifiers to trigger it. "Previous character was x" maybe.

hexagon < hexadecimal because x triggers "hex" so it's he, x!, 0xa, gon < he, x!, 0xadec, imal. lol. So err not that then. Maybe only _x? I think this wants some experimentation/brainstorming. Sorry to think of this at this late stage in the process...

[joshtriplett]

We want to sort numeric values without regard for leading zeroes, so x0a < x00b < x0c.

Is that because it's like hex?

No, this is nothing to do with hex; that the examples I gave started with an x was just the convention of picking x for an unknown variable. :)

I don't think we actually care very much about the precise ordering of strings which differ only in number of leading zeroes in numerical components.

The proposal I was making was "when comparing numeric components, compare by numeric value and ignore the number of leading zeroes". We don't have to do a second pass, though; if they're equal, we go on to the next group, but we remember if one or the other had less leading zeroes, and if we get to the end and the two strings happened to be equal in every group, we base the return value on that first difference.

Can we make it so that identifiers containing hex values come out in the right order? Identifiers with hex in aren't rare.

That's a reasonable thought. As you noted, I don't think there's any way to make hex sort numerically without attempting to detect hexadecimal. And I think we're not going to be able to do that reliably. My current inclination would be to not attempt to handle hexadecimal numbers at all.

[joshtriplett]

@yaahc I've addressed the concern; could you please review?

[yaahc]

@rfcbot resolve discuss edge-cases raised by ijackson

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[ijackson]

LGTM too now, thanks for the updates :-)

[compiler-errors]

@bors r+

[bors]

📌 Commit 2e931b5 has been approved by compiler-errors

It is now in the queue for this repository.

[compiler-errors]

@bors rollup