Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BracketedSort a new, faster algorithm for partialsort and friends #52006

Merged
merged 41 commits into from
Nov 23, 2023

Conversation

LilithHafner
Copy link
Member

@LilithHafner LilithHafner commented Nov 2, 2023

See the docstring of BracketedSort for an explanation of this algorithm.

The diff is kinda big but only 40% of it is code, the rest are comments, docstrings, and tests.

I produced it without reference to any sources that describe such an algorithm but I do not claim original authorship—somebody has certainly come up with this before.

Here's an earlier version: JuliaStats/Statistics.jl#154

Robust benchmarks are available further down in the thread. Here's an appetizer

Some benchmarks vs master

julia> @btime partialsort!(x, 1:100) setup=(x=rand(Int, 100_000_000)) evals=1 samples=100 seconds=20;
  173.980 ms (3 allocations: 762.94 MiB)=>  65.699 ms (5 allocations: 4.49 MiB)

julia> @btime partialsort(x, 1:100) setup=(x=rand(Int, 100_000_000)) evals=1 samples=100 seconds=20;
  239.999 ms (6 allocations: 1.49 GiB)  => 132.764 ms (8 allocations: 767.43 MiB)

julia> @btime partialsortperm(x, 1:100) setup=(x=rand(Int, 100_000_000)) evals=1 samples=100 seconds=20;
  327.099 ms (8 allocations: 1.49 GiB)  => 321.387 ms (8 allocations: 1.49 GiB)

julia> @btime partialsort!(copy(x), 1:100) setup=(x=rand(10_000)) evals=1000 samples=1000 seconds=20;
  24.198 μs (6 allocations: 156.38 KiB) => 18.521 μs (6 allocations: 82.06 KiB)

julia> @btime partialsort(x, 1:100) setup=(x=rand(10_000)) evals=1000 samples=1000 seconds=20;
  18.803 μs (6 allocations: 156.38 KiB) => 18.105 μs (6 allocations: 82.06 KiB)

julia> @btime partialsortperm(x, 1:100) setup=(x=rand(10_000)) evals=1000 samples=1000 seconds=20;
  24.164 μs (8 allocations: 156.42 KiB) => 24.302 μs (8 allocations: 156.42 KiB)

A detailed view of a special case using @benchmark and including 1.9.3

julia> @benchmark partialsort!(x, 1:100) setup=(x=rand(Int, 100_000_000)) evals=1 samples=100 seconds=20

# PR
BenchmarkTools.Trial: 100 samples with 1 evaluation.
 Range (min … max):  68.893 ms … 111.625 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     69.499 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   70.898 ms ±   6.030 ms  ┊ GC (mean ± σ):  0.00% ± 0.00%

  █▆▂                                                           
  ████▇▄▁▁▄▁▁▁▁▁▁▁▁▁▁▁▄▁▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄ ▄
  68.9 ms       Histogram: log(frequency) by time       103 ms <

 Memory estimate: 4.49 MiB, allocs estimate: 5.

# Master
BenchmarkTools.Trial: 49 samples with 1 evaluation.
 Range (min … max):  163.071 ms …    1.067 s  ┊ GC (min … max): 0.23% … 1.17%
 Time  (median):     216.896 ms               ┊ GC (median):    0.17%
 Time  (mean ± σ):   305.018 ms ± 204.787 ms  ┊ GC (mean ± σ):  0.20% ± 0.15%

  ▃▆█▆                                                           
  ████▇▇▇▄▁▆▁▄▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▄▄▁▁▁▁▄▄▁▁▁▁▄▆▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄ ▁
  163 ms           Histogram: frequency by time          1.07 s <

 Memory estimate: 762.94 MiB, allocs estimate: 3.

# 1.9.3
BenchmarkTools.Trial: 54 samples with 1 evaluation.
 Range (min … max):  161.129 ms … 894.473 ms  ┊ GC (min … max): 0.13% … 0.02%
 Time  (median):     213.916 ms               ┊ GC (median):    0.09%
 Time  (mean ± σ):   265.121 ms ± 142.888 ms  ┊ GC (mean ± σ):  0.08% ± 0.02%

     █▃▆                                                         
  ▇▄████▇▅▅▅▄▅▄▄▁▁▄▄▁▁▁▁▄▁▁▁▁▁▁▁▄▁▁▁▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▄ ▁
  161 ms           Histogram: frequency by time          742 ms <

 Memory estimate: 762.94 MiB, allocs estimate: 2.

I'm not touching the *perm varients because

  • The cache locality makes me go "ewww!" and I don't want to touch them until I (or someone else) cleans that up
  • When I tried I got wonky performance results including regressions that I don't want to investigate
  • I want to pick good thresholds for partialsortperm! and I will only be able to do that once the overall approach is improved.

This makes sortperm unstable once again, with no option to make it stable. It was unstable in 1.9 with no option to make it stable, stable in 1.10 with no option to make it unstable, and has never been documented one way or the other.

Lilith Hafner added 16 commits October 28, 2023 13:55
This commit speeds up my_partialsort! to be faster than it was before, but partialsort! is slower than any version of my_partialsort!

julia> using Random; Random.seed!(1); x = rand(1000); hash(x)
0x13843e5a90ce1e51

julia> @Btime partialsort!(copy($x), 500);
  3.838 μs (3 allocations: 7.88 KiB)

julia> @Btime my_partialsort!(copy($x), 500);
  1.946 μs (3 allocations: 7.88 KiB)
…ble IEEE opt. because it makes the kernel slower)
julia> using Random; Random.seed!(1); x = rand(1000); hash(x)
0x13843e5a90ce1e51

julia> @Btime partialsort!(copy($x), 500);
  2.671 μs (3 allocations: 7.88 KiB)

julia> @Btime my_partialsort!(copy($x), 500);
  2.000 μs (3 allocations: 7.88 KiB)

julia> @Btime partialsort!(copy($x), 500, lt=<);
  1.733 μs (3 allocations: 7.88 KiB)
@LilithHafner LilithHafner added performance Must go faster sorting Put things in order labels Nov 2, 2023
@LilithHafner
Copy link
Member Author

Here's a much more comprehensive benchmark
using Random, Printf
get_length() = round(Int, exp(17*rand()))
get_eltype() = rand([Float64, Float32, Int, UInt8, Int128, String])
get_function() = rand([partialsort!, partialsort])
get_element(x) = rand(x)
get_element(::Type{String}) = randstring()
get_k(n) = rand([
        1,
        n,
        rand(1:n),
        rand(1:1+n÷100),
        rand(n-n÷100:n),

        iseven(n) ? (n÷2:n÷2+1) : (n+1)÷2,

        UnitRange(sort((rand(1:n), rand(1:n)))...),
        1:rand(1:n),
        1:rand(1:1+n÷100),
        rand(n-n÷100:n):n,
        (1:rand(1:1+n÷100)) .+ rand(0:(n-n÷100-1)),
    ])

new!(v, k) = partialsort!(v, k)
new(v, k) = partialsort(v, k)
function old!(v, k)
    Base.Sort._sort!(v, Base.Sort.InitialOptimizations(Base.Sort.ScratchQuickSort(k)), Base.Order.Forward, (;))
    Base.Sort.maybeview(v, k)
end
old(v::AbstractVector, k::Union{Integer,OrdinalRange}; kws...) =
    old!(Base.copymutable(v), k; kws...)

function test(length::Int, eltype, func)
    func === partialsort && (length ÷= 2) # benchmarking is inefficient
    data = [get_element(eltype) for _ in 1:length]
    ks = [get_k(length) for _ in 1:5]
    sort!(ks, by = x->(x isa Number, x))
    unique!(ks)
    println("\n$func(rand($eltype, $length), k)")
    for k in ks
        (t_old, t_new) = func === partialsort! ? test!(similar(data), data, k) : test(data, k)
        a = 1e9t_old/length
        b = 1e9t_new/length
        c = log(b/a)*100
        @printf "k = %-18s %6.3f %6.3f  %+05.1f%%\n" k a b c
    end
end

estimate_runtime(data::Vector{T}, k) where T =
    round(Int, (10 + length(data) + length(k) * log(length(k))) * (T === String ? 3 : T == Int128 ? 2 : 1))
function test(data::Vector{T}, k::K) where {T,K}
    estimated_runtime = estimate_runtime(data, k)
    if estimated_runtime > 10^7
        t_old = @elapsed old_res = old(data, k)
        t_new = @elapsed new_res = new(data, k)
        old_res == new_res || error("Bad sort")
        t_old, t_new
    elseif estimated_runtime > 10^5
        truth = old(data, k)
        t_old, t_new = Inf, Inf
        for _ in 1:2*10^7 ÷ estimated_runtime
            t1 = @elapsed res1 = old(data, k)
            res1 == truth || error("Bad old sort")
            t_old = min(t_old, t1)

            t2 = @elapsed res2 = new(data, k)
            res2 == truth || error("Bad sort")
            t_new = min(t_new, t2)
        end
        t_old, t_new
    else
        old(data, k) == new(data, k) || error("Bad sort")
        evals = max(1, 10^4 ÷ estimated_runtime)
        samples = 10^7 ÷ evals ÷ estimated_runtime
        t_old = @belapsed old($data, $k) evals=evals samples=samples gctrial=false
        t_new = @belapsed new($data, $k) evals=evals samples=samples gctrial=false
        t_old, t_new
    end
end

function test!(d::Vector{T}, data::Vector{T}, k::K) where {T,K}
    estimated_runtime = estimate_runtime(data, k)
    if estimated_runtime > 10^7
        copyto!(d, data)
        t_old = @elapsed old_res = old!(d, k)
        copyto!(d, data)
        t_new = @elapsed new_res = new!(d, k)
        old_res == new_res || error("Bad sort")
    elseif estimated_runtime > 10^6
        truth = old(data, k)
        t_old, t_new = Inf, Inf
        for _ in 1:2*10^7 ÷ estimated_runtime
            copyto!(d, data)
            t1 = @elapsed res1 = old!(d, k)
            res1 == truth || error("Bad old sort")
            t_old = min(t_old, t1)

            copyto!(d, data)
            t2 = @elapsed res2 = new!(d, k)
            res2 == truth || error("Bad sort")
            t_new = min(t_new, t2)
        end
    else
        old!(copyto!(d, data), k) == new!(copyto!(d, data), k) || error("Bad sort")
        evals = max(1, 10^4 ÷ estimated_runtime)
        samples = 10^7 ÷ evals ÷ estimated_runtime
        if evals == 1
            t_old = @belapsed old($d, $k) setup=(copyto!($d, $data)) evals=evals samples=samples gctrial=false
            t_new = @belapsed new($d, $k) setup=(copyto!($d, $data)) evals=evals samples=samples gctrial=false
        else
            t_old = @belapsed old(copyto!($d, $data), $k) evals=evals samples=samples gctrial=false
            t_new = @belapsed new(copyto!($d, $data), $k) evals=evals samples=samples gctrial=false
        end
    end
    t_old, t_new
end

function test(n, seed=1729)
    Random.seed!(seed)
    samples = [(get_length(), get_eltype(), get_function()) for _ in 1:n]
    sort!(samples, by = x->(x[3] === partialsort, string(x[2]), x[1]))
    try
        for s in samples
            test(s...)
        end
    catch x
        x isa InterruptException || rethrow()
    end
    nothing
end

test(100)
And the results

The columns are k | old runtime | new runtime | change. Runtime is measured in nanoseconds per element. Changes are measured in logarithmic percent (a reported 1% change is a 1.01x or 0.99x, but a +100% is 2.71x and -100% is 0.37x) this makes positive and negative symmetric. rand(String, 10) is an abbreviation for [randstring() for _ in 1:10].

partialsort!(rand(Float32, 3), k)
k = 1:1                11.595 17.104  +38.9%
k = 2:3                13.092 18.849  +36.5%
k = 2                  12.010 16.833  +33.8%
k = 3                  11.523 15.749  +31.2%

partialsort!(rand(Float32, 120), k)
k = 1:1                 2.207  2.632  +17.6%
k = 1:68                6.718  8.666  +25.5%
k = 120:120             3.148  4.162  +27.9%

partialsort!(rand(Float32, 356), k)
k = 1:1                 2.176  2.575  +16.8%
k = 178:179             2.822  3.377  +18.0%
k = 222:225             2.926  3.565  +19.8%
k = 1                   2.480  2.072  -18.0%
k = 356                 2.493  2.107  -16.8%

partialsort!(rand(Float32, 764), k)
k = 1:612               7.090  9.490  +29.2%
k = 69:736              7.526 10.034  +28.8%
k = 3                   2.227  1.636  -30.8%

partialsort!(rand(Float32, 1341), k)
k = 1:7                 3.773  2.250  -51.7%
k = 414:418             3.924  3.058  -24.9%
k = 683:1321            5.345  6.773  +23.7%
k = 1331                1.744  1.491  -15.7%
k = 1341                1.731  1.447  -17.9%

partialsort!(rand(Float32, 2417), k)
k = 2315:2339           1.954  1.787  -08.9%
k = 1                   2.129  1.228  -55.0%
k = 25                  2.133  1.302  -49.4%
k = 1209                2.383  3.599  +41.2%

partialsort!(rand(Float32, 1412894), k)
k = 706447:706448       2.456  1.606  -42.5%
k = 781870:784710       2.638  1.690  -44.5%
k = 849750:1250424      8.591 17.445  +70.8%
k = 265276              1.924  1.646  -15.6%
k = 1410938             1.768  1.091  -48.3%

partialsort!(rand(Float32, 3469856), k)
k = 1734928:1734929     3.008  1.761  -53.6%
k = 3441966:3469856     1.896  1.394  -30.7%
k = 2650041             1.752  1.737  -00.9%
k = 3461652             1.762  1.077  -49.2%
k = 3469697             1.774  1.047  -52.8%

partialsort!(rand(Float64, 3), k)
k = 1                  10.096 13.835  +31.5%
k = 2                  10.547 15.117  +36.0%
k = 3                   9.771 13.798  +34.5%

partialsort!(rand(Float64, 34), k)
k = 12:12               5.809  5.032  -14.4%
k = 1                   5.566  5.010  -10.5%
k = 5                   5.588  5.096  -09.2%
k = 32                  4.902  5.107  +04.1%

partialsort!(rand(Float64, 6994), k)
k = 1:843               3.390  3.289  -03.0%
k = 1:2826              6.077 10.396  +53.7%
k = 61                  2.693  1.364  -68.0%
k = 6707                1.638  1.531  -06.8%
k = 6994                1.567  1.418  -10.0%

partialsort!(rand(Float64, 1046517), k)
k = 161433:167648       2.730  2.151  -23.9%
k = 500996:870065      10.586 22.474  +75.3%
k = 6515                2.220  1.072  -72.8%
k = 351940              2.712  1.988  -31.1%
k = 1037448             2.267  1.292  -56.2%

partialsort!(rand(Float64, 1051336), k)
k = 1:4416              2.092  1.114  -63.0%
k = 455127:1037517     17.144 38.654  +81.3%
k = 1042624:1051336     5.153  1.371  -132.4%
k = 1                   2.063  0.947  -77.9%
k = 913974              4.896  1.893  -95.0%

partialsort!(rand(Float64, 3029236), k)
k = 1514618:1514619     3.359  1.977  -53.0%
k = 2999085:3029236     2.490  1.655  -40.8%
k = 160737              3.498  1.986  -56.6%
k = 3001131             2.621  1.981  -28.0%
k = 3029236             2.310  1.096  -74.6%

partialsort!(rand(Float64, 6212361), k)
k = 3649468:5431358    10.937 20.295  +61.8%
k = 6168786:6212361     2.770  1.464  -63.8%
k = 1                   2.592  1.079  -87.7%
k = 6212361             2.529  1.167  -77.4%

partialsort!(rand(Int128, 56), k)
k = 1:1                 5.784  8.571  +39.3%
k = 56:56               2.758  5.446  +68.0%
k = 16                  5.704  5.119  -10.8%
k = 50                  2.788  4.811  +54.6%
k = 56                  2.827  4.960  +56.2%

partialsort!(rand(Int128, 84), k)
k = 42:43               4.865  6.782  +33.2%
k = 62:62               3.650  5.840  +47.0%
k = 1                   3.875  4.305  +10.5%
k = 84                  3.182  2.630  -19.0%

partialsort!(rand(Int128, 271), k)
k = 1:2                 2.605  3.129  +18.3%
k = 108:110             3.283  3.898  +17.2%
k = 245:246             3.735  4.477  +18.1%
k = 271:271             4.016  4.766  +17.1%
k = 3                   2.614  2.686  +02.7%

partialsort!(rand(Int128, 4647), k)
k = 1                   4.035  2.035  -68.4%
k = 2270                4.430  2.385  -61.9%
k = 3279                4.178  2.457  -53.1%
k = 4644                1.740  1.766  +01.5%
k = 4647                1.757  1.784  +01.5%

partialsort!(rand(Int128, 6493), k)
k = 1:24                3.138  2.092  -40.5%
k = 1161:3576           7.900  7.887  -00.2%
k = 1722                2.971  2.394  -21.6%
k = 6493                1.906  1.668  -13.3%

partialsort!(rand(Int128, 11572), k)
k = 370:9931           18.839 18.244  -03.2%
k = 5786:5787           2.474  2.412  -02.5%
k = 11527:11572         2.675  1.718  -44.3%
k = 1602                2.744  2.247  -20.0%
k = 11528               2.679  1.617  -50.5%

partialsort!(rand(Int128, 24908), k)
k = 12454:12455         4.756  2.325  -71.6%
k = 21710               3.663  2.136  -53.9%
k = 24908               1.522  1.504  -01.2%

partialsort!(rand(Int128, 3802413), k)
k = 1:24959             3.391  1.685  -69.9%
k = 1:3621317          41.984 37.486  -11.3%
k = 1815638:1841498     4.307  1.962  -78.6%
k = 3789561:3802413     4.025  1.335  -110.3%
k = 3802413             3.904  1.226  -115.8%

partialsort!(rand(Int128, 5462941), k)
k = 1:21462             5.965  1.648  -128.7%
k = 1:5099526          39.881 38.254  -04.2%
k = 16598               6.804  1.565  -146.9%
k = 2731471             6.221  1.902  -118.5%
k = 5462941             6.516  1.262  -164.2%

partialsort!(rand(Int128, 22252660), k)
k = 1:90642            14.692  1.615  -220.8%
k = 1:3951357          11.860  8.545  -32.8%
k = 3032737:3090032     4.914  1.815  -99.6%
k = 1                   5.119  1.448  -126.3%
k = 22187610            4.722  1.216  -135.7%

partialsort!(rand(Int64, 4), k)
k = 1:1                 7.893  8.578  +08.3%
k = 4:4                 8.972  9.366  +04.3%
k = 4                   7.966  8.170  +02.5%

partialsort!(rand(Int64, 247), k)
k = 1                   2.406  2.530  +05.0%
k = 2                   1.558  2.135  +31.5%
k = 3                   2.215  2.131  -03.9%
k = 124                 3.476  2.357  -38.8%

partialsort!(rand(Int64, 885), k)
k = 96:381              5.446  5.948  +08.8%
k = 566:570             4.250  4.370  +02.8%
k = 878:885             1.605  2.076  +25.7%
k = 1                   1.331  1.558  +15.7%
k = 2                   1.391  1.806  +26.1%

partialsort!(rand(Int64, 9113), k)
k = 1:6373             10.813 10.644  -01.6%
k = 9081                1.280  1.559  +19.7%
k = 9103                1.262  1.084  -15.2%
k = 9113                1.257  1.079  -15.3%

partialsort!(rand(Int64, 66063), k)
k = 1:300               1.883  1.036  -59.8%
k = 36275:53130         5.594  5.700  +01.9%
k = 1                   1.706  0.887  -65.4%
k = 28981               2.393  1.278  -62.7%
k = 33032               1.647  1.279  -25.3%

partialsort!(rand(Int64, 80969), k)
k = 1:17084             5.396  4.470  -18.8%
k = 23302:70376        12.018 12.612  +04.8%
k = 33166:33845         2.900  1.404  -72.6%
k = 40                  1.894  0.913  -73.0%
k = 526                 1.894  0.918  -72.4%

partialsort!(rand(Int64, 170571), k)
k = 1:1038              1.144  1.022  -11.2%
k = 1:1629              1.177  1.025  -13.9%
k = 1:117683           15.013 14.777  -01.6%
k = 100458:145299       7.945  6.151  -25.6%
k = 85286               1.871  1.286  -37.5%

partialsort!(rand(Int64, 864519), k)
k = 2399:826475        23.491 23.279  -00.9%
k = 13365:242859        7.745  7.076  -09.0%
k = 203249:209420       2.948  1.701  -55.0%
k = 246445:252872       3.132  1.736  -59.0%
k = 432260              1.988  1.692  -16.1%

partialsort!(rand(String, 2), k)
k = 1:2                15.082 15.000  -00.5%
k = 1                  15.191 14.968  -01.5%
k = 2                  15.343 15.193  -01.0%

partialsort!(rand(String, 13), k)
k = 1:1                13.444 11.886  -12.3%
k = 1:12               13.023 11.193  -15.1%
k = 13:13              12.575 11.351  -10.2%
k = 1                  12.910 11.619  -10.5%
k = 13                 13.511 11.597  -15.3%

partialsort!(rand(String, 22), k)
k = 1:1                22.199 23.420  +05.4%
k = 11:12              22.083 23.769  +07.4%
k = 16:16              22.472 23.419  +04.1%
k = 22                  7.066 23.783  +121.4%

partialsort!(rand(String, 44), k)
k = 1:30               23.358 27.462  +16.2%
k = 34:34               8.259 11.860  +36.2%
k = 1                   8.321 12.730  +42.5%
k = 39                  8.259 13.506  +49.2%

partialsort!(rand(String, 138), k)
k = 1:1                15.508 16.140  +04.0%
k = 1:2                15.014 16.881  +11.7%
k = 1:111              28.005 29.438  +05.0%
k = 8:8                15.014 16.990  +12.4%
k = 1                  15.014  7.672  -67.1%

partialsort!(rand(String, 1159), k)
k = 1:8                 9.347  7.406  -23.3%
k = 1:10               10.443  7.370  -34.9%
k = 907:916            12.044 13.463  +11.1%
k = 10                 10.012  6.669  -40.6%
k = 890                12.116 11.558  -04.7%

partialsort!(rand(String, 1778), k)
k = 889:890            15.490 12.444  -21.9%
k = 1                   8.178  5.718  -35.8%
k = 4                   8.226  5.741  -36.0%
k = 1442                7.804 10.217  +26.9%
k = 1765                7.756  6.913  -11.5%

partialsort!(rand(String, 6996), k)
k = 1:9                 8.165  5.598  -37.7%
k = 480:500             9.636  8.541  -12.1%
k = 1143:6613          86.442 86.151  -00.3%
k = 2800:3871          30.410 24.085  -23.3%
k = 6996               10.447  5.664  -61.2%

partialsort!(rand(String, 9178), k)
k = 4589:4590          18.518 11.027  -51.8%
k = 7108:7329          18.082 11.772  -42.9%
k = 9112:9178          14.251  5.897  -88.2%
k = 1                   6.519  5.293  -20.8%
k = 9178               14.773  5.212  -104.2%

partialsort!(rand(String, 41621), k)
k = 19610:19752        19.027 10.953  -55.2%
k = 339                13.291  4.650  -105.0%
k = 20811              18.025 10.404  -55.0%
k = 29484              13.994  9.245  -41.5%
k = 41323              14.500  5.066  -105.2%

partialsort!(rand(String, 145977), k)
k = 135563:136325       7.751  9.293  +18.1%
k = 1088               11.616  4.787  -88.7%
k = 29000              16.266  8.101  -69.7%
k = 145913              5.996  4.815  -21.9%
k = 145977              5.875  4.828  -19.6%

partialsort!(rand(String, 6803515), k)
k = 1:3811305          181.665 184.072  +01.3%
k = 2206222:2240415    22.658 10.770  -74.4%
k = 6388849:6420065    22.926  9.106  -92.3%
k = 1581624            21.180  8.640  -89.7%
k = 1865233            19.512  8.863  -78.9%

partialsort!(rand(UInt8, 21), k)
k = 1:2                 5.456  4.547  -18.2%
k = 1                   5.675  4.523  -22.7%
k = 11                  5.539  4.504  -20.7%
k = 21                  5.607  4.523  -21.5%

partialsort!(rand(UInt8, 106), k)
k = 1:2                 1.535  2.904  +63.7%
k = 1:16                2.802  4.165  +39.6%
k = 1:77                5.235  6.700  +24.7%
k = 82:83               3.043  4.537  +39.9%
k = 34                  2.230  2.473  +10.3%

partialsort!(rand(UInt8, 71092), k)
k = 30602:31046         1.325  1.156  -13.6%
k = 35546:35547         1.281  1.185  -07.8%
k = 1                   0.797  0.554  -36.3%
k = 12198               0.753  1.015  +29.9%

partialsort!(rand(UInt8, 176199), k)
k = 175743:176199       1.066  0.677  -45.4%
k = 1                   0.850  0.589  -36.7%
k = 88100               1.389  1.141  -19.6%
k = 176199              1.028  0.723  -35.1%

partialsort!(rand(UInt8, 2649352), k)
k = 1:891915            6.867  6.232  -09.7%
k = 1324676:1324677     1.189  0.990  -18.3%
k = 1                   1.630  0.538  -110.8%
k = 687451              2.050  1.108  -61.5%
k = 2648225             0.916  0.653  -33.8%

partialsort!(rand(UInt8, 19380151), k)
k = 1:1974500           3.210  2.316  -32.7%
k = 1                   1.144  0.591  -66.1%
k = 9690076             1.424  1.101  -25.7%
k = 19315724            1.196  0.694  -54.4%

partialsort(rand(Float32, 42), k)
k = 11:12               4.654  8.016  +54.4%
k = 16:27               6.125 10.700  +55.8%
k = 11                  4.454  5.157  +14.6%
k = 42                  4.299  5.549  +25.5%

partialsort(rand(Float32, 49), k)
k = 1:10                5.924  8.946  +41.2%
k = 1:15                7.160 10.553  +38.8%
k = 1                   3.452  4.272  +21.3%
k = 49                  3.069  4.946  +47.7%

partialsort(rand(Float32, 254), k)
k = 3:34                2.726  3.003  +09.7%
k = 127:128             2.802  3.369  +18.4%
k = 252:254             2.350  2.687  +13.4%
k = 118                 2.908  3.064  +05.2%
k = 254                 2.345  2.696  +13.9%

partialsort(rand(Float32, 311), k)
k = 138:176             4.236  5.416  +24.6%
k = 169:170             3.570  4.486  +22.8%
k = 1                   2.213  2.070  -06.7%
k = 156                 3.375  2.273  -39.5%

partialsort(rand(Float32, 849), k)
k = 1:557               7.362  9.545  +26.0%
k = 448:453             3.467  2.802  -21.3%
k = 483:490             3.547  2.806  -23.4%
k = 1                   1.919  1.566  -20.3%
k = 848                 3.101  1.669  -62.0%

partialsort(rand(Float32, 1179), k)
k = 1:3                 2.748  1.807  -41.9%
k = 270:472             5.425  4.780  -12.7%
k = 837:921             3.935  3.540  -10.6%
k = 1                   2.774  1.529  -59.6%
k = 1179                1.427  1.577  +10.0%

partialsort(rand(Float32, 266404), k)
k = 133202:133203       1.984  1.654  -18.2%
k = 263867:266404       1.681  1.337  -22.9%
k = 266216:266404       1.494  1.024  -37.7%
k = 100179              1.954  1.650  -16.9%

partialsort(rand(Float32, 1769209), k)
k = 88264:771345       11.808 25.203  +75.8%
k = 999524:1000381      2.175  2.017  -07.6%
k = 1768923:1769209     2.472  1.319  -62.8%
k = 3977                2.974  1.242  -87.4%
k = 1769209             2.311  1.344  -54.2%

partialsort(rand(Float64, 4), k)
k = 2:2                 6.944  9.687  +33.3%
k = 2:3                 7.445 10.807  +37.3%
k = 4:4                 8.068  9.556  +16.9%
k = 4                   7.047 10.168  +36.7%

partialsort(rand(Float64, 68), k)
k = 34:35               4.737  6.278  +28.2%
k = 1                   4.294  4.529  +05.3%
k = 10                  4.921  4.553  -07.8%
k = 68                  3.059  3.581  +15.7%

partialsort(rand(Float64, 163), k)
k = 1:1                 3.260  3.789  +15.0%
k = 1                   3.225  2.740  -16.3%
k = 2                   3.251  2.484  -26.9%
k = 97                  3.588  2.606  -32.0%

partialsort(rand(Float64, 302), k)
k = 1:243               7.910 10.371  +27.1%
k = 299:302             3.583  4.126  +14.1%
k = 136                 3.432  3.600  +04.8%
k = 302                 3.027  2.639  -13.7%

partialsort(rand(Float64, 2613), k)
k = 1:26                2.264  1.712  -28.0%
k = 1388:1403           2.875  2.785  -03.2%
k = 1307                2.876  2.328  -21.1%
k = 2613                1.696  1.462  -14.8%

partialsort(rand(Float64, 2900), k)
k = 1:485               3.678  4.095  +10.7%
k = 1:1370              6.437  8.764  +30.9%
k = 1450:1451           3.218  2.950  -08.7%
k = 76                  2.371  1.580  -40.5%
k = 361                 2.376  2.634  +10.3%

partialsort(rand(Float64, 6168), k)
k = 1:2218              5.499  6.735  +20.3%
k = 1318:4528           7.863 11.119  +34.6%
k = 1                   2.331  1.331  -56.0%
k = 22                  2.344  1.338  -56.1%
k = 6168                1.898  1.405  -30.1%

partialsort(rand(Float64, 7112), k)
k = 3374:3381           3.615  2.384  -41.6%
k = 3556:3557           3.492  2.437  -36.0%
k = 26                  1.775  1.306  -30.7%
k = 57                  1.793  1.353  -28.1%
k = 4544                3.158  2.197  -36.3%

partialsort(rand(Float64, 8773), k)
k = 4378:4433           2.033  2.612  +25.1%
k = 8698:8773           1.985  1.586  -22.4%
k = 49                  2.189  1.387  -45.7%
k = 4387                2.000  2.223  +10.6%
k = 7094                1.909  2.151  +11.9%

partialsort(rand(Float64, 27548), k)
k = 13774:13775         2.310  2.204  -04.7%
k = 27448:27548         1.768  1.339  -27.8%
k = 1980                2.084  2.036  -02.4%
k = 3915                1.995  2.107  +05.5%
k = 27548               1.750  1.317  -28.4%

partialsort(rand(Float64, 28179), k)
k = 1378:13101          8.279 19.063  +83.4%
k = 1                   2.324  1.220  -64.5%
k = 103                 2.335  1.226  -64.4%
k = 14090               1.993  2.066  +03.6%
k = 14103               2.001  2.007  +00.3%

partialsort(rand(Float64, 28808), k)
k = 1:9188              6.584 13.262  +70.0%
k = 14404:14405         2.702  2.197  -20.7%
k = 27501:27573         3.097  2.171  -35.5%
k = 11                  1.830  1.231  -39.6%
k = 28636               2.953  1.335  -79.4%

partialsort(rand(Float64, 29910), k)
k = 1:172               1.746  1.383  -23.3%
k = 185                 1.720  1.216  -34.7%
k = 273                 1.718  1.220  -34.2%
k = 29703               2.034  1.316  -43.5%
k = 29908               2.021  1.277  -45.9%

partialsort(rand(Float64, 430947), k)
k = 1:23609             2.869  4.027  +33.9%
k = 426654:430947       3.735  1.646  -81.9%
k = 172991              2.890  2.222  -26.3%
k = 236437              3.251  2.120  -42.7%
k = 430947              3.341  1.301  -94.3%

partialsort(rand(Float64, 1731857), k)
k = 213972              3.156  2.483  -24.0%
k = 859089              4.200  2.588  -48.4%
k = 1349429             4.172  2.489  -51.6%
k = 1731857             2.574  1.610  -46.9%

partialsort(rand(Int128, 6239), k)
k = 1:22                2.391  2.124  -11.9%
k = 1:987               4.294  4.301  +00.2%
k = 1:4122             13.036 12.455  -04.6%
k = 1:5968             16.970 17.023  +00.3%
k = 5466                2.598  2.337  -10.6%

partialsort(rand(Int128, 43356), k)
k = 1                   2.839  1.821  -44.4%
k = 42975               1.993  1.499  -28.5%
k = 42989               1.975  1.455  -30.6%
k = 43356               1.923  1.491  -25.5%

partialsort(rand(Int128, 47581), k)
k = 1:183               3.129  1.909  -49.4%
k = 1:937               3.244  2.093  -43.8%
k = 310:37621          23.330 23.535  +00.9%
k = 47178:47581         1.962  1.654  -17.0%
k = 15970               7.137  3.616  -68.0%

partialsort(rand(Int128, 141841), k)
k = 1:912               3.167  1.956  -48.2%
k = 141227:141841       2.053  1.701  -18.8%
k = 141394:141841       2.030  1.631  -21.9%
k = 1                   3.045  1.834  -50.7%
k = 132529              1.986  2.095  +05.3%

partialsort(rand(Int128, 234381), k)
k = 1:466               2.003  1.860  -07.4%
k = 10909:144086       19.509 19.817  +01.6%
k = 1                   1.996  1.812  -09.7%
k = 232422              3.225  1.661  -66.3%
k = 233023              3.195  1.608  -68.7%

partialsort(rand(Int128, 3699957), k)
k = 1:2480865          32.084 31.112  -03.1%
k = 1:2648196          40.711 50.277  +21.1%
k = 1                   8.191  3.768  -77.6%
k = 3684223             6.407  2.854  -80.9%

partialsort(rand(Int128, 10134995), k)
k = 1:7311             11.225  4.345  -94.9%
k = 4762326:4798250    16.755  3.402  -159.4%
k = 7208105:8030984    30.537 12.322  -90.8%
k = 5067498            10.386  3.179  -118.4%
k = 10112027            6.250  2.756  -81.9%

partialsort(rand(Int64, 3), k)
k = 1:2                 8.248  8.170  -00.9%
k = 3:3                 9.157  9.319  +01.8%
k = 1                   8.127  8.073  -00.7%

partialsort(rand(Int64, 14), k)
k = 7:8                 5.008  4.129  -19.3%
k = 1                   5.001  4.035  -21.5%
k = 5                   4.901  4.042  -19.3%
k = 14                  4.972  4.014  -21.4%

partialsort(rand(Int64, 72), k)
k = 1:40                5.087  6.931  +30.9%
k = 1:42                4.954  7.593  +42.7%
k = 60:60               2.850  5.055  +57.3%
k = 1                   3.061  3.181  +03.8%
k = 72                  1.640  2.626  +47.0%

partialsort(rand(Int64, 59667), k)
k = 1:90                1.973  1.423  -32.7%
k = 1                   1.946  0.925  -74.4%
k = 29834               2.024  1.323  -42.5%
k = 59667               1.061  0.960  -10.0%

partialsort(rand(Int64, 687637), k)
k = 1                   2.170  0.970  -80.5%
k = 6872                2.164  1.036  -73.7%
k = 343819              1.935  1.429  -30.3%
k = 681048              1.210  1.128  -07.0%

partialsort(rand(Int64, 1317944), k)
k = 1:50669             3.428  1.997  -54.0%
k = 1:338036            9.349  7.138  -27.0%
k = 9103:899321        18.610 18.670  +00.3%
k = 1247468             2.066  1.703  -19.3%
k = 1317944             2.003  1.365  -38.4%

partialsort(rand(Int64, 2385485), k)
k = 1:1911              2.084  1.312  -46.3%
k = 1:711890           11.337  8.545  -28.3%
k = 765906:1233854      8.038  6.507  -21.1%
k = 8768                2.093  1.308  -47.0%
k = 1192743             3.561  1.722  -72.6%

partialsort(rand(String, 1), k)
k = 1                  21.452 21.178  -01.3%

partialsort(rand(String, 2), k)
k = 1:1                14.065 14.289  +01.6%
k = 1:2                14.166 14.416  +01.7%
k = 1                  14.139 14.440  +02.1%
k = 2                  14.139 14.291  +01.1%

partialsort(rand(String, 76), k)
k = 1:1                12.595 15.639  +21.6%
k = 1:69               24.808 27.069  +08.7%
k = 38:39              13.836 15.437  +11.0%
k = 1                  13.346  9.392  -35.1%

partialsort(rand(String, 177), k)
k = 1:97               19.068 20.056  +05.1%
k = 77                  7.865 10.621  +30.0%
k = 89                  9.347 10.538  +12.0%
k = 177                 9.804  7.104  -32.2%

partialsort(rand(String, 631), k)
k = 1:520              42.328 43.450  +02.6%
k = 1                  10.961  8.346  -27.3%
k = 147                11.939  9.681  -21.0%
k = 316                17.816 12.071  -38.9%
k = 631                 4.952  7.316  +39.0%

partialsort(rand(String, 21232), k)
k = 10616:10617        16.677 10.911  -42.4%
k = 12562:12573        15.980 10.813  -39.1%
k = 21094:21232         6.445  5.752  -11.4%
k = 204                16.649  4.841  -123.5%
k = 17497              10.326  8.187  -23.2%

partialsort(rand(String, 1168148), k)
k = 477727:492176      23.248 13.192  -56.7%
k = 40                 20.478  5.983  -123.0%
k = 6153               19.729  6.043  -118.3%
k = 7714               20.117  5.968  -121.5%
k = 1168148             5.235  5.996  +13.6%

partialsort(rand(String, 1671468), k)
k = 587216:650171      30.025 18.322  -49.4%
k = 892834:1098407     48.032 40.643  -16.7%
k = 16234              11.980  7.993  -40.5%
k = 1660847            19.883  6.625  -109.9%
k = 1671468            20.015  6.304  -115.5%

partialsort(rand(UInt8, 1), k)
k = 1:1                20.490 20.856  +01.8%
k = 1                  21.911 21.635  -01.3%

partialsort(rand(UInt8, 21), k)
k = 1:1                 4.406  3.672  -18.2%
k = 1                   4.289  3.555  -18.8%
k = 11                  4.264  3.525  -19.0%

partialsort(rand(UInt8, 160), k)
k = 1:1                 1.531  2.487  +48.5%
k = 1:2                 1.594  2.474  +44.0%
k = 63:64               1.899  2.999  +45.7%
k = 80:81               2.267  3.170  +33.5%
k = 1                   1.589  1.773  +11.0%

partialsort(rand(UInt8, 300967), k)
k = 66275:68273         1.599  1.096  -37.7%
k = 231768:287281       4.817  3.223  -40.2%
k = 150484              1.934  3.183  +49.8%
k = 242115              2.255  1.118  -70.1%
k = 300967              2.026  0.670  -110.6%

There are a few too many regressions right now, I'm going to try to squash a few.

base/sort.jl Show resolved Hide resolved
@LilithHafner
Copy link
Member Author

LilithHafner commented Nov 3, 2023

Rerunning the benchmarks
using BenchmarkTools, Random, Printf
get_length() = round(Int, exp(17*rand()))
get_eltype() = rand([Float64, Float32, Int, UInt8, Int128, String])
get_function() = rand([partialsort!, partialsort])
get_element(x) = rand(x)
get_element(::Type{String}) = randstring()
get_k(n) = rand([
        1,
        n,
        rand(1:n),
        rand(1:1+n÷100),
        rand(n-n÷100:n),

        iseven(n) ? (n÷2:n÷2+1) : (n+1)÷2,

        UnitRange(sort((rand(1:n), rand(1:n)))...),
        1:rand(1:n),
        1:rand(1:1+n÷100),
        rand(n-n÷100:n):n,
        (1:rand(1:1+n÷100)) .+ rand(0:(n-n÷100-1)),
    ])

new!(v, k) = partialsort!(v, k)
new(v, k) = partialsort(v, k)
function old!(v, k)
    Base.Sort._sort!(v, Base.Sort.InitialOptimizations(Base.Sort.ScratchQuickSort(k)), Base.Order.Forward, (;))
    Base.Sort.maybeview(v, k)
end
old(v::AbstractVector, k::Union{Integer,OrdinalRange}; kws...) =
    old!(Base.copymutable(v), k; kws...)

function test(length::Int, eltype, func)
    func === partialsort && (length ÷= 2) # benchmarking is inefficient
    length == 0 && return
    data = [get_element(eltype) for _ in 1:length]
    ks = [get_k(length) for _ in 1:5]
    sort!(ks, by = x->(x isa Number, x))
    unique!(ks)
    println("\n$func(rand($eltype, $length), k)")
    for k in ks
        (t_old, t_new) = func === partialsort! ? test!(similar(data), data, k) : test(data, k)
        a = 1e9t_old/length
        b = 1e9t_new/length
        c = log(b/a)*100
        @printf "k = %-18s %6.3f %6.3f  %+05.1f%%\n" k a b c
    end
end

estimate_runtime(data::Vector{T}, k) where T =
    round(Int, (10 + length(data) + length(k) * log(length(k))) * (T === String ? 3 : T == Int128 ? 2 : 1))
function test(data::Vector{T}, k::K) where {T,K}
    estimated_runtime = estimate_runtime(data, k)
    if estimated_runtime > 10^7
        t_old = @elapsed old_res = old(data, k)
        t_new = @elapsed new_res = new(data, k)
        old_res == new_res || error("Bad sort")
        t_old, t_new
    elseif estimated_runtime > 10^5
        truth = old(data, k)
        t_old, t_new = Inf, Inf
        for _ in 1:2*10^7 ÷ estimated_runtime
            t1 = @elapsed res1 = old(data, k)
            res1 == truth || error("Bad old sort")
            t_old = min(t_old, t1)

            t2 = @elapsed res2 = new(data, k)
            res2 == truth || error("Bad sort")
            t_new = min(t_new, t2)
        end
        t_old, t_new
    else
        old(data, k) == new(data, k) || error("Bad sort")
        evals = max(1, 10^4 ÷ estimated_runtime)
        samples = 10^7 ÷ evals ÷ estimated_runtime
        t_old = @belapsed old($data, $k) evals=evals samples=samples gctrial=false
        t_new = @belapsed new($data, $k) evals=evals samples=samples gctrial=false
        t_old, t_new
    end
end

function test!(d::Vector{T}, data::Vector{T}, k::K) where {T,K}
    estimated_runtime = estimate_runtime(data, k)
    if estimated_runtime > 10^7
        copyto!(d, data)
        t_old = @elapsed old_res = old!(d, k)
        copyto!(d, data)
        t_new = @elapsed new_res = new!(d, k)
        old_res == new_res || error("Bad sort")
    elseif estimated_runtime > 10^6
        truth = old(data, k)
        t_old, t_new = Inf, Inf
        for _ in 1:2*10^7 ÷ estimated_runtime
            copyto!(d, data)
            t1 = @elapsed res1 = old!(d, k)
            res1 == truth || error("Bad old sort")
            t_old = min(t_old, t1)

            copyto!(d, data)
            t2 = @elapsed res2 = new!(d, k)
            res2 == truth || error("Bad sort")
            t_new = min(t_new, t2)
        end
    else
        old!(copyto!(d, data), k) == new!(copyto!(d, data), k) || error("Bad sort")
        evals = max(1, 10^4 ÷ estimated_runtime)
        samples = 10^7 ÷ evals ÷ estimated_runtime
        if evals == 1
            t_old = @belapsed old($d, $k) setup=(copyto!($d, $data)) evals=evals samples=samples gctrial=false
            t_new = @belapsed new($d, $k) setup=(copyto!($d, $data)) evals=evals samples=samples gctrial=false
        else
            t_old = @belapsed old(copyto!($d, $data), $k) evals=evals samples=samples gctrial=false
            t_new = @belapsed new(copyto!($d, $data), $k) evals=evals samples=samples gctrial=false
        end
    end
    t_old, t_new
end

function test(n, seed=1729)
    Random.seed!(seed)
    samples = [(get_length(), get_eltype(), get_function()) for _ in 1:n]
    sort!(samples, by = x->(x[3] === partialsort, string(x[2]), x[1]))
    try
        for s in samples
            test(s...)
        end
    catch x
        x isa InterruptException || rethrow()
    end
    nothing
end

test(100)
Produces better results
partialsort!(rand(Float32, 3), k)
k = 1:1                13.257 12.516  -05.8%
k = 2:3                14.006 12.119  -14.5%
k = 2                  12.119 13.130  +08.0%
k = 3                  13.275 13.961  +05.0%

partialsort!(rand(Float32, 120), k)
k = 1:1                 2.353  2.659  +12.2%
k = 1:68                6.899  6.929  +00.4%
k = 120:120             3.385  3.655  +07.7%

partialsort!(rand(Float32, 356), k)
k = 1:1                 2.176  2.241  +02.9%
k = 178:179             2.904  3.411  +16.1%
k = 222:225             2.931  3.124  +06.4%
k = 1                   2.172  2.185  +00.6%
k = 356                 2.562  2.401  -06.5%

partialsort!(rand(Float32, 764), k)
k = 1:612               7.171  7.199  +00.4%
k = 69:736              7.635  7.907  +03.5%
k = 3                   2.318  1.759  -27.6%

partialsort!(rand(Float32, 1341), k)
k = 1:7                 3.880  1.722  -81.2%
k = 414:418             3.919  2.867  -31.3%
k = 683:1321            5.282  5.344  +01.2%
k = 1331                1.749  1.696  -03.1%
k = 1341                1.784  1.713  -04.1%

partialsort!(rand(Float32, 2417), k)
k = 2315:2339           1.827  1.776  -02.9%
k = 1                   2.164  1.383  -44.7%
k = 25                  2.095  1.547  -30.3%
k = 1209                2.103  2.125  +01.0%

partialsort!(rand(Float32, 1412894), k)
k = 706447:706448       2.379  1.412  -52.2%
k = 781870:784710       2.556  1.478  -54.8%
k = 849750:1250424      7.945  7.571  -04.8%
k = 265276              1.936  1.521  -24.2%
k = 1410938             1.737  1.013  -53.9%

partialsort!(rand(Float32, 3469856), k)
k = 1734928:1734929     2.770  1.424  -66.6%
k = 3441966:3469856     1.795  1.153  -44.2%
k = 2650041             1.616  1.525  -05.8%
k = 3461652             1.699  0.955  -57.5%
k = 3469697             1.708  0.960  -57.7%

partialsort!(rand(Float64, 3), k)
k = 1                  10.493 10.385  -01.0%
k = 2                  10.277 11.866  +14.4%
k = 3                  10.981 11.577  +05.3%

partialsort!(rand(Float64, 34), k)
k = 12:12               6.014  6.025  +00.2%
k = 1                   5.857  4.173  -33.9%
k = 5                   5.917  4.465  -28.2%
k = 32                  4.891  3.892  -22.8%

partialsort!(rand(Float64, 6994), k)
k = 1:843               3.402  2.663  -24.5%
k = 1:2826              7.304  6.279  -15.1%
k = 61                  2.734  1.591  -54.2%
k = 6707                1.632  1.567  -04.1%
k = 6994                1.692  1.579  -06.9%

partialsort!(rand(Float64, 1046517), k)
k = 161433:167648       2.782  1.872  -39.6%
k = 500996:870065      10.050  9.708  -03.5%
k = 6515                2.156  1.085  -68.6%
k = 351940              2.833  1.765  -47.3%
k = 1037448             2.433  1.120  -77.6%

partialsort!(rand(Float64, 1051336), k)
k = 1:4416              1.940  1.091  -57.6%
k = 455127:1037517     14.790 14.658  -00.9%
k = 1042624:1051336     4.602  1.121  -141.2%
k = 1                   1.871  0.951  -67.7%
k = 913974              4.487  1.712  -96.4%

partialsort!(rand(Float64, 3029236), k)
k = 1514618:1514619     2.940  1.683  -55.8%
k = 2999085:3029236     2.263  1.199  -63.5%
k = 160737              3.280  1.718  -64.7%
k = 3001131             2.111  1.099  -65.3%
k = 3029236             2.073  1.012  -71.7%

partialsort!(rand(Float64, 6212361), k)
k = 3649468:5431358    10.719  8.271  -25.9%
k = 6168786:6212361     2.657  1.174  -81.7%
k = 1                   2.290  0.960  -87.0%
k = 6212361             2.281  1.042  -78.3%

partialsort!(rand(Int128, 56), k)
k = 1:1                 6.310  5.655  -11.0%
k = 56:56               4.186  3.194  -27.0%
k = 16                  7.321  2.966  -90.4%
k = 50                  3.155  3.105  -01.6%
k = 56                  2.996  2.550  -16.1%

partialsort!(rand(Int128, 84), k)
k = 42:43               6.124  4.769  -25.0%
k = 62:62               3.743  3.781  +01.0%
k = 1                   5.307  3.210  -50.3%
k = 84                  3.060  1.694  -59.2%

partialsort!(rand(Int128, 271), k)
k = 1:2                 2.505  2.894  +14.4%
k = 108:110             3.374  3.500  +03.7%
k = 245:246             3.997  4.206  +05.1%
k = 271:271             4.124  4.441  +07.4%
k = 3                   2.605  2.650  +01.7%

partialsort!(rand(Int128, 4647), k)
k = 1                   4.026  2.600  -43.7%
k = 2270                4.510  2.636  -53.7%
k = 3279                4.214  2.529  -51.1%
k = 4644                1.775  1.820  +02.5%
k = 4647                1.829  1.784  -02.5%

partialsort!(rand(Int128, 6493), k)
k = 1:24                3.433  1.694  -70.6%
k = 1161:3576           9.587  9.311  -02.9%
k = 1722                2.971  2.631  -12.2%
k = 6493                2.047  1.758  -15.2%

partialsort!(rand(Int128, 11572), k)
k = 370:9931           18.367 18.529  +00.9%
k = 5786:5787           2.492  2.592  +04.0%
k = 11527:11572         2.650  2.204  -18.5%
k = 1602                2.758  2.380  -14.7%
k = 11528               2.650  1.613  -49.6%

partialsort!(rand(Int128, 24908), k)
k = 12454:12455         4.761  2.319  -71.9%
k = 21710               3.657  2.300  -46.4%
k = 24908               1.586  1.534  -03.3%

partialsort!(rand(Int128, 3802413), k)
k = 1:24959             4.389  1.298  -121.9%
k = 1:3621317          34.975 33.500  -04.3%
k = 1815638:1841498     3.626  1.833  -68.2%
k = 3789561:3802413     3.365  1.173  -105.4%
k = 3802413             6.394  1.123  -173.9%

partialsort!(rand(Int128, 5462941), k)
k = 1:21462             6.762  1.170  -175.4%
k = 1:5099526          36.968 38.356  +03.7%
k = 16598               3.776  1.142  -119.6%
k = 2731471            16.004  1.716  -223.3%
k = 5462941             4.313  1.141  -133.0%

partialsort!(rand(Int128, 22252660), k)
k = 1:90642            19.009  1.131  -282.2%
k = 1:3951357          13.614  7.313  -62.1%
k = 3032737:3090032     9.632  1.641  -177.0%
k = 1                   4.160  1.029  -139.7%
k = 22187610            4.244  1.138  -131.7%

partialsort!(rand(Int64, 4), k)
k = 1:1                 7.688  8.316  +07.8%
k = 4:4                 8.316  8.622  +03.6%
k = 4                   7.572  7.630  +00.8%

partialsort!(rand(Int64, 247), k)
k = 1                   1.971  1.709  -14.3%
k = 2                   1.558  2.029  +26.4%
k = 3                   1.549  1.709  +09.8%
k = 124                 3.938  1.940  -70.8%

partialsort!(rand(Int64, 885), k)
k = 96:381              5.398  5.571  +03.2%
k = 566:570             4.190  2.363  -57.3%
k = 878:885             1.591  1.977  +21.7%
k = 1                   1.395  1.618  +14.8%
k = 2                   1.314  1.605  +20.0%

partialsort!(rand(Int64, 9113), k)
k = 1:6373             10.086 11.522  +13.3%
k = 9081                1.285  1.148  -11.3%
k = 9103                1.294  1.106  -15.7%
k = 9113                1.266  1.079  -16.0%

partialsort!(rand(Int64, 66063), k)
k = 1:300               1.962  1.074  -60.3%
k = 36275:53130         6.044  6.020  -00.4%
k = 1                   1.699  1.130  -40.8%
k = 28981               2.395  1.311  -60.2%
k = 33032               1.650  1.431  -14.2%

partialsort!(rand(Int64, 80969), k)
k = 1:17084             5.291  4.722  -11.4%
k = 23302:70376        12.617 12.582  -00.3%
k = 33166:33845         2.909  1.372  -75.2%
k = 40                  1.946  0.936  -73.2%
k = 526                 1.890  1.059  -58.0%

partialsort!(rand(Int64, 170571), k)
k = 1:1038              1.151  0.984  -15.7%
k = 1:1629              1.213  1.327  +08.9%
k = 1:117683           13.952 13.964  +00.1%
k = 100458:145299       7.542  6.197  -19.6%
k = 85286               1.885  1.322  -35.5%

partialsort!(rand(Int64, 864519), k)
k = 2399:826475        20.629 20.534  -00.5%
k = 13365:242859        7.230  6.542  -10.0%
k = 203249:209420       2.432  1.493  -48.8%
k = 246445:252872       2.890  1.476  -67.2%
k = 432260              1.642  1.440  -13.2%

partialsort!(rand(String, 2), k)
k = 1:2                15.750 15.418  -02.1%
k = 1                  15.417 15.569  +01.0%
k = 2                  15.643 16.096  +02.9%

partialsort!(rand(String, 13), k)
k = 1:1                12.932 13.021  +00.7%
k = 1:12               12.668 13.074  +03.2%
k = 13:13              13.421 13.555  +01.0%
k = 1                  14.134 11.529  -20.4%
k = 13                 14.245 11.796  -18.9%

partialsort!(rand(String, 22), k)
k = 1:1                22.181 22.345  +00.7%
k = 11:12              22.841 22.651  -00.8%
k = 16:16              22.345 22.163  -00.8%
k = 22                  7.339 14.860  +70.5%

partialsort!(rand(String, 44), k)
k = 1:30               23.043 23.810  +03.3%
k = 34:34               9.035  8.321  -08.2%
k = 1                   8.259 10.665  +25.6%
k = 39                  8.398 11.674  +32.9%

partialsort!(rand(String, 138), k)
k = 1:1                15.138 15.069  -00.5%
k = 1:2                15.728 14.987  -04.8%
k = 1:111              28.306 28.382  +00.3%
k = 8:8                15.755 15.055  -04.5%
k = 1                  15.069  6.986  -76.9%

partialsort!(rand(String, 1159), k)
k = 1:8                 9.455  7.352  -25.2%
k = 1:10               10.102  7.370  -31.5%
k = 907:916            12.421 13.140  +05.6%
k = 10                 10.030  7.100  -34.5%
k = 890                12.133 12.582  +03.6%

partialsort!(rand(String, 1778), k)
k = 889:890            15.841 13.663  -14.8%
k = 1                   8.319  6.327  -27.4%
k = 4                   8.319  6.304  -27.7%
k = 1442                7.593 10.967  +36.8%
k = 1765                7.898  6.655  -17.1%

partialsort!(rand(String, 6996), k)
k = 1:9                 8.130  5.890  -32.2%
k = 480:500             9.767 10.214  +04.5%
k = 1143:6613          86.145 86.508  +00.4%
k = 2800:3871          31.631 27.295  -14.7%
k = 6996               10.286  5.640  -60.1%

partialsort!(rand(String, 9178), k)
k = 4589:4590          18.632 14.432  -25.5%
k = 7108:7329          18.450 11.913  -43.7%
k = 9112:9178          14.414  5.802  -91.0%
k = 1                   6.492  5.457  -17.4%
k = 9178               14.405  5.489  -96.5%

partialsort!(rand(String, 41621), k)
k = 19610:19752        18.747 12.522  -40.4%
k = 339                13.222  5.039  -96.5%
k = 20811              18.066 11.486  -45.3%
k = 29484              13.945 10.496  -28.4%
k = 41323              14.962  5.548  -99.2%

partialsort!(rand(String, 145977), k)
k = 135563:136325       7.565  9.835  +26.2%
k = 1088               12.334  5.173  -86.9%
k = 29000              16.495  9.233  -58.0%
k = 145913              6.136  5.146  -17.6%
k = 145977              6.231  4.929  -23.4%

partialsort!(rand(String, 6803515), k)
k = 1:3811305          165.445 163.866  -01.0%
k = 2206222:2240415    20.343 11.169  -60.0%
k = 6388849:6420065    21.737  8.672  -91.9%
k = 1581624            18.846  8.649  -77.9%
k = 1865233            16.361  9.331  -56.1%

partialsort!(rand(UInt8, 21), k)
k = 1:2                 5.406  5.723  +05.7%
k = 1                   5.527  4.831  -13.5%
k = 11                  5.681  4.714  -18.7%
k = 21                  5.379  4.677  -14.0%

partialsort!(rand(UInt8, 106), k)
k = 1:2                 1.521  1.595  +04.8%
k = 1:16                2.809  2.904  +03.3%
k = 1:77                5.199  5.342  +02.7%
k = 82:83               2.964  2.987  +00.8%
k = 34                  2.098  1.623  -25.7%

partialsort!(rand(UInt8, 71092), k)
k = 30602:31046         1.354  1.232  -09.5%
k = 35546:35547         1.322  1.093  -19.0%
k = 1                   1.035  0.591  -56.1%
k = 12198               0.760  1.087  +35.8%

partialsort!(rand(UInt8, 176199), k)
k = 175743:176199       1.175  0.764  -43.0%
k = 1                   0.893  0.715  -22.2%
k = 88100               1.397  1.144  -20.0%
k = 176199              1.095  0.679  -47.8%

partialsort!(rand(UInt8, 2649352), k)
k = 1:891915            6.003  5.336  -11.8%
k = 1324676:1324677     1.025  0.947  -07.9%
k = 1                   1.385  0.513  -99.3%
k = 687451              1.737  0.996  -55.6%
k = 2648225             0.821  0.588  -33.4%

partialsort!(rand(UInt8, 19380151), k)
k = 1:1974500           2.606  2.099  -21.7%
k = 1                   1.116  0.524  -75.5%
k = 9690076             1.192  0.988  -18.8%
k = 19315724            1.120  0.649  -54.5%

partialsort(rand(Float32, 42), k)
k = 11:12               4.554  5.018  +09.7%
k = 16:27               5.871  6.051  +03.0%
k = 11                  4.490  4.004  -11.4%
k = 42                  4.485  4.196  -06.7%

partialsort(rand(Float32, 49), k)
k = 1:10                6.086  6.430  +05.5%
k = 1:15                7.389  7.287  -01.4%
k = 1                   3.738  3.205  -15.4%
k = 49                  3.482  3.819  +09.2%

partialsort(rand(Float32, 254), k)
k = 3:34                2.681  2.688  +00.2%
k = 127:128             2.868  2.926  +02.0%
k = 252:254             2.447  2.368  -03.3%
k = 118                 3.095  2.594  -17.7%
k = 254                 2.425  2.305  -05.1%

partialsort(rand(Float32, 311), k)
k = 138:176             4.383  4.606  +05.0%
k = 169:170             3.635  3.859  +06.0%
k = 1                   2.230  2.321  +04.0%
k = 156                 3.436  2.606  -27.6%

partialsort(rand(Float32, 849), k)
k = 1:557               7.435  7.485  +00.7%
k = 448:453             3.542  3.137  -12.2%
k = 483:490             3.556  3.154  -12.0%
k = 1                   1.910  1.669  -13.5%
k = 848                 3.052  1.811  -52.2%

partialsort(rand(Float32, 1179), k)
k = 1:3                 2.774  1.820  -42.2%
k = 270:472             5.442  5.478  +00.6%
k = 837:921             4.011  3.787  -05.7%
k = 1                   2.796  1.639  -53.4%
k = 1179                1.427  1.732  +19.4%

partialsort(rand(Float32, 266404), k)
k = 133202:133203       1.951  1.660  -16.2%
k = 263867:266404       1.672  1.237  -30.1%
k = 266216:266404       1.522  1.064  -35.8%
k = 100179              1.931  1.597  -19.0%

partialsort(rand(Float32, 1769209), k)
k = 88264:771345       10.340 10.380  +00.4%
k = 999524:1000381      1.936  1.630  -17.2%
k = 1768923:1769209     2.104  1.172  -58.5%
k = 3977                2.541  1.184  -76.3%
k = 1769209             2.442  1.344  -59.7%

partialsort(rand(Float64, 4), k)
k = 2:2                 6.842  6.901  +00.8%
k = 2:3                 6.929  6.882  -00.7%
k = 4:4                 6.871  6.813  -00.9%
k = 4                   7.192  7.616  +05.7%

partialsort(rand(Float64, 68), k)
k = 34:35               4.421  4.532  +02.5%
k = 1                   4.318  3.389  -24.2%
k = 10                  4.897  3.767  -26.2%
k = 68                  3.116  3.126  +00.3%

partialsort(rand(Float64, 163), k)
k = 1:1                 3.242  3.229  -00.4%
k = 1                   3.301  2.426  -30.8%
k = 2                   3.247  2.624  -21.3%
k = 97                  3.565  2.399  -39.6%

partialsort(rand(Float64, 302), k)
k = 1:243               8.623  8.830  +02.4%
k = 299:302             3.721  3.819  +02.6%
k = 136                 3.372  3.346  -00.8%
k = 302                 3.173  2.255  -34.2%

partialsort(rand(Float64, 2613), k)
k = 1:26                2.280  1.680  -30.6%
k = 1388:1403           2.812  2.971  +05.5%
k = 1307                2.764  2.530  -08.8%
k = 2613                1.674  1.589  -05.2%

partialsort(rand(Float64, 2900), k)
k = 1:485               3.290  3.233  -01.8%
k = 1:1370              6.710  6.724  +00.2%
k = 1450:1451           3.099  2.711  -13.4%
k = 76                  2.328  1.624  -36.0%
k = 361                 2.543  2.658  +04.4%

partialsort(rand(Float64, 6168), k)
k = 1:2218              5.479  7.370  +29.7%
k = 1318:4528           7.512  7.397  -01.5%
k = 1                   2.304  1.452  -46.1%
k = 22                  2.364  1.520  -44.2%
k = 6168                1.912  1.466  -26.6%

partialsort(rand(Float64, 7112), k)
k = 3374:3381           3.609  2.443  -39.0%
k = 3556:3557           3.498  2.478  -34.5%
k = 26                  1.763  1.476  -17.8%
k = 57                  1.781  1.476  -18.8%
k = 4544                3.146  2.367  -28.5%

partialsort(rand(Float64, 8773), k)
k = 4378:4433           2.033  2.617  +25.3%
k = 8698:8773           1.981  1.544  -24.9%
k = 49                  2.185  1.525  -36.0%
k = 4387                1.990  2.299  +14.4%
k = 7094                1.919  2.370  +21.1%

partialsort(rand(Float64, 27548), k)
k = 13774:13775         2.314  2.239  -03.3%
k = 27448:27548         1.762  1.376  -24.7%
k = 1980                2.090  2.118  +01.3%
k = 3915                2.010  2.119  +05.3%
k = 27548               1.762  1.326  -28.4%

partialsort(rand(Float64, 28179), k)
k = 1378:13101          7.921  7.921  +00.0%
k = 1                   2.313  1.418  -48.9%
k = 103                 2.333  1.436  -48.6%
k = 14090               2.007  2.091  +04.1%
k = 14103               1.987  2.106  +05.8%

partialsort(rand(Float64, 28808), k)
k = 1:9188              6.765  9.377  +32.7%
k = 14404:14405         2.595  2.259  -13.8%
k = 27501:27573         3.074  2.206  -33.2%
k = 11                  1.833  1.429  -24.9%
k = 28636               2.952  1.374  -76.5%

partialsort(rand(Float64, 29910), k)
k = 1:172               1.758  1.422  -21.2%
k = 185                 1.723  1.385  -21.9%
k = 273                 1.705  1.452  -16.1%
k = 29703               2.045  1.358  -40.9%
k = 29908               2.030  1.332  -42.1%

partialsort(rand(Float64, 430947), k)
k = 1:23609             3.045  2.725  -11.1%
k = 426654:430947       3.341  1.433  -84.6%
k = 172991              2.829  2.067  -31.4%
k = 236437              3.067  2.085  -38.6%
k = 430947              3.225  1.293  -91.4%

partialsort(rand(Float64, 1731857), k)
k = 213972              2.909  2.211  -27.4%
k = 859089              3.807  2.306  -50.1%
k = 1349429             3.775  2.291  -49.9%
k = 1731857             2.390  1.512  -45.8%

partialsort(rand(Int128, 6239), k)
k = 1:22                2.371  1.830  -25.9%
k = 1:987               4.294  4.314  +00.5%
k = 1:4122             14.078 14.011  -00.5%
k = 1:5968             18.887 18.907  +00.1%
k = 5466                2.638  2.772  +04.9%

partialsort(rand(Int128, 43356), k)
k = 1                   3.038  1.501  -70.5%
k = 42975               1.942  2.118  +08.7%
k = 42989               2.802  1.511  -61.8%
k = 43356               2.035  1.676  -19.4%

partialsort(rand(Int128, 47581), k)
k = 1:183               3.200  1.691  -63.8%
k = 1:937               3.447  1.913  -58.9%
k = 310:37621          24.287 24.679  +01.6%
k = 47178:47581         1.999  1.617  -21.2%
k = 15970               7.190  2.077  -124.2%

partialsort(rand(Int128, 141841), k)
k = 1:912               3.196  1.670  -64.9%
k = 141227:141841       2.076  1.649  -23.1%
k = 141394:141841       2.042  1.619  -23.2%
k = 1                   3.131  1.547  -70.5%
k = 132529              2.037  2.295  +11.9%

partialsort(rand(Int128, 234381), k)
k = 1:466               2.032  1.561  -26.4%
k = 10909:144086       19.149 19.170  +00.1%
k = 1                   2.575  1.569  -49.5%
k = 232422              3.253  1.691  -65.4%
k = 233023              3.240  1.673  -66.1%

partialsort(rand(Int128, 3699957), k)
k = 1:2480865          30.849 29.229  -05.4%
k = 1:2648196          32.474 34.413  +05.8%
k = 1                   3.860  2.257  -53.7%
k = 3684223             5.563  2.272  -89.6%

partialsort(rand(Int128, 10134995), k)
k = 1:7311              9.158  3.000  -111.6%
k = 4762326:4798250    28.317  3.188  -218.4%
k = 7208105:8030984    25.721  7.617  -121.7%
k = 5067498            23.809 12.697  -62.9%
k = 10112027           19.700  2.532  -205.2%

partialsort(rand(Int64, 3), k)
k = 1:2                 7.373  8.617  +15.6%
k = 3:3                 8.218  8.651  +05.1%
k = 1                   8.507  8.453  -00.6%

partialsort(rand(Int64, 14), k)
k = 7:8                 5.149  5.387  +04.5%
k = 1                   4.815  4.278  -11.8%
k = 5                   4.965  4.328  -13.7%
k = 14                  4.915  4.386  -11.4%

partialsort(rand(Int64, 72), k)
k = 1:40                4.885  5.033  +03.0%
k = 1:42                5.858  5.519  -06.0%
k = 60:60               2.798  2.788  -00.3%
k = 1                   3.305  2.047  -47.9%
k = 72                  1.908  1.105  -54.7%

partialsort(rand(Int64, 59667), k)
k = 1:90                2.793  1.070  -96.0%
k = 1                   2.624  0.960  -100.5%
k = 29834               2.084  1.325  -45.3%
k = 59667               1.088  1.187  +08.7%

partialsort(rand(Int64, 687637), k)
k = 1                   2.000  0.985  -70.8%
k = 6872                1.961  1.075  -60.1%
k = 343819              1.909  1.400  -31.0%
k = 681048              1.241  1.175  -05.5%

partialsort(rand(Int64, 1317944), k)
k = 1:50669             3.349  1.933  -54.9%
k = 1:338036            8.383  6.749  -21.7%
k = 9103:899321        16.587 16.428  -01.0%
k = 1247468             1.823  1.591  -13.6%
k = 1317944             1.857  1.213  -42.6%

partialsort(rand(Int64, 2385485), k)
k = 1:1911              1.883  1.206  -44.6%
k = 1:711890           11.192  7.965  -34.0%
k = 765906:1233854      7.826  6.471  -19.0%
k = 8768                1.921  1.288  -40.0%
k = 1192743             3.347  1.670  -69.5%

partialsort(rand(String, 1), k)
k = 1                  19.112 19.116  +00.0%

partialsort(rand(String, 2), k)
k = 1:1                13.011 13.236  +01.7%
k = 1:2                13.416 13.082  -02.5%
k = 1                  12.935 13.161  +01.7%
k = 2                  13.462 13.236  -01.7%

partialsort(rand(String, 76), k)
k = 1:1                12.797 12.913  +00.9%
k = 1:69               24.877 24.877  +00.0%
k = 38:39              12.985 12.941  -00.3%
k = 1                  12.841  8.137  -45.6%

partialsort(rand(String, 177), k)
k = 1:97               19.020 19.114  +00.5%
k = 77                  7.837  9.873  +23.1%
k = 89                  9.402 10.330  +09.4%
k = 177                10.538  6.550  -47.5%

partialsort(rand(String, 631), k)
k = 1:520              43.713 44.043  +00.8%
k = 1                  10.341  7.871  -27.3%
k = 147                12.110 13.471  +10.6%
k = 316                17.895 12.507  -35.8%
k = 631                 4.807  7.356  +42.5%

partialsort(rand(String, 21232), k)
k = 10616:10617        17.287 13.144  -27.4%
k = 12562:12573        16.182 12.322  -27.3%
k = 21094:21232         6.403  6.142  -04.2%
k = 204                17.234  5.675  -111.1%
k = 17497              10.780  9.914  -08.4%

partialsort(rand(String, 1168148), k)
k = 477727:492176      23.706 15.129  -44.9%
k = 40                 20.416  6.508  -114.3%
k = 6153               18.556  5.953  -113.7%
k = 7714               18.317  5.861  -114.0%
k = 1168148             5.151  5.650  +09.3%

partialsort(rand(String, 1671468), k)
k = 587216:650171      26.276 18.482  -35.2%
k = 892834:1098407     53.598 38.543  -33.0%
k = 16234              11.151  6.411  -55.4%
k = 1660847            17.204  6.086  -103.9%
k = 1671468            16.548  5.551  -109.2%

partialsort(rand(UInt8, 1), k)
k = 1:1                21.636 23.147  +06.8%
k = 1                  19.573 20.627  +05.2%

partialsort(rand(UInt8, 21), k)
k = 1:1                 4.375  4.634  +05.7%
k = 1                   4.584  4.048  -12.4%
k = 11                  4.652  3.666  -23.8%

partialsort(rand(UInt8, 160), k)
k = 1:1                 1.486  1.504  +01.2%
k = 1:2                 1.504  1.580  +05.0%
k = 63:64               1.895  1.917  +01.2%
k = 80:81               2.254  2.254  -00.0%
k = 1                   1.675  1.284  -26.6%

partialsort(rand(UInt8, 300967), k)
k = 66275:68273         1.603  1.134  -34.6%
k = 231768:287281       4.326  3.218  -29.6%
k = 150484              1.936  1.020  -64.1%
k = 242115              2.252  1.193  -63.6%
k = 300967              2.037  0.735  -101.9%

partialsort!(rand(String, 22), 22) regressed 70.5%* and partialsort(rand(String, 631), 631) regressed 42.5%* because of differences in performance characteristics between PartialQuickSort and ScratchQuickSort. Using many random seeds results in +regression -improvment of [-180, -79, -35, 24, 108] in measured in %* for quantiles 0:.25:1. So, an improvement typically but sometimes a regression for the length 22 one, and [-65, -29, -6, 19, 60] for the length 631 one. The signposting never fails in these cases, so that variance is coming from something else.

The noise comes from both ScratchQuickSort and PartialQuickSort. Maybe worth looking into, but niether a merge blocking regression nor particularly closely related to this PR more closely related to #45222.

Edit: I looked into it a bit. I think that for this random seed, ScratchQuickSort gets lucky and can find the target very quickly. It is very hard to compete with ScratchQuickSort on a partialsort when ScratchQuickSort gets lucky because it may randomly pick great signposts. I'm not worried about regressions on a specific random seed when there are larger improvements on that same random seed and, I suspect, an overall decrease in noise.

There were 4 regressions in the 30% range that look to be from the same cause. I did not investigate smaller regressions.

@@ -91,7 +91,15 @@ issorted(itr;
issorted(itr, ord(lt,by,rev,order))

function partialsort!(v::AbstractVector, k::Union{Integer,OrdinalRange}, o::Ordering)
_sort!(v, InitialOptimizations(ScratchQuickSort(k)), o, (;))
# TODO move k from `alg` to `kw`
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a TODO that predates this PR, I'm just adding a note because this PR touches target range handling and reminded me that this is not the best approach.

get_next::F
end

# TODO: this composition between BracketedSort and ScratchQuickSort does not bring me joy
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be avoided via moving k from alg to kw.

base/sort.jl Outdated Show resolved Hide resolved
There are some pretty compelling reasons to use PartialQuickSort instead of ScratchQuickSort as a base case for BracketedSort.
However, PartialQuickSort can segfault for invalid but seemingly innocuous lt functions.
I don't want to talk about introducing segfaults to the default sorting algorithms or introducing a regression to PartialQuickSort in this PR
Revert this commit it you want `partialsort!(::Vector{Int}, ::Int)` not to allocate.
@LilithHafner
Copy link
Member Author

Here's my latest benchmark results
partialsort!(rand(Float32, 3), k)
k = 1:1                15.749 14.647  -07.3%
k = 2:3                15.717 12.994  -19.0%
k = 2                  14.033 13.239  -05.8%
k = 3                  11.541 11.306  -02.1%

partialsort!(rand(Float32, 120), k)
k = 1:1                 2.275  2.348  +03.2%
k = 1:68                6.929  7.186  +03.6%
k = 120:120             3.303  3.381  +02.3%

partialsort!(rand(Float32, 356), k)
k = 1:1                 2.163  2.605  +18.6%
k = 178:179             2.844  3.108  +08.9%
k = 222:225             2.935  3.205  +08.8%
k = 1                   2.202  2.636  +18.0%
k = 356                 2.584  2.614  +01.2%

partialsort!(rand(Float32, 764), k)
k = 1:612               7.144  7.226  +01.1%
k = 69:736              7.526  7.690  +02.2%
k = 3                   2.272  2.045  -10.5%

partialsort!(rand(Float32, 1341), k)
k = 1:7                 3.773  1.935  -66.8%
k = 414:418             3.897  2.592  -40.8%
k = 683:1321            5.313  5.406  +01.7%
k = 1331                1.767  1.869  +05.6%
k = 1341                1.718  1.758  +02.3%

partialsort!(rand(Float32, 2417), k)
k = 2315:2339           1.827  1.741  -04.8%
k = 1                   2.306  1.625  -35.0%
k = 25                  2.090  1.767  -16.8%
k = 1209                2.224  2.547  +13.6%

partialsort!(rand(Float32, 1412894), k)
k = 706447:706448       2.412  1.411  -53.6%
k = 781870:784710       2.474  1.444  -53.9%
k = 849750:1250424      8.233  7.825  -05.1%
k = 265276              1.910  1.412  -30.2%
k = 1410938             1.753  0.891  -67.6%

partialsort!(rand(Float32, 3469856), k)
k = 1734928:1734929     2.825  1.442  -67.2%
k = 3441966:3469856     1.866  1.056  -56.9%
k = 2650041             1.608  1.500  -06.9%
k = 3461652             1.927  0.953  -70.4%
k = 3469697             1.930  1.002  -65.6%

partialsort!(rand(Float64, 3), k)
k = 1                   9.789  9.898  +01.1%
k = 2                  10.873  9.572  -12.7%
k = 3                  10.096 10.511  +04.0%

partialsort!(rand(Float64, 34), k)
k = 12:12               5.884  6.360  +07.8%
k = 1                   5.571  6.230  +11.2%
k = 5                   5.890  6.333  +07.2%
k = 32                  5.215  5.129  -01.7%

partialsort!(rand(Float64, 6994), k)
k = 1:843               3.503  2.568  -31.1%
k = 1:2826              6.833  6.833  -00.0%
k = 61                  2.687  1.638  -49.5%
k = 6707                1.603  1.650  +02.9%
k = 6994                1.573  1.537  -02.3%

partialsort!(rand(Float64, 1046517), k)
k = 161433:167648       2.575  1.769  -37.5%
k = 500996:870065       9.698  9.566  -01.4%
k = 6515                2.198  1.002  -78.5%
k = 351940              2.511  1.660  -41.4%
k = 1037448             2.434  0.993  -89.6%

partialsort!(rand(Float64, 1051336), k)
k = 1:4416              1.924  1.009  -64.5%
k = 455127:1037517     14.672 14.707  +00.2%
k = 1042624:1051336     4.510  1.128  -138.6%
k = 1                   1.924  0.940  -71.7%
k = 913974              4.510  1.650  -100.6%

partialsort!(rand(Float64, 3029236), k)
k = 1514618:1514619     3.073  1.691  -59.8%
k = 2999085:3029236     2.253  1.194  -63.5%
k = 160737              3.059  1.707  -58.3%
k = 3001131             2.335  1.019  -82.9%
k = 3029236             2.056  0.977  -74.4%

partialsort!(rand(Float64, 6212361), k)
k = 3649468:5431358    11.010  8.654  -24.1%
k = 6168786:6212361     2.522  1.140  -79.4%
k = 1                   2.250  0.917  -89.8%
k = 6212361             2.463  1.017  -88.5%

partialsort!(rand(Int128, 56), k)
k = 1:1                 7.490  5.655  -28.1%
k = 56:56               4.196  2.718  -43.4%
k = 16                  5.625  5.685  +01.1%
k = 50                  2.778  2.817  +01.4%
k = 56                  2.728  4.236  +44.0%

partialsort!(rand(Int128, 84), k)
k = 42:43               4.731  4.750  +00.4%
k = 62:62               3.650  3.931  +07.4%
k = 1                   3.893  3.884  -00.2%
k = 84                  2.602  2.695  +03.5%

partialsort!(rand(Int128, 271), k)
k = 1:2                 2.560  2.840  +10.4%
k = 108:110             3.093  3.536  +13.4%
k = 245:246             3.789  4.151  +09.1%
k = 271:271             4.368  4.513  +03.3%
k = 3                   2.560  2.849  +10.7%

partialsort!(rand(Int128, 4647), k)
k = 1                   4.259  1.865  -82.6%
k = 2270                4.483  2.788  -47.5%
k = 3279                4.116  2.878  -35.8%
k = 4644                1.748  1.919  +09.3%
k = 4647                1.766  2.071  +15.9%

partialsort!(rand(Int128, 6493), k)
k = 1:24                3.157  2.124  -39.6%
k = 1161:3576           9.286  9.786  +05.3%
k = 1722                2.939  3.087  +04.9%
k = 6493                2.041  2.073  +01.6%

partialsort!(rand(Int128, 11572), k)
k = 370:9931           19.890 19.569  -01.6%
k = 5786:5787           2.646  3.115  +16.3%
k = 11527:11572         2.654  2.391  -10.4%
k = 1602                2.863  2.510  -13.2%
k = 11528               3.302  2.279  -37.1%

partialsort!(rand(Int128, 24908), k)
k = 12454:12455         4.794  3.026  -46.0%
k = 21710               3.632  2.446  -39.5%
k = 24908               1.559  1.618  +03.7%

partialsort!(rand(Int128, 3802413), k)
k = 1:24959             4.602  1.203  -134.2%
k = 1:3621317          36.712 41.577  +12.4%
k = 1815638:1841498     3.802  1.852  -71.9%
k = 3789561:3802413     3.530  1.173  -110.2%
k = 3802413             3.530  1.066  -119.7%

partialsort!(rand(Int128, 5462941), k)
k = 1:21462             6.329  1.255  -161.8%
k = 1:5099526          38.144 35.609  -06.9%
k = 16598               8.874  1.401  -184.6%
k = 2731471             4.520  1.618  -102.8%
k = 5462941             4.417  1.100  -139.1%

partialsort!(rand(Int128, 22252660), k)
k = 1:90642            11.833  1.112  -236.5%
k = 1:3951357           9.932 14.094  +35.0%
k = 3032737:3090032     4.046  1.684  -87.7%
k = 1                   3.844  1.037  -131.0%
k = 22187610            3.947  1.060  -131.5%

partialsort!(rand(Int64, 4), k)
k = 1:1                 8.710  9.644  +10.2%
k = 4:4                 9.250 10.359  +11.3%
k = 4                   8.724  9.104  +04.3%

partialsort!(rand(Int64, 247), k)
k = 1                   2.468  1.594  -43.7%
k = 2                   1.651  2.313  +33.7%
k = 3                   1.589  1.567  -01.4%
k = 124                 4.102  3.316  -21.3%

partialsort!(rand(Int64, 885), k)
k = 96:381              5.414  5.775  +06.5%
k = 566:570             4.092  2.512  -48.8%
k = 878:885             1.572  1.836  +15.5%
k = 1                   1.340  1.776  +28.2%
k = 2                   1.395  1.858  +28.6%

partialsort!(rand(Int64, 9113), k)
k = 1:6373             10.882 11.549  +06.0%
k = 9081                1.321  1.523  +14.2%
k = 9103                1.298  1.166  -10.8%
k = 9113                1.280  1.148  -10.9%

partialsort!(rand(Int64, 66063), k)
k = 1:300               1.929  1.290  -40.2%
k = 36275:53130         5.729  6.246  +08.6%
k = 1                   1.735  0.974  -57.7%
k = 28981               2.407  1.425  -52.5%
k = 33032               1.861  1.345  -32.5%

partialsort!(rand(Int64, 80969), k)
k = 1:17084             5.965  4.637  -25.2%
k = 23302:70376        12.578 12.708  +01.0%
k = 33166:33845         2.938  1.705  -54.4%
k = 40                  1.922  0.963  -69.0%
k = 526                 1.916  0.989  -66.2%

partialsort!(rand(Int64, 170571), k)
k = 1:1038              1.541  1.067  -36.7%
k = 1:1629              1.200  1.053  -13.0%
k = 1:117683           14.285 14.036  -01.8%
k = 100458:145299       7.805  6.447  -19.1%
k = 85286               1.839  1.542  -17.6%

partialsort!(rand(Int64, 864519), k)
k = 2399:826475        20.802 20.863  +00.3%
k = 13365:242859        7.262  6.709  -07.9%
k = 203249:209420       3.011  1.614  -62.4%
k = 246445:252872       2.744  1.708  -47.4%
k = 432260              1.844  1.539  -18.1%

partialsort!(rand(String, 2), k)
k = 1:2                16.166 16.082  -00.5%
k = 1                  16.170 16.170  +00.0%
k = 2                  15.644 16.096  +02.8%

partialsort!(rand(String, 13), k)
k = 1:1                12.999 14.000  +07.4%
k = 1:12               13.074 12.821  -02.0%
k = 13:13              12.998 13.710  +05.3%
k = 1                  13.332 12.932  -03.1%
k = 13                 12.865 13.666  +06.0%

partialsort!(rand(String, 22), k)
k = 1:1                22.272 22.345  +00.3%
k = 11:12              22.254 23.049  +03.5%
k = 16:16              21.708 22.600  +04.0%
k = 22                  7.758  7.503  -03.3%

partialsort!(rand(String, 44), k)
k = 1:30               22.727 23.719  +04.3%
k = 34:34               8.305  8.367  +00.7%
k = 1                   9.004  8.414  -06.8%
k = 39                  8.336  8.336  +00.0%

partialsort!(rand(String, 138), k)
k = 1:1                15.673 15.783  +00.7%
k = 1:2                15.179 15.042  -00.9%
k = 1:111              28.154 28.230  +00.3%
k = 8:8                15.001 15.000  -00.0%
k = 1                  15.659 14.946  -04.7%

partialsort!(rand(String, 1159), k)
k = 1:8                 9.113  7.262  -22.7%
k = 1:10               10.066  7.496  -29.5%
k = 907:916            12.205 11.720  -04.1%
k = 10                  9.887  7.190  -31.8%
k = 890                12.313 12.331  +00.1%

partialsort!(rand(String, 1778), k)
k = 889:890            15.350 12.163  -23.3%
k = 1                   8.319  6.421  -25.9%
k = 4                   8.085  6.398  -23.4%
k = 1442                7.710 10.756  +33.3%
k = 1765                8.015  6.632  -18.9%

partialsort!(rand(String, 6996), k)
k = 1:9                 8.237  6.313  -26.6%
k = 480:500             9.571  9.839  +02.8%
k = 1143:6613          86.323 86.496  +00.2%
k = 2800:3871          31.167 28.254  -09.8%
k = 6996               10.506  5.551  -63.8%

partialsort!(rand(String, 9178), k)
k = 4589:4590          18.922 11.931  -46.1%
k = 7108:7329          18.414 12.539  -38.4%
k = 9112:9178          14.437  5.897  -89.5%
k = 1                   6.360  5.806  -09.1%
k = 9178               14.478  5.593  -95.1%

partialsort!(rand(String, 41621), k)
k = 19610:19752        18.755 12.312  -42.1%
k = 339                13.448  5.687  -86.1%
k = 20811              18.243 11.670  -44.7%
k = 29484              14.020 11.365  -21.0%
k = 41323              14.492  5.243  -101.7%

partialsort!(rand(String, 145977), k)
k = 135563:136325       7.923  9.966  +22.9%
k = 1088               11.517  5.656  -71.1%
k = 29000              15.786  9.875  -46.9%
k = 145913              5.930  4.992  -17.2%
k = 145977              5.905  4.707  -22.7%

partialsort!(rand(String, 6803515), k)
k = 1:3811305          165.338 176.441  +06.5%
k = 2206222:2240415    20.491 10.289  -68.9%
k = 6388849:6420065    31.945  9.184  -124.7%
k = 1581624            18.836  8.710  -77.1%
k = 1865233            17.371  8.955  -66.3%

partialsort!(rand(UInt8, 21), k)
k = 1:2                 5.520  5.711  +03.4%
k = 1                   5.724  5.687  -00.6%
k = 11                  5.755  5.651  -01.8%
k = 21                  5.176  5.577  +07.5%

partialsort!(rand(UInt8, 106), k)
k = 1:2                 1.628  1.600  -01.7%
k = 1:16                2.701  2.745  +01.6%
k = 1:77                5.235  5.217  -00.3%
k = 82:83               2.964  3.034  +02.3%
k = 34                  2.363  2.125  -10.6%

partialsort!(rand(UInt8, 71092), k)
k = 30602:31046         1.330  1.189  -11.2%
k = 35546:35547         1.202  1.119  -07.1%
k = 1                   0.923  0.642  -36.2%
k = 12198               0.886  1.135  +24.8%

partialsort!(rand(UInt8, 176199), k)
k = 175743:176199       1.147  0.727  -45.7%
k = 1                   0.894  0.580  -43.3%
k = 88100               1.459  1.101  -28.2%
k = 176199              1.047  0.678  -43.4%

partialsort!(rand(UInt8, 2649352), k)
k = 1:891915            5.941  5.628  -05.4%
k = 1324676:1324677     1.002  0.955  -04.8%
k = 1                   1.510  0.519  -106.8%
k = 687451              1.754  0.939  -62.5%
k = 2648225             0.922  0.656  -34.0%

partialsort!(rand(UInt8, 19380151), k)
k = 1:1974500           2.740  2.101  -26.5%
k = 1                   0.963  0.536  -58.6%
k = 9690076             1.255  0.951  -27.8%
k = 19315724            1.030  0.627  -49.7%

partialsort(rand(Float32, 42), k)
k = 11:12               4.892  4.918  +00.5%
k = 16:27               6.633  6.452  -02.8%
k = 11                  5.022  5.162  +02.7%
k = 42                  4.568  5.007  +09.2%

partialsort(rand(Float32, 49), k)
k = 1:10                6.325  6.852  +08.0%
k = 1:15                7.407  7.611  +02.7%
k = 1                   3.829  4.091  +06.6%
k = 49                  3.593  3.547  -01.3%

partialsort(rand(Float32, 254), k)
k = 3:34                2.940  2.662  -09.9%
k = 127:128             3.121  2.842  -09.4%
k = 252:254             2.421  2.407  -00.6%
k = 118                 2.948  3.334  +12.3%
k = 254                 2.461  2.496  +01.4%

partialsort(rand(Float32, 311), k)
k = 138:176             4.223  4.906  +15.0%
k = 169:170             3.548  3.920  +10.0%
k = 1                   2.165  2.641  +19.8%
k = 156                 3.470  3.738  +07.4%

partialsort(rand(Float32, 849), k)
k = 1:557               7.190  7.534  +04.7%
k = 448:453             3.377  3.003  -11.8%
k = 483:490             3.489  3.150  -10.2%
k = 1                   1.905  1.851  -02.9%
k = 848                 3.137  1.954  -47.3%

partialsort(rand(Float32, 1179), k)
k = 1:3                 2.801  1.957  -35.8%
k = 270:472             5.434  5.619  +03.4%
k = 837:921             4.100  3.446  -17.4%
k = 1                   2.827  1.833  -43.3%
k = 1179                1.409  1.789  +23.9%

partialsort(rand(Float32, 266404), k)
k = 133202:133203       2.336  1.852  -23.2%
k = 263867:266404       1.662  1.252  -28.3%
k = 266216:266404       1.547  1.105  -33.6%
k = 100179              2.162  1.845  -15.9%

partialsort(rand(Float32, 1769209), k)
k = 88264:771345       10.366 10.440  +00.7%
k = 999524:1000381      2.186  1.703  -25.0%
k = 1768923:1769209     2.219  1.081  -71.9%
k = 3977                2.547  1.127  -81.5%
k = 1769209             2.239  1.136  -67.8%

partialsort(rand(Float64, 4), k)
k = 2:2                 7.397  7.178  -03.0%
k = 2:3                 7.852  8.133  +03.5%
k = 4:4                 8.389  8.068  -03.9%
k = 4                   8.652  7.834  -09.9%

partialsort(rand(Float64, 68), k)
k = 34:35               4.600  4.654  +01.2%
k = 1                   4.419  4.471  +01.2%
k = 10                  4.725  4.873  +03.1%
k = 68                  3.068  3.365  +09.2%

partialsort(rand(Float64, 163), k)
k = 1:1                 3.211  3.238  +00.8%
k = 1                   3.139  3.287  +04.6%
k = 2                   3.202  3.310  +03.3%
k = 97                  3.713  3.677  -01.0%

partialsort(rand(Float64, 302), k)
k = 1:243               7.933  8.324  +04.8%
k = 299:302             3.440  3.850  +11.2%
k = 136                 3.311  3.716  +11.5%
k = 302                 3.195  3.380  +05.6%

partialsort(rand(Float64, 2613), k)
k = 1:26                2.264  1.775  -24.3%
k = 1388:1403           2.775  2.721  -01.9%
k = 1307                2.828  2.605  -08.2%
k = 2613                1.531  1.680  +09.3%

partialsort(rand(Float64, 2900), k)
k = 1:485               3.333  3.491  +04.6%
k = 1:1370              6.293  6.739  +06.8%
k = 1450:1451           3.103  2.792  -10.6%
k = 76                  2.404  1.743  -32.1%
k = 361                 2.371  2.763  +15.3%

partialsort(rand(Float64, 6168), k)
k = 1:2218              6.168  8.424  +31.2%
k = 1318:4528           7.748  7.958  +02.7%
k = 1                   2.675  1.655  -48.0%
k = 22                  2.432  1.581  -43.1%
k = 6168                2.054  1.506  -31.0%

partialsort(rand(Float64, 7112), k)
k = 3374:3381           3.650  2.390  -42.3%
k = 3556:3557           3.504  2.490  -34.2%
k = 26                  1.752  1.594  -09.5%
k = 57                  1.828  1.570  -15.2%
k = 4544                3.105  2.297  -30.2%

partialsort(rand(Float64, 8773), k)
k = 4378:4433           2.061  2.546  +21.1%
k = 8698:8773           2.042  1.833  -10.8%
k = 49                  2.147  1.653  -26.2%
k = 4387                2.047  2.584  +23.3%
k = 7094                1.933  2.579  +28.8%

partialsort(rand(Float64, 27548), k)
k = 13774:13775         2.326  2.488  +06.7%
k = 27448:27548         2.296  1.401  -49.4%
k = 1980                2.111  2.432  +14.1%
k = 3915                2.031  2.264  +10.9%
k = 27548               1.786  1.411  -23.6%

partialsort(rand(Float64, 28179), k)
k = 1378:13101          7.874  7.889  +00.2%
k = 1                   2.342  1.496  -44.8%
k = 103                 2.431  1.591  -42.4%
k = 14090               1.975  2.236  +12.4%
k = 14103               2.011  2.147  +06.5%

partialsort(rand(Float64, 28808), k)
k = 1:9188              6.620  9.592  +37.1%
k = 14404:14405         2.638  2.180  -19.1%
k = 27501:27573         3.034  2.136  -35.1%
k = 11                  1.844  1.563  -16.5%
k = 28636               2.985  1.338  -80.3%

partialsort(rand(Float64, 29910), k)
k = 1:172               1.783  1.555  -13.7%
k = 185                 1.747  1.505  -14.9%
k = 273                 1.740  1.546  -11.8%
k = 29703               2.014  1.376  -38.1%
k = 29908               2.038  1.371  -39.7%

partialsort(rand(Float64, 430947), k)
k = 1:23609             2.734  2.695  -01.4%
k = 426654:430947       3.867  1.485  -95.7%
k = 172991              2.905  2.033  -35.7%
k = 236437              3.479  2.018  -54.4%
k = 430947              3.222  1.332  -88.3%

partialsort(rand(Float64, 1731857), k)
k = 213972              2.864  2.216  -25.7%
k = 859089              3.830  2.183  -56.2%
k = 1349429             3.796  2.192  -54.9%
k = 1731857             2.352  1.446  -48.6%

partialsort(rand(Int128, 6239), k)
k = 1:22                2.397  2.044  -16.0%
k = 1:987               4.308  4.361  +01.2%
k = 1:4122             14.532 12.662  -13.8%
k = 1:5968             17.912 17.992  +00.4%
k = 5466                2.558  2.565  +00.3%

partialsort(rand(Int128, 43356), k)
k = 1                   3.235  1.541  -74.2%
k = 42975               2.093  1.652  -23.7%
k = 42989               2.238  1.631  -31.7%
k = 43356               1.995  1.601  -22.0%

partialsort(rand(Int128, 47581), k)
k = 1:183               3.196  1.577  -70.6%
k = 1:937               3.293  1.909  -54.5%
k = 310:37621          24.123 23.471  -02.7%
k = 47178:47581         1.944  1.647  -16.6%
k = 15970               7.137  2.338  -111.6%

partialsort(rand(Int128, 141841), k)
k = 1:912               3.126  1.874  -51.2%
k = 141227:141841       2.062  1.665  -21.4%
k = 141394:141841       2.067  1.682  -20.6%
k = 1                   3.325  1.647  -70.2%
k = 132529              1.999  2.212  +10.1%

partialsort(rand(Int128, 234381), k)
k = 1:466               2.012  1.596  -23.2%
k = 10909:144086       17.633 17.974  +01.9%
k = 1                   2.017  1.554  -26.1%
k = 232422              3.205  1.671  -65.1%
k = 233023              3.158  1.621  -66.7%

partialsort(rand(Int128, 3699957), k)
k = 1:2480865          31.268 29.099  -07.2%
k = 1:2648196          37.210 44.789  +18.5%
k = 1                   7.652  4.192  -60.2%
k = 3684223             5.363  2.154  -91.2%

partialsort(rand(Int128, 10134995), k)
k = 1:7311             12.499  3.882  -116.9%
k = 4762326:4798250    18.055  3.122  -175.5%
k = 7208105:8030984    13.014 20.994  +47.8%
k = 5067498            12.321  2.874  -145.6%
k = 10112027            5.683  2.935  -66.1%

partialsort(rand(Int64, 3), k)
k = 1:2                 8.384  8.403  +00.2%
k = 3:3                 8.434  9.356  +10.4%
k = 1                   9.446  8.778  -07.3%

partialsort(rand(Int64, 14), k)
k = 7:8                 5.528  5.506  -00.4%
k = 1                   6.825  5.373  -23.9%
k = 5                   6.954  6.038  -14.1%
k = 14                  6.918  5.409  -24.6%

partialsort(rand(Int64, 72), k)
k = 1:40                4.899  5.612  +13.6%
k = 1:42                5.589  4.968  -11.8%
k = 60:60               2.855  2.855  +00.0%
k = 1                   3.123  3.133  +00.3%
k = 72                  1.679  1.607  -04.4%

partialsort(rand(Int64, 59667), k)
k = 1:90                2.200  1.018  -77.0%
k = 1                   2.008  1.221  -49.7%
k = 29834               2.039  1.795  -12.7%
k = 59667               1.096  0.959  -13.3%

partialsort(rand(Int64, 687637), k)
k = 1                   1.996  0.974  -71.7%
k = 6872                1.993  1.000  -69.0%
k = 343819              1.872  1.322  -34.8%
k = 681048              1.494  1.068  -33.5%

partialsort(rand(Int64, 1317944), k)
k = 1:50669             3.556  2.061  -54.5%
k = 1:338036            8.978  7.532  -17.6%
k = 9103:899321        21.655 16.751  -25.7%
k = 1247468             1.988  1.622  -20.4%
k = 1317944             2.017  1.274  -46.0%

partialsort(rand(Int64, 2385485), k)
k = 1:1911              2.015  1.261  -46.9%
k = 1:711890           11.731  8.535  -31.8%
k = 765906:1233854      7.810  5.993  -26.5%
k = 8768                2.005  1.261  -46.3%
k = 1192743             3.374  1.662  -70.8%

partialsort(rand(String, 1), k)
k = 1                  24.066 24.205  +00.6%

partialsort(rand(String, 2), k)
k = 1:1                16.170 15.493  -04.3%
k = 1:2                15.668 16.500  +05.2%
k = 1                  15.267 16.094  +05.3%
k = 2                  16.171 16.245  +00.5%

partialsort(rand(String, 76), k)
k = 1:1                12.768 13.677  +06.9%
k = 1:69               24.808 24.260  -02.2%
k = 38:39              13.951 14.024  +00.5%
k = 1                  14.096 13.721  -02.7%

partialsort(rand(String, 177), k)
k = 1:97               18.739 19.162  +02.2%
k = 77                  8.073  7.921  -01.9%
k = 89                 10.067  9.389  -07.0%
k = 177                 9.665 10.538  +08.6%

partialsort(rand(String, 631), k)
k = 1:520              43.250 42.854  -00.9%
k = 1                  10.552  8.571  -20.8%
k = 147                12.058 13.603  +12.1%
k = 316                17.908 14.118  -23.8%
k = 631                 4.873  8.558  +56.3%

partialsort(rand(String, 21232), k)
k = 10616:10617        17.509 12.842  -31.0%
k = 12562:12573        16.347 12.169  -29.5%
k = 21094:21232         7.059  6.682  -05.5%
k = 204                16.706  5.554  -110.1%
k = 17497              10.419  9.959  -04.5%

partialsort(rand(String, 1168148), k)
k = 477727:492176      20.877 23.185  +10.5%
k = 40                 18.609  6.037  -112.6%
k = 6153               17.853  5.894  -110.8%
k = 7714               17.804  6.044  -108.0%
k = 1168148             5.103  5.651  +10.2%

partialsort(rand(String, 1671468), k)
k = 587216:650171      27.566 18.496  -39.9%
k = 892834:1098407     44.234 38.733  -13.3%
k = 16234              10.982  6.334  -55.0%
k = 1660847            18.325  6.219  -108.1%
k = 1671468            18.555  5.910  -114.4%

partialsort(rand(UInt8, 1), k)
k = 1:1                23.561 25.074  +06.2%
k = 1                  23.469 24.386  +03.8%

partialsort(rand(UInt8, 21), k)
k = 1:1                 4.640  4.560  -01.7%
k = 1                   5.484  4.523  -19.3%
k = 11                  5.484  4.400  -22.0%

partialsort(rand(UInt8, 160), k)
k = 1:1                 1.540  1.545  +00.3%
k = 1:2                 1.603  1.562  -02.6%
k = 63:64               1.881  1.899  +01.0%
k = 80:81               2.231  2.267  +01.6%
k = 1                   1.540  1.612  +04.6%

partialsort(rand(UInt8, 300967), k)
k = 66275:68273         1.615  1.135  -35.2%
k = 231768:287281       4.357  3.415  -24.4%
k = 150484              1.948  1.027  -64.0%
k = 242115              2.262  1.033  -78.4%
k = 300967              2.100  0.719  -107.2%

Which are mostly regression-free. The largest reported regression is a 1.76x regression on partialsort(rand(String, 631), 631).

Investigating that regression more closely, it is due to random noise and I expect all the other significant regressions are as well:

Screenshot 2023-11-04 at 12 43 07 PM
Methodology
using BenchmarkTools, Random, Printf
get_length() = round(Int, exp(17*rand()))
get_eltype() = rand([Float64, Float32, Int, UInt8, Int128, String])
get_function() = rand([partialsort!, partialsort])
get_element(x) = rand(x)
get_element(::Type{String}) = randstring()
get_k(n) = rand([
        1,
        n,
        rand(1:n),
        rand(1:1+n÷100),
        rand(n-n÷100:n),

        iseven(n) ? (n÷2:n÷2+1) : (n+1)÷2,

        UnitRange(sort((rand(1:n), rand(1:n)))...),
        1:rand(1:n),
        1:rand(1:1+n÷100),
        rand(n-n÷100:n):n,
        (1:rand(1:1+n÷100)) .+ rand(0:(n-n÷100-1)),
    ])

new!(v, k) = partialsort!(v, k)
new(v, k) = partialsort(v, k)
function old!(v, k)
    Base.Sort._sort!(v, Base.Sort.InitialOptimizations(Base.Sort.ScratchQuickSort(k)), Base.Order.Forward, (;))
    Base.Sort.maybeview(v, k)
end
old(v::AbstractVector, k::Union{Integer,OrdinalRange}; kws...) =
    old!(Base.copymutable(v), k; kws...)

function test(length::Int, eltype, func; zoom=false)
    func === partialsort && (length ÷= 2) # benchmarking is inefficient
    length == 0 && return
    data = [get_element(eltype) for _ in 1:length]
    ks = [get_k(length) for _ in 1:5]
    sort!(ks, by = x->(x isa Number, x))
    unique!(ks)
    println("\n$func(rand($eltype, $length), k)")
    for k in ks
        (t_old, t_new) = func === partialsort! ? test!(similar(data), data, k; zoom) : test(data, k; zoom)
        a = 1e9t_old/length
        b = 1e9t_new/length
        c = log(b/a)*100
        @printf "k = %-18s %6.3f %6.3f  %+05.1f%%\n" k a b c
    end
end

estimate_runtime(data::Vector{T}, k) where T =
    round(Int, (10 + length(data) + length(k) * log(length(k))) * (T === String ? 3 : T == Int128 ? 2 : 1))
function test(data::Vector{T}, k::K; zoom=false) where {T,K}
    estimated_runtime = estimate_runtime(data, k)
    if estimated_runtime > 10^7 || zoom
        t_old = @elapsed old_res = old(data, k)
        t_new = @elapsed new_res = new(data, k)
        old_res == new_res || error("Bad sort")
        t_old, t_new
    elseif estimated_runtime > 10^5
        truth = old(data, k)
        t_old, t_new = Inf, Inf
        for _ in 1:2*10^7 ÷ estimated_runtime
            t1 = @elapsed res1 = old(data, k)
            res1 == truth || error("Bad old sort")
            t_old = min(t_old, t1)

            t2 = @elapsed res2 = new(data, k)
            res2 == truth || error("Bad sort")
            t_new = min(t_new, t2)
        end
        t_old, t_new
    else
        old(data, k) == new(data, k) || error("Bad sort")
        evals = max(1, 10^4 ÷ estimated_runtime)
        samples = 10^7 ÷ evals ÷ estimated_runtime
        t_old = @belapsed old($data, $k) evals=evals samples=samples gctrial=false
        t_new = @belapsed new($data, $k) evals=evals samples=samples gctrial=false
        t_old, t_new
    end
end

function test!(d::Vector{T}, data::Vector{T}, k::K; zoom=false) where {T,K}
    estimated_runtime = estimate_runtime(data, k)
    if estimated_runtime > 10^7 || zoom
        copyto!(d, data)
        t_old = @elapsed old_res = old!(d, k)
        copyto!(d, data)
        t_new = @elapsed new_res = new!(d, k)
        old_res == new_res || error("Bad sort")
    elseif estimated_runtime > 10^6
        truth = old(data, k)
        t_old, t_new = Inf, Inf
        for _ in 1:2*10^7 ÷ estimated_runtime
            copyto!(d, data)
            t1 = @elapsed res1 = old!(d, k)
            res1 == truth || error("Bad old sort")
            t_old = min(t_old, t1)

            copyto!(d, data)
            t2 = @elapsed res2 = new!(d, k)
            res2 == truth || error("Bad sort")
            t_new = min(t_new, t2)
        end
    else
        old!(copyto!(d, data), k) == new!(copyto!(d, data), k) || error("Bad sort")
        evals = max(1, 10^4 ÷ estimated_runtime)
        samples = 10^7 ÷ evals ÷ estimated_runtime
        if evals == 1
            t_old = @belapsed old($d, $k) setup=(copyto!($d, $data)) evals=evals samples=samples gctrial=false
            t_new = @belapsed new($d, $k) setup=(copyto!($d, $data)) evals=evals samples=samples gctrial=false
        else
            t_old = @belapsed old(copyto!($d, $data), $k) evals=evals samples=samples gctrial=false
            t_new = @belapsed new(copyto!($d, $data), $k) evals=evals samples=samples gctrial=false
        end
    end
    t_old, t_new
end

function test(n, seed=1729; zoom=false)
    Random.seed!(seed)
    samples = [(get_length(), get_eltype(), get_function()) for _ in 1:n]
    sort!(samples, by = x->(x[3] === partialsort, string(x[2]), x[1]))
    try
        for s in samples
            test(s...; zoom)
        end
    catch x
        x isa InterruptException || rethrow()
    end
    nothing
end

test(100)

function validate_sample()
    d = [randstring() for _ in 1:631]
    k = 631
    ol = @belapsed old($d, $k) evals=5 samples=100 gctrial=false
    nw = @belapsed new($d, $k) evals=5 samples=100 gctrial=false
    ol*1e9/631, nw*1e9/631
end
function validate(n)
    # data = [validate_sample() for _ in 1:n]
    mx = maximum(maximum.(data))
    bins = LinRange(0, mx, 40)
    histogram(first.(data); bins, label="old", alpha=.7, color=:lightblue)
    histogram!(last.(data); bins, label="new", alpha=.7, color=:pink)
    vline!([mean(first.(data))], label="mean old", color=:blue, linewidth=3)
    vline!([mean(last.(data))], label="mean new", color=:red, linewidth=3)
    vline!([4.873], label="reported old", color=:blue, linewidth=2, linestyle=:dash)
    vline!([8.558], label="reported new", color=:red, linewidth=2, linestyle=:dash)
    xlabel!("Runtime (ns / element)")
    ylabel!("Frequency")
    title!("x = [randstring() for _ in 1:631]\n@elapsed partialsort(x, 631)")
end

display(validate(1000))

At this point I'd say runtime is adequately regression-free; this PR no longer re-introduces PartialQuickSort (since 5c18e25) so there are no fun new segfaults; uses @inbounds sparingly and responsibly; and provides significant performance improvements in several common cases.

The two remaining downsides I see are increased code complexity and instability. Instability is surmountable but not without a non-negligible performance and code complexity penalty. I think it's okay to drop stability given that we have never promised it and have only provided it since ScratchQuickSort was introduced.

@LilithHafner
Copy link
Member Author

Does anyone think the stability thing is a problem? If not, this is a fairly solid performance win an should be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Must go faster sorting Put things in order
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants