Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not search hidden folders / files #2

Closed
sharkdp opened this issue May 9, 2017 · 1 comment
Closed

Do not search hidden folders / files #2

sharkdp opened this issue May 9, 2017 · 1 comment

Comments

@sharkdp
Copy link
Owner

sharkdp commented May 9, 2017

No description provided.

@sharkdp sharkdp mentioned this issue May 9, 2017
@sharkdp
Copy link
Owner Author

sharkdp commented May 12, 2017

Closed via 12bfee0

@sharkdp sharkdp closed this as completed May 12, 2017
tavianator added a commit to tavianator/fd that referenced this issue Apr 26, 2019
…data

This should partially address sharkdp#432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '.h$' -j1 /usr -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    100.00    3.700169           3    983938     38022 stat
    $ strace -c -f ./fd-after '.h$' -j1 /usr -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    100.00    0.671723           4    162052     38021 stat

Though it's not as good as possible:

    $ strace -c -f find /usr -name '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    18.75    0.449866           3    136199           newfstatat
    $ strace -c -f bfs /usr -name '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    15.01    0.216024           3     60211         1 statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '.h$' -j1 /usr -S +1k"
    Benchmark sharkdp#1: ./fd-before '.h$' -j1 /usr -S +1k
      Time (mean ± σ):      2.707 s ±  0.042 s    [User: 890.8 ms, System: 1939.7 ms]
      Range (min … max):    2.659 s …  2.786 s    10 runs

    Benchmark sharkdp#2: ./fd-after '.h$' -j1 /usr -S +1k
      Time (mean ± σ):      1.562 s ±  0.034 s    [User: 726.2 ms, System: 957.9 ms]
      Range (min … max):    1.536 s …  1.648 s    10 runs

    Summary
      './fd-after '.h$' -j1 /usr -S +1k' ran
        1.73 ± 0.05 times faster than './fd-before '.h$' -j1 /usr -S +1k'

While remaining the same when it's not:

    tavianator@graviton $ hyperfine ./fd-{before,after}" '.h$' -j1 /usr"
    Benchmark sharkdp#1: ./fd-before '.h$' -j1 /usr
      Time (mean ± σ):      1.341 s ±  0.016 s    [User: 664.3 ms, System: 761.2 ms]
      Range (min … max):    1.309 s …  1.361 s    10 runs

    Benchmark sharkdp#2: ./fd-after '.h$' -j1 /usr
      Time (mean ± σ):      1.338 s ±  0.012 s    [User: 684.1 ms, System: 741.1 ms]
      Range (min … max):    1.310 s …  1.350 s    10 runs

    Summary
      './fd-after '.h$' -j1 /usr' ran
        1.00 ± 0.02 times faster than './fd-before '.h$' -j1 /usr'
tavianator added a commit to tavianator/fd that referenced this issue Apr 29, 2019
…data

This should partially address sharkdp#432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     15.71    8.831948           7   1192279     46059 stat
    $ strace -c -f ./fd-after '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      7.92    1.972474          10    183907     46046 stat

Though it's not as few as possible:

    $ strace -c -f find /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     19.01    0.946500           5    161649           newfstatat
    $ strace -c -f bfs /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     13.73    0.406565           5     69005           statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1 -S +1k"
    Benchmark sharkdp#1: ./fd-before '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      4.623 s ±  0.154 s    [User: 1.465 s, System: 3.354 s]
      Range (min … max):    4.327 s …  4.815 s    10 runs

    Benchmark sharkdp#2: ./fd-after '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      2.650 s ±  0.058 s    [User: 1.258 s, System: 1.592 s]
      Range (min … max):    2.568 s …  2.723 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1 -S +1k' ran
        1.74 ± 0.07 times faster than './fd-before '\.h$' /usr -j1 -S +1k'

While remaining the same when it's not:

    $ hyperfine ./fd-{before,after}" '.h$' /usr -j1"
    Benchmark sharkdp#1: ./fd-before '.h$' /usr -j1
      Time (mean ± σ):      2.314 s ±  0.052 s    [User: 1.185 s, System: 1.291 s]
      Range (min … max):    2.260 s …  2.441 s    10 runs

    Benchmark sharkdp#2: ./fd-after '.h$' /usr -j1
      Time (mean ± σ):      2.316 s ±  0.040 s    [User: 1.162 s, System: 1.315 s]
      Range (min … max):    2.263 s …  2.375 s    10 runs

    Summary
      './fd-before '.h$' /usr -j1' ran
        1.00 ± 0.03 times faster than './fd-after '.h$' /usr -j1'
tavianator added a commit to tavianator/fd that referenced this issue Apr 30, 2019
…data

This should partially address sharkdp#432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     15.71    8.831948           7   1192279     46059 stat
    $ strace -c -f ./fd-after '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      7.92    1.972474          10    183907     46046 stat

Though it's not as few as possible:

    $ strace -c -f find /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     19.01    0.946500           5    161649           newfstatat
    $ strace -c -f bfs /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     13.73    0.406565           5     69005           statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1 -S +1k"
    Benchmark sharkdp#1: ./fd-before '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      4.623 s ±  0.154 s    [User: 1.465 s, System: 3.354 s]
      Range (min … max):    4.327 s …  4.815 s    10 runs

    Benchmark sharkdp#2: ./fd-after '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      2.650 s ±  0.058 s    [User: 1.258 s, System: 1.592 s]
      Range (min … max):    2.568 s …  2.723 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1 -S +1k' ran
        1.74 ± 0.07 times faster than './fd-before '\.h$' /usr -j1 -S +1k'

While remaining the same when it's not:

    $ hyperfine ./fd-{before,after}" '.h$' /usr -j1"
    Benchmark sharkdp#1: ./fd-before '.h$' /usr -j1
      Time (mean ± σ):      2.314 s ±  0.052 s    [User: 1.185 s, System: 1.291 s]
      Range (min … max):    2.260 s …  2.441 s    10 runs

    Benchmark sharkdp#2: ./fd-after '.h$' /usr -j1
      Time (mean ± σ):      2.316 s ±  0.040 s    [User: 1.162 s, System: 1.315 s]
      Range (min … max):    2.263 s …  2.375 s    10 runs

    Summary
      './fd-before '.h$' /usr -j1' ran
        1.00 ± 0.03 times faster than './fd-after '.h$' /usr -j1'
tavianator added a commit to tavianator/fd that referenced this issue May 3, 2019
…data

This should partially address sharkdp#432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     15.71    8.831948           7   1192279     46059 stat
    $ strace -c -f ./fd-after '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      7.92    1.972474          10    183907     46046 stat

Though it's not as few as possible:

    $ strace -c -f find /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     19.01    0.946500           5    161649           newfstatat
    $ strace -c -f bfs /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     13.73    0.406565           5     69005           statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1 -S +1k"
    Benchmark sharkdp#1: ./fd-before '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      4.623 s ±  0.154 s    [User: 1.465 s, System: 3.354 s]
      Range (min … max):    4.327 s …  4.815 s    10 runs

    Benchmark sharkdp#2: ./fd-after '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      2.650 s ±  0.058 s    [User: 1.258 s, System: 1.592 s]
      Range (min … max):    2.568 s …  2.723 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1 -S +1k' ran
        1.74 ± 0.07 times faster than './fd-before '\.h$' /usr -j1 -S +1k'

While remaining the same when it's not:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1"
    Benchmark sharkdp#1: ./fd-before '\.h$' /usr -j1
      Time (mean ± σ):      2.382 s ±  0.038 s    [User: 1.221 s, System: 1.286 s]
      Range (min … max):    2.325 s …  2.433 s    10 runs

    Benchmark sharkdp#2: ./fd-after '\.h$' /usr -j1
      Time (mean ± σ):      2.362 s ±  0.034 s    [User: 1.193 s, System: 1.294 s]
      Range (min … max):    2.307 s …  2.422 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1' ran
        1.01 ± 0.02 times faster than './fd-before '\.h$' /usr -j1'
sharkdp pushed a commit that referenced this issue May 8, 2019
…data

This should partially address #432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     15.71    8.831948           7   1192279     46059 stat
    $ strace -c -f ./fd-after '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      7.92    1.972474          10    183907     46046 stat

Though it's not as few as possible:

    $ strace -c -f find /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     19.01    0.946500           5    161649           newfstatat
    $ strace -c -f bfs /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     13.73    0.406565           5     69005           statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1 -S +1k"
    Benchmark #1: ./fd-before '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      4.623 s ±  0.154 s    [User: 1.465 s, System: 3.354 s]
      Range (min … max):    4.327 s …  4.815 s    10 runs

    Benchmark #2: ./fd-after '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      2.650 s ±  0.058 s    [User: 1.258 s, System: 1.592 s]
      Range (min … max):    2.568 s …  2.723 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1 -S +1k' ran
        1.74 ± 0.07 times faster than './fd-before '\.h$' /usr -j1 -S +1k'

While remaining the same when it's not:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1"
    Benchmark #1: ./fd-before '\.h$' /usr -j1
      Time (mean ± σ):      2.382 s ±  0.038 s    [User: 1.221 s, System: 1.286 s]
      Range (min … max):    2.325 s …  2.433 s    10 runs

    Benchmark #2: ./fd-after '\.h$' /usr -j1
      Time (mean ± σ):      2.362 s ±  0.034 s    [User: 1.193 s, System: 1.294 s]
      Range (min … max):    2.307 s …  2.422 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1' ran
        1.01 ± 0.02 times faster than './fd-before '\.h$' /usr -j1'
sharkdp added a commit that referenced this issue Sep 15, 2019
Benchmark #1: ./fd-sys-alloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     246.8 ms ±   3.4 ms    [User: 960.1 ms, System: 810.0 ms]
  Range (min … max):   244.1 ms … 257.1 ms    12 runs

Benchmark #2: ./fd-jemalloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     201.0 ms ±   3.0 ms    [User: 833.9 ms, System: 666.9 ms]
  Range (min … max):   196.1 ms … 206.9 ms    14 runs

Summary
  './fd-jemalloc '[0-9]\.jpg$' /home/shark' ran
    1.23 ± 0.03 times faster than './fd-sys-alloc '[0-9]\.jpg$' /home/shark'
sharkdp added a commit that referenced this issue Sep 15, 2019
Benchmark #1: ./fd-sys-alloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     246.8 ms ±   3.4 ms    [User: 960.1 ms, System: 810.0 ms]
  Range (min … max):   244.1 ms … 257.1 ms    12 runs

Benchmark #2: ./fd-jemalloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     201.0 ms ±   3.0 ms    [User: 833.9 ms, System: 666.9 ms]
  Range (min … max):   196.1 ms … 206.9 ms    14 runs

Summary
  './fd-jemalloc '[0-9]\.jpg$' /home/shark' ran
    1.23 ± 0.03 times faster than './fd-sys-alloc '[0-9]\.jpg$' /home/shark'
sharkdp added a commit that referenced this issue Sep 15, 2019
Benchmark #1: ./fd-sys-alloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     246.8 ms ±   3.4 ms    [User: 960.1 ms, System: 810.0 ms]
  Range (min … max):   244.1 ms … 257.1 ms    12 runs

Benchmark #2: ./fd-jemalloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     201.0 ms ±   3.0 ms    [User: 833.9 ms, System: 666.9 ms]
  Range (min … max):   196.1 ms … 206.9 ms    14 runs

Summary
  './fd-jemalloc '[0-9]\.jpg$' /home/shark' ran
    1.23 ± 0.03 times faster than './fd-sys-alloc '[0-9]\.jpg$' /home/shark'
aswild referenced this issue in aswild/fd Jul 10, 2021
I don't know why fd was disabling jemalloc on musl, when that's the libc
where jemalloc really helps (about 3.6x faster than the default malloc
on musl 1.2.2). On glibc 2.33 (arch) and 2.28 (buster), jemalloc is only
2-3% faster than the default allocator, which isn't worth the extra
compile time.

Hyperfine runs searching in a linux kernel checkout where the Arch
5.12.15 config has been compiled on an i7-7700K running at stock speeds.

glibc:
    Benchmark #1: ./fd.gnu-default asdfasdfasdfasdf /workspace/linux-mainline
      Time (mean ± σ):      83.4 ms ±   2.9 ms    [User: 506.0 ms, System: 106.3 ms]
      Range (min … max):    79.8 ms … 111.6 ms    500 runs

    Benchmark #2: ./fd.gnu-jemalloc asdfasdfasdfasdf /workspace/linux-mainline
      Time (mean ± σ):      81.1 ms ±   3.1 ms    [User: 483.4 ms, System: 103.5 ms]
      Range (min … max):    77.6 ms … 118.3 ms    500 runs

    Summary
      './fd.gnu-jemalloc asdfasdfasdfasdf /workspace/linux-mainline' ran
	1.03 ± 0.05 times faster than './fd.gnu-default asdfasdfasdfasdf /workspace/linux-mainline'

musl:
    Benchmark #1: ./fd.musl-default asdfasdfasdfasdf /workspace/linux-mainline
      Time (mean ± σ):     274.0 ms ±   3.8 ms    [User: 2.040 s, System: 0.069 s]
      Range (min … max):   266.7 ms … 305.9 ms    500 runs

    Benchmark #2: ./fd.musl-jemalloc asdfasdfasdfasdf /workspace/linux-mainline
      Time (mean ± σ):      76.1 ms ±   3.8 ms    [User: 480.4 ms, System: 71.6 ms]
      Range (min … max):    72.1 ms … 112.1 ms    500 runs

    Summary
      './fd.musl-jemalloc asdfasdfasdfasdf /workspace/linux-mainline' ran
	3.60 ± 0.18 times faster than './fd.musl-default asdfasdfasdfasdf /workspace/linux-mainline'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant