burntsushi/walkdir

A cross platform Rust library for efficiently walking a directory recursively

Comes with support for following symbolic links, controlling the number of

walkdir

A cross platform Rust library for efficiently walking a directory recursively. Comes with support for following symbolic links, controlling the number of open file descriptors and efficient mechanisms for pruning the entries in the directory tree.

Dual-licensed under MIT or the UNLICENSE.

Documentation

docs.rs/walkdir

Usage

To use this crate, add walkdir as a dependency to your project's Cargo.toml:

Example

The following code recursively iterates over the directory given and prints the path for each entry:

Or, if you'd like to iterate over all entries and ignore any errors that may arise, use filter_map. (e.g., This code below will silently skip directories that the owner of the running process does not have permission to access.)

Example: follow symbolic links

The same code as above, except follow_links is enabled:

Example: skip hidden files and directories efficiently on unix

This uses the filter_entry iterator adapter to avoid yielding hidden files and directories efficiently:

Minimum Rust version policy

This crate's minimum supported rustc version is 1.34.0.

The current policy is that the minimum Rust version required to use this crate can be increased in minor version updates. For example, if crate 1.0 requires Rust 1.20.0, then crate 1.0.z for all values of z will also require Rust 1.20.0 or newer. However, crate 1.y for y > 0 may require a newer minimum version of Rust.

In general, this crate will be conservative with respect to the minimum supported version of Rust.

Performance

The short story is that performance is comparable with find and glibc's nftw on both a warm and cold file cache. In fact, I cannot observe any performance difference after running find /, walkdir / and nftw / on my local file system (SSD, ~3 million entries). More precisely, I am reasonably confident that this crate makes as few system calls and close to as few allocations as possible.

I haven't recorded any benchmarks, but here are some things you can try with a local checkout of walkdir:

On my system, the performance of walkdir, find and nftw is comparable.

Issues

Collection of the latest Issues

kpcyrd

kpcyrd

Comment Icon0

This is currently recommended to skip hidden directories in the docs:

This works well for foo, but breaks if . is provided instead (playground):

Output:

I'm wondering if walkdir::FilterEntry could be extended with a method to make the initial path skip the filter. Alternatively is_hidden could be enhanced to check for . and .. since filter_entry should never try these unless the path was the initial folder.

The example code in the docs should be updated to include this change.

Thanks!

lemmih

lemmih

Comment Icon1

Hi,

Line 109 in src/lib.rs uses cfg(doctest) which didn't stabilize until Rust version 1.40.

matklad

matklad

Comment Icon0

Meta: this isn't remotely important, logging the issue just for completeness sake.

In rust-analyzer, I am writing a test that needs to list all Rust files in rust-analyzer's repo. I think in this context using stdlib and doing something like the following should be fine:

However I am not sure -- there might be some pitfalls about std::fs::read_dir I am not sure about (similarly to how, eg, remove_dir_all doesn't always work on Windows). It would be helpful if walkdir (the primary alternative to "do it yourself") docs contained a section explaining when walkdir isn't actually needed.

silasb

silasb

enhancement
Comment Icon2

I have a folder structure like

Doing

results in

I'd expect to see

per what the documentation is suggesting.

andrewhickman

andrewhickman

Comment Icon0

The docs say

This causes the iterator to stop traversing the contents of the least recently yielded directory.

Looking at the example I think this should be

This causes the iterator to stop traversing the contents of the most recently yielded directory

Am I misunderstanding? If not I'm happy to create a PR

zicklag

zicklag

enhancement
Comment Icon5

I am using this library and have run into the need to capture a potential error inside of the filter_entry predicate. I asked in a forum topic about my problem and the response was that the best way to accomplish it would be with a try_filter_entry iterator so that I could return a result inside of the filter predicate.

I did a quick check of the source code to see how I might be able to implement that, but I was wondering if you might have any guidance on how you would want that implemented.

I haven't ever written an iterator before, but it looks like, if I wanted to create a FilterEntry iterator that it would be nearly identical to the existing TryFilterEntry iterator, but not quite the same.

Is there any way to share most of the implementation without duplicating code, or is it better just to create a copy of FilterEntry and the make the changes necessary.

CAD97

CAD97

bug
Comment Icon2

This requires the P for the filter_entry to be the same as the one already applied. So this requires a second application of filter_entry to use the same type as the first. This means it's not possible to call filter_entry on a WalkDir twice with two different closures; instead, function pointers (or boxed closures) need to be used instead.

I find it amusing it took so long to find this 😆 I guess 99.99% of use cases only need one filter. My workaround is to combine the two filters I was applying, though readability suffers for it.

The snippet in question

The better impl would give FilterEntry::filter_entry the same signature as IntoIter::filter_entry:

IIUC this is a "major breaking" change as someone could have used an empty turbofish (e.g. let _: fn(_, fn(&_) -> _) -> _ = walkdir::FilterEntry::filter_entry::<>;) to observe the lack of generic arguments.

metsuke0

metsuke0

help wanted
Comment Icon2

When iterating over the WalkDir dictionary in Windows 10, rust stable 1.36, if the user does not have permission to read a directory, an error is given, but not the actual path. When hitting an error with files, the path in the error is displayed but not with directories for some reason:

Error on entry 'Err(Error { depth: 2, inner: Io { path: None, err: Os { code: 5, kind: PermissionDenied, message: "Access is denied." } } })': Error { depth: 2, inner: Io { path: None, err: Os { code: 5, kind: PermissionDenied, message: "Access is denied." } } }

joshtriplett

joshtriplett

enhancement
Comment Icon15

I recently wrote a short test program using walkdir to count 6M files in a single directory. I then wrote the same test program using std::fs::read_dir, and found a surprisingly large performance difference.

To create the test data:

Using walkdir:

Performance:

Using std::fs::read_dir:

Note that the two programs use almost the same system time, and strace shows them making almost exactly the same syscalls other than an extra lstat (which makes sense, as walkdir uses std::fs::read_dir underneath). However, walkdir uses a lot more user time.

This seems worth investigating, to try to reduce the overhead.

Information - Updated May 20, 2022

Stars: 752
Forks: 83
Issues: 18

quest-hook-template

A template for writing mods for Quest il2cpp games in Rust using cargo generate to clone the template:

quest-hook-template

Command line json text parsing and processing utility

parsing json compliant with rust and cargo

Command line json text parsing and processing utility

Clone this repo: git clone

If you don't have Rust and cargo-make installed,

Clone this repo: git clone

Rustup: the Rust installer and version management tool

To test that you have Rust and Cargo installed, you can run this in your terminal of choice: cargo --version

Rustup: the Rust installer and version management tool

Address generator in Rust

If you have Rust: cargo install gemgen

Address generator in Rust

First, complete the basic Rust setup instructions

Use Rust's native cargo command to build and launch the template node:

First, complete the basic Rust setup instructions

cargo rssc - Rust scripts for crates building

will copy the template_basic into scripts_rssc folder

cargo rssc - Rust scripts for crates building

NES Emulator in Rust-WASM

Requires Rust with cargo, nodejs, and wasm-pack

NES Emulator in Rust-WASM

Terraformer Engine

A 3D game engine but completely in rust/std/cargo

Terraformer Engine

Rustup: the Rust installer and version management tool

To test that you have Rust and Cargo installed, you can run this in your terminal of choice: cargo --version

Rustup: the Rust installer and version management tool
Facebook Instagram Twitter GitHub Dribbble
Privacy