What's the preferred way of running tests for packagers?

[orhun]

Hey!

I'm maintaining cargo-nextest for Arch Linux and we are currently running tests like this after building the package:

build() {
  cd "$pkgname-$pkgver"
  cargo build --release --frozen --package "$pkgname" --no-default-features --features default-no-update
}

check() {
  cd "$pkgname-$pkgver"
  cargo run --package cargo-nextest -- nextest run -- --skip test_version_info
}

I adopted this from the GitHub workflows from this repository a while ago.

However, with the last release (0.9.100), running the tests take a long time and get stuck after a while, maybe due to the new global timeout feature?

More specifically, after this point:

        SLOW [>1620.000s] nextest-runner::integration basic::test_retries::retry_overrides_ignored
        SLOW [>1620.000s] nextest-runner::integration basic::test_retries::retry_overrides_obeyed
        SLOW [>1620.000s] nextest-runner::integration basic::test_run
        SLOW [>1620.000s] nextest-runner::integration basic::test_run_ignored
        SLOW [>1620.000s] nextest-runner::integration basic::test_string_filters_without_filter_expr
        SLOW [>1620.000s] nextest-runner::integration basic::test_termination
        SLOW [>1620.000s] nextest-runner::integration target_runner::test_listing_with_target_runner
        SLOW [>1620.000s] nextest-runner::integration target_runner::test_run_with_target_runn

So I have a couple of questions:

• What do you suggest we do as packagers to avoid long-running tests like this? Is there a filter command or something similar to skip these?
• Is my way of running tests correct & can it be improved?

Thank you :)

[sunshowers]

Thanks -- I think this is a genuine bug in nextest.

[sunshowers]

Interesting! 96 is a lot (is that a Threadripper?)

Might be worth me setting up an Arch container to repro this.

[sunshowers]

I can't repro this in an Arch container either...

@orhun would you be willing to fetch main (340c902) and then try:

cargo run --package cargo-nextest -- nextest run --profile ci --skip test_version_info

(The important thing is --profile ci and updating to main. Thank you!)

[orhun]

Yeah, that's the build server that we are using at Arch Linux.

Might be worth me setting up an Arch container to repro this.

You can use the docker container or use a clean chroot.

To build in the container, you can probably do this:

FROM archlinux:base-devel
WORKDIR nextest
RUN pacman --noconfirm -Syu rust
RUN curl https://gitlab.archlinux.org/archlinux/packaging/packages/cargo-nextest/-/raw/main/PKGBUILD?ref_type=heads > PKGBUILD
# edit pkgver and checksums for 0.9.100
ENV EUID=1 # this is really bad but it's the easiest way of running makepkg as root x_x
RUN makepkg -si

[sunshowers]

What does uname -a say on the build server?

[orhun]

Linux build.archlinux.org 6.15.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Tue, 10 Jun 2025 21:32:33 +0000 x86_64 GN
U/Linux

[sunshowers]

Let's discuss this in #2463 since this appears to be a real issue.

[sunshowers]

Closing in favor of #2463.

[orhun]

Going back to the original question:

Is my way of running tests correct & can it be improved?

Specifically I'm asking regarding this command: cargo run --package cargo-nextest -- nextest run -- --skip test_version_info

[sunshowers]

Thanks @orhun -- what happens with test_version_info? Overall you should be able to run the whole test suite (definitely worth documenting this explicitly too)

[orhun]

I don't remember the exact problem with that case. I tried building the package again without that --skip flag and now I'm getting an entirely different test suite failure (related to timeouts):

test.log

[sunshowers]

Have you updated to Rust 1.89? That failure looks like the same as the one in #2463.

[orhun]

Ah sorry you are right... Here is how test_version_info fails:

        FAIL [   0.014s] integration-tests::integration test_version_info
  stdout ───

    running 1 test
    cargo-nextest 0.9.102
    test test_version_info ... FAILED

    failures:

    failures:
        test_version_info

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 24 filtered out; finished in 0.01s

  stderr ───
    *** executing: /build/cargo-nextest/src/cargo-nextest-0.9.102/target/debug/cargo-nextest-dup nextest -V
    *** command /build/cargo-nextest/src/cargo-nextest-0.9.102/target/debug/cargo-nextest-dup nextest -V exited with status exit status: 0

    thread 'test_version_info' panicked at integration-tests/tests/integration/main.rs:59:28:
    short version matches regex: cargo-nextest 0.9.102

    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

  Cancelling due to test failure: 8 tests still running
        PASS [   0.350s] integration-tests::integration test_setup_script_error
        PASS [   0.235s] integration-tests::integration test_setup_scripts_not_enabled
        PASS [   0.335s] integration-tests::integration test_show_config_version
        PASS [   0.695s] integration-tests::integration test_target_arg
        PASS [   1.088s] integration-tests::integration test_show_config_test_groups
        PASS [   3.230s] nextest-runner::integration basic::test_termination
        PASS [   2.338s] integration-tests::integration test_run
        PASS [   1.572s] integration-tests::integration test_target_dir
────────────
     Summary [  20.489s] 340 tests run: 339 passed, 1 failed, 1 skipped
        FAIL [   0.014s] integration-tests::integration test_version_info
error: test run failed