Race condition in integration tests #528

New Issue

Closed

opened 2023-03-13 12:53:55 +00:00 by teutat3s · 0 comments

teutat3s commented

2023-03-13 12:53:55 +00:00

Contributor

As mentioned in the Matrix #garage chat, there's currently a race condition in the integration tests, that can occur when running the tests in parallel on various threads.

The failing test looks like this:

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/lib.rs (target/debug/deps/integration-2d3fba86f796d156)

running 16 tests
test bucket::test_bucket_all ... FAILED
test s3::streaming_signature::test_create_bucket_streaming ... ok
test s3::streaming_signature::test_put_website_streaming ... ok
test s3::objects::test_getobject ... ok
test s3::simple::test_simple ... ok
test s3::streaming_signature::test_putobject_streaming ... ok
test s3::objects::test_putobject ... ok
test s3::website::test_website_s3_api ... ok
test s3::objects::test_deleteobject ... ok
test s3::list::test_listmultipart ... ok
test s3::website::test_website ... ok
test s3::list::test_listobjectsv2 ... ok
test s3::list::test_listobjectsv1 ... ok
test admin::test_admin_bucket_perms ... ok
test s3::multipart::test_uploadlistpart ... ok
test s3::multipart::test_uploadpartcopy ... ok

failures:

---- bucket::test_bucket_all stdout ----
thread 'bucket::test_bucket_all' panicked at 'assertion failed: r.is_err()', src/garage/tests/bucket.rs:22:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    bucket::test_bucket_all

test result: FAILED. 15 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.59s

error: test failed, to rerun pass `--test integration`

The current workaround is:

cargo test -- --test-threads=1

As mentioned in the Matrix #garage chat, there's currently a race condition in the integration tests, that can occur when running the tests in parallel on various threads. The failing test looks like this: ``` running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s Running tests/lib.rs (target/debug/deps/integration-2d3fba86f796d156) running 16 tests test bucket::test_bucket_all ... FAILED test s3::streaming_signature::test_create_bucket_streaming ... ok test s3::streaming_signature::test_put_website_streaming ... ok test s3::objects::test_getobject ... ok test s3::simple::test_simple ... ok test s3::streaming_signature::test_putobject_streaming ... ok test s3::objects::test_putobject ... ok test s3::website::test_website_s3_api ... ok test s3::objects::test_deleteobject ... ok test s3::list::test_listmultipart ... ok test s3::website::test_website ... ok test s3::list::test_listobjectsv2 ... ok test s3::list::test_listobjectsv1 ... ok test admin::test_admin_bucket_perms ... ok test s3::multipart::test_uploadlistpart ... ok test s3::multipart::test_uploadpartcopy ... ok failures: ---- bucket::test_bucket_all stdout ---- thread 'bucket::test_bucket_all' panicked at 'assertion failed: r.is_err()', src/garage/tests/bucket.rs:22:9 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace failures: bucket::test_bucket_all test result: FAILED. 15 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.59s error: test failed, to rerun pass `--test integration` ``` The current workaround is: ``` cargo test -- --test-threads=1 ```

👍 1