Deuxfleurs/garage
34
65
Fork 82
Code Issues 94 Pull requests 16 Releases 32 Wiki Activity

Added a disk to a single node and gone extremely slow #977

New issue
Open
opened 2025-03-06 09:00:45 +00:00 by bodleytunes99 · 8 comments
bodleytunes99 commented 2025-03-06 09:00:45 +00:00
Copy link

I tested adding a filepath on a 4tb spinning rust disk last night to existing single node but since doing this all read and write performance has gone unusable slow. I used to get read and write via cyber duck at 116MB second but now it instantly drops to about 200KB sec and I find that most of my files are no longer readable however strangely some are still readable. Its used for movie sitorage for plex and jellyfin.

I'm wondering if I restarted the docker container during some kind of operation that may have been under way after applying new layout.

How to debug this and worst case scenario is there a way I can safely revert to the last working setup? Its been 8 hours since the changes and everything is still unusable.

Thanks
Jon

I tested adding a filepath on a 4tb spinning rust disk last night to existing single node but since doing this all read and write performance has gone unusable slow. I used to get read and write via cyber duck at 116MB second but now it instantly drops to about 200KB sec and I find that most of my files are no longer readable however strangely some are still readable. Its used for movie sitorage for plex and jellyfin. I'm wondering if I restarted the docker container during some kind of operation that may have been under way after applying new layout. How to debug this and worst case scenario is there a way I can safely revert to the last working setup? Its been 8 hours since the changes and everything is still unusable. Thanks Jon
bodleytunes99 commented 2025-03-06 12:33:42 +00:00
Author
Copy link

part of the garage.toml that was modified

metadata_dir = "/var/lib/garage/meta"
#data_dir = "/var/lib/garage/data"
data_dir = [
    { path = "/var/lib/garage/data", capacity = "250G" },
    { path = "/mnt/data400/garaged400", capacity = "100G" },
    { path = "/mnt/4tb-backups/garage4tb", capacity = "150G" },
]
db_engine = "lmdb"

#replication_mode = "3"
#replication_mode = "1"
replication_factor = 1
part of the garage.toml that was modified ``` metadata_dir = "/var/lib/garage/meta" #data_dir = "/var/lib/garage/data" data_dir = [ { path = "/var/lib/garage/data", capacity = "250G" }, { path = "/mnt/data400/garaged400", capacity = "100G" }, { path = "/mnt/4tb-backups/garage4tb", capacity = "150G" }, ] db_engine = "lmdb" #replication_mode = "3" #replication_mode = "1" replication_factor = 1 ```
bodleytunes99 commented 2025-03-06 12:37:05 +00:00
Author
Copy link
garage status
==== HEALTHY NODES ====
ID                Hostname    Address           Tags    Zone  Capacity  DataAvail
6e7446cc7e53c903  srv-lounge  10.12.89.15:3901  [lsk1]  lsk1  500.0 GB  3.1 TB (59.5%)

garage stats

Garage version: v1.0.1 [features: k2v, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs]
Rust compiler version: 1.77.0

Database engine: LMDB (using Heed crate)

Table stats:
  Table      Items   MklItems  MklTodo  GcTodo
  bucket_v2  8       9         0        0
  key        8       9         0        0
  object     40005   44801     8        72
  version    38268   46203     57       1269
  block_ref  541024  615252    60       8300

Block manager stats:
  number of RC entries (~= number of blocks): 516881
  resync queue length: 3756
  blocks with resync errors: 302

Storage nodes:
  ID                Hostname    Zone  Capacity  Part.  DataAvail              MetaAvail
  6e7446cc7e53c903  srv-lounge  lsk1  500.0 GB  256    3.1 TB/5.3 TB (59.5%)  15.8 GB/999.7 GB (1.6%)

Estimated available storage space cluster-wide (might be lower in practice):
  data: 3.1 TB
  metadata: 15.8 GB

garage block list-errors
Hash                                                              RC  Errors  Last error    Next try
02b52c1bbefabdcf86cab566101c5c54cac619cbddb4245e098d2ee6877ea4f9  1   5876    12 hours ago  in asap
03c4e65bbad1e8e17e6cc342a48699c0e407ed285d8e4b8520fb07e9684be3d7  1   4131    12 hours ago  in asap
06937b090a2a1a9999238e00d81d02eb408dc9595c2cfec8dd26629e67447639  1   5875    12 hours ago  in asap
071879cd2de4ced18ef06cbd60e6aa9a4fc0511c7769fa70070c7014a506a152  1   5874    12 hours ago  in asap
085f5bc730cddface4bbf1590d987c315fbdbfc1d950ed5fb345016198f77f73  2   5875    12 hours ago  in asap

^^^ big list there spans a few pages

garage block retry-now --all
302 blocks returned in queue for a retry now (check logs to see results

2025-03-06T12:28:02.795379Z  INFO garage_api::generic_server: [::ffff:10.12.89.191]:34594 GET /movies/?delimiter=/&max-keys=1000&prefix=v%20for%20vendetta-443ce0/
2025-03-06T12:28:02.795389Z DEBUG garage_api::generic_server: Request { method: GET, uri: /movies/?delimiter=/&max-keys=1000&prefix=v%20for%20vendetta-443ce0/, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))", "accept": "*/*", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=56836372436b197fbe6866fb1a89ce8456eca09112f05999b3520e08fa5e7eb0", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T122802Z"}, body: Body(Empty) }
2025-03-06T12:28:02.795411Z DEBUG garage_api::generic_server: Endpoint: ListObjects
2025-03-06T12:28:02.795575Z DEBUG garage_api::s3::list: ListObjects ListObjectsQuery { is_v2: false, marker: None, continuation_token: None, start_after: None, common: ListQueryCommon { bucket_name: "movies", bucket_id: 309b123c46e399c6, delimiter: Some("/"), page_size: 1000, prefix: "v for vendetta-443ce0/", urlencode_resp: false } }
2025-03-06T12:28:02.795688Z DEBUG garage_api::s3::list: List: get range Some("v for vendetta-443ce0/") (max 1001), results: 6
2025-03-06T12:28:02.795754Z DEBUG garage_api::generic_server: 200 OK {"content-type": "application/xml"}
2025-03-06T12:28:02.796349Z  INFO garage_api::generic_server: [::ffff:10.12.89.191]:34594 GET /movies/?delimiter=/&max-keys=1000&prefix=young%20guns%202-e8c541/
2025-03-06T12:28:02.796357Z DEBUG garage_api::generic_server: Request { method: GET, uri: /movies/?delimiter=/&max-keys=1000&prefix=young%20guns%202-e8c541/, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))", "accept": "*/*", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e14a587ee89ef7dfe61ec1368ffd49c612d02181c239cec2cea84ba0dd2cc964", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T122802Z"}, body: Body(Empty) }
2025-03-06T12:28:02.796377Z DEBUG garage_api::generic_server: Endpoint: ListObjects
2025-03-06T12:28:02.796532Z DEBUG garage_api::s3::list: ListObjects ListObjectsQuery { is_v2: false, marker: None, continuation_token: None, start_after: None, common: ListQueryCommon { bucket_name: "movies", bucket_id: 309b123c46e399c6, delimiter: Some("/"), page_size: 1000, prefix: "young guns 2-e8c541/", urlencode_resp: false } }
2025-03-06T12:28:02.796602Z DEBUG garage_api::s3::list: List: get range Some("young guns 2-e8c541/") (max 1001), results: 4
2025-03-06T12:28:02.796653Z DEBUG garage_api::generic_server: 200 OK {"content-type": "application/xml"}
2025-03-06T12:28:02.797224Z  INFO garage_api::generic_server: [::ffff:10.12.89.191]:34594 GET /movies/?delimiter=/&max-keys=1000&prefix=young%20guns-c00b03/
2025-03-06T12:28:02.797232Z DEBUG garage_api::generic_server: Request { method: GET, uri: /movies/?delimiter=/&max-keys=1000&prefix=young%20guns-c00b03/, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))", "accept": "*/*", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=c9c84af9e9f3fde02a4686f6d059b6e25ea926aaa2e340d87f546a38119d6528", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T122802Z"}, body: Body(Empty) }
2025-03-06T12:28:02.797254Z DEBUG garage_api::generic_server: Endpoint: ListObjects
2025-03-06T12:28:02.797402Z DEBUG garage_api::s3::list: ListObjects ListObjectsQuery { is_v2: false, marker: None, continuation_token: None, start_after: None, common: ListQueryCommon { bucket_name: "movies", bucket_id: 309b123c46e399c6, delimiter: Some("/"), page_size: 1000, prefix: "young guns-c00b03/", urlencode_resp: false } }
2025-03-06T12:28:02.797460Z DEBUG garage_api::s3::list: List: get range Some("young guns-c00b03/") (max 1001), results: 2
2025-03-06T12:28:02.797508Z DEBUG garage_api::generic_server: 200 OK {"content-type": "application/xml"}
2025-03-06T12:30:21.614379Z DEBUG garage_api::generic_server: Error handling HTTP connection: error from user's Body stream
2025-03-06T12:30:21.715017Z  INFO garage_api::generic_server: [::ffff:172.20.0.2]:44740 GET /kopiabackups/kopiakopia.maintenance
2025-03-06T12:30:21.715027Z DEBUG garage_api::generic_server: Request { method: GET, uri: /kopiabackups/kopiakopia.maintenance, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "MinIO (linux; amd64) minio-go/v7.0.69", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9a8636b2402f6b2d48a4c3b5c6378766fa66e8a489b8d00422ebaa231ac9e0cf", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T123021Z", "accept-encoding": "gzip"}, body: Body(Empty) }
2025-03-06T12:30:21.715052Z DEBUG garage_api::generic_server: Endpoint: GetObject
2025-03-06T12:30:21.715258Z DEBUG garage_api::s3::get: Version meta: ObjectVersionMeta { size: 3191, etag: "90ea1235cf7fe63660cb716375dca8bc", encryption: Plaintext { inner: ObjectVersionMetaInner { headers: [("content-type", "application/x-kopia")], checksum: None } } }
2025-03-06T12:30:21.715311Z DEBUG garage_api::generic_server: 200 OK {"last-modified": "Wed, 05 Mar 2025 20:07:36 GMT", "accept-ranges": "bytes", "etag": "\"90ea1235cf7fe63660cb716375dca8bc\"", "content-type": "application/x-kopia", "content-length": "3191"}
2025-03-06T12:35:21.716428Z DEBUG garage_api::generic_server: Error handling HTTP connection: error from user's Body stream
2025-03-06T12:35:21.867094Z  INFO garage_api::generic_server: [::ffff:172.20.0.2]:33030 GET /kopiabackups/kopiakopia.maintenance
2025-03-06T12:35:21.867107Z DEBUG garage_api::generic_server: Request { method: GET, uri: /kopiabackups/kopiakopia.maintenance, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "MinIO (linux; amd64) minio-go/v7.0.69", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=967eb02e9813fe9e77e24c508463b8f57a91ced8fb6f4df82602997e3ee53f84", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T123521Z", "accept-encoding": "gzip"}, body: Body(Empty) }
2025-03-06T12:35:21.867132Z DEBUG garage_api::generic_server: Endpoint: GetObject
2025-03-06T12:35:21.867346Z DEBUG garage_api::s3::get: Version meta: ObjectVersionMeta { size: 3191, etag: "90ea1235cf7fe63660cb716375dca8bc", encryption: Plaintext { inner: ObjectVersionMetaInner { headers: [("content-type", "application/x-kopia")], checksum: None } } }
2025-03-06T12:35:21.867380Z DEBUG garage_api::generic_server: 200 OK {"last-modified": "Wed, 05 Mar 2025 20:07:36 GMT", "accept-ranges": "bytes", "etag": "\"90ea1235cf7fe63660cb716375dca8bc\"", "content-type": "application/x-kopia", "content-length": "3191"}
``` garage status ==== HEALTHY NODES ==== ID Hostname Address Tags Zone Capacity DataAvail 6e7446cc7e53c903 srv-lounge 10.12.89.15:3901 [lsk1] lsk1 500.0 GB 3.1 TB (59.5%) ``` ``` garage stats Garage version: v1.0.1 [features: k2v, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs] Rust compiler version: 1.77.0 Database engine: LMDB (using Heed crate) Table stats: Table Items MklItems MklTodo GcTodo bucket_v2 8 9 0 0 key 8 9 0 0 object 40005 44801 8 72 version 38268 46203 57 1269 block_ref 541024 615252 60 8300 Block manager stats: number of RC entries (~= number of blocks): 516881 resync queue length: 3756 blocks with resync errors: 302 Storage nodes: ID Hostname Zone Capacity Part. DataAvail MetaAvail 6e7446cc7e53c903 srv-lounge lsk1 500.0 GB 256 3.1 TB/5.3 TB (59.5%) 15.8 GB/999.7 GB (1.6%) Estimated available storage space cluster-wide (might be lower in practice): data: 3.1 TB metadata: 15.8 GB ``` ``` garage block list-errors Hash RC Errors Last error Next try 02b52c1bbefabdcf86cab566101c5c54cac619cbddb4245e098d2ee6877ea4f9 1 5876 12 hours ago in asap 03c4e65bbad1e8e17e6cc342a48699c0e407ed285d8e4b8520fb07e9684be3d7 1 4131 12 hours ago in asap 06937b090a2a1a9999238e00d81d02eb408dc9595c2cfec8dd26629e67447639 1 5875 12 hours ago in asap 071879cd2de4ced18ef06cbd60e6aa9a4fc0511c7769fa70070c7014a506a152 1 5874 12 hours ago in asap 085f5bc730cddface4bbf1590d987c315fbdbfc1d950ed5fb345016198f77f73 2 5875 12 hours ago in asap ``` ^^^ big list there spans a few pages ``` garage block retry-now --all 302 blocks returned in queue for a retry now (check logs to see results ``` ``` 2025-03-06T12:28:02.795379Z INFO garage_api::generic_server: [::ffff:10.12.89.191]:34594 GET /movies/?delimiter=/&max-keys=1000&prefix=v%20for%20vendetta-443ce0/ 2025-03-06T12:28:02.795389Z DEBUG garage_api::generic_server: Request { method: GET, uri: /movies/?delimiter=/&max-keys=1000&prefix=v%20for%20vendetta-443ce0/, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))", "accept": "*/*", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=56836372436b197fbe6866fb1a89ce8456eca09112f05999b3520e08fa5e7eb0", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T122802Z"}, body: Body(Empty) } 2025-03-06T12:28:02.795411Z DEBUG garage_api::generic_server: Endpoint: ListObjects 2025-03-06T12:28:02.795575Z DEBUG garage_api::s3::list: ListObjects ListObjectsQuery { is_v2: false, marker: None, continuation_token: None, start_after: None, common: ListQueryCommon { bucket_name: "movies", bucket_id: 309b123c46e399c6, delimiter: Some("/"), page_size: 1000, prefix: "v for vendetta-443ce0/", urlencode_resp: false } } 2025-03-06T12:28:02.795688Z DEBUG garage_api::s3::list: List: get range Some("v for vendetta-443ce0/") (max 1001), results: 6 2025-03-06T12:28:02.795754Z DEBUG garage_api::generic_server: 200 OK {"content-type": "application/xml"} 2025-03-06T12:28:02.796349Z INFO garage_api::generic_server: [::ffff:10.12.89.191]:34594 GET /movies/?delimiter=/&max-keys=1000&prefix=young%20guns%202-e8c541/ 2025-03-06T12:28:02.796357Z DEBUG garage_api::generic_server: Request { method: GET, uri: /movies/?delimiter=/&max-keys=1000&prefix=young%20guns%202-e8c541/, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))", "accept": "*/*", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e14a587ee89ef7dfe61ec1368ffd49c612d02181c239cec2cea84ba0dd2cc964", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T122802Z"}, body: Body(Empty) } 2025-03-06T12:28:02.796377Z DEBUG garage_api::generic_server: Endpoint: ListObjects 2025-03-06T12:28:02.796532Z DEBUG garage_api::s3::list: ListObjects ListObjectsQuery { is_v2: false, marker: None, continuation_token: None, start_after: None, common: ListQueryCommon { bucket_name: "movies", bucket_id: 309b123c46e399c6, delimiter: Some("/"), page_size: 1000, prefix: "young guns 2-e8c541/", urlencode_resp: false } } 2025-03-06T12:28:02.796602Z DEBUG garage_api::s3::list: List: get range Some("young guns 2-e8c541/") (max 1001), results: 4 2025-03-06T12:28:02.796653Z DEBUG garage_api::generic_server: 200 OK {"content-type": "application/xml"} 2025-03-06T12:28:02.797224Z INFO garage_api::generic_server: [::ffff:10.12.89.191]:34594 GET /movies/?delimiter=/&max-keys=1000&prefix=young%20guns-c00b03/ 2025-03-06T12:28:02.797232Z DEBUG garage_api::generic_server: Request { method: GET, uri: /movies/?delimiter=/&max-keys=1000&prefix=young%20guns-c00b03/, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))", "accept": "*/*", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=c9c84af9e9f3fde02a4686f6d059b6e25ea926aaa2e340d87f546a38119d6528", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T122802Z"}, body: Body(Empty) } 2025-03-06T12:28:02.797254Z DEBUG garage_api::generic_server: Endpoint: ListObjects 2025-03-06T12:28:02.797402Z DEBUG garage_api::s3::list: ListObjects ListObjectsQuery { is_v2: false, marker: None, continuation_token: None, start_after: None, common: ListQueryCommon { bucket_name: "movies", bucket_id: 309b123c46e399c6, delimiter: Some("/"), page_size: 1000, prefix: "young guns-c00b03/", urlencode_resp: false } } 2025-03-06T12:28:02.797460Z DEBUG garage_api::s3::list: List: get range Some("young guns-c00b03/") (max 1001), results: 2 2025-03-06T12:28:02.797508Z DEBUG garage_api::generic_server: 200 OK {"content-type": "application/xml"} 2025-03-06T12:30:21.614379Z DEBUG garage_api::generic_server: Error handling HTTP connection: error from user's Body stream 2025-03-06T12:30:21.715017Z INFO garage_api::generic_server: [::ffff:172.20.0.2]:44740 GET /kopiabackups/kopiakopia.maintenance 2025-03-06T12:30:21.715027Z DEBUG garage_api::generic_server: Request { method: GET, uri: /kopiabackups/kopiakopia.maintenance, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "MinIO (linux; amd64) minio-go/v7.0.69", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9a8636b2402f6b2d48a4c3b5c6378766fa66e8a489b8d00422ebaa231ac9e0cf", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T123021Z", "accept-encoding": "gzip"}, body: Body(Empty) } 2025-03-06T12:30:21.715052Z DEBUG garage_api::generic_server: Endpoint: GetObject 2025-03-06T12:30:21.715258Z DEBUG garage_api::s3::get: Version meta: ObjectVersionMeta { size: 3191, etag: "90ea1235cf7fe63660cb716375dca8bc", encryption: Plaintext { inner: ObjectVersionMetaInner { headers: [("content-type", "application/x-kopia")], checksum: None } } } 2025-03-06T12:30:21.715311Z DEBUG garage_api::generic_server: 200 OK {"last-modified": "Wed, 05 Mar 2025 20:07:36 GMT", "accept-ranges": "bytes", "etag": "\"90ea1235cf7fe63660cb716375dca8bc\"", "content-type": "application/x-kopia", "content-length": "3191"} 2025-03-06T12:35:21.716428Z DEBUG garage_api::generic_server: Error handling HTTP connection: error from user's Body stream 2025-03-06T12:35:21.867094Z INFO garage_api::generic_server: [::ffff:172.20.0.2]:33030 GET /kopiabackups/kopiakopia.maintenance 2025-03-06T12:35:21.867107Z DEBUG garage_api::generic_server: Request { method: GET, uri: /kopiabackups/kopiakopia.maintenance, version: HTTP/1.1, headers: {"host": "10.12.89.15:3900", "user-agent": "MinIO (linux; amd64) minio-go/v7.0.69", "authorization": "AWS4-HMAC-SHA256 Credential=GKdddf1c2a97b4a2daf181e5ed/20250306/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=967eb02e9813fe9e77e24c508463b8f57a91ced8fb6f4df82602997e3ee53f84", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20250306T123521Z", "accept-encoding": "gzip"}, body: Body(Empty) } 2025-03-06T12:35:21.867132Z DEBUG garage_api::generic_server: Endpoint: GetObject 2025-03-06T12:35:21.867346Z DEBUG garage_api::s3::get: Version meta: ObjectVersionMeta { size: 3191, etag: "90ea1235cf7fe63660cb716375dca8bc", encryption: Plaintext { inner: ObjectVersionMetaInner { headers: [("content-type", "application/x-kopia")], checksum: None } } } 2025-03-06T12:35:21.867380Z DEBUG garage_api::generic_server: 200 OK {"last-modified": "Wed, 05 Mar 2025 20:07:36 GMT", "accept-ranges": "bytes", "etag": "\"90ea1235cf7fe63660cb716375dca8bc\"", "content-type": "application/x-kopia", "content-length": "3191"} ```
lx commented 2025-03-06 20:16:05 +00:00
Owner
Copy link

Hi, not sure what is going on here.

In all cases, before doing anything, make a snapshot of your metadata directory (using garage meta snapshot, then copy the file from $metadata_dir/snapshots to somewhere safe) and if possible also take a copy of all of your data directories so that you can come back to the current garage state if needed.

To improve performance after adding a disk, you should invoke garage repair rebalance. Doing this, Garage will move files between your various data directories, so this is why you should make a copy before. This will help Garage know of a single unique location for each file, instead of having to look through multiple disks.

However, the fact that your r/w performance dropped so significantly seems to indicate that there is at least one of your disks that is extremely slow compared to the other. You should try to benchmark the IO performance of your disks to see if one of them is problematic.

Also, you should increase the block_size parameter if you haven't already. For media files, 10M is a minimum, you can even go higher especially in a single-node setup.

Hi, not sure what is going on here. In all cases, before doing anything, make a snapshot of your metadata directory (using `garage meta snapshot`, then copy the file from $metadata_dir/snapshots to somewhere safe) and if possible also take a copy of all of your data directories so that you can come back to the current garage state if needed. To improve performance after adding a disk, you should invoke `garage repair rebalance`. Doing this, Garage will move files between your various data directories, so this is why you should make a copy before. This will help Garage know of a single unique location for each file, instead of having to look through multiple disks. However, the fact that your r/w performance dropped so significantly seems to indicate that there is at least one of your disks that is extremely slow compared to the other. You should try to benchmark the IO performance of your disks to see if one of them is problematic. Also, you should increase the `block_size` parameter if you haven't already. For media files, `10M` is a minimum, you can even go higher especially in a single-node setup.
lx commented 2025-03-06 20:19:21 +00:00
Owner
Copy link

Also, if you just want to go back to the previous steup, you can:

  1. modify your configuration file and set read_only = true for the disk you added (and remove capacity = "xxx")
  2. run garage repair rebalance
  3. check the progress of the rebalance using garage worker list/ garage worker info
  4. once the rebalance is done, modify your configuration file again to remove the added disk
Also, if you just want to go back to the previous steup, you can: 1. modify your configuration file and set `read_only = true` for the disk you added (and remove `capacity = "xxx"`) 2. run `garage repair rebalance` 3. check the progress of the rebalance using `garage worker list`/ `garage worker info` 4. once the rebalance is done, modify your configuration file again to remove the added disk
bodleytunes99 commented 2025-03-06 21:08:24 +00:00
Author
Copy link

OK will give it a shot!

The disk I added isn't that slow it is slower as its a usb spinning rust, but its fast enough when I'm backing up to it with proxmox. The others are SSD's standard ones.

OK will give it a shot! The disk I added isn't that slow it is slower as its a usb spinning rust, but its fast enough when I'm backing up to it with proxmox. The others are SSD's standard ones.
bodleytunes99 commented 2025-03-06 21:16:27 +00:00
Author
Copy link

how to monitor the progress of a repair?
Sorry just seen, I need to use worker list and worker info!

how to monitor the progress of a repair? Sorry just seen, I need to use worker list and worker info!
bodleytunes99 commented 2025-03-06 23:30:08 +00:00
Author
Copy link

tried restoring the meta db from a snapshot, created a new layout and now seems worse as doesn't even authenticate when I'm connect and don't see anything in the logs...

[root@srv-lounge garagefs]$ garage stats

Garage version: v1.0.1 [features: k2v, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs]
Rust compiler version: 1.77.0

Database engine: LMDB (using Heed crate)

Table stats:
  Table      Items   MklItems  MklTodo  GcTodo
  bucket_v2  8       9         0        0
  key        8       9         0        0
  object     39912   44684     0        0
  version    36879   44354     0        0
  block_ref  527281  601011    0        0

Block manager stats:
  number of RC entries (~= number of blocks): 512846
  resync queue length: 391
  blocks with resync errors: 289

Storage nodes:
  ID                Hostname    Zone  Capacity  Part.  DataAvail                  MetaAvail
  6e7446cc7e53c903  srv-lounge  lsk1  500.0 GB  256    100.8 GB/999.7 GB (10.1%)  100.8 GB/999.7 GB (10.1%)

Estimated available storage space cluster-wide (might be lower in practice):
  data: 100.8 GB
  metadata: 100.8 GB

[root@srv-lounge garagefs]$ garage layout show
==== CURRENT CLUSTER LAYOUT ====
ID                Tags  Zone  Capacity  Usable capacity
6e7446cc7e53c903  lsk1  lsk1  500.0 GB  500.0 GB (100.0%)

Zone redundancy: maximum

Current cluster layout version: 10
[root@srv-lounge garagefs]$ garage layout show
==== CURRENT CLUSTER LAYOUT ====
ID                Tags  Zone  Capacity  Usable capacity
6e7446cc7e53c903  lsk1  lsk1  500.0 GB  500.0 GB (100.0%)

Zone redundancy: maximum

Current cluster layout version: 10
tried restoring the meta db from a snapshot, created a new layout and now seems worse as doesn't even authenticate when I'm connect and don't see anything in the logs... ``` [root@srv-lounge garagefs]$ garage stats Garage version: v1.0.1 [features: k2v, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs] Rust compiler version: 1.77.0 Database engine: LMDB (using Heed crate) Table stats: Table Items MklItems MklTodo GcTodo bucket_v2 8 9 0 0 key 8 9 0 0 object 39912 44684 0 0 version 36879 44354 0 0 block_ref 527281 601011 0 0 Block manager stats: number of RC entries (~= number of blocks): 512846 resync queue length: 391 blocks with resync errors: 289 Storage nodes: ID Hostname Zone Capacity Part. DataAvail MetaAvail 6e7446cc7e53c903 srv-lounge lsk1 500.0 GB 256 100.8 GB/999.7 GB (10.1%) 100.8 GB/999.7 GB (10.1%) Estimated available storage space cluster-wide (might be lower in practice): data: 100.8 GB metadata: 100.8 GB [root@srv-lounge garagefs]$ garage layout show ==== CURRENT CLUSTER LAYOUT ==== ID Tags Zone Capacity Usable capacity 6e7446cc7e53c903 lsk1 lsk1 500.0 GB 500.0 GB (100.0%) Zone redundancy: maximum Current cluster layout version: 10 [root@srv-lounge garagefs]$ garage layout show ==== CURRENT CLUSTER LAYOUT ==== ID Tags Zone Capacity Usable capacity 6e7446cc7e53c903 lsk1 lsk1 500.0 GB 500.0 GB (100.0%) Zone redundancy: maximum Current cluster layout version: 10 ```
maximilien commented 2025-03-18 15:02:20 +00:00
Owner
Copy link

What do you mean by "fail to authenticate"?

What do you mean by "fail to authenticate"?
maximilien added the
scope
ops
 label 2025-03-18 15:02:46 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
next-v2
fix-bucket-aliases
add-unsigned-payload-trailer-test
v1.99-internal
feat-metrics-metadata-engine
main-0.9.x
db-no-unsafe
main-0.8.x
k2v/shared_http_client
feat/website-redir
smithy2
test/compilation
poc/nomad-ci
test-ci-wait
pnet_datalink-0.33.0
k2v-watch-range
db-debug-log
build-convert-db
demonstrate-cargo2nix-bug
dangerous/no-fsync
dev0.7.3
doc/benchmarks
bug/check_static
bucket-cors-more
storage-optimizations
old-main
v1.99.1-internal
v1.1.0
v1.1.0-rc1
v1.99.0-internal
v1.0.1
v1.0.0
v0.9.4
v1.0.0-rc1
v0.8.7
v0.9.3
v0.9.2
v0.8.6
v0.9.2-rc1
v0.9.1
v0.8.5
v0.10.0-beta1
v0.9.0
v0.9.0-rc2
v0.9.0-rc1
v0.9.0-beta4
v0.9.0-beta3
v0.9.0-beta2
v0.8.4
v0.9.0-beta1
v0.8.3
v0.8.3-rc1
format_table-v0.1.1
format_table-v0.1.0
v0.8.2
v0.8.1
v0.8.0
v0.8.0-rc2
v0.8.0-rc1
v0.8.0-dangerous-no-fsync
v0.8.0-beta2-k2v
v0.8.0-beta2
v0.8.0-beta1-k2v
v0.8.0-beta1
v0.7.99.3-k2v
v0.7.3
v0.7.3-beta2
v0.7.3-beta1
v0.7.2.1
v0.7.2
v0.7.2_ci-test-2
v0.7.2+ci-test-version
v0.7.99.2-k2v
v0.7.99.1-k2v
v0.7.99-k2v
v0.7.2-k2v
v0.7.1-admin-k2v-2
v0.7.1-admin-k2v
v0.7.1-k2v
v0.7.1
v0.7.0
v0.7.0-rc1
v0.6.1
v0.6.0
v0.6.0-rc1
v0.5.1
v0.5.0.1
v0.5.0
v0.5-beta1
v0.4.0
v0.4-rc2
v0.4-rc1
0.4-beta
0.4-alpha
v0.3.0.2
v0.3.0.1
v0.3.0
v0.2.1.5
v0.2.1
v0.2.0
0.1.1b
0.1.1
0.1.0b
0.1.0
Labels
Clear labels   
action
check-aws
The behaviour should be checked against AWS S3

  
action
discussion-needed
We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  
action
for-external-contributors
Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  
action
for-newcomers
An issue that should be newcomer friendly.

  
action
more-info-needed
More information is needed to investigate this issue

  
action
need-funding
This issue represents lots of work, we need to find money to pay our core team work

  
action
triage-required
It's not very clear what we want to do with this issue yet, more thoughts is required

  
kind
correctness
Issues of theoretical correctness that may impact edge-case scenarios

  
kind/experimental

PRs implementing experimental features / features used only for debugging

  
kind
ideas
For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  
kind
improvement
Feature requests that do not fall in other categories

  
kind
performance
Deceptive performance observed OR idea to improve performances

  
kind
testing
Functional tests, integration tests, Jepsen, etc.

  
kind
usability
Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  
kind
wrong-behavior
Unintended behavior

  
prio
critical
Could impact integrity or availability of production clusters

  
prio
low
For issues that we don't want to spend time on until made absolutely necessary

  
scope
admin-api
Issues concerning the administration API

  
scope
admin-sdk
Related to the admin SDKs that provide API bindings for various languages

  
scope
background-healing
Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  
scope
build
Issues with Nix, Drone, Rust Compiler, etc.

  
scope
documentation

  
scope
k8s
Kubernetes related, including Helm Charts

  
scope
layout
Linked to the cluster layout and storage management

  
scope
metadata
Metadata Engine (LMDB, sqlite, etc.)

  
scope
ops
Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  
scope
rpc
Linked to RPC

  
scope
s3-api
Related to the S3 API (especially compatibility issues).

  
scope
security
Cryptography-related, auth-related, authorization-related issues

  
scope
telemetry
OpenTelemetry (traces, metrics, etc.)

No labels
action
check-aws
action
discussion-needed
action
for-external-contributors
action
for-newcomers
action
more-info-needed
action
need-funding
action
triage-required
kind
correctness
kind/experimental
kind
ideas
kind
improvement
kind
performance
kind
testing
kind
usability
kind
wrong-behavior
prio
critical
prio
low
scope
admin-api
scope
admin-sdk
scope
background-healing
scope
build
scope
documentation
scope
k8s
scope
layout
scope
metadata
scope
ops
scope
rpc
scope
s3-api
scope
security
scope
telemetry
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#977
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?