Deuxfleurs
/
garage
43
37
Fork 55
Code Issues 69 Pull Requests 4 Releases 29 Wiki Activity

Garage S3 API is not able to GET certain file formats (.png, .jpg) when used in Directus #579

New Issue
Closed
opened 2023-05-21 16:43:20 +00:00 by best-user-name-on-demand · 4 comments
best-user-name-on-demand commented 2023-05-21 16:43:20 +00:00
Copy Link

https://directus.io/ (it seems to be using @aws-sdk/client-s3)

Context: If I use minio node js (with Garage S3 API!), I can upload and download images without issues. In Directus, I can upload and download .txt without issues.

Problem: I can upload .png or .jpg, but then I receive an error after the upload (possibly because Directus tries to GET the image after the POST). I'm not able to download .png or .jpg in Directus. Weirdly enough, in Directus, I'm able to listObjects to get metadata about the .jpg, .png and I'm also able to delete .jpg and .png. Just the fetching / GET doesn't work.

Error message in Directus logs:

[15:57:23.538] TRACE: [292.741ms] insert into "directus_revisions" ("activity", "collection", "data", "delta", "item") values ($1, $2, $3, $4, $5) returning "id" [49, directus_files, {"folder":null,"title":"Licensed Image","filename_download":"licensed-image.jpg","type":"image/jpeg","storage":"s3"}, {"folder":null,"title":"Licensed Image","filename_download":"licensed-image.jpg","type":"image/jpeg","storage":"s3"}, c264afcf-c4bf-48b9-9d9c-5ab6bcab385d]
[15:57:24] POST /files 500 1.8s
[15:57:24.948] ERROR: UnknownError
err: {
    "type": "S3ServiceException",
    "message": "UnknownError",
    "stack":
    403: UnknownError
    at throwDefaultError (/directus/node_modules/.pnpm/@aws-sdk+smithy-client@3.292.0/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at deserializeAws_restXmlHeadObjectCommandError (/directus/node_modules/.pnpm/@aws-sdk+client-s3@3.292.0/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4745:51)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /directus/node_modules/.pnpm/@aws-sdk+middleware-serde@3.292.0/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /directus/node_modules/.pnpm/@aws-sdk+middleware-signing@3.292.0/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
    at async /directus/node_modules/.pnpm/@aws-sdk+middleware-retry@3.292.0/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /directus/node_modules/.pnpm/@aws-sdk+middleware-logger@3.292.0/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async DriverS3.stat (file:///directus/node_modules/.pnpm/file+packages+storage-driver-s3/node_modules/@directus/storage-driver-s3/dist/index.js:68:49)
    at async FilesService.uploadOne (file:///directus/node_modules/.pnpm/file+api/node_modules/@directus/api/dist/services/files.js:71:26)
    at async Multipart.<anonymous> (file:///directus/node_modules/.pnpm/file+api/node_modules/@directus/api/dist/controllers/files.js:75:32)
    "name": "403",
    "$fault": "client",
    "$metadata": {
        "httpStatusCode": 403,
        "attempts": 1,
        "totalRetryDelay": 0
    }
}

Error Message in Garage Logs:

2023-05-21T16:32:24.061381Z  INFO garage_api::generic_server: REMOVED PUT /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg?x-id=PutObject
2023-05-21T16:32:24.061498Z DEBUG garage_api::generic_server: Request { method: PUT, uri: /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg?x-id=PutObject, version: HTTP/1.1, headers: {"host": "REMOVED", "x-request-id": "423b819992b01821e85182e867a2bfb1", "x-real-ip": "REMOVED", "x-forwarded-for": "REMOVED", "x-forwarded-host": "REMOVED", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "x-original-forwarded-for": "REMOVED", "content-length": "275237", "cf-ipcountry": "DE", "cdn-loop": "REMOVED", "accept-encoding": "gzip", "cf-ray": "7cae3715dec130f6-FRA", "cf-visitor": "{\"scheme\":\"https\"}", "content-type": "image/jpeg", "x-amz-user-agent": "aws-sdk-js/3.292.0", "user-agent": "aws-sdk-js/3.292.0 os/linux/5.15.0-69-generic lang/js md/nodejs/18.16.0 api/s3/3.292.0", "amz-sdk-invocation-id": "37b9497c-8bb6-4462-ad40-9927ce0716d8", "amz-sdk-request": "attempt=1; max=3", "x-amz-date": "20230521T163223Z", "x-amz-content-sha256": "92d754bd93a140c7039892bc54958ce26b06258788fb020bb62bdb5c4bb99a31", "authorization": "AWS4-HMAC-SHA256 Credential=REMOVED/20230521/garage/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=b775574b5004e5b0bcbb36ce248dbfac25b0462cd9b8634c08dd3a3097781f2c", "cf-connecting-ip": "REMOVED"}, body: Body(Streaming) }
2023-05-21T16:32:24.062143Z DEBUG garage_api::s3::router: Received an unknown query parameter: 'x-id'
2023-05-21T16:32:24.062189Z DEBUG garage_api::generic_server: Endpoint: PutObject
2023-05-21T16:32:24.062587Z TRACE garage_api::signature::payload: canonical request:
PUT
/test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg
x-id=PutObject
amz-sdk-invocation-id:37b9497c-8bb6-4462-ad40-9927ce0716d8
amz-sdk-request:attempt=1; max=3
content-length:275237
content-type:image/jpeg
host:REMOVED
x-amz-content-sha256:92d754bd93a140c7039892bc54958ce26b06258788fb020bb62bdb5c4bb99a31
x-amz-date:20230521T163223Z
x-amz-user-agent:aws-sdk-js/3.292.0

amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent
92d754bd93a140c7039892bc54958ce26b06258788fb020bb62bdb5c4bb99a31
2023-05-21T16:32:24.062617Z TRACE garage_api::signature::payload: string to sign:
AWS4-HMAC-SHA256
20230521T163223Z
20230521/garage/s3/aws4_request
82253201991b9aac18d1e6b59be0bd1754023f1963a2c5e30180296d93b296ca
2023-05-21T16:32:24.064090Z DEBUG garage_api::s3::put: Object headers: ObjectVersionHeaders { content_type: "image/jpeg", other: {} }
2023-05-21T16:32:24.064160Z TRACE garage_api::s3::put: Body next: 6920 bytes
2023-05-21T16:32:24.064256Z TRACE garage_api::s3::put: Body next: 16384 bytes
2023-05-21T16:32:24.064316Z TRACE garage_api::s3::put: Body next: 32768 bytes
2023-05-21T16:32:24.064384Z TRACE garage_api::s3::put: Body next: 40846 bytes
2023-05-21T16:32:24.065627Z TRACE garage_api::s3::put: Body next: 37094 bytes
2023-05-21T16:32:24.065770Z TRACE garage_api::s3::put: Body next: 26562 bytes
2023-05-21T16:32:24.066122Z TRACE garage_api::s3::put: Body next: 1398 bytes
2023-05-21T16:32:24.066536Z TRACE garage_api::s3::put: Body next: 36168 bytes
2023-05-21T16:32:24.067526Z TRACE garage_api::s3::put: Body next: 65536 bytes
2023-05-21T16:32:24.067707Z TRACE garage_api::s3::put: Body next: 11561 bytes
2023-05-21T16:32:24.928215Z TRACE garage_api::s3::put: Successfully validated x-amz-content-sha256
2023-05-21T16:32:25.142363Z DEBUG garage_api::generic_server: 200 OK {"x-amz-version-id": "61d482a4441400b2d9d0885d29a2448983b010d4c66a4a92b4ba588f19a38fa9", "etag": "\"9613a70a6b20ae0f4eaba238a28145f9\""}
2023-05-21T16:32:25.218857Z  INFO garage_api::generic_server: REMOVED GET /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg
2023-05-21T16:32:25.218960Z DEBUG garage_api::generic_server: Request { method: GET, uri: /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg, version: HTTP/1.1, headers: {"host": "REMOVED", "x-request-id": "b24e9d66f5a40b2762f71191fb7c1e9c", "x-real-ip": "REMOVED", "x-forwarded-for": "REMOVED", "x-forwarded-host": "REMOVED", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "x-original-forwarded-for": "REMOVED", "cf-ipcountry": "DE", "cdn-loop": "REMOVED", "accept-encoding": "gzip", "cf-ray": "7cae371d4aef30f6-FRA", "cf-visitor": "{\"scheme\":\"https\"}", "x-amz-user-agent": "aws-sdk-js/3.292.0", "user-agent": "aws-sdk-js/3.292.0 os/linux/5.15.0-69-generic lang/js md/nodejs/18.16.0 api/s3/3.292.0", "amz-sdk-invocation-id": "ed0225d2-f47b-49bb-815e-c9127905d04a", "amz-sdk-request": "attempt=1; max=3", "x-amz-date": "20230521T163225Z", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "authorization": "AWS4-HMAC-SHA256 Credential=REMOVED/20230521/garage/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=9aaf3b2046f17312a7841b7878a8ffe4fca9eadfa41aebefe00b83ddad5db7d2", "cf-connecting-ip": "REMOVED"}, body: Body(Empty) }
2023-05-21T16:32:25.219254Z DEBUG garage_api::generic_server: Endpoint: GetObject
2023-05-21T16:32:25.219630Z TRACE garage_api::signature::payload: canonical request:
GET
/test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg

amz-sdk-invocation-id:ed0225d2-f47b-49bb-815e-c9127905d04a
amz-sdk-request:attempt=1; max=3
host:REMOVED
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230521T163225Z
x-amz-user-agent:aws-sdk-js/3.292.0

amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-05-21T16:32:25.219663Z TRACE garage_api::signature::payload: string to sign:
AWS4-HMAC-SHA256
20230521T163225Z
20230521/garage/s3/aws4_request
8436cb6a6b96fc74ec060df136100cb075f67ff9f0b73ac7e2f2e27413601187
2023-05-21T16:32:25.220768Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature

Garage Setup in Directus (running in Kubernetes):

- name: STORAGE_LOCATIONS
  value: "s3"
- name: STORAGE_S3_DRIVER
  value: "s3"
- name: STORAGE_S3_KEY
  value: "{{ .Values.kube.directus.storage_s3_key }}"
- name: STORAGE_S3_SECRET
  value: "{{ .Values.kube.directus.storage_s3_secret }}"
- name: STORAGE_S3_BUCKET
  value: "{{ .Values.kube.directus.storage_s3_bucket }}"
- name: STORAGE_S3_REGION
  value: "{{ .Values.kube.directus.storage_s3_region }}"
- name: STORAGE_S3_ENDPOINT
  value: "{{ .Values.kube.directus.storage_s3_endpoint }}"
- name: STORAGE_S3_FORCE_PATH_STYLE
  value: "{{ .Values.kube.directus.storage_s3_force_path_style }}"

force path style is on, because I didn't succeed yet in providing sub domain wildcard TLS in Kubernetes. Maybe that could be the issue. I kinda doubt it though.

I'm investigating the error myself and will do a pull request if/when I figure it out.

I just wanted to let you know and maybe you even have an idea what the problem could be.

https://directus.io/ (it seems to be using @aws-sdk/client-s3) Context: If I use minio node js (with Garage S3 API!), I can upload and download images without issues. In Directus, I can upload and download .txt without issues. Problem: I can upload .png or .jpg, but then I receive an error after the upload (possibly because Directus tries to GET the image after the POST). I'm not able to download .png or .jpg in Directus. Weirdly enough, in Directus, I'm able to listObjects to get metadata about the .jpg, .png and I'm also able to delete .jpg and .png. Just the fetching / GET doesn't work. Error message in Directus logs: ```JavaScript [15:57:23.538] TRACE: [292.741ms] insert into "directus_revisions" ("activity", "collection", "data", "delta", "item") values ($1, $2, $3, $4, $5) returning "id" [49, directus_files, {"folder":null,"title":"Licensed Image","filename_download":"licensed-image.jpg","type":"image/jpeg","storage":"s3"}, {"folder":null,"title":"Licensed Image","filename_download":"licensed-image.jpg","type":"image/jpeg","storage":"s3"}, c264afcf-c4bf-48b9-9d9c-5ab6bcab385d] [15:57:24] POST /files 500 1.8s [15:57:24.948] ERROR: UnknownError err: { "type": "S3ServiceException", "message": "UnknownError", "stack": 403: UnknownError at throwDefaultError (/directus/node_modules/.pnpm/@aws-sdk+smithy-client@3.292.0/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22) at deserializeAws_restXmlHeadObjectCommandError (/directus/node_modules/.pnpm/@aws-sdk+client-s3@3.292.0/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4745:51) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /directus/node_modules/.pnpm/@aws-sdk+middleware-serde@3.292.0/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24 at async /directus/node_modules/.pnpm/@aws-sdk+middleware-signing@3.292.0/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20 at async /directus/node_modules/.pnpm/@aws-sdk+middleware-retry@3.292.0/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46 at async /directus/node_modules/.pnpm/@aws-sdk+middleware-logger@3.292.0/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26 at async DriverS3.stat (file:///directus/node_modules/.pnpm/file+packages+storage-driver-s3/node_modules/@directus/storage-driver-s3/dist/index.js:68:49) at async FilesService.uploadOne (file:///directus/node_modules/.pnpm/file+api/node_modules/@directus/api/dist/services/files.js:71:26) at async Multipart.<anonymous> (file:///directus/node_modules/.pnpm/file+api/node_modules/@directus/api/dist/controllers/files.js:75:32) "name": "403", "$fault": "client", "$metadata": { "httpStatusCode": 403, "attempts": 1, "totalRetryDelay": 0 } } ``` Error Message in Garage Logs: ```Rust 2023-05-21T16:32:24.061381Z INFO garage_api::generic_server: REMOVED PUT /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg?x-id=PutObject 2023-05-21T16:32:24.061498Z DEBUG garage_api::generic_server: Request { method: PUT, uri: /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg?x-id=PutObject, version: HTTP/1.1, headers: {"host": "REMOVED", "x-request-id": "423b819992b01821e85182e867a2bfb1", "x-real-ip": "REMOVED", "x-forwarded-for": "REMOVED", "x-forwarded-host": "REMOVED", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "x-original-forwarded-for": "REMOVED", "content-length": "275237", "cf-ipcountry": "DE", "cdn-loop": "REMOVED", "accept-encoding": "gzip", "cf-ray": "7cae3715dec130f6-FRA", "cf-visitor": "{\"scheme\":\"https\"}", "content-type": "image/jpeg", "x-amz-user-agent": "aws-sdk-js/3.292.0", "user-agent": "aws-sdk-js/3.292.0 os/linux/5.15.0-69-generic lang/js md/nodejs/18.16.0 api/s3/3.292.0", "amz-sdk-invocation-id": "37b9497c-8bb6-4462-ad40-9927ce0716d8", "amz-sdk-request": "attempt=1; max=3", "x-amz-date": "20230521T163223Z", "x-amz-content-sha256": "92d754bd93a140c7039892bc54958ce26b06258788fb020bb62bdb5c4bb99a31", "authorization": "AWS4-HMAC-SHA256 Credential=REMOVED/20230521/garage/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=b775574b5004e5b0bcbb36ce248dbfac25b0462cd9b8634c08dd3a3097781f2c", "cf-connecting-ip": "REMOVED"}, body: Body(Streaming) } 2023-05-21T16:32:24.062143Z DEBUG garage_api::s3::router: Received an unknown query parameter: 'x-id' 2023-05-21T16:32:24.062189Z DEBUG garage_api::generic_server: Endpoint: PutObject 2023-05-21T16:32:24.062587Z TRACE garage_api::signature::payload: canonical request: PUT /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg x-id=PutObject amz-sdk-invocation-id:37b9497c-8bb6-4462-ad40-9927ce0716d8 amz-sdk-request:attempt=1; max=3 content-length:275237 content-type:image/jpeg host:REMOVED x-amz-content-sha256:92d754bd93a140c7039892bc54958ce26b06258788fb020bb62bdb5c4bb99a31 x-amz-date:20230521T163223Z x-amz-user-agent:aws-sdk-js/3.292.0 amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent 92d754bd93a140c7039892bc54958ce26b06258788fb020bb62bdb5c4bb99a31 2023-05-21T16:32:24.062617Z TRACE garage_api::signature::payload: string to sign: AWS4-HMAC-SHA256 20230521T163223Z 20230521/garage/s3/aws4_request 82253201991b9aac18d1e6b59be0bd1754023f1963a2c5e30180296d93b296ca 2023-05-21T16:32:24.064090Z DEBUG garage_api::s3::put: Object headers: ObjectVersionHeaders { content_type: "image/jpeg", other: {} } 2023-05-21T16:32:24.064160Z TRACE garage_api::s3::put: Body next: 6920 bytes 2023-05-21T16:32:24.064256Z TRACE garage_api::s3::put: Body next: 16384 bytes 2023-05-21T16:32:24.064316Z TRACE garage_api::s3::put: Body next: 32768 bytes 2023-05-21T16:32:24.064384Z TRACE garage_api::s3::put: Body next: 40846 bytes 2023-05-21T16:32:24.065627Z TRACE garage_api::s3::put: Body next: 37094 bytes 2023-05-21T16:32:24.065770Z TRACE garage_api::s3::put: Body next: 26562 bytes 2023-05-21T16:32:24.066122Z TRACE garage_api::s3::put: Body next: 1398 bytes 2023-05-21T16:32:24.066536Z TRACE garage_api::s3::put: Body next: 36168 bytes 2023-05-21T16:32:24.067526Z TRACE garage_api::s3::put: Body next: 65536 bytes 2023-05-21T16:32:24.067707Z TRACE garage_api::s3::put: Body next: 11561 bytes 2023-05-21T16:32:24.928215Z TRACE garage_api::s3::put: Successfully validated x-amz-content-sha256 2023-05-21T16:32:25.142363Z DEBUG garage_api::generic_server: 200 OK {"x-amz-version-id": "61d482a4441400b2d9d0885d29a2448983b010d4c66a4a92b4ba588f19a38fa9", "etag": "\"9613a70a6b20ae0f4eaba238a28145f9\""} 2023-05-21T16:32:25.218857Z INFO garage_api::generic_server: REMOVED GET /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg 2023-05-21T16:32:25.218960Z DEBUG garage_api::generic_server: Request { method: GET, uri: /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg, version: HTTP/1.1, headers: {"host": "REMOVED", "x-request-id": "b24e9d66f5a40b2762f71191fb7c1e9c", "x-real-ip": "REMOVED", "x-forwarded-for": "REMOVED", "x-forwarded-host": "REMOVED", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "x-original-forwarded-for": "REMOVED", "cf-ipcountry": "DE", "cdn-loop": "REMOVED", "accept-encoding": "gzip", "cf-ray": "7cae371d4aef30f6-FRA", "cf-visitor": "{\"scheme\":\"https\"}", "x-amz-user-agent": "aws-sdk-js/3.292.0", "user-agent": "aws-sdk-js/3.292.0 os/linux/5.15.0-69-generic lang/js md/nodejs/18.16.0 api/s3/3.292.0", "amz-sdk-invocation-id": "ed0225d2-f47b-49bb-815e-c9127905d04a", "amz-sdk-request": "attempt=1; max=3", "x-amz-date": "20230521T163225Z", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "authorization": "AWS4-HMAC-SHA256 Credential=REMOVED/20230521/garage/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=9aaf3b2046f17312a7841b7878a8ffe4fca9eadfa41aebefe00b83ddad5db7d2", "cf-connecting-ip": "REMOVED"}, body: Body(Empty) } 2023-05-21T16:32:25.219254Z DEBUG garage_api::generic_server: Endpoint: GetObject 2023-05-21T16:32:25.219630Z TRACE garage_api::signature::payload: canonical request: GET /test-directus/dcb3ca5f-6439-406f-ba4b-1f8adfa30bae.jpg amz-sdk-invocation-id:ed0225d2-f47b-49bb-815e-c9127905d04a amz-sdk-request:attempt=1; max=3 host:REMOVED x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20230521T163225Z x-amz-user-agent:aws-sdk-js/3.292.0 amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-21T16:32:25.219663Z TRACE garage_api::signature::payload: string to sign: AWS4-HMAC-SHA256 20230521T163225Z 20230521/garage/s3/aws4_request 8436cb6a6b96fc74ec060df136100cb075f67ff9f0b73ac7e2f2e27413601187 2023-05-21T16:32:25.220768Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature ``` Garage Setup in Directus (running in Kubernetes): ``` - name: STORAGE_LOCATIONS value: "s3" - name: STORAGE_S3_DRIVER value: "s3" - name: STORAGE_S3_KEY value: "{{ .Values.kube.directus.storage_s3_key }}" - name: STORAGE_S3_SECRET value: "{{ .Values.kube.directus.storage_s3_secret }}" - name: STORAGE_S3_BUCKET value: "{{ .Values.kube.directus.storage_s3_bucket }}" - name: STORAGE_S3_REGION value: "{{ .Values.kube.directus.storage_s3_region }}" - name: STORAGE_S3_ENDPOINT value: "{{ .Values.kube.directus.storage_s3_endpoint }}" - name: STORAGE_S3_FORCE_PATH_STYLE value: "{{ .Values.kube.directus.storage_s3_force_path_style }}" ``` force path style is on, because I didn't succeed yet in providing sub domain wildcard TLS in Kubernetes. Maybe that could be the issue. I kinda doubt it though. I'm investigating the error myself and will do a pull request if/when I figure it out. I just wanted to let you know and maybe you even have an idea what the problem could be.
best-user-name-on-demand commented 2023-05-21 22:15:32 +00:00
Author
Copy Link

So, I cloned the Directus repository (https://github.com/directus/directus) and implemented the S3 Driver with Minio (Node JS), instead of @aws-sdk/client-s3 (Node JS).

Turns out, ... I get basically the same error.

Directus Logs:

[21:43:01] POST /files 500 1.8s
[21:43:01] GET /admin/assets/FiraMono-Medium-33e88fa9.woff2 200 2ms
[21:43:01.291] ERROR: Valid and authorized credentials required
    err: {
      "type": "S3Error",
      "message": "Valid and authorized credentials required",
      "stack":
          S3Error: Valid and authorized credentials required
              at getError (/directus/node_modules/.pnpm/minio@7.1.0/node_modules/minio/dist/main/transformers.js:149:15)
              at /directus/node_modules/.pnpm/minio@7.1.0/node_modules/minio/dist/main/transformers.js:161:14
              at DestroyableTransform._flush (/directus/node_modules/.pnpm/minio@7.1.0/node_modules/minio/dist/main/transformers.js:91:10)
              at DestroyableTransform.prefinish (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_transform.js:129:10)
              at DestroyableTransform.emit (node:events:513:28)
              at prefinish (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_writable.js:611:14)
              at finishMaybe (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_writable.js:620:5)
              at endWritable (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_writable.js:643:3)
              at Writable.end (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_writable.js:571:22)
              at IncomingMessage.onend (node:internal/streams/readable:705:10)
      "code": "AccessDenied",
      "amzRequestid": null,
      "amzId2": null,
      "amzBucketRegion": null
    }

Garage Logs:

2023-05-21T21:43:00.192148Z  INFO garage_api::generic_server: REMOVED PUT /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg
2023-05-21T21:43:00.192355Z DEBUG garage_api::generic_server: Request { method: PUT, uri: /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg, version: HTTP/1.1, headers: {"host": REMOVED, "x-request-id": "07e5e595711ad21833527f3aadb5b8e1", "x-real-ip": "REMOVED", "x-forwarded-for": "REMOVED", "x-forwarded-host": REMOVED, "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "x-original-forwarded-for": REMOVED, "content-length": "133809", "cf-ipcountry": "DE", "cdn-loop": "cloudflare", "accept-encoding": "gzip", "cf-ray": "7caffe11dd370374-FRA", "cf-visitor": "{\"scheme\":\"https\"}", "user-agent": "MinIO (linux; x64) minio-js/7.1.0", "content-md5": "7y0ofTgSDK9xeNHV9DJjow==", "x-amz-date": "20230521T214300Z", "x-amz-content-sha256": "UNSIGNED-PAYLOAD", "authorization": "AWS4-HMAC-SHA256 Credential=REMOVED/20230521/garage/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=c14426516a24fc26e3787199eda2a9c69eae31437ed8d0daef805f9f73686f33", "cf-connecting-ip": REMOVED}, body: Body(Streaming) }
2023-05-21T21:43:00.193212Z DEBUG garage_api::generic_server: Endpoint: PutObject
2023-05-21T21:43:00.193955Z TRACE garage_api::signature::payload: canonical request:
PUT
/test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg

content-md5:7y0ofTgSDK9xeNHV9DJjow==
host:REMOVED
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230521T214300Z

content-md5;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2023-05-21T21:43:00.194017Z TRACE garage_api::signature::payload: string to sign:
AWS4-HMAC-SHA256
20230521T214300Z
20230521/garage/s3/aws4_request
5df924be79b7d70c9012366829c41be256775619f63d0febdc45d21d71475121
2023-05-21T21:43:00.197269Z DEBUG garage_api::s3::put: Object headers: ObjectVersionHeaders { content_type: "blob", other: {} }
2023-05-21T21:43:00.197434Z TRACE garage_api::s3::put: Body next: 7228 bytes
2023-05-21T21:43:00.197684Z TRACE garage_api::s3::put: Body next: 16384 bytes
2023-05-21T21:43:00.197865Z TRACE garage_api::s3::put: Body next: 32768 bytes
2023-05-21T21:43:00.198037Z TRACE garage_api::s3::put: Body next: 25906 bytes
2023-05-21T21:43:00.198616Z TRACE garage_api::s3::put: Body next: 1398 bytes
2023-05-21T21:43:00.198817Z TRACE garage_api::s3::put: Body next: 37838 bytes
2023-05-21T21:43:00.199244Z TRACE garage_api::s3::put: Body next: 4194 bytes
2023-05-21T21:43:00.199643Z TRACE garage_api::s3::put: Body next: 8093 bytes
2023-05-21T21:43:00.954974Z TRACE garage_api::s3::put: Successfully validated content-md5
2023-05-21T21:43:01.210786Z DEBUG garage_api::generic_server: 200 OK {"x-amz-version-id": "6c1cd549ea6d319e00a739a60ca4d0fabc6d8049dd9fe079b4fbda4650f41b85", "etag": "\"ef2d287d38120caf7178d1d5f43263a3\""}
2023-05-21T21:43:01.280687Z  INFO garage_api::generic_server: REMOVED GET /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg
2023-05-21T21:43:01.280737Z DEBUG garage_api::generic_server: Request { method: GET, uri: /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg, version: HTTP/1.1, headers: {"host": REMOVED, "x-request-id": "faaafad551535dd40d7208541181945c", "x-real-ip": "REMOVED", "x-forwarded-for": "REMOVED", "x-forwarded-host": REMOVED, "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "x-original-forwarded-for": REMOVED, "cf-ipcountry": "DE", "cdn-loop": "cloudflare", "accept-encoding": "gzip", "cf-ray": "7caffe18cd483604-FRA", "cf-visitor": "{\"scheme\":\"https\"}", "user-agent": "MinIO (linux; x64) minio-js/7.1.0", "x-amz-date": "20230521T214301Z", "x-amz-content-sha256": "UNSIGNED-PAYLOAD", "authorization": "AWS4-HMAC-SHA256 Credential=REMOVED/20230521/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7463775a77db8092430d6bd41e0888d61cdb1f80188d96438eeeb46a0047a3ef", "cf-connecting-ip": REMOVED}, body: Body(Empty) }
2023-05-21T21:43:01.280863Z DEBUG garage_api::generic_server: Endpoint: GetObject
2023-05-21T21:43:01.280989Z TRACE garage_api::signature::payload: canonical request:
GET
/test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg

host:REMOVED
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20230521T214301Z

host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2023-05-21T21:43:01.281010Z TRACE garage_api::signature::payload: string to sign:
AWS4-HMAC-SHA256
20230521T214301Z
20230521/garage/s3/aws4_request
868ed552367b07661c31f08d76331df4abc8a93d5966ebbadafc32716dba51ef
2023-05-21T21:43:01.281403Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature

Super confusing.

I assume this is something Directus does internally?

So, I cloned the Directus repository (https://github.com/directus/directus) and implemented the S3 Driver with Minio (Node JS), instead of @aws-sdk/client-s3 (Node JS). Turns out, ... I get basically the same error. Directus Logs: ```JavaScript [21:43:01] POST /files 500 1.8s [21:43:01] GET /admin/assets/FiraMono-Medium-33e88fa9.woff2 200 2ms [21:43:01.291] ERROR: Valid and authorized credentials required err: { "type": "S3Error", "message": "Valid and authorized credentials required", "stack": S3Error: Valid and authorized credentials required at getError (/directus/node_modules/.pnpm/minio@7.1.0/node_modules/minio/dist/main/transformers.js:149:15) at /directus/node_modules/.pnpm/minio@7.1.0/node_modules/minio/dist/main/transformers.js:161:14 at DestroyableTransform._flush (/directus/node_modules/.pnpm/minio@7.1.0/node_modules/minio/dist/main/transformers.js:91:10) at DestroyableTransform.prefinish (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_transform.js:129:10) at DestroyableTransform.emit (node:events:513:28) at prefinish (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_writable.js:643:3) at Writable.end (/directus/node_modules/.pnpm/readable-stream@3.6.0/node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (node:internal/streams/readable:705:10) "code": "AccessDenied", "amzRequestid": null, "amzId2": null, "amzBucketRegion": null } ``` Garage Logs: ```Rust 2023-05-21T21:43:00.192148Z INFO garage_api::generic_server: REMOVED PUT /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg 2023-05-21T21:43:00.192355Z DEBUG garage_api::generic_server: Request { method: PUT, uri: /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg, version: HTTP/1.1, headers: {"host": REMOVED, "x-request-id": "07e5e595711ad21833527f3aadb5b8e1", "x-real-ip": "REMOVED", "x-forwarded-for": "REMOVED", "x-forwarded-host": REMOVED, "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "x-original-forwarded-for": REMOVED, "content-length": "133809", "cf-ipcountry": "DE", "cdn-loop": "cloudflare", "accept-encoding": "gzip", "cf-ray": "7caffe11dd370374-FRA", "cf-visitor": "{\"scheme\":\"https\"}", "user-agent": "MinIO (linux; x64) minio-js/7.1.0", "content-md5": "7y0ofTgSDK9xeNHV9DJjow==", "x-amz-date": "20230521T214300Z", "x-amz-content-sha256": "UNSIGNED-PAYLOAD", "authorization": "AWS4-HMAC-SHA256 Credential=REMOVED/20230521/garage/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=c14426516a24fc26e3787199eda2a9c69eae31437ed8d0daef805f9f73686f33", "cf-connecting-ip": REMOVED}, body: Body(Streaming) } 2023-05-21T21:43:00.193212Z DEBUG garage_api::generic_server: Endpoint: PutObject 2023-05-21T21:43:00.193955Z TRACE garage_api::signature::payload: canonical request: PUT /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg content-md5:7y0ofTgSDK9xeNHV9DJjow== host:REMOVED x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20230521T214300Z content-md5;host;x-amz-content-sha256;x-amz-date UNSIGNED-PAYLOAD 2023-05-21T21:43:00.194017Z TRACE garage_api::signature::payload: string to sign: AWS4-HMAC-SHA256 20230521T214300Z 20230521/garage/s3/aws4_request 5df924be79b7d70c9012366829c41be256775619f63d0febdc45d21d71475121 2023-05-21T21:43:00.197269Z DEBUG garage_api::s3::put: Object headers: ObjectVersionHeaders { content_type: "blob", other: {} } 2023-05-21T21:43:00.197434Z TRACE garage_api::s3::put: Body next: 7228 bytes 2023-05-21T21:43:00.197684Z TRACE garage_api::s3::put: Body next: 16384 bytes 2023-05-21T21:43:00.197865Z TRACE garage_api::s3::put: Body next: 32768 bytes 2023-05-21T21:43:00.198037Z TRACE garage_api::s3::put: Body next: 25906 bytes 2023-05-21T21:43:00.198616Z TRACE garage_api::s3::put: Body next: 1398 bytes 2023-05-21T21:43:00.198817Z TRACE garage_api::s3::put: Body next: 37838 bytes 2023-05-21T21:43:00.199244Z TRACE garage_api::s3::put: Body next: 4194 bytes 2023-05-21T21:43:00.199643Z TRACE garage_api::s3::put: Body next: 8093 bytes 2023-05-21T21:43:00.954974Z TRACE garage_api::s3::put: Successfully validated content-md5 2023-05-21T21:43:01.210786Z DEBUG garage_api::generic_server: 200 OK {"x-amz-version-id": "6c1cd549ea6d319e00a739a60ca4d0fabc6d8049dd9fe079b4fbda4650f41b85", "etag": "\"ef2d287d38120caf7178d1d5f43263a3\""} 2023-05-21T21:43:01.280687Z INFO garage_api::generic_server: REMOVED GET /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg 2023-05-21T21:43:01.280737Z DEBUG garage_api::generic_server: Request { method: GET, uri: /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg, version: HTTP/1.1, headers: {"host": REMOVED, "x-request-id": "faaafad551535dd40d7208541181945c", "x-real-ip": "REMOVED", "x-forwarded-for": "REMOVED", "x-forwarded-host": REMOVED, "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-forwarded-scheme": "https", "x-scheme": "https", "x-original-forwarded-for": REMOVED, "cf-ipcountry": "DE", "cdn-loop": "cloudflare", "accept-encoding": "gzip", "cf-ray": "7caffe18cd483604-FRA", "cf-visitor": "{\"scheme\":\"https\"}", "user-agent": "MinIO (linux; x64) minio-js/7.1.0", "x-amz-date": "20230521T214301Z", "x-amz-content-sha256": "UNSIGNED-PAYLOAD", "authorization": "AWS4-HMAC-SHA256 Credential=REMOVED/20230521/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7463775a77db8092430d6bd41e0888d61cdb1f80188d96438eeeb46a0047a3ef", "cf-connecting-ip": REMOVED}, body: Body(Empty) } 2023-05-21T21:43:01.280863Z DEBUG garage_api::generic_server: Endpoint: GetObject 2023-05-21T21:43:01.280989Z TRACE garage_api::signature::payload: canonical request: GET /test-directus/7eafdbbe-ee4c-4701-835a-519fb4305d42.jpg host:REMOVED x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20230521T214301Z host;x-amz-content-sha256;x-amz-date UNSIGNED-PAYLOAD 2023-05-21T21:43:01.281010Z TRACE garage_api::signature::payload: string to sign: AWS4-HMAC-SHA256 20230521T214301Z 20230521/garage/s3/aws4_request 868ed552367b07661c31f08d76331df4abc8a93d5966ebbadafc32716dba51ef 2023-05-21T21:43:01.281403Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature ``` Super confusing. I assume this is something Directus does internally?
lx added the
Bug
 label 2023-05-22 08:23:24 +00:00
lx commented 2023-05-22 08:26:47 +00:00
Owner
Copy Link

I don't see anything strange in the signature calculation, however it looks like you are behind some kind of cloudflare proxy? Could it be that the proxy is rewriting some of your headers in the GET request (e.g. the Host header) and this is the reason why signatures do not match on either sides?

I don't see anything strange in the signature calculation, however it looks like you are behind some kind of cloudflare proxy? Could it be that the proxy is rewriting some of your headers in the GET request (e.g. the Host header) and this is the reason why signatures do not match on either sides?
🎉 1
best-user-name-on-demand commented 2023-05-22 10:03:45 +00:00
Author
Copy Link

Eyoooooooooo!

That was the issue.

So I jumped to another DNS (Bunny CDN) that's easier to manage with and without proxy than CloudFlare:

CloudFlare with Proxy: Does not work
CloudFlare without Proxy: No idea, don't get TLS to work without proxy, CloudFlare (free tier) sometimes sucks hard

Bunny with Proxy: Does not work
Bunny without Proxy: -->>WORKS<<--

Weird.

Is this somehow fixable?

Technically, I intend to use the Garage API internally in the Kubernetes network as soon as I manage to make it work (had some issues with internal IP). So the problem can be fully avoided and I see the problem as solved for my use case.

However, is there not an incentive to make this work with Proxies and CDNs as well? That could be a real legitimate use case, e.g. caching.

Eyoooooooooo! That was the issue. So I jumped to another DNS (Bunny CDN) that's easier to manage with and without proxy than CloudFlare: CloudFlare with Proxy: **Does not work** CloudFlare without Proxy: No idea, don't get TLS to work without proxy, CloudFlare (free tier) sometimes sucks hard Bunny with Proxy: **Does not work** Bunny without Proxy: **-->>WORKS<<--** Weird. Is this somehow fixable? Technically, I intend to use the Garage API internally in the Kubernetes network as soon as I manage to make it work (had some issues with internal IP). So the problem can be fully avoided and I see the problem as solved for my use case. However, is there not an incentive to make this work with Proxies and CDNs as well? That could be a real legitimate use case, e.g. caching.
lx commented 2023-05-22 10:12:13 +00:00
Owner
Copy Link

Not sure this is fixable, it seems it depends on the way the proxy changes HTTP headers which is vendor-dependant and hard to predict. I think Garage works fine with some other reverse proxies such as Caddy. For the CDN use case, if the idea is to broadcast read-only content to a wide audience, then you can try using Garage's web endpoint which is publicly accessible without authentication.

Not sure this is fixable, it seems it depends on the way the proxy changes HTTP headers which is vendor-dependant and hard to predict. I think Garage works fine with some other reverse proxies such as Caddy. For the CDN use case, if the idea is to broadcast read-only content to a wide audience, then you can try using Garage's web endpoint which is publicly accessible without authentication.
lx closed this issue 2023-05-22 10:12:13 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
main-0.9.x
db-no-unsafe
main-0.8.x
k2v/shared_http_client
feat/website-redir
smithy2
test/compilation
poc/nomad-ci
test-ci-wait
pnet_datalink-0.33.0
optimal-layout
k2v-watch-range
db-debug-log
build-convert-db
demonstrate-cargo2nix-bug
dangerous/no-fsync
dev0.7.3
doc/benchmarks
bug/check_static
bucket-cors-more
storage-optimizations
old-main
v1.0.0
v0.9.4
v1.0.0-rc1
v0.8.7
v0.9.3
v0.9.2
v0.8.6
v0.9.2-rc1
v0.9.1
v0.8.5
v0.10.0-beta1
v0.9.0
v0.9.0-rc2
v0.9.0-rc1
v0.9.0-beta4
v0.9.0-beta3
v0.9.0-beta2
v0.8.4
v0.9.0-beta1
v0.8.3
v0.8.3-rc1
format_table-v0.1.1
format_table-v0.1.0
v0.8.2
v0.8.1
v0.8.0
v0.8.0-rc2
v0.8.0-rc1
v0.8.0-dangerous-no-fsync
v0.8.0-beta2-k2v
v0.8.0-beta2
v0.8.0-beta1-k2v
v0.8.0-beta1
v0.7.99.3-k2v
v0.7.3
v0.7.3-beta2
v0.7.3-beta1
v0.7.2.1
v0.7.2
v0.7.2_ci-test-2
v0.7.2+ci-test-version
v0.7.99.2-k2v
v0.7.99.1-k2v
v0.7.99-k2v
v0.7.2-k2v
v0.7.1-admin-k2v-2
v0.7.1-admin-k2v
v0.7.1-k2v
v0.7.1
v0.7.0
v0.7.0-rc1
v0.6.1
v0.6.0
v0.6.0-rc1
v0.5.1
v0.5.0.1
v0.5.0
v0.5-beta1
v0.4.0
v0.4-rc2
v0.4-rc1
0.4-beta
0.4-alpha
v0.3.0.2
v0.3.0.1
v0.3.0
v0.2.1.5
v0.2.1
v0.2.0
0.1.1b
0.1.1
0.1.0b
0.1.0
Labels
Clear labels   
AdminAPI

Issues concerning the administration API

  
Bug

   
Check AWS

The behaviour should be checked against AWS S3

  
CI

Issues with Nix, Drone, Rust Compiler, etc.

  
Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  
Critical

   
Documentation

   
Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  
Improvement

   
Low priority

For issues that we don't want to spend time on until made absolutely necessary

  
Newcomer

A bug that should be newcomer friendly.

  
Performance

   
S3 Compatibility

   
Testing

   
Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No Label
AdminAPI
Bug
Check AWS
CI
Correctness
Critical
Documentation
Ideas
Improvement
Low priority
Newcomer
Performance
S3 Compatibility
Testing
Usability
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#579
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?