Trying to see if Garage works as a backend state store for Apache Flink #975

New issue

Open

opened 2025-03-05 11:30:53 +00:00 by bodleytunes99 · 1 comment

bodleytunes99 commented

2025-03-05 11:30:53 +00:00

Hi There,

I was testing with my local instance, seeing if I can use Garage as a target bucket for Flink, but I've tried many different combo's of settings and keep getting a 403 as folllows.

Just wondering if this is something that can be got working by tuning some settings or whether I'm flogging a dead horse and should resign myself to use MinIo?

garage server:

[::ffff:10.12.177.4]:58362 GET /stg-flink-state-bucket/?list-type=2&delimiter=%2F&max-keys=2&prefix=ha%2Fjob-result-store%2Fbasic-example%2F32b41b4a-e68f-4d38-8e6b-7023feab9baa%2F&fetch-owner=false
2025-03-04T23:32:50.942505Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature
2025-03-04T23:34:04.374787Z  INFO garage_api::generic_server: [::ffff:10.12.177.4]:58482 GET /stg-flink-state-bucket/?list-type=2&delimiter=%2F&max-keys=2&prefix=ha%2Fjob-result-store%2Fbasic-example%2F32b41b4a-e68f-4d38-8e6b-7023feab9baa%2F&fetch-owner=false
2025-03-04T23:34:04.374981Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature

flink errors:

│ flink-main-container Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden: Invalid signature (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: null; S3 Extended Request ID: null; Proxy: null)                             │

    s3.endpoint: http://10.12.89.15:3900
    s3.path.style.access: "true"
    s3.endpoint-region: "garage"
    fs.s3a.endpoint.region: "garage"
    s3.connection.ssl.enabled: "false"
    s3.access-key: G___76d
    s3.secret-key: de0ec___

Thanks,

Jon.

Hi There, I was testing with my local instance, seeing if I can use Garage as a target bucket for Flink, but I've tried many different combo's of settings and keep getting a 403 as folllows. Just wondering if this is something that can be got working by tuning some settings or whether I'm flogging a dead horse and should resign myself to use MinIo? garage server: ``` [::ffff:10.12.177.4]:58362 GET /stg-flink-state-bucket/?list-type=2&delimiter=%2F&max-keys=2&prefix=ha%2Fjob-result-store%2Fbasic-example%2F32b41b4a-e68f-4d38-8e6b-7023feab9baa%2F&fetch-owner=false 2025-03-04T23:32:50.942505Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature 2025-03-04T23:34:04.374787Z INFO garage_api::generic_server: [::ffff:10.12.177.4]:58482 GET /stg-flink-state-bucket/?list-type=2&delimiter=%2F&max-keys=2&prefix=ha%2Fjob-result-store%2Fbasic-example%2F32b41b4a-e68f-4d38-8e6b-7023feab9baa%2F&fetch-owner=false 2025-03-04T23:34:04.374981Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature ``` flink errors: ``` │ flink-main-container Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden: Invalid signature (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: null; S3 Extended Request ID: null; Proxy: null) │ ``` ``` s3.endpoint: http://10.12.89.15:3900 s3.path.style.access: "true" s3.endpoint-region: "garage" fs.s3a.endpoint.region: "garage" s3.connection.ssl.enabled: "false" s3.access-key: G___76d s3.secret-key: de0ec___ ``` Thanks, Jon.

lx commented

2025-03-05 12:06:32 +00:00

Owner

Hello,

Thanks for trying out Garage.

Unless Flink is trying to use unsupported s3 features, which is not the case here, this should work. Invalid signature errors are generally caused by a bad configuration of the access credentials, or by reverse proxies that mess with the headers.

I don't see an obvious error in the configuration you gave, but it's worth double-checking. In particular, the region parameter should match the value you put in your garage.toml configuration file.

You can enable debug mode for the API part by setting the environment variable RUST_LOG=garage_api=debug,garage_api_common=debug,garage_api_s3=debug, your logs should then contain more details about the signature which could help us troubleshoot the issue.

Hello, Thanks for trying out Garage. Unless Flink is trying to use [unsupported s3 features](https://garagehq.deuxfleurs.fr/documentation/reference-manual/s3-compatibility/), which is not the case here, this should work. Invalid signature errors are generally caused by a bad configuration of the access credentials, or by reverse proxies that mess with the headers. I don't see an obvious error in the configuration you gave, but it's worth double-checking. In particular, the region parameter should match the value you put in your `garage.toml` configuration file. You can enable debug mode for the API part by setting the environment variable `RUST_LOG=garage_api=debug,garage_api_common=debug,garage_api_s3=debug`, your logs should then contain more details about the signature which could help us troubleshoot the issue.