Deuxfleurs
/
garage
42
37
Fork 55
Code Issues 70 Pull Requests 7 Releases 29 Wiki Activity

helm chart improvements #425

Merged
maximilien merged 8 commits from patrickjahns/garage:helm-improvements into main 2023-01-27 10:51:05 +00:00
Conversation 3 Commits 8 Files Changed 5 +141 -34
patrickjahns commented 2022-11-17 22:43:14 +00:00
Contributor
Copy Link

This PR contains a couple of improvements to the helm chart:

  • The ability to override the (default) configuration template so that custom values can easily be injected. In the end, this simplifies the configuration in the helm chart, as it drops the need for exposing every garage configuration value into yaml.
    I would also like to discuss, that some of the values we currently have in the garage key can now be dropped and just moved into that configuration file

  • Some unused variables from the garage configuration key (namely metadataDir and dataDir ) were removed and hardcoded in the configuration (as the mountpoint in the pod was hardcoded anyway)

  • A change to the configuration will now trigger a rollout // rolling update of the pods (previously manual effort was required by restarting the pods to have configuration changes take effect)

  • The currently securityContext and podSecurityContext configuration lead to permissions issues on a couple of CSI drivers (tested with longhorn, openebs, openebs-lvm) as the user/group (1000/1000) of garage would not match the default permissions of the filesystem (root). By moving the configuration into the podSecurityContext and defining the default user for the group the filesystem will have when mounted, the problems are gone and this raises compatibility

Previous errors observed:

2022-11-16T20:38:07.348526Z ERROR garage: panicked at 'Unable to open sled DB: Io(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })', garage.rs:98:22
2022-11-16T20:38:07.348533Z ERROR garage: panicked at 'Unable to open sled DB: Io(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })', garage.rs:98:22

  • Ability to override the init container image - this allows to now specify a different registry (and tag) for the image, so that it's not bound to dockerhub

  • Added the ability to easily integrate with prometheus (operator) in kubernetes. A kubernetes service exposing the admin api is created, and if wanted, users can choose to also use ServiceMonitors so that the prometheus operator automatically detects the garage pods

TODO

  • bump version in Chart.yaml
This PR contains a couple of improvements to the helm chart: - The ability to override the (default) configuration template so that custom values can easily be injected. In the end, this simplifies the configuration in the helm chart, as it drops the need for exposing every garage configuration value into yaml. I would also like to discuss, that some of the values we currently have in the `garage` key can now be dropped and just moved into that configuration file - Some unused variables from the `garage` configuration key (namely `metadataDir` and `dataDir` ) were removed and hardcoded in the configuration (as the mountpoint in the pod was hardcoded anyway) - A change to the configuration will now trigger a rollout // rolling update of the pods (previously manual effort was required by restarting the pods to have configuration changes take effect) - The currently `securityContext` and `podSecurityContext` configuration lead to permissions issues on a couple of CSI drivers (tested with longhorn, openebs, openebs-lvm) as the user/group (1000/1000) of garage would not match the default permissions of the filesystem (root). By moving the configuration into the podSecurityContext and defining the default user for the group the filesystem will have when mounted, the problems are gone and this raises compatibility Previous errors observed: ``` 2022-11-16T20:38:07.348526Z ERROR garage: panicked at 'Unable to open sled DB: Io(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })', garage.rs:98:22 2022-11-16T20:38:07.348533Z ERROR garage: panicked at 'Unable to open sled DB: Io(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })', garage.rs:98:22 ``` - Ability to override the init container image - this allows to now specify a different registry (and tag) for the image, so that it's not bound to dockerhub - Added the ability to easily integrate with prometheus (operator) in kubernetes. A kubernetes service exposing the admin api is created, and if wanted, users can choose to also use `ServiceMonitors` so that the prometheus operator automatically detects the garage pods ### TODO - [ ] bump version in Chart.yaml
patrickjahns force-pushed helm-improvements from a2b30bd63f to e53693607b 2022-11-17 22:53:08 +00:00 Compare
maximilien self-assigned this 2022-11-18 09:22:12 +00:00
maximilien requested review from maximilien 2022-12-12 00:10:22 +00:00
maximilien reviewed 2022-12-12 00:26:54 +00:00
maximilien left a comment
Owner
Copy Link

Looks good, I rebased it over main so you'll have to force-pull sorry, some comments but nothing blocking, let me know what you think

Looks good, I rebased it over main so you'll have to force-pull sorry, some comments but nothing blocking, let me know what you think
script/helm/garage/templates/service.yaml Outdated
@ -20,0 +22,4 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "garage.fullname" . }}-metrics
maximilien commented 2022-12-12 00:18:19 +00:00
Owner
Copy Link

Actually this is the admin API endpoint, so maybe we should simply call that -admin?

Actually this is the admin API endpoint, so maybe we should simply call that `-admin`?
patrickjahns commented 2022-12-28 17:11:36 +00:00
Author
Contributor
Copy Link

The initial thought here was, to keep the service definitions for admin and metrics by their own. So having two services - which can have individual configurations if need be. A thought here was, that if the user would for example expose the admin api via a LoadBalancer service, the metrics service would not be impacted.
Or adding annotations to the "admin service" i.e. for istio would not affect prometheus and vice/versa

Let me know if this changes your thoughts - or if you would like to proceed with renaming the ports

The initial thought here was, to keep the service definitions for admin and metrics by their own. So having two services - which can have individual configurations if need be. A thought here was, that if the user would for example expose the admin api via a LoadBalancer service, the metrics service would not be impacted. Or adding annotations to the "admin service" i.e. for istio would not affect prometheus and vice/versa Let me know if this changes your thoughts - or if you would like to proceed with renaming the ports
script/helm/garage/templates/service.yaml Outdated
@ -20,0 +34,4 @@
- port: 3903
targetPort: 3903
protocol: TCP
name: metrics
maximilien commented 2022-12-12 00:18:35 +00:00
Owner
Copy Link

Same as above

Same as above
script/helm/garage/values.yaml Outdated
@ -44,1 +77,4 @@
initImage:
repository: busybox
tag: 1.28
maximilien commented 2022-12-12 00:22:58 +00:00
Owner
Copy Link

Maybe we should target "stable" by default?

Maybe we should target "stable" by default?
patrickjahns marked this conversation as resolved
script/helm/garage/values.yaml Outdated
@ -148,0 +193,4 @@
serviceMonitor:
# If true, a ServiceMonitor CRD is created for a prometheus operator
# https://github.com/coreos/prometheus-operator
#
maximilien commented 2022-12-12 00:24:40 +00:00
Owner
Copy Link

Remove newline?

Remove newline?
patrickjahns marked this conversation as resolved
patrickjahns commented 2022-12-28 17:33:17 +00:00
Author
Contributor
Copy Link

As mentioned in #409 (comment) - we need to align both PRs to resolve minor conflicts and add the correct version to the helm chart

There is now also health endpoints (see: #440 ) - I would create a follow up pr to utilize them, once this is merged

As mentioned in https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/409#issuecomment-4705 - we need to align both PRs to resolve minor conflicts and add the correct version to the helm chart There is now also health endpoints (see: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/440 ) - I would create a follow up pr to utilize them, once this is merged
lx commented 2023-01-26 16:30:25 +00:00
Owner
Copy Link

Is this alive?

Is this alive?
lx commented 2023-01-26 21:09:03 +00:00
Owner
Copy Link

Conflicts with changes made in #409, can this be rebased?

Conflicts with changes made in #409, can this be rebased?
maximilien force-pushed helm-improvements from 94772a7c54 to 50bce43f25 2023-01-26 23:43:36 +00:00 Compare
maximilien merged commit df30f3df4b into main 2023-01-27 10:51:05 +00:00
patrickjahns deleted branch helm-improvements 2023-02-08 19:41:59 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
maximilien
Labels
Clear labels   
AdminAPI

Issues concerning the administration API

  
Bug

   
Check AWS

The behaviour should be checked against AWS S3

  
CI

Issues with Nix, Drone, Rust Compiler, etc.

  
Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  
Critical

   
Documentation

   
Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  
Improvement

   
Low priority

For issues that we don't want to spend time on until made absolutely necessary

  
Newcomer

A bug that should be newcomer friendly.

  
Performance

   
S3 Compatibility

   
Testing

   
Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No Label
AdminAPI
Bug
Check AWS
CI
Correctness
Critical
Documentation
Ideas
Improvement
Low priority
Newcomer
Performance
S3 Compatibility
Testing
Usability
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
maximilien
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#425
No description provided.
Delete Branch "patrickjahns/garage:helm-improvements"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?