Deuxfleurs/garage
41
56
Fork 73
Code Issues 99 Pull requests 14 Releases 30 Wiki Activity

helm chart improvements #425

Merged
maximilien merged 8 commits from patrickjahns/garage:helm-improvements into main 2023-01-27 10:51:05 +00:00
Conversation 3 Commits 8 Files changed 5 +141 -34
patrickjahns commented 2022-11-17 22:43:14 +00:00
Contributor
Copy link

This PR contains a couple of improvements to the helm chart:

  • The ability to override the (default) configuration template so that custom values can easily be injected. In the end, this simplifies the configuration in the helm chart, as it drops the need for exposing every garage configuration value into yaml.
    I would also like to discuss, that some of the values we currently have in the garage key can now be dropped and just moved into that configuration file

  • Some unused variables from the garage configuration key (namely metadataDir and dataDir ) were removed and hardcoded in the configuration (as the mountpoint in the pod was hardcoded anyway)

  • A change to the configuration will now trigger a rollout // rolling update of the pods (previously manual effort was required by restarting the pods to have configuration changes take effect)

  • The currently securityContext and podSecurityContext configuration lead to permissions issues on a couple of CSI drivers (tested with longhorn, openebs, openebs-lvm) as the user/group (1000/1000) of garage would not match the default permissions of the filesystem (root). By moving the configuration into the podSecurityContext and defining the default user for the group the filesystem will have when mounted, the problems are gone and this raises compatibility

Previous errors observed:

2022-11-16T20:38:07.348526Z ERROR garage: panicked at 'Unable to open sled DB: Io(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })', garage.rs:98:22
2022-11-16T20:38:07.348533Z ERROR garage: panicked at 'Unable to open sled DB: Io(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })', garage.rs:98:22

  • Ability to override the init container image - this allows to now specify a different registry (and tag) for the image, so that it's not bound to dockerhub

  • Added the ability to easily integrate with prometheus (operator) in kubernetes. A kubernetes service exposing the admin api is created, and if wanted, users can choose to also use ServiceMonitors so that the prometheus operator automatically detects the garage pods

TODO

  • bump version in Chart.yaml
This PR contains a couple of improvements to the helm chart: - The ability to override the (default) configuration template so that custom values can easily be injected. In the end, this simplifies the configuration in the helm chart, as it drops the need for exposing every garage configuration value into yaml. I would also like to discuss, that some of the values we currently have in the `garage` key can now be dropped and just moved into that configuration file - Some unused variables from the `garage` configuration key (namely `metadataDir` and `dataDir` ) were removed and hardcoded in the configuration (as the mountpoint in the pod was hardcoded anyway) - A change to the configuration will now trigger a rollout // rolling update of the pods (previously manual effort was required by restarting the pods to have configuration changes take effect) - The currently `securityContext` and `podSecurityContext` configuration lead to permissions issues on a couple of CSI drivers (tested with longhorn, openebs, openebs-lvm) as the user/group (1000/1000) of garage would not match the default permissions of the filesystem (root). By moving the configuration into the podSecurityContext and defining the default user for the group the filesystem will have when mounted, the problems are gone and this raises compatibility Previous errors observed: ``` 2022-11-16T20:38:07.348526Z ERROR garage: panicked at 'Unable to open sled DB: Io(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })', garage.rs:98:22 2022-11-16T20:38:07.348533Z ERROR garage: panicked at 'Unable to open sled DB: Io(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })', garage.rs:98:22 ``` - Ability to override the init container image - this allows to now specify a different registry (and tag) for the image, so that it's not bound to dockerhub - Added the ability to easily integrate with prometheus (operator) in kubernetes. A kubernetes service exposing the admin api is created, and if wanted, users can choose to also use `ServiceMonitors` so that the prometheus operator automatically detects the garage pods ### TODO - [ ] bump version in Chart.yaml
patrickjahns force-pushed helm-improvements from a2b30bd63f to e53693607b 2022-11-17 22:53:08 +00:00 Compare
maximilien self-assigned this 2022-11-18 09:22:12 +00:00
maximilien requested review from maximilien 2022-12-12 00:10:22 +00:00
maximilien reviewed 2022-12-12 00:26:54 +00:00
maximilien left a comment
Owner
Copy link

Looks good, I rebased it over main so you'll have to force-pull sorry, some comments but nothing blocking, let me know what you think

Looks good, I rebased it over main so you'll have to force-pull sorry, some comments but nothing blocking, let me know what you think
script/helm/garage/templates/service.yaml Outdated
@ -20,0 +22,4 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "garage.fullname" . }}-metrics
maximilien commented 2022-12-12 00:18:19 +00:00
Owner
Copy link

Actually this is the admin API endpoint, so maybe we should simply call that -admin?

Actually this is the admin API endpoint, so maybe we should simply call that `-admin`?
patrickjahns commented 2022-12-28 17:11:36 +00:00
Author
Contributor
Copy link

The initial thought here was, to keep the service definitions for admin and metrics by their own. So having two services - which can have individual configurations if need be. A thought here was, that if the user would for example expose the admin api via a LoadBalancer service, the metrics service would not be impacted.
Or adding annotations to the "admin service" i.e. for istio would not affect prometheus and vice/versa

Let me know if this changes your thoughts - or if you would like to proceed with renaming the ports

The initial thought here was, to keep the service definitions for admin and metrics by their own. So having two services - which can have individual configurations if need be. A thought here was, that if the user would for example expose the admin api via a LoadBalancer service, the metrics service would not be impacted. Or adding annotations to the "admin service" i.e. for istio would not affect prometheus and vice/versa Let me know if this changes your thoughts - or if you would like to proceed with renaming the ports
script/helm/garage/templates/service.yaml Outdated
@ -20,0 +34,4 @@
- port: 3903
targetPort: 3903
protocol: TCP
name: metrics
maximilien commented 2022-12-12 00:18:35 +00:00
Owner
Copy link

Same as above

Same as above
script/helm/garage/values.yaml Outdated
@ -44,1 +77,4 @@
initImage:
repository: busybox
tag: 1.28
maximilien commented 2022-12-12 00:22:58 +00:00
Owner
Copy link

Maybe we should target "stable" by default?

Maybe we should target "stable" by default?
patrickjahns marked this conversation as resolved
script/helm/garage/values.yaml Outdated
@ -148,0 +193,4 @@
serviceMonitor:
# If true, a ServiceMonitor CRD is created for a prometheus operator
# https://github.com/coreos/prometheus-operator
#
maximilien commented 2022-12-12 00:24:40 +00:00
Owner
Copy link

Remove newline?

Remove newline?
patrickjahns marked this conversation as resolved
patrickjahns commented 2022-12-28 17:33:17 +00:00
Author
Contributor
Copy link

As mentioned in #409 (comment) - we need to align both PRs to resolve minor conflicts and add the correct version to the helm chart

There is now also health endpoints (see: #440 ) - I would create a follow up pr to utilize them, once this is merged

As mentioned in https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/409#issuecomment-4705 - we need to align both PRs to resolve minor conflicts and add the correct version to the helm chart There is now also health endpoints (see: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/440 ) - I would create a follow up pr to utilize them, once this is merged
lx commented 2023-01-26 16:30:25 +00:00
Owner
Copy link

Is this alive?

Is this alive?
lx commented 2023-01-26 21:09:03 +00:00
Owner
Copy link

Conflicts with changes made in #409, can this be rebased?

Conflicts with changes made in #409, can this be rebased?
maximilien force-pushed helm-improvements from 94772a7c54 to 50bce43f25 2023-01-26 23:43:36 +00:00 Compare
maximilien merged commit df30f3df4b into main 2023-01-27 10:51:05 +00:00
patrickjahns deleted branch helm-improvements 2023-02-08 19:41:59 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
maximilien
Labels
Clear labels   
action
check-aws
The behaviour should be checked against AWS S3

  
action
discussion-needed
We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  
action
for-external-contributors
Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  
action
for-newcomers
An issue that should be newcomer friendly.

  
action
more-info-needed
More information is needed to investigate this issue

  
action
need-funding
This issue represents lots of work, we need to find money to pay our core team work

  
action
triage-required
It's not very clear what we want to do with this issue yet, more thoughts is required

  
kind
correctness
Issues of theoretical correctness that may impact edge-case scenarios

  
kind
ideas
For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  
kind
improvement
Feature requests that do not fall in other categories

  
kind
performance
Deceptive performance observed OR idea to improve performances

  
kind
testing
Functional tests, integration tests, Jepsen, etc.

  
kind
usability
Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  
kind
wrong-behavior
Unintended behavior

  
prio
critical
Could impact integrity or availability of production clusters

  
prio
low
For issues that we don't want to spend time on until made absolutely necessary

  
scope
admin-api
Issues concerning the administration API

  
scope
background-healing
Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  
scope
build
Issues with Nix, Drone, Rust Compiler, etc.

  
scope
documentation

  
scope
k8s
Kubernetes related, including Helm Charts

  
scope
layout
Linked to the cluster layout and storage management

  
scope
metadata
Metadata Engine (LMDB, sqlite, etc.)

  
scope
ops
Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  
scope
rpc
Linked to RPC

  
scope
s3-api
Related to the S3 API (especially compatibility issues).

  
scope
security
Cryptography-related, auth-related, authorization-related issues

  
scope
telemetry
OpenTelemetry (traces, metrics, etc.)

No labels
action
check-aws
action
discussion-needed
action
for-external-contributors
action
for-newcomers
action
more-info-needed
action
need-funding
action
triage-required
kind
correctness
kind
ideas
kind
improvement
kind
performance
kind
testing
kind
usability
kind
wrong-behavior
prio
critical
prio
low
scope
admin-api
scope
background-healing
scope
build
scope
documentation
scope
k8s
scope
layout
scope
metadata
scope
ops
scope
rpc
scope
s3-api
scope
security
scope
telemetry
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
maximilien
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#425
No description provided.
Delete branch "patrickjahns/garage:helm-improvements"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?