Blog post inroducing Garage v0.7 #6

Merged
lx merged 10 commits from v0.7 into master 2022-04-08 14:52:21 +00:00
4 changed files with 35 additions and 33 deletions
Showing only changes of commit b7cfe7c09c - Show all commits

View file

After

Width:  |  Height:  |  Size: 28 KiB

View file

After

Width:  |  Height:  |  Size: 117 KiB

View file

@ -9,27 +9,27 @@ date=2022-04-04
--- ---
Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our initial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute, and all requests were served by Garage without cache! Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our initial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, peaking at 100 views/minute, and all requests were served by Garage, without even using a caching frontend!
Since this event, we continued to improve Garage, and - 2 months after the initial release - we are happy to announce version 0.7.0. Since this event, we continued to improve Garage, and — 2 months after the initial release — we are happy to announce version 0.7.0.
But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a.
This is also our first time welcoming contributors external to the core team, and as we wish for Garage to be a community-driven project, we encourage it! This is also our first time welcoming contributors external to the core team, and as we wish for Garage to be a community-driven project, we encourage it!
You can get this release using our binaries or the package provided by your distribution. You can get this release using our binaries or the package provided by your distribution.
We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for most Linux architectures (amd64, i386, aarch64 and armv6) and associated [Docker containers](https://hub.docker.com/u/dxflrs). We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for most common Linux architectures (amd64, i386, aarch64 and armv6) and associated [Docker containers](https://hub.docker.com/u/dxflrs).
Garage now is also packaged by third parties on some OS/distributions. We are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage). Garage now is also packaged by third parties on some OS/distributions. We are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage).
Feel free to [reach out to us](mailto:garagehq@deuxfleurs.fr) if you are packaging (or planning to package) Garage, we welcome maintainers and will upstream specific patches if that can help. If you already did package Garage, tell us and we'll add it to the documentation. Feel free to [reach out to us](mailto:garagehq@deuxfleurs.fr) if you are packaging (or planning to package) Garage; we welcome maintainers and will upstream specific patches if that can help. If you already did package Garage, please inform us and we'll add it to the documentation.
Speaking about the changes of this new version, it obviously includes many bug fixes. Speaking about the changes of this new version, it obviously includes many bug fixes.
We listed them in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), take a look, we might have fixed something that annoyed you! We listed them in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), so take a look, we might have fixed some issues you were having!
Besides bugfixes, there is two new features: a better Kubernetes integration and support for OpenTelemetry. Besides bugfixes, there are two new major features in this release: better integration with Kubernetes, and support for observability via OpenTelemetry.
## Kubernetes integration ## Kubernetes integration
Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on [Kubernetes](https://kubernetes.io) (K8S). Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on [Kubernetes](https://kubernetes.io) (K8S).
In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CR) to simplify cluster discovery. In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CR) to simplify cluster discovery.
CR discovery can be quickly enabled by configuring the name of the desired service (`kubernetes_namespace`) and which namespace to look for (`kubernetes_service_name`) in your Garage config: CR discovery can be quickly enabled in Garage, by configuring the name of the desired service (`kubernetes_namespace`) and which namespace to look for (`kubernetes_service_name`) in your Garage configuration file:
```toml ```toml
kubernetes_namespace = "default" kubernetes_namespace = "default"
@ -37,8 +37,8 @@ kubernetes_service_name = "garage-daemon"
``` ```
Custom Resources must be defined *a priori* with [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD). Custom Resources must be defined *a priori* with [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD).
If the CRD does not exist, Garage can create it for you. It is enabled by default but it requires some additional permissions. If the CRD does not exist, Garage will create it for you. Automatic CRD creation is enabled by default, but it requires giving additional permissions to Garage to work.
If you prefer limiting accesses to your K8S cluster, you can create the resource manually and prevent Garage from automatically creating it: If you prefer strictly controlling access to your K8S cluster, you can create the resource manually and prevent Garage from automatically creating it:
```toml ```toml
kubernetes_skip_crd = true kubernetes_skip_crd = true
@ -59,14 +59,14 @@ kubectl exec -it garage-0 --container garage -- /garage status
You can then follow the [regular documentation](https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/#creating-a-cluster-layout) to complete the configuration of your cluster. You can then follow the [regular documentation](https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/#creating-a-cluster-layout) to complete the configuration of your cluster.
If you target a production deployment, you should avoid binding admin rights to your cluster to create Garage's CRD. You will also need to expose some [Services](https://kubernetes.io/docs/concepts/services-networking/service/) to make your cluster reachable. Keep also in mind that Garage is a stateful service, so you must be very careful how you handle your data in Kubernetes to not lose them. In the near future, we plan to release a proper Helm chart and write "best practises" on our documentation. If you target a production deployment, you should avoid binding admin rights to your cluster to create Garage's CRD. You will also need to expose some [Services](https://kubernetes.io/docs/concepts/services-networking/service/) to make your cluster reachable. Keep also in mind that Garage is a stateful service, so you must be very careful of how you handle your data in Kubernetes in order not to lose it. In the near future, we plan to release a proper Helm chart and write "best practises" in our documentation.
If Kubernetes is not your thing, know that we are running Garage on a Nomad+Consul cluster. If Kubernetes is not your thing, know that we are running Garage on a Nomad+Consul cluster, which is also well supported.
We have not documented it yet but you can get a look at [our Nomad service](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/commit/1e5e4af35c073d04698bb10dd4ad1330d6c62a0d/app/garage/deploy/garage.hcl). We have not documented it yet but you can get a look at [our Nomad service](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/commit/1e5e4af35c073d04698bb10dd4ad1330d6c62a0d/app/garage/deploy/garage.hcl).
## OpenTelemetry support ## OpenTelemetry support
[OpenTelemetry](https://opentelemetry.io/) standardizes how software generate and collect system telemetry, namely metrics, logs and traces. [OpenTelemetry](https://opentelemetry.io/) standardizes how software generates and collects system telemetry information, namely metrics, logs and traces.
By implementing this standard in Garage, we hope that it will help you to better monitor, manage and tune your cluster. By implementing this standard in Garage, we hope that it will help you to better monitor, manage and tune your cluster.
Note that to fully leverage this feature, you must be already familiar with monitoring stacks like [Prometheus](https://prometheus.io/)+[Grafana](https://grafana.com/) or [ElasticSearch](https://www.elastic.co/elasticsearch/)+[Kibana](https://www.elastic.co/kibana/). Note that to fully leverage this feature, you must be already familiar with monitoring stacks like [Prometheus](https://prometheus.io/)+[Grafana](https://grafana.com/) or [ElasticSearch](https://www.elastic.co/elasticsearch/)+[Kibana](https://www.elastic.co/kibana/).
@ -78,55 +78,57 @@ api_bind_addr = "127.0.0.1:3903"
trace_sink = "http://localhost:4317" trace_sink = "http://localhost:4317"
``` ```
The first line, `api_bind_address`, instructs Garage to expose an HTTP endpoint from which metrics can be obtained in Prometheus' data format.
The second line, `trace_sink`, instructs Garage to export tracing information to an OpenTelemetry collector at the given address.
These two options work independently and you can use them separately, depending on if you are interested only in metrics, traces, or both.
We provide [some files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/script/telemetry) to help you quickly bootstrap a testing monitoring stack. We provide [some files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/script/telemetry) to help you quickly bootstrap a testing monitoring stack.
It includes a docker-compose file and a pre-configured Grafana dashboard. It includes a docker-compose file and a pre-configured Grafana dashboard.
You can use them if you want to reproduce the following examples. You can use them if you want to reproduce the following examples.
Now that your telemetry data is collected and stored, you can visualize it.
Grafana is particularly adapted to understand how your cluster is performing from a "bird's eye view". Grafana is particularly adapted to understand how your cluster is performing from a "bird's eye view".
For example, the following graph shows S3 API calls sent to your node per time-unit, For example, the following graph shows S3 API calls sent to your node per time-unit.
you can use it to better understand how your users are interacting with your cluster. You can use it to better understand how your users are interacting with your cluster.
![A screenshot of a plot made by Grafana depicting the number of requests per time units grouped by endpoints](/images/blog/api_rate.png) ![A screenshot of a plot made by Grafana depicting the number of requests per time units grouped by endpoints](api_rate.png)
Thanks to this graph, we know that starting at 14:55, an important upload has been started. Thanks to this graph, we know that starting at 14:55, an important upload has been started.
This upload is made of many small files, as we see many PutObject calls that are often used for small files. This upload is made of many small files, as we see many PutObject calls that are often used for small files.
It also has some large objects, as we observe some Multipart Uploads requests. It also has some large objects, as we observe some multipart uploads requests.
Conversely, at this time, no read are done as the corresponding read enpoints (ListBuckets, ListObjectsv2, etc.) receive 0 request per time unit. Conversely, at this time, no read are done as the corresponding read enpoints (ListBuckets, ListObjectsV2, etc.) receive 0 request per time unit.
Garage also collects metrics from lower level parts of the system. Garage also collects metrics from lower level parts of the system.
You can use them to better understand how Garage is interacting with your OS and your hardware. You can use them to better understand how Garage is interacting with your OS and your hardware.
![A screenshot of a plot made by Grafana depicting the write speed (in MB/s) during time.](/images/blog/writes.png) ![A screenshot of a plot made by Grafana depicting the write speed (in MB/s) during time.](writes.png)
This plot has been captured at the same moment than the previous one. This plot has been captured at the same moment than the previous one.
We do not see a correlation between the writes and the API requests for the full upload but only for its beginning. We do not see a correlation between the writes and the API requests for the full upload but only for its beginning.
More precisely, it maps well to Multipart Uploads requests, and this is expected. More precisely, it maps well to multipart upload requests, and this is expected.
Large files (of the Multipart Uploads) will saturate the writes of your disk but the uploading of small files (via the PutObject endpoint) will be throttled by other parts of the system. Large files (of the multipart uploads) will saturate the writes of your disk but the uploading of small files (via the PutObject endpoint) will be limited by other parts of the system.
This simple example covers only 2 metrics over the 20+ ones that we already defined, but we were still able to precisely describe our cluster usage and identifies where bottlenecks could be. This simple example covers only 2 metrics over the 20+ ones that we already defined, but it still allowed us to precisely describe our cluster usage and identify where bottlenecks could be.
We are confident that cleverly using these metrics on a production cluster will give you many more valuable insights on your cluster. We are confident that cleverly using these metrics on a production cluster will give you many more valuable insights on your cluster.
While metrics are good to have a large, general overview of your system, they are however not adapted to dig and pinpoint a specific performance issue on a specific code path. While metrics are good for having a large, general overview of your system, they are however not adapted for digging and pinpointing a specific performance issue on a specific code path.
Thankfully, we also have a solution for this problem: tracing. Thankfully, we also have a solution for this problem: tracing.
Using [Application Performance Monitoring](https://www.elastic.co/observability/application-performance-monitoring) (APM) in conjunction with Kibana, Using [Application Performance Monitoring](https://www.elastic.co/observability/application-performance-monitoring) (APM) in conjunction with Kibana,
we get the following visualization: we can get for instance the following visualization of what happens during a PutObject call (click to enlarge):
![A screenshot of APM depicting the trace of a PutObject call](/images/blog/apm.png) [![A screenshot of APM depicting the trace of a PutObject call](apm.png)](apm.png)
On the top of the screenshot, we see the latency distribution of all PutObject requests. On the top of the screenshot, we see the latency distribution of all PutObject requests.
We learn that the selected request took ~1ms to execute, while 95% of all requests took less than 80ms to run. We learn that the selected request took ~1ms to execute, while 95% of all requests took less than 80ms to run.
Having some dispersion between requests is expected as Garage does not run on a strong real-time system, but in this case, you must also consider that Having some dispersion between requests is expected as Garage does not run on a strong real-time system, but in this case, you must also consider that
a request duration is impacted by the size of the object that is sent (a 10B object will be quicker to process than a 10MB one). a request duration is impacted by the size of the object that is sent (a 10B object will be quicker to process than a 10MB one).
Consequently, this request corresponds probably to a very tiny file. Consequently, this request probably corresponds to a very small file.
Below this first histogram, you can select the request you want to inspect, and then see its stacktrace on the bottom part. Below this first histogram, you can select the request you want to inspect, and then see its trace on the bottom part.
You can break down this trace in 4 parts: fetching the API key to check authentication (`key get`), fetching the bucket identifier from its name (`bucket_alias get`), fetching the bucket configuration to check authorizations (`bucket_v2 get`), and finally inserting the object in the storage (`object insert`). The trace shown above can be broken down in 4 parts: fetching the API key to check authentication (`key get`), fetching the bucket identifier from its name (`bucket_alias get`), fetching the bucket configuration to check authorizations (`bucket_v2 get`), and finally inserting the object in the storage (`object insert`).
With this example, we demonstrated that we can inspect Garage internals to find slow requests, then see which codepath has been taken by a request, to finally identify which part of the code took time. With this example, we demonstrated that we can inspect Garage internals to find slow requests, then see which codepath has been taken by a request, and finally to identify which part of the code took time.
Keep in mind that this is our first iteration on telemetry for Garage, so things are a bit rough around the edges (step by step documentation is missing, our Grafana dashboard is a work in a progress, etc.). Keep in mind that this is our first iteration on telemetry for Garage, so things are a bit rough around the edges (step by step documentation is missing, our Grafana dashboard is a work in a progress, etc.).
In all cases, your feedback is welcome on our Matrix channel. In all cases, your feedback is welcome on our Matrix channel.
@ -134,8 +136,8 @@ In all cases, your feedback is welcome on our Matrix channel.
## Conclusion ## Conclusion
This is only the first iteration of the Kubernetes and OpenTelemetry into Garage, so things are still a bit rough. This is only the first iteration of the Kubernetes and OpenTelemetry integrations in Garage, so things are still a bit rough.
We plan to polish their integration in the coming months based on our experience and your feedback. We plan to polish their integration in the coming months based on our experience and your feedback.
You may also ask yourself what will be the other works we plan to conduct: stay tuned, we will release a roadmap soon! You may also ask yourself what will be the other works we plan to conduct: stay tuned, we will soon release information on our roadmap!
In the mean time, we hope you will enjoy Garave v0.7! In the mean time, we hope you will enjoy using Garage v0.7.

View file

After

Width:  |  Height:  |  Size: 18 KiB