cargo: Updated gethostname v0.2.3 -> v0.4.3.

* addr2line v0.19.0 -> v0.20.0
 * async-compression v0.4.0 -> v0.4.1
 * clap v4.3.8 -> v4.3.19
 * hyper v0.14.26 -> v0.14.27
 * ipnet v2.7.2 -> v2.8.0
 * rmp v0.8.11 -> v0.8.12
 * serde v1.0.164 -> v1.0.188
 * tokio v1.29.0 -> v1.31.0
 * zstd v0.12.3+zstd.1.5.2 -> v0.12.4
 * Others in `cargo update`

doc/book/connect/backup.md

@@ -105,6 +105,7 @@ restic restore 79766175 --target /var/lib/postgresql
 Restic has way more features than the ones presented here.
 You can discover all of them by accessing its documentation from the link below.
 
+Files on Android devices can also be backed up with [restic-android](https://github.com/lhns/restic-android).
 
 *External links:* [Restic Documentation > Amazon S3](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#amazon-s3)
 

doc/book/cookbook/reverse-proxy.md

@@ -378,6 +378,47 @@ admin.garage.tld {
 But at the same time, the `reverse_proxy` is very flexible.
 For a production deployment, you should [read its documentation](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy) as it supports features like DNS discovery of upstreams, load balancing with checks, streaming parameters, etc.
 
+### Caching
+
+Caddy can compiled with a
+[cache plugin](https://github.com/caddyserver/cache-handler) which can be used
+to provide a hot-cache at the webserver-level for static websites hosted by
+Garage.
+
+This can be configured as follows:
+
+```caddy
+# Caddy global configuration section
+{
+	# Bare minimum configuration to enable cache.
+	order cache before rewrite
+
+	cache
+
+	#cache
+	#	allowed_http_verbs GET
+	#	default_cache_control public
+	#	ttl 8h
+	#}
+}
+
+# Site specific section
+https:// {
+	cache
+
+	#cache {
+	#	timeout {
+	#		backend 30s
+	#	}
+	#}
+
+	reverse_proxy ...
+}
+```
+
+Caching is a complicated subject, and the reader is encouraged to study the
+available options provided by the plugin.
+
 ### On-demand TLS
 
 Caddy supports a technique called
@@ -428,3 +469,6 @@ https:// {
 	reverse_proxy localhost:3902 192.168.1.2:3902 example.tld:3902
 }
 ```
+
+More information on how this endpoint is implemented in Garage is available
+in the [Admin API Reference](@/documentation/reference-manual/admin-api.md) page.

doc/book/reference-manual/admin-api.md

@@ -39,11 +39,95 @@ Authorization: Bearer <token>
 
 ## Administration API endpoints
 
-### Metrics-related endpoints
-
-#### Metrics `GET /metrics`
+### Metrics `GET /metrics`
 
 Returns internal Garage metrics in Prometheus format.
+The metrics are directly documented when returned by the API.
+
+**Example:**
+
+```
+$ curl -i http://localhost:3903/metrics
+HTTP/1.1 200 OK
+content-type: text/plain; version=0.0.4
+content-length: 12145
+date: Tue, 08 Aug 2023 07:25:05 GMT
+
+# HELP api_admin_error_counter Number of API calls to the various Admin API endpoints that resulted in errors
+# TYPE api_admin_error_counter counter
+api_admin_error_counter{api_endpoint="CheckWebsiteEnabled",status_code="400"} 1
+api_admin_error_counter{api_endpoint="CheckWebsiteEnabled",status_code="404"} 3
+# HELP api_admin_request_counter Number of API calls to the various Admin API endpoints
+# TYPE api_admin_request_counter counter
+api_admin_request_counter{api_endpoint="CheckWebsiteEnabled"} 7
+api_admin_request_counter{api_endpoint="Health"} 3
+# HELP api_admin_request_duration Duration of API calls to the various Admin API endpoints
+...
+```
+
+### Health `GET /health`
+
+Returns `200 OK` if enough nodes are up to have a quorum (ie. serve requests),
+otherwise returns `503 Service Unavailable`.
+
+**Example:**
+
+```
+$ curl -i http://localhost:3903/health
+HTTP/1.1 200 OK
+content-type: text/plain
+content-length: 102
+date: Tue, 08 Aug 2023 07:22:38 GMT
+
+Garage is fully operational
+Consult the full health check API endpoint at /v0/health for more details
+```
+
+### On-demand TLS `GET /check`
+
+To prevent abuses for on-demand TLS, Caddy developpers have specified an endpoint that can be queried by the reverse proxy
+to know if a given domain is allowed to get a certificate. Garage implements this endpoints to tell if a given domain is handled by Garage or is garbage.
+
+Garage responds with the following logic:
+ - If the domain matches the pattern `<bucket-name>.<s3_api.root_domain>`, returns 200 OK
+ - If the domain matches the pattern `<bucket-name>.<s3_web.root_domain>` and website is configured for `<bucket>`, returns 200 OK
+ - If the domain matches the pattern `<bucket-name>` and website is configured for `<bucket>`, returns 200 OK
+ - Otherwise, returns 404 Not Found, 400 Bad Request or 5xx requests.
+
+*Note 1: because in the path-style URL mode, there is only one domain that is not known by Garage, hence it is not supported by this API endpoint.
+You must manually declare the domain in your reverse-proxy. Idem for K2V.*
+
+*Note 2: buckets in a user's namespace are not supported yet by this endpoint. This is a limitation of this endpoint currently.*
+
+**Example:** Suppose a Garage instance configured with `s3_api.root_domain = .s3.garage.localhost` and `s3_web.root_domain = .web.garage.localhost`.
+
+With a private `media` bucket (name in the global namespace, website is disabled), the endpoint will feature the following behavior:
+
+```
+$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=media.s3.garage.localhost
+200
+$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=media
+400
+$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=media.web.garage.localhost
+400
+```
+
+With a public `example.com` bucket (name in the global namespace, website is activated), the endpoint will feature the following behavior:
+
+```
+$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=example.com.s3.garage.localhost
+200
+$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=example.com
+200
+$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=example.com.web.garage.localhost
+200
+```
+
+
+**References:**
+ - [Using On-Demand TLS](https://caddyserver.com/docs/automatic-https#using-on-demand-tls)
+ - [Add option for a backend check to approve use of on-demand TLS](https://github.com/caddyserver/caddy/pull/1939)
+ - [Serving tens of thousands of domains over HTTPS with Caddy](https://caddy.community/t/serving-tens-of-thousands-of-domains-over-https-with-caddy/11179)
 
 ### Cluster operations
 

src/api/admin/api_server.rs

@@ -26,6 +26,7 @@ use crate::admin::cluster::*;
 use crate::admin::error::*;
 use crate::admin::key::*;
 use crate::admin::router::{Authorization, Endpoint};
+use crate::helpers::host_to_bucket;
 
 pub struct AdminApiServer {
 	garage: Arc<Garage>,
@@ -78,10 +79,7 @@ impl AdminApiServer {
 			.body(Body::empty())?)
 	}
 
-	async fn handle_check_website_enabled(
-		&self,
-		req: Request<Body>,
-	) -> Result<Response<Body>, Error> {
+	async fn handle_check_domain(&self, req: Request<Body>) -> Result<Response<Body>, Error> {
 		let query_params: HashMap<String, String> = req
 			.uri()
 			.query()
@@ -102,12 +100,43 @@ impl AdminApiServer {
 			.get("domain")
 			.ok_or_internal_error("Could not parse domain query string")?;
 
+		// Resolve bucket from domain name, inferring if the website must be activated for the
+		// domain to be valid.
+		let (bucket_name, must_check_website) = if let Some(bname) = self
+			.garage
+			.config
+			.s3_api
+			.root_domain
+			.as_ref()
+			.and_then(|rd| host_to_bucket(domain, rd))
+		{
+			(bname.to_string(), false)
+		} else if let Some(bname) = self
+			.garage
+			.config
+			.s3_web
+			.as_ref()
+			.and_then(|sw| host_to_bucket(domain, sw.root_domain.as_str()))
+		{
+			(bname.to_string(), true)
+		} else {
+			(domain.to_string(), true)
+		};
+
 		let bucket_id = self
 			.garage
 			.bucket_helper()
-			.resolve_global_bucket_name(domain)
+			.resolve_global_bucket_name(&bucket_name)
 			.await?
-			.ok_or(HelperError::NoSuchBucket(domain.to_string()))?;
+			.ok_or(HelperError::NoSuchBucket(bucket_name.to_string()))?;
+
+		if !must_check_website {
+			return Ok(Response::builder()
+				.status(StatusCode::OK)
+				.body(Body::from(format!(
+					"Domain '{domain}' is managed by Garage"
+				)))?);
+		}
 
 		let bucket = self
 			.garage
@@ -123,11 +152,11 @@ impl AdminApiServer {
 				Ok(Response::builder()
 					.status(StatusCode::OK)
 					.body(Body::from(format!(
-						"Bucket '{domain}' is authorized for website hosting"
+						"Domain '{domain}' is managed by Garage"
 					)))?)
 			}
 			None => Err(Error::bad_request(format!(
-				"Bucket '{domain}' is not authorized for website hosting"
+				"Domain '{domain}' is not managed by Garage"
 			))),
 		}
 	}
@@ -229,7 +258,7 @@ impl ApiHandler for AdminApiServer {
 
 		match endpoint {
 			Endpoint::Options => self.handle_options(&req),
-			Endpoint::CheckWebsiteEnabled => self.handle_check_website_enabled(req).await,
+			Endpoint::CheckDomain => self.handle_check_domain(req).await,
 			Endpoint::Health => self.handle_health(),
 			Endpoint::Metrics => self.handle_metrics(),
 			Endpoint::GetClusterStatus => handle_get_cluster_status(&self.garage).await,

src/api/admin/router.rs

@@ -17,7 +17,7 @@ router_match! {@func
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub enum Endpoint {
 	Options,
-	CheckWebsiteEnabled,
+	CheckDomain,
 	Health,
 	Metrics,
 	GetClusterStatus,
@@ -92,7 +92,7 @@ impl Endpoint {
 
 		let res = router_match!(@gen_path_parser (req.method(), path, query) [
 			OPTIONS _ => Options,
-			GET "/check" => CheckWebsiteEnabled,
+			GET "/check" => CheckDomain,
 			GET "/health" => Health,
 			GET "/metrics" => Metrics,
 			GET "/v0/status" => GetClusterStatus,
@@ -138,7 +138,7 @@ impl Endpoint {
 	pub fn authorization_type(&self) -> Authorization {
 		match self {
 			Self::Health => Authorization::None,
-			Self::CheckWebsiteEnabled => Authorization::None,
+			Self::CheckDomain => Authorization::None,
 			Self::Metrics => Authorization::MetricsToken,
 			_ => Authorization::AdminToken,
 		}

src/api/s3/post_object.rs

@@ -30,7 +30,7 @@ pub async fn handle_post_object(
 		.get(header::CONTENT_TYPE)
 		.and_then(|ct| ct.to_str().ok())
 		.and_then(|ct| multer::parse_boundary(ct).ok())
-		.ok_or_bad_request("Counld not get multipart boundary")?;
+		.ok_or_bad_request("Could not get multipart boundary")?;
 
 	// 16k seems plenty for a header. 5G is the max size of a single part, so it seems reasonable
 	// for a PostObject
@@ -64,15 +64,13 @@ pub async fn handle_post_object(
 				"tag" => (/* tag need to be reencoded, but we don't support them yet anyway */),
 				"acl" => {
 					if params.insert("x-amz-acl", content).is_some() {
-						return Err(Error::bad_request(
-							"Field 'acl' provided more than one time",
-						));
+						return Err(Error::bad_request("Field 'acl' provided more than once"));
 					}
 				}
 				_ => {
 					if params.insert(&name, content).is_some() {
 						return Err(Error::bad_request(format!(
-							"Field '{}' provided more than one time",
+							"Field '{}' provided more than once",
 							name
 						)));
 					}
@@ -149,7 +147,7 @@ pub async fn handle_post_object(
 		.ok_or_bad_request("Invalid expiration date")?
 		.into();
 	if Utc::now() - expiration > Duration::zero() {
-		return Err(Error::bad_request("Expiration date is in the paste"));
+		return Err(Error::bad_request("Expiration date is in the past"));
 	}
 
 	let mut conditions = decoded_policy.into_conditions()?;
@@ -330,7 +328,7 @@ impl Policy {
 					if map.len() != 1 {
 						return Err(Error::bad_request("Invalid policy item"));
 					}
-					let (mut k, v) = map.into_iter().next().expect("size was verified");
+					let (mut k, v) = map.into_iter().next().expect("Size could not be verified");
 					k.make_ascii_lowercase();
 					params.entry(k).or_default().push(Operation::Equal(v));
 				}

src/garage/tests/s3/website.rs

@@ -72,7 +72,7 @@ async fn test_website() {
 		res_body,
 		json!({
 			"code": "InvalidRequest",
-			"message": "Bad request: Bucket 'my-website' is not authorized for website hosting",
+			"message": "Bad request: Domain 'my-website' is not managed by Garage",
 			"region": "garage-integ-test",
 			"path": "/check",
 		})
@@ -91,24 +91,29 @@ async fn test_website() {
 		BODY.as_ref()
 	);
 
-	let admin_req = || {
-		Request::builder()
-			.method("GET")
-			.uri(format!(
-				"http://127.0.0.1:{0}/check?domain={1}",
-				ctx.garage.admin_port,
-				BCKT_NAME.to_string()
-			))
-			.body(Body::empty())
-			.unwrap()
-	};
+	for bname in [
+		BCKT_NAME.to_string(),
+		format!("{BCKT_NAME}.web.garage"),
+		format!("{BCKT_NAME}.s3.garage"),
+	] {
+		let admin_req = || {
+			Request::builder()
+				.method("GET")
+				.uri(format!(
+					"http://127.0.0.1:{0}/check?domain={1}",
+					ctx.garage.admin_port, bname
+				))
+				.body(Body::empty())
+				.unwrap()
+		};
 
-	let mut admin_resp = client.request(admin_req()).await.unwrap();
-	assert_eq!(admin_resp.status(), StatusCode::OK);
-	assert_eq!(
-		to_bytes(admin_resp.body_mut()).await.unwrap().as_ref(),
-		format!("Bucket '{BCKT_NAME}' is authorized for website hosting").as_bytes()
-	);
+		let mut admin_resp = client.request(admin_req()).await.unwrap();
+		assert_eq!(admin_resp.status(), StatusCode::OK);
+		assert_eq!(
+			to_bytes(admin_resp.body_mut()).await.unwrap().as_ref(),
+			format!("Domain '{bname}' is managed by Garage").as_bytes()
+		);
+	}
 
 	ctx.garage
 		.command()
@@ -142,7 +147,7 @@ async fn test_website() {
 		res_body,
 		json!({
 			"code": "InvalidRequest",
-			"message": "Bad request: Bucket 'my-website' is not authorized for website hosting",
+			"message": "Bad request: Domain 'my-website' is not managed by Garage",
 			"region": "garage-integ-test",
 			"path": "/check",
 		})
@@ -397,7 +402,7 @@ async fn test_website_s3_api() {
 }
 
 #[tokio::test]
-async fn test_website_check_website_enabled() {
+async fn test_website_check_domain() {
 	let ctx = common::context();
 
 	let client = Client::new();