Compare commits

..

5 commits

Author SHA1 Message Date
Jonathan Davies 2b1e66afc8 cargo: Updated gethostname v0.2.3 -> v0.4.3.
Some checks failed
continuous-integration/drone/pr Build is failing
2023-08-27 19:20:38 +01:00
Jonathan Davies e306af6414 cargo: Updated pretty_env_logger v0.4.0 -> v0.5.0. 2023-08-27 19:20:38 +01:00
Jonathan Davies 5fccab5a48 garage/api: cargo: Updated idna dependency to 0.4. 2023-08-27 19:20:38 +01:00
Jonathan Davies 3c9f4d5519 garage/db: cargo: Updated rusqlite to 0.29. 2023-08-27 19:20:38 +01:00
Jonathan Davies 1f1a8fa428 cargo: Updated:
* addr2line v0.19.0 -> v0.20.0
 * async-compression v0.4.0 -> v0.4.1
 * clap v4.3.8 -> v4.3.19
 * hyper v0.14.26 -> v0.14.27
 * ipnet v2.7.2 -> v2.8.0
 * rmp v0.8.11 -> v0.8.12
 * serde v1.0.164 -> v1.0.188
 * tokio v1.29.0 -> v1.31.0
 * zstd v0.12.3+zstd.1.5.2 -> v0.12.4
 * Others in `cargo update`
2023-08-27 19:20:33 +01:00
7 changed files with 42 additions and 203 deletions

View file

@ -105,7 +105,6 @@ restic restore 79766175 --target /var/lib/postgresql
Restic has way more features than the ones presented here.
You can discover all of them by accessing its documentation from the link below.
Files on Android devices can also be backed up with [restic-android](https://github.com/lhns/restic-android).
*External links:* [Restic Documentation > Amazon S3](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#amazon-s3)

View file

@ -378,47 +378,6 @@ admin.garage.tld {
But at the same time, the `reverse_proxy` is very flexible.
For a production deployment, you should [read its documentation](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy) as it supports features like DNS discovery of upstreams, load balancing with checks, streaming parameters, etc.
### Caching
Caddy can compiled with a
[cache plugin](https://github.com/caddyserver/cache-handler) which can be used
to provide a hot-cache at the webserver-level for static websites hosted by
Garage.
This can be configured as follows:
```caddy
# Caddy global configuration section
{
# Bare minimum configuration to enable cache.
order cache before rewrite
cache
#cache
# allowed_http_verbs GET
# default_cache_control public
# ttl 8h
#}
}
# Site specific section
https:// {
cache
#cache {
# timeout {
# backend 30s
# }
#}
reverse_proxy ...
}
```
Caching is a complicated subject, and the reader is encouraged to study the
available options provided by the plugin.
### On-demand TLS
Caddy supports a technique called
@ -469,6 +428,3 @@ https:// {
reverse_proxy localhost:3902 192.168.1.2:3902 example.tld:3902
}
```
More information on how this endpoint is implemented in Garage is available
in the [Admin API Reference](@/documentation/reference-manual/admin-api.md) page.

View file

@ -39,95 +39,11 @@ Authorization: Bearer <token>
## Administration API endpoints
### Metrics `GET /metrics`
### Metrics-related endpoints
#### Metrics `GET /metrics`
Returns internal Garage metrics in Prometheus format.
The metrics are directly documented when returned by the API.
**Example:**
```
$ curl -i http://localhost:3903/metrics
HTTP/1.1 200 OK
content-type: text/plain; version=0.0.4
content-length: 12145
date: Tue, 08 Aug 2023 07:25:05 GMT
# HELP api_admin_error_counter Number of API calls to the various Admin API endpoints that resulted in errors
# TYPE api_admin_error_counter counter
api_admin_error_counter{api_endpoint="CheckWebsiteEnabled",status_code="400"} 1
api_admin_error_counter{api_endpoint="CheckWebsiteEnabled",status_code="404"} 3
# HELP api_admin_request_counter Number of API calls to the various Admin API endpoints
# TYPE api_admin_request_counter counter
api_admin_request_counter{api_endpoint="CheckWebsiteEnabled"} 7
api_admin_request_counter{api_endpoint="Health"} 3
# HELP api_admin_request_duration Duration of API calls to the various Admin API endpoints
...
```
### Health `GET /health`
Returns `200 OK` if enough nodes are up to have a quorum (ie. serve requests),
otherwise returns `503 Service Unavailable`.
**Example:**
```
$ curl -i http://localhost:3903/health
HTTP/1.1 200 OK
content-type: text/plain
content-length: 102
date: Tue, 08 Aug 2023 07:22:38 GMT
Garage is fully operational
Consult the full health check API endpoint at /v0/health for more details
```
### On-demand TLS `GET /check`
To prevent abuses for on-demand TLS, Caddy developpers have specified an endpoint that can be queried by the reverse proxy
to know if a given domain is allowed to get a certificate. Garage implements this endpoints to tell if a given domain is handled by Garage or is garbage.
Garage responds with the following logic:
- If the domain matches the pattern `<bucket-name>.<s3_api.root_domain>`, returns 200 OK
- If the domain matches the pattern `<bucket-name>.<s3_web.root_domain>` and website is configured for `<bucket>`, returns 200 OK
- If the domain matches the pattern `<bucket-name>` and website is configured for `<bucket>`, returns 200 OK
- Otherwise, returns 404 Not Found, 400 Bad Request or 5xx requests.
*Note 1: because in the path-style URL mode, there is only one domain that is not known by Garage, hence it is not supported by this API endpoint.
You must manually declare the domain in your reverse-proxy. Idem for K2V.*
*Note 2: buckets in a user's namespace are not supported yet by this endpoint. This is a limitation of this endpoint currently.*
**Example:** Suppose a Garage instance configured with `s3_api.root_domain = .s3.garage.localhost` and `s3_web.root_domain = .web.garage.localhost`.
With a private `media` bucket (name in the global namespace, website is disabled), the endpoint will feature the following behavior:
```
$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=media.s3.garage.localhost
200
$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=media
400
$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=media.web.garage.localhost
400
```
With a public `example.com` bucket (name in the global namespace, website is activated), the endpoint will feature the following behavior:
```
$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=example.com.s3.garage.localhost
200
$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=example.com
200
$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=example.com.web.garage.localhost
200
```
**References:**
- [Using On-Demand TLS](https://caddyserver.com/docs/automatic-https#using-on-demand-tls)
- [Add option for a backend check to approve use of on-demand TLS](https://github.com/caddyserver/caddy/pull/1939)
- [Serving tens of thousands of domains over HTTPS with Caddy](https://caddy.community/t/serving-tens-of-thousands-of-domains-over-https-with-caddy/11179)
### Cluster operations

View file

@ -26,7 +26,6 @@ use crate::admin::cluster::*;
use crate::admin::error::*;
use crate::admin::key::*;
use crate::admin::router::{Authorization, Endpoint};
use crate::helpers::host_to_bucket;
pub struct AdminApiServer {
garage: Arc<Garage>,
@ -79,7 +78,10 @@ impl AdminApiServer {
.body(Body::empty())?)
}
async fn handle_check_domain(&self, req: Request<Body>) -> Result<Response<Body>, Error> {
async fn handle_check_website_enabled(
&self,
req: Request<Body>,
) -> Result<Response<Body>, Error> {
let query_params: HashMap<String, String> = req
.uri()
.query()
@ -100,43 +102,12 @@ impl AdminApiServer {
.get("domain")
.ok_or_internal_error("Could not parse domain query string")?;
// Resolve bucket from domain name, inferring if the website must be activated for the
// domain to be valid.
let (bucket_name, must_check_website) = if let Some(bname) = self
.garage
.config
.s3_api
.root_domain
.as_ref()
.and_then(|rd| host_to_bucket(domain, rd))
{
(bname.to_string(), false)
} else if let Some(bname) = self
.garage
.config
.s3_web
.as_ref()
.and_then(|sw| host_to_bucket(domain, sw.root_domain.as_str()))
{
(bname.to_string(), true)
} else {
(domain.to_string(), true)
};
let bucket_id = self
.garage
.bucket_helper()
.resolve_global_bucket_name(&bucket_name)
.resolve_global_bucket_name(domain)
.await?
.ok_or(HelperError::NoSuchBucket(bucket_name.to_string()))?;
if !must_check_website {
return Ok(Response::builder()
.status(StatusCode::OK)
.body(Body::from(format!(
"Domain '{domain}' is managed by Garage"
)))?);
}
.ok_or(HelperError::NoSuchBucket(domain.to_string()))?;
let bucket = self
.garage
@ -152,11 +123,11 @@ impl AdminApiServer {
Ok(Response::builder()
.status(StatusCode::OK)
.body(Body::from(format!(
"Domain '{domain}' is managed by Garage"
"Bucket '{domain}' is authorized for website hosting"
)))?)
}
None => Err(Error::bad_request(format!(
"Domain '{domain}' is not managed by Garage"
"Bucket '{domain}' is not authorized for website hosting"
))),
}
}
@ -258,7 +229,7 @@ impl ApiHandler for AdminApiServer {
match endpoint {
Endpoint::Options => self.handle_options(&req),
Endpoint::CheckDomain => self.handle_check_domain(req).await,
Endpoint::CheckWebsiteEnabled => self.handle_check_website_enabled(req).await,
Endpoint::Health => self.handle_health(),
Endpoint::Metrics => self.handle_metrics(),
Endpoint::GetClusterStatus => handle_get_cluster_status(&self.garage).await,

View file

@ -17,7 +17,7 @@ router_match! {@func
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Endpoint {
Options,
CheckDomain,
CheckWebsiteEnabled,
Health,
Metrics,
GetClusterStatus,
@ -92,7 +92,7 @@ impl Endpoint {
let res = router_match!(@gen_path_parser (req.method(), path, query) [
OPTIONS _ => Options,
GET "/check" => CheckDomain,
GET "/check" => CheckWebsiteEnabled,
GET "/health" => Health,
GET "/metrics" => Metrics,
GET "/v0/status" => GetClusterStatus,
@ -138,7 +138,7 @@ impl Endpoint {
pub fn authorization_type(&self) -> Authorization {
match self {
Self::Health => Authorization::None,
Self::CheckDomain => Authorization::None,
Self::CheckWebsiteEnabled => Authorization::None,
Self::Metrics => Authorization::MetricsToken,
_ => Authorization::AdminToken,
}

View file

@ -30,7 +30,7 @@ pub async fn handle_post_object(
.get(header::CONTENT_TYPE)
.and_then(|ct| ct.to_str().ok())
.and_then(|ct| multer::parse_boundary(ct).ok())
.ok_or_bad_request("Could not get multipart boundary")?;
.ok_or_bad_request("Counld not get multipart boundary")?;
// 16k seems plenty for a header. 5G is the max size of a single part, so it seems reasonable
// for a PostObject
@ -64,13 +64,15 @@ pub async fn handle_post_object(
"tag" => (/* tag need to be reencoded, but we don't support them yet anyway */),
"acl" => {
if params.insert("x-amz-acl", content).is_some() {
return Err(Error::bad_request("Field 'acl' provided more than once"));
return Err(Error::bad_request(
"Field 'acl' provided more than one time",
));
}
}
_ => {
if params.insert(&name, content).is_some() {
return Err(Error::bad_request(format!(
"Field '{}' provided more than once",
"Field '{}' provided more than one time",
name
)));
}
@ -147,7 +149,7 @@ pub async fn handle_post_object(
.ok_or_bad_request("Invalid expiration date")?
.into();
if Utc::now() - expiration > Duration::zero() {
return Err(Error::bad_request("Expiration date is in the past"));
return Err(Error::bad_request("Expiration date is in the paste"));
}
let mut conditions = decoded_policy.into_conditions()?;
@ -328,7 +330,7 @@ impl Policy {
if map.len() != 1 {
return Err(Error::bad_request("Invalid policy item"));
}
let (mut k, v) = map.into_iter().next().expect("Size could not be verified");
let (mut k, v) = map.into_iter().next().expect("size was verified");
k.make_ascii_lowercase();
params.entry(k).or_default().push(Operation::Equal(v));
}

View file

@ -72,7 +72,7 @@ async fn test_website() {
res_body,
json!({
"code": "InvalidRequest",
"message": "Bad request: Domain 'my-website' is not managed by Garage",
"message": "Bad request: Bucket 'my-website' is not authorized for website hosting",
"region": "garage-integ-test",
"path": "/check",
})
@ -91,29 +91,24 @@ async fn test_website() {
BODY.as_ref()
);
for bname in [
BCKT_NAME.to_string(),
format!("{BCKT_NAME}.web.garage"),
format!("{BCKT_NAME}.s3.garage"),
] {
let admin_req = || {
Request::builder()
.method("GET")
.uri(format!(
"http://127.0.0.1:{0}/check?domain={1}",
ctx.garage.admin_port, bname
))
.body(Body::empty())
.unwrap()
};
let admin_req = || {
Request::builder()
.method("GET")
.uri(format!(
"http://127.0.0.1:{0}/check?domain={1}",
ctx.garage.admin_port,
BCKT_NAME.to_string()
))
.body(Body::empty())
.unwrap()
};
let mut admin_resp = client.request(admin_req()).await.unwrap();
assert_eq!(admin_resp.status(), StatusCode::OK);
assert_eq!(
to_bytes(admin_resp.body_mut()).await.unwrap().as_ref(),
format!("Domain '{bname}' is managed by Garage").as_bytes()
);
}
let mut admin_resp = client.request(admin_req()).await.unwrap();
assert_eq!(admin_resp.status(), StatusCode::OK);
assert_eq!(
to_bytes(admin_resp.body_mut()).await.unwrap().as_ref(),
format!("Bucket '{BCKT_NAME}' is authorized for website hosting").as_bytes()
);
ctx.garage
.command()
@ -147,7 +142,7 @@ async fn test_website() {
res_body,
json!({
"code": "InvalidRequest",
"message": "Bad request: Domain 'my-website' is not managed by Garage",
"message": "Bad request: Bucket 'my-website' is not authorized for website hosting",
"region": "garage-integ-test",
"path": "/check",
})
@ -402,7 +397,7 @@ async fn test_website_s3_api() {
}
#[tokio::test]
async fn test_website_check_domain() {
async fn test_website_check_website_enabled() {
let ctx = common::context();
let client = Client::new();