forked from Deuxfleurs/garage
Document multi-hdd support
This commit is contained in:
parent
bca347a1e8
commit
6595efd82f
3 changed files with 116 additions and 8 deletions
|
@ -75,16 +75,11 @@ to store 2 TB of data in total.
|
|||
|
||||
- For the metadata storage, Garage does not do checksumming and integrity
|
||||
verification on its own. If you are afraid of bitrot/data corruption,
|
||||
put your metadata directory on a BTRFS partition. Otherwise, just use regular
|
||||
put your metadata directory on a ZFS or BTRFS partition. Otherwise, just use regular
|
||||
EXT4 or XFS.
|
||||
|
||||
- Having a single server with several storage drives is currently not very well
|
||||
supported in Garage ([#218](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/218)).
|
||||
For an easy setup, just put all your drives in a RAID0 or a ZFS RAIDZ array.
|
||||
If you're adventurous, you can try to format each of your disk as
|
||||
a separate XFS partition, and then run one `garage` daemon per disk drive,
|
||||
or use something like [`mergerfs`](https://github.com/trapexit/mergerfs) to merge
|
||||
all your disks in a single union filesystem that spreads load over them.
|
||||
- Servers with multiple HDDs are supported natively by Garage without resorting
|
||||
to RAID, see [our dedicated documentation page](@/documentation/operations/multi-hdd.md).
|
||||
|
||||
## Get a Docker image
|
||||
|
||||
|
|
100
doc/book/operations/multi-hdd.md
Normal file
100
doc/book/operations/multi-hdd.md
Normal file
|
@ -0,0 +1,100 @@
|
|||
+++
|
||||
title = "Multi-HDD support"
|
||||
weight = 15
|
||||
+++
|
||||
|
||||
|
||||
Since v0.9, Garage natively supports nodes that have several storage drives
|
||||
for storing data blocks (not for metadata storage).
|
||||
|
||||
## Initial setup
|
||||
|
||||
To set up a new Garage storage node with multiple HDDs,
|
||||
format and mount all your drives in different directories,
|
||||
and use a Garage configuration as follows:
|
||||
|
||||
```toml
|
||||
data_dir = [
|
||||
{ path = "/path/to/hdd1", capacity = "2T" },
|
||||
{ path = "/path/to/hdd2", capacity = "4T" },
|
||||
]
|
||||
```
|
||||
|
||||
Garage will automatically balance all blocks stored by the node
|
||||
among the different specified directories, proportionnally to the
|
||||
specified capacities.
|
||||
|
||||
## Updating the list of storage locations
|
||||
|
||||
If you add new storage locations to your `data_dir`,
|
||||
Garage will not rebalance existing data between storage locations.
|
||||
Newly written blocks will be balanced proportionnally to the specified capacities,
|
||||
and existing data may be moved between drives to improve balancing,
|
||||
but only opportunistically when a data block is re-written (e.g. an object
|
||||
is re-uploaded, or an object with a duplicate block is uploaded).
|
||||
|
||||
To understand precisely what is happening, we need to dive in to how Garage
|
||||
splits data among the different storage locations.
|
||||
|
||||
First of all, Garage divides the set of all possible block hashes
|
||||
in a fixed number of slices (currently 1024), and assigns
|
||||
to each slice a primary storage location among the specified data directories.
|
||||
The number of slices having their primary location in each data directory
|
||||
is proportionnal to the capacity specified in the config file.
|
||||
|
||||
When Garage receives a block to write, it will always write it in the primary
|
||||
directory of the slice that contains its hash.
|
||||
|
||||
Now, to be able to not lose existing data blocks when storage locations
|
||||
are added, Garage also keeps a list of secondary data directories
|
||||
for all of the hash slices. Secondary data directories for a slice indicates
|
||||
storage locations that once were primary directories for that slice, i.e. where
|
||||
Garage knows that data blocks of that slice might be stored.
|
||||
When Garage is requested to read a certain data block,
|
||||
it will first look in the primary storage directory of its slice,
|
||||
and if it doesn't find it there it goes through all of the secondary storage
|
||||
locations until it finds it. This allows Garage to continue operating
|
||||
normally when storage locations are added, without having to shuffle
|
||||
files between drives to place them in the correct location.
|
||||
|
||||
This relatively simple strategy works well but does not ensure that data
|
||||
is correctly balanced among drives according to their capacity.
|
||||
To rebalance data, two strategies can be used:
|
||||
|
||||
- Lazy rebalancing: when a block is re-written (e.g. the object is re-uploaded),
|
||||
Garage checks whether the existing copy is in the primary directory of the slice
|
||||
or in a secondary directory. If the current copy is in a secondary directory,
|
||||
Garage re-writes a copy in the primary directory and deletes the one from the
|
||||
secondary directory.
|
||||
|
||||
- Active rebalancing: an operator of a Garage node can explicitly launch a repair
|
||||
procedure that rebalances the data directories, moving all blocks to their
|
||||
primary location. Once done, all secondary locations for all hash slices are
|
||||
removed so that they won't be checked anymore when looking for a data block.
|
||||
|
||||
## Read-only storage locations
|
||||
|
||||
If you would like to move all data blocks from an existing data directory to one
|
||||
or several new data directories, mark the old directory as read-only:
|
||||
|
||||
```toml
|
||||
data_dir = [
|
||||
{ path = "/path/to/old_data", read_only = true },
|
||||
{ path = "/path/to/new_hdd1", capacity = "2T" },
|
||||
{ path = "/path/to/new_hdd2", capacity = "4T" },
|
||||
]
|
||||
```
|
||||
|
||||
Garage will be able to read requested blocks from the read-only directory.
|
||||
Garage will also move data out of the read-only directory either progressively
|
||||
(lazy rebalancing) or if requested explicitly (active rebalancing).
|
||||
|
||||
Once an active rebalancing has finished, your read-only directory should be empty:
|
||||
it might still contain subdirectories, but no data files. You can check that
|
||||
it contains no files using:
|
||||
|
||||
```bash
|
||||
find -type f /path/to/old_data
|
||||
```
|
||||
|
||||
at which point it can be removed from the `data_dir` list in your config file.
|
|
@ -91,6 +91,19 @@ This folder can be placed on an HDD. The space available for `data_dir`
|
|||
should be counted to determine a node's capacity
|
||||
when [adding it to the cluster layout](@/documentation/cookbook/real-world.md).
|
||||
|
||||
Since `v0.9.0`, Garage supports multiple data directories with the following syntax:
|
||||
|
||||
```toml
|
||||
data_dir = [
|
||||
{ path = "/path/to/old_data", read_only = true },
|
||||
{ path = "/path/to/new_hdd1", capacity = "2T" },
|
||||
{ path = "/path/to/new_hdd2", capacity = "4T" },
|
||||
]
|
||||
```
|
||||
|
||||
See [the dedicated documentation page](@/documentation/operations/multi-hdd.md)
|
||||
on how to operate Garage in such a setup.
|
||||
|
||||
### `db_engine` (since `v0.8.0`)
|
||||
|
||||
By default, Garage uses the Sled embedded database library
|
||||
|
|
Loading…
Reference in a new issue