Multi-DC Use Case vs Ceph/Minio #233

New Issue

Closed

opened 2022-02-09 22:13:53 +00:00 by Ghost · 3 comments

Ghost commented

2022-02-09 22:13:53 +00:00

Hey guys,

Great work/great thinking behind Garage.

Sounds like it could be ideal for a multi-dc scenario where the connections between the nodes go over the public internet?

Let's say I have one node in each of 5 regions, and I don't need/want to encrypt the traffic between them (does Garage do that automatically?) then how much better performance and resilience than say Ceph or Minio?

Have you quantified that?

Thank u.

Hey guys, Great work/great thinking behind Garage. Sounds like it could be ideal for a multi-dc scenario where the connections between the nodes go over the public internet? Let's say I have one node in each of 5 regions, and I don't need/want to encrypt the traffic between them (does Garage do that automatically?) then how much better performance and resilience than say Ceph or Minio? Have you quantified that? Thank u.

quentin commented

2022-02-10 15:05:59 +00:00

Owner

Hi innerop,

Sounds like it could be ideal for a multi-dc scenario where the connections between the nodes go over the public internet?

Indeed, it's our use case (we want to self-host multiple nodes across different cities in France/Europe).

Does Garage encrypts the traffic between nodes?

Garage already encrypts the traffic between nodes. The protocol used is named Secret Handshake. Your secret is the rpc_secret you define in your configuration. To join a Garage cluster, you must know this rpc_secret and the ID of one of the nodes of the network. There is no way to deactivate encryption on Garage (but if you copy paste the rpc_secret in our example config, your encryption will not be very strong).

how much better performance and resilience than say Ceph or Minio?

We have a micro benchmark where we simulate a network with links of ~100ms and we measure the time to first byte of multiple requests. Our results are available in our documentation: https://garagehq.deuxfleurs.fr/documentation/design/benchmarks/

Of couse, we would like to do additional benchmarks and be able to provide more "high level" recommendations but for now, that's all we have :-)

Does it answer your questions?

notes

we could add to our documentation that traffic between nodes is encrypted and how

Hi innerop, > Sounds like it could be ideal for a multi-dc scenario where the connections between the nodes go over the public internet? Indeed, it's our use case (we want to self-host multiple nodes across different cities in France/Europe). > Does Garage encrypts the traffic between nodes? Garage already encrypts the traffic between nodes. The protocol used is named [Secret Handshake](https://dominictarr.github.io/secret-handshake-paper/shs.pdf). Your secret is the `rpc_secret` you define in your configuration. To join a Garage cluster, you must know this rpc_secret and the ID of one of the nodes of the network. There is no way to deactivate encryption on Garage (but if you copy paste the rpc_secret in our example config, your encryption will not be very strong). > how much better performance and resilience than say Ceph or Minio? We have a micro benchmark where we simulate a network with links of ~100ms and we measure the time to first byte of multiple requests. Our results are available in our documentation: https://garagehq.deuxfleurs.fr/documentation/design/benchmarks/ Of couse, we would like to do additional benchmarks and be able to provide more "high level" recommendations but for now, that's all we have :-) Does it answer your questions? ## notes we could add to our documentation that traffic between nodes is encrypted and how

quentin added the

Documentation

label 2022-02-10 15:06:48 +00:00

Ghost commented

2022-02-10 15:39:07 +00:00

Author

hi @quentin

Yes, it answers my question. Thank you.

However, i had not thought much about how inter-node connections are done. I assumed something maybe TCP based like "secure websockets" that has encryption already as part of the standard (TLS) -- I did not consider that you may be using a custom UDP protocol, so then it makes sense to add encryption.

Yes, it would be helpful to shed some light on how it's implemented.

Thanks again.

hi @quentin Yes, it answers my question. Thank you. However, i had not thought much about how inter-node connections are done. I assumed something maybe TCP based like "secure websockets" that has encryption already as part of the standard (TLS) -- I did not consider that you may be using a custom UDP protocol, so then it makes sense to add encryption. Yes, it would be helpful to shed some light on how it's implemented. Thanks again.

quentin closed this issue

2022-02-11 14:31:03 +00:00

quentin commented

2022-02-11 14:34:55 +00:00

Owner

Thanks for your advices.

We do not use a UDP protocol but TCP sockets, with message pack to serialize/deserialize messages, and scheduling on the network is handled by netapp, our internal network library.

I am closing this issue in favor of #228 because I want to centralize all the information that are missing in our documentation in a single issue. Feel free to ask additional questions in this issue, I will try my best to answer them, and later add the answers in the documentation.

And hopefully, in a not so long future, we will be able to release a whitepaper on Garage internals :-)

Thanks for your advices. We do not use a UDP protocol but TCP sockets, with message pack to serialize/deserialize messages, and scheduling on the network is handled by [netapp](https://git.deuxfleurs.fr/lx/netapp), our internal network library. I am closing this issue in favor of #228 because I want to centralize all the information that are missing in our documentation in a single issue. Feel free to ask additional questions in this issue, I will try my best to answer them, and later add the answers in the documentation. And hopefully, in a not so long future, we will be able to release a whitepaper on Garage internals :-)