MD5 sums are broken and s3cmd does not like that #1

New Issue

Closed

opened 2020-07-05 19:50:12 +00:00 by quentin · 4 comments

quentin commented

2020-07-05 19:50:12 +00:00

Owner

[quentin@lheureduthe nomad]$ s3cmd --no-check-certificate put ~/Images/happy.jpg s3://nextcloud/happy.jpg
upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg'  [1 of 1]
 270593 of 270593   100% in    0s     2.16 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of /home/quentin/Images/happy.jpg
upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg'  [1 of 1]
 270593 of 270593   100% in    0s     3.03 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of /home/quentin/Images/happy.jpg
upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg'  [1 of 1]
 270593 of 270593   100% in    0s     2.22 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of /home/quentin/Images/happy.jpg
upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg'  [1 of 1]
 270593 of 270593   100% in    0s     2.66 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of /home/quentin/Images/happy.jpg
upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg'  [1 of 1]
 270593 of 270593   100% in    0s     3.11 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of /home/quentin/Images/happy.jpg
upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg'  [1 of 1]
 270593 of 270593   100% in    0s     2.65 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Too many failures. Giving up on '/home/quentin/Images/happy.jpg'
ERROR: Upload of '/home/quentin/Images/happy.jpg' failed too many times (Last reason: )

[quentin@lheureduthe tmp]$ s3cmd --no-check-certificate get s3://nextcloud/happy.jpg lol.jpg
download: 's3://nextcloud/happy.jpg' -> 'lol.jpg'  [1 of 1]
 270593 of 270593   100% in    0s     4.41 MB/s  done
WARNING: MD5 signatures do not match: computed=cd3d85d7eda3d58bde2107dae5873242, received=

``` [quentin@lheureduthe nomad]$ s3cmd --no-check-certificate put ~/Images/happy.jpg s3://nextcloud/happy.jpg upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg' [1 of 1] 270593 of 270593 100% in 0s 2.16 MB/s done WARNING: MD5 Sums don't match! WARNING: Retrying upload of /home/quentin/Images/happy.jpg upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg' [1 of 1] 270593 of 270593 100% in 0s 3.03 MB/s done WARNING: MD5 Sums don't match! WARNING: Retrying upload of /home/quentin/Images/happy.jpg upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg' [1 of 1] 270593 of 270593 100% in 0s 2.22 MB/s done WARNING: MD5 Sums don't match! WARNING: Retrying upload of /home/quentin/Images/happy.jpg upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg' [1 of 1] 270593 of 270593 100% in 0s 2.66 MB/s done WARNING: MD5 Sums don't match! WARNING: Retrying upload of /home/quentin/Images/happy.jpg upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg' [1 of 1] 270593 of 270593 100% in 0s 3.11 MB/s done WARNING: MD5 Sums don't match! WARNING: Retrying upload of /home/quentin/Images/happy.jpg upload: '/home/quentin/Images/happy.jpg' -> 's3://nextcloud/happy.jpg' [1 of 1] 270593 of 270593 100% in 0s 2.65 MB/s done WARNING: MD5 Sums don't match! WARNING: Too many failures. Giving up on '/home/quentin/Images/happy.jpg' ERROR: Upload of '/home/quentin/Images/happy.jpg' failed too many times (Last reason: ) ``` ``` [quentin@lheureduthe tmp]$ s3cmd --no-check-certificate get s3://nextcloud/happy.jpg lol.jpg download: 's3://nextcloud/happy.jpg' -> 'lol.jpg' [1 of 1] 270593 of 270593 100% in 0s 4.41 MB/s done WARNING: MD5 signatures do not match: computed=cd3d85d7eda3d58bde2107dae5873242, received= ```

quentin added the

Newcomer

label 2020-07-05 19:50:54 +00:00

lx commented

2020-07-08 12:12:34 +00:00

Owner

Related: Garage currently does not implement checking of the MD5 sum of the uploaded data, which should match the Content-MD5 header. Should be implemented.

Related: Garage currently does not implement checking of the MD5 sum of the uploaded data, which should match the `Content-MD5` header. Should be implemented.

lx commented

2020-07-08 12:14:13 +00:00

Owner

WRT the original bug, Amazon says in the docs that the returned ETag can be compared against the MD5 of the file for the PutObject API call and that they should match. This is inconsistent with the fact that ETags are supposed to be opaque identifiers attributed by the system, but if S3 clients rely on this behaviour we have to respect it. Garage doesn't care about ETags so we can do whatever with that field.

😆 1

lx commented

2020-07-08 12:16:03 +00:00

Owner

Some information about ETags: https://teppen.io/2018/06/23/aws_s3_etags/
We should implement the same thing for maximum compatibility.

Some information about ETags: https://teppen.io/2018/06/23/aws_s3_etags/ We should implement the same thing for maximum compatibility.

👍 1

lx commented

2020-07-13 14:52:56 +00:00

Owner

Fixed for PutObject, original issue is solved. Creating other tickets for the other things.