UploadPartCopy fails with Harbor v1 (a docker registry) #248

New issue

Closed

opened 2022-02-22 17:16:34 +00:00 by quentin · 0 comments

quentin commented

2022-02-22 17:16:34 +00:00

Owner

We have a problem as UploadPartCopy fails with Harbor.

The error:

 INFO  garage_api::api_server  > Response: error 400 Bad Request, Bad request: Not enough data to copy: 821344 bytes (minimum: 1MB)

We have a trace here:

 INFO  garage_api::api_server  > 192.168.128.11:42216 POST /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?uploads=
 
 DEBUG garage_api::api_server  > Request { method: POST, uri: /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?uploads=, version: HTTP/1.1, headers: {"host": "garage-cluster.svc.aab0f485-69e7-467f-a86e-9b4d4a4875ad.lev-1.int.greenbaum.zone:3900", "user-agent": "aws-sdk-go/1.15.11 (go1.15.6; linux; amd64)", "content-length": "0", "authorization": "AWS4-HMAC-SHA256 Credential=GK9f1194f03fe3f6111028c46f/20220222/garage/s3/aws4_request, SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-storage-class, Signature=22d7f929c980c84be14317bd31c1d716697f11e894fb82bdbe5f8043be0dfe0d", "content-type": "application/octet-stream", "x-amz-acl": "private", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20220222T165531Z", "x-amz-storage-class": "STANDARD", "accept-encoding": "gzip"}, body: Body(Empty) }

 DEBUG garage_api::api_server  > Endpoint: CreateMultipartUpload { key: "docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data" }

DEBUG garage_api::api_server  > 200 OK {}

 INFO  garage_api::api_server  > 192.168.128.11:42216 PUT /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?partNumber=2&uploadId=dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070

 DEBUG garage_api::api_server  > Request { method: PUT, uri: /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?partNumber=2&uploadId=dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070, version: HTTP/1.1, headers: {"host": "garage-cluster.svc.aab0f485-69e7-467f-a86e-9b4d4a4875ad.lev-1.int.greenbaum.zone:3900", "user-agent": "aws-sdk-go/1.15.11 (go1.15.6; linux; amd64)", "content-length": "0", "authorization": "AWS4-HMAC-SHA256 Credential=GK9f1194f03fe3f6111028c46f/20220222/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-range;x-amz-date, Signature=8e9661be2d4474dc7a3f0b6b3cb068d1d407cc222420d3fa90416eed45e8214b", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-copy-source": "harbor-registry/docker/registry/v2/repositories/gc/mdbook-multilang/_uploads/ed5b86b2-e9b1-4147-8cfc-e9d11c781fc6/data", "x-amz-copy-source-range": "bytes=33554432-34375775", "x-amz-date": "20220222T165531Z", "accept-encoding": "gzip"}, body: Body(Empty) }

DEBUG garage_api::api_server  > Endpoint: UploadPartCopy { key: "docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data", part_number: 2, upload_id: "dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070" }

 INFO  garage_api::api_server  > Response: error 400 Bad Request, Bad request: Not enough data to copy: 821344 bytes (minimum: 1MB)
 
 INFO  garage_api::api_server  > 192.168.128.11:42216 PUT /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?partNumber=1&uploadId=dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070

DEBUG garage_api::api_server  > Request { method: PUT, uri: /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?partNumber=1&uploadId=dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070, version: HTTP/1.1, headers: {"host": "garage-cluster.svc.aab0f485-69e7-467f-a86e-9b4d4a4875ad.lev-1.int.greenbaum.zone:3900", "user-agent": "aws-sdk-go/1.15.11 (go1.15.6; linux; amd64)", "content-length": "0", "authorization": "AWS4-HMAC-SHA256 Credential=GK9f1194f03fe3f6111028c46f/20220222/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-range;x-amz-date, Signature=82baf287b143ece8959ffae99ac6e41ac691714a3fceee4908239484287b2e1b", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-copy-source": "harbor-registry/docker/registry/v2/repositories/gc/mdbook-multilang/_uploads/ed5b86b2-e9b1-4147-8cfc-e9d11c781fc6/data", "x-amz-copy-source-range": "bytes=0-33554431", "x-amz-date": "20220222T165531Z", "accept-encoding": "gzip"}, body: Body(Empty) }

DEBUG garage_api::api_server  > Endpoint: UploadPartCopy { key: "docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data", part_number: 1, upload_id: "dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070" }

The check: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_copy.rs#L305

Harbor version:

 registry:
    image: goharbor/registry-photon:v1.10.10

Harbor conf:

version: 0.1
log:
  level: info
  fields:
    service: registry
storage:
  cache:
    layerinfo: redis
  s3:
    accesskey: redacted
    secretkey: redacted
    region: garage
    regionendpoint: http://redacted:3900
    bucket: harbor-registry
    secure: False
    v4auth: true
    rootdirectory: /
  maintenance:
    uploadpurging:
      enabled: false
  delete:
    enabled: true
redis:
  addr: redis:6379
  password:
  db: 1
http:
  addr: :5000
  secret: placeholder
  debug:
    addr: localhost:5001
auth:
  token:
    issuer: harbor-token-issuer
    realm: https://redacted
    rootcertbundle: /etc/registry/root.crt
    service: harbor-registry
validation:
  disabled: true
notifications:
  endpoints:
  - name: harbor
    disabled: false
    url: http://core:8080/service/notifications
    timeout: 3000ms
    threshold: 5
    backoff: 1s
    ignoredmediatypes:
        - application/vnd.docker.image.rootfs.diff.tar.gzip
        - application/vnd.docker.image.rootfs.foreign.diff.tar.gzip
        - application/vnd.oci.image.layer.v1.tar
        - application/vnd.oci.image.layer.v1.tar+gzip
        - application/vnd.oci.image.layer.v1.tar+zstd
        - application/vnd.oci.image.layer.nondistributable.v1.tar
        - application/vnd.oci.image.layer.nondistributable.v1.tar+gzip
        - application/vnd.oci.image.layer.nondistributable.v1.tar+zstd
        - application/octet-stream
compatibility:
  schema1:
    enabled: true

We have a problem as UploadPartCopy fails with Harbor. The error: ``` INFO garage_api::api_server > Response: error 400 Bad Request, Bad request: Not enough data to copy: 821344 bytes (minimum: 1MB) ``` We have a trace here: ``` INFO garage_api::api_server > 192.168.128.11:42216 POST /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?uploads= DEBUG garage_api::api_server > Request { method: POST, uri: /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?uploads=, version: HTTP/1.1, headers: {"host": "garage-cluster.svc.aab0f485-69e7-467f-a86e-9b4d4a4875ad.lev-1.int.greenbaum.zone:3900", "user-agent": "aws-sdk-go/1.15.11 (go1.15.6; linux; amd64)", "content-length": "0", "authorization": "AWS4-HMAC-SHA256 Credential=GK9f1194f03fe3f6111028c46f/20220222/garage/s3/aws4_request, SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-storage-class, Signature=22d7f929c980c84be14317bd31c1d716697f11e894fb82bdbe5f8043be0dfe0d", "content-type": "application/octet-stream", "x-amz-acl": "private", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20220222T165531Z", "x-amz-storage-class": "STANDARD", "accept-encoding": "gzip"}, body: Body(Empty) } DEBUG garage_api::api_server > Endpoint: CreateMultipartUpload { key: "docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data" } DEBUG garage_api::api_server > 200 OK {} INFO garage_api::api_server > 192.168.128.11:42216 PUT /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?partNumber=2&uploadId=dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070 DEBUG garage_api::api_server > Request { method: PUT, uri: /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?partNumber=2&uploadId=dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070, version: HTTP/1.1, headers: {"host": "garage-cluster.svc.aab0f485-69e7-467f-a86e-9b4d4a4875ad.lev-1.int.greenbaum.zone:3900", "user-agent": "aws-sdk-go/1.15.11 (go1.15.6; linux; amd64)", "content-length": "0", "authorization": "AWS4-HMAC-SHA256 Credential=GK9f1194f03fe3f6111028c46f/20220222/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-range;x-amz-date, Signature=8e9661be2d4474dc7a3f0b6b3cb068d1d407cc222420d3fa90416eed45e8214b", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-copy-source": "harbor-registry/docker/registry/v2/repositories/gc/mdbook-multilang/_uploads/ed5b86b2-e9b1-4147-8cfc-e9d11c781fc6/data", "x-amz-copy-source-range": "bytes=33554432-34375775", "x-amz-date": "20220222T165531Z", "accept-encoding": "gzip"}, body: Body(Empty) } DEBUG garage_api::api_server > Endpoint: UploadPartCopy { key: "docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data", part_number: 2, upload_id: "dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070" } INFO garage_api::api_server > Response: error 400 Bad Request, Bad request: Not enough data to copy: 821344 bytes (minimum: 1MB) INFO garage_api::api_server > 192.168.128.11:42216 PUT /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?partNumber=1&uploadId=dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070 DEBUG garage_api::api_server > Request { method: PUT, uri: /harbor-registry/docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data?partNumber=1&uploadId=dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070, version: HTTP/1.1, headers: {"host": "garage-cluster.svc.aab0f485-69e7-467f-a86e-9b4d4a4875ad.lev-1.int.greenbaum.zone:3900", "user-agent": "aws-sdk-go/1.15.11 (go1.15.6; linux; amd64)", "content-length": "0", "authorization": "AWS4-HMAC-SHA256 Credential=GK9f1194f03fe3f6111028c46f/20220222/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-range;x-amz-date, Signature=82baf287b143ece8959ffae99ac6e41ac691714a3fceee4908239484287b2e1b", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-copy-source": "harbor-registry/docker/registry/v2/repositories/gc/mdbook-multilang/_uploads/ed5b86b2-e9b1-4147-8cfc-e9d11c781fc6/data", "x-amz-copy-source-range": "bytes=0-33554431", "x-amz-date": "20220222T165531Z", "accept-encoding": "gzip"}, body: Body(Empty) } DEBUG garage_api::api_server > Endpoint: UploadPartCopy { key: "docker/registry/v2/blobs/sha256/71/71bfb2a59c65a1e0f1b8b0a93c39a14a9ff1aebb7cf01a0b63c57a8b660f7c9f/data", part_number: 1, upload_id: "dccdfd59233b5c87957d7681cc2fc2a0665ff1faa87003df21af323da6373070" } ``` The check: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_copy.rs#L305 Harbor version: ``` registry: image: goharbor/registry-photon:v1.10.10 ``` Harbor conf: ``` version: 0.1 log: level: info fields: service: registry storage: cache: layerinfo: redis s3: accesskey: redacted secretkey: redacted region: garage regionendpoint: http://redacted:3900 bucket: harbor-registry secure: False v4auth: true rootdirectory: / maintenance: uploadpurging: enabled: false delete: enabled: true redis: addr: redis:6379 password: db: 1 http: addr: :5000 secret: placeholder debug: addr: localhost:5001 auth: token: issuer: harbor-token-issuer realm: https://redacted rootcertbundle: /etc/registry/root.crt service: harbor-registry validation: disabled: true notifications: endpoints: - name: harbor disabled: false url: http://core:8080/service/notifications timeout: 3000ms threshold: 5 backoff: 1s ignoredmediatypes: - application/vnd.docker.image.rootfs.diff.tar.gzip - application/vnd.docker.image.rootfs.foreign.diff.tar.gzip - application/vnd.oci.image.layer.v1.tar - application/vnd.oci.image.layer.v1.tar+gzip - application/vnd.oci.image.layer.v1.tar+zstd - application/vnd.oci.image.layer.nondistributable.v1.tar - application/vnd.oci.image.layer.nondistributable.v1.tar+gzip - application/vnd.oci.image.layer.nondistributable.v1.tar+zstd - application/octet-stream compatibility: schema1: enabled: true ```

quentin added the

scope

s3-api

label 2022-02-22 17:16:34 +00:00

quentin changed title from ~~UploadPartCopy fails with Harbor (a docker registry)~~ to UploadPartCopy fails with Harbor v1 (a docker registry)

2022-02-23 10:39:42 +00:00