Race condition -> bucket might not be completely initialized #59

New issue

Open

opened 2024-02-13 11:31:00 +00:00 by quentin · 0 comments

quentin commented

2024-02-13 11:31:00 +00:00

Owner

Sometimes upon initialization, we have:

PIM Builder failed
local alias does not exist in garage or points to the wrong bucket.
missing bucket local garage alias in LDAP.

I think /pim/setup went successfully, redirected to /pim/inspect, that rerun the builder and failed in the second branch:

	if pm.bucketId == "" {
         // ...
        bInfo, err = grgCreateLocalBucket(candidateName, *keyInfo.AccessKeyId)
        // ...
	} else {
		binfo, err := grgGetBucket(pm.bucketId)
		if err != nil {
			pm.errors = append(pm.errors, err)
			return pm
		}
		pm.bucketInfo = binfo
		//@TODO find my key, check that pm.bucketName exists in bucketLocalAliases
		nameFound := false
		for _, k := range binfo.Keys {
			if *k.AccessKeyId != *keyInfo.AccessKeyId {
				// not my key
				continue
			}
			if slices.Contains(k.BucketLocalAliases, pm.bucketName) {
				nameFound = true
				break
			}
		}
		if !nameFound {
			pm.errors = append(pm.errors, ErrPimBucketLocalAliasNotFound)
			return pm
		}

I think we should fix this bug by calling grgGetBucket in the first branch in a loop with an exponential backoff and a limit, and do the validation we do in the normal branch each time, to be sure that we make progress only once the bucket is fully setup.

Sometimes upon initialization, we have: ``` PIM Builder failed local alias does not exist in garage or points to the wrong bucket. missing bucket local garage alias in LDAP. ``` I think /pim/setup went successfully, redirected to /pim/inspect, that rerun the builder and failed in the second branch: ```go if pm.bucketId == "" { // ... bInfo, err = grgCreateLocalBucket(candidateName, *keyInfo.AccessKeyId) // ... } else { binfo, err := grgGetBucket(pm.bucketId) if err != nil { pm.errors = append(pm.errors, err) return pm } pm.bucketInfo = binfo //@TODO find my key, check that pm.bucketName exists in bucketLocalAliases nameFound := false for _, k := range binfo.Keys { if *k.AccessKeyId != *keyInfo.AccessKeyId { // not my key continue } if slices.Contains(k.BucketLocalAliases, pm.bucketName) { nameFound = true break } } if !nameFound { pm.errors = append(pm.errors, ErrPimBucketLocalAliasNotFound) return pm } ``` I think we should fix this bug by calling grgGetBucket in the first branch in a loop with an exponential backoff and a limit, and do the validation we do in the normal branch each time, to be sure that we make progress only once the bucket is fully setup.

quentin added the

cat

bug

label 2024-02-13 11:31:04 +00:00

quentin added this to the Deuxfleurs Email project 2024-02-13 11:31:19 +00:00

quentin changed title from ~~Race condition: bucket might not appear~~ to Race condition -> bucket might not be completely setup.

2024-02-13 11:31:51 +00:00

quentin changed title from ~~Race condition -> bucket might not be completely setup.~~ to Race condition -> bucket might not be completely initialized

2024-02-13 11:32:44 +00:00