Deuxfleurs/dxfl
15
1
Fork 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity Actions

deploy: detect unmodified files and avoid uploading them again #10

Merged
Armael merged 3 commits from deploy-incremental into main 2025-02-23 11:19:59 +00:00
Conversation 0 Commits 3 Files changed 1 +90 -11
Armael commented 2025-02-22 18:18:30 +00:00
Owner
Copy link

The strategy I'm implementing here is similar to what rclone is doing (AFAICT):

  • use the metadata available from ListObjects in a best-effort fashion (file size here, rclone also uses modification dates)
  • when uploading objects, compute their checksum and store it in a custom metadata field of the object
  • if comparing file sizes is not enough to decide, fetch (with HeadObject) the object metadata and use the checksum field stored there.
The strategy I'm implementing here is similar to what rclone is doing (AFAICT): - use the metadata available from ListObjects in a best-effort fashion (file size here, rclone also uses modification dates) - when uploading objects, compute their checksum and store it in a custom metadata field of the object - if comparing file sizes is not enough to decide, fetch (with HeadObject) the object metadata and use the checksum field stored there.
Armael added 3 commits 2025-02-22 18:18:30 +00:00
Armael force-pushed deploy-incremental from 8231dc6e1f to 8e734b7f5c 2025-02-22 18:34:05 +00:00 Compare
Armael force-pushed deploy-incremental from 8e734b7f5c to a3496f1542 2025-02-22 18:37:18 +00:00 Compare
Armael force-pushed deploy-incremental from a3496f1542 to 408c6e0fe2 2025-02-22 18:43:56 +00:00 Compare
Armael force-pushed deploy-incremental from 408c6e0fe2 to 79c7ef3194 2025-02-23 11:11:34 +00:00 Compare
Armael added 1 commit 2025-02-23 11:17:49 +00:00
Armael merged commit 2ccfa63e7a into main 2025-02-23 11:19:59 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/dxfl#10
No description provided.
Delete branch "deploy-incremental"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?