Repository navigation
Conversation
upload caches the checksum of each file, keyed by its path, size and modification time, and looks up the upload history by it. Editing an image can keep both: exiftool -P, when it rewrites the GPS values of an image copied from a camera card (whose modification time has whole seconds), keeps the size and the modification time to the nanosecond. The cached checksum of the image before the edit was then taken, so the edited image matched the upload history of the old one and was skipped as already uploaded. Key the cache by the change time and the file ID as well, as git does. The system sets the change time on every change of a file, and it can not be set back; a file replaced by another gets a new file ID. On Windows, st_ctime is the creation time, so there only the file ID tells an edited file: on NTFS, one replaced by the editor (as exiftool does by default), not one edited in place. FAT and exFAT, as on memory cards, store no change time, and Windows makes their file IDs from the position of the file in its folder, which the replacing file can take: there an edit that keeps the size and modification time may not be told. A file that changes while it is read is not cached, as its checksum may be of the contents before or during the change. Entries cached before this change are not found by the new keys; they are read again once and expire. The command reference lists the new key.
check_upload_history computed the checksum of every file by reading it in full, and kept none, so a check of a folder of dashcam videos on a memory card took minutes, and the upload that followed read every file again to compute the same checksums. Take the checksums from the checksum cache that upload uses, and cache those computed, as upload does: a file is read once for its checksum, by the check, and a later check or the upload looks it up. As in upload, the cache is used only when the upload history is, and a file is taken as unchanged while its path, size, modification time, change time and file ID are. A cache that can not be opened or written is warned about and not used, as in upload. The check still writes no upload history; only the checksum cache. Its output is unchanged, and the progress of the checksums goes to stderr, as in upload. The README and the command reference said that the check changes nothing: they now say that it writes the checksum cache.
caglarpir
force-pushed
the
check-history-checksum-cache
branch
from
November 16, 2026 13:04
3365220 to
9e04495
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Desktop Uploader runs
check_upload_historybeforeupload, and both read every file in full to compute its checksum, so a folder of dashcam videos on a memory card was read twice per session, minutes each time.check_upload_historyusesupload's checksum cache (second commit). It takes checksums from the cache and caches those it computes, so a file is read once for its checksum; a later check, or the upload, looks it up.upload, the cache is used only when the upload history is, and a cache that can't be opened or written gives a warning and is not used.checksum_cache. Its stdout is unchanged; the checksum progress goes to stderr, as inupload. The integration test that asserted the history folder is not created now asserts it holds only the checksum cache. The README and the command reference no longer say that the check changes nothing.The cache key adds the change time and file ID (first commit). The check needs it to avoid false positives, and it fixes
uploadsince #943. The key was path, size and modification time. Editing the GPS of an image withexiftool -Pkeeps both when the image was copied from a camera card (whole-second modification times), so the checksum of the old image was taken and the edited image was skipped as already uploaded.st_ctimeis the creation time, so there a replaced file is caught on NTFS (exiftool replaces by default), not one edited in place. Not tested on FAT or exFAT (memory cards): Windows makes their file IDs from the position of the file in its folder, which the replacing file can take.Checked by hand with 936 GoPro images, a BlackVue video, a GoPro MAX .360 and a CAMM video (
process, thencheck_upload_history):upload --dry_runafter the check computed no checksums; the check after the upload gave the same output as main's.exiftool -Pand uploaded, then edited again (same size and modification time): main'suploadskipped it as already uploaded, and the check with the cache but the old key reported it as uploaded. With both commits, the check reports it as not uploaded anduploaduploads it.Tests: cache hits read nothing; size, modification time, in-place edit (not on Windows) and replacement are misses; a file changed while read is not cached; an edited image is uploaded again; for the check: a second run reads nothing and gives the same results, upload after a check reads nothing, unusable caches, no cache without history, stdout of the command.