Skip to content

Measure the remaining upload bytes once, not before every chunk - #954

Open
caglarpir wants to merge 1 commit into
mapillary:mainfrom
caglarpir:fix-upload-remaining-once
Open

caglarpir wants to merge 1 commit into
mapillary:mainfrom
caglarpir:fix-upload-remaining-once

Conversation

@caglarpir

@caglarpir caglarpir commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Since #833, UploadService.chunkize_byte_stream seeks to the end of the stream and back before every chunk, to cap the read to the remaining bytes. A video is uploaded as a ChainedIO of one slice per sample: its SEEK_END walks every slice, and its SEEK_SET starts again at the first one and walks forward, so each chunk walked them all and uploading a large video became quadratic. This affects real uploads and --dry_run alike; #833 is not in a release yet.

This measures the remaining bytes once, when the first chunk is read, and counts them down by the bytes read. Reads stay capped as in #833.

Chunking a 21.5 GB GoPro HERO12 video (95,249 samples, 10,190 chunks of 2 MB):

per chunk whole video
before #833 0.5 ms 4.7-5.6 s
main 70 ms on average (40 ms over the first 300 chunks) 715 s
this PR 0.5 ms 5.2-5.3 s

A new test checks that chunking seeks twice whatever the number of chunks (main seeks 8 times for 3 chunks), and another one covers streams that cannot seek.

Since mapillary#833, chunkize_byte_stream seeks to the end of the stream and back
before reading each chunk, to cap the read to the remaining bytes. A
video is uploaded as a stream chained from one slice per sample, and
seeking it walks the slices, so every chunk walked them all and
uploading a large video became quadratic. For a 21.5 GB GoPro HERO12
video (95,249 samples, 10,190 chunks of 2 MB), a chunk took 40 ms
instead of 0.5 ms, and more further into the file.

Measure the remaining bytes once, when the first chunk is read, and
count them down by the bytes read. Reads are still capped as in mapillary#833,
and chunking that video takes 5 s again, as before mapillary#833.
@meta-cla meta-cla Bot added the cla signed label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant