Repository navigation
Conversation
Since mapillary#833, chunkize_byte_stream seeks to the end of the stream and back before reading each chunk, to cap the read to the remaining bytes. A video is uploaded as a stream chained from one slice per sample, and seeking it walks the slices, so every chunk walked them all and uploading a large video became quadratic. For a 21.5 GB GoPro HERO12 video (95,249 samples, 10,190 chunks of 2 MB), a chunk took 40 ms instead of 0.5 ms, and more further into the file. Measure the remaining bytes once, when the first chunk is read, and count them down by the bytes read. Reads are still capped as in mapillary#833, and chunking that video takes 5 s again, as before mapillary#833.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #833,
UploadService.chunkize_byte_streamseeks to the end of the stream and back before every chunk, to cap the read to the remaining bytes. A video is uploaded as aChainedIOof one slice per sample: itsSEEK_ENDwalks every slice, and itsSEEK_SETstarts again at the first one and walks forward, so each chunk walked them all and uploading a large video became quadratic. This affects real uploads and--dry_runalike; #833 is not in a release yet.This measures the remaining bytes once, when the first chunk is read, and counts them down by the bytes read. Reads stay capped as in #833.
Chunking a 21.5 GB GoPro HERO12 video (95,249 samples, 10,190 chunks of 2 MB):
A new test checks that chunking seeks twice whatever the number of chunks (main seeks 8 times for 3 chunks), and another one covers streams that cannot seek.