Skip to content

Cap upload stream reads to remaining file bytes - #833

Merged
caglarpir merged 2 commits into
mapillary:mainfrom
rnissl:mly/01-upload-read-cap
Nov 11, 2026
Merged

caglarpir merged 2 commits into
mapillary:mainfrom
rnissl:mly/01-upload-read-cap

Conversation

@rnissl

@rnissl rnissl commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Cap FileIO.read to the remaining file size so a large chunk size cannot allocate a 1 GiB buffer per JPEG worker.

Test plan

  • tests/unit/test_upload_api_v4.py (includes test_chunkize_caps_read_to_remaining_bytes)

Rebased onto current mapillary/mapillary_tools main (d1b82ab). Independent of the GoPro Max / custom FFmpeg stack.

A huge chunk size used to make FileIO allocate the full buffer and OOM with many JPEG workers.

Co-authored-by: Cursor <cursoragent@cursor.com>
@meta-cla meta-cla Bot added the cla signed label Sep 20, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
@caglarpir

Copy link
Copy Markdown
Contributor

Thanks for this batch of PRs, and for rebasing and splitting them up so each one stands on its own. I went through all six and left a note on each.

This one looks good. With the default 2 MB chunks it doesn't change anything, but if someone sets a large MAPILLARY_TOOLS_UPLOAD_CHUNK_SIZE_MB there's no reason to allocate more than what's left of the file. CI is green, so I'll merge it.

@caglarpir
caglarpir merged commit 34316a9 into mapillary:main Nov 11, 2026
16 checks passed
@rnissl

rnissl commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for going through the six pulls, and for merging #833.

The sampling stack is easier to judge by running it. Two zips:

Stock FFmpeg cannot stitch a GoPro .360. This build provides gopromax2_cuda. Install the tools from the first zip and point them at the binary:

set MAPILLARY_TOOLS_FFMPEG_PATH=C:\path\to\ffmpeg.exe
set MAPILLARY_TOOLS_FFPROBE_PATH=C:\path\to\ffprobe.exe

GPU JPEG also needs nvjpeg64_13.dll and cudart64_13.dll next to ffmpeg.exe, copied from a CUDA toolkit bin directory. The driver does not ship nvJPEG. Without those two DLLs the stitch still runs and JPEG falls back to software mjpeg.

One call on a directory that holds a few .360 files:

mapillary_tools video_process_and_upload C:\clips --user_name YOUR_NAME

That samples by distance, writes the JPEGs, and uploads them. An NVIDIA GPU is required for the CUDA stitch.

If that is interesting, I would not reopen these branches as they are. 0.15.0 moved the video clock and sample_video, so the next step is new pulls cut against that release. On the FFmpeg side, the approach that looks most promising is a Vulkan implementation of the stitch.

@caglarpir

Copy link
Copy Markdown
Contributor

Thanks for putting the builds together and writing up the steps, that's a lot of work. I won't be able to try them, though: I don't have a Windows machine with an NVIDIA GPU to test on, and I can't install prebuilt binaries from outside links anyway.

From your description, the stitch needs a patched FFmpeg build, an NVIDIA GPU and the CUDA DLLs. mapillary_tools has to work with the FFmpeg people install from their package manager or ffmpeg.org, so video_process can't depend on a filter that only exists in a custom build. If the stitch makes it into upstream FFmpeg one day, for example the Vulkan version you mention, that would change the picture.

In the meantime, GoPro MAX and MAX 2 .360 files can be uploaded as they are with mapillary_tools process_and_upload, and Mapillary stitches them after upload, so there's no need to stitch locally.

As for new pulls against 0.15.0: small changes that work with stock FFmpeg and are wired into a command end to end, like the glob import paths from #834, are the easiest for us to review and test.

@rnissl

rnissl commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

The invitation is the tools adapted to the FFmpeg 9.1 line, main 0e930b6. 9.1 is not tagged yet. It is current git master, which is where the jpegapp bitstream filter and its sidecar option are. The latest release, 9.0.2, is older than that and does not have the filter.

Those master builds are the ones linked from https://ffmpeg.org/download.html :

Point the tools at that binary with MAPILLARY_TOOLS_FFMPEG_PATH and MAPILLARY_TOOLS_FFPROBE_PATH. This is what the jpegapp path needs for mp4 files. The sidecar is passed from a file, because the hex does not fit on the command line.

I am not asking for a review of the source yet. I would like to show the work on real files. If that is useful, we can decide what a pull against 0.15.0 should contain.

If .360 files should be part of that look, the stitch is not in these builds. It needs a Vulkan FFmpeg, which I expect to be in 9.2:

git clone https://code.ffmpeg.org/rnissl/FFmpeg.git
git checkout pr/v360-vulkan-gopromax

That branch is 2f5e9b8bc1. It builds with --enable-vulkan --enable-libplacebo. The same tools then use v360_vulkan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants