My starting point would be Chromium's documentation, as - presumably - chrome is one of the most widely used and battle tested, user-facing, third party sandboxes running on end user machines.
With the caveat that I wouldn't necessairly assume this is the cutting edge at this point, and there might be other resources to invest in for server-side sandboxing involving containers or hypervisors, and that I've only actually engaged with the Windows APIs based on that reading.
macOS provides native sandboxing; you can use capabilities at the app level[1] or the sandbox-exec CLI to wrap an existing tool.
For Windows, you probably want WSB[2] or AppContainer isolation[3].
For Linux, the low-level primitives for sandboxing are seccomp and namespaces. You can use tools like Firejail and bubblewrap to wrap individual tool invocations, similar to sandbox-exec on macOS.
Pseudo sandboxing on the fly is an old idea and with its own issues, as proven by classical UNIX approach to launching daemons.