Is there some way how express downloading and updating of all sources (including inventories) in some kind of _service file (possibly disabled per default)?

Honestly, I don't know, but it would require patching and moving anyway. Thankfully, those inventories evolve slowly, so using an slightly outdated inventory is much better than not using any at all (current state).

Also, how is this working with different versions of Python? Is that inventory same for multiple versions of Python?

No, it isn't, but again, we don't build the docs for different python versions anyway. Do we really want that? I think, that the implicit policy of using the current major version to build the docs is fully sufficient.

I'm just trying to do the minimal invasive surgery to explicitly providing the missing stuff within the limits of the build service.

But I'm open for ideas, of course.