The theory with crawling is it has discovery built into it, no? You follow outbound links and discover domains that way. So you need some seeds, but otherwise you discover based on what other people already know about.

To me the problem seems like a few submarines in a cave. They can each see a little bit of what’s around them, and then they can share maps. Like the minimum knowledge of the internet is one’s own explorations. As one browses the web, their sensors are storing everything they see. It also actively searches with other agents, automatically crawls on its own like active sensors on a submarine always mapping out the environment.

Then, in the presence of other friendly subs, you can trade information. So one’s own personal and small map of the internet can get merged and mixed with others to get a more and more complete version.

Obviously this can be automated and batched, but that’s sort of the analogy I see in the real world: multiple parties exploring an unknown/changing space and sharing their data to make a map.