Again, isn’t that the site’s prerogative?

I think there should at least be a recognized way to opt-out that archive.org actually follows. For years they told people to put

User-agent: ia_archiver
Disallow:
``` /

in robots.txt, but they still archived content from those sites.  They refuse to publish what IP addresses they pull content down from, but that would be a trivial thing to do.  They refuse to use a UserAgent that you can filter on.  

If you want to be a library, be open and honest about it.  There's no need to sneak around.