Recent article in American Scientist on concerns regarding end-to-end science systems, driven by AI:
The accelerating AI age is driving a push to increase the involvement of AI in every stage of research, creating so-called end-to-end science systems.
Emphasizing speed and productivity at the expense of more deliberative human processes threatens to undermine rigorous investigation, creative discovery, and trust.
Gaffes, disasters, or malfeasance may result unless we design governance into such systems before the stampede to be first and most widely adopted make reform impossible.
Self-Driving Science, by Brian Uzzi, Julio M. Ottino
There are undeniable parallels to the automation tradeoffs in other areas, such as software engineering. Choice quote from the article:
The question is not whether to build end-to-end systems; it is whether we are wise enough to build them in ways that preserve the conditions for wisdom.
Indeed. Incentives can and do lead to formidable conditions.