By design, an "instructed" model is no longer a model of the base training corpus, but rather of an idealized version of it, never clearly defined. It can still score sequences of natural language but what it now models is a vision of what language should be in a specific setup.
A good summary.
That GPT-4 was launched as a product when it's at the far edge of LM science is obviously a concern given its transformative potential.
“I’m not telling you to make the world better I’m just telling you to live in it. Not just to endure it, not just to suffer it, not just to pass through it, but to live in it. To look at it. To take chances. To make your own work and take pride in it."
—Joan Didion
It's a good question, what am I worried about? Well, let me share a scenario.
It's April 2026. I wake up in the morning and check Hacker News. "Hundreds of Starlink satellites burn up in the atmosphere."