Infrastructure as code makes one promise: the repository is the truth. Read the code, and you know what is running.
Drift is what happens when that stops being true.
It is not a rare failure mode. Run terraform plan against a production account that has been live for two years and has never had scheduled drift detection, and you will almost certainly get changes you did not write. The interesting question is not whether you have drift — it is where it came from, because the sources are more legitimate than people expect.
Where drift actually comes from
The 3 a.m. fix
An incident is in progress. Someone widens a security group in the console to restore access, or bumps an ASG's max size to absorb load. It works. The incident closes.
The change is correct. The follow-up pull request is the part that does not happen, because by the time the postmortem is written the urgency is gone.
This is the most common source of drift, and it is worth being clear: the engineer did the right thing. You do not want a policy that makes people route an outage fix through code review. You want a system that notices the divergence afterwards.
Cloud-side defaults you never declared
You create a resource with a subset of its attributes. The provider fills in the rest. Later, AWS changes a default, or adds a field, or applies a service-level setting automatically.
Your code never mentioned the attribute. The cloud has an opinion about it. Terraform now reports a difference on a field you have never thought about, and it will report it on every single plan until someone either declares it or ignores it.
Controllers that mutate by design
This one breaks people's mental model. If Terraform provisions an EKS cluster and a Kubernetes autoscaler manages the node group's desired capacity, then two systems own the same field. That is not a bug in either one.
Terraform sees a value it did not set and wants to correct it. The autoscaler sees Terraform's correction and adjusts again. Left alone, they fight, and the fight shows up as a permanent diff.
resource "aws_autoscaling_group" "workers" {
min_size = 2
max_size = 20
# The cluster autoscaler owns this at runtime. Without the ignore,
# every apply drags capacity back to whatever it was at plan time —
# which, during a scale-out, means actively fighting the scaler.
lifecycle {
ignore_changes = [desired_capacity]
}
}
The rule of thumb: if something other than Terraform legitimately writes to a field, Terraform must be told to stop reading it. ignore_changes is not a workaround here — it is the correct declaration of ownership.
Out-of-band automation
Backup tooling that tags resources. A security scanner that adds a rule. A cost tool that applies a scheduling tag. Another team's pipeline that has permission to your account.
Each is doing its job. Each writes to resources Terraform believes it controls.
Why the second-order damage is worse
The drifted resource itself is usually harmless. A tag, an unexpected default, a capacity number.
The damage is what noise does to the review process.
A clean terraform plan is a security control. When it says 3 to add, 1 to change, 0 to destroy, a human can read all four and confirm they are intended. That review is often the last checkpoint before a change reaches production.
A plan with forty pre-existing diffs is not a security control. It is a wall of text people scroll past. And once people scroll past plan output as a matter of routine, the one line that says a security group is opening to 0.0.0.0/0 scrolls past with everything else.
Drift does not usually cause the incident. It removes the check that would have caught the incident.
Detection has to be scheduled
The instinct is to check drift during deploys. That is too late and too infrequent.
Consider the timeline. Drift is introduced on a Tuesday. Your next apply to that module is three weeks out, because the module is stable and nobody has needed to touch it. For three weeks, your infrastructure does not match your code and nothing in your tooling knows.
If the drift was a widened security group from an incident fix, that is three weeks of exposure that no scan reports, because the scan checks your repository and your repository is fine.
The fix is a scheduled, read-only run against every workspace, on its own cadence — daily for production, weekly is defensible elsewhere:
# Read-only. Refreshes state against the provider, reports differences,
# changes nothing. Exit code 2 means "drift detected".
terraform plan -detailed-exitcode -refresh-only
-refresh-only is the important flag. It asks the narrow question — does reality match state? — without mixing in pending code changes that have not been applied yet. That separation matters, because "someone changed the cloud" and "someone changed the repo" need different responses.
Exit codes: 0 means no changes, 1 means error, 2 means drift. Wire 2 to an alert that names the workspace and the resource. Do not wire it to a dashboard nobody opens.
Responding without creating a second problem
When detection fires, there are exactly three valid responses. Pick one deliberately.
Codify it. The change was correct and should persist. Write it into the module, open a PR, apply. The security group stays wide because it needs to be, and now the code says so and the next reviewer can see it.
Revert it. The change was wrong or unauthorised. Apply the module and let Terraform restore the declared state. Then find out how it happened — a drift you can revert but not explain will come back.
Declare shared ownership. Something else legitimately owns the field. Add ignore_changes with a comment saying what owns it. The comment is not decoration; the next person to read that block needs to know whether the ignore is a considered decision or a suppressed alarm.
The response you must not pick is the fourth one: leaving it. Every un-triaged diff makes the next plan harder to read, and the noise compounds until nobody reads any of it.
Prevention, in order of what actually works
Remove console write access from production. Read access stays — people need to debug. Write access goes behind a break-glass role that is time-limited and logs loudly. This single change eliminates most human-origin drift, because the friction arrives at the moment of the change rather than in a policy document.
Declare every attribute you care about. An attribute you do not set is an attribute the provider decides. Explicit beats inherited.
Make ownership boundaries explicit in code. Every field written by a runtime controller gets an ignore_changes and a comment. Do this when you write the module, not after the first fight.
Treat a drift alert like a failed test. Something to triage today, not to add to a backlog. The moment drift alerts become a queue, they become noise, and you are back where you started.
DevLift's infrastructure agent, Sage, runs continuous drift detection across Terraform, CloudFormation and Kubernetes — and when it finds a divergence, it proposes the reconciliation as a reviewable pull request rather than an alert you have to interpret. See it against your own stack.