Most AI governance frameworks in Australian businesses rest on a single load-bearing assumption: that a human at the end of the decision chain provides meaningful oversight of the automated systems that fed into it. It is a reassuring idea. It is also, in most real-world configurations, structurally unsound. The reasons have nothing to do with individual competence or bad faith. They are baked into the architecture itself.
If you are a founder, director, or business owner using AI-assisted tools to handle customer data, make staffing decisions, or manage any kind of structured workflow, understanding where this assumption breaks down is now a practical governance question, not just a philosophical one. Australia's Privacy Act reforms have introduced new automated decision-making transparency obligations, and regulators are beginning to look past formal governance documents toward what systems actually do in practice.
What happens before the human decides
The place where AI systems do most of their influential work is not at the decision point. It is upstream of it. Modern enterprise software handles an expanding share of its work through intake screening, triage, prioritisation, summarisation, and routing. All of this happens before a human formally makes a call.
Here is a concrete example that will be familiar to anyone using Microsoft 365. A staff member is asked to make a judgement call on a customer matter. They open the email thread, but before they read it, Copilot has already produced a summary. They read the summary. They form a view. They act on it.
A summary is not a neutral compression of information. It selects, omits, and characterises. Where the human's view is formed on the summary rather than on independent reading of the underlying record, the summarisation has shaped the decision in substance, not merely in form. The decision-maker may not, after the fact, be able to tell whether their view was independently formed or was anchored by the way the system characterised the situation.
That inability to trace the system's contribution retrospectively is not a reason to treat the system as sitting outside the scope of governance. It is an aggravating feature. Decision-shaping that cannot be traced after the fact is decision-shaping that cannot be contested after the fact. For businesses handling personal information, that creates real exposure under Australian privacy law.
The accountability sink
This is where the structural problem becomes visible. Automated systems shape decisions but cannot bear responsibility for them. Responsibility, in any legal sense, attaches only to human or corporate actors.
When a decision pipeline routes visible accountability to a human at the end of the chain, while leaving the upstream systems and the organisation that procured, deployed, and configured them at one or more removes from direct accountability, the result is predictable. Responsibility ends up concentrated on the actor with the least actual influence over the substantive content of the decision.
This is what might be called an accountability sink. It is not the product of anyone acting dishonestly. It is the predictable product of an architecture where the human ratifier is the only visible actor at the moment of decision, and every upstream contribution is too diffuse or invisible to attribute clearly. For directors with governance obligations, or founders whose Privacy Policy commits to responsible data handling, that gap between formal accountability and functional influence is worth taking seriously.
Why override capacity does not fix it
The most common reassurance in these conversations is that the human can always override the system. In principle, yes. In practice, override capacity is present in most modern systems but rarely exercised, particularly under time pressure or where diverging from the system requires the human to document reasons for doing so.
Three operational facts compound this:
- Presentation order matters. A system output that reaches the human before they form an independent view anchors the decision. A post-hoc check is a fundamentally different cognitive exercise.
- Concordance is self-reinforcing. The proportion of cases where the human matches the system's recommendation tends to be high, and once it is high it tends to stay high. High concordance looks like confirmation that the system is working well. It may also mean the system is effectively determinative.
- The cost of disagreement is asymmetric. Where overriding requires a person to document reasons, escalate, or personally own a divergent outcome, the system is functionally in charge even where it is formally advisory.
None of this appears in a governance document that records "human in the loop." All of it is visible from watching how the system is actually used day to day.
Five questions every executive should be asking
If you sit on a board, run an executive team, or carry responsibility for how your business uses customer or employee data, the useful questions are not whether AI is "in use" or whether a human is involved. Going forward, the answer to both will almost always be yes. The useful questions are about the functional influence of the system on actual decisions.
- What information does the system curate before the human sees the file, and can the human detect what was filtered or characterised?
- What proportion of decisions match the system's recommendation, and is that proportion trending upward over time?
- How much time does the human have to engage with the underlying record, and what is the real cost to them of disagreeing?
- Where the system performs intake, triage, or summarisation, is that activity recognised as decision-shaping, or is it being treated as purely administrative?
- If a customer or staff member challenged a decision, could you reconstruct what the system contributed at the time the decision was made?
If the answers to those questions are unclear, you do not yet have oversight. You have a ratifier, and you have an accountability sink behind them.
Moving from formal governance to functional governance
The shift this requires is from asking whether a human is involved to asking what the human can actually do with the information environment the system has produced for them. That is the same shift that mature businesses have already made in safety, in financial controls, and in privacy practice more broadly. The presence of a control is not the operation of a control.
For businesses handling personal information, this framing connects directly to obligations under the Privacy Act. If your organisation's automated systems are shaping decisions about customers or individuals, and those contributions cannot be identified or reconstructed, you are unlikely to be meeting the spirit of the new transparency requirements even if your documentation looks clean. A well-drafted Privacy Policy sets the right foundation, but governance practice needs to match what the policy says. Similarly, where AI tools are being used in contexts involving confidential information from third parties, the protections in a Confidentiality Agreement are only as strong as your ability to demonstrate that information was handled as agreed.
For AI governance specifically, the most useful investments are not at the decision point. They are upstream, in understanding what your systems are doing before the human gets there, and downstream, in the ability to reconstruct what the system did when a decision is later challenged. That is a different governance posture from the one most Australian businesses are currently running, and it is the posture that the next phase of regulation is pointing toward.
Get the right documents in place
Good governance starts with getting your foundational documents right. Mode.law is an Australian legal document platform built for founders and business owners, offering plain-English documents drafted for Australian law. If you are reviewing how your business handles personal data, automated workflows, or third-party information, explore the Mode.law document library at /documents to find the agreements and policies that fit your situation.