When public sector AI projects stall, the code is rarely the problem. They stall on questions nobody answered early enough. Who is this for? Who is accountable? What data does it use? How do we know it’s working? What happens when it gets something wrong?
We have been answering those questions with government teams across Canada since 2018. GovLab, our public sector practice, was founded as a collaboration between AltaML, the Government of Alberta and Mitacs. The work runs from wildfire risk prediction to Mountain Pine Beetle detection from satellite imagery to international admissions screening at the University of Alberta, with the City of Edmonton, the City of Calgary and Alberta Blue Cross, alongside the province.
Answering those questions, day to day, is what AI governance in the public sector means in practice. It is the work of agreeing who decides and who carries the responsibility when something goes wrong.
Here are six lessons from that work.
The best AI projects started with a conversation, not a pitch. We sit down with the people who do the job every day, like the caseworker, the analyst, and the reviewer. We ask what slows them down, what they work around, and what keeps them up at night.
What we hear from staff is often different from what leadership expects. A leader might want a dashboard or faster approvals. The person doing the work might just want to stop typing the same information into three systems. Both are valid, but they lead to completely different solutions.
This is where governance matters most. When staff pain points and the leader's wish list don't match, governance gives everyone a fair way to sort it out. It asks:
When we skip this step, we see two things happen:
Listening early, and writing down what we agreed to, prevents both.
The small stuff matters more than you'd think. On one project, analysts asked us to use the same labels as their source documents and to show results as a simple pass or fail. That's how they already think about the work. No leader would put that on a feature list, but it decided whether the tool got used.
Even “accurate” can mean different things to different people. A leader may care about the overall number, while staff care about a single metric that could mislead a client. That's why we agree with our clients on how we'll measure success from the very start, and we name who decides when a solution is good enough.
Privacy and security reviews should start in the first few weeks of an AI project, before anything is built.The most common mistake I see is treating governance as the last hurdle to clear. The team builds something great, then hands it to privacy and security for sign-off. That's when the questions start, and a launch date that looked close is suddenly months away.
In the public sector, most AI tools need a privacy impact assessment and a security risk assessment before they can go live. These reviews take time—and they should. Start them in the first few weeks of the project.
The same goes for upgrades. Moving a live tool to a newer AI model can mean a whole new round of security reviews. Plan for it, or a routine upgrade can quickly turn into a crisis.
AI governance works best when the program area, privacy, IT security, and records management are involved from the start, with one business owner accountable for decisions. No one person “owns” AI governance. Your program area knows the problem. Privacy knows the law. IT security knows the risks. Records management knows what has to be kept. If any of them hears about your project late, they have every right to slow it down.
What works is simple: get these people in the room early and keep them there. Short, regular check-ins, with one clear thread for each project. It's not fancy, but it's the single biggest thing that keeps projects moving.
Name one accountable owner on the business side, too. Someone who can say, “Yes, this is the problem we're solving, and yes, we accept this level of risk.” Without that person, decisions drift.
And plan for people changing jobs. Projects stall when ownership lives in an inbox. Write down who owns what, and plan handoffs for when people move on. Do the same for the solutions themselves: plan for how you'll retire a solution just as much as for how you'll launch it.
Before building anything, know where your data will be stored, who can see it, and whether it could be used to train outside AI models. Trust in public services starts with how information is protected. This means that the first questions any reviewer asks will be about data. Where is it stored? Who can see it? Is it being used to train someone else's AI?
Answer these before anyone asks. Know exactly where your data is stored and who can reach it. Match your security to how sensitive the data is. Get it in writing that your data won't be used to train outside AI models. And build with privacy laws in mind from week one.
A one-page data map goes a long way. Show what data comes in, where it goes, who touches it, and when it's deleted. Reviewers can say yes faster when they can see the whole picture.
The newest model isn't always the right fit for the rules you work under. On one project, we picked an older AI model because it was the one that met the privacy and security requirements. On another, we found more sensitive data than expected partway through and had to stop and rethink where the solution could run. Understand how sensitive your data is first, then plan the technology around it.
The AI tools that earn trust are the ones that help people do their jobs. They don't make the decisions. A tool can draft a document, search through pages of policy, or flag information to remove, but a public servant still makes the final call.
Human review only counts if staff can see where an answer came from and why. If a real person can't check the work, they can't own the decision.
A few years ago, we built an early prototype of one tool just to get feedback. Analysts told us that checking every line took as long as doing the work by hand. Fair enough! The next version flagged only what needed a closer look and showed the source document right beside each answer. That became the feature that frontline staff valued most. One analyst even caught a mistake because she checked items one by one instead of clicking “approve all.”
Here's the tricky part: AI tools help newer staff the most, because they don't know where to look yet. But newer staff are also the least likely to spot a wrong answer. That's why we always show sources, and why we train end-users to check them.
Governance doesn't stop at launch, either. People give plenty of feedback during testing, but it slows to a trickle after launch. And when staff flag an answer as unhelpful, it’s usually the busiest, most engaged users doing the flagging, not unhappy ones. So plan how you'll keep hearing from staff after a solution goes live. Make sure you can see the full story behind a complaint. And And when you make a fix, tell the people who asked for it.
Agentic AI, which can take action on its own, needs stricter guardrails than AI that only answers questions.
The first wave of government AI answered questions and made predictions. A person read the output and decided what to do with it. Agentic systems (the second wave of government AI) remove that step: they search systems, fill in forms, and draft and send documents on their own. When AI can act, mistakes can act too, so clear guardrails matter even more.
Everything above still applies. It just needs to stretch. For these tools, we ask a few extra questions:
Clear limits and good records turn a risky tool into one you can manage.
Good governance doesn't slow AI down. It's what gets AI into the hands of public servants, and keeps it there. Planning an AI project? Here’s where we’d start:
What does keeping a human in the loop mean in practice?
A person makes the final call and can see what the system based it on. The AI drafts, searches or flags. A public servant decides. The review only counts if staff can trace each answer to its source, so the system has to show its working.
How is governance different for agentic AI in government?
Agentic AI can take actions, so governance covers five things a drafting tool never raised: what it may and may never do, which actions need approval, who can change its instructions, how each step is logged, and how fast it can be stopped.
Should government teams always use the newest AI model?
No. The right model is the one that fits the sensitivity of the data and the department's privacy and security rules. Classify the data first, then choose. For sensitive information that is often an older or locally hosted model the security team has already reviewed.
How long does it take to set up an AI governance team?
Convening the group takes two to four weeks, because the people are already on staff. Agreeing how they work together takes about three months. Plan on six months before the group can clear a new use case without escalating every question.
Do you need an AI governance team before starting an AI project?
No. A first project needs a named business owner and the reviews that already apply: the privacy impact assessment, the security threat and risk assessment, and an algorithmic impact assessment where the system supports decisions about people. The group makes later projects faster.
Who should be on an AI governance team in a government department?
Six seats cover most cases: the program area, the privacy office, IT security, records management, procurement, and legal where the system affects entitlements. One seat has to be the accountable business owner, because reviewers can identify risk but accepting it is a business decision.
How do you know if AI governance is working in a government department?
AI governance is working when four things are true at once. There is a current inventory of every AI system in use, including features switched on inside existing licences. Staff can state what they are allowed to do. Reviews close on time. Leadership sees both risk and value.
What rules should public sector staff follow when using ChatGPT or Copilot?
Three rules cover most day-to-day use. Nothing goes into a general AI tool unless it is already public, which rules out personal information, health information and anything security-classified. Every output is a draft until a person checks it. Whoever uses the output owns what it says.
How can AltaML help a public sector team set up AI governance?
AltaML builds the documentation package all three reviews need from one source, so privacy, security and records can run in parallel. AI Foundations and AI Navigator cover readiness and roadmap. The Agentic AI Lab and GovLab put a live use case in front of the group early.
The rules are changing as quickly as the technology, and we’re continuing to learn alongside our clients. If you're working through these same questions, I'd love to compare notes.