6. Traceability and human-in-the-loop
In any productive process with business decisions, human oversight is not a nice-to-have: it is a governance obligation and, in many sectors (banking, healthcare, insurance, public sector), also a legal one. The EU AI Act (Regulation (EU) 2024/1689) explicitly reinforces the need for meaningful human oversight for high-risk systems, and Article 22 of the GDPR (Regulation (EU) 2016/679) protects individuals from purely automated decisions with legal or similarly significant effects.
The human-in-the-loop pattern in BPM with agents works like this. Each agent returns, alongside its output, a confidence score (0-1). If confidence exceeds the configured threshold (for example 0.85), the process continues automatically. Below that, the flow routes the task to a human inbox with full context: original input, agent proposal, justification and comparable data.
The full audit trail includes: timestamp of each invocation, LLM model used, prompt version, exact input sent (sanitised if it contains personal data), structured output, confidence score, final decision (automatic or human) and who took it if human. This record is what enables responding, months later, to an internal audit or a regulatory request.
Without explicit traceability of the agent's reasoning and without confidence-based human escalation, there is no productive AI project. There is only a demo.