Before AI can help, it needs to know the rules

In the last Insight, I described using AI to help transform a technical standard, a referenced federal regulation, and an existing checklist into a sophisticated first draft of an assessment—one that can ultimately be implemented as an assessment application within GapCross.

It can be created remarkably quickly.

But there is an important part of the process that occurs before the source documents are ever provided to AI.

AI needs instructions.

Asking an AI agent to "create an assessment from this standard" may produce something that looks impressive. It may have sections, questions, explanations, and references.

That doesn't mean it is a good assessment.

Before beginning the work, AI needs to understand what you are trying to build, how it should be structured, what rules it must follow, and where its boundaries are.

In many ways, this isn't very different from bringing a new person onto a project.

You wouldn't hand a new employee three complex documents and say, "Build our assessment."

You would first explain the objective.

You would describe how your assessments are structured.

You would explain terminology and conventions.

You would establish what information belongs in a question and what belongs in guidance.

You would explain how requirements should be referenced, what evidence needs to be captured, how different sources should be treated, and what the person should do when something is unclear.

AI needs much of the same direction.

Define the assessment before building it

Before providing source documents, we establish specifications that define the assessment AI is being asked to create. We provide a similar assessment so AI can learn the expected structure. It needs to understand:

What constitutes a section, question, tips, and references?

How should applicability or conditional requirements be addressed?

What answer options are appropriate?

What information should and should not appear in the final report?

What assumptions is AI permitted to make?

What should happen when two sources appear to conflict?

These decisions create boundaries.

They also create consistency.

Without them, two prompts using the same source material can produce very different results. With an established structure and rules, AI is working toward a defined target rather than simply generating something that resembles an assessment.

Source material comes next

Only after the structure, expectations, and boundaries have been established does the source material enter the process.

A standard establishes requirements.

A referenced regulation may add legally enforceable requirements, definitions, exceptions, or conditions.

An existing checklist may provide another valuable source: the organization's previous experience applying those requirements.

Before development begins, there is another important step: ensuring AI understands the source material and its relevance to the intended assessment.

Again, think about assigning the project to a knowledgeable employee. You would expect that person to review the source documents and discuss their understanding with you before spending days building something.

AI should be treated much the same way.

Does it understand the purpose and organization of the source documents? How do they relate to one another? Are there introductory or summary sections that provide context but do not establish requirements? Are some requirements conditional? Do references provide supporting information, additional requirements, or both?

The answers can affect how the assessment should be constructed.

This creates an important alignment point before development begins. The SME can correct misunderstandings, clarify relationships, and establish expectations so time is not wasted reviewing results that are not on target.

Only then should AI begin turning the source material into the assessment.

AI can then do something humans find extremely time-consuming. It can work across all of that material simultaneously, looking for relationships and developing a structured first draft.

But the same principle from the previous Insight still applies:

AI can identify relationships and propose interpretations. It cannot assume responsibility for whether those interpretations are correct.

That responsibility belongs to the subject matter expert.

The SME is still in charge

The SME now has a very different starting point.

Instead of spending days extracting requirements, organizing them, drafting questions, copying references, and creating initial guidance, the SME can begin by reviewing a substantially developed assessment.

That doesn't make the SME's role less important.

It makes the SME's time more valuable.

The expert can concentrate on questions such as:

Does this accurately represent the requirement?

Does it preserve the intent?

Would this question produce meaningful evidence?

Is the guidance technically correct?

Has an important condition or exception been overlooked?

Would this actually work in the field?

And perhaps most importantly:

What does my experience tell me that isn't written in these documents?

That is where AI-generated content begins to become an assessment built on real expertise.

The preparation is reusable

There is another important productivity gain that can easily be overlooked.

The specifications, structures, and rules used to guide AI do not have to be recreated for every assessment.

Once an organization has defined what a good assessment should look like, those expectations can become part of a repeatable development process.

The next project doesn't start from zero.

Neither does the AI.

This is where the economics of assessment development begin to change significantly.

At GapCross, we have developed specifications, structure, and rule sets that allow intelligent tools to create sophisticated draft assessments from customer source materials. What once required days of manual development can now be accomplished much faster and at a relatively low cost.

The customer's subject matter expert then does what the expert is best qualified to do: review, correct, refine, and add knowledge.

The result isn't an assessment created by AI.

It is an assessment created through a controlled process in which AI handles much of the development work and the SME remains responsible for the expertise.

That distinction will matter as AI becomes a larger part of quality management.