Automated Coding for Qualitative Data: Can You Reproduce It?

Automated coding for qualitative data is the use of AI to assign themes and codes to transcripts and open-ended responses, instead of tagging them by hand. The argument about whether to do it is over. Automation won, and almost nobody hand-tags four hundred transcripts anymore.

What is not settled is what came next. Automated coding became infrastructure faster than it became accountable. Most teams now run it like a spellchecker: constantly, invisibly, and without anyone asking what it did or whether it would do the same thing twice.

Key Takeaways

• Automating the coding is settled. Checking the coding never caught up.
• Models get updated. If two waves were coded by different versions, a shift in your themes might be a shift in your coder.
• When every team uses similar models on similar transcripts, the codes stop being a differentiator. What you do with them is all that is left.
• Intercoder reliability works fine on a machine. It is the cheapest credibility you can buy back.

We Dropped the Checks

Hand coding was slow, but it came with a way to prove it was sound: a written codebook, a second coder, an agreement score, a record of decisions. That existed because qual findings get challenged and someone always asks how you know.

Automation took away the work and, in most teams, the checks went with it. Ask which model coded your last study, what the prompt was, and whether anyone read a sample by hand. You will usually get a shrug. Not because people stopped caring about rigor, but because the tool makes coding feel like plumbing. It is still an analysis decision. It is just one nobody writes down.

Your Tracker Has a Coder You Did Not Control For

This is the one that should worry you.

A tracker works because the coding stays the same, so a change in the themes means a change in the market. But the models behind these tools get updated, sometimes quietly, and an updated model will not always sort a borderline answer the same way. That is how the technology works. The problem is that almost no methodology section mentions it.

So when wave three shows rising price sensitivity, you now have two possible explanations instead of one. The market moved, or the coder did. And most teams cannot rule out the second, because nobody wrote down which version coded wave one. Comparability across waves was always the strength of qualitative feedback analysis over time. It now depends on something most teams do not track.

The fix is boring and cheap. Record the model and version next to the codebook. Treat a version change like changing field agency. And when it changes, re-code a sample of the last wave, so you can see whether the coder moved before you decide the market did.

Everyone Is Using the Same Coder

When three agencies pitch the same client using similar models on similar transcripts, the themes come back looking alike. Coding used to be where a team's style showed, in how you drew a line or split a category. It is turning into a layer that gives everyone the same middle-of-the-road read.

There is a quieter version inside a single study. Models are good at producing the expected structure, so the themes you get tend to be the ones the category already talks about. A researcher with ten years in a category notices when something does not fit. A model has no reason to.

The answer is not to tag by hand again. It is to accept that codes are the cheap part now, and compete on what survives: framing, weighting, and the argument you build. Frequency is not importance. One cost complaint from the segment carrying your margin beats eight from anywhere else. No model reads your brief.

Bring Back the Reliability Check

The good news is you do not need a new method. You need to point an old one at a new coder.

O'Connor and Joffe's guidelines on intercoder reliability argue that reliability checks earn their keep by making coding more systematic and transparent, and by helping convince an audience that the analysis can be trusted. None of that depends on both coders being human.

So treat the model as your second coder. Code a sample by hand without looking at its output, compare, and score the agreement as you always would. That gives you a number for the methodology section instead of a vendor's promise.

The disagreements are the useful half. Where you and the model differ, it is usually a code definition that was always fuzzy, an answer that genuinely reads two ways, or context the model never had. All three are worth knowing, and the first one improves your codebook for every wave after. Do it once per study design, not once per study, and it costs almost nothing.

What a machine does to your data should be as easy to inspect as what a junior analyst used to do. In most teams right now, it is a lot harder.

Book a demo with Enumerate to see how coded output traces back to the exchange behind it.