Building Conversational AI Beyond the Demo

The First Transcript Is the Easy Part

A prepared audio file goes in.
Text comes out.
A summary follows.
The demo “works”.

Then the system meets a real conversation.

A caller moves between English and another language; some phrases are easier to explain that way. A product name is pronounced differently. Background noise shifts from a quiet office to a busy street. A new speaker disrupts the conversation. A request that worked flawlessly in testing is now battle-tested, breaking down and exposing weak spots.

The model has not necessarily changed.

The environment has.

“In theory, there is no difference between theory and practice. In practice, there is.” — A well-known engineering adage

A demo proves that a component works. Production tests whether the system holds together.

The Problem Is Larger Than Recognition

Speech recognition attracts attention because its output is visible. But recognition quality is only one part of the operating problem.

Teams also need to understand where audio travels, how a request moves between services, what can be replaced, where configuration lives, how failures are observed, and how a deployment can change without bringing the whole workflow down.

When this knowledge is spread across repositories, services, and individual team members, every change becomes slower and riskier.

Multilingual Speech Makes the Weak Points Visible

Code-switching, where a speaker moves between two or more languages inside a single conversation or even a single sentence, is still an open problem in speech recognition. Researchers continue to identify code-switching as one of the more difficult challenges in speech recognition. Reviews of the field have found significant variation in system performance, while recent work continues to focus on reducing language confusion when speakers move between languages within the same conversation. ¹ ²

This matters because the conversation can seem natural to the people speaking, even when the underlying system loses context silently.

APAC does not create this problem. It exposes it more often.

The Question We Started With

We did not begin by asking how to build another all-in-one conversational AI platform.

We began with a narrower question:

How do we let developers change one part of a conversational system without rebuilding everything around it?

That question led us away from a single product surface and towards a set of replaceable building blocks.

Explore Makimoto

Use these links as the next step for readers who want to explore the project or join the community.

• Join the Makimoto Discord community

Tomorrow, we will explain the architecture decision that followed.

---

References:

¹ Mustafa, M. B., Yusoof, M. A., Khalaf, H. K., Abdel, A., Kiah, M. L., Ting, H. N., & Muthaiyah, S. (2022). Code-Switching in Automatic Speech Recognition: The Issues and Future Directions. Applied Sciences, 12(19), 9541. https://doi.org/10.3390/app12199541

² Liu, H., Zhang, X., Zhang, H., Garcia, L. P., Khong, A. W., Chng, E. S., & Watanabe, S. (2024). Aligning Speech to Languages to Enhance Code-switching Speech Recognition. ArXiv. https://arxiv.org/abs/2403.05887