I have shipped two AI products on my own: theMixtr, which turns a guided chat into a full brand book, and myigirlfriend, an AI companion app with long-term memory. These are the lessons I would give any team adding LLM features.
Keep the model behind your own API
Never call the model directly from the browser. Route every request through your backend so you can check the user’s plan, apply rate limits, log usage and switch providers without shipping a new frontend.
Decide what “good” looks like first
Before tuning prompts, write ten example inputs and what a good output would be. Run them every time you change a prompt. Without this, you are guessing whether a change made things better.
Structure the output
If the result feeds into your product (a palette, a list of tasks, a document section), ask the model for structured data and validate it. Free text is fine for chat; structured output is what makes generation features reliable.
Memory is a design problem
Sending the whole conversation every time gets slow and expensive. Instead, store what matters (names, preferences, important events) as structured facts and include only the relevant ones. That is how a companion can remember a user’s job without re-reading weeks of chat.
Price for usage
AI features have a real cost per request. Free tiers need limits, and heavy users need a plan that covers their usage. Both of my AI products give a small free allowance and then move people onto weekly or monthly plans.
Design for waiting and failure
Stream responses so users see progress. Show clear messages when a limit is reached or a provider is down. A slow answer with feedback feels faster than a quick answer that occasionally hangs.
If you want to add AI to your product, see my AI integration service or read about AI products I have built.