You’ll start with a project that is already built out using Next.js TypeScript (of course), MongDB through the Mongoose ORM, and Redis for caching.

The first couple of days you will implement fundamental AI app features building out a Naive agent. You’ll start by hooking up an LLM of your choice to the Node.js app and stream responses to your Next.js app using the Vercel AI SDK and you'll implement a search tool it can use to supplement its knowledge when conversing with users.

Chats with an LLM don’t save themselves so you will also save conversations to the database as well.

The first real differentiator between a vibe-coded side project and a production-ready product you can feel confident putting in front of customers is observability and evals.

You need to know what is going on with your LLM calls as well as need an objective means to judge the output that LLMs are producing.

This is exactly what the next few days are about. You’ll hook up your app to LangFuse and get familiar with looking through traces produced by the application.

Once you can see what your LLM is doing, now’s the time to test inputs and outputs of your agent using evals. Evals are the unit test of the AI application world and we’ll start by wiring up Evalite which is an open-source, vitest-based runner. You’ll learn about what makes good success criteria and build out your evals including implementing LLM-as-a-Judge and custom Datasets specific to your product. We’ll also discuss how you can capture negative feedback intro traces that you can feed into your app to make it better.

Up until this point, your app is driven by an increasingly large prompt within your Node.js app and some of the prompts coming from your Next.js app, that will become unwieldy and impossible to test and iterate on when your app complexity grows.

We’ll take these next two days to revisit the over-all architecture of our application and refactor it to better handle complex multi-step AI processes. The primary idea behind this refactor is called Task Decomposition where you allow a smart LLM model determine the next steps to take based on the current conversation but allow room to hand off actual action to focused or cheaper models.

The last few days we will evaluate what our deepsearch agent is and how we can further optimize output. You’ll learn the differences between “Agent” and “Workflow” and see how in this use-case we’ll lean harder into workflow patterns to build a more reliable product.

In ai-land, this pattern is called the evaluator-optimizer loop which effectively means that if the agent has enough information, it will answer the question presented but if it doesn’t it will search for more information. With this pattern defined we’ll embrace and optimize around its design.

By the end of this cohort you will be confident in building out AI applications that are reliable and will improve through iteration and user feedback. LLM models and the whole AI field is changing rapidly so understanding these fundamentals will give you a foundation to navigate building applications for years to come.