"This prompt could have been a function | by Yair Morgenstern | Nov, 2025 | Medium"

Get app

Unknown user

Context

Mcp Server

LLM

This prompt could have been a function

Yair Morgenstern

3 min read

Nov 25, 2025

Listen

Many parts of LLM prompts are deterministic steps that are better done by code.
Converting these to code makes them fast, cheap, consistent and debuggable, and minimizes the context the LLM needs to hold

Context

My team is responsible for over 50 asynchronous processes, so every week there is an issue somewhere in some pipeline. The investigation path is the same every time — on failure a ticket is generated, so the team oncall goes from ticket > workflow run > run logs > stacktrace to investigate.

This consistent pattern marked it as “ripe for LLM automation”, so we set up an LLM with MCP endpoints to fetch ticket data, run data, and logs, but there was only one problem — it didn’t work. There were about 5 tasks that needed to happen in order to provide value, and every time something else tripped it up. The prompt got longer and more explicit, and we still weren’t providing value.

Determinism

This is down to a fundamental aspect of LLMs, which is that unlike the clear investigation process, they are non-deterministic.

So we changed track — instead of having the LLM fetch all the data, we fetch it up-front — using the exact same MCP endpoints the LLM has, but using them as regular HTTP APIs.

For the final stage of investigation, we pass over everything we found to the LLM as part of its prompt. Not only does this mean we could account for context fetching variances, it also freed up the LLM’s context window, so it had less to get “confused” about, which made it do its actual job — the non-deterministic part — better.

Combining contexts

This doesn’t preclude MCP-style LLM data fetching — we can separate data that we can consistently, deterministically fetch, from data that we cannot, and have the LLM build up the missing context we cannot provide it.

Press enter or click to view image in full size

In certain cases, you want the LLM to make a decision whether to fetch some data — in which case as much of the logic as possible should be in the tool rather than the LLM, to again pass the responsibility to code.

Get Yair Morgenstern’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Another possible decision is “what process are we doing here” — if these are distinct enough, one small prompt for decision + one prompt for each process is more manageable than one large multi-branch prompt.

This looks familiar

In 2017 🦕 I discussed that different parts of the ML algorithm pipeline take different views of “which part is The Algorithm”, and this prompt-breaking exercise feels to me like the moving down the stack — from “the algorithm is the process” to “the algorithm is an input-output box”.

Just like with “classical” ML algorithms, the input-output box has disadvantages (there, cost/GPU time — here, that plus non-determinism) that encourage us to push as much into pre-processing and post-processing as possible.

Press enter or click to view image in full size

Geez, nothing changes, huh

The end-to-end process

Gather deterministic data
Convert to token-efficient format e.g. yaml or toon
Trigger LLM process, which
Fetches non-deterministic data, and
Returns results / acts upon the world (ideally creating an approvable action plan rather than actually mutating state)

Changing our LLM interfaces

Much of the work of the past 2 years has been expanding what LLMs can do. Our work, as users of LLMs, is determining what LLMs should do.

Our thinking is effected by both our interface with the LLM, and the LLM’s interface with the outside world — before, during, and after activation.

As our tools evolve to provide us with better human and software interfaces to LLM models, we will be better able to break tasks down and choose LLMs because they are the right tool for the job, rather than the only option that covers all bases.

Context

Mcp Server

LLM