How to Evaluate an ML Engineer With No ML Team
Your board deck has an AI line item. Or a customer asked for something that needs a model instead of a rule. Or your data finally got big enough that somebody should be doing something with it. Whatever the trigger, you now have to hire an ML engineer, and the actual problem is that nobody on your team can tell a strong one from someone who has completed a lot of tutorials.
Your standard loop will not save you here. A candidate who handles a systems design round well and writes clean Python tells you very little about whether the model they build will survive contact with production data. This post gives you a screening process you can run without an ML person on staff — and starts with the question most vendors skip, which is whether you need an ML engineer at all.
First: you may not need an ML engineer
At seed through Series A, most features described internally as “AI” are an API call wrapped in careful product engineering. That is not a criticism of the feature. It is a hiring fact, and getting it wrong costs you a quarter.
One of our own client projects is an AI-powered virtual assistant for real estate agents — lead capture, scheduling, SMS, CRM integration, generated property descriptions. The hires that moved that product forward were a full-stack engineer and a UX/UI designer, not an ML engineer. The intelligence was bought. The product around it had to be built.
Three questions, in order:
- Does a general-purpose model already do this acceptably? Then what you need is a strong backend or full-stack engineer with good judgment about latency, cost, prompt regression testing, and failure modes. Hire that.
- Do you have the data in principle, but it is scattered, untrusted, or never logged? Then you need a data engineer first. An ML engineer hired into that situation spends six months building a data platform, badly, and resents every week of it.
- Do you have proprietary, reasonably clean data and a prediction problem where being five percent better is worth real money? Now hire an ML engineer.
Google’s 2025 DORA report found that AI’s “primary role is as an amplifier, magnifying an organization’s existing strengths and weaknesses.” The same is true of the hire. An ML engineer amplifies whether your problem is well-defined and your data is trustworthy. They do not supply either one.
The four roles people call “ML engineer”
Job posts blur these constantly, which is how teams end up interviewing researchers for a pipeline job.
| Role | Owns | Does not own | You need them when |
|---|---|---|---|
| Research / applied scientist | Novel methods, experimentation from first principles | Uptime, production systems | The model is the product. Rare before Series B. |
| ML engineer (production) | Framing, training pipelines, evaluation, deployment, monitoring | Novel research, the warehouse | You have a prediction problem and usable data. |
| ML platform / MLOps | Serving infrastructure, feature consistency, model CI/CD, observability | The models themselves | You already run three or more models in production. |
| Data engineer | Ingestion, warehousing, quality, lineage | Modeling | Your data is not yet trustworthy. |
Most postings titled “ML engineer” at this stage actually describe rows two and four combined. If that is your situation, say so in the job description. Candidates who are strong at both exist; candidates who want only one will self-select out, which saves everyone time.
What to screen for when you cannot screen for math
The instinct is to test the thing you cannot evaluate. Resist it. Ranked by how well each predicts whether the hire works out:
1. Problem framing
Can they take “reduce churn” and return a target variable, a prediction window, a population, a baseline, and the decision the prediction will drive? Most modeling failures are framing failures. Somebody predicts churn at ninety days when the business can only intervene at thirty, and the model is useless while being technically accurate.
You do not need ML knowledge to assess this. You need to hear a good product engineer’s questions asked in ML vocabulary.
2. Evaluation design
Highest signal, least tested. Ask how they would know the model was working. A strong answer covers a baseline worth beating (often “the current rule” or “predict the majority class”), an offline metric tied to the actual decision, a holdout that respects time ordering, and a plan for measuring online what offline cannot.
Leading with accuracy on an imbalanced problem is a flag. No baseline at all is a bigger one.
Then probe leakage directly: “Tell me about a model that looked great offline and failed in production. What happened?” Every practitioner has that story. If they do not have one, they have not shipped one.
3. The unglamorous eighty percent
Pipelines, feature computation that matches between training and serving, versioning, retraining triggers, drift monitoring, rollback. Ask what broke and how they found it — the same question you would ask any senior engineer, which means you can judge the answer.
4. Knowing when not to use ML
A strong ML engineer will tell you a heuristic is fine. Ask outright: “When have you argued against building a model?” Someone who has never made that argument will happily build you one you do not need.
Architecture recall and math depth sit at the bottom of the list — not because they are worthless, but because they are the easiest things to perform in an interview and the easiest things to look up on the job.
One reason plausibility is the specific risk here: in Stack Overflow’s 2025 Developer Survey, 84% of respondents said they use or plan to use AI tools, and the single most-cited frustration, at 66%, was output that is “almost right, but not quite.” That is also the failure mode of a weak ML hire — work that reads correctly and requires someone senior to catch. Your loop has to be built to catch it.
A four-stage loop you can run without an ML person
Stage 1 — Written framing exercise (async, 45 minutes of their time)
Send one paragraph describing a real problem at your company. Ask for a one-page response: target definition, data they would want, baseline, evaluation plan, what they would ship first, and the biggest risk. No code.
Strong: narrows the problem, names what they do not know, proposes something shippable in weeks.
Weak: jumps to model selection, proposes a six-month research effort, no baseline anywhere.
You can grade this without ML expertise, and it filters harder than any coding round.
Stage 2 — Shipped-work walkthrough (60 minutes, you run it)
Have them walk through one ML system they actually shipped. Work down this list and listen for the italicized thing:
- What decision did this model inform, and who made it before? — a human process being replaced or assisted
- What was the baseline, and by how much did you beat it? — a number, and a baseline that existed
- How was the training data labeled, and by whom? — labeling described as the messy problem it is
- What was in your holdout, and why that split? — time-based splits on temporal data
- What did offline evaluation miss? — a specific war story
- How did you know it was still working three months later? — monitoring, drift, a retraining cadence
- What did you cut in order to ship? — scope discipline
- What would you do differently? — a real answer, not humility theater
If they can only answer the first two, they contributed to a model. They did not own one. That distinction is most of what you are buying.
Stage 3 — Practitioner peer screen (60 minutes)
This is the stage you cannot run yourself, and the one teams skip. Options: a senior ML engineer in your investor network, a technical advisor, or a staffing partner whose technical screen is conducted by a practitioner in the same discipline. What does not work is a generalist recruiter with a question sheet — they can verify that someone says the right things, not whether there is depth behind it.
This is why our technical screen is run by a senior engineer in the candidate’s own stack, scored against a rubric, as one of five gates. Between 5% and 10% of applicants finish all five.
Stage 4 — Reference calls, narrowly scoped
One question to former managers: “What did they build that is still running?” Models decay faster than services. Something still in production two years later is a meaningful signal.
Where this approach breaks down
Four situations where the advice above is the wrong advice:
- You need genuine research. New architectures, frontier work, publishable results. This loop will not find that person, and neither will most staffing firms. Hire through the research community and budget accordingly.
- You have no data foundation. Hiring any ML person before the pipes work is premature. Fix ingestion and quality first.
- Nobody internal will own the model afterward. Staff augmentation works well for building and shipping alongside your team. It works badly when the resulting system has no internal home — model ownership decays fast, and a retraining job nobody understands becomes a liability in month eight.
- You have regulatory obligations around model documentation, fairness review, or explainability that you cannot staff. Budget for that separately, and get counsel before you build. This post is not legal advice.
The cost side, briefly
ML engineers price at a premium to general backend engineers in every market, including Latin America. Our published rates by role and seniority run 49–54% below equivalent US base salaries, with payroll, benefits and local compliance included — and the method is published on that page rather than hidden behind a form.
Timezone alignment matters more for this role than for most. Debugging a model is conversational work: someone notices a distribution looks wrong, pulls up a notebook, and talks it through. That does not happen well across a twelve-hour gap. Engineers we place work EST, CST or PST hours.
Start with the framing exercise
If you take one thing from this: send the stage-one written exercise before you schedule a single call. It costs you fifteen minutes to write, it costs a serious candidate forty-five minutes, and it will tell you more than three hours of interviews built around questions you cannot grade.
Then get the peer screen covered by someone who does the work. If that is not on your team, it can be ours — see the five gates every engineer clears before you meet them, including what disqualifies an ML candidate outright. Tell us the role and we will show you who cleared all five.