AI talent
AI Data Engineer, Retrieval and RAG Pipelines
Builds and owns the retrieval layer: the corpus, the chunking, the index and the pipeline that keeps all three current.
The role
What the job actually is.
This role owns what the model is allowed to know. It covers ingestion from the systems that hold the documents, the chunking and metadata decisions that determine whether a passage can be found at all, the embedding and index choices underneath, and the pipeline that reflects a change in the source rather than requiring a rebuild. Most retrieval disappointment is a data problem wearing a model problem's clothes: the answer was wrong because the passage was never retrievable, not because the prompt was weak. The part that separates this role from a conventional data engineer is that correctness here is measured on whether the right passage surfaced, not on whether the load completed. Permissions are part of the job rather than adjacent to it, because a retrieval layer that ignores who may see which document is a data-leak path with a chat interface on it.
What we screen for
The questions that separate the field.
Asked by someone who has built the thing, and designed to catch this role's specific failure rather than to confirm a general impression.
- How they chose a chunking strategy on a real corpus, what the first version got wrong, and what they measured to know.
- Whether they can show a retrieval evaluation set and say what recall looked like before and after a change.
- How document permissions survive the pipeline, and what happens when a source document's access changes after it was indexed.
- What their re-index and freshness path is, and whether a source change reaches the index without a human remembering.
The common mis-hire
The one you have probably already made.
A capable warehouse engineer who treats the index as another load target. The pipeline runs, the dashboards are green, and nobody has measured whether the right passage comes back, because the definition of done was arrival rather than retrievability. The tell is an interview with no retrieval evaluation in it.
In the estate
Where this role works, and what we screen it against.
The layers this family works at, lit. These are the tools we screen against. Naming one says we can test for it, not that we have delivered on it.
Evaluation and observability
Spans every layer. Without it a system is shipped on impressions.
Also screened against
- Ragas
Experience and delivery
Copilots and agents inside a business process, and the interaction design that makes an uncertain system usable.
Orchestration and agents
Where an agent's steps, tools and state are defined, and where its failures are caught before a user meets them.
Models
The models themselves, and the platforms an enterprise hosts them through.
Data and grounding
What the model is grounded in, and the integration work that gets enterprise data to where it can reach it.
- Databricks Mosaic AI
- Snowflake Cortex
- Microsoft Fabric
- Azure AI Search
- Pinecone
- Elasticsearch
Also screened against
- pgvector
- Weaviate
- Qdrant
- LlamaIndex
Systems you already run
The seven platform desks Yallo staffs. Almost no AI work is greenfield; it lands here.
Governance, risk and safety
Spans every layer. Named as what governance roles are screened against; what any of them obliges is your counsel's call.
- EU AI Act
- ISO/IEC 42001
- ISO/IEC 23894
- NIST AI Risk Management Framework
- OWASP Top 10 for LLM Applications
Seniority
What changes between mid, senior and lead.
The grade is a description of what the person owns, not a band. Rates come with the shortlist.
- Mid
- Builds and maintains ingestion and indexing against a design and an evaluation set someone else owns.
- Senior
- Owns the retrieval design: chunking, metadata, index choice and the freshness path, and the evaluation that decides whether a change to any of them ships.
- Lead
- Sets the grounding standard across teams, owns the permission model the retrieval layer inherits, and holds recall as sources and access rules change.
In a programme
When this role is needed, and what blocks it.
Blocked by data access rather than by model choice, so it starts at whatever point the programme can get at the documents, which is usually earlier than the plan assumes and later than it should be. This is the role whose absence is discovered during evaluation, when the prompt layer has been tuned twice against a corpus that cannot answer the question. Keep it retained after go-live: sources change, access rules change, and an index that is correct on the cutover weekend drifts from that day onward.
Adjacent
The roles this one is confused with.
Ask
Send the brief, get a screened AI Data Engineer shortlist.
Tell us the programme, the stack and the timeline.