Mapping the Unseen: How Meta Deployed an AI Agent Swarm to Document Tribal Knowledge in Massive Codebases

The Challenge: AI Tools Without Context

AI coding assistants are highly capable, but their effectiveness depends on how well they comprehend the underlying codebase. At Meta, a large-scale data processing pipeline spanning four repositories, three programming languages, and over 4,100 files presented a daunting test. When the team directed AI agents at this pipeline, they quickly discovered a critical gap: the agents lacked the contextual understanding required to make meaningful edits efficiently. Time and again, the AI would guess, explore, and guess again, often producing code that compiled but introduced subtle, hard-to-detect errors.

This pipeline is built on a config-as-code paradigm, mixing Python configurations, C++ services, and Hack automation scripts. A single operation, such as onboarding a new data field, touches six subsystems that must stay perfectly synchronized: configuration registries, routing logic, DAG composition, validation rules, C++ code generation, and automation scripts. The AI could manage operational tasks—scanning dashboards, pattern-matching historical incidents, and suggesting mitigations—but when extended to development tasks, it faltered. It had no map of the tribal knowledge that engineers carried in their heads: for instance, that two configuration modes use different field names for the same operation (swap them and you get silent wrong output), or that dozens of “deprecated” enum values must never be removed because serialization compatibility depends on them.

The Solution: A Pre-Compute Engine with 50+ Specialized Agents

To solve this, Meta built a pre-compute engine powered by a swarm of more than 50 specialized AI agents. The system used a large-context-window model combined with careful task orchestration to systematically read every file in the codebase and produce 59 concise context files that encoded the tribal knowledge previously locked in engineers’ minds. The result: AI agents now have structured navigation guides for 100% of the code modules—up from a mere 5%—covering all 4,100+ files across three repositories. The system also documented 50+ “non-obvious patterns”, underlying design choices and relationships not immediately apparent from the code itself.

This knowledge layer is intentionally model-agnostic, meaning it works with most leading AI models without requiring proprietary integrations. In preliminary tests, the system reduced AI agent tool calls per task by 40%, as agents no longer needed to explore blindly.

How the System Works

The creation of the context files unfolded in a structured, multi-phase workflow orchestrated by a large-context-window model:

• Two explorer agents mapped the entire codebase, identifying file relationships and dependencies.
• 11 module analysts read every file and answered five key questions, extracting essential knowledge.
• Two writers synthesized the findings into the 59 context files.
• 10+ critic passes ran three rounds of independent quality review to catch errors and gaps.
• Four fixers applied corrections based on critic feedback.
• Eight upgraders refined the routing layer that connects agents to the right context.
• Three prompt testers validated more than 55 queries across five different user personas.
• Four gap-fillers covered remaining directories that had been missed.
• Three final critics ran integration tests to ensure the system worked end-to-end.

All 50+ specialized tasks were orchestrated in a single session, demonstrating the feasibility of large-scale AI collaboration.

Results and Impact

The immediate impact was dramatic: AI agents could now navigate the entire codebase with confidence, reducing exploration time and error rates. The 40% reduction in tool calls means faster development cycles and fewer wasted resources. Moreover, the documented non-obvious patterns serve as a living knowledge base for both human engineers and AI systems, ensuring that critical design decisions are never lost when team members move on or when the code evolves over time.

Self-Maintaining Infrastructure

A key innovation is that the system maintains itself. Automated jobs run every few weeks to validate file paths, detect coverage gaps, re-run quality critics, and auto-fix stale references. The AI is not just a consumer of this infrastructure; it is the engine that runs it. The system becomes a continuously updated map of tribal knowledge, adapting as the code changes, and ensuring that both human and AI developers always have the most accurate and complete context.

By turning the problem of undocumented knowledge into a solvable automation challenge, Meta has shown that AI can be used not only to write code but to understand it at scale—paving the way for more reliable and efficient development in complex, multi-language environments.