i'm a software engineer based in the philippines, driven by curiosity and whatever i find interesting. i learn fast with the caveat of goldfish memory (which i call my context lol) and i'm usually self-motivated that i forget to eat.
this is not like other portfolios. feel free to have a look around and explore what makes me, me.
@schmayterling
open githubtypescript, python, rust, go, infrastructure, agentic systems.
memory system, subsystem model, weeks of neuroscience papers.
someone else is
on the other end.
structural code intelligence for developers and agents, with answers grounded in the repository graph.
an experimental memory layer for long-running agents, state tracking, and better recall.
native macos music library and playback app.
2,239-trial benchmark across 8 OSS repos: Atlas beats a text-search baseline by +0.223 deterministic, +0.127 LLM-judge, at 42% fewer tokens.
markdown-native personal knowledgebase with an agent that reads and mutates notes through explicit tools.
i wrote a whole WFD about how fast diffusion models are. turns out, at GPT 5.2 quality, the speed advantage is negligible.
local llm wrapper that reads and controls a live chromium page through the devtools protocol.
i watched a kdrama about a girl who forgets everything every morning and realized my AI agent has the same condition. so i built it a brain.
unfolding paper…
wfd / Apr 19, 2026 / 17 min
draft, partially incomplete, unlisted this is a working draft. sections are still being filled in and numbers may drift between versions. Atlas is a project i've been tinkering with on the side. if you're reading this, i must think you're cool, so please don't share it :3
pre-publication, do not share i'm collaborating with other researchers to publish this on arxiv and get it peer reviewed. target is ~june/july 2026 as i keep refining Atlas. please don't share this first draft in the meantime.
We evaluate Atlas, an MCP server that exposes a tree-sitter + TS-compiler-derived code graph to LLM agents, against three baselines on a 2,239-trial head-to-head benchmark across 8 OSS repositories spanning 5 programming languages. On gpt-5.4-nano, Atlas achieves a deterministic score of 0.62 vs the text-search baseline's 0.40 (paired delta +0.223, 95% CI [0.185, 0.262]) and an LLM-judge score of 0.54 vs 0.42 (paired delta +0.127, 95% CI [0.087, 0.167]). Both intervals exclude zero. Atlas also uses 42% fewer tokens and 49% fewer tool calls than the text-search baseline. Competing structural-code-intelligence MCP servers (codebase-memory-mcp, Chunkhound) score significantly below the baseline on the same task set. Atlas wins on every individual corpus including six it was authored without seeing, and on 5 of 7 capabilities on the LLM-judge axis (losing only on graph-querying and call-tracing, where its terse symbol-list answers are penalized by judge verbosity bias despite higher deterministic correctness).
LLM agents increasingly call into external tools to navigate codebases. The quality of the tool surface shapes the agent's ability to answer structural questions (where is X defined? what calls X? what does X depend on?) and content questions (how many files mention Y?) without burning context on grep traversal. Multiple paradigms have emerged:
read_file + grep + glob, the lower bound an LLM with shell access can achieveThis benchmark answers two questions:
We make no claims about strong-model performance (Claude Opus 4.7, GPT-5.4-pro, Gemini 3.1 Pro, or the upcoming GPT-5.5 "spud"); those would close the gap between agents because better routing decisions reduce baseline failure modes. We also do not test mixed-mode agents (Atlas + grep), which would be production-realistic but methodologically muddier (see §3.1 design decisions).
The lower bound on tool-augmented coding agents is grep + glob + read_file. This is what a vanilla LLM with shell access can do. It works well for content-defined questions (substring search, comment scans) and breaks down on structural questions (call traces, blast-radius) where ad-hoc regex over file contents fails to match aliased imports, polymorphic dispatch, or factory-pattern flow. Atlas's design assumes this is the floor, not a ceiling.
Chunkhound, Sourcegraph Cody, Vexp, and Augment all bias toward dense retrieval over a chunked code corpus. The model retrieves k similar chunks and reasons over them. Strengths: handles fuzzy intent ("find error-handling logic"). Weaknesses: chunk boundaries lose structural context (a function body retrieved without its containing class), retrieval recall is bounded by chunking quality, and embeddings degrade with project size (more chunks = more noise per query). Chunkhound is included in this benchmark as the strongest representative of this paradigm.
codebase-memory-mcp (cbm), Atlas, and a handful of academic systems use tree-sitter or a language-specific compiler to extract symbols + edges into a queryable graph. Atlas adds TS-compiler resolution for cross-file edges, a typed scoring layer (file-path, symbol-set, count, structural predicates) for deterministic eval, and exposes 11 MCP tools shaped per-question-type. cbm uses a 14-tool surface biased toward graph queries and free-text intent. Both are included.
LLM-judge has documented biases (verbosity, position, self-preference; see arxiv 2306.05685, judgebiasbench). We mitigate by:
read_file/grep/glob to spot-check claims (still rubric by default for methodology continuity)Deltas are reported only when the 95% CI from a paired bootstrap excludes zero.
Three load-bearing methodology choices precede every number below.
3.1.1 Tool-augmented agents have NO text-tool fallback. Earlier iterations gave Atlas, cbm, and Chunkhound a grep + glob + read_file fallback alongside their native tools. Instrumented runs showed the Atlas agent reaching for text tools on 44% of calls, meaning some of Atlas's apparent advantage was carried by grep. In this benchmark, text tools are available only to the baseline agent. Atlas, cbm, and Chunkhound are tested on their native tool surface alone. Tradeoff: this is a clean "Atlas-pure" capability measurement, not a production-realistic mixed-mode bench. Mixed-mode is future work.
3.1.2 Dual judges, never cross-aggregated. Every trial is scored twice:
min-results, has-symbol, contains-file). Objective, reproducible, no LLM in the loop.Both are reported. When they agree directionally, the signal is credible; when they diverge, the diff itself is a finding.
3.1.3 Paired-bootstrap confidence intervals. Every Atlas-vs-baseline delta has a 95% CI from 10,000 paired resamples. Pairing is by (taskId, trial) so task difficulty stays constant across agents. We observed 5-trial Atlas-vs-baseline deltas swinging ±0.04 across runs on the same commit, so tight CIs are the difference between real progress and noise. Throughout this report, we report deltas ONLY when the CI excludes zero.
8 OSS repositories, pinned to specific SHAs. Selected to span 5 languages and 4 mixed-language combinations.
| name | language(s) | files indexed | symbols | edges | repo | ref |
|---|---|---|---|---|---|---|
| ripgrep | rust | 98 | 3,476 | 11,093 | BurntSushi/ripgrep | 4649aa97 |
| zod | typescript | 339 | 5,217 | 21,126 | colinhacks/zod | 7baee4e1 |
| pydantic | python | 180 | 5,056 | 8,827 | pydantic/pydantic v2.10.0 | 910bc54b |
| hugo | go | 875 | 13,227 | 84,939 | gohugoio/hugo v0.140.0 | 3f35721f |
| gradio | python + typescript | 1,050 | 6,772 | 25,678 | gradio-app/gradio v5.50.1 | 275cd048 |
| turbo | rust + typescript | 1,274 | 9,363 | 30,630 | vercel/turborepo v2.5.0 | df394be8 |
| unleash | typescript + go | 4,617 | 25,251 | 93,669 | Unleash/unleash v6.4.0 | 8a79b527 |
| expo | typescript + jsx (react native) | 6,844 | 40,714 | 103,244 | expo/expo sdk-52 main | 4f07effc |
Total: 15,277 files, 109,076 symbols, 379,206 edges across 8 repositories.
112 tasks total: 14 per corpus, distributed across 7 capabilities (2 per capability per corpus):
| capability | n per corpus | typical question |
|---|---|---|
| indexing | 2 | "how many python files does the project have?" |
| discovery | 2 | "find the BaseModel class" |
| code-access | 2 | "where is validate_email defined?" |
| call-tracing | 2 | "what calls Searcher::new?" |
| graph-querying | 2 | "what depends on ZodNumber?" |
| file-navigation | 2 | "list rust files under crates/searcher/" |
| text-content | 2 | "how many files mention pcre2?" |
Each task declares expected (typed), intent (prose, shown to the agent), and a text_search_strategy (if the task is comparable to text-search). The deterministic scorer uses expected; the agent never sees it; the LLM judge sees both.
Ripgrep and Zod tasks are hand-authored; the 6 newer corpora use auto-generated tasks from a templating script that mines top-inbound symbols from the indexed sqlite db and emits canonical templates. Auto-generated tasks have wider tolerance bands and are explicitly marked for transparency. Results are reported separately when relevant.
| agent | tool surface | size |
|---|---|---|
| baseline | read_file, grep, glob | 3 |
| atlas | 11 atlas_* MCP tools (search, semantic_search, content_search, overview, symbol_detail, deps, blast_radius, trace, call_sites, test_coverage, files) | 11 |
| cbm | 14 cbm MCP tools | 14 |
| chunkhound | 4 chunkhound MCP tools | 4 |
All four agents share the same system prompt, the same model, the same trial budget, and the same scoring. Tool descriptions are each agent's own (Atlas's by us, cbm's and Chunkhound's verbatim from upstream); we explicitly do not prompt-engineer cbm or Chunkhound to make their tools easier to use. That's the same test Atlas faces.
openai/gpt-5.4-nano via OpenRouter| agent | det score | LLM-judge score | det Δ | LLM Δ | tokens (M) | $ inference | $ judge | tools/task |
|---|---|---|---|---|---|---|---|---|
| baseline | 0.40 | 0.42 | n/a | n/a | 8.58 | $1.36 | $0.54 | 3.0 |
| atlas | 0.62 | 0.54 | +0.223 ✓ | +0.127 ✓ | 5.02 | $0.74 | $0.52 | 1.5 |
| cbm | 0.14 | 0.16 | −0.260 ✓ | −0.256 ✓ | 11.57 | $1.24 | $0.44 | 5.1 |
| chunkhound | 0.28 | 0.24 | −0.113 ✓ | −0.172 ✓ | 6.24 | $1.09 | $0.40 | 2.7 |
Paired-bootstrap 95% CIs (10,000 resamples, paired by (taskId, trial)):
Atlas is the only tool-augmented agent that beats the text-search baseline on both axes, at 42% fewer tokens and 49% fewer tool calls. cbm and Chunkhound both score significantly below baseline despite costing more in tokens (cbm) or being similarly efficient (Chunkhound).
Atlas vs baseline, mean over 70 trials per corpus (14 tasks × 5 trials).
| corpus | atlas det/LLM | baseline det/LLM | det Δ | LLM Δ |
|---|---|---|---|---|
| zod | 0.87 / 0.79 | 0.55 / 0.56 | +0.319 | +0.232 |
| ripgrep | 0.80 / 0.82 | 0.61 / 0.71 | +0.189 | +0.109 |
| hugo | 0.66 / 0.57 | 0.43 / 0.39 | +0.235 | +0.175 |
| gradio | 0.64 / 0.57 | 0.41 / 0.38 | +0.233 | +0.190 |
| turbo | 0.59 / 0.50 | 0.42 / 0.42 | +0.169 | +0.077 |
| pydantic | 0.54 / 0.51 | 0.41 / 0.47 | +0.129 | +0.042 |
| expo | 0.43 / 0.33 | 0.28 / 0.30 | +0.149 | +0.033 |
| unleash | 0.42 / 0.27 | 0.06 / 0.12 | +0.363 | +0.158 |
Atlas wins every corpus on both axes. The smallest LLM-judge wins (expo +0.033, pydantic +0.042) are within rough noise tolerance and should be read as ties. The largest wins are on zod (+0.319 det), the original hand-authored corpus, and unleash (+0.363 det), a typescript+go monorepo with 25k symbols where baseline collapsed to 0.06.
Importantly, the gap holds on the 6 corpora Atlas was authored without seeing. If the original 28-task selection had simply been favorable, the gap would shrink on new corpora. It does not. It widens on some (gradio +0.190 LLM-judge, hugo +0.175, unleash +0.158) and matches ripgrep's hand-tuned baseline (+0.109 LLM-judge) on others.
Aggregating across all 8 corpora.
| capability | atlas det/LLM | baseline det/LLM | det Δ | LLM Δ |
|---|---|---|---|---|
| graph-querying | 0.86 / 0.20 | 0.42 / 0.36 | +0.438 | −0.156 |
| text-content | 0.60 / 0.55 | 0.28 / 0.17 | +0.321 | +0.382 |
| indexing | 0.45 / 0.37 | 0.20 / 0.16 | +0.246 | +0.213 |
| discovery | 0.98 / 0.96 | 0.76 / 0.76 | +0.227 | +0.198 |
| code-access | 0.83 / 0.85 | 0.66 / 0.65 | +0.171 | +0.203 |
| file-navigation | 0.48 / 0.44 | 0.35 / 0.30 | +0.122 | +0.136 |
| call-tracing | 0.13 / 0.43 | 0.09 / 0.51 | +0.038 | −0.087 |
Atlas wins on all 7 capabilities deterministically but only on 5 of 7 on the LLM-judge axis. The two LLM-judge losses are structurally informative:
atlas_trace returns 0 paths from factory functions to methods on returned classes, and baseline's grep-then-read-the-file approach happens to surface adjacent context that the judge credits.The largest combined (det + LLM-judge agree) wins are text-content (+0.321 det / +0.382 LLM) where Atlas's atlas_content_search (rg-backed, scoped to the indexed file set) outperforms unscoped grep on noisy repositories, and indexing (+0.246 / +0.213) where Atlas's atlas_status answers structural inventory questions baseline has to estimate by globbing.
| agent | total tokens | per-task tokens | per-task tool calls | per-trial wall (median) |
|---|---|---|---|---|
| baseline | 8.58M | 15.3k | 3.0 | 5.57 s |
| atlas | 5.02M | 9.0k | 1.5 | 4.50 s |
| cbm | 11.57M | 20.7k | 5.1 | 13.73 s |
| chunkhound | 6.24M | 11.2k | 2.7 | 26.47 s |
Atlas uses ~58% of baseline's tokens and ~43% of cbm's tokens. Mean tool calls per task: 1.5 (atlas) vs 3.0 (baseline) vs 5.1 (cbm). Atlas converges on an answer in fewer cycles because individual tool returns carry more structural signal per byte. Wall-time advantage is smaller than token advantage (Atlas 4.5s vs baseline 5.6s median) because Atlas's per-call cost includes a sqlite query, not just a file read; the win is in number of round trips, not per-call latency. Chunkhound's 26.5s median reflects its semantic-search calls hitting embedding inference per query.
| agent | natural exit | max-iters | timeout | error |
|---|---|---|---|---|
| baseline | 550 | 10 | 0 | 0 |
| atlas | 560 | 0 | 0 | 0 |
| cbm | 521 | 38 | 0 | 1 |
| chunkhound | 529 | 2 | 28 | 0 |
Atlas hit zero max-iters and zero timeouts across all 560 trials. Chunkhound timed out on 28 trials (5%), all on the larger corpora (expo, unleash); its tool surface scales poorly past 5,000-file repositories. cbm hit max-iters on 38 trials (7%), driven primarily by repeated list_projects({}) calls that consumed tool-call budget without converging.
Structural questions have a clear ceiling for text-search: regex over file contents cannot reliably traverse aliased imports, factory closures, or polymorphic dispatch. Atlas wins graph-querying by +0.438 deterministic, the largest single capability gap in the benchmark, but loses graph-querying on the LLM-judge axis by −0.156 because the structural answer (a list of qualified names) is terser than the baseline's prose-style "I found these via grep" answer. Call-tracing is essentially tied (+0.038 det) and a slight LLM-judge loss (−0.087); this is the factory-pattern weakness called out in §5.5 future work. Content questions (literal substring counts) had previously been a structural gap (text-search wins by definition); Atlas's atlas_content_search (rg-backed, scoped to the indexed file set) closes this and converts it to the second-largest Atlas win (+0.321 det / +0.382 LLM).
Questions where Atlas and baseline both score high (code-access at 0.83 / 0.66, discovery at 0.98 / 0.76) are ones where the answer is unambiguous given a single grep + read_file pair, but Atlas still adds measurable value (+0.171 det on code-access, +0.227 on discovery) because direct symbol lookup beats grep-then-validate.
cbm scores 0.14 deterministic vs baseline's 0.40 despite having a 14-tool MCP surface explicitly designed for this kind of work. Instrumented runs show 555 calls to list_projects({}) across 560 trials; the model is repeatedly checking "are there projects" before every substantive query, an artifact of cbm's tool description design. Without prompt-engineering on top, the LLM agent cannot learn this pattern. We explicitly do not prompt-engineer cbm or Chunkhound for this benchmark; the comparison is "what happens when an agent gets the published tool surface verbatim", not "what happens when an expert tunes the prompts."
Chunkhound scores 0.28. Its 4-tool surface skews toward semantic search and regex, with 47% of calls being search_regex, effectively text search with a different API. The tool surface is small enough that the LLM doesn't have routing options, and the embedding-based queries that should differentiate Chunkhound from baseline don't seem to land on the question types in this task set.
The most contentious choice in this benchmark is removing text-tool fallback from Atlas, cbm, and Chunkhound. This measures what each MCP server's native tools can do alone, not what a production agent (which has both) can do.
We made this choice because earlier mixed-mode runs showed the Atlas agent reaching for grep on 44% of calls; meaning some of Atlas's apparent +0.126 deterministic advantage was carried by grep, not Atlas. The pure-tool numbers are smaller (Atlas-pure +0.038 det in the original 12-task mixed-mode run) but cleaner. Atlas's true contribution is what survives when text fallback is removed. That survival is, on this benchmark, +0.223 deterministic and +0.127 LLM-judge across 8 corpora, both significant.
Internal, task authoring overfit. Ripgrep + Zod tasks (28 of 112) were hand-authored before any of the v3 Atlas improvements landed. Concern: Atlas's tools were tuned in response to per-task failures. Mitigation: we authored 84 tasks for the 6 new corpora using a templating script that mines top-inbound symbols from the indexed db without any reference to Atlas's tool weaknesses. The per-corpus deltas (§4.2) show Atlas wins are not concentrated on the hand-authored corpora; gradio (+0.190 LLM-judge) and hugo (+0.175) are auto-generated and beat ripgrep (+0.109).
Internal, auto-generated tasks favor a subset. The script picks the top-inbound class per corpus for discovery tasks; this favors structural tools. Counter-argument: text-search can find any class by class FooBar regex, so the pattern doesn't intrinsically favor Atlas. And Atlas wins the text-content capability (+0.321 det) which is the scriptable opposite of structural.
External, model dependency. All numbers are for openai/gpt-5.4-nano. Earlier 12-task runs on claude-haiku-4.5 showed a smaller Atlas advantage (+0.17pp vs +0.22pp here) because better models reduce baseline failure modes. Atlas's value scales inversely with model quality. Opus 4.7 and GPT-5.4-pro would likely close the gap further. We explicitly do not claim model independence.
External, corpus selection. 8 corpora is wider than typical code-intelligence benchmarks (most use 1 to 3) but still finite. All are OSS, all have english code/comments, none are ML/scientific computing-heavy. Results may not generalize to embedded systems, ML training code, or non-english codebases.
Construct, LLM-judge bias. Verbosity bias in LLM-judges is documented (~15% inflation per labelyourdata). We mitigate with a 4-level rubric prompt and dual-judge cross-validation. Atlas's outputs are systematically more concise than baseline's (1.5 vs 3.0 tool calls/task, less raw text to summarize), so verbosity bias would under-rate Atlas, not over-rate it. §4.3 confirms this empirically: on graph-querying Atlas wins det by +0.438 but loses LLM-judge by −0.156, a 0.59-point gap consistent with a verbosity penalty on terse symbol-list answers. The +0.127 aggregate LLM-judge gap is therefore a lower bound on Atlas's true structural advantage.
Construct, text-search strength. Baseline uses read_file + grep + glob. An experienced human with ast-grep, ripgrep advanced flags, and a custom indexer could likely beat both Atlas and our baseline. We use the floor, not the ceiling, because we want to measure what an LLM agent gets for free, not what a tuned human can do.
claude-opus-4.7 released 2026-04-16, claude-sonnet-4.6, gpt-5.4-pro, gpt-5.4 standard, gemini-3.1-pro) and one open-weight 70B+ model to characterize how Atlas's advantage scales. Our gpt-5.4-nano numbers are a sub-frontier baseline; the 12-task claude-haiku-4.5 run already showed the delta shrinking from +0.22pp to +0.17pp, and pro-tier models should compress it further.atlas_trace returns 0 paths from a factory function (e.g. Zod's string()) to methods on its returned class. Adding a returns edge kind would close this gap.Cost per full 4-agent 8-corpus 5-trial run on gpt-5.4-nano: $6.21 ($4.31 inference + $1.90 judge). Wall time at concurrency 20: 23 minutes.
Every run records per-trial scores, full tool traces, token usage, LLM-judge rationales, and abnormal-exit reasons, keyed to the commit and timestamp that produced them.