What if Wikipedia could show its train of thought?

In 2019, I was given a fairly typical university exercise: use breadth-first search to move between words in a thesaurus. What stayed with me was not the assignment itself, but how funny the result felt. Follow enough related words and almost any word could be made to lead to almost any other. In that loose, chain-of-association sense, anything could mean anything.

That raised the question that became WikiCrawl. If words could be connected that way through a thesaurus, could anything on Wikipedia be connected to anything else? Every article could be treated as a node and every hyperlink as an edge. Choose two subjects, crawl outward, and see whether Wikipedia contained a route between them.

The first crawl worked. It also took far too long.

A live Wikipedia crawl expands brutally. One article produces hundreds of links, which produce thousands more. The earliest version spent most of its life waiting on the internet and repeatedly downloading work it had already done. Caching became necessary almost immediately: first whole pages, then the outgoing link lists extracted from them, and eventually completed paths and the evidence surrounding each hop.

The crawler also began searching from both ends. Bidirectional breadth-first search let one frontier grow from each article until they met, reducing how much of Wikipedia had to be explored. But finding the path immediately exposed a deeper problem: a chain of article titles could prove that a route existed without telling its story.

So the crawler started collecting the words around every link.

For each hop, WikiCrawl fetched the parent article, found the exact anchor leading to the next article, and tried to reconstruct the piece of the page that gave that link meaning. Sometimes the useful context was a sentence. Sometimes it was a paragraph, a table, or an infobox. The extraction code had to climb through inconsistent HTML, recover from missing elements and redirects, and turn whatever it found into one stable evidence box.

Even that did not fully solve the explanation problem. The excerpts contained the evidence, but a person still had to read several awkward fragments and assemble the larger connection in their head. The links found the route. The excerpts supplied clues. What was missing was a bridge that could turn those pieces into a coherent account without separating the account from its evidence.

The frontend rendered that preserved Wikipedia markup through React’s ominously named dangerouslySetInnerHTML. The name was apt: the approach was brittle and demanded care, but flattening everything into plain text destroyed the very structure that made the excerpt understandable. The result highlighted the linking sentence while keeping the surrounding material and its working Wikipedia links intact.

Then came my table-of-contents problem. Knowing the paragraph still was not enough when the meaning depended on where it appeared. WikiCrawl read Wikipedia’s headings, excluded sections such as References and External links from traversal, and walked backward through heading levels from each discovered link. That produced a breadcrumb—article section, subsection, and so on—so an excerpt arrived with its subject intact.

The links could find the connection. They could not tell the story.

THE SECOND HALF OF THE ORIGINAL WIKICRAWL IDEA

Between 2020 and 2021, “connected” split into three different questions.

Wikipedia links have direction. An article may link to another without the second article linking back. Once backlinks entered the crawler, a relationship could be explored in three distinct ways:

The forbidden-path search grew alongside those modes. After WikiCrawl found a meeting article, it could mark that meeting point as forbidden and run the search again. Repeating that process forced the crawler away from the first obvious hub and exposed alternate explanations. “Find a connection” became “show me different ways these things can be connected.”

Backlinks also changed the intellectual model of the project. An article does not necessarily contain its whole story. Other articles may supply the missing context by referring to it from a different subject, period, discipline, or point of view. Wikipedia was not just a collection of documents; it was a directed network of assertions about relevance.

The evidence pipeline was waiting for a narrator.

The 2019 system was already moving toward the ingredients an AI narrator would need: the actual path, the linking sentence, its larger paragraph or structured box, and the section hierarchy around it. AI was not meant to replace the search or declare that a connection existed. It was the missing interpretive layer—the bridge that could piece together what otherwise remained scattered across titles and difficult excerpts.

When narration was finally added, the model did not have to guess why a chain made sense from article names alone. It could receive the same evidence the interface showed to the user, explain the transitions in ordinary language, and leave the underlying trail available for inspection.

That final step still needed iteration. Early narratives were confused by meaningless fragments and incorrectly reconstructed paths. Later versions passed the full paragraph and section breadcrumb, rebuilt the chain in its true visited order, centered oversized excerpts on the relevant link, and attached visible source receipts. The current interface puts the explanation first, but the graph trail remains the proof underneath it.

The production work also made the original system faster and less fragile: concurrent frontier expansion, constant-time intersection checks, database indexes, faster parsing, Wikipedia’s links API, pooled connections, retries and polite rate limits. One recorded long uncached query fell from 13.5 seconds to 1.3 seconds. A separate failure investigation established an important limit: capping or truncating a BFS frontier makes the answer incomplete, so WikiCrawl must preserve the search invariant or honestly report that it found nothing.

The future is not “AI knows everything.” It is AI you can inspect.

The next research direction is a RAG version of Wikipedia: let someone ask a natural-language question, retrieve relevant articles and passages, use the link graph to discover adjacent evidence, and generate an answer whose claims sit beside visible citations. The goal is not to promise that hallucinations disappear. It is to make unsupported claims easier to detect, challenge and correct.

Wikipedia is not ground truth, and a citation is not automatically a good citation. Its practical advantage is inspectability. Articles are public, linked, referenced, revised, debated and watched by large communities. A reader can open the article, inspect its sources and history, and disagree with the evidence. Future work could go further by checking whether Wikipedia’s cited source actually supports the claim and by distinguishing stronger references from weaker ones.

That matters because ordinary web search often hides a source-quality decision inside ranking. A polished answer may ultimately rest on an obscure commercial blog, an interested party, or someone the reader would never consider credible if they saw the source first. Remove provenance and the reader inherits that source’s incentives without knowing it. WikiCrawl’s wager is that the route to an answer should remain visible enough for a person to judge where it came from.

A citation does not make an answer true. It makes the answer inspectable.

THE PRINCIPLE BEHIND THE NEXT WIKICRAWL

Why keep building it?

The honest answer is that I hope it finds something mind-blowing one day. Maybe two subjects produce a trail that exposes an overlooked history, a buried influence, an unexpected conflict of interest, or a pattern nobody thought to search for. Maybe following the surrounding links reveals a story that was always public but too scattered for anyone to notice as a whole.

A strange path is not proof of a conspiracy. Wikipedia links encode all kinds of ordinary relationships, and coincidence can look meaningful when enough data is connected. But a surprising trail can be a lead: something worth reading, verifying and investigating further. WikiCrawl can show the route and the evidence that produced it instead of asking anyone to trust the surprise on instinct.

That was the original ambition beneath the university exercise: dig through the connections until the graph helps us notice something we could not have noticed alone. The best possible result is not merely a short path. It is a moment when the path changes how we understand the subjects at either end.