How Karpathy Built HackerTube from HN Data

How Karpathy built HackerTube: public discussion records

A video archive begins with candidate links rather than a hand-built list of titles. Each Hacker News comment can contain a YouTube URL, a link shortener, or text that points to a video. An ingestion process has to normalize those forms and extract a stable video identifier. This makes two different URLs for the same clip resolve to one catalog record.

The project write-up describes processing roughly 40,000 comment records. That is a useful scale marker, but it should not be confused with 40,000 unique videos. Many comments can mention one video, while some comments may be contextual rather than recommendations. The archive groups those references so repeated discussion can be inspected without copying the full comment text into every downstream page.

Group and validate stable video IDs

The next step is deduplication. YouTube URLs can use watch parameters, short links, playlist parameters, or embed paths, so the visible URL is a poor database key. A stable eleven-character video ID provides a better join between title, channel, category, mention, and related-video records. Invalid or unavailable IDs should be retained only when the data model can explain their status.

Karpathy’s public export contains 6,454 video records across its snapshot. SignalTube uses 6,386 video URLs from the public sitemap for its current indexed set. That distinction matters when reproducing counts: use the same inclusion rule before comparing results. A reliable catalog records the source snapshot and the filter applied, so a later refresh can be compared consistently rather than silently replacing history.

Label comment roles without flattening context

After grouping mentions, the dataset labels how a comment uses a video. The available roles include strong recommendation, recommendation, reference, critical mention, and joke. Those categories answer a narrow question about the comment’s role in its thread. They do not establish that the video is accurate, useful to every viewer, or endorsed by the entire community.

A downstream interface should keep the label beside the date and source-thread link. If it reduces every occurrence to “recommended,” it can turn an example or criticism into an endorsement. SignalTube displays a concise interpretation and returns readers to the original Hacker News comment for nuance. This gives pages unique explanatory value without republishing a commenter’s full wording.

Organize records with a topic taxonomy

Topic labels create the browse layer. The HackerTube export describes 150 topics grouped into 16 areas, ranging from software and hardware to science, history, and design. A video can belong to more than one category, so category totals may overlap and should not be summed as unique video counts. Topic pages are entry points, not mutually exclusive partitions.

For a useful taxonomy, each category needs a stable identifier, human-readable name, and association to its broader area. A video page can link to all of its assigned topics, while an index can expose category counts. The taxonomy should be treated as a navigation aid: automated labels can be wrong, ambiguous talks can fit several subjects, and users should be able to move between related topics.

Publish a useful interface around the export

A searchable interface adds practical value through canonical video pages, filters, pagination, and fast previews. It should separate Hacker News dates from YouTube publication dates and show when runtime or availability is unknown. If verified metadata is added later, record its source and update time. Do not fill a blank release date using the year of a Hacker News comment.

Karpathy’s reported build combines data organization with a browser that makes recommendations accessible. SignalTube adds its own topic routes, local playlists, watch history, and original aggregate summaries. A separate interface remains a derived product, so attribution and license terms still matter. Keep the source credit visible and avoid implying that two projects share ownership or maintenance.

What the two-hour build does and does not prove

The public project notes describe a roughly two-hour build and name Opus 5.5 and Sonnet in the workflow. That is a report about one implementation, not a universal estimate for rebuilding the whole data pipeline. The archive benefited from an existing public export and source context; a fresh crawl, data audit, taxonomy review, and ongoing metadata refresh are separate work.

The durable lesson is to split the task into visible stages: acquire a lawful source, normalize IDs, preserve provenance, label carefully, publish useful navigation, then test edge cases. Pages with unique explanations and links back to sources create more value than a thin copy of the source catalog. Start with the project explainer, then review the dataset guide and topic index.

Frequently asked questions