HackerTube Dataset: Fields, Counts, and Use

HackerTube dataset video fields

A video record centers on a stable video ID and may include a title, channel, thumbnail URL, content type, level, tags, and topic categories. The ID is the field to use for deduplication and joins. Titles and channel names are descriptive metadata; they can change, contain inconsistent spelling, or be missing. Preserve the original values when reproducibility matters, and normalize into separate derived fields instead of overwriting source data.

Category membership can be many-to-many. A video may appear under functional programming and programming-language design, for example, so counting rows after an expanded join can inflate totals. Keep a unique video-ID count when reporting coverage. The taxonomy has 150 topics in 16 areas, but each topic count is a local view, not a mutually exclusive share of the entire collection.

Mention records and source context

Mention rows connect a video to a Hacker News comment ID, source URL, month, story title, and stance. The public export contains about 40,000 mention rows overall; the SignalTube-filtered sitemap set has 39,397 mention records. This is not the same as the number of unique users or unique comments. The distinct recommender metric is a separate aggregate supplied by the dataset.

Keep source URLs with every note. A short summary can explain the recorded context, but it should not replace the original comment or imply that its role is obvious from the title alone. A date such as 2018-10 describes when the link appeared in Hacker News. It does not establish that the video was published in 2018.

Recommendation signals are not view metrics

The dataset includes counts such as mentions, distinct recommenders, recommendation months, strong recommendations, and stance totals. These fields describe how the video appeared in Hacker News discussions. They can support ranking and discovery, but they do not estimate YouTube views, audience retention, or video quality. A video mentioned by many commenters may still be difficult to watch or mismatched to a reader’s needs.

Use precise labels in charts and page copy. “Recommended by 18 HN commenters” is clear; “18 views” is incorrect unless it came from YouTube. If YouTube statistics are added, record retrieval time and the API source, because view counts change. If a field is absent, expose it as unavailable rather than filling it with an inferred number.

Topic taxonomy and overlapping totals

The topic taxonomy supports browsing and analysis by subject. Each category has an ID and a display name and belongs to a broad area. Categories can overlap, so the total of category counts is expected to exceed the unique video count. When creating topic charts, state whether the chart counts unique IDs inside each topic and explain that one video can appear in more than one bar.

Taxonomy assignments are useful but imperfect. A talk about compilers may also fit programming language design, and an interview may cover multiple disciplines. Treat labels as recorded classifications rather than ground truth. For a custom analysis, keep both the original category IDs and your own revised labels so another reader can reproduce the source view or compare alternative taxonomies.

A safe workflow for dataset reuse

Start by loading the video, mention, taxonomy, and related-video records separately. Validate that IDs join as expected, measure duplicates, and compare unique video counts before and after applying a sitemap or availability filter. Keep the snapshot date in your output. If a later export changes counts, explain the new inclusion rule instead of presenting the difference as unexplained growth.

For a public derivative, preserve required attribution to HackerTube and open-index/hacker-news under ODC-By 1.0. Keep comment text on its original platform and link to the source rather than republishing large excerpts. Titles, thumbnails, and platform metadata have their own rights and terms. This guide is informational; check the license and applicable platform rules before redistributing a new dataset.

Fields that need a separate source

The public HackerTube export does not supply verified YouTube upload dates, duration, view counts, or current availability for every record. Those fields need a separate metadata source and must remain null until verified. For structured data, a page should omit an unknown upload date instead of borrowing an HN mention year. A missing value is more honest and easier to audit than an attractive guess.

A YouTube Data API refresh can batch video IDs and store a metadata snapshot with a fetch date. Requests should be paced and errors handled without exposing the API key. Private, deleted, or otherwise inaccessible records should be marked according to the returned result and removed from public video sitemaps. Continue with the project background or inspect a video record in context.

Frequently asked questions