UPDATE (February 4, 2024): This is the discussion about this project on HN: here. Please specifically read @dang's comment regarding the core assumption of this project: here. On a personal note, the number of Stories removed yesterday (Saturday, February 3, 2024) was the lowest ever recorded by the service. This includes 2 duplicate Stories. As a side note, in the list always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
The purpose of this project is to try to understand the type and scale of the moderation of the Hacker News Front Page.
NOTE: I love Hacker News. I try to read it every day. In the case of OnnxStream (here for example), 95% of the comments were helpful and intelligent. I also understand that moderating a site with huge traffic and where users are basically anonymous must be a very difficult task.
Returning to the purpose of this project, from what I have been able to see, the "public" (i.e. observable from the outside) moderation of the Front Page consists of two main tools: modification of the title of a Story (voluntarily or involuntarily influencing its growth in terms of rank) or directly its removal.
Regarding the first type of moderation, an excellent site is already available that tracks changes to Story titles. Here instead I will focus on the second type.
For the reasons explained in the "Why?" section below, I have developed a small application that logs all the Stories that are removed from the Front Page, for personal use. I later discovered that there is no tool/website that provides this type of information and I decided to make it public here. It was a difficult decision but my rationale is: is it better to have more transparency or less transparency?
If you know of a tool/website similar to this, please let me know: I will archive this repo or set it to private.
A possible very positive outcome for this project could be to have a list similar to this, but available directly among the HN lists. Or even to notify a user when a Story is penalized on the Front Page, perhaps indicating the number of flags and/or the reason, for example.
Feel free to skip this part or click to expand
A friend of mine posted two Stories on Hacker News related to OnnxStream (31 days apart), the first related to SDXL Turbo support and the second related to TinyLlama and Mistral 7B support.
In the case of the first, the Story was among the first on the Front Page, until its title was changed from "Stable Diffusion Turbo on a Raspberry Pi Zero 2 generates an image in 29 minutes" to "OnnxStream: Stable Diffusion XL 1.0 Base on a Raspberry Pi Zero 2". This effectively "killed" the Story. One user pointed out that the new title didn't reflect the spirit of the Story (thanks @practice9).
In the case of the second, the Story was in third place on the Front Page, less than an hour after the submission. In this case it was simply removed from the Front Page.
Having discovered this, perplexed, I sent an email to the moderator. @dang, who was very kind and quick in his response, explained to me that the Story had been flagged by users even without being explicitly [flagged], and that he could therefore only hypothesize the causes of the flag. His hypothesis was that (some?) users might be fed up with news related to LLMs.
While I have no reason to doubt Daniel's good faith, it's hard to believe that HN users would be tired of LLM-related news.
So I decided to develop a small console application to determine the frequency of this phenomenon (actually I was also motivated by the prospect of writing some C# code, after more than 2 years of complete abstinence). I subsequently discovered that there were no tools/websites that monitored this specific phenomenon and I therefore decided to make it public here.
Using the official HN API, the service fetches 90 Top Stories every minute and makes a comparison with the first 30 Top Stories (i.e. the Front Page) fetched the previous minute. It logs all missing Stories here. The assumption is that a Story cannot go from the top 30 to a position greater than 90 in a single minute, without having been explicitly removed. If a Story reappears on the Front Page, it is removed from this log. All Stories present in the second-chance pool are excluded from the log. Title and URL are those from when the Story first appeared in the top 30. The number of points and comments and the rank are those from when the Story was removed from the Front Page. The ID points to the news.social-protocols.org page for that Story, which provides a graph of the Story's position on the Front Page over time.
NOTE: always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
- 41536041 #22 15 points 29 comments -> Tesla Cybertruck sales surge 61% in July, outsells direct rivals
- 41486107 #29 53 points 8 comments -> Build Simple Fuzzer (2020)
- 41537358 #22 40 points 40 comments -> Brazil Supreme Court unfreezes X, Starlink's bank accounts after taking $3.3M
- 41538326 #21 -> Losses widen at Stripe Europe to $1.1B after employee 'liquidity event'
- 41535694 #13 175 points 87 comments -> OpenAI o1 Results on ARC-AGI-Pub
- 41501912 #15 111 points 50 comments -> Jerry Neumann: Resignation Letter
- 41537994 #15 29 points 26 comments -> Who is responsible for the greatest change to our language? Teenage girls
- 41538572 #21 7 points 1 comments -> Self-Verification Theory
- 41538595 #12 8 points 3 comments -> Sunamp's thermal battery uses a chemical found in salt-and-vinegar potato chips
- 41538273 #23 38 points 79 comments -> Show HN: I removed politics from Twitter with AI
- 41539653 #28 4 points 0 comments -> Canadian military admits new sleeping bags are not suited to Canadian winters
- 41539553 #17 15 points 3 comments -> Thomas Piketty: 'The Draghi report is a step in the right direction'
- 41541481 #7 5 points 6 comments -> Sweden to pay refugees up to 341,00 USD to return
- 41541678 #30 3 points 2 comments -> Accha FM – World's First 100% AI Generated Audio Entertainment Super App
- 41543382 #24 24 points 40 comments -> A cartoon butt clenching a bar of soap has invaded my online ads
- 41545541 #24 6 points 0 comments -> Show HN: I made a game testing your prompting skills
- 41507521 #29 12 points 0 comments -> Designing Organisations That Work
- 41547015 #2 7 points 0 comments -> I moved my emails to Proton Mail
- 41546242 #19 65 points 36 comments -> European Consumer Organization goes after publishers for their in-game currency
- 41545335 #18 226 points 114 comments -> Show HN: I Made Bullshit Remover
- 41518567 #27 163 points 133 comments -> USPS' long-awaited new mail truck makes its debut
- 41512910 #30 19 points 14 comments -> After 155 years, the Campbell Soup company is changing its name
- 41547637 #23 3 points 1 comments -> Honorary Citizenship of the United States
- 41547200 #20 32 points 40 comments -> US House votes to bar new DJI drones
- 41548213 #29 3 points 0 comments -> Kevin Hart's Vegan Fast-Food Chain Suddenly Closed All Locations Overnight
- 41547566 #14 9 points 0 comments -> Designing mathematics standards in agreement with science [pdf]
- 41548677 #7 18 points 24 comments -> U.S. fleet welcomes newest USS New Jersey, the first gender-neutral submarine
- 41548016 #13 -> Our New Religion Isn't Enough
- 41548997 #1 74 points 3 comments -> Terence Tao: creative strategies, this aspect of LLM tools is still weak
- 41550406 #10 3 points 2 comments -> The Unfortunate Physics of Male Urination (2016)
- 41505528 #20 150 points 92 comments -> The Disunity of Consciousness in Everyday Experience
- 41553840 #5 9 points 4 comments -> Startups Shouldn't Start with Figma
- 41554285 #23 9 points 0 comments -> Thierry Breton Has Resigned
- 41552642 #22 96 points 39 comments -> Data center emissions probably 662% higher than big tech claims
- 41546769 #14 239 points 80 comments -> Learn Git Branching
- 41529292 #14 95 points 46 comments -> Using the Topics API (Chrome Only, Opposed by Firefox/Safari)
- 41551387 #15 92 points 23 comments -> What Firefox trains are we in?
- 41531033 #19 67 points 7 comments -> The A-Z of Product Psychology
- 41553031 #23 63 points 61 comments -> Diesel shop fined $10M for making and selling Emissions Defeat Devices
- 41548985 #16 119 points 16 comments -> Data Engineering Vault: A 1000 Node Second Brain for DE Knowledge
- 41524932 #25 281 points 325 comments -> USPS' long-awaited new mail truck makes its debut to rave reviews from carriers
- 41554692 #22 6 points 0 comments -> date-fns 4.0 is out with first-class time zones support
- 41520950 #26 95 points 19 comments -> The art of niwaki (Japanese tree pruning)
- 41550426 #29 13 points 0 comments -> CSS Performance Optimization
- 41512421 #17 3 points 1 comments -> Where the Crops Grow
- 41551776 #28 38 points 62 comments -> New tech can make air-conditioning less harmful to the planet
- 41557178 #6 3 points 0 comments -> Matrix Multiplication in Finite Fields
- 41556910 #30 5 points 0 comments -> Ubisoft is using bots to promote new Assassin's Creed
- 41558704 #24 5 points 0 comments -> Windows 12.1 Pro
- 41558540 #7 4 points 0 comments -> When will AI outthink humans?
- 41559350 #6 45 points 1 comments -> Amazon Tells Workers to Return to Office Five Days a Week
- 41560027 #10 15 points 4 comments -> Amazon mandates five days a week in office starting next year
- 41560847 #28 3 points 0 comments -> New type of illusion is hard to make [video]
- 41561127 #12 7 points 0 comments -> Amazon mandates 5 days per week in office, starting next year
- 41560662 #30 6 points 1 comments -> The Things They Carried
- 41560491 #25 37 points 41 comments -> The Subprime AI Crisis
- 41559870 #30 12 points 2 comments -> CATL launches ultra-high-energy-density EV bus battery that lasts ~1M miles
- 41562543 #12 14 points 5 comments -> Firefox no longer works after upgrading to macOS Sequoia
- 41563076 #8 5 points 0 comments -> OpenAI O1 Blog
- 41562111 #25 9 points 0 comments -> Open Source Form Back End
- 41563205 #5 20 points 44 comments -> The reason why music is getting worse
- 41564611 #2 70 points 32 comments -> We have mathematically proven that transformers can solve any problem
- 41517455 #6 26 points 9 comments -> ChessDream, a place for chess players to improve
- 41562481 #19 151 points 62 comments -> Intel foundry business to make custom chip for Amazon, chipmaker's shares jump
- 41567615 #26 7 points 0 comments -> Java 23 General-Availability Release
- 41568052 #19 7 points 0 comments -> Hezbollah Operatives' Pagers Explode in Attack Across Lebanon
- 41567980 #29 4 points 0 comments -> Caution Earth to Capture a New Moon for 53 Days: Meet Asteroid 2024 PT5
- 41567939 #30 20 points 1 comments -> Hezbollah members reportedly injured by exploding pagers
- 41568731 #25 3 points 1 comments -> Mastodeck – Mastodon for Power Users
- 41568019 #2 83 points 57 comments -> Exploding pagers injure hundreds in attack targeting Hezbollah members
- 41570732 #15 13 points 1 comments -> Stuxnet
- 41567573 #15 677 points 3 comments -> Dozens of Hezbollah members said to be seriously injured as devices explode
- 41524517 #8 34 points 9 comments -> Ribbit: A portable, compact and extensible R4RS Scheme implementation
- 41552132 #12 5 points 1 comments -> The Illusion of a Pagan West
- 41577717 #4 8 points 11 comments -> Show HN: Free AI tailored workout generator
- 41578117 #3 24 points 22 comments -> Tax the Rich – European Citizens' Initiative
- 41529435 #3 86 points 19 comments -> Haxe lang – build cross-platform application with fast cross-compiler
- 41528598 #23 67 points 4 comments -> Scrolling Text Display
- 41579730 #19 5 points 0 comments -> U.S. Shrugs as World War III Approaches
- 41580375 #19 7 points 0 comments -> Israel detonates Hezbollah walkie-talkies in second wave after pager attack
- 41580214 #19 30 points 7 comments -> Walkie talkies explode across Lebanon a day after pager explosions
- 41583139 #27 7 points 1 comments -> How Wikipedia Became a Propaganda Site
- 41582958 #13 6 points 0 comments -> The San Francisco Freedom Club
- 41583294 #30 13 points 1 comments -> Walkie-talkie explosions rock Lebanon a day after pager attacks
- 41585624 #6 5 points 1 comments -> Seventh Day Adventists, fighting onanism, and selling veganism to America
- 41538009 #11 13 points 5 comments -> Are You Qualified to Use Null in SQL?
- 41587900 #11 10 points 26 comments -> Strange Design Decision: An Oven with French Doors
- 41590030 #7 11 points 4 comments -> Diversification is a negative price lunch
- 41580502 #11 255 points 147 comments -> 0day Contest for End-of-Life Devices Announced
- 41580166 #12 210 points 107 comments -> Bento: Jupyter Notebooks at Meta
- 41583689 #15 108 points 126 comments -> iOS 18 breaks IMAPS self-signed certs
- 41574844 #12 1570 points 799 comments -> Apple mobile processors are now made in America by TSMC
- 41584486 #16 383 points 209 comments -> LinkedIn is now using everyone's content to train their AI tool
- 41539841 #17 117 points 4 comments -> A overview of binaries, ELF, and NoMMU on Linux
- 41583062 #21 156 points 33 comments -> Qwen2.5: A Party of Foundation Models
- 41590688 #15 32 points 15 comments -> Tech Jobs Have Dried Up–-and Aren't Coming Back Soon
- 41582461 #23 160 points 161 comments -> Microplastics in the olfactory bulb of the human brain
- 41581942 #27 240 points 102 comments -> RabbitMQ 4.0
- 41586836 #9 64 points 19 comments -> Debugging Behind the Iron Curtain (2010)
- 41573034 #26 616 points 341 comments -> Independent directors of 23andMe resign from board
- 41591024 #11 5 points 0 comments -> A modest proposal for the regulation of comedy
- 41590544 #27 3 points 0 comments -> Lenovo Yoga with 'Lunar Lake' chip smokes M3 MacBook Air in battery life test
- 41577156 #30 301 points 120 comments -> Gentle Guide to Self-Hosting
- 41590319 #25 11 points 9 comments -> Apple Gets EU Warning to Open Up iPhone Operating System
- 41583028 #29 206 points 96 comments -> Meta AI: "The Future of AI Is Open Source and Decentralized"
- 41590469 #30 7 points 3 comments -> Nirvana fallacy (Perfect solution fallacy)
- 41582278 #24 209 points 45 comments -> Senate Vote Tomorrow Could Give Helping Hand to Patent Trolls
- 41579354 #29 38 points 2 comments -> Five hundred and seven Mechanical Movements
- 41575323 #29 403 points 161 comments -> Scramble: Open-Source Alternative to Grammarly
- 41587510 #30 53 points 39 comments -> YouTube confirms your pause screen is now fair game for ads
- 41590771 #21 7 points 0 comments -> Using RP2040 PIO to drive a poorly-designed display
- 41590460 #23 5 points 0 comments -> Satisficing
- 41590384 #16 14 points 6 comments -> Rolls-Royce wins pioneering deal to build mini nuclear plants in Czech Republic
- 41591237 #2 7 points 0 comments -> Nebula: Brave's differentially private system for privacy-preserving analytics
- 41591664 #18 3 points 1 comments -> Dumb Home
- 41580481 #14 5 points 0 comments -> Android 15 for Raspberry Pi 5 is here
- 41592008 #5 4 points 2 comments -> Iran linked hacker group Handala Hack Team claim pager explosions
- 41591765 #20 22 points 41 comments -> WSJ: Tech jobs have dried up
- 41593146 #30 3 points 0 comments -> 'It's not fun anymore': Fentanyl users rush for treatment as drug deaths drop
- 41591991 #26 6 points 0 comments -> The "R" in MTTR: Repair or Recover? What's the Difference?
- 41595042 #29 3 points 0 comments -> Homeless people often choose the street over a bed – find out why
- 41594523 #27 10 points 0 comments -> There is no antimimetics division V2
- 41540542 #12 170 points 104 comments -> MicroPython on Flipper Zero
- 41548551 #21 5 points 0 comments -> Sculpture commemorates 16th century drowning that inspired Shakespeare
- 41598274 #21 15 points 5 comments -> How Stripe Built One of Silicon Valley’s Best Engineering Teams
- 41599259 #19 8 points 3 comments -> Pov: Living in My Car in SF
- 41560225 #7 33 points 22 comments -> Show HN: Microtonal Piano
- 41598170 #19 176 points 92 comments -> Why Apple Uses JPEG XL in the iPhone 16 and What It Means for Your Photos
- 41600170 #23 16 points 6 comments -> Young women are starting to leave men behind
- 41600483 #28 3 points 0 comments -> 30th-anniversary limited-run PS5 and PS5 Pro bring back mid-'90s gray plastic
- 41600388 #14 124 points 202 comments -> Foundations: Why Britain Has Stagnated
- 41602181 #17 7 points 0 comments -> A year in exile: The fading hopes of Nagorno-Karabakh Armenians
- 41602024 #23 10 points 14 comments -> Dry Cask Storage
- 41603237 #18 3 points 0 comments -> Amazon says workers need to be in the office. Most of Silicon Valley disagrees
- 41602877 #12 20 points 6 comments -> Comedian John Mulaney brutally roasts SF techies at Dreamforce
- 41603448 #12 5 points 1 comments -> Researcher reveals 'catastrophic' security flaw in the Arc browser
- 41604241 #26 6 points 2 comments -> Updates to the Google Photos API
- 41604161 #30 20 points 12 comments -> Evidence points to Wuhan market as source of Covid-19 outbreak
- 41603920 #23 13 points 4 comments -> Re-opened Three Mile Island will power AI data centers under new deal
- 41605202 #24 6 points 1 comments -> Qualcomm Approached Intel About a Takeover in Recent Days
- 41550404 #30 14 points 5 comments -> Ultrasonic Chips Could Cut Interference in the IoT