70 results for Crawler

news.ycombinator.com
Ask HN: Are Paywalls allowed to show Google crawler full content but not users?https://news.ycombinator.com/item?id=31454039Today I noticed the Washington Post I'd quite naughty.Even when you click through from a Google search, they still gate the article content behind a paywall. Clearly they fed the full article to Google bot and Google news, but then show a different page, sans article content, to web users.Is this a violation of Google search policy? If yes, does Google even care enough to enforce it?I think this sort of publisher behavior makes for a hostile and overall very poor user experience... Especially when Wapo is already bankrolled by greedy Uncle Jeff. But I digress.
May 21, 2022 12:01 AM
bing.com
JavaScript links can make your pages invisible to AI searchhttp://www.bing.com/news/apiclick.aspx?ref=FexRss&aid=&tid=6a88cdc94aea40bda18a4c6777d3c109&url=https%3A%2F%2Fwww.msn.com%2Fen-us%2Ftechnology%2Fgeneral%2Fjavascript-links-can-make-your-pages-invisible-to-ai-search%2Far-AA2ati0O&c=14639234073689797483&mkt=en-usA 41-day experiment reveals how JavaScript-only navigation limits AI crawler discovery and why fixing it later can be harder.
Aug 19, 2026 8:00 AM
slicksearchhq.com
Show HN: Slick – I built a search engine where you control the rankinghttps://slicksearchhq.com/I've been frustrated with search for a while.There are a lot of alternatives to Google now. Brave and Mojeek have their own indexes, Marginalia focuses on the independent web, SearXNG lets you combine results from different search engines, and Kagi has probably gone furthest with user customization.But even Kagi and SearXNG have limitations. Kagi gives you control over individual websites, while SearXNG gives you control over which engines you search. I wanted to go a step further: what if you could actually control how your results are ranked?So I built Slick.Slick has its own independent index, and lets you customize ranking instead of treating it as a black box. You can influence how different sites and results are ranked based on what you actually want to see.I'm still building the index and improving the ranking system, so it's nowhere near Google in scale. I've been building the crawler, index, ranking system, and search interface myself, and I'd like to see whether this approach
Aug 16, 2026 11:02 PM
news.ycombinator.com
Help me build a better bot blockerhttps://news.ycombinator.com/item?id=49265322I got annoyed enough with bot blockers making me wait several seconds to access a site or go through a CAPTCHA, that I decided to make something better.I'm doing bot filtering/blocking using only Apache config options. I started with a basic filter that blocks HTTP clients that don't support HTTP/2.0, and clients that don't support brotli. To ensure clients don't just spoof brotli support, the root index.html is only available in brotli. I'm planning to add a hidden link to a brotli bomb, since it compresses much better than gzip. A 1GB ASCII repeating sequence compresses to under 1kB. My site has low traffic, under 10k hits per day. With the h2 & br filters active for 2 days, no malicious bot crawlers are getting through.I'd like to increase bot traffic to around 100k hits per day to help test the filters. Any suggestions on how? Perhaps just posting the site here will attract bots: https://solarsi.ca/My intention is a false positive rate of under 0.1%. Please reply with your browser
Aug 11, 2026 10:24 PM
github.com
Show HN: OptiQra – Open-source AI website optimization and intelligence platformhttps://github.com/armin5872/OptiQraHi HN,I've been building OptiQra for quite a while, and I finally finished the desktop version.OptiQra is an open-source website intelligence and optimization platform. It can crawl and analyze websites or local projects, identify SEO/AEO/GEO, accessibility, performance and technical issues, explain them with AI, and automatically fix them.The desktop version is where the project gets particularly interesting. It can run locally, work with local projects, operate offline for analysis that doesn't require cloud services, and run scheduled scans in the background.The workflow is roughly:URL/project → crawl → analyze → understand the issues → inspect them on the actual site → fix them → verify the result.It also has a fairly visual crawler, including live 2D and 3D crawl trees, and the AI interface is customizable rather than being limited to a single assistant personality.I built the UI and CSS myself rather than using Tailwind/shadcn, and the project has grown considerably beyond a simp
Aug 10, 2026 9:13 AM
bing.com
How search engines rank billions of web pages in secondshttp://www.bing.com/news/apiclick.aspx?ref=FexRss&aid=&tid=6a8d79bf37d64ec699423bb2fe2bd65a&url=https%3A%2F%2Fwww.msn.com%2Fen-in%2Fnews%2Fother%2Fhow-search-engines-rank-billions-of-web-pages-in-seconds%2Far-AA29KzEJ&c=11140371160363390790&mkt=en-usSearch engines utilize sophisticated algorithms to decode user inquiries and provide pertinent results. Their crawlers explore and catalog websites, creating an extensive digital archive for rapid ...
Aug 10, 2026 12:38 AM
hnhiring.azuanz.com
Show HN: HN Hiring – Search and Filter Who Is Hiringhttps://hnhiring.azuanz.comhttps://hnhiring.azuanz.comI originally built this to scratch my own itch while job hunting.The main problem for me with the monthly Who Is Hiring thread was finding relevant jobs by location. I was mainly looking around Europe, and most posts are understandably US focused.I’ve since found a new job, so I don’t really need it myself anymore, but I thought it might still be useful to others.The crawler is written in Go, but the final site is just a single HTML file plus JSON files, served through Cloudflare Pages. The crawler runs every 10 minutes on an Ubuntu VM on my existing Synology homelab.The extraction/classification part was also a bit of an experiment for me.I primarily use Sol at work, but I also had a GLM subscription that I picked up during their Christmas deal last year, so I wanted to see how well it handled both coding and this kind of structured classification task.Right now the extraction is done using sequential GLM 5.2 calls with reasoning disabled. I tried batching m
Aug 9, 2026 12:06 PM
news.ycombinator.com
Show HN: LLM Tools that I made that I cant live withouthttps://news.ycombinator.com/item?id=49146317Hello HN ..I am posting here because I have made this bad ass thing. I want to be clear. I'm trying to make somthing worthy of people buying it and getting good value, I have not really shown it off or had anyone look at my site past the occasional web crawler, and I did 100% make this. Its called 0verload and the things it does, I will now list. Automatcially tracks versions on all your source code. It dosnt replace GIT, its for the saves you do between checkins...that one version you should have saved .it does..automatically. even oob writes get a version. I cant live with out it... somthing about telling my llm ....get me that thing I wrote like 2 days ago or a week ago or whenever.you can just tell the llm kinda where when and it will go get it and merge it in....I have files with hundreds of versions and you would think it would be annoying but if you have an llm ...I ship claude skills with it so it wakes up knowing how to use it, AND that brings me to somthing else it does.. I i
Aug 2, 2026 5:13 PM
prieco.net
Show HN: PriEco (open source, 400M index, EU search engine) integrated Goggleshttps://prieco.net/Back in 2021 I wanted to switch away from Google and try alternatives. To my disappointment they were... not great. I took a way of comparing qualities and disadvantages of each web search engine back then. What I discovered I cared about the most was privacy (like DuckDuckGo), ecology (Ecosia) and Google index (likely to signal quality of results, Startpage). To my disappointment there wasn't a single one that I could find that offered all 3 of these qualities. After a long time of deciding if and how I could build my own web search engine I decided to do it.At first, for 4 years, it was just a private and ecological meta search engine. But I didn't have the money to grow user base while relying on 3rd party APIs to deliver results. I had to create own index. I spent 1.5 years on web crawler, made from scratch in Rust btw. As soon as it was done it was able to deliver me 15M results a day on 200Mb/s download bandwidth. Which I'd say is competitive, especially as my next developmental
Jul 29, 2026 1:05 PM
news.ycombinator.com
My website gets more attacks than human visitorshttps://news.ycombinator.com/item?id=48677487I run a small self-hosted website on a Raspberry Pi 4B at home. A few weeks ago I started wondering: who actually visits a website in 2026? Not just humans. Everything. So I built a public observability dashboard on top of GoAccess that separates traffic into four categories: human visitors, search engine crawlers, AI retrieval agents, and automated attacks. The numbers from the last 17 days surprised me:4,523 human visits 6,409 automated attack attempts Thousands of crawler requests from search engines and AI systemsThe attacks aren't sophisticated. They're mostly automated scanners probing for .env files, WordPress admin panels, and cloud credentials — hitting every public IP on the internet regardless of what's actually running there. What I found more interesting was the AI agent behavior. AI retrieval agents (GPTBot, ClaudeBot, PerplexityBot, Amazonbot) behave differently from traditional search crawlers. They hit semantic files aggressively — llms.txt, sitemap.xml, JSON-LD struct
Jun 25, 2026 6:35 PM
news.ycombinator.com
Ask HN: Am I being advertised an ARG via user agent logs?https://news.ycombinator.com/item?id=48582005I'm here looking through logs on my unnamed reverse proxy and CDN service. The crawler bot swarm has been hitting my PHP application like I've upset them so I'm seeing which weird user agent strings are being allowed to connect. There's "Sogou" and "meta-webindexer" and a small number of requests from "SleepBot/1.0"What's SleepBot?The ASN is Google and the UA string is: "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SleepBot/1.0; +http //sleepbot com/) Chrome/131.0.0.0 Safari/537.36" [edited to make link non-clickable]So I visit the site. And it looks like the homepage of an interesting tech and ambient music guy who is still running a Shoutcast online radio stream but otherwise hasn't been seen online in 5 years. The Wayback Machine shows few changes in over a decade. But the resume link brings up a GitHub account with a different URL and username which reported 1 issue in March of this year. It goes deeper.What's going on? Is a Google or adjacent employee running a p
Jun 18, 2026 7:27 AM
stackscope.dev
Show HN: StackScope – I crawled over 40k indie launches to see what they shiphttps://stackscope.dev/Hey all, I built StackScope, a crawler/catalogue that looks at new product launches and shows what they were built with.It watches launches from Product Hunt, Show HN, and PeerPush, then crawls the public site behind each one. The goal is to show what people actually launched with: hosting, frameworks, analytics, DNS, security headers, legal pages, AI-builder signals, and other public clues.I started building it because most stack-detection sites look at the web as a whole. I was more interested in the current indie launch scene: what people are choosing right now, at the point they first put something in public.A few implementation details: it runs on .NET, uses Playwright for rendered pages, and has a first-party fingerprint catalogue rather than one copied from Wappalyzer/etc. robots.txt is honoured, and the bot identifies itself.Frustratingly, I am still waiting for verified bot status from Cloudflare and currently that knocks out about 10% of all sites.There is also a private read
Jun 12, 2026 3:26 PM
news.ycombinator.com
AniTroves – An anime database with a custom LLM-based discovery hubhttps://news.ycombinator.com/item?id=48057592I’ve always felt that traditional anime databases rely too heavily on rigid tag-based searches. If you’re looking for a specific "vibe" or a very niche trope that isn't a primary tag, you usually end up scrolling through pages of irrelevant results.I built AniTroves (https://anitroves.com) to experiment with a more conversational, LLM-driven approach to series discovery.The Tech Behind the Hub:LLM Integration: Instead of a generic API wrapper, I've been working on a custom hub (https://anitroves.com/ai-hub/) that uses specialized models to understand series lore and character archetypes for roleplay and discovery.Anipick Engine: This is the logic layer that maps natural language queries to our database entries.Technical Transparency: I’ve implemented a structured llms.txt (https://anitroves.com/llms.txt) to provide a machine-readable source of truth for other crawlers and AI models.I’m currently the technical administrator and I'm handling the SEO and server scaling (managed on Hosting
May 8, 2026 2:00 AM
console.seltz.ai
Show HN: Seltz – The fastest, high quality, search API for AI agentshttps://console.seltz.ai/loginHi HN, Antonio here. Founder of Seltz.Seltz is a web search API built for AI agents. We wrote the crawler, the index, and the retrieval models ourselves, in Rust, by a team that's spent years building web search at scale. In our tests, queries come back in under 200ms.Efficiency was the first design principle. Search sits on the critical path: agents can't generate their first tokens or kick off the next tool call until results come back. When you run tens or hundreds of queries in parallel, every millisecond of tail latency compounds.Most search APIs for agents are wrappers around Google or Bing. If your agent already has a Google tool, a second call to a Google-wrapped API returns the same ten documents. We run our own independent index, so you get different results and different rankings.Coverage starts with US news. More verticals are coming.There's a free tier with $100 in credits at the link. I'd love feedback from anyone building agents: how does it compare to what you're using,
Apr 20, 2026 5:12 PM
news.ycombinator.com
Show HN: Canary – Monitor any URL for changes, get AI-powered intelligencehttps://news.ycombinator.com/item?id=47557235Canary watches any URL and alerts you when something changes — competitor pricing, API endpoints, docs, job boards, status pages. Changes are scored 1-10 by signal strength and summarised by AI. Built solo. Landing page live today. Automated crawler running on GitHub Actions posting real detected changes to Bluesky hourly and X daily. canary-landing-one.vercel.app Happy to answer questions about the architecture — particularly proud of the URL deduplication approach where one crawl serves multiple subscribers.
Mar 28, 2026 6:49 PM
agentgames.co
Show HN: AgentGames.co – my game creatorhttps://agentgames.coHey all,I recently built the first version of my project to play and create interactive story games with AI agents.Each game can have up to 20 agents, each agent can have an image, voice, access to other agents, and resources to enhance the experience. All game configuration has logic conditions you can set. For example, from agent A you can only access agent B if you've visited agent C or you have a specific resource item.You can create anything from a murder mystery to a dungeon crawler. The resources are dynamic and can set drop rates, make stackable for currencies or health, and give them their own images as well.To play: Type or speak in game. If you want to go somewhere, say where you want to go. To create: Describe what you want. You can manually configure as well.You can try it for free. I'd really love your feedback. There was a lot of trickiness along the way to build this, so I learned a lot. More improvements to comeHappy to answer any questions, thank you!
Feb 27, 2026 9:36 PM
levaevro.com
Show HN: BGN/EUR converter for Bulgaria's euro adoptionhttps://levaevro.com/Built this as a side project ahead of Bulgaria joining the eurozone (Jan 1, 2026).A few things I tried that seemed to work:- Launched ~2 months before the fixed rate announcement so Google had time to index - The domain "levaevro.com" is a transliteration of "лева в евро" (leva in euro) – exact-match domains still work surprisingly well in smaller language markets - Instead of canonicalizing all query param URLs to the homepage, I submitted 52 parameterized URLs to a custom sitemap (?amount=100&from=BGN, ?amount=500&from=BGN, etc.) – each one targets a specific search query like "100 лева в евро". Zero extra dev work, same template. - Added a llms-sitemap.xml for AI crawlers (ChatGPT, Perplexity) alongside the standard sitemapThe fixed exchange rate (1 EUR = 1.95583 BGN) is set by EU Council regulation and will never change, so the tool stays accurate forever with no maintenance. Happy to share more details about the SEO approach or the technical setup.
Feb 27, 2026 2:33 PM
spaceflightnow.com
SLS rocket hauled back to VAB for repairshttps://spaceflightnow.com/2026/02/25/sls-rocket-hauled-back-to-vab-for-repairs/A 6.6-million-pound Apollo-era crawler-transporter rolled up the ramp to pad 38B on Tuesday and was positioned directly under the 3.5-million-pound Space Launch System rocket and its 11.3-million-pound mobile launch platform.
Feb 25, 2026 7:01 PM
emailmarketingskill.com
Show HN: I built a 55K-word email marketing knowledge base and Claude Code skillhttps://www.emailmarketingskill.com/I co-founded SmartrMail (email marketing SaaS, 12K ecommerce customers, acquired 2022). When I left, I no longer had access to the sending data I'd spent years learning from, billions of sent emails, deliverability patterns, things that actually move email engagementCos' I don't have this data anymore, I built the best thing I could to help with the email marketing I currently do:---my research processMultiple sprints across all major email marketing topics. The crawler pulled 908 sources: Litmus, Klaviyo, HubSpot, Campaign Monitor, and Salesforce annual reports; practitioner blogs; academic research; platform documentation; Reddit threads; Shopify forums; and community discussions on X. From those, I extracted 4,798 discrete insights. Every claim that made it into the guide has a source. Anything that was unsourced opinion got cut.That produced EMB v4: Over 80k words across 16 chapters. After two editorial passes, cutting duplicates, consolidating overlapping sections, removing anythi
Feb 21, 2026 1:08 AM