Wednesday, April 16, 2025
Extreme automobiles, Ft. Lauderdale edition
Bunny and I stopped by Josef & Joseph to have them fix Bunny's watch (it turned out to just need a new battery). Inside is a large collection of clocks, watches and jewelry. Outside, however, was this!
![[A silver car, longer than a limousine with an assortment of ornamants adorning the outside] “That's not a limousine, this is a limousine!”](/2025/04/16/artcar1.jpg "“That's not a limousine, this is a limousine!”")
![[The frontend of a silver car with nine, count them, nine, headlights spanning the front grill] I don't think I'd want this barrelling towards me from behind at night!](/2025/04/16/artcar2.jpg "I don't think I'd want this barrelling towards me from behind at night!")
![[The other side of a three-times longer than normal silver car with three baloon-like things leaned up against it] This is so unusual of a car that three aliens have come down to examine it closely.](/2025/04/16/artcar3.jpg "This is so unusual of a car that three aliens have come down to examine it closely.")
![[Closeup detail of the car, sporting emblems from VW, BMW, and Cadilac, with a model air plane strapped to the top, all silver] I don't think those wings are capable of supporting flight. Probably for the best.](/2025/04/16/artcar4.jpg "I don't think those wings are capable of supporting flight. Probably for the best.")
When I asked about the car inside, I was informed that indeed, it could drive, so it's a working car, made of a VW, BMW and Cadilac welded together and painted silver. And it was delivered as part of an Elvis Presley exhibit at Josef & Joseph. What Elvis has to do with a Mad-Max inspired limousine is beyond me. But it's there, and it's awesome.
It's also been there so long you can see it on Google's street view!
Thursday, April 03, 2025
God, I feel like I'm an old man wearing a tin foil hat yelling at the world
A few months ago our ice maker broke and the upshot—we got charged for a replacement unit that was never installed and we're using ice trays.
Last week, our dryer stopped working. It turned on, you can set the controls, but it fails to spin up and do any actual drying. We thought about maybe getting it repaired, but in all likelyhood, by the time we pay for the repair man and the replacement widget that isn't working, we would probably be looking at the price of a new dryer anyway. And if it can't be fixed, then we're definitely out the prirce of a new dryer and the repair man.
So early this week we went out to one of the two remaining home improvement stores (and it wouldn't surprise me to find out they're both owned by the same shadow company—can you tell I'm gettin cynical here?) and bought a replacement. Of course, we couldn't just buy a new dryer—no. We had to buy a new electrical cord (what? It doesn't come as part of the dryer?) and a new dryer hose (what? What's wrong with the one we already have? This stuff is pretty much standardized by now). Then there was the delivery fee, the installation fee, and the removal-of-the-old-unit fee. All that tacked on an additional 25% on the shelf price, not including taxes.
Maybe it would have been cheaper to get it repaired.
Anyway, today we received the unit. Two men unloaded the new dryer from the truck, brought it inside the garage, took the old dryer off the “pedestal” (which we purchased several years ago when we got a new washer and dryer—each are on a pedestal that acts as storage space), and then said they would NOT install the new dryer on the old pedestal. In fact, they insisted they COULD NOT install on the older pedestal, and then they left.
Well, thank you very much.
We called the home improvement store and they said it was company policy not to install a dryer on a pre-existing pedetal. They didn't say which company mandated this policy—the manufacturer of the dryer (which, at this point, I wouldn't be surprised if there was only one shadow company that owned all the appliance manufacturers) or the home improvement store. And of course, the removal of the pedestal would require an additional removal-of-the-old-unit fee, because it was considered a separate unit. It didn't matter that we had a perfectly good pedestal whose size matches the new dryer. And it didn't matter that the new dryer hose matched the size of the existing hose. Oh, and if we installed the dryer ourselves, we would immediately void the one-year warrantee on the dryer.
It seems like our only real choices were to spend even more money on a new pededstal, more money for delivery, more money to remove the old pedestal, more money to install the new pedestal. and probably an additional fee to install the new dryer (which we had already paid for installation) to install the dryer on said pedestal, or we could elect to void the warrantee and install the dryer on our own.
Well, XXXX me!
Bunny thinks I'm too cynical, but at this point, how do you not be cynical?
We ended up calling a friend over to help install the dryer on the old pedestal. It fit fine. So did the old hose. I'm not sure if it's even worth attempting to get the installation fee back—they might just void our warrantee on the spot (I can hear Bunny calling out “Stop being so cynical!”). I'm glad we didn't spring for the three or five year warrantees—the unit will probably last that long (and no longer, because then the same shadow company that owns all the appliance manufacturers gets another sale, and thus the line goes up, because XXXX you, think of the poor billionaires).
I hope it lasts longer. I don't want to be this cynical.
Wednesday, March 26, 2025
Notes on blocking spam by filtering on ASN
So now that I can classify IP addresses by ASN, I thought I might see how it could help with spam email. I'm already using an ansi-spam agent to cut down on spam, so maybe filtering by ASN could cut down even more. The last time I looked into additional means of spam avoidance, the use of SPF wasn't worth the effort.
And I'm afraid the effort of blocking via ASN won't be worth the effort either. Looking over email attempts over the past month, the top 10 networks who sent email to my server, from 5,181 individual emails:
| AS | Count |
|---|---|
| IOMART-AS, | 375 |
| IDNIC-IDCLOUDHOST-AS-ID | 369 |
| PAIR-NETWORKS, | 263 |
| MICROSOFT-CORP-MSN-AS-BLOCK, | 246 |
| AS-COLOCROSSING, | 152 |
| EMERALD-ONION, | 124 |
| GOOGLE, | 122 |
| SPARKPOST, | 120 |
| TZULO, | 112 |
| AMAZON-02, | 106 |
Unlike the web (or even Gemini or gopher) there isn't one dominant network here—it's all spread out. I don't think it's really worth the effort to block via ASN for spam. At least for my email server.
Discussions about this entry
Friday, March 21, 2025
Now a bit about feed readers
There are a few bots acting less than optimally that aren't some LLM-based company scraping my site. I think. Anyway, the first one I mentioned:
| Agent | Requests |
|---|---|
| Feedly/1.0 (+https://feedly.com/poller.html; 16 subscribers; ) | 1667 |
| Feedly/1.0 (+https://feedly.com/poller.html; 6 subscribers; ) | 1419 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 6 subscribers; ) | 938 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 16 subscribers; ) | 811 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 8 subscribers; ) | 94 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 37 subscribers; ) | 17 |
| Agent | Requests |
|---|---|
| Feedly/1.0 (+https://feedly.com/poller.html; 16 subscribers; ) | 1579 |
| Feedly/1.0 (+https://feedly.com/poller.html; 6 subscribers; ) | 1481 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 6 subscribers; ) | 905 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 16 subscribers; ) | 741 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 8 subscribers; ) | 90 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 37 subscribers; ) | 11 |
This is feedly, a company that offers a news reader (and I'd like to thank the 67 subscribers I have—thank you). The first issue I have about this client is the apparent redundant requests from six different clients. An issue because I only have three different feeds, the Atom feed, the RSS feed and the the JSON feed. The poller seems to be acting correctly—16 subscribers to my Atom feed and 6 to the RSS feed. The other four? The fetchers? I'm not sure what's going on there. There's one for the RSS feed, and three for the Atom feed. And one of them is a typo—it's requesting “//index.atom” instead of the proper “/index.atom” (but apparently Apache allows it). How do I have 16 subscribers to “/index.atom” and another 37 for “/index.atom”? What exactly, is the difference between the two? And can't you fix the “//index.atom” reference? To me, that's an obvious typo, one that could be verified by retreiving both “/index.atom” and “//index.atom” and seeing they're the same.
Anyway, the second issue I have with feedly is their apparent lack of caching on their end. They do not do a conditional request and while they aren't exactly slamming my server, they are making multiple requests per hour, and for a resource that doesn't change all that often (excluding today that is).
Then there's the bot at IP address 4.231.104.62. It made 43,236 requests to get “/index.atom”, 5 invalid requests in the form of “/gopher://gopher.conman.org/0Phlog:2025/02/…” and one other valid request for this page. It's not the 5 invalid requests or the 1 valid request that has me weirded out—it's the 43,236 to my Atom feed. That's one request every 55 seconds! And even worse—it's not a conditional request! Of all the bots, this is the one I feel most like blocking at the firewall level—just have it drop the packets entirely.
At least it supports compressed results.
Sheesh.
As for the rest—of the 109 bots that fetched the Atom feed at least once per day (I put the cut off at 28 requests or more durring February), only 31 did so conditionally. That's a horrible rate. And of the 31 that did so conditionally, most don't support compression. So on the one hand, the majority of bots that fetch the Atom feed do so compressed. On the other hand, it appears that the bots that do fetch conditionally most don't support compression.
Sigh.
Still no information on who “The Knowledge AI” is or was
Back in July 2019 I was investigating some bad bots on my website when I came across the bot that identified itself simply as “The Knowledge AI” that was the number one robot hitting my site. Most bots that identify themselves will give a URL to a page that describes their usage like Barkrowler (to pick one that recently crawled my site). But not so “The Knowledge AI”. That was all it said, “The Knowledge AI”. It was very hard to Google, but I wouldn’t be surprised if it was OpenAI.
The earliest I can find “The Knowledge AI” crawling my site was April of 2018, and despite starting on April 16th, it was the second most active robot that month. In May it was the number one bot, and it stayed there through October of 2022, after which it pretty much dropped—from 32,000+ in October of 2022 to 85 in November of 2022 (about 4½ years). It was sporadic, showing up in single digit hits until January of 2024. It may be still crawling my site, but if it is, it is no longer identifying itself.
I don’t know if “The Knowledge AI” was an LLM company crawling, but if it was, not giving a link to explain the bot is suspicious. It’s the rare crawler that doesn’t identify itself with at least a URL to describe it. The fact that it took the number one crawling spot on my site for 4 ½ years is suspicious. As robots go, it didn’t affect the web server all that much (I’ve come across worse ones), and well over 90% of its requests were valid (unlike MJ12, which had a 75% failure rate). And my /robots.txt file doesn’t exclude any robot from scanning, so I can’t really complain about it.
My comment on “Mitigating SourceHut's partial outage caused by aggressive crawlers | Lobsters”
Even though the log data is a few years old, I don't think that IPs change from ASN to ASN all that much (but I could be wrong on that). I checked the IPs used by “The Knowledge AI” in May 2018, and in October 2022, and they didn't change that much. They were still the same /24 networks across that time.
Looking up the information today is very disappointing—Hurricane Electric LLC., a backbone provider.
So no real information about who “The Knowledge AI” might have been.
Sigh.
A deeper dive into mapping web requests via ASN, not by IP address
I went ahead and replaced IP addresses with ASNs in the log file to find the network that sent the most requests to my blog for the month of February.
| MICROSOFT-CORP-MSN-AS-BLOCK, US | 78889 |
| OVH, FR | 31837 |
| ALIBABA-CN-NET Alibaba US Technology Co., Ltd., CN | 25019 |
| HETZNER-AS, DE | 23840 |
| GOOGLE-CLOUD-PLATFORM, US | 21431 |
| CSTL, US | 17225 |
| HURRICANE, US | 15495 |
| AMAZON-AES, US | 14430 |
| FACEBOOK, US | 13736 |
| AKAMAI-LINODE-AP Akamai Connected Cloud, SG | 12673 |
Even though Alibaba US has the most unique IPs hitting my blog, Microsoft is still the network making the most requests. So let's see how Microsoft presents itself to my web server. Here are the user agents it sends:
| agent | requests |
|---|---|
| Go-http-client/2.0 | 43236 |
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | 23978 |
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 | 7953 |
| Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0 | 2955 |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot | 210 |
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot | 161 |
| DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) | 123 |
| 'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)' | 122 |
| Python/3.9 aiohttp/3.10.6 | 28 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.36 Safari/537.36 | 14 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.114 Safari/537.36 | 14 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/112.0.1722.68 | 10 |
| DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) | 10 |
| DuckAssistBot/1.1; (+http://duckduckgo.com/duckassistbot.html) | 10 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 | 6 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.143 Safari/537.36 | 6 |
| python-requests/2.32.3 | 5 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.142 Safari/537.36 | 5 |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 | 4 |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0 | 4 |
| DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot) | 4 |
| Twingly Recon | 3 |
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot) | 3 |
| Mozilla/5.0 (compatible; Twingly Recon; twingly.com) | 3 |
| python-requests/2.28.2 | 2 |
| newspaper/0.9.1 | 2 |
| Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36 | 2 |
| Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b | 2 |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 | 2 |
| http.rb/5.1.1 (Mastodon/4.2.10; +https://trystero.social/) Bot | 1 |
| http.rb/5.1.1 (Mastodon/4.2.10; +https://trystero.social/) | 1 |
| Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5 skype-url-preview@microsoft.com | 1 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 | 1 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36 | 1 |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/112.0.1722.48 | 1 |
| Mastodon/4.4.0-alpha.2 (http.rb/5.2.0; +https://sns.mszpro.com/) Bot | 1 |
| Mastodon/4.4.0-alpha.2 (http.rb/5.2.0; +https://sns.mszpro.com/) | 1 |
| Mastodon/4.3.3 (http.rb/5.2.0; +https://the.voiceover.bar/) Bot | 1 |
| Mastodon/4.3.3 (http.rb/5.2.0; +https://the.voiceover.bar/) | 1 |
| Mastodon/4.3.3 (http.rb/5.2.0; +https://discuss.systems/) Bot | 1 |
| Mastodon/4.3.3 (http.rb/5.2.0; +https://discuss.systems/) | 1 |
The top result comes from a single IP address and probably requires a separate post about it, since it's weird and annoying. But the rest—you got Bing, you got OpenAI, you got several Mastodon instances—it seems like most of these are from Microsoft's cloud offering. A mixture of things.
What about Facebook?
| agent | requests |
|---|---|
| meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) | 13497 |
| facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) | 207 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 | 12 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 | 4 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 | 4 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 | 4 |
| Mozilla/5.0 (Windows NT 10.0; WOW64; rv:58.0) Gecko/20100101 Firefox/59.0 | 4 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0 | 2 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 | 2 |
Hmm … looks like I have a few readers at Facebook, but other than that, nothing terribly interesting.
Alibaba,
on the other hand,
is frightening.
Out of 25,019 requests,
it presented 581 different user agents.
From looking at what was requested,
I don't think it's 500 Chinese people reading my blog—it's defintely bots crawling my site
(and amusingly, there are requests to /robots.txt file,
but without a proper user agent to go by,
it's hard to block it via that file).
I can think of one conclusion here—if you do filter by ASN, it can help tremendously, but it also comes with possibly blocking legitimate traffic.
Discussions about this entry
A different approach to blocking bad webbots by IP address
Web crawlers for LLM-based companies, as well as some specific solutions to blocking them, have been making the rounds in the past few days. I was curious to see just how many were hitting my web site, so I ran a few queries over the log files. To ensure consistent results, I decided to query the log file for last month:
| total requests | 468439 |
| unique IPs | 24654 |
| IP | Requests |
|---|---|
| 4.231.104.62 | 43242 |
| 198.100.155.33 | 26650 |
| 66.55.200.246 | 9057 |
| 74.80.208.170 | 8631 |
| 74.80.208.59 | 8407 |
| 216.244.66.239 | 5998 |
| 4.227.36.126 | 5832 |
| 20.171.207.130 | 5817 |
| 8.29.198.26 | 4946 |
| 8.29.198.25 | 4807 |
(Note: I'm not concerned about protecting any privacy here—given the number of results, there is no way these are any individual. These are all companies hitting my site, and if companies are mining their data for my information, I'm going to do the same to them. So there.)
But it became apparent that it's hard to determine which requests are coming from a single entity—it's clear that a company can employ a large pool of IP addresses to crawl the web, and it's hard to figure out what IPs are under control of which company.
Or is it?
An idea suddenly hit me—a stray thought from the days when I was wearing a network admin hat I recalled that BGP routing basically knows the network boundaries for networks as it's based on policy routing via ASNs. I wonder if I could map IP addresses to ASNs? A quick search and I found my answer—yes! Within a few minutes, I had converted a list of 24,654 unique IP addresses to 1,490 unique networks, I was then able to rework my initial query to include the ASN (or rather, the human readable version instead of just the number):
| IP | Requests | AS |
|---|---|---|
| 4.231.104.62 | 43242 | MICROSOFT-CORP-MSN-AS-BLOCK, US |
| 198.100.155.33 | 26650 | OVH, FR |
| 66.55.200.246 | 9057 | BIDDEFORD1, US |
| 74.80.208.170 | 8631 | CSTL, US |
| 74.80.208.59 | 8407 | CSTL, US |
| 216.244.66.239 | 5998 | WOW, US |
| 4.227.36.126 | 5832 | MICROSOFT-CORP-MSN-AS-BLOCK, US |
| 20.171.207.130 | 5817 | MICROSOFT-CORP-MSN-AS-BLOCK, US |
| 8.29.198.26 | 4946 | FEEDLY-DEVHD, US |
| 8.29.198.25 | 4807 | FEEDLY-DEVHD, US |
Now, I was curious as to how they identified themselves, so I reran the query to include the user agent string. The top eight identified themselves consistently:
| Agent | Requests |
|---|---|
| Go-http-client/2.0 | 43236 |
| Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/132.0.0.0 Safari/537.36 | 26650 |
| WF search/Nutch-1.12 | 9057 |
| Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | 8631 |
| Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | 8407 |
| Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com) | 5998 |
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | 5832 |
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | 5817 |
The last two, however had a changing user agent string:
| Agent | Requests |
|---|---|
| Feedly/1.0 (+https://feedly.com/poller.html; 16 subscribers; ) | 1667 |
| Feedly/1.0 (+https://feedly.com/poller.html; 6 subscribers; ) | 1419 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 6 subscribers; ) | 938 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 16 subscribers; ) | 811 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 8 subscribers; ) | 94 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 37 subscribers; ) | 17 |
| Agent | Requests |
|---|---|
| Feedly/1.0 (+https://feedly.com/poller.html; 16 subscribers; ) | 1579 |
| Feedly/1.0 (+https://feedly.com/poller.html; 6 subscribers; ) | 1481 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 6 subscribers; ) | 905 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 16 subscribers; ) | 741 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 8 subscribers; ) | 90 |
| Feedly/1.0 (+http://www.feedly.com/fetcher.html; 37 subscribers; ) | 11 |
I'm not sure what the difference is between polling and fetching (checking the URLs shows two identical pages, only differing in “Poller” and “Fetcher.” But looking deeper into that is for another post.
The next request I did was to see how many IPs (that hit my site in February) map to a particular ASN, and the top 10 are:
| AS | Count |
|---|---|
| ALIBABA-CN-NET Alibaba US Technology Co., Ltd., CN | 4034 |
| AMAZON-02, US | 1733 |
| HWCLOUDS-AS-AP HUAWEI CLOUDS, HK | 1527 |
| GOOGLE-CLOUD-PLATFORM, US | 996 |
| COMCAST-7922, US | 895 |
| AMAZON-AES, US | 719 |
| TENCENT-NET-AP-CN Tencent Building, Kejizhongyi Avenue, CN | 635 |
| MICROSOFT-CORP-MSN-AS-BLOCK, US | 615 |
| AS-VULTR, US | 599 |
| ATT-INTERNET4, US | 472 |
So Alibaba US crawled my site from 4,034 different IP addresses—I haven't done the query to figure out how many requests each ASN did, but it should be a straightforward thing to just replace IP address with the ASN to get a better count of which company is crawling my site the hardest.
And now I'm thinking, I wonder if instead of a form of ad-hoc banning of single IP addresses, or blocking huge swaths of IP addresses (like 47.0.0.0/8, it might not be better to block per ASN? The IP to ASN mapping service I found makes it quite easy to get the ASN of an IP address (and to map the ASN to an human-readable name), Instead of, for example, blocking 101.32.0.0/16, 119.28.0.0/16, 43.128.0.0/14, 43.153.0.0/16 and 49.51.0.0/16 (which isn't an exaustive list by any means) just block IPs belonging to ASN 132203, otherwise known as “TENCENT-NET-AP-CN Tencent Building, Kejizhongyi Avenue, CN.”
I don't know how effective that idea is, but the IP-to-ASN site I found does offer the information via DNS, so it shouldn't be that hard to do.
Discussions about this entry
Wednesday, March 19, 2025
How I vibe code
There's a new kind of coding I call "vibe coding", where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper so I barely even touch the keyboard. I ask for the dumbest things like "decrease the padding on the sidebar by half" because I'm too lazy to find it. I "Accept All" always, I don't read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension, I'd have to really read through it for a while. Sometimes the LLMs can't fix a bug so I just work around it or ask for random changes until it goes away. It's not too bad for throwaway weekend projects, but still quite amusing. I'm building a project or webapp, but it's not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.
Via Flutterby, Andrej Karpathy on X
Good Lord! If you thought software today was bloated and slow, this sounds like it would produce software that is gigantically glacial in comparison (and by “embrace exponentials” I think he means “accept code with O(n2), O(2n) or even O(n!) behavior”).
That's not how I would “vibe code.” No, to me, “vibe coding” is:
- Don't necessarily worry about the behavior of the code—make it work but at least try to avoid O(2n) or worse algorithms, then make it right, then fast.
- Don't use version control! If you make a mistake and need to revert, revert by hand, or carry on through the bad code. And avoid using directores like “src.1/”, “src.2/“ or “src-no-really-this-works/”—that's still a form of version control (albeit a poor man's version control). Power through your mistakes.
- Don't bother with “unit tests,” “integration tests,” TDD or even BDD. I'm not saying don't test, just don't write tests. Want to refactor? Go ahead—bull through the changes, or don't. It's your code. Yes, this does mean mostly manual testing, and having a file of test data is fine—just don't write test code.
- Format the code however you want! Form your own opinions on formatting. Have some soul in your code for once.
- This isn't a team sport, so no pair programming! This is vibe coding, not vibe partying.
- Remember the words of Bob Ross: “we don't make mistakes, just happy little accidents.”
- Go with the flow. Just Do It™!
Now that I think about it, this is pretty much how programmers wrote code on home computers in the late 70s/early 80s. Funny that. But just blindly accepting LLM-written code? Good luck in getting anything to run correctly.
Sheesh.
Tuesday, March 18, 2025
Who serves whom?
The narrative around these bots is that [AIs] are there to help humans. In this story, the hospital buys a radiology bot that offers a second opinion to the human radiologist. If they disagree, the human radiologist takes another look. In this tale, AI is a way for hospitals to make fewer mistakes by spending more money. An AI assisted radiologist is less productive (because they re-run some x-rays to resolve disagreements with the bot) but more accurate.
In automation theory jargon, this radiologist is a "centaur" – a human head grafted onto the tireless, ever-vigilant body of a robot
Of course, no one who invests in an AI company expects this to happen. Instead, they want reverse-centaurs: a human who acts as an assistant to a robot. The real pitch to hospital is, "Fire all but one of your radiologists and then put that poor bastard to work reviewing the judgments our robot makes at machine scale."
Pluralistic: AI can't do your job (18 Mar 2025) – Pluralistic: Daily links from Cory Doctorow
This has always been my fear of the recent push of LLM backed AI—not that they would help me do my job better, but that I existed to help it do its job better (if I'm even there).
A network of bloggers, a reel of YouTubers and other collective nouns
While I just made up the “network of bloggers” and “reel of YouTubers,” other collective nouns for groups, like a gaggle of geese, a murder of crows, or a pod of whales, are not quite as old as they may seem, and were largely made up just a few hundred years ago, and there were a lot more than we use today, according to this video. Neat.
Measuring the cosmos, part II
Last month, I mentioned part one of how we measured the night sky, and now, part two of “Terence Tao on how we measure the cosmos”.
Monday, March 03, 2025
A quirk of the Motorola 6809 assemblers
I just learned an interesting bit of trivia about 6809 assembly language on a Discord server today.
When Motorola designed the 6809 assembler,
they made a distinction between the use of n,PC and n,PCR in the indexing mode.
Both of those make a reference based off the PC register,
but in assembly language they defined,
using n,PC means use the literal value of n as the distance,
whereas n,PCR means generate the distance between n and the current value of the PC register.
I never knew that.
I just looked and all the materials I had on the 6809 use the n,PCR method everywhere,
yet when I wrote my assembler,
I only support n,PC and it always calculates the distance.
I think I forgot that it should have been n,PCR because on the 68000
(which I also programmed,
and was also made by Motorola) it always used n,PC.
And I don't think I'll change my assembler as there does exist a method to use an arbitrary value of n as a distance:
LDA (*+3)+n,PC.
The asterisk evaluates to the address of the current instruction,
and by adding 3 you get the address of the next instruction,
which in the PC-relative addressing mode,
is a distance of 0.
Then n will be the actual offset used in the instruction.
Yes,
it's a bit convoluted,
but it's a way to get how Motorola originally defined n,PC.
And apparently, Motorola defined it that way to make up for less intelligent assemblers back in the day due to memory constraints. We are long past those days.
Yelling at clouds
I will admit—these are kneejerk reactions, but they're honestly my reactions to reading the following statements. I know, I know, hanging onions off our belt is long out of style.
And get off my lawn!
Anyway … statment the first:
Think
jq, but without having to ask an LLM to write the query for you.
Via Lobsters, A float walks into a gradual type system
So … using jq is so hard you need to use a tool that will confabulate ¼ of the time in order to construct a simple query?
Is that what you are saying?
That you can't be bothered to use your brain?
Just accept the garbage spewed forth by a probabilistic text slinger?
Really?
And did you use an LLM to help write the code? If not, why not?
Sigh.
And statement the second:
… and most importantly, coding can be social and fun again.
Via Lobsters, introducing tangled
If I had known that programming would become a team sport, I, an introvert, would have choosen a different career. Does XXXXXXX everything have to be social? Why can't it just be fun? I need to be micromanaged as well?
