Tuesday, August 05, 2025
A bit of a deep dive into the Feedly bot
This Lobsters thread got me looking at Feedly again. Last month, the Feedly bot made the following requests to the Atom feed for my site:
Resource | Response | Agent | Requests |
---|---|---|---|
/index.atom | 200 | Feedly/1.0 | 2058 |
//index.atom | 200 | Feedly/1.0 (poller) | 86 |
/index.atom | 200 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 37 subscribers; ) | 19 |
/index.atom | 200 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 16 subscribers; ) | 19 |
//index.atom | 200 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 8 subscribers; ) | 19 |
/index.atom | 200 | Feedly/1.0 (poller; 37 subscribers; ) | 11 |
/index.atom | 200 | Feedly/1.0 (+https://feedly.com/poller.html; 37 subscribers; ) | 1 |
/index.atom | 200 | Feedly/1.0 (+https://feedly.com/poller.html; 16 subscribers; ) | 1 |
One suggestion for one request for /index.atom
having 37 subscribers,
and another for /index.atom
having 16 subscribers was one was originally for http:
and the other for https:
.
That's a decent explanation,
given we have 8 subscribers for //index.atom
,
telling me that Feedly is treating http://boston.conman.org/index.atom
,
https://boston.conman.org/index.atom
and https://boston.conman.org//index.atom
as entirely separate feeds,
even though I now redirect http:
to https:
.
But even though I do redirect http:
to https:
,
it's with a temporary redirect,
not a permanent one
(because I'm still wary about making the redirect permanent)
so that one is totally on me;
the //index.atom
is obviously a typo so that one is totally on Feedly.
I still can't tell the difference between the fetcher and the poller. Even the pages describing the two are identical, except one says “Fetcher” and the other says “Poller.” That's just really weird. And what's with the plain “Feedly/1.0” bot?
The 200 response means that Feedly did not do a conditional fetch of the feed
(Feedly can ask “Did the file change since I last requested it?” and my server can reply with either
“Yes, here it is” with a 200 response, or “No, it did not” with a 304 response).
I did go back before I {^2022/12/04.1 switch to https:
) and there,
(from November of 2022)
I get a completely different Feedly:
Resource | Respose | Agent | Requests |
---|---|---|---|
/index.atom | 304 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 27 subscribers; like FeedFetcher-Google) | 2656 |
/index.atom | 304 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 28 subscribers; like FeedFetcher-Google) | 477 |
/index.atom | 200 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 27 subscribers; like FeedFetcher-Google) | 40 |
/index.atom | 200 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 28 subscribers; like FeedFetcher-Google) | 8 |
Aside from picking up one more subscription from Feedly,
it's what I would expect—most requests are conditional with none of that “fetcher/poller” stuff.
I can explain the current lack of conditional requests on the http:
to https:
redirect throwing off the request code,
since that seems to hold true starting with December of 2022.
But in June of 2023,
when I get my first https:
subscriber,
there is no conditional requests:
Resource | Response | Agent | Requests |
---|---|---|---|
/index.atom | 200 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 32 subscribers; ) | 227 |
/index.atom | 200 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 1 subscribers; ) | 136 |
/index.atom | 200 | Feedly/1.0 (+http://www.feedly.com/fetcher.html; 33 subscribers; ) | 74 |
So maybe their https:
request code fails to do conditional requests?
Odd,
but it does explain why there are no conditional requests.
It also appears that they tuned their polling down from November of 2022 to June 2023.
Seems like a few easy bugs to fix to me.
But then in September of 2023,
the number of requests for the https:
version shoots up over 4,000,
almost 6,000 in October 2023,
and in November of 2023 the “poller” first shows up,
only to go away in December of 2023,
only to show up again in March 2024 and stick around from there.
So it's clear to me that the backend at Feedly changed,
and from my point of view,
not for the better.