NetNewsWire and Conditional GET Issues 3 Aug 2024, 9:21 pm

I had thought that NetNewsWire’s conditional GET support was rock-solid — and so my first reaction was to be very surprised to learn that it’s not!

My second reaction was to be appreciative — Rachel’s work here on setting up a test server and reporting on the results is really great. My goal has always been to make NetNewsWire a model net citizen, and learning where it’s not is super valuable. So: much respect and thanks to Rachel for this.

The Data

Let’s look at some data and try to figure out what’s happening. Here’s Rachel’s report for NetNewsWire.

Things to know: these are all requests for a NetNewsWire-specific feed, and the copy of NetNewsWire making these requests is on my personal laptop. That laptop is occasionally used for development, which can throw things off, but not often. You can even see in the data a gap lasting just over two weeks where there were no requests (I was on vacation).

(You can also see some anomalies from when I had it on my dev machine also — ignore every row where ip is v6, since that’s my dev machine.)

Another thing to know: this is testing direct feed-reading, as with the On My Mac (or iPhone/iPad) and iCloud accounts. With systems such as Feedly, Feedbin, and so on, we get the data from the sync system and not directly from the site.

Ignoring Timing Issues

Let’s set aside, at least for today, the timing issues. That situation could be improved, but it very much reflects that this is a desktop app with a command that allows you to refresh feeds manually, without having to wait for the next poll.

Conditional GET Issues

First, a refresher on how this should work.

When a server returns a Last-Modified header, the client should return that exact same string in follow-up requests in an If-Modified-Since header. The server then looks at the If-Modified-Since header and decides to either return a 200 plus the feed — if it has been modified since — or return a 304 Not Modified response and an empty body.

It’s the same story with the Etag header. The client should save it and return it in follow-up requests in an If-None-Match header.

This is great because it can save a ton of bandwidth, which is great for server and app alike. And NetNewsWire’s been doing this since the early 2000s.

But clearly there’s a bug! In some cases, NetNewsWire is not picking up and saving the changed Last-Modified and Etag headers. Sometimes it does, and sometimes it keeps using whatever it already had and ignores the new ones.

What could account for this? Let’s look at the logic.

Feed processing logic

Here’s what happens when a feed download completes without errors and the content is non-empty:

First we check the hash of the raw feed data against the hash of the raw feed data the last time it actually changed. If those hashes match, then the app stops processing, because the feed hasn’t changed: it’s exactly the same as last time.

This is an optimization that deals with the fact that many servers unfortunately don’t support conditional GET. It allows the app to skip feed parsing and updating the database. Saves a bunch of work. Good for battery life.

If the hashes don’t match, then processing continues: it parses the feed and then sends the parsed articles to the code that updates the database.

After that it updates and saves the hash of the raw feed data, and finally it stores the conditional GET info — it saves any Last-Modified and Etag header values to send with the next request.

This isn’t actually the code, but it’s what the logic looks like:

downloadDidComplete(httpResponse, feed, feedData)
	hash = feedData.md5
	if hash == feed.previousHash then return
	parsedFeed = parse(feedData)
	updateDatabase(feed, parsedFeed)
	feed.previousHash = hash
	feed.conditionalGetInfo = conditionalGetInfoFromResponse(httpResponse)

My theory

There’s a great chance you’ve already spotted what I think is the issue: it’s that optimization where we check the hash of the raw feed data and return if it matches the previous hash.

Here’s what I think has happened in some of the tests: the raw feed data was unchanged, but one or both of the Last-Modified and Etag header values did change.

NetNewsWire never picked up the changes to those headers, because that code didn’t run — it had already bailed when it saw that the raw feed data was unchanged.

The assumption I made when I wrote this code was that if the raw feed data was unchanged then of course the Last-Modified and Etag header values would be unchanged too, so there was no need to check to see if they were new.

And I think that in real-world situations this is probably true pretty much all the time, and it’s only in tests like this where my assumption wouldn’t be true.

But I can’t say that for sure! This is a real bug, and we’ll fix it and add a test or tests to make sure it doesn’t happen again.

Here’s what the new logic should look like:

downloadDidComplete(httpResponse, feedData, feed)
	hash = feedData.md5
	if hash != feed.previousHash {
		parsedFeed = parse(feedData)
		updateDatabase(feed, parsedFeed)
		feed.previousHash = hash
	}
	feed.conditionalGetInfo = conditionalGetInfoFromResponse(httpResponse)

With the above logic, conditionalGetInfo gets updated no matter what.

PS There could be other bugs

My theory does point to a bug that should get fixed. But is it the only bug? Is it even the bug that causes the issues in these tests?

Though I’m pretty confident that this is the bug — seems pretty obvious, right? — more investigation and testing is warranted.

Reruns 8 Feb 2024, 4:16 pm

It’s not a bug in your RSS reader if recent articles in this feed all suddenly appeared as unread. You may even have seeming duplicates.

Sorry about that! It’s due to my changing settings in my blog generator. Pages now have a .html suffix where before they had none. This change impacts permalinks, which also changes the guids in my RSS feed — and NetNewsWire and other RSS readers use the guid property to identify articles, which means these will show up as new articles.

(Note: I’ve created redirects so that old links pointing in will still work.)

Why NetNewsWire Isn’t Available for Vision Pro 5 Feb 2024, 6:39 am

I’ve been getting questions about NetNewsWire’s unavailability on Vision Pro. Why isn’t it there? When might it be there?

Here’s the scoop:

I consider it risky to support an app running on a device I don’t own. Imagine writing a Mac or iPhone app and not actually running it on one of those machines — you wouldn’t.

I realize that the app would act as if it were running on an iPad — but the Vision Pro is not really an iPad. It’s a device with very different interactions from the direct manipulation we’re used to on iPad. And the compatibility mode is a new thing because this device is a new thing — we don’t know how well it works and what the gotchas might be.

I could test it on the simulator, sure, but the simulator is a convenience for developers. It’s no replacement for running it on a real device.

And, yes, the app is open source, which could mean that other developers with a Vision Pro could help support it — but it’s important to remember that I’m the only person who has to support it. The NetNewsWire team is awesome, but I’m the one on the line for this.

So I want to be careful and go slowly, because if I made it available it would be extremely difficult to reverse the decision and take it away, even with an excellent reason.

I’m hoping that a consensus will form among developers that running apps on Vision Pro as iPad apps is fine, that it’s a cakewalk. If that happens, I’d go ahead and do it too. But it’s too soon for us to know that.

PS Why don’t I have a Vision Pro? I’m sure it’s an incredible technical achievement and an amazing experience — and pretty damn wonderful in just about every way — but it’s just not my thing. I like reading and writing mostly, plus making apps for reading and writing. My personal future of computing has been here for all these years — the Mac.

PPS Eventually the price will come down to where I’d consider buying one as a test device and for a little fun — but that may be a few years away. I’m hoping that we’ll find, sooner than that, that running as iPad on Vision Pro is a-okay.

Corporations Are Not To Be Loved 17 Jan 2024, 6:25 pm

I started using Apple computers — and writing code for them, starting with BASIC — 43 years ago, before the Macintosh, even, and I’ve made this my career. I’ve had all these decades to really, thoroughly delight in these incredible machines and software, and to give a little back with my own apps.

Apple’s positive effect on my life should not be underestimated. My Mom once (lovingly, teasingly) said to me that my alternate career, had all this never happened, was “criminal genius.” Which might have been fun too, but possibly more stressful than I might have liked. At any rate, Apple has saved me from a life of crime, and I should love Apple for that.

But I need to remember, now and again, that Apple is a corporation, and corporations aren’t people, and they can’t love you back. You wouldn’t love GE or Exxon or Comcast — and you shouldn’t love Apple. It’s not an exception to the rule: there are no exceptions.

Apple doesn’t care about you personally in the least tiny bit, and if you were in their way somehow, they would do whatever their might — effectively infinite compared to your own — enables them to deal with you.

Luckily, Apple has just provided us all with a reminder. Just like the sixth finger in an AI-rendered hand, Apple’s policies for Distributing apps in the U.S. that provide an external purchase link are startlingly graceless and a jarring, but not surprising, reminder that Apple is not a real person and not worthy of your love.

On Mastodon Support in NetNewsWire 18 Dec 2023, 2:56 am

Tim Chambers, admin of the Mastodon server indieweb.social, which hosts my personal account and the NetNewsWire account, asked me if ActivityPub support is on the roadmap for NetNewsWire. Great question!

Tim was responding to Richard MacManus:

Are there any RSS Readers that are trying to integrate ActivityPub too? Pondering the glut of email newsletters in our inboxes these days, and wondering if we are due for an RSS Reader renaissance, except with a fediverse twist this time…

I hear this. It’s not the first time this has come up, and I’ve spent the past year or so thinking about it.

Since RSS is an open web thing that brings you stuff people write, and ActivityPub is also an open web thing that brings you stuff people write, it’s an obvious good idea to do both in the same app. Totally.

But the question was specifically about NetNewsWire.

We will make NetNewsWire work better with Mastodon-generated RSS feeds. There are some things we’re not doing yet that will make reading these a better experience.

But should NetNewsWire become a Mastodon client? That’s where the answer is not so obvious.

* * *

Let’s compare the NetNewsWire and Mastodon experiences.

NetNewsWire is a three-paned RSS reader. Sidebar (accounts, folders, feeds) on the left. Timeline (titles and/or first few lines) in the middle. Article on the right. See these screenshots to get the idea.

Each article has a read/unread status. Selecting an article marks it as read, and there are commands to mark all as read (and mark all as read above or below the selection).

This is not the only good model for an RSS reader, but it’s how NetNewsWire works — and it’s a bad fit for a Mastodon reading experience.

Whether you use Mastodon on the web or in an app you’re used to a single timeline per account (with perhaps side streams for mentions and such). All posts from everyone you follow appear in that single timeline. You don’t deal with read/unread status — instead you have a chronological position.

The timeline shows the entire post (usually), instead of an optional title and first few lines. There is no third article view, because the entire post is displayed in the timeline.

This experience isn’t unique to Mastodon, of course — that’s how X, Threads, Micro.blog, Bluesky, and other social media apps work.

But it’s not how NetNewsWire works, and if we try to mash that experience into the app — but have it on only when a Mastodon account is selected — we’d end up with a confusion that feels like two very different apps. Because it would be.

* * *

A second option: we could not change the experience and just assume that some people would like to use Mastodon in a NetNewsWire way instead of in the usual social media way. I’m extremely skeptical of this idea — I think it would be a pain, and I think most people who try it would drop it.

The third option, which I think is best, which we already do, is to support Mastodon via RSS. (Mastodon feeds are usually available as RSS — you can add .rss to the end of a URL to get a feed.)

There might be some Mastodon accounts you want to follow in your RSS reader instead of in your Mastodon client. Think of an account for a website that doesn’t have an RSS feed, but that reliably posts links to articles. Or a person you want to follow — but you don’t want to show up in their follower list. (Sounds a little antisocial, sure, but you can imagine good reasons.)

We can, and will, do more to make Mastodon-over-RSS a good experience in NetNewsWire. Right now we’re not picking up avatars or image attachments, which we can fix. There might be more things like that to do. We might add the ability to post to Mastodon, since it’s a natural thing to want to share stuff from your RSS reader to Mastodon.

But I don’t think we’ll make NetNewsWire a real Mastodon client.

* * *

It still seems like RSS and Mastodon could fit in the same app, though! If I were designing it, I’d start with the social media experience: the single timeline of posts. Very simple sidebar. No article view. No read/unread status — just position in the timeline.

You could add RSS feeds, but they’d be treated like Mastodon posts. Any article short enough would appear in full in the timeline, but most would probably have to be truncated. You’d open articles in your browser, just like you do now with social media apps (there’d be no third article pane).

Such an app could be a nice unified experience. Get your Mastodon, Threads, RSS feeds, Micro.blog and, hopefully, other services — anything that supports ActivityPub, RSS, or some other open format or API — all in one place, in a way that’s already familiar to everyone.

Sounds pretty great! But it’s not NetNewsWire.