Dynamically create sitemap.xml

[veganstraightedge]

Issues

• Sitemap ping failing #3780
• Figure out process for updating/uploading the sitemap.xml.gz #1192

closes #3780
closes #1192

An investigation and discoveries

So, it turns out that we maybe never had a live sitemap.xml or .xml.gz in production, this whole time. 🤦🏻

The reason that it seemed to work in development and seemed to succeed in the deploy/release in production but not actually available in production is because… Heroku's ephemeral filesystem.

Ephemeral Disk
Heroku has an “ephemeral” hard drive, this means that you can write files to disk, but those files will not persist after the application is restarted.

So, what was happening was in development, running bundle exec rails sitemap:create or sitemap:refresh etc, would create the sitemap.xml.gz in our local /public folders. And stay there. Seems good.

In production, during the release stage of a production deploy (as defined in the Procfile), the sitemap:refresh would "succeed", but then the /public folder it was created in wouldn't necessarily be in the actual dyno/s serving any real requests. AFAICT.

---

My preferred requirements

When working on this, I went round and round trying to make it work with all of these conditions:

• sitemap_generator gem (great gem, love it, would use in non-Heroku deployed apps)
• Heroku's ephemeral filesystem
• and keeping its location on the same domain at the root, where it can be auto-discovered without first reading robots.txt

The gem suggests and has functionality to store the generated file somewhere else (say, S3), but I'd like to keep it in its well known location.

---

Conclusion

In the end, I decided to create a sitemaps controller and dynamically create the file to:

• keep it at /sitemap.xml
• not have to deal with Heroku's ephemeral filesystem
• keep its discovery well known and not dependent on robots.txt (a separate issue/pr to do!)

The challenge and risk, of course, is performance. Namely around articles and some of the tools (zines, etc) which have the biggest tables to scan. Especially since most items in the sitemap never change.

But that not ever really changing-ness is what allowed me to use Rails' fragment caching around each <url> item in the long list of <urlset> and reduce the page load time from ~1s to ~200ms (depending on warm cache, etc). Even at 1s, it's not the end of the world, since (I'm suspecting) that this file doesn't get read a ton.

TODO follow up

• remove sitemap:refresh from Procfile
• remove config/sitemap.rb
• remove sitemap_generator gem
• keep an eye on this in production
• add any files/paths that i missed (or skipped for now because i ran out of steam)

[veganstraightedge]

✅ https://crimethinc.com/sitemap.xml