Migrating to an org-mode website

Introduction

This is a story… a story of me changing the way I code and publish my website. In the past, I’ve switch from Jekyll to “orgmode and Jekyll” to Hugo (sorry those are written in french). The past year, I’ve written and documented myself a little bit about minimalism and digital minimalism. Although I don’t see myself as a minimalist, it helped me realize some issues I had.

I also realized if I want to write more, I need to lower the barrier between writing and publishing my content ; if I want it to be published, of course. This post is about what I’m putting in place for this, with a premise : I spend my life in Emacs and thus in orgmode. And orgmode is feature-full and has this badass feature : org-publish.

To build and publish this website, we will try to rely on a reproducible setup, meaning Emacs and orgmode of course, GNU Make of course but most importantly, Nix (in the near future 👼).

Update

There is now an article about it, that uses literate programming: publishing this website. The content of the post might no be up-to-date at some point.

Requirements

Let’s list the requirements I feel I have for my website:

Full control over the URL of the published posts.
This is a golden rule of the web: should I change the publishing system, I want to be able to stick to the same URLs or else all external references would be broken. This is a big no-no and in my opinion it makes most blogging systems unacceptable.
Top-notch Org support.
I believe generators like Jekyll and Hugo only have partial Org support. You end up requiring some conversion tooling to get the job done.
Simple publishing pipeline.
I want the generation process to be as simple as possible. This is important for maintenance. Should I someday switch host, I want to be sure that I can set up the same pipeline.
Full control over the publishing system.
I want maximum control over the generation process. I don’t want to be restricted by a non-Turing-complete configuration file or a dumb programming language.
Ease of use.
The process as a whole must be as immediate and friction-less as possible, or else I take the risk of feeling too lazy to publish new posts and update the content.
Hackability.
Last but not least, and this probably supersedes all other requirements: The system must be hackable. Lisp-based systems are prime contenders in that area.

DONE Organizing

Logbook
  • State “DONE” from “TODO” [2020-03-21 Sat 17:57]

Let’s describe what I do want to publish here:

Posts
this is what we call blog these days : short to medium article that are valid at a point of time, as may contain deprecated content, or content that do not reflect my views at a later point in time.
Articles
medium to long article about a topic. Those should be up-to-date or explicitly mark as deprecated or invalid. In a ideal world this is my ready for the public knowledge database, a bit like Jethro’s Braindump.
Configurations
medium to long article about my configurations. Those are base on my home configuration mono-repository, and usually follow literate programming principles.
About
an about page about the author of the website (aka me), linking external contributions (GitHub/Gitlab/… profiles, Talks, …).

DONE Publishing

Logbook
  • State “DONE” from “TODO” [2020-03-25 Wed 15:54]

As said above, the goal is to publish everything using only Emacs and orgmode (with the help of some standard GNU tools).

The publish.el file is where all the magic happens:

  • I want to generate something that is html5 (almost?). The preamble, content and postamble div can be customized using org-html-div. Same goes for the container element that is used for “wrapping top level sections”, it is customized using org-html-container-helement (we want <section>). There is a few additional variable that I might document one day 😛. 

    (setq org-export-use-babel nil)
(setq org-link-abbrev-alist '(("att" . org-attach-expand-link)))

;; setting to nil, avoids "Author: x" at the bottom
(setq org-export-with-section-numbers nil
      org-export-with-smart-quotes t
      org-export-with-toc nil)

(defvar sbr-date-format "%b %d, %Y")

(setq org-html-divs '((preamble "header" "top")
                      (content "main" "content")
                      (postamble "footer" "postamble"))
      org-html-container-element "section"
      org-html-metadata-timestamp-format sbr-date-format
      org-html-checkbox-type 'unicode
      org-html-html5-fancy t
      org-html-doctype "html5"
      org-html-htmlize-output-type 'css
      org-html-htmlize-font-prefix "org-"
      org-src-fontify-natively t
      org-html-coding-system 'utf-8-unix)
  • Part of the <head>, preamble and postamble are customized for the website.
    head
    (defvar sbr-website-html-head
  "<link rel='icon' type='image/x-icon' href='/images/favicon.ico'/>
<meta name='viewport' content='width=device-width, initial-scale=1'>
<link rel='stylesheet' href='/css/new.css' type='text/css'/>
<link rel='stylesheet' href='/css/syntax.css' type='text/css'/>
<link href='/index.xml' rel='alternate' type='application/rss+xml' title='Vincent Demeester' />")
    premable
    (defun sbr-website-html-preamble (plist)
  "PLIST: An entry."
  ;; Skip adding subtitle to the post if :KEYWORDS don't have 'post' has a
  ;; keyword
  (when (string-match-p "post" (format "%s" (plist-get plist :keywords)))
    (plist-put plist
               :subtitle (format "Published on %s by %s."
                                 (org-export-get-date plist sbr-date-format)
                                 (car (plist-get plist :author)))))

  ;; Below content will be added anyways
  "<nav>
<img src=\"/images/favicon.ico\" id=\"sitelogo\"/> <a href='/'>home</a> /
<a href='/posts/'>posts</a> (<a href='/index.xml'>rss</a>) /
<a href='/articles/'>articles</a> /
<a href='/configurations/'>configurations</a> /
<a href='https://dl.sbr.pm/'>files</a> /
<a href='/about/'>about</a></li>
</nav>")
    postamble
    (defvar sbr-website-html-postamble
  "<footer>
     <span class='questions'>Questions, comments ? Please use my <a href=\"https://lists.sr.ht/~vdemeester/public-inbox\">public inbox</a> by sending a plain-text email to <a href=\"mailto:~vdemeester/public-inbox@lists.sr.ht\">~vdemeester/public-inbox@lists.sr.ht</a>.</span>
     <span class='opinions'>Opinions stated here are my own and do not express the views of my employer, spouse, children, pets, neighbors, secret crushes, favorite authors, or anyone else who is not me. And maybe not even me, depending on how old this is.</span>
     <span class='copyright'>
      Content and design by Vincent Demeester
      (<a rel='licence' href='http://creativecommons.org/licenses/by-nc-sa/3.0/'>Some rights reserved</a>)
    </span><br />
    <span class='engine'>
      Powered by <a href='https://www.gnu.org/software/emacs/'>Gnu Emacs</a> and <a href='https://orgmode.org'>orgmode</a>
    </span>
</footer>")

  • orgmode is able to generate a site-map. This is what we are going to use to generate the index files for posts/ and articles mainly. 

    (defun sbr/org-sitemap-format-entry (entry style project)
  "Format posts with author and published data in the index page.

ENTRY: file-name
STYLE:
PROJECT: `posts in this case."
  (cond ((not (directory-name-p entry))
         (format "%s — [[file:%s][%s]]
                 :PROPERTIES:
                 :PUBDATE: [%s]
                 :END:"
                 (format-time-string "%Y-%m-%d"
                                     (org-publish-find-date entry project))
                 entry
                 (org-publish-find-title entry project)
                 (format-time-string "%Y-%m-%d"
                                     (org-publish-find-date entry project))))
        ((eq style 'tree) (file-name-nondirectory (directory-file-name entry)))
        (t entry)))

(defun sbr/org-publish-sitemap (title list)
  ""
  (concat "#+TITLE: " title "\n\n"
          (org-list-to-subtree list)))

  • orgmode is able to generate a rss, with ox-rss. This is what we are going to use to generate the rss files for posts and articles. This is heavily customized. 

    (defun sbr/org-get-first-paragraph (file)
  "Get string content of first paragraph of file."
  (ignore-errors
    (with-temp-buffer
    (insert-file-contents file)
    (goto-char (point-min))
    (show-all)
    (let ((first-begin (progn
                         (org-forward-heading-same-level 1)
                         (next-line)
                         (point)))
          (first-end (progn
                       (org-next-visible-heading 1)
                       (point))))
      (buffer-substring first-begin first-end)))))

(defun sbr/org-rss-publish-to-rss (plist filename pub-dir)
  "Prepare rss.org file before exporting."
  (let* ((postsdir (plist-get plist :base-directory)))
    (with-current-buffer (find-file filename)
      (erase-buffer)
      (insert "#+TITLE: Posts\n")
      (insert "#+AUTHOR: Vincent Demeester\n")
      (insert "#+OPTIONS: toc:nil\n")
      (let* ((files-all
              (reverse (directory-files "." nil
                                        "[0-9-]+.*\\.org$")))
             (files (subseq files-all 0 (min (length files-all) 30))))
        (message (format "foo: %s" filename))
        (dolist (post files)
          (let* ((post-file post)
                 (post-title (org-publish-find-title post-file plist))
                 (preview-str (sbr/org-get-first-paragraph post-file))
                 (date (replace-regexp-in-string
                        "\\([0-9]\\{4\\}-[0-9]\\{2\\}-[0-9]\\{2\\}\\)-.*"
                        "\\1" post)))
            (insert (concat "* [[file:" postsdir "/" post "][" post-title "]]\n\n"))
            (org-set-property "ID" post)
            (org-set-property "RSS_TITLE" post-title)
            ;; ox-rss prepends html-link-home to permalink
            (org-set-property "RSS_PERMALINK"
                              (concat postsdir "/"
                                      (file-name-sans-extension post)
                                      ".html"))
            (org-set-property
             "PUBDATE"
             (format-time-string
              "<%Y-%m-%d %a %H:%M>"
              (org-time-string-to-time
               (replace-regexp-in-string
                "\\([0-9]\\{4\\}-[0-9]\\{2\\}-[0-9]\\{2\\}\\)-.*"
                "\\1" post))))
            (insert preview-str)
            (newline 1)
            (insert (concat "[[file:" postsdir "/" post "][(Read more)]]\n\n"))))
        (save-buffer))))
  (let ((user-mail-address "t")
        (org-export-with-broken-links t)
        (org-rss-use-entry-url-as-guid nil))
    (org-rss-publish-to-rss plist filename pub-dir)))

  • Finally let’s set the org-publish-project-alist to publish our projects 

    (setq org-publish-project-alist
      `(("posts"
         :base-directory "posts"
         :base-extension "org"
         :recursive t
         :publishing-function org-html-publish-to-html
         :publishing-directory "./public/posts"
         :exclude ,(regexp-opt '("README.org" "draft"))
         :auto-sitemap t
         :with-footnotes t
         :with-toc nil
         :with-drawers t
         :sitemap-filename "index.org"
         :sitemap-title "Posts"
         :sitemap-format-entry sbr/org-sitemap-format-entry
         :sitemap-style list
         :sitemap-sort-files anti-chronologically
         :sitemap-function sbr/org-publish-sitemap
         :html-head-include-scripts nil
         :html-head-include-default-style nil
         :html-head ,sbr-website-html-head
         :html-preamble sbr-website-html-preamble
         :html-postamble ,sbr-website-html-postamble)
        ("posts-rss"
         :base-directory "posts"
         :base-extension "org"
         :recursive t
         :html-link-home "https://vincent.demeester.fr/"
         :rss-link-home "https://vincent.demeester.fr/posts/"
         :html-link-use-abs-url t
         :rss-extension "xml"
         :publishing-directory "./public"
         :publishing-function (sbr/org-rss-publish-to-rss)
         :section-number nil
         :exclude ".*"
         :include ("index.org"))
        ("articles"
         :base-directory "articles"
         :base-extension "org"
         :recursive t
         :publishing-function org-html-publish-to-html
         :publishing-directory "./public/articles"
         :exclude ,(regexp-opt '("README.org" "draft"))
         :auto-sitemap t
         :with-footnotes t
         :with-toc nil
         :with-drawers t
         :sitemap-filename "sitemap.org"
         :sitemap-title "Articles"
         :sitemap-style tree
         :sitemap-sort-files anti-chronologically
         ;;:sitemap-format-entry sbr/org-sitemap-format-entry
         ;;:sitemap-function sbr/org-publish-sitemap
         :html-head-include-scripts nil
         :html-head-include-default-style nil
         :html-head ,sbr-website-html-head
         :html-preamble sbr-website-html-preamble
         :html-postamble ,sbr-website-html-postamble)
        ("articles-assets"
         :exclude ,(regexp-opt '("*.org"))
         :base-directory "articles"
         :base-extension ,site-attachments
         :publishing-directory "./public/articles"
         :publishing-function org-publish-attachment
         :recursive t)
        ("about"
         :base-directory "about"
         :base-extension "org"
         :exclude ,(regexp-opt '("README.org" "draft"))
         :index-filename "index.org"
         :recursive nil
         :with-footnotes t
         :with-toc nil
         :with-drawers t
         :publishing-function org-html-publish-to-html
         :publishing-directory "./public/about"
         :html-head-include-scripts nil
         :html-head-include-default-style nil
         :html-head ,sbr-website-html-head
         :html-preamble sbr-website-html-preamble
         :html-postamble ,sbr-website-html-postamble)
        ("index"
         :base-directory ""
         :base-extension "org"
         :exclude ,(regexp-opt '("README.org" "draft"))
         :index-filename "index.org"
         :recursive nil
         :with-footnotes t
         :with-toc nil
         :with-drawers t
         :with-title nil
         :publishing-function org-html-publish-to-html
         :publishing-directory "./public"
         :html-head-include-scripts nil
         :html-head-include-default-style nil
         :html-head ,sbr-website-html-head
         :html-preamble sbr-website-html-preamble
         :html-postamble ,sbr-website-html-postamble)
        ("css"
         :base-directory "./css"
         :base-extension ,site-attachments
         :recursive t
         :publishing-directory "./public/css"
         :publishing-function org-publish-attachment
         :recursive t)
        ("images"
         :base-directory "./images"
         :base-extension ,site-attachments
         :publishing-directory "./public/images"
         :publishing-function org-publish-attachment
         :recursive t)
        ("assets"
         :base-directory "./assets"
         :base-extension ,site-attachments
         :publishing-directory "./public/assets"
         :publishing-function org-publish-attachment
         :recursive t)
        ("legacy"
         :base-directory "./legacy"
         :base-extension ,site-attachments
         :publishing-directory "./public/"
         :publishing-function org-publish-attachment
         :recursive t)
        ("all" :components ("posts" "about" "index" "articles" "articles-assets" "css" "images" "assets" "legacy" "posts-rss"))))

Here are some inspiration I took for this publishing code:

DONE Styling

Logbook
  • State “DONE” from “STARTED” [2020-03-23 Mon 19:02]

The style of the website has be as simple as possible, and also really light. This means:

  • use default system font as much as possible
  • have a small stylesheet, rely on the default as much as we can

In addition, I want support for:

  • side notes
  • code syntax highlight
  • table of content

The inspiration for this website, in term of style are the following:

To be able to define the style a bit, let’s try some things below. From this point on, this is random content just to try my style out. 👼

There is more in the sandbox.

Let’s dig into how I setup my development environment when working on tektoncd/pipeline

sub-heading 1

Checking for errors is very common in Go, having Comparison function for it was a requirement.

  • Error fails if the error is nil or the error message is not the expected one.
  • ErrorContains fails if the error is nil or the error message does not contain the expected substring.
  • ErrorType fails if the error is nil or the error type is not the expected type.

Let’s first look at the most used : Error and ErrorContains. When you’re working on pipeline, usually you want :

  1. make sure it compiles : go build -v ./..
  2. Running unit tests : go test ./... (bonus use ram for continuous testing)

  3. End-to-end tests : go test -tags e2e ./... (or simply using `./test/` package)

    Make sure you re-deploy before running the e2e tests using ko apply -f ./config, otherwise you’re testing the wrong code. 

var err error
// will fail with : expected an error, got nil
assert.Check(t, cmp.Error(err, "message in a bottle"))
err = errors.Wrap(errors.New("other"), "wrapped")
// will fail with : expected error "other", got "wrapped: other"
assert.Check(t, cmp.Error(err, "other"))
// will succeed
assert.Check(t, cmp.ErrorContains(err, "other"))
2020-02-29-13-46-08.png
Figure 1: This is the caption for the next figure link (or table)

If this is the case, then what makes dynamic languages feel easy? Can we take what we learn in answering this question and improve the ergonomics of our static languages? For example, in 2018, I don’t think there’s as strong an argument for “you don’t have to write out the types,” since modern type inference eliminates most of the keystrokes required (even though many major languages still lack such facilities). Plus, saving a few keystrokes does not seem like a critical bottleneck in the programming process.

2020-02-29-14-41-59.png

sub-heading 2

Some content from my other org-mode files.

I already wrote 2 previous posts about golang and testing. It’s something I care deeply about and I wanted to continue writing about it. It took me a bit more time than I thought, but getting back to it. Since the last post, Daniel Nephin and I worked (but mainly Daniel 🤗) on bootstrapping a testing helper library.

Package assert provides assertions for comparing expected values to actual values. When assertion fails a helpful error message is printed.

There is already some good testing helpers in the Go ecosystem : testify, gocheck, ginkgo and a lot more — so why create a new one ? There is multiple reason for it, most of them can be seen in the following GitHub issue.

Daniel also wrote a very useful 1converter if your code base is currently using testify : gty-migrate-from-testify. 

$ go get -u gotest.tools/assert/cmd/gty-migrate-from-testify
# […]
$ go list \
     -f '{{.ImportPath}} {{if .XTestGoFiles}}{{"\n"}}{{.ImportPath}}_test{{end}}' \
     ./... | xargs gty-migrate-from-testify

We’ll Use Assert for the rest of the section but any example here would work with Check too. When we said Comparison above, it’s mainly the BoolOrComparison interface — it can either be a boolean expression, or a cmp.Comparison type. Assert and Check code will be smart enough to detect which one it is. 

assert.Assert(t, ok)
assert.Assert(t, err != nil)
assert.Assert(t, foo.IsBar())

What’s next ?

One thing is to import old blog posts from vincent.demeester.fr. This is easily done with Pandoc and a small bash loop — and some manual adjusting later on 😛. 

for post in ~/src/github.com/vdemeester/blog/content/posts/*.md; do
    pandoc -f markdown -t org -o posts/$(basename -s .md ${post}).org ${post}
done

What is still to do after this initial take.

  • ☐ List FILETAGS for taximony
  • ☐ Maybe use css grid for the UI

Footnotes:

1

foo is bar, bar is baz