Practial Guides on jneidel

Guide to Modern Emacs Completion: vertico, corfu & friends

web@jneidel.com (Jonathan Neidel) — Wed, 04 Feb 2026 00:00:00 +0000

A good completion system can make navigating an app like walking over clouds. A terrible one is like walking barefoot over spiky stones. Every step of the way will hurt. Unfortunately Emacs’ defaults make for quite the thorny walk. Fortunately that is very fixable.

This guide will explain the necessary concepts and go through all of my configuration for the relevant packages of the “minad stack” (vertico, consult, corfu & friends) and some more. The setup is tailored toward modern sensibilities (fuzzy finding, LSP completions) and assumes you can tweak an elisp configuration to your needs.

Types of Emacs completion #

The most important thing to understand about auto-completion in Emacs is that there are two kinds:

Completing-read: this is your minibuffer (M-x), find-file, and all those other functions that prompt you at the bottom of the screen.
Completion-at-point (and completion-in-region): these complete text inside of a buffer (usually assigned to TAB.)

With that understood, suddenly those cryptic package headlines make more sense:

“🔍 consult.el - Consulting completing-read”
“🏝️ corfu.el - COmpletion in Region FUnction”

Completion styles #

completion-styles define the matching algorithms used. After years of fzf (ab)use I instinctively use spaces to separate queries for completing-read prompt (e.g. “a b”.) This is supported by orderless. Default styles plus orderless and I’m content to the point that I had not felt the need to fiddle with it.

Configuration code for completion styles + orderless

completion-styles checks for matches in order. So go from strictest to losest.

(use-package orderless
 :custom
 (orderless-matching-styles '(orderless-prefixes))
 (completion-ignore-case t)
 :config
 (set-face-attribute 'orderless-match-face-0 nil
 :foreground "#d70000")
 (set-face-attribute 'orderless-match-face-1 nil
 :foreground "#005fd7")
 (set-face-attribute 'orderless-match-face-2 nil
 :foreground "#007f3a")
 (set-face-attribute 'orderless-match-face-3 nil
 :foreground "#d700d7"))
 
(setq completion-styles '(basic substring initials orderless)
 completion-category-overrides '((file (styles basic partial-completion))))

Completing-read #

I consider the configuration of the minibuffer UI critical In my first 6 months of Emacs, the god-awful user experience of fido-mode and find-file really made me not want to use Emacs. I was used to insanely fast fuzzy finders everywhere. Emacs’ default completing-read felt like from a different century (which it is.)

Vertico + marginalia goodness.

Vertico is a great improvement. Marginalia adds context. Just that already makes for a much improved experience. Even if you don’t like the vertical list style, vertico got you covered (visual demo of the different styles by Karthink.)

Configuration code for vertico and marginalia

(use-package vertico
 :demand t
 :bind (:map minibuffer-local-map
 ("<tab>" . vertico-next)
 ("<backtab>" . vertico-previous))
 :custom
 (vertico-cycle t) ; C-n at the bottom of the list loops around to the top
 :config
 (vertico-mode t)
 (vertico-reverse-mode t)
 
 ; Verticos highlight font was clashing with my completion part, so adjust the font faces
 (set-face-attribute 'vertico-current nil
 :weight 'bold
 :inverse-video nil
 :foreground "#ffbf00"
 :background "grey20")
 (set-face-attribute 'completions-common-part nil
 :weight 'bold
 :foreground "#d70000")
 (set-face-attribute 'minibuffer-prompt nil
 :foreground "#ffbf00")
 )


(use-package marginalia
 :custom
 (marginalia-align 'right)
 :config
 (marginalia-mode t)

 (set-face-attribute 'marginalia-documentation nil
 :inherit nil
 :slant 'italic
 :foreground "grey43")
 (set-face-attribute 'marginalia-key nil
 :foreground "#00ff5f")
 )

For whatever reason my vertico has some indicators in the right fringe and this is the only way (I found) to get rid of them:

(defun jn/vertico-hide-fringe-and-truncation-indicators ()
 (when vertico-mode
 (setq-local fringe-indicator-alist
 (let ((a (copy-alist fringe-indicator-alist)))
 (setf (cdr (assq 'continuation a)) nil)
 (setf (cdr (assq 'truncation a)) nil)
 a))
 (setq-local truncation-string "")))
(add-hook 'minibuffer-setup-hook
 #'jn/vertico-hide-fringe-and-truncation-indicators)

Many functions will benefit from a nice completing-read UI setup.

Fuzzy file search #

For 98% of my file search I use a fuzzy finder. It gets me where I want to go quicker than traversing the directory tree. For this I use affe.¹

Affe search. Permissions, sizes and dates are courtesy of marginalia.

affe-find only searches the current directory (or project.) To use it for the whole system some customization is needed. There is a reason that affe-find from the home directory is not offered by the package author. Without some cleanup you will be overwhelmed by irrelevant matches. Though with ripgrep as our blazing fast file finder we can exclude matches like this:

pass a --glob flag to the command, with a pattern to ignore
create a .rgignore with a list of files in any directory root

With that, the number of matched files gets reduced to an amount where search feels instant.

I have these different bindings to invoke affe (ordered by frequency of use):

Search for a file on the system
Search for a file in the current project
Search for a directory on the system

Configuration code for affe

For one-off exclusions I use local .rgignore files (e.g. in ~ and ~/.config.) For file type or repeated directory exclusions I use ignore flags.

(defvar rg-ignore-flags
 "-g \"!*.mp3\" -g \"!*.jpg\" -g \"!*.JPG\" -g \"!*.jpeg\" -g \"!*.png\" \
 -g \"!*.mkv\" -g \"!*.mp4\" -g \"!*.avi\" -g \"!*.zip\" -g \"!*.ddl\" \
 -g \"!*.ods\" -g \"!*.xlsx\" -g \"!*.m3u\" -g \"!*.url\" -g \"!*.aac\" \
 -g \"!*.mpc\" -g \"!*.sql\" -g \"!*.ydb\" -g \"!dist/\" \
 -g \"!.git/\" -g \"!git/*\" -g \"!node_modules/\" -g \"!*cache/\" \
 -g \"!.cache\" -g \"!vendor/\" \
 -g \"!.pki/\" -g \"!.local/share/*/\" \
 -g \"!coverage\" -g \"!build/\" -g \"!var/\" -g \"!npm/\" \
 -g \"!Library/\" -g \"!.DS_Store\" -g \"!.stfolder\""
 "Exclusion flags for usage with ripgrep commands.")
(defvar rg-find-files-command
 (format "rg -L --ignore --hidden --files --color=never %s" rg-ignore-flags)
 "Command for finding files with ripgrep.")
(defvar rg-find-directories-command
 (format "rg-dir -L --ignore --hidden --color=never %s" rg-ignore-flags)
 "Command for finding directories with ripgrep.")

(defun affe-find-file (&optional dir) (interactive) ; default dir is cwd
 (setq affe-find-command rg-find-files-command)
 (affe-find dir))
(defun affe-find-directory (&optional dir) (interactive) ; default dir is cwd
 (setq affe-find-command rg-find-directories-command)
 (affe-find dir))
(defun affe-find-file-home () (interactive)
 (affe-find-file (substitute-in-file-name "$HOME")))
(defun affe-find-directory-home () (interactive)
 (affe-find-directory (substitute-in-file-name "$HOME")))

(use-package affe
 :bind (("H-f" . affe-find-file-home)
 ("H-M-f" . affe-find-file)
 ("H-s" . affe-find-directory-home))
 :custom
 (affe-count 5000))

If you want to have the H (Hyper) modifier, here is how I set it up on Linux.

Improved find-file #

I do use find-file to initiate tramp connections. find-files default TAB and RET behaviors leave a lot to be desired. Might be a skill issue, but I found myself always writing the path by hand for those deeply nested files.

vertico-directory gives me exactly the intuitive navigation I expect:

(use-package vertico-directory
 :after vertico
 :ensure nil
 :bind (:map vertico-map
 ("RET" . vertico-directory-enter)
 ("DEL" . vertico-directory-delete-char)
 ("M-DEL" . vertico-directory-delete-word))
 :hook (rfn-eshadow-update-overlay . vertico-directory-tidy))

Consult for better functions #

Consult offers a host of useful interactive functions built on completing-read. Many of them are superior replacements for built-in functions.

Fuzzy finder to select a line to jump to (with context preview.)

Configuration code for consult

(use-package consult
 :bind (;; new functionality
 ("M-g l" . consult-line)
 ("M-g m" . consult-mark)
 ("H-r" . consult-ripgrep)
 ;; replace built-in functions with superior versions
 ("M-y" . consult-yank-pop)
 ("C-x 4 b" . consult-buffer-other-window)
 ("C-x t b" . consult-buffer-other-tab)
 ([remap Info-search] . consult-info)
 ([remap list-buffers] . ibuffer)
 ))

If you want to have the H (Hyper) modifier, here is how I set it up on Linux.

The consult interface I rely on most heavily is consult-buffer. It can be modded to achieve this:

What are we looking at? We got multiple sections “Perspective Buffer”, “Special Buffer”, “File” and “Bookmark” off-screen. So, we get filtering by perspective workspace, access to recent files and bookmarks. Already great. The “Special Buffer” is where the real flexibility of this system comes in.

I use popper to relegate help buffers, compilation output and those sorts of buffers to a separate “popup stack”. Some get hidden completely and the rest don’t disturb my layout and are easily hidden. All well and good. A popup buffer, even a suppressed one, will still show up at the top of your buffer list, as the most recently accessed. Distracting clutter. Well, consult-buffer allows you to customize the sections. With this I segregated popup buffer off into their own section. Separate and not equal.

Configuration code for consult-buffer + popper + perspective

(setq jn/secondary-buffer-regexes ; alist in popper format
 '("\\*Messages\\*"
 ("\\*Warnings\\*" . hide)
 "\\*Man.*"
 "\\*Help.*"
 "\\*helpful.*"
 ("\\*Compile.*" . hide)
 ("\\*Org-Babel Error Output\\*" . hide)
 ("\\*Async Shell Command\\*" . hide)
 "\\*slime.*"
 ("\\*inferior-lisp\\*" . hide)
 ("\\*sldb.*" . hide)
 "\\*Backtrace\\*"
 "\\*org-search\\*"
 "\\* Merriam-Webster"
 "\\*Flymake diagnostics"))
 
(use-package popper
 :custom
 (popper-reference-buffers jn/secondary-buffer-regexes)
 (popper-mode-line '(:eval ""))
 (popper-window-height 12) ; default function maxes out at 33%, which seems too small to me
 :defer nil
 :config
 (popper-mode +1)
 
 (defhydra popper-hydra (:hint nil)
 "
_n_ext _p_revious promo_t_e/demo_t_e show/_u_nshow"
 ("n" popper-cycle)
 ("p" popper-cycle-backwards)
 ("t" popper-toggle-type)
 ("u" popper-toggle)
 ("q" nil nil)
 ("C-g" nil nil)
 ("ESC" nil nil))
 :bind (("H-u" . popper-toggle) ; quick show/hide popups
 ("H-U" . popper-hydra/body)))
 
(defun jn/strip-alists (buffers)
 (mapcar
 (lambda (item)
 (if (consp item)
 (car item)
 item))
 buffers))

(setq jn/secondary-buffer-pure-regexes (jn/strip-alists jn/secondary-buffer-regexes)) ; normalize into regexes

(use-package consult
 :custom
 (popper-group-function #'popper-group-by-perspective)
 :config
 (consult-customize consult-source-buffer :hidden t :default nil) ; disable
 (consult-customize persp-consult-source
 :name "Perspective Buffer"
 :default t
 :items '(lambda () (consult--buffer-query
 :sort 'visibility
 :as #'buffer-name
 :predicate (lambda (buf) (persp-is-current-buffer buf t))
 :exclude (append jn/secondary-buffer-pure-regexes '("^ \\*"))))) ; hidden buffers like minibuffer
 
 (defvar consult-special-buffer
 `( :name "Special Buffer"
 :default nil
 :face consult-buffer
 :category buffer
 :history buffer-name-history
 :state ,#'consult--buffer-state
 :items ,(lambda () (consult--buffer-query
 :sort 'visibility
 :as #'buffer-name
 :predicate (lambda (buf) (persp-is-current-buffer buf t))
 :include jn/secondary-buffer-pure-regexes
 ))))

 (add-to-list 'consult-buffer-sources consult-special-buffer)
 (add-to-list 'consult-buffer-sources persp-consult-source)
 
 (consult-customize consult-source-bookmark
 :enabled (lambda () (equal (persp-current-name) "main")))
 (consult-customize consult-source-recent-file
 :enabled (lambda () (equal (persp-current-name) "main")))
)

In your own scripts #

completing-read can be powerfully used in your own elisp scripts, as demonstrated by Álvaro Ramírez. I use it all the time. Here to pick a type from a pool of candidates:

(completing-read "What type? " '("Project" "Design" "Resource") nil t)

You can reach for gum if you want a similarly nice experience in your shell scripts.

In-buffer completion #

I use corfu as my completion-at-point UI. I would recommend it over company, because corfu uses the built-in completion API (company implements it’s own API incompatible to Emacs’.) Even if a you depend on a package that only provides a company-backend, corfu got you covered.

Since corfu enhances the built-in API, it will just work out of the box for modes that bring their own capfs (completion at point functions.) Like account completion in ledger-mode:

Completion at point with corfu. (Error overlay by flymake-hledger.)

Configuration code for corfu

;; TAB key: fix indentation if needed, otherwise perform completion
(setq tab-always-indent 'complete)

(use-package corfu
 :hook (after-init . global-corfu-mode)
 :custom
 (corfu-cycle t) ; cycle around to first entry after reaching the last
 (corfu-preview-current nil) ; don't expand text at point until I press return
 (corfu-min-width 20)
 (corfu-on-exact-match 'insert) ; complete if there is only a single candidate
 (corfu-quit-no-match t)
 (corfu-quit-at-boundary t)
 :config
 (setq corfu-popupinfo-delay '(1.25 . 0.5))
 (corfu-popupinfo-mode 1) ; shows documentation next to completions

 ;; sort by input history
 (with-eval-after-load 'savehist
 (corfu-history-mode 1)
 (add-to-list 'savehist-additional-variables 'corfu-history))
 )

Add more capfs #

With cape we can easily add further completion sources (completion at point functions) to be used by corfu. I use: words in the buffer and file system paths.

Configuration code for cape

(use-package cape
 :defer 1
 :config
 (add-hook 'completion-at-point-functions #'cape-dabbrev 20) ; words from buffer
 (add-hook 'completion-at-point-functions #'cape-file 20))

Code completion #

Thanks to language servers (see LSP explainer), nice auto-completion for code is now easy to achieve consistently.

Completion on a string in JavaScript.

The icons in the UI are added by kind-icon.

Configuration code for eglot and kind-icon

(use-package eglot
 :ensure nil
 :functions (eglot-ensure)
 :commands (eglot)
 :hook (prog-mode . eglot-ensure)
 :config
 (set-face-attribute 'eglot-highlight-symbol-face nil
 :foreground "#ffd700"
 :underline t)
 )

(use-package kind-icon
 :after corfu
 :config
 (add-to-list 'corfu-margin-formatters #'kind-icon-margin-formatter))

;; fix overly large icons (https://github.com/jdtsmith/kind-icon/issues/22)
(setq kind-icon-default-style
 '(:padding -1 :stroke 0 :margin 0 :radius 0 :height 0.4 :scale 1))

GitHub Copilot #

GitHub Copilot can be used as an evolved auto-completion. It not only completes the “word” at hand, but “finishes the whole thought.” Often it does what you want. Faster than you could ever type it.

If you got free access, it is worth to trying, to see if it fits your development style. That is if you are a are a student or your employer pays for it.

For me was not a good fit. I always turned it off and never turned it on again. I felt it constantly interrupted my thought. Imagine you get a proposal before you even had time to think about what the piece of code should do. You have no basis to evaluate it on, since you don’t know what you want. And at that point the proposal influences what you think is best. I much prefer the agentic coding loop: I think, write it down as specs/idea, pass it to the agent and only, have it work and only then see the generated code.

Conclusions #

Hopefully this guide gave you all the hints to successfully set up a feature-complete completions system that meets your needs. Most of the linked packages have great documentation, if you should run into problems.

For more from me, feel free to subscribe to one of the RSS feeds.
For corrections or suggestions on this article, please email me🙂

Have a great day!

References #

This is an entry for the Emacs Carnival. See other contributions on the topic of completion.

My complete literate config
Protesilaos literate config (completions section)

I tried fzf.el, but it does not play along well with the rest of Emacs. Affe on the other hand integrates seamlessly. ↩︎

Window management on MacOS with AeroSpace

web@jneidel.com (Jonathan Neidel) — Thu, 15 Jan 2026 00:00:00 +0000

On Linux I’ve had a nice way to manage my windows for a long time now. On my work computer, a Macbook, I was using the inadequate built-in “Spaces” feature. An unsatisfying experience for to a lack of consistency and sane keyboard controls. This is where AeroSpace comes in, providing a nicer window management experience than what Apple offers.

Before we setup the app, let’s take a deeper look at the problem it solves first.

What is window management? #

On a computer there are multiple application windows open at the same time. In my case, at a minimum I always have a browser, Emacs, terminal and signal messaging client open. Window management is what you do when you select an application window, move another out of the way or arrange multiple one the screen together. This can be done more or less effectively.

While everyone know the application window, the concept of the workspace is both unfamiliar and underutilized by most people. Instead of cramming every open window into one workspace, layering them on top of each other or constantly minimizing them, one can split them up among many workspaces.

For example, the below shows that I have two programs open in fullscreen, at the same time, each on a separate workspaces.

Workspace 2: the browser

Workspace 5: the mail client

Instead of switching between programs, I would switch between workspaces. The advantage is that workspaces can contain arrangements of windows that can be switched between just as easily. You can also think of workspaces in terms of function (browser) instead of program (chromium, firefox, safari, etc.)

(Workspaces do not require AeroSpace and can also be utilized with the native MacOS Spaces.)

Why bother with this? #

Using the mouse is a user friendly and easily understandable way of selecting a window, moving it somewhere or making it bigger. But it is not very efficient, automatable or reproducible.

Using a keyboard driven approach has all of those advantages after you got used to the few basic keybindings. It’s more efficient because you can perform one or many keypresses much faster, your hand does not need to leave the keyboard. It’s automatable because you don’t need to think. You get to your browser on muscle-memory alone. (AeroSpace additionally provides rules for automation.) You get a sane reproducible window management because everything is always where you expect it.

Window management is simply a part of every computer users life. Might as well make it as pleasant and fluid as possible, since you use it literally all the time.

To see every app in fullscreen eliminates distractions. There is nothing else on the screen. The same is true for the consistency and muscle-memory. I don’t have to think about the computer interface and can fully focus on what I’m doing.

AeroSpace #

On Linux there exists the distinction between Windows and MacOS style mouse-driven “Desktop Environments” and keyboard-driven “Window Managers”. Usage of one of the many available window managers makes for a stellar native experience. On MacOS using “advanced” keyboard-driven window management was not intended by Apple and is provided by one of multiple third-party packages that hacked it in. This carries some unfortunate and unavoidable negative performance implications.

I chose AeroSpace as my window management solution. It fulfills my needs and provided me with a close equivalent to what is used on my Linux machine (i3.)

Why not use MacOS Spaces?

It’s what I used prior. While the experience is more polished, there are several flaws which make their usage frustrating. Among their several shortcomings are:

Fixed keybindings and the defaults use the unergonomic arrow keys.
Workspaces are inconsistent. Fullscreening an app messes up the order. Workspace two will not always contain the same thing.
Lack of features like splits, stack, move to workspace, all readily accessible via keybinds.
Switching comes with an unavoidable animation. I want instant switching and that is just not possible.
Lack of automation as described above.

Demo #

Here is my 2nd workspace, which is dedicated to the browser.

I like only having one app (and window) open per workspace. It’s the least distracting arrangement and any app I need is only a short keypress away.

For the rare case that I need it, split screen windows are of course available. The window resizing happens automatically, as soon as I open a new browser window. (Vertical/horizontal splits can be toggled with option+e.)

If we still want to keep both windows in the same workspace and have them be in fullscreen, we can switch to the stacked (option+s) layout.

(Return back to the split layout anytime with option+e.)

In workspace 5 I have my mail. To switch to over I simply press option+5.

Let’s say I want to view my email program and browser side by side. I can easily move the mails over to workspace 2 with option+shift+2 and then switch to that workspace with option+2.

To switch from one window to the other I use option+j and I can change the order with option+shift+j. (j/k are up down, h/l are left right. If unintuitive, this is easily configurable.)

To resize use option+r.

All of these keybindings are arbitrary and just what I use. They can be changed to your liking. You could use another prefix (instead of option.) You could use mnemonic workspace keys instead of 1..10, like b=browser, t=terminal, m=mail, e=Emacs, etc. Choose what fits you.

Getting started #

Install the app as per official install instructions. If you have brew, do:

brew install --cask nikitabobko/tap/aerospace

Then put the config below into ~/.aerospace.toml or ~/.config/AeroSpace/aerospace.toml.

If you don’t know how:

Download this file as aerospace.toml.
Open the Finder in the Downloads directory.
Press command+shift+. to show hidden files.
Rename the file to .aerospace.toml (i.e. add a leading dot “.”) and confirm.
Move the file .aerospace.toml into your home/user directory, in my case “jneidel”.

# See help and documentation at: https://nikitabobko.github.io/AeroSpace/guide
# Note that the modifier "alt" is invoked by the "option" key.

### Configuration
start-at-login = true

# Mouse follows focus when focused monitor changes
on-focused-monitor-changed = ['move-mouse monitor-lazy-center']

[mode.main.binding]
# change focus: option + h/j/k/l
alt-h = 'focus --boundaries-action wrap-around-the-workspace left'
alt-j = 'focus --boundaries-action wrap-around-the-workspace down'
alt-k = 'focus --boundaries-action wrap-around-the-workspace up'
alt-l = 'focus --boundaries-action wrap-around-the-workspace right'

# move window: option + H/J/K/L
alt-shift-h = 'move left'
alt-shift-j = 'move down'
alt-shift-k = 'move up'
alt-shift-l = 'move right'

# change layout
alt-s = 'layout v_accordion' # option + s = vertical tabs
alt-w = 'layout h_accordion' # option + w = horizontal tabs
alt-e = 'layout tiles horizontal vertical' # option + e = change between vertical and horizontal splits

alt-f = 'fullscreen' # option + f to toggle fullscreen
alt-space = 'layout floating tiling' # toggle between floating and regular window

# go to workspace
alt-1 = 'workspace 1'
alt-2 = 'workspace 2'
alt-3 = 'workspace 3'
alt-4 = 'workspace 4'
alt-5 = 'workspace 5'
alt-6 = 'workspace 6'
alt-7 = 'workspace 7'
alt-8 = 'workspace 8'
alt-9 = 'workspace 9'
alt-0 = 'workspace 10'

# send active window to to workspace
alt-shift-1 = 'move-node-to-workspace 1'
alt-shift-2 = 'move-node-to-workspace 2'
alt-shift-3 = 'move-node-to-workspace 3'
alt-shift-4 = 'move-node-to-workspace 4'
alt-shift-5 = 'move-node-to-workspace 5'
alt-shift-6 = 'move-node-to-workspace 6'
alt-shift-7 = 'move-node-to-workspace 7'
alt-shift-8 = 'move-node-to-workspace 8'
alt-shift-9 = 'move-node-to-workspace 9'
alt-shift-0 = 'move-node-to-workspace 10'

# interactively resize split
alt-r = 'mode resize'
[mode.resize.binding]
h = 'resize width -50'
j = 'resize height +50'
k = 'resize height -50'
l = 'resize width +50'
enter = 'mode main'
esc = 'mode main'
q = 'mode main'


# Move app windows to specific workspaces.
# My workspace assignment is the same as on Linux:
# Workspace 1: Emacs
# Workspace 2: Browser
# Workspace 3: Terminal
# Workspace 4: Communication
# Workspace 5: Misc

# You can find out the app id in Karabiner EventViewer or if the app
# is running using: $ aerospace list-apps

# uncomment to activate
# [[on-window-detected]]
# if.app-id = 'org.gnu.Emacs'
# run = ['move-node-to-workspace 1']

# [[on-window-detected]]
# if.app-id = 'org.chromium.Chromium'
# run = ['move-node-to-workspace 2']

# [[on-window-detected]]
# if.app-id = 'net.kovidgoyal.kitty'
# run = ['move-node-to-workspace 3']

# [[on-window-detected]]
# if.app-id = 'com.microsoft.teams2'
# run = ['move-node-to-workspace 4']

# [[on-window-detected]]
# if.app-id = 'com.apple.mail'
# run = ['move-node-to-workspace 5']

Then start the app. Try a keybinding like option+2 to confirm that it works.

You can make changes to the configuration file with your editior of choice and load those changes through the AeroSpace icon on the task bar (the icon is a number of the current workspace.)

Caveats #

I already mentioned the performance implication inherit to MacOS’s closed design. AeroSpace is way slower than it’s Linux counterpart. This is annoying, but still much better than not using AeroSpace.

Sometimes you will displace a window. You misclicked with the move command, your automatic rules moved the new window to an unexpected workspace or you it is hidden in a stack somewhere. You will find that with practice it is easier to locate these missing windows. Usually what happened was quite logical. My recommendation would be to stay calm and do not fall back on the tools MacOS provides. Mechanisms like the window selector (four fingers pinching inward) will not work well and are ultimately an unnecessary crutch.

It is recommended that you back up the configuration file, so you don’t lose them when you migrate systems or lose the computer. I do that through my dotfiles.

The above config uses option as it’s prefix key. That is AeroSpaces recommendation and probably fits for most people. Personally I use the command key.

Why do I use command my prefix key?

I would not recommend this to non-hackers or hackers without specific requirements.

Command on Mac is used all over the place. Mapping AeroSpace stuff to command could compromise keyboard shortcuts across the OS or inside applications. Doing this will require significant tinkering.

What makes it worth the effort to me is:

Super is my i3 window management prefix under Linux. Super and command are the same key on my external split keyboard. By using the exact same keybindings between MacOS and Linux I reduce the cognitive load of switching between them. (I also remap cmd+c to ctrl+c for the same reason.)
I want my window management to use a single key as my prefix. AeroSpace has a limited amount of prefix keys available: command, option, control, shift. Command is used by MacOS and various apps. Option and control are filled by Emacs binding (not a problem for most people.) Shift can’t be used by itself. Other keys like fn or hyper are not available as prefixes.

My setup to use command as my prefix key

I use Karabiner-Elements to translate cmd+1 into cmd+ctrl+alt+1. My AeroSpace configuration then binds “switch to workspace 1” to cmd+ctrl+alt+1. In the same way I translate alt+1 to cmd+1 to make the ability to navigate to the first browser tab (cmd+1) and others available.

Here is my Karabiner config for translating command keybindings:

{
 "description": "Map aerospace bindings to cmd",
 "manipulators": [
 {
 "from": {
 "key_code": "1",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "1",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "1",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "1",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "2",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "2",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "2",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "2",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "3",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "3",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "3",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "3",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "4",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "4",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "4",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "4",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "5",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "5",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "5",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "5",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "6",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "6",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "6",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "6",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "7",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "7",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "7",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "7",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "8",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "8",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "8",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "8",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "9",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "9",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "9",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "9",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "0",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "0",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "0",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "0",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "p",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "3",
 "modifiers": ["left_command", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "p",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "4",
 "modifiers": ["left_command", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "p",
 "modifiers": { "mandatory": ["command", "control"] }
 },
 "to": [
 {
 "key_code": "5",
 "modifiers": ["left_command", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "h",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "h",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "j",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "j",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "k",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "k",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "l",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "l",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "h",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "h",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "j",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "j",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "k",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "k",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "l",
 "modifiers": { "mandatory": ["command", "shift"] }
 },
 "to": [
 {
 "key_code": "l",
 "modifiers": ["left_command", "left_option", "left_control", "left_shift"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "f",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "f",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "s",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "s",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "w",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "w",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "e",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "e",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 },
 {
 "from": {
 "key_code": "r",
 "modifiers": { "mandatory": ["command"] }
 },
 "to": [
 {
 "key_code": "r",
 "modifiers": ["left_command", "left_option", "left_control"]
 }
 ],
 "type": "basic"
 }
 ]
}

You probably also want to take a look at the Karabiner config I use for translating copy, paste, new tab, etc. to their Linux control/alt counterpart. See that article.

And here is the AeroSpace config I actually use:

# i3-like window management for MacOS.
# Source: https://nikitabobko.github.io/AeroSpace/goodies#i3-like-config

### On the prefix key.

# On Linux I use the super key as a prefix.
# Unlike control and alt/option it is unused by applications.
# MacOS makes heavy use of command (super), so there is no easy empty prefix key available.
# Since I want one a single prefix key, that can add shift as a modifier, I used Karabiner Elements
# to put MacOS's cmd+... stuff on control and alt, replicating the Linux commands and freeing the cmd
# to be used as the window management prefix.
# In my setup karabiner expands a cmd keypress to "cmd-ctrl-alt", which stuff is mapped to here.
# You may forgo the Karabiner translation layer and use another key combination as you prefix.


### Configuration
start-at-login = true

# Mouse follows focus when focused monitor changes
on-focused-monitor-changed = ['move-mouse monitor-lazy-center']

[mode.main.binding]
# change focus: cmd + h/j/k/l
cmd-ctrl-alt-h = 'focus --boundaries-action wrap-around-the-workspace left'
cmd-ctrl-alt-j = 'focus --boundaries-action wrap-around-the-workspace down'
cmd-ctrl-alt-k = 'focus --boundaries-action wrap-around-the-workspace up'
cmd-ctrl-alt-l = 'focus --boundaries-action wrap-around-the-workspace right'

# move window: cmd + H/J/K/L
cmd-ctrl-alt-shift-h = 'move left'
cmd-ctrl-alt-shift-j = 'move down'
cmd-ctrl-alt-shift-k = 'move up'
cmd-ctrl-alt-shift-l = 'move right'

# change layout
cmd-ctrl-alt-s = 'layout v_accordion' # cmd + s = vertical tabs
cmd-ctrl-alt-w = 'layout h_accordion' # cmd + w = horizontal tabs
cmd-ctrl-alt-e = 'layout tiles horizontal vertical' # cmd + e = change between vertical and horizontal splits

cmd-ctrl-alt-f = 'fullscreen' # cmd + f to toggle fullscreen
# alt-shift-space = 'layout floating tiling' # 'floating toggle' in i3

cmd-ctrl-alt-1 = 'workspace 1'
cmd-ctrl-alt-2 = 'workspace 2'
cmd-ctrl-alt-3 = 'workspace 3'
cmd-ctrl-alt-4 = 'workspace 4'
cmd-ctrl-alt-5 = 'workspace 5'
cmd-ctrl-alt-6 = 'workspace 6'
cmd-ctrl-alt-7 = 'workspace 7'
cmd-ctrl-alt-8 = 'workspace 8'
cmd-ctrl-alt-9 = 'workspace 9'
cmd-ctrl-alt-0 = 'workspace 10'

cmd-ctrl-alt-shift-1 = 'move-node-to-workspace 1'
cmd-ctrl-alt-shift-2 = 'move-node-to-workspace 2'
cmd-ctrl-alt-shift-3 = 'move-node-to-workspace 3'
cmd-ctrl-alt-shift-4 = 'move-node-to-workspace 4'
cmd-ctrl-alt-shift-5 = 'move-node-to-workspace 5'
cmd-ctrl-alt-shift-6 = 'move-node-to-workspace 6'
cmd-ctrl-alt-shift-7 = 'move-node-to-workspace 7'
cmd-ctrl-alt-shift-8 = 'move-node-to-workspace 8'
cmd-ctrl-alt-shift-9 = 'move-node-to-workspace 9'
cmd-ctrl-alt-shift-0 = 'move-node-to-workspace 10'

# interactively resize split
cmd-ctrl-alt-r = 'mode resize'
[mode.resize.binding]
h = 'resize width -50'
j = 'resize height +50'
k = 'resize height -50'
l = 'resize width +50'
enter = 'mode main'
esc = 'mode main'
q = 'mode main'

# Move app windows to specific workspaces.
# The workspace assignment is the same as on Linux:
# Workspace 1: Emacs
# Workspace 2: Browser
# Workspace 3: Terminal
# Workspace 4: Communication
# Workspace 5: Misc

# You can find out the app id in Karabiner EventViewer or if the app
# is running using: $ aerospace list-apps
[[on-window-detected]]
if.app-id = 'org.gnu.Emacs'
run = ['move-node-to-workspace 1']

[[on-window-detected]]
if.app-id = 'org.chromium.Chromium'
run = ['move-node-to-workspace 2']

[[on-window-detected]]
if.app-id = 'net.kovidgoyal.kitty'
run = ['move-node-to-workspace 3']

[[on-window-detected]]
if.app-id = 'com.microsoft.teams2'
run = ['move-node-to-workspace 4']

[[on-window-detected]]
if.app-id = 'com.apple.mail'
run = ['move-node-to-workspace 5']

Conclusion #

I showed that what keyboard-driven window management can do for you and why it’s well worth learning and using. Using the provided minimal config you can get started right away.

Complete Email Setup with isync, notmuch and friends

web@jneidel.com (Jonathan Neidel) — Mon, 03 Nov 2025 00:00:00 +0000

This is a complete guide to my email setup. I wanted to provide and end-to-end experience showing everything and every step involved. That means the configuration at my hosting and DNS providers, mail sending, syncing and indexing tools and the user interface. This is not for a single mailbox, but sending and receiving from any address on multiple domains.

All of my tool/provider choices in this guide can be replaced once you understand the role that particular part fulfills.

Hosting providers #

These providers make sure mail is collectable from a mailbox and that it can be sent.

Domains #

I use namecheap. Good prices and easy enough user interface.

I have the following domains with them:

jneidel.com
jneidel.de
neidel.xyz

Mailserver #

I use uberspace. They got great docs and a flexible pricing model that, though a model of solidary, provides hosting even to those who would be otherwise unable to afford it. I have been with them for many years.

With the command uberspace mail domain add neidel.xyz I can add a mail domain:

The mailservers configuration has been adapted.
Now you can use the following records for your DNS:

neidel.xyz. IN MX 0 pizarro.uberspace.de.
neidel.xyz. IN TXT "v=spf1 include:spf.uberspace.de ~all"
uberspace._domainkey.neidel.xyz. IN TXT "v=DKIM1;t=s;n=core;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAonmVoIli72aSHhW9LyMrnrkcYfCddeqHyQkK8aIkpBDhJObkbX+9b+p+V/+dJz86IUd3zi1Flj41Wt3TqS8/WQg2NVUKsAJMUSrO00gz+3Gqw0e0cT0BJ4YXSFr8kzwro79A9rh6ZJta6/E4xg+7DgQcIFh8Qy1XcLV2wPimQDQRWAAFMO5nXqQea9Vuvid2n2SoCTE9BXhNg4TwDRiZZTxSzTigU7iDSTBuS75wH0XtOBIMpc0OeK+2Jz3E/33/0V28rCGcPLX2FPSXvfDPY/9VmQzc0jDW8mz4SJ4/bS6Gj8qWtVlVXEEVZ3Fa2aWQ1W8vJrAelo/31dq9kcd+aQIDAQAB"

The output tell me what to add in namecheap in the domain under “Advanced DNS”:

This configuration includes DKIM signatures and SPF headers. Without a valid DKIM signature emails sent to e.g. Gmail addresses will be blocked. The validity of these records are tested later.

Mailboxes #

To gain access to mail delivered to a configured domain we need to create a mailbox:

uberspace mail user add jneidel

I want mail sent to any address of my domains to go into the same mailbox/maildir, so I set it up as a catch-all:

uberspace mail catchall set jneidel

(See the uberspace docs.)

Mail utilities #

These tools are necessary to secure, retrieve, send and index mail.

GPG #

To securely store my mail password I use pass, which encrypts the passwords with my GPG key.

You can create your own key, ideally with a secure password generated with the dice method:

gpg --full-generate-key

Install the relevant programs for the usage:

yay -S pass pinentry

Add the mail server password to pass:

pass add jneidel@jneidel.de

Configure your favorite pin entry program in ~/.config/gnupg/gpg-agent.conf:

pinentry-program /bin/pinentry-gtk
default-cache-ttl 43200 # 12h, refreshes with each use of the key
max-cache-ttl 259200  # 3d

Do restart the gpg-agent after changes (find the command in htop, kill the process and execute the command again.)

(Here is some more information about pin entry.)

Retrival and sync #

We need to configure a program to download email from the IMAP server for offline use. This program will also sync any changes we do back to the server.

I use mbsync, which is part of isync.

yay -S isync

In ~/.config/isyncrc I have:

IMAPStore jneidel@jneidel.de-remote
Host pizarro.uberspace.de
Port 993
User jneidel@jneidel.de
PassCmd "pass jneidel@jneidel.de"
AuthMechs LOGIN
TLSType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt

MaildirStore jneidel@jneidel.de-local
Subfolders Verbatim
Path /home/jneidel/.local/share/mail/jneidel@jneidel.de/
Inbox /home/jneidel/.local/share/mail/jneidel@jneidel.de/INBOX

Channel jneidel@jneidel.de
Expunge Both
Far :jneidel@jneidel.de-remote:
Near :jneidel@jneidel.de-local:
Patterns * !"*virtual*"
Create Both
SyncState *
MaxMessages 0
ExpireUnread no

The host is the one from the domain setup.
The port value can be found in the uberspace docs.
The password was stored in pass in the previous step.
I designated ~/.local/share/mail as my local maildir.

I use ~/scripts/mail/mailsync invoking mbsync to synchronize my mail:

#!/bin/sh

MAILSYNC_MUTE=true

if [ "$1" = "--help" ] || [ "$1" = "-h" ] || [ "$1" = "help" ]; then
 cat <<EOF
$ mailsync [ACCOUNT]
Sync mail and index it.

Run it:
- via goimapnotify (server has received new mail hook)
- via cron (timer)

Source: https://github.com/LukeSmithxyz/mutt-wizard
Changes: pop removed, notifications disabled, .config/neomvim and .config/isync as config dirs.

Parameters:
 [\$1]: account to sync, by default syncs all

Example:
 $ mailsync
 $ mailsync jneidel@jneidel.com
EOF
 exit
fi

hash mbsync notmuch || exit 127

# There are many arbitrary and ugly features in this script because it is
# inherently difficult to pass environmental variables to cronjobs and other
# issues. It also should at least be compatible with Linux (and maybe BSD) with
# Xorg and MacOS as well.

# Run only if user logged in (prevent cron errors)
pgrep -u "${USER:=$LOGNAME}" >/dev/null || { echo "$USER not logged in; sync will not run."; exit ;}
# Run only if not already running in other instance
pgrep mbsync >/dev/null && { echo "mbsync is already running."; exit ;}

# First, we have to get the right variables for the mbsync file, the pass
# archive, notmuch and the GPG home. This is done by searching common profile
# files for variable assignments. This is ugly, but there are few options that
# will work on the maximum number of machines.
eval "$(grep -h -- \
 "^\s*\(export \)\?\(MBSYNCRC\|PASSWORD_STORE_DIR\|NOTMUCH_CONFIG\|GNUPGHOME\)=" \
 "$HOME/.profile" "$HOME/.bash_profile" "$HOME/.zprofile" "$HOME/.config/zsh/.zprofile" "$HOME/.zshenv" \
 "$HOME/.config/zsh/.zshenv" "$HOME/.bashrc" "$HOME/.zshrc" "$HOME/.config/zsh/.zshrc" \
 "$HOME/.pam_environment" 2>/dev/null)"

# export GPG_TTY="$(tty)"

[ -n "$MBSYNCRC" ] && alias mbsync="mbsync -c $MBSYNCRC" || MBSYNCRC=$HOME/.config/isyncrc

lastrun="${XDG_CONFIG_HOME:-$HOME/.config}/neomutt/.mailsynclastrun"

# Settings are different for MacOS (Darwin) systems.
case "$(uname)" in
 Darwin)
 notify() { osascript -e "display notification \"$2 in $1\" with title \"You've got Mail\" subtitle \"Account: $account\"" && sleep 2 ;}
 ;;
 *)
 case "$(readlink -f /sbin/init)" in
 *systemd*|*openrc*) export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus ;;
 esac
 # remember if a display server is running since `ps` doesn't always contain a display
 pgrepoutput="$(pgrep -a X\(org\|wayland\))"
 displays="$(echo "$pgrepoutput" | grep -wo "[0-9]*:[0-9]\+" | sort -u)"
 notify() { [ -n "$pgrepoutput" ] && for x in ${displays:-0:}; do
 export DISPLAY=$x
 notify-send --app-name="mutt-wizard" "New mail!" "📬 $2 new mail(s) in \`$1\` account."
 done ;}
 ;;
esac

# Check account for new mail. Notify if there is new content.
syncandnotify() {
 acc="$(echo "$account" | sed "s/.*\///")"
 if [ -z "$opts" ]; then mbsync "$acc"; else mbsync "$opts" "$acc"; fi
 new=$(find\
 "$HOME/.local/share/mail/$acc/INBOX/new/"\
 "$HOME/.local/share/mail/$acc/Inbox/new/"\
 "$HOME/.local/share/mail/$acc/inbox/new/"\
 "$HOME/.local/share/mail/$acc/INBOX/cur/"\
 "$HOME/.local/share/mail/$acc/Inbox/cur/"\
 "$HOME/.local/share/mail/$acc/inbox/cur/"\
 -type f -newer "$lastrun" 2> /dev/null)
 newcount=$(echo "$new" | sed '/^\s*$/d' | wc -l)
 case 1 in
 $((newcount > 0)) ) [ -z "$MAILSYNC_MUTE" ] && notify "$acc" "$newcount" ;;
 esac
}

# Sync accounts passed as argument or all.
if [ "$#" -eq "0" ]; then
 accounts="$(awk '/^Channel/ {print $2}' "$MBSYNCRC")"
else
 for arg in "$@"; do
 [ "${arg%${arg#?}}" = '-' ] && opts="${opts:+${opts} }${arg}" && shift 1
 done
 accounts=$*
fi

# Parallelize multiple accounts
for account in $accounts; do
 syncandnotify &
done

wait

notmuch new
afew --tag --new
afew --move-mails --all

# Create a touch file that indicates the time of the last run of mailsync
touch "$lastrun"

This script could be simpler, but for now it just works. It was adapted from mutt-wizard.

The script could be invoke at a regular interval through cron, but we can do it even better than that.

Sync on IMAP change #

Why not let the IMAP server inform us of new mail and only sync then? goimapnotify does exactly that.

yay -S goimapnotify

In ~/.config/imapnotify/jneidel@jneidel.de we tell it what to do:

{
"host": "pizarro.uberspace.de",
"port": 993,
"tls": true,
"tlsOptions": {
"rejectUnauthorized": false
},
"username": "jneidel@jneidel.de",
"password": "",
"passwordCmd": "pass jneidel@jneidel.de",
"onNewMail": "/home/jneidel/scripts/mail/mailsync",
"onNewMailPost": "",
"boxes": [ "INBOX" ]
}

To activate run this on startup (I have it in my ~/.xinitrc):

goimapnotify -conf ~/.config/imapnotify/jneidel.de.yaml >/dev/null 2>&1 &

Sending mail #

For sending mail (SMTP) I use msmtp:

yay -S msmtp

It is configured in ~/.config/msmtp/config:

account jneidel@jneidel.de
host pizarro.uberspace.de
port 465
from jneidel@jneidel.de
user jneidel@jneidel.de
passwordeval "pass jneidel@jneidel.de"
auth on
tls on
tls_trust_file	/etc/ssl/certs/ca-certificates.crt
logfile /home/jneidel/.local/share/msmtp/jneidel.de.log
tls_starttls off

account default : jneidel@jneidel.de

The account default acts like a catch-all for sending, so I don’t have to configure each sender address individually. This way I can respond to a mail I received on “abc@jneidel.com” from “abc@jneidel.com”, not just from “jneidel@jneidel.de”.

The SMTP port was also found in the docs.

Mail indexing #

I use notmuch to index my email. It makes it very quick to search. Since I use notmuch as my mail client this is required for me, but optional otherwise.

yay -S notmuch

Configuration is handled in this redefined config location: export NOTMUCH_CONFIG=~/.config/notmuch-config:

[database]
path=/home/jneidel/.local/share/mail
[user]
name=Jonathan Neidel
primary_email=jneidel@jneidel.de
other_email=jneidel@jneidel.com;j@neidel.xyz
[new]
tags=unread;inbox;
ignore=.mbsyncstate;.uidvalidity
[search]
exclude_tags=deleted;spam;
[maildir]
synchronize_flags=true
[crypto]
gpg_path=gpg

Notmuch does not provide tagging automation or mail moving. For that I use afew.

yay -S afew

Configured in ~/.config/afew/config:

[Filter.1]
message = Remove new tag everywhere but INBOX folder
query = NOT folder:jneidel@jneidel.de/INBOX
tags = -unread;-inbox

[Filter.2]
message = Hide jobdirecto mails
query = folder:jneidel@jneidel.de/INBOX AND to:wilfredocasas100@gmail.com
tags = -unread;+spam;-inbox

[Filter.3]
message = Remove Lidl Pay confirmations
query = subject:Abbuchungsankündigung AND from:lidlpay@lidl.de
tags = +trash;-unread;-inbox

[Filter.4]
message = Hide bellicon
query = from:affiliate@bellicon.com
tags = +spam;-unread;-inbox

[Filter.5]
message = Remove IB trade confirmations
query = subject:Trade Confirmation Flex AND from:donotreply@interactivebrokers.com
tags = +trash;-unread;-inbox

[Filter.6]
message = Move Telekom invoices
query = subject:'Telekom Festnetz-Rechnung' AND from:rechnungonline@telekom.de
tags = +invoice;-unread;-inbox

[Filter.7]
message = Blutspende
query = to:spenderservice@jneidel.com
tags = +trash;-unread;-inbox

[Filter.8]
message = Tag mail in sent dir as sent
query = folder:jneidel@jneidel.com/Sent OR folder:jneidel@jneidel.de/Sent
tags = +sent;-unread;-inbox

[Filter.9]
message = Tag mail in archive dir as archieved
query = folder:jneidel@jneidel.com/archive OR folder:jneidel@jneidel.de/archive
tags = +archived;-unread;-inbox

[Filter.10]
message = Tag mail in invoices dir as invoice
query = folder:jneidel@jneidel.com/invoices OR folder:jneidel@jneidel.de/invoices
tags = +invoice;-unread;-inbox

[ArchiveSentMailsFilter]
sent_tag = sent

[MailMover]
folders = jneidel@jneidel.de/INBOX jneidel@jneidel.de/archive jneidel@jneidel.de/spam jneidel@jneidel.de/invoices
rename = True

jneidel@jneidel.de/INBOX = 'tag:spam':jneidel@jneidel.de/spam 'tag:trash':jneidel@jneidel.de/Trash 'tag:invoice':jneidel@jneidel.de/invoices 'tag:archived':jneidel@jneidel.de/archive
jneidel@jneidel.de/spam = 'tag:trash':jneidel@jneidel.de/Trash
jneidel@jneidel.de/archive = 'tag:trash':jneidel@jneidel.de/Trash
jneidel@jneidel.de/invoices = 'tag:trash':jneidel@jneidel.de/Trash

# Test queries with notmuch search

The Filters at the top are used to get mail that I don’t want to see out of my inbox. Recurring invoices (Telekom) and unsubscribable spam (jobdirecto, bellicon) also get tagged respectively.

I use the tags “spam, trash, archived, invoice” and these correspond to directories in my maildir: “spam, Trash, archive, invoices”. The MailMover at the bottom is responsible for moving mail according to the tags.

Mail client #

I use the notmuch Emacs client. The Emacs package was already installed alongside the system package:

(use-package notmuch
 :ensure nil
 :defer t
 :commands (notmuch notmuch-mua-new-mail))

In the previous section I already configured notmuch and autotagging with afew. That means all that is left to configure is the Emacs interface and a little wiring for persona and msmtp:

(setq user-mail-address "jneidel@jneidel.com"
 user-full-name "Jonathan Neidel"
 notmuch-identities '("Jonathan Neidel <jneidel@jneidel.com>"))

(setq send-mail-function 'sendmail-send-it
 message-send-mail-function 'sendmail-send-it
 sendmail-program "/bin/msmtp"

 mail-specify-envelope-from t
 message-sendmail-envelope-from 'header
 mail-envelope-from 'header)

(setq notmuch-fcc-dirs "jneidel@jneidel.de/Sent"
 notmuch-draft-folder "jneidel@jneidel.de/Drafts")
;; paths are relative to notmuch index base root aka the maildir at ~/.local/share/mail

I have replaced the default notmuch hello screen with a collection of queries easily accessible through keybindings. A lot of my this notmuch config this was inspired by Prots Emacs configuration.

The notmuch-hello screen after the below configuration

(use-package notmuch
 :defer t
 :config
 (setq notmuch-show-logo nil
 notmuch-column-control 1.0
 notmuch-hello-auto-refresh t
 notmuch-hello-thousands-separator ""
 notmuch-hello-sections '(notmuch-hello-insert-saved-searches)
 notmuch-search-oldest-first nil)

 (setq notmuch-show-empty-saved-searches t)
 (setq notmuch-saved-searches
 `(( :name "📥 inbox"
 :query "tag:inbox"
 :sort-order newest-first
 :key ,(kbd "i"))
 ( :name "👀 all unread (inbox)"
 :query "tag:unread and tag:inbox"
 :sort-order newest-first
 :key ,(kbd "u"))
 ( :name "✏️ drafts"
 :query "tag:draft and not:tag:trash"
 :sort-order newest-first
 :key ,(kbd "d"))
 ( :name "🗃️ archive"
 :query "tag:archived"
 :sort-order newest-first
 :key ,(kbd "a"))
 ( :name "🛒 invoices"
 :query "tag:invoice"
 :sort-order newest-first
 :key ,(kbd "n"))
 ( :name "📨 sent"
 :query "tag:sent"
 :sort-order newest-first
 :key ,(kbd "s"))
 ( :name "🪓 spam"
 :query "tag:spam and not:tag:trash"
 :sort-order newest-first
 :key ,(kbd "m"))
 ( :name "🗑️ trash"
 :query "tag:trash"
 :sort-order newest-first
 :key ,(kbd "t"))
 ( :name "🌍 all"
 :query "not:tag:trash and not:tag:spam"
 :sort-order newest-first
 :key ,(kbd "A"))
 ))

 (keymap-set notmuch-common-keymap (kbd "g") #'notmuch-jump-search)
 )

The interface shows each emails tags. To make these less verbose and easy to parse I turn them into emoji. This snippet configures everthing around tags.

(use-package notmuch
 :defer t
 :config
 (setq notmuch-archive-tags '("+archived")
 notmuch-message-replied-tags '("+replied")
 notmuch-message-forwarded-tags '("+forwarded")
 notmuch-show-mark-read-tags '("-unread")
 notmuch-draft-tags '("+draft")
 notmuch-draft-save-plaintext 'ask)

 (setq notmuch-tag-formats
 '(("unread" "👀")
 ("attachment" "📎")
 ("signed" "🔑")
 ("encrypted" "🔒")
 ("replied" "💬")
 ("forwarded" "⏩")
 ("sent" "📨")
 ("inbox" "📥")
 ("spam" "🪓")
 ("invoice" "🛒")
 ("archived" "🗃️")
 ("draft" "✏️")
 ("trash" "🗑️")
 ("lists" "📜lists")
 ("flag" (propertize tag 'face 'notmuch-tag-flagged)
 (concat tag "🚩")))
 notmuch-tag-deleted-formats
 '(("unread" (notmuch-apply-face bare-tag 'notmuch-tag-deleted)
 (concat "👀" tag))
 (".*" (notmuch-apply-face tag 'notmuch-tag-deleted)
 (concat "🚫" tag)))
 notmuch-tag-added-formats
 '(("del" (notmuch-apply-face tag 'notmuch-tag-added)
 (concat "💥" tag))
 (".*" (notmuch-apply-face tag 'notmuch-tag-added)
 (concat "🏷️" tag))))

 (setq notmuch-tagging-keys
 `((,(kbd "d") ("+trash" "-inbox" "-unread") "🗑️ Mark for deletion")
 (,(kbd "s") ("+spam" "-inbox" "-unread") "🪓 Mark as spam")
 (,(kbd "f") ("+flag") "🚩 Flag as important")
 (,(kbd "i") ("+invoice" "-inbox" "-unread") "🛒 Mark as invoice")
 (,(kbd "a") ("+archived" "-inbox" "-unread") "🗃️ Mark as archived")
 (,(kbd "r") ("-unread") "👀 Mark as read")
 (,(kbd "u") ("+unread") "👀 Mark as unread")))

 (keymap-set notmuch-common-keymap (kbd "u") #'notmuch-tag-undo)

 (defun notmuch-search-mode-delete-goto-next ()
 "Tag current mail as trash and move to the next."
 (interactive)
 (notmuch-search-tag '("+trash" "-unread" "-inbox"))
 (forward-line)
 )
 (defun notmuch-show-mode-delete-goto-next ()
 "Tag current mail as trash and move to the next."
 (interactive)
 (notmuch-show-tag '("+trash" "-unread" "-inbox"))
 (notmuch-show-next-message t)
 (notmuch-search-show-thread)
 )

 (keymap-set notmuch-search-mode-map (kbd "d") #'notmuch-search-mode-delete-goto-next)
 (keymap-set notmuch-show-mode-map (kbd "d") #'notmuch-show-mode-delete-goto-next)
 )

Often times I want to open a HTML email in the browser. That needs an entry in ~/.mailcap:

text/html; brave %s; nametemplate=%s.html

I added a wrapper around the native notmuch-show-view-part so it does what I want from opening email in the browser:

(defun notmuch-show-mode-open-html ()
 "Open HTML part of message in browser (via ~/.mailcap)."
 (interactive)
 (save-excursion
 (notmuch-show-previous-message)
 (condition-case nil
 (search-forward "text/html")
 (error (message "No HTML part."))
 (:success (notmuch-show-view-part)
 (shell-command "open-i3-workspace 2")
 (message "Opened HTML in browser")
 ))))
(defun notmuch-search-mode-open-html ()
 "Open HTML part of message in browser (via ~/.mailcap)."
 (interactive)
 (save-excursion
 (notmuch-search-show-thread)
 (notmuch-show-previous-message)
 (condition-case nil
 (search-forward "text/html")
 (error (message "No HTML part."))
 (:success (notmuch-show-view-part)
 (shell-command "open-i3-workspace 2")
 (message "Opened HTML in browser")
 ))))

(keymap-set notmuch-search-mode-map (kbd "o") #'notmuch-search-mode-open-html)
(keymap-set notmuch-show-mode-map (kbd "o") #'notmuch-show-mode-open-html)

Mutt has an option called reverse_name. If activated the From: line in a reply will be built using the address the message was originally sent To:. If somebody writes me an email to “hi@jneidel.com”, I want to answer from “hi@jneidel.com”, even though “jneidel@jneidel.de” is my configured as my default sender. The code below does exactly that.

(use-package notmuch
 :defer t
 :config
 (defun notmuch--get-message-to ()
 "Return the raw To: header of the message or nil."
 (let ((message-properties (cond
 ((derived-mode-p 'notmuch-show-mode)
 (notmuch-show-get-message-properties))
 ((derived-mode-p 'notmuch-search-mode)
 (notmuch-search-show-thread)
 (notmuch-show-get-message-properties))
 ((derived-mode-p 'notmuch-tree-mode)
 (notmuch-tree-get-message-properties))
 (t nil))))
 (if (eq message-properties nil)
 nil
 (plist-get (plist-get message-properties :headers) :To))))

 (defun notmuch-reply-reverse-name-advice (orig-fn &rest args)
 "Advice 'notmuch-mua-reply' to use the original To: addresses as the From:."
 (let ((to-address (notmuch--get-message-to)))
 (apply orig-fn args)
 (when to-address
 (message-replace-header "From" to-address))))

 (advice-add 'notmuch-mua-reply :around #'notmuch-reply-reverse-name-advice))

One of the killer feature of having email in Emacs is being able to reference an email with a clickable link from org-mode files. This is achieved using ol-notmuch.

(use-package ol-notmuch
 :bind
 ("C-c l" . org-store-link))

Oftentimes PDFs are attached to incoming mail. By default these would need to be manually saved and then opened. This script extracts the text data from the PDF attachment and pops it into a new buffer (source).

(defun user/mm-pipe-- (handle cmd)
 ;; conveniently, '-' '-' a args to pdftotext and docx2txt.pl work fine
 ;; fixme: naming inconsistency (fn name and buffer name)
 (let ((buffer (get-buffer-create "*attachment-to-text*")))
 (with-current-buffer buffer
 (setq buffer-read-only nil)
 (erase-buffer))
 (with-temp-buffer
 ;; "based on mm-pipe-part in mm-decode.el"
 (mm-with-unibyte-buffer
 (mm-insert-part handle)
 (mm-add-meta-html-tag handle)
 (let ((coding-system-for-write 'binary))
 (call-process-region (point-min) (point-max)
 cmd nil buffer nil "-" "-"))))
 (pop-to-buffer buffer)
 (goto-char (point-min))
 (text-mode)
 (visual-line-mode)
 (view-mode)))

(defun user/notmuch-show-pop-attachment-to-buffer ()
 ;; "based on notmuch-show-apply-to-current-part-handle"
 (interactive)
 (let ((handle (notmuch-show-current-part-handle)))
 ;;(message "%s" handle)
 (unwind-protect
 (pcase (car (nth 1 handle))
 ("application/pdf"
 (user/mm-pipe-- handle "pdftotext"))
 ("application/vnd.openxmlformats-officedocument.wordprocessingml.document"
 (user/mm-pipe-- handle "docx2txt.pl"))
 (_ (notmuch-show-save-part)))
 (kill-buffer (mm-handle-buffer handle)))))

(setq notmuch-show-part-button-default-action
 #'user/notmuch-show-pop-attachment-to-buffer)

The latest version of my Emacs config (and thus all of the above snippets) can always be found in my dotfiles.

Android client #

For the occasional access on my phone I use K-9 Mail. The same values as in the above IMAP and SMTP sections are used.

Testing #

Obviously we need to test that outgoing mail from the different domains are delivered and that incoming mail arrives, is properly tagged and displayed.

Uberspaces webmail can be used for testing if needed.

DKIM

To make sure DKIM and SPF are working correctly used this website for testing.

Conclusions #

Email was at first a daunting task because of the many moving parts and the endless testing on every change. Once I understood what components I needed and what they were responsible for they were not hard to configure. I hope you were able to take that understanding away from this.

Being able to have email in Emacs was definitely worth it to me. It is now easier than ever to reach inbox zero.

Easily collect website feedback

web@jneidel.com (Jonathan Neidel) — Sat, 11 May 2024 00:00:00 +0000

I have this website. It will necessarily include errors. I would like to make it as easy as possible for other people to help me fix them. If somebody spots an error, there

I would categorize people that want to help into the categories of:

Known vs unknown: if the person knows the author directly
Technical vs non-technical: if the person is capable of editing the source and submitting a patch

If you don’t provide people who are willing to help with another option, they will communicate change requests through these communication channels:

	Unknown	Known
Non-technical	Email	Email or private messenge
Technical	Email	Email or private messenge

When I see a typo while browsing, I will navigate to the contact page and write the website owner an email (or use a contact form, if applicable.)

The issue with email #

It’s better than nothing, but a few problems will crop up:

People have to leave the website to write an email or take notes.
Corrections will be inconsistent or have information missing.
Visual defects are hard to describe through just text.
It’s a missed opportunity. Technical people might be willing to do the correction for you.

We can do better than that.

Overview of solutions #

Let’s go through our options for each type of person:

Unknown, non-technical person: They are the people least likely to be interested in helping and it is the hardest to set something up for them.
Known, non-technical person: Much more willing to help and we can provide them credentials to a tool.
Technical person: The most interested in helping and most capable at doing so. Give them an easy way to submit patches.

By putting this in a table, we get:

	Unknown	Known
Non-technical	Email	Tool
Technical	Submit a patch	Submit a patch

The concrete solutions I have implemented are detailed below. How hard or easy they are for you to adopt, depends entirely on your setup. There might be comparable variants in whatever you are using. Even if you can not make use of them directly, the principles will be the same.

These are the solutions that I am using and will detail below:

	Unknown	Known
Non-technical	Email	Netlify Drawer
Technical	Edit on Github button	Edit on Github button or Netlify Drawer

Netlify Drawer #

Netlify has embedded a tool for submitting feedback directly without leaving the page.

The obvious requirement: Your site has to be hosted on Netlify.

How does it work? #

At the bottom of the page there is a UI element.

The drawer on the bottom of a page.

Clicking it will expand it, to reveal a form. Through this form you can submit text, take screenshots or record video¹. What ever fits best to document the defect, submit a correction or suggest something.

Submit a correction through the expanded drawer.

Make sure to assign an epic, so the owner will be notified:

That’s it from the user perspective. They just send of their report and close the drawer. As the site operator, I get an email with all the info I need to fix the reported issue:

Setup #

This is how you can configure your own site to use the Netlify Drawer just like this.

1. Setup a parallel deployment with the Netlify Drawer activated #

We don’t want have the Netlify Drawer active on the main site, since we only want the reviewers to be able to see it. We need a parallel deployment, which mirrors production, and has the Netlify Drawer enabled.

Do configure that we log into Netlify and:

Create a new site
Supply the same repo as build source
(Optional: Change the build command -> I also make drafts visible.)
Under “Branches and deploy contexts”:
- Set the production branch to “none” or something invalid.
- Add your master branch under “Branch deploys” (Netlify Drawer only works on branches deployments, not on the production deployment.)
Scroll down to “Collaboration tools” and make sure it is enable on branch deployments.

Now whenever you deploy push to your master branch, you will also get a secondary deployment with Netlify Drawer enabled. The working link will be branch + site name. Like this:

https://master--jneidel-stage.netlify.app

2. Create a reviewer Netlify account #

We want to have a reviewer account whose credentials we can just pass along to all our reviewers. Let’s create a new account for that. It’s easiest if you start by inviting the new account as a reviewer to your team under “Team > Members > Reviewers > Add Reviewer.” Then just sign-up from the email you received.

3. Create two Shortcut accounts #

Head over to Shortcut, to first create an account for yourself. If you sign-up with an email from Kill the Newsletter you can get all communication (i.e. future notifications of changes) through a RSS feed.

After your primary account is signed-up, we want to create another account for our reviewers to use. From the primary account invite the reviewer email to your shortcut team. Then go through that accounts sign-up flow as well.

4. Configure Shortcut accounts #

While on your primary account, go into “Settings > Notifications” and enable “Most Events.”

Select the “Epics” section in the sidebar under “Teams > All Work” and create a new epic for your reviewers to create issues in. Assign yourself as owner, so that you will be notified of any new issues that are created in that epic.

While on the secondary reviewer account, go into “Settings > API Tokens” to create a new token and save it for the next step.

5. Configure Netlify reviewer account #

Open your secondary Netlify deployment, open the drawer and log in with the Netlify reviewer account. Go into integrations and connect to Shortcut by pasting in the API Token for the Shortcut reviewer account created in the previous step.

6. Profit #

Now everything is setup and you can pass the credentials to the Netlify reviewer account and linkt to the secondary deployment along to your reviewers. You will be notified of their reports via email.

Edit on Github #

If you website sources are hosted publicly, everyone will be able to suggest improvements. Otherwise you need to grant access to individual reviewers to be able to receive their suggestions

In my case the website’s repo is hosted on Github.

How does it work? #

I have a button at the top of the page that leads to the markdown source file of that page.

That buttons leads to this page:

https://github.com/jneidel/jneidel.com/blob/master/content/guide/collect-website-feedback/index.md

A technical user can edit the page and submit a pull request. You can try it yourself at the top of this page.

Setup #

You need to find a way to generate and embed the link the source file into the page. That is something my hugo theme (congo) does for me.

Setup with congo #

I just supply this config:

[params.article]
 showEdit = true
 editURL = "https://github.com/jneidel/blog/tree/master/content/"
 editAppendPath = true

Within congo, to change the icon to the GitHub logo: copy layouts/partials/meta/edit.html in your own project and edit -> github. To change the hover title, add this to your i18n/en.yaml:

article:
 edit_title: "Edit content on GitHub - Thanks for submitting a pull request!"

Conclusions #

If come across problems on this website feel to use the edit button below the title or let me know via email :slightly_smiling_face:

You can read more about it in Netlify’s docs. ↩︎

My Backup strategy

web@jneidel.com (Jonathan Neidel) — Fri, 05 Jan 2024 00:00:00 +0000

Everyone is responsible for backing up their own data. In this guide I’ll walk you through how I do it.

Code and configurations #

My software projects and linux configurations are versioned via git and use GitHub as a remote backup. This relies on me committing and pushing changes regularly.

My dotfiles allow me to completely recreate my linux setup in a few hours on a new device.

Threat vectors covered: Main machine drive failure

Secrets #

I also version my secrets (password database, api keys, private configurations) in git. Once a day a cronjob runs a copy of my dotfiles repo management scripts that collect the relevant files, commits any changes and pushes to a private server. You could also use private GitHub repos, but I rather keep my unencrypted secrets on my own server. Alternatively you could encrypt the files then you can store it where ever. But this wouldn’t play well with git, since all files will be different every time.

Threat vectors covered: Main machine drive failure, revert undesired changes

Notes and laptop/phone media #

As detailed in my post about my Note synchronization setup I use Syncthing to sync them to my homeserver (and from there to other devices.) The same applies to music and any other media on my phone or computer.

Syncthing offers multiple ways to version your files as to allow for retrival of changed or deleted files. I only use this versioning on the homesever and vary it based on the type of files being synced. For my notes I use “Simple” 45 day versioning and for my media 20 day “Trash Can” versioning.

Synchronization is not a backup though. By being synced through my homeserver everything is also covered by it’s backup strategy.

Threat vectors covered: Main machine drive failure, revert undesired changes

Homeserver #

To project against disk failure I’m running UNRAID. I opted for UNRAID over RAID for the ease of upgrading. RAID would have required me to buy all of my hard-drives together. I wasn’t sure how much storage I needed and wanted to reserve the option to upgrade slowly instead of having a big upfront investment.

RAID, or UNRAID in my case is not a backup though. So besides my single parity disk I also have two external WD Elements drives for on-site and off-site backups. On-site is backed-up weekly and swapped with the off-site one at a relatives place whenever I visit. Any safe place outside of your house that you regularly visit will do.

Threat vectors covered: Single drive failure¹, multiple drive failures², Randsomware, environmental hazards at server location (fire, water, burglary)

Reference:

Technical details #

By encrypting our external backup drives we can secure our data against abuse.

Threat vectors covered: External backup in the wrong hands

Initial setup #

Whenever I get a new drive that is to serve as a backup I have to test the drive and put an encrypted filesystem on it.

Testing the drive #

To verify that the disk is fine run an extended SMART test on it. I do this through the UNRAID UI, but I’m sure there is a way to do it on the command-line. If there are any errors the drive gets returned to the seller.

Next I’ll preclear the drive. This will:

read out every byte
write a zero to every byte
read out every byte again

This is technically unnecessary³ but will serve as a stresstest for the drive. If it will fail early I want to know as soon as possible so I can return it.

On UNRAID this can be done through the ‘Unassigned Devices Preclear’ plugin⁴.

Format the drive #

To access the drive without having to enter our passphrase every time⁵ we will generate a keyfile that will be placed on the server.

dd bs=512 count=16 if=/dev/random of=keyfile iflag=fullblock

With this we can now put an encrypted ext4 filesystem on our drive:

# optional: switch to admin
sudo su

# get the device id for the drive you want to format
# -> replace sdX below
lsblk

# format drive, generate and enter a passphrase
cryptsetup -s 512 luksFormat /dev/sdX

# for keyfile unlock
cryptsetup luksAddKey /dev/sdX keyfile

# create filesystem
cryptsetup luksOpen /dev/sdX cryptext --key-file keyfile
mkfs.ext4 /dev/mapper/cryptext
cryptsetup luksClose cryptext

Now you’re ready to backup your server to this drive.

Reference: Arch wiki section on disk encryption

Update external backup #

To only transfer the changes between to the backup I use rsync – the standard utility for this kind of job.

On my server I run: $ ./sync-backup sdd where sdd specifies the device id of my backup drive. Here are the scripts contents:

#! /bin/sh

keyfile=/mnt/user/system/user/keyfile
cryptmapper=cryptbackup
mapper=/dev/mapper/$cryptmapper
mount_location=/ext/backup

mkdir -p $mount_location /ext/hd

if [ "$1" = "--help" ] || [ "$1" = "-h" ] || [ "$1" = "help" ] || [ -z "$1" ]; then
 cat <<EOF
$ sync-backup device_id
Backup data onto exsting backup drive

Parameters:
 \$1: like sdd, obtained from lsblk

Example:
 $ sync-backup sdd
EOF
 exit
fi

command -v rsync >/dev/null || { echo "rsync is not installed" 1>&2; exit 127; }

device_id=$1
device=/dev/$device_id

cryptsetup luksOpen $device $cryptmapper --key-file $keyfile && echo Opened luks && sleep 4s &&
 mount $mapper $mount_location && echo Mounted backup drive &&
 rsync -avr --delete /mnt/user /boot $mount_location &&
 sync &&
 umount $mount_location && echo Unmounted backup drive &&
 cryptsetup luksClose $cryptmapper && echo Closed luks

Open threat vectors #

These I am aware off, but haven’t addressed yet, or we’re not worth addressing.

Data corruption due to electricity blackout:

Could be addressed by having the main power supply run through a battery. So that if the power suddenly stops the battery keeps the server running so it can gently shut-down. I don’t know how the intermediate battery would communicate this shutdown to the server though and relying on me to manually initiate shut-down would be error prone.
Software project environment files:

While the project is backed up these env files are not necessarily. I have not yet build a consistent way to best back these up. Critical ones would be included in the secrets programme though.

If you think I’m missing something here, please let me know at: backups@jneidel.com

Covered by UNRAID’s parity. Plug in a new drive and the failed one will be recreated. ↩︎
With a single parity drive this mean UNRAID is failing. Data on the lost drives is not recoverable and will have to be recreated from the backup. ↩︎
If you want to add a new drive to an UNRAID array it is required to be zeroed so it matches the parity. There is no such need in external drives. ↩︎
Install the plugins ‘Unassigned Devices’ and ‘Unassigned Devices Preclear’. Then in ‘Tools > Disk Utilities’ you will have the option to preclear a disk. ↩︎
This is optional. You can also skip this and enter a passphrase every time you mount the drive. ↩︎

Syncing text notes between multiple devices

web@jneidel.com (Jonathan Neidel) — Mon, 13 Nov 2023 00:00:00 +0000

If you’re not using a note-taking app that handles synchronization for you, you’ll have to find a different solution.

This guide is going to detail how I use Syncthing for all my synchronization needs and how I use Signal as my notes inbox.

Syncthing #

Why Syncthing #

it just works™
open source
feels-like-instant sync
works behind CGNAT routers (no outgoing IP address)
nice web-inteface for management
decent memory/cpu usage
- Has configurable run conditions on the phone to e.g. only run while being charged
you don’t have worry about storage costs with a cloud provider
- Dropbox’s free plan is a paltry 2GB
- my notes are 30GB and I don’t have to care
ignore patterns
- very useful for caches, node_modules, vendor, etc.
nested folders (more on that later)
built-in versioning
- so you can easily revert unintentional changes/deletions

Between computers #

My most basic use case was syncing my notes between my work and personal computer. My setup is running on both machines, with some tweaks depending specific on the device, all I need is to plug in my notes.

You can sync directly between the machines:

stateDiagram-v2 Work --> Personal Personal --> Work

But I would recommend you put a server in between, so you can also sync without them both being online at the same time (e.g. you close the one machine and open the other.)

stateDiagram-v2 Server --> Personal Server --> Work Work --> Server Personal --> Server

Between phone and computer #

I’m using Syncthing-Fork on my phone. The dynamic is much the same:

stateDiagram-v2 Server --> Phone Phone --> Server Server --> Computer Computer --> Server

I have a media sync with my music library, podcasts, movies, etc. To only sync parts of this I can use nested folders and only expose a subset to my phone:

.
└── media <-- syncthing folder 1
 ├── movies
 ├── music
 │ └── symlink to ../phone/music
 └── phone <-- syncthing folder 2
 ├── podcasts
 └── music

Like this I can add new music and podcasts from my computer and delete stuff I heard podcasts from my phone. Much easier than the manual ftp moving I did before.

With the symlink I have the phones music integrated into my mpd setup, which will automatically have deletion from the phone of song I don’t like synced.

The server #

Syncthing runs fine even on a Raspberry Pi. I have it running on my homeserver, where the data is included in my regular backups.

Signal as an inbox #

I only use my phone to jot down notes, not to read them.

My unoptimized workflow was me writing into a Signal chat with myself and copying out the notes on my computer. The obvious improvement was to automate that by having a script read out the signal chat and create notes out of the messages.

I have done that by creating a script that parses the signal cli app. You can find out everything about it in it’s repository: signal-cli-to-file

With my current workflow I can easily jot down a note in Signal and have it delivered into my note-taking system. I also routinely forward messages from other chats, emails, photos, PDFs and whatever else I might run across on my phone.