title: "September 2026"

September 2026

BlueBubbles read receipts — 23 September

Investigation, repair, verification, and deployment log.

systemd re-visitied

Context is trying to figure out why sabnzbd-categories is failing at systemd...

So i asked claude last time and got most of it but writing it myself always helps....

What am I trying to understand here? a) why booting takes so long b) how does after/depends-on/requires/after/before works in the dependency tree of systemd

Output of systemd-analyze critical-chain

graphical.target @5.292s
└─multi-user.target @5.291s
  └─tailscaled.service @4.006s +1.253s
    └─NetworkManager.service @2.066s +736ms
      └─basic.target @2.063s
        └─dbus-broker.service @1.980s +67ms
          └─dbus.socket @1.969s +73us
            └─sysinit.target @1.955s
              └─systemd-resolved.service @952ms +949ms
                └─systemd-sysctl.service @927ms +22ms
                  └─systemd-modules-load.service @835ms +88ms
                    └─systemd-journald.socket
                      └─system.slice
                        └─-.slice

right so....

  1. systemd-modules-load.services starts at 835ms and takes 88ms
  2. sytemd-sysctl-.service starts at 927ms and takes 22ms

this makes sense as 835ms + 88ms = 923ms, which i guess... is ~= 927ms

so apparently the above is for boot ^

according to codex the approx oder from power in is:

  1. firmware / bootloader / rgub
  2. kernel
  3. systemd
  4. tuigreet (login)
  5. USWM
  6. hyprland
  7. graphical-session.target

for the user its systemd-analyze --user critical-chain

➜ jollof dev-setup (main) ✗ systemd-analyze --user critical-chain
  1 The time when unit became active or started is printed after the "@" character.
  2 The time the unit took to start is printed after the "+" character.
  3
  4 default.target @222ms
  5 └─nixos-activation.service @212ms +10ms
  6   └─basic.target @209ms
  7     └─dbus-broker.service @221ms +22ms
  8       └─session.slice @220ms
  9         └─-.slice @171ms

hmmm i guess this is the critical chain not actually hyprland? as hyprland does NOT take 222ms

well this is MUCH more reasonable

➜ jollof dev-setup (main) ✗ systemd-analyze --user critical-chain graphical-session.target
  1 The time when unit became active or started is printed after the "@" character.
  2 The time the unit took to start is printed after the "+" character.
  3
  4 graphical-session.target @3.040s
  5 └─basic.target @209ms
  6   └─dbus-broker.service @221ms +22ms
  7     └─session.slice @220ms
  8       └─-.slice @171ms

but still doesn't tell me exactly where that 3s comes from? anyway i digress....

systemd rules

right so under systemd [Unit]

  • wants/requires is should the unit actually be run?
  • so A wants B means A ... needs B to run, but can be started concurrently (along for the ride)
  • A wants B is more like if A started, start B, A wont fail if B failes
  • A wants=B After=B means needs B and should run after B has started
  • wantedby is the reverse of wants, i.e. network-manager wantedBy=tailscale, as tailsacle will need networkmanagewr

ahhhhh so

  • A requires=B means A wants=B but unlike wants if B fails A also fails

right table from codex explaining... for the "starting" relationships, i.e. when units should be switched on or not

ForwardReverseExplanation
Wants=BWantedBy=AStarting A also requests B. B may fail; A continues.
Requires=BRequiredBy=AStarting A also requests B. Stronger failure/stop coupling.
BindsTo=BBoundBy=ALike Requires, but A also stops if B disappears/crashes.
PartOf=BConsistsOf=AStopping/restarting B also stops/restarts A.
Requisite=BRequisiteOf=AB must already be active; does not start B.

according to codex, a for A Requires=B. on start by B means A does not start, but later on if B crashes A will NOT stop

not 100% sure about bindsTo and partOf honestly

then for ordering

ForwardReverseExplanation
After=BBefore=AIf both start, wait for B before starting A.

kitty breaking escape keys

How weird... in a broken tmux state where escape is being malformed?

➜ jollof dev-setup (main) ✗ cat -v
^[[27u^[[27u^[[27u^[[27u^[[27u^[[27u^C%
➜ jollof dev-setup (main) ✗ cat -v
^[^[^[^[^C%

the 2nd is escaped esacpe code (from a working kitty window) - but the first is malforming them?

nix paths

god this is a long time coming...

claude can seemingly cook up all the nix paths required to debug but i never know how it works - why does nix eval take so long? where do these store paths come from?

the most obvious one is with systemd units already built... just do systemctl status sabnzbd-categories.service or systemctl cat and the path will be in ExecStart=/nix/store/.....

god.... this nightmare again... nix eval.. evaluates, but apparently the format to evaluate from this repo is in the form of a module NOT a flake. I.e. accoridng to (nixos modules)[https://nixos.wiki/wiki/NixOS_modules]

thus, we can do nix eval to get the networking resolved options for pi-box

➜ claude@worker207 dev-setup (main) ✗ nix eval --json "#nixosConfigurations.pi-box.options.services.resolved"
warning: Git tree '/home/claude/dev-setup' is dirty
{
  "dnsDelegates": "services.resolved.dnsDelegates",
  "dnsovertls": "services.resolved.dnsovertls",
  "dnssec": "services.resolved.dnssec",
  "domains": "services.resolved.domains",
  "enable": "services.resolved.enable",
  "extraConfig": "services.resolved.extraConfig",
  "fallbackDns": "services.resolved.fallbackDns",
  "llmnr": "services.resolved.llmnr",
  "settings": {
    "Resolve": "services.resolved.settings.Resolve"
  }
}

so my understanding is the let in... nixosConfigurations produces the output nixosConfigurations

`the # is short form for nix eval --json "/home/joelyboy/dev-setup#nixosConfigurations...."

claude separates these into two categories, nix "evaluations" which doesnt build anything it just "computes" it

CommandDoesExampleOutput
nix evalPrint a Nix valuenix eval --impure --expr '1 + 1'2
nix replInteractive: :lf . then tab-completenix replnix-repl>
nix flake metadataInputs and pinned revs (reads flake.lock, no eval)nix flake metadata --json | jq -r '.locks.nodes.nixpkgs.locked.rev'59e69648d345…
nix flake showList a flake's outputsnix flake showtree of nixosConfigurations, packages…

claude then talkes about "build" where we actuall build something

CommandDoesExampleOutput
nix buildBuild a derivation, symlink ./resultnix build "$W.config.system.build.toplevel"creates ./result
nix runBuild then immediately run itnix run nixpkgs#helloHello, world!
nix shellBuild, drop into a shell with it on $PATHnix shell nixpkgs#jqnew shell prompt

ahhh so this helps... in nix repl, we run :lf . to load this flake here at dev-setup

then can simply type nixosConfigurations and the interprester will give us the list:

nix-repl> nixosConfigurations
{
  degen-bot = { ... };
  degen-home = { ... };
  degen-work = { ... };
  desktop-work = { ... };
  installer = { ... };
  jollof-home = { ... };
  pi-box = { ... };
  streaming-server = { ... };
  worker207 = { ... };
}

although the more "correct" way would be builtins.attrNames nixosConfigurations. From this we can use builtins.attrNames nixosConfigurations.pi-box.options to figure out there is nixflix

thus. grabbing all in a json in a single expression becomes: nix eval --json .#nixosConfigurations.pi-box.options.nixflix

Vimium

So annoyingly vimium synchronises to a file somewhere in /home/jollof/.config/BraveSoftware/Brave-Browser/Default/Extensions/... but its a binary file? so is not just a plain old list of configuratrions.

I did find this but it requires home manager (which I don't want to use)

Then there is surfingkeys, which DOES have a plain text option for blocked sites but you have to enable developer mode in brave and then user scripts for the extension, which i don't think is configurable via nixos

See chromium policies, for details on the chrome policy to block vimium on those pages

Now i (could) try tridactl but i did get it to work with a nice hack from codex?

Agent sidebars

Interactive architecture map and remote setup

So apparently someone has built what ive been wondering about a) tmux combined with remote session b) status indicator for coding (claude-code, codex, opencode) whether they're waiting idle etc

I did try tmux-agent-sidebar before but wasn't overly impressed with the UI

quick recap on TTYs.... If i open a new terminal and type tty then i get /dev/pts/10

This means kitty is bound to that pseudo-terminal device? whatever that means... And can see it belongs to the tty group:

➜ jollof ~ ls -la /dev/pts/10
crw--w---- 1 jollof tty 136, 10 Sep 13 11:37 /dev/pts/10

Then, can do the same thing for tmux and get its tty

➜ jollof ~ tmux display-message -p '#{pane_tty}'
/dev/pts/12

To view all of these options like pane_tty, do tmux display-message -a

quite cool to view the process tree and see the tmux attach and the associated pts with it, from ps -ef --forest

root     3945642    1393  0 Sep07 ?        00:00:00  |   \_ sshd-session: claude [priv]
claude   3945646 3945642  0 Sep07 ?        00:00:56  |   |   \_ sshd-session: claude@pts/2
claude   3945647 3945646  0 Sep07 pts/2    00:00:00  |   |       \_ -zsh
claude   3945691 3945647  0 Sep07 pts/8    00:00:00  |   |           \_ -zsh
claude   3945996 3945647  0 Sep07 pts/2    00:00:00  |   |           \_ tmux attach
root     3648552    1393  0 10:49 ?        00:00:00  |   \_ sshd-session: claude [priv]
claude   3648566 3648552  0 10:49 ?        00:00:06  |       \_ sshd-session: claude@pts/5
claude   3648567 3648566  0 10:49 pts/5    00:00:00  |           \_ -zsh
claude   3648611 3648567  0 10:49 pts/9    00:00:00  |               \_ -zsh
claude   3649060 3648567  0 10:49 pts/5    00:00:00  |               \_ tmux attach

So a comparison table so far of what I'm looking for:

  • unified view of local + remote sessions in tmux
  • ability to preview remote session in the switcher (like choose tree)
  • see which coding LLM sessions are running, waiting, which ones were last active

So far:

namelinklast commitcontributorscommitsstarsunified viewpreview session
tmaxgithublast week111143✅⚠️
tmmxgithub2 days ago1321⚠️❌
tmux-agent-statusgithub3 days ago9155276⚠️✅
iTerm2githubtoday100+—18051⚠️✅
tmux-remote-sessionsgithub4 years ago31841❌❌
seshgithub3 days ago614162821❌✅
tmux-sessionxgithub13 days ago362401378❌✅
tap (tmux-agent-panel)github2 days ago11081❌✅

My additional thoughts on each one

Tmax

Really nice piece of work, but im not a fan of how the local tmux emulates remote sessions.

I already had to hack the code to make it accept an SSH key (not password), and discovered a bug where splitting panes doesn't respect the current dir

also a bit of a 🟥 that it clobbers the claude settings.json file

Tmmx

Not an agent sidebar, instead a remote connection manager. Works by nesting remote sessions inside local ones.

Made by one guy, 1 star, basically what I hacked together myself.

Annoyingly, it does not integrate with tmux-agent-sidebar with nested sessions...

tmux-agent-sidebar

Seems good so far , stable which is the most important thing. although the UI is hard to read...

Feels like having a loading/waiting/idle status per LLM session would be simplest

Bit of a 🟥 that last release was 3 months ago given how fast this space moves, and plenty of PRs sitting un-merged... but it DOES seem the most trusty workhorse at the moment...

annoyingly, it will only load agents in the flat tmux server - doing nested tmux sessions with remotes it doesnt pick them up.. not a fault but means it wont work with my end goal of sidebar for all agents in all remotes

claude squad

only tried briefly. makes its own tmux sessions and has its own TUI... doesn't match my workflow as I want something to integrate not replace my workflow

agenmux

very intersting new project, very new (and very janky)

I tried to use tmux zoom pane in the sidebar pane and it crashed it... not the best start. ok maybe i overlooked this one.

i think this is the future but it doesnt have timestamps which is the most annoying thing right now...

tmux-agent-status

yeah i just could not get this to work cleanly. granted i got codex to do the heavy lifting of setting it up but I couldnt expand panes within remote sessions, the loading prefix for statuses didnt update (although i wasnt logged in).

makes sense as its all shell script with not that much activity

sesh / tmux-sessionx / tap: local-only, so ❌ on unified, but could be useful for inspiration on the picker

verdict on agent sidebars

ok so what im convering on is:

  • tmax is a bit janky... i think its control and hack of keys is quiestionable.
  • HOWEVER- tmax can se the built in c-a for searching is EXCELLENT
  • re-making my own searcher is a bit of a faff
  • tmax also binds c-a L which is also excellent
  • bug just happened with tmax! pretty annoying, splits didnt work and not in the same path...

what is a happier middle ground (I think) is:

  • let someone else handle the agent statuses? like running/idle etc?
  • whats hard is c-a L? to track remote sessions?
  • could have a c-a E - which is like c-a L but for remotes? i.e. go to last remote?
  • trying to remember the combination of the last local session and last remote session is just too much of a faff...

opensessions

this looks very promising but the fact there's been 1 release 3 months ago on alpha and no commits since then is a bit a 🟥

Seems like there's either janky maintained personal projects or ones that dont fit my workflow 😥

agent-deck

hmmm - gave it a try, it has a pretty heavy handed TUI.

I had problems straight away with it not connecting to tmux... granted I probably didn't configure it properly but I also don't NEED a TUI to spin up agents? I just need a manager to track my agents and let me switch between them

pasting issues

Stil getting these random issues with pasting just doesnt work, even though the entry at the top of cliphist via super-v does work?

so (hopefully) this is fixed now with the addition of wl-clip-persist. have added clipdebug to zsh incase we need to debug it in the future where clipboard fails again

nvim command searching

something i forgot, can do : to bring up the command view in nvim now i have the binding for c-f to bring up the search pane for historic commands. as i always forget the weird q: binding thats' already built in

image trimming

So the purpose of this is to try and bring down the size of the container to save on space when I want to spin up a test?

But a few points to clarify which I seem to be getting confused with

  1. doing pacman -S gdu inside Dockerfile is NOT the same as doing it in a container
    • both install files on disk: the first in an image layer, the second in the container's writable layer
    • running the installer or scanning files can increase RAM usage, including caches that can remain afterwards and be reclaimed when needed
  2. docker image layers stay on disk; files are loaded into RAM as programs use them, not all at container startup

however, trimming down the absolutely necessary neovim dependencies and keybindings to allow me to keep the container RAM small simpy because it won't be spamming language servers

nvim tabs

right so been meaning to do this for a while...

I was initially hesitant to an approach like this? given that i have harpoon but never actually use it (dead plugin), but this could be good? ok tabby is actually quite good - the naming of tabs like DiffviewFilePanel is excellent - having a way to search for a tab <space>Tp is lovely

So a sumamry of changes to keys:

  • <space>Tp search for tabs
  • <space>T for tab menu
  • 4gt go to tab 4
  • g<tab> go to previously used tab

and as a refresher:

  • mA marks A (capital is across all files)
  • 'A goes to mark A
  • dmA deletes mark A

and mini surround 🤣, covering all the stops here

  1. the first step is sa for adding chars? sd for deletion, sr replace and highligh sh

ok so followups... a

  1. if i have and have selected the space text and wanted to delete the <>, why doesnt sd< work, sd just deletes the work straight away?
  2. how does the selection work? with sa? is it more implicit plugin hell? im just supposed ot know?

nix ssh config

annoyingly, ssh config configurations is only available via home-manager, so having to rawdog it (or get claude to do it)

future work for agent status

TODO: Right now the switcher is a bit of a hack reading through temp files to get the last change on agents..., exposed by tmux-agent-sidebar but not in the metadata...

in the future would be nice ot just have this "available", either via a PR or different package. annoyingly, agenmux doesnt poll if sidebar is closed so unsure if thats the right model as it doesnt use hooks

tmux selection

So up until now ive been doing c-a + [ + k + V to get the last line... so there IS a better way!

  1. ctrl-a tab to go into extrakto which has different modes an fzf
  2. extrakto also has tab to insert to enter to copy if we want to use it stragith away nice
  3. tmux-finger has ctrl-a F to copy specific work via key, or ctral-a J to jump to position by vim key

unsure about that one tmux finger? seems specific to files, git etc not very generic

  1. copycat has ctrl-a ctrl-f for files, ctrl-a ctrl-u for urls - and goes back up the buffer unlike the fzf plugin
  2. another binding for me is the extrakto ctrl-a ctrl-l to go straight into line mode

pi-box → streaming-server migration

So pi-box is underpowered, so moving the big harddrive to streaming-server (beelink bit more powerful) and all the configs with it.

This is the rough list of migration steps:

  1. nix configurations copied across, iamb points to stremaing-server, set mac-mini destination to streaming-server (98e14ce)
  2. go to the sparkyfitness admin dashboard on pi-box and click "backup now" in the admin section
  3. copy to backup tar file from the pi-box to the /mnt/big-hdd (currently mounted on pi-box)
  4. turn off pi and plug in big HDD to streaming server box
  5. rebuild the streaming-server box with task os:build
  6. copy the backup from the big HDD box (mounted with the new config) to downloads
  7. use the manual restore from db in admin from sparkyfitness

And for the manual steps:

  1. change bitwarden entries for jellyfin/seer etc from pi-box to streaming-server
  2. invite the rooms and login:
:invite send @whatsappbot:jollof.chat
:invite send @signalbot:jollof.chat
:invite send @telegrambot:jollof.chat
:invite send @facebookbot:jollof.chat
  1. prompt mac to restart and re-add itself as a user:
# stop it (launchd would restart it straight away if you just killed it)
launchctl bootout gui/$(id -u)/org.nixos.mautrix-imessage
# move the DB and its side files out of the way
cd ~/.local/share/mautrix-imessage
mkdir -p ~/imessage-db-old
mv mautrix-imessage.db* ~/imessage-db-old/
# start it again
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.nixos.mautrix-imessage.plist

todo:

  • matrix pwd to store?
  • next time... can copy iamb?
  • nixflix?
  • remove everything from pi-box

Nix evaluations again...

right back here agin, so it turns out? this mess of eval / inspect etc...

commandnotesformat
:lf .loads flake into current replflake OUTPUTs
outputsits just the same as :lf ., just have to prefix with outputs.flake outputs