This is a list of things I learned over time. I mainly created it for my own reference, but maybe others find it useful too.
2025-11-30: git add with the --edit option
Today I learned that git add has a --edit (or -e) option that allows you to edit the patch to be applied, i. e. to remove changes that you don't want to add. This is really helpful for me as I have a tendency to change more than one thing at a time, but I still want each of my commits to contain one change only.
2025-12-06: Dump data read / written by a program (from / to files or sockets) with strace
Today I learned that strace lets you dump data (as hex) read / written by a program (from / to files or sockets). With the --read and --write options you can specify a set of file descriptors to dump (after you've found out which file descriptors you're interested in with e. g. lsof orsysdig`).
Very useful for debugging / reverse engineering :-) For example, if you want to dump all the data a network server exchanges with its client(s), you can use a command like this:
strace -qq -p <server PID> --read=<fd(s) of interest> --write=<fd(s) of interest> -o >(grep -P '^ \|' | tee /tmp/data.hex)
If need be, you can use xxd -r to convert the hex dump back to binary data.
2025-12-30: DuckDB and jc
Today I learned that there is https://duckdb.org/ that allows you to query, among other things, JSON data with SQL. So you can do for example something like this:
journalctl -o json | duckdb -c "select SYSLOG_IDENTIFIER, count(*) from read_json('/dev/stdin') group by 1 order by 2 desc;"
As someone who's been using SQL for more than 30 years, I really like this, especially in combination with https://github.com/kellyjonbrazil/jc
2026-01-11: File system of a container visible at /proc/<PID>/root
Today I learned that you can access the file system of a container on Linux via the path /proc/<PID>/root, where PID is the id of the process that runs in the container (can be retrieved with docker / podman inspect -f '{{.State.Pid}}' <id or name of the container>).
No need to start a shell in the container, which is not possible with distroless containers anyway.
2026-03-06: Creating a Kerberos keytab file for Wireshark
I've long been interested in Kerberos, the protocol that Microsoft's Active Directory uses for authentication. In an attempt to really understand how it works I decided to write my own (very simple) Key Distribution Center in Python (you can find the code here). Naturally I had to do some debugging and of course I used Wireshark for that. But there are some parts of the Kerberos protocol messages that are encrypted and can only be read by someone who possesses the corresponding keys. As it turned out, Wireshark can import and use such keys if they are stored in a so-called Kerberos keytab file. You can create such a file as follows. Note that this only works with the MIT version of Kerberos that Linux uses, not with the Heimdal version that comes with macOS.
- Start the
ktutil program in interactive mode (it then shows the prompt ktutil:).
- Enter the command
add_entry -password -p <principal> -k 1 -e <encryption type> to create the key for the specified Kerberos principal (derived from the principal's password) with the specified encryption type, e. g. add_entry -password -p krbtgt@CWTEST.LOCAL -k 1 -e aes256-cts-hmac-sha1-96. The command asks you for the principal's password.
- Write the created key(s) to a keytab file with the command
write_kt <file>
To import this file in Wireshark go to Preferences -> Protocols -> KRB5, check the option Try to decrypt Kerberos blobs and enter the file path. That's it.
2026-03-06: Trace mode for Kerberos commands
All the Kerberos commands, like kinit, support a trace mode that can be enabled by setting the environment variable KRB5_TRACE to the file the trace output should be written to, e. g. /dev/stderr. This is really helpful as the normal output of these commands can be misleading at times, e. g. kinit reports a wrong password when in fact the TGT returned by the KDC exceeds the requested life time.
2026-09-07: Running commands in a container which it doesn't have
What if you want to run a command in a container, but the container doesn't have this command? If it's your own container image, you could build a new image with this command included, at least in theory. In reality this might not be so easy (for example, due to processes or restrictions in production environments) or not something you're willing to do if you need this command only once in order to debug something. nsenter to the rescue - with nsenter you can run a command in one or more namespaces of a certain process. If this process runs in a container, you basically run the command in this container. Unless you also enter the mount namespace, the command's executable doesn't need to exist in the container image, it's taken from the root namespace - the host where the container runs on. See here for more details.
So, for example, if you'd like to capture the network traffic in any container, including distroless ones, just run this command. Note that you need the PID on the host, not in the container.
nsenter --net --target <PID on the host> -- tshark -i eth0
I find this really helpful when I want to see what's actually going on in a container while debugging a problem.
2026-09-07: Listing processes in a container without ps
Unfortunately the nsenter trick above doesn't work with ps. The problem is that ps reads the process information from the /proc file system. So if you don't enter the mount namespace, you get the process list of the host, not the container. If you do enter the mount namespace, you're back to the original problem, namely that the container doesn't have ps. Luckily there is a solution - just access the container's /proc file system directly, like ps does. The command below lists all processes with their complete command line (tr is necessary because the /proc file system uses null bytes instead of spaces as separator).
for proc in /proc/[0-9]*; do printf "${proc#/proc/} "; cat $proc/cmdline | tr '\0' ' '; printf "\n"; done