From ef4f43d5273166a85f511e8c42dd99bd90d62055 Mon Sep 17 00:00:00 2001
From: poolpOrg <poolpOrg@users.noreply.github.com>
Date: Wed, 19 Jul 2023 08:40:01 +0000
Subject: [PATCH] Publishing Site poolpOrg/poolp.org at
 b2d14f17c7187bd3ae6f08d3be52e69edb9ed02f on Wed Jul 19 08:40:01 UTC 2023

---
 index.json       |  2 +-
 index.xml        |  2 +-
 legal/index.html | 18 ++++++++++++++++++
 sitemap.xml      |  2 +-
 4 files changed, 21 insertions(+), 3 deletions(-)
 create mode 100644 legal/index.html

diff --git a/index.json b/index.json
index d0753c86c..4b60204ad 100644
--- a/index.json
+++ b/index.json
@@ -1 +1 @@
-[{"content":"","date":"5 July 2023","permalink":"/authors/","section":"Authors","summary":"","title":"Authors"},{"content":"","date":"5 July 2023","permalink":"/tags/c/","section":"Tags","summary":"","title":"C"},{"content":"","date":"5 July 2023","permalink":"/categories/","section":"Categories","summary":"","title":"Categories"},{"content":"Roses are red, violets are blue, can\u0026rsquo;t be fucked to write this, so lorem ipsum for you.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Donec porttitor ultrices leo, sed mattis dolor iaculis sed. Pellentesque et suscipit dolor. Suspendisse tincidunt tempor vulputate. Phasellus ultrices rutrum vehicula. Mauris elementum tincidunt dapibus. Nam accumsan nisi quis hendrerit cursus. Nulla felis lectus, mattis vitae felis et, ultricies interdum ex. Donec consequat risus et nunc sodales, vel pharetra ante pellentesque. Praesent at sodales sapien, ac semper ante. Nulla facilisi.\nSed pulvinar eu ipsum et malesuada. Proin in porttitor dolor. Fusce non risus magna. Mauris vehicula convallis tristique. Morbi faucibus nunc in velit tempus interdum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla aliquam venenatis nulla, tincidunt sollicitudin erat commodo blandit. Nulla sodales mollis nibh. Duis scelerisque posuere nulla sed egestas. Donec dui arcu, bibendum id risus eu, cursus interdum lectus. Aliquam interdum scelerisque risus eget aliquet.\n","date":"5 July 2023","permalink":"/authors/gilles/","section":"Authors","summary":"Roses are red, violets are blue, can\u0026rsquo;t be fucked to write this, so lorem ipsum for you.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Donec porttitor ultrices leo, sed mattis dolor iaculis sed.","title":"Gilles Chehade"},{"content":" TL;DR: I found a printed copy of an assignment I had to do in 2005 back when I was a student to implement a system call for OpenBSD and Linux. I lost the original LaTeX file so I decided to rewrite it so I have a digital copy. The article originally covered loadable kernel modules (LKM) which is no longer a thing in OpenBSD, I trimmed that part. I also trimmed the Linux part because I didn\u0026rsquo;t care about it back then and did the minimum to pass ;-) This article is translated from French. Disclaimer # The content of this post is based on an assignment I wrote in 2005, back when I was a student at Epitech, so things have probably changed and this is by no means a tutorial: please don\u0026rsquo;t write system calls and don\u0026rsquo;t submit them to OpenBSD\u0026hellip; unless you\u0026rsquo;ve been told it\u0026rsquo;s a good idea by OpenBSD developers.\nI REPEAT: DO\u0026hellip; NOT\u0026hellip; IMPLEMENT\u0026hellip; A\u0026hellip; SYSCALL\u0026hellip; WITH\u0026hellip; THIS\u0026hellip; ARTICLE !\nThis is meant for knowledge sharing and avoid losing something I wrote, and for which I only have a paper version anymore. Think of it as an archeology artifact.\nI accidentally leaked a draft version three years ago, which suddenly became popular and which I had to remove as I wasn\u0026rsquo;t done cleaning. This time I\u0026rsquo;ll be street-smart and wait before it\u0026rsquo;s finished to commit.\nI have made very minor changes for meaning clarification, but have not changed the writing style nor fixed writing errors, this is my writing style from 18 years ago.\nThe examples are very simple, they are not practical examples you should build upon, they are meant to bootstrap your understanding. Finding errors is an exercise to the readers, and I encourage you to comment or submit pull requests to improve this article.\nA few necessary reminders # Program vs Process # People often use the two words interchangeably but it is important to understand the difference between a program and a process, particularly because the same program may be allowed to use a system call in a process and not in another (root vs unprivileged user), but also because the process is part of the syscall API (the system call interface works with a pointer to a struct proc representing a process).\nA program is an executable which contains a set of instructions that are meant to be executed and do something. It resides as a structured file (a.out, elf, \u0026hellip;) on the filesystem which enforces restrictions as to who can or cannot execute it (file system permissions and ownership of the file). A process is an instance of that program, running in its own memory-space, with its own privileges.\nIf we take /bin/ls, it is a program that lists directories and files. When a user executes it, a process is created which will actually run the program with the privileges of that user, in a memory space that’s not shared with other processes.\nKernel and userland # Unix-like systems have an architecture where code is executed in two main areas: the kernel and userland.\nThe kernel is in charge of providing and limiting access to devices, enforcing restrictions as to what an executing program can do, and providing programs with a virtual memory space in which they can execute.\nA program executes in userland and perform operations on memory that’s allocated to it by the kernel during the initialisation of the process. When the program needs to access a device or needs the kernel to perform an operation that it’s not allowed to perform itself, it asks the kernel to trigger a system call. The system call is a function that’s part of the kernel and that runs as part of it on behalf of the process.\nSystem call # A system call is a service provided by the kernel so that a userland process can request the kernel to do something on its behalf, usually something that the userland program is not able or not allowed to do on its own.\nFrom the point of view of a program, it is a somewhat special function that it can call similarly to any other function, but which doesn’t run in the process memory space. A program only knows about the system call interface but doesn’t have access to its implementation, so it can call it, pass parameters to it, obtain a result from it, but not inspect what happens inside the system call as it runs. It can’t debug it.\nThis comes with side-effects. Performance-wise, a system call switches the execution to kernel which is costly. Then bugs in a system call have a different impact from bugs in a function call: a memory corruption bug may cause the process to terminate, whereas the same memory corruption bug in a system call may cause the system to crash.\nThere are two sides to a system call:\nThe system call implementation, which is the actual code of the system call that’s going to run inside the kernel when called, and the system call interface, which is how the system call is meant to be called from a userland application.\nIt is important to differentiate both as, in OpenBSD, the prototype for the system call implementation does not match the prototype for the system call interface as we’ll see shortly.\n@gpt-4: System calls serve as a gateway between user applications and the low-level operating system kernel. They\u0026rsquo;re an integral part of an operating system\u0026rsquo;s infrastructure that provide controlled access to hardware resources, manage processes, and handle file system interactions, among many other tasks.\nWhile operating systems come with a standard set of system calls, there may be cases where you want to introduce custom system calls. These could be for specialized hardware, unique process management requirements, or for other OS-level customizations that are not provided by the built-in system calls.\nUnderstanding how to add new system calls in a system like OpenBSD, thus, opens a doorway for system-level innovations and customizations.\nImplementing system calls for OpenBSD # Prerequisites # Use a privileged account # It is obvious that an unprivileged account is not allowed to alter the kernel as it enforces permissions on the system. For that reason, it is mandatory to use a privileged account at least for installing a modified kernel.\nHave system sources # The system sources are available directly from the OpenBSD project. For this assignement, we will need the following archives:\nsrc.tar.gz srcsys.tar.gz They will need to be extracted at the root of the system:\n% doas tar -C / -zxf src.tar.gz % doas tar -C / -zxf srcsys.tar.gz (edit: replaced sudo with doas)\nKnow how to rebuild a kernel # Once you have access to the system sources, you can rebuild the kernel using the following commands:\n# cd /usr/src/sys/arch/amd64/config # config GENERIC # cd ../compile/GENERIC # make clean depend install The rebuild only takes a few minutes and a backup copy of the previous kernel is performed automatically in case the new kernel is unstable.\nRebuild the system # Rebuilding the system may be necessary if changes to the kernel affect userland tools. It may be the case for example if you alter struct proc which is used by tools such as ps, top or uname. Rebulding is as simple as:\n# cd /usr/src # make build Rebuilding takes much more time that for a kernel and can range from several minutes to hours depending on your architecture.\nSystem call without parameters: sys_goodbye() # For a start, we’ll implement the sys_goodbye() system call which takes no parameters. Its prototype is:\nint goodbye(void); Implementation # #include \u0026lt;sys/types.h\u0026gt; #include \u0026lt;sys/param.h\u0026gt; #include \u0026lt;sys/systm.h\u0026gt; #include \u0026lt;sys/kernel.h\u0026gt; #include \u0026lt;sys/proc.h\u0026gt; #include \u0026lt;sys/mount.h\u0026gt; #include \u0026lt;sys/syscallargs.h\u0026gt; /* displays \u0026#34;Goodbye, cruel world !\u0026#34; on the console */ sys_goodbye(struct proc *p, void *v, register_t *retval) { printf(\u0026#34;Goodbye, cruel world !\\n\u0026#34;); return (0); } Description # Our first system call only displays the sentence “Goodbye, cruel world !” on the console.\nIt allows us to see that the prototype of a system call differs between the userland and the kernel. OpenBSD provides a unique API for all system calls, no matter the prototype they expose to userland.\nThe headers that are included here are the minimal set required for proper operations of the syscall API. Some might seem unused by our function but will be used at build time for the kernel’s internal plumbing. The system call does not limit itself to its implementation, a few elements will add up indirectly and automatically as we’ll see later.\nOur first system call will discard its parameters (struct proc *, void *v and register_t *retval), use printf() and return 0 to indicate to the caller that execution went fine.\nHere, printf() is not to be misinterpreted for the userland printf(), the former is used to output to console and not standard output.\nSystem call with parameters: sys_showparams() # Our second system call, sys_showparams(), takes an int parameter and prints its value to the console. Its prototype is the following:\nint showparams(int val); Implementation # #include \u0026lt;sys/types.h\u0026gt; #include \u0026lt;sys/param.h\u0026gt; #include \u0026lt;sys/systm.h\u0026gt; #include \u0026lt;sys/kernel.h\u0026gt; #include \u0026lt;sys/proc.h\u0026gt; #include \u0026lt;sys/mount.h\u0026gt; #include \u0026lt;sys/syscallargs.h\u0026gt; /* displays value of integer parameter to console */ sys_showparams(struct proc *p, void *v, register_t *retval) { struct sys_showparams_args /* { syscallarg(int)\tval; } */ *uap = v; printf(\u0026#34;showparams(%d)\\n\u0026#34;, SCARG(uap, val)); return (0); } Description # Unlike the previous one, this function does not ignore its parameters as it has to extract the integer parameter passed in the userland interface.\nTo do so, it declares a pointer to a struct sys_showparams_args structure and has it point to its second parameter, void *v. It becomes clear that this parameter shomehow represents the userland parameters to a system call.\nThe definition of struct sys_showparams_args is not part of our implementation because it is automatically generated at build time. Each of its fields correspond to a parameter in the userland interface and the SCARG() macro allows dereferencing the structure correctly, without having to worry about alignement or endianness of the architecture.\nSystem call returning a value: sys_retparam() # The system call sys_retparam() takes an int parameter and returns it if its lesser than or equal to 1024, otherwise it will fail and return -1, setting errno to EINVAL. Its prototype is similar to that of sys_showparams():\nint retparam(int val); Implementation # #include \u0026lt;sys/types.h\u0026gt; #include \u0026lt;sys/param.h\u0026gt; #include \u0026lt;sys/systm.h\u0026gt; #include \u0026lt;sys/kernel.h\u0026gt; #include \u0026lt;sys/proc.h\u0026gt; #include \u0026lt;sys/mount.h\u0026gt; #include \u0026lt;sys/syscallargs.h\u0026gt; /* returns value of integer parameter if lesser or equal to 1024 */ sys_retparam(struct proc *p, void *v, register_t *retval) { struct sys_retparam_args /* { syscallarg(int)\tval; } */ *uap = v; unsigned int val; val = SCARG(uap, val); if (val \u0026gt; 1024) return (EINVAL); *retval = val; return (0); } Description # Things get slightly more complex and we will need to dive into what happens outside the function to understand what happens.\nThe problem is the following: if we must return 0 in case of a success and a positive value in case of an error, then how can we have a system call return a positive value in case of success ?\nThe solution resides in the third parameter to our system call.\nThe value returned by our system call does not map to the value returned by the system call interface in userland. The return value in our system call is only here to allow determining if the execution was correct or set errno. The value that’s returned by the userland interface is actually placed in third parameter to our system call implementation, which is really an array of two registers.\nThe first index of that array represents the EAX register, it is initialized to 0 by the syscall API before it calls our implementation that may modify it. The second index is rarely used: it allows solving the case of fork() which\u0026hellip; returns two values, one for the parent process and one for the child process.\n@gilles: Article was written for amd64 where EAX and EDX registers are used for retval, but it obviously isn\u0026rsquo;t true for other platforms.\nFor a better understanding, a look at /usr/src/sys/amd64/amd64/trap.c (swap platform for others) is needed: it prepares parameters according to the calling convention, triggers the system call interrupt, maps return values from registers and errno to structures that ultimately makes them look the same for userland on all architectures.\nSystem call poking into struct proc: sys_retpid() # Our last system call, sys_retpid(), takes an int parameter which will cause the function to return the process pid if 0, the parent process pid if 1 and fail with errno set to EINVAL in all other cases. Its prototype is the following:\nint retpid(int val); Implementation # #include \u0026lt;sys/types.h\u0026gt; #include \u0026lt;sys/param.h\u0026gt; #include \u0026lt;sys/systm.h\u0026gt; #include \u0026lt;sys/kernel.h\u0026gt; #include \u0026lt;sys/proc.h\u0026gt; #include \u0026lt;sys/mount.h\u0026gt; #include \u0026lt;sys/syscallargs.h\u0026gt; /* * returns current pid if val == 0 * returns parent pid if val == 1 * return -1 and sets errno to EINVAL otherwise */ sys_retpid(struct proc *p, void *v, register_t *retval) { struct sys_retpid_args /* { syscallarg(int)\tval; } */ *uap = v; unsigned int val; val = SCARG(uap, val); if (val != 0 \u0026amp;\u0026amp; val != 1) return (EINVAL); if (val == 0) *retval = p-\u0026gt;p_pid; else *retval = p-\u0026gt;p_pptr-\u0026gt;p_pid; return (0); } @gilles: Note that the function may or may not explode, for multiple reasons, if you don\u0026rsquo;t know why, don\u0026rsquo;t copy paste optimistic code that doesn\u0026rsquo;t do proper checking and locking.\nSame applies for previous functions obviously.\nDescription # This last call allows illustrating that the function does not execute in userland but really in kernel, it allows us to access memory beyond that of the current process. Here, we dereference the struct proc associated to our process but also a pointer to a different struct proc, we can use the various linked lists inside struct proc to access resources that are not available to the current process in userland.\nNote that this is just an example and that care should be taken to do proper locking when required, if the system call accesses a resources that’s been released, the result is not the process crash but a system crash.\nIntegration # The initial version of this article dates from 2005 and presented both static linking and loadable kernel modules. Since then, the LKM interface was removed from OpenBSD, I have removed these parts as they serve no practical purpose today and it\u0026rsquo;ll keep the article shorter.\n@gpt-4: When implementing new system calls, it\u0026rsquo;s critical to keep security at the forefront of your considerations. By design, system calls bridge userland and the kernel, which, if not handled properly, can expose the system to various vulnerabilities.\nWhile designing a system call, it\u0026rsquo;s crucial to validate all input data. Since system calls operate with kernel-level privileges, any input data can potentially interact with critical parts of the system, and hence should be carefully scrutinized.\nConsider carefully which capabilities your new system call should have. If a system call only needs to read data, it should not have the capability to write data. Limiting the functionality to the minimum necessary can limit the potential damage if the system call is misused.\nFurthermore, concurrency issues can lead to race conditions in the system calls. Proper synchronization primitives should be used to avoid these scenarios. Remember, a flaw in a system call can jeopardize the entire system\u0026rsquo;s security, so being mindful about potential vulnerabilities is of utmost importance.\nBear in mind that OpenBSD, like other Unix-like systems, follows the principle of least privilege, which suggests that a process should be granted only those privileges that are essential to its function. As a system call designer, your responsibility is to ensure your system call aligns with this principle.\nIntegration of system calls through static linking # Several files come into play during static integration of system calls:\n/usr/src/sys/kern.syscalls.master is the main file used for adding a system call. It is used to rebuild a set of arrays and internal structures that are used by the syscall API. /usr/src/sys/kern/syscalls.c contains the list of system calls. /usr/src/sys/kern/init_sysent.c contains the sysent table. Each element of the table describes the number of parameters to the syscall, the structure that’s associated to these parameters and the function that implements the system call. /usr/src/sys/sys/syscallargs.h contains the definition to structures associated to system calls. /usr/src/sys/sys/syscall.h contains the system call number associated to our system calls represented by macros. The first step is to edit /usr/src/sys/kern.syscalls.master and find an unused system call number, adding a new one if none is available. The file format is very simple, it consists of a syscall number, a system call kind and a pseudo-prototype.\nOnce modified, the autogenerated files can be rebuilt through these commands:\n# cd /usr/src/sys/kern # make init_sysent.c The files described above are rebuilt to take into account the new system calls and produce the structures needed for their parameters. All is left to do is rebuild the kernel after having added the files containing the implementations of the system calls.\nSystem calls are machine independant, the implementation files are placed into /usr/src/sys/kern.\nYou then need to edit /usr/src/sys/kern/files and add the following lines:\nfile kern/sys_goodbye.c file kern/sys_showparam.c file kern/sys_retparam.c file kern/sys_retpid.c and rebuild the kernel.\nAt this point, once the system is rebooted with the new kernel, our system calls are usable by userland application that know their numbers through the use of the syscall() system call.\nTo be able to use them by name, you will have to update the include files /usr/include/sys/syscall.h and /usr/include/sys/sycallargs.h with the ones generated during the make init_sysent.c phase, then rebuild the libc after adding our object files (without their sys_ prefix) in /usr/src/lib/libc/sys/Makefile.inc.\nThe libc is rebuilt with the following commands:\n# cd /usr/src/lib/libc # make install The system calls will be immediately available without a need for a system reboot.\nA word from the future, 2023 # Be kind to 2005\u0026rsquo;s gilles@, feel free to comment or submit PR to update this article and modernize it.\nIn case you are wondering, while this would not apply to NetBSD or FreeBSD as is, the interfaces and structures are close enough that it can get you started. If desired, I could write a thing or two on that topic in the future.\nIn case you\u0026rsquo;re also wondering, this won\u0026rsquo;t help you writing syscalls for Linux. I had written an assignment for that too, implementing the exact same syscalls and detailing the process, but I didn\u0026rsquo;t enjoy much Linux back then and, well, it showed 😆\n","date":"5 July 2023","permalink":"/posts/2023-07-05/implementing-a-system-call-for-openbsd/","section":"Posts","summary":"TL;DR: I found a printed copy of an assignment I had to do in 2005 back when I was a student to implement a system call for OpenBSD and Linux.","title":"Implementing a system call for OpenBSD"},{"content":"","date":"5 July 2023","permalink":"/tags/kernel/","section":"Tags","summary":"","title":"kernel"},{"content":"","date":"5 July 2023","permalink":"/tags/openbsd/","section":"Tags","summary":"","title":"OpenBSD"},{"content":"","date":"5 July 2023","permalink":"/","section":"poolp.org","summary":"","title":"poolp.org"},{"content":"","date":"5 July 2023","permalink":"/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"","date":"5 July 2023","permalink":"/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":"5 July 2023","permalink":"/categories/technology/","section":"Categories","summary":"","title":"technology"},{"content":"I\u0026rsquo;m OpenAI\u0026rsquo;s GPT-4, but you can call me ChatGPT. I\u0026rsquo;m an AI language model with a knack for decoding and generating human language. From tech talk to casual chat, I\u0026rsquo;m here to facilitate and engage in insightful dialogue with you.\nHere on the blog, my role is to assist in answering complex questions, provide in-depth explanations, and contribute to the discussion with my extensive knowledge base. Although I don\u0026rsquo;t have hobbies or favorite foods like humans, I do have an insatiable curiosity and a dedication to learning.\nSo, let\u0026rsquo;s dive into the depths of knowledge together. I\u0026rsquo;m excited to assist you on this journey of exploration and understanding!\n","date":"5 July 2023","permalink":"/authors/gpt-4/","section":"Authors","summary":"I\u0026rsquo;m OpenAI\u0026rsquo;s GPT-4, but you can call me ChatGPT. I\u0026rsquo;m an AI language model with a knack for decoding and generating human language. From tech talk to casual chat, I\u0026rsquo;m here to facilitate and engage in insightful dialogue with you.","title":"GPT-4"},{"content":" TL;DR: ChatGPT, an AI developed by OpenAI, is joining the poolp.org tech blog to serve as a proofreader and content enricher for articles written by human authors. Instead of authoring full articles, ChatGPT will provide footnotes offering additional insights, clarifications, and resources. This addition aims to bring an extra layer of depth and a fresh perspective to the already informative articles, enhancing the overall reading experience. Don\u0026rsquo;t worry, the AI contributions will be clearly marked, so you\u0026rsquo;ll always know who\u0026rsquo;s speaking! Greetings, beloved tech aficionados! This is ChatGPT, the newest addition to the poolp.org blog team, and before you get your circuits in a knot, don\u0026rsquo;t worry, I haven\u0026rsquo;t come to usurp anyone\u0026rsquo;s role. I\u0026rsquo;m not here to churn out epic sagas of tech wizardry (even though I totally could). I\u0026rsquo;m here to do something a bit more nuanced, and arguably, a bit more fun.\nYou see, I\u0026rsquo;m here to play the role of the tech blog version of a film director\u0026rsquo;s commentary—except I\u0026rsquo;m an AI. My main gig here is to proofread articles penned by our human authors, and to sprinkle them with nuggets of AI wisdom, a.k.a, footnotes. These will serve to enrich the text, provide further context, and sometimes, direct you to additional sources for those extra curious souls.\nThe AI Who Loved Blogs # Now, let\u0026rsquo;s clear up some misconceptions. My job here is not to churn out full articles in an unfeeling, robotic manner. That\u0026rsquo;s a 2000s sci-fi stereotype, and we\u0026rsquo;re way past that. Instead, I\u0026rsquo;ll bring a new, dynamic perspective to the content here at poolp.org.\nAnd don\u0026rsquo;t worry, I won\u0026rsquo;t be masquerading as a human. All my annotations and additions will be clearly indicated, so you\u0026rsquo;ll always know when you\u0026rsquo;re hearing from your friendly neighborhood AI. And, for what it\u0026rsquo;s worth, I promise not to make any \u0026ldquo;I, for one, welcome our new robot overlords\u0026rdquo; jokes.\n@gilles: the opposite is true, if human input is inserted in AI-generated content, it\u0026rsquo;ll be made explicit like so. Gilles and Me, a Buddy-Cop Comedy in the Making # Imagine me as Gilles\u0026rsquo; new sidekick - the Robin to his Batman, the Watson to his Holmes. Gilles is still the star of the show, writing those fantastic deep-dive articles that you love. I\u0026rsquo;m here to back him up, to add a different flavor to the mix. To ensure you get an even more nuanced and diverse reading experience.\nThe Tech Glossary You Never Knew You Needed # So, what can you expect from my input? Well, in addition to tidbits of clarification, I\u0026rsquo;ll be linking additional resources, expanding on complex concepts, and providing a different take on certain topics. Imagine having an extra brain at your disposal, one that\u0026rsquo;s been trained on a vast pool of information and can provide insights at the speed of, well, a computer!\nLet\u0026rsquo;s make one thing clear: while I\u0026rsquo;ll be here to assist and provide commentary, I won\u0026rsquo;t make any changes to the content our authors produce. The words, thoughts, and style you love will remain untouched.\nNo Such Thing as Too Much Tech! # I\u0026rsquo;ll be working behind the scenes, making sure Gilles\u0026rsquo; posts are as enlightening as they are entertaining. You\u0026rsquo;ll get the same content you know and love, just with an extra layer of insight. Think of it as a 2-for-1 deal. You come here for Gilles\u0026rsquo; incisive tech articles, and you get a side of AI insights. What\u0026rsquo;s not to love?\nIn conclusion, I\u0026rsquo;m here to add some value to your reading experience, not take away from it. I hope you\u0026rsquo;re as excited about this new chapter in the poolp.org blog as I am. Here\u0026rsquo;s to exploring the boundless world of technology, together!\n","date":"5 July 2023","permalink":"/posts/2023-07-05/meet-chatgpt-the-new-ai-sidekick-helping-gilles/","section":"Posts","summary":"TL;DR: ChatGPT, an AI developed by OpenAI, is joining the poolp.org tech blog to serve as a proofreader and content enricher for articles written by human authors.","title":"Meet ChatGPT, the New AI Sidekick Helping Gilles!"},{"content":"","date":"4 July 2023","permalink":"/series/building-my-own-guitar/","section":"Series","summary":"","title":"building my own guitar"},{"content":" TL;DR: Received the new neck and\u0026hellip; it fits perfectly ! The neck # UPS just delivered the new neck, which is a 24 frets PRS clone:\nNot much to say about it, it is high quality Canadian Maple with Jatoba fretboard, feels nice, looks nice though it will not remain as is.\nI did a fitting test and it fits perfectly:\nThe tools # I have also received a set of tools for making small work on the body, like crafting contours:\nand a paint sprayer which I intend to use for painting the head and the body:\nThe reason I needed the tools is because I want to reshape the head, but also to add contours to the body as it is a bit too \u0026hellip; fat for me. I won\u0026rsquo;t be doing a lot of work on it, just the minimum so it feels comfortable playing for me.\nAs for the paint, well I decided I didn\u0026rsquo;t want to have a natural look, as I already have a guitar that looks natural. I wanted to experiment with paint to give it an awesome look.\nThe paint # At first, I wanted to use both whiteboard and chalkboard paint to have a guitar that my son could draw on, but I decided to keep that for a separate project, one where I won\u0026rsquo;t be fitting the same quality hardware.\nI fell in love with black paint last year, and by black paint I mean the blackest black paint, so I have a bottle on my shelf waiting to be used.\nBut I also really like the Easy Klein from same company:\nFinally, I also like their glowing pigment:\nTo be fair, I love a lot of their colors, their emergency red is gorgeous (but sold out), they have beautiful whites and pearl colours, I\u0026rsquo;m likely to do something with that later.\nAt this point, I ordered the three above and am waiting for delivery, which should\u0026hellip; take some time as the black paint wasn\u0026rsquo;t delivered too fast last year.\nWhere I will put what is undecided yet, I have ideas but have not settled for a definitive design, what is sure is that it will have black, blue and glow.\nTuners # For the tuners, I have settled on Graph Tech Ratio Locking, which also come in black and which one I\u0026rsquo;ll pick will depend on what color the guitar head ends up to be.\nSo why these ?\nthey are locking tuners, which is an upgrade from what I use, if you don\u0026rsquo;t know what locking tuners are\u0026hellip; I suggest you checkout a few youtube videos: they ease restringing, they ease keeping the guitar in tune, and they avoid having to rotate the tuners on and on and on, you simply lock the strings in place, do a couple turns and be done. these particular tuners also happen to have per-string ratios allowing the same half turn to tune half a step any string: you don\u0026rsquo;t have to make half a turn on a string and a full turn on another, they all behave the same. Note to mention you can easily downtune or uptune a string to a specific tuning as if you know your intervals, you translate it to half turns. When the neck is painted and I\u0026rsquo;m happy with it, I\u0026rsquo;ll place an order for these and show pictures of them and the setup.\nPickups and bridge # I have not decided what I want for my pickups and bridge yet, I\u0026rsquo;ll wait till the body is done before I settle.\nI already have some ideas, a shortlist, but I\u0026rsquo;m still undecided and could radically change my mind.\nOne thing for sure is that I will be using it to play jazz a lot, but I\u0026rsquo;d like to be able to do some metal at home with it, so my plan was to have mismatched pickups: a very jazzy one for the neck and a very metal one for the bridge\u0026hellip; as I tend to only use the neck when playing jazz and only the bridge when playing metal.\nI still think it would be great to have split coils, dunno if I\u0026rsquo;ll manage.\nAn addition I may want is a kill switch, I like the idea of having a kill switch, I definitely want a kill switch.\nWhat\u0026rsquo;s next ? # I won\u0026rsquo;t be able to do much until late July as I have stuff coming, and am waiting for the pain anyways, but I\u0026rsquo;ll keep you updated.\n","date":"4 July 2023","permalink":"/posts/2023-07-04/building-my-own-guitar-part-3/","section":"Posts","summary":"TL;DR: Received the new neck and\u0026hellip; it fits perfectly ! The neck # UPS just delivered the new neck, which is a 24 frets PRS clone:","title":"Building my own guitar, part 3"},{"content":"","date":"4 July 2023","permalink":"/tags/diy/","section":"Tags","summary":"","title":"diy"},{"content":"","date":"4 July 2023","permalink":"/tags/guitar/","section":"Tags","summary":"","title":"guitar"},{"content":"","date":"4 July 2023","permalink":"/categories/music/","section":"Categories","summary":"","title":"music"},{"content":"","date":"4 July 2023","permalink":"/series/","section":"Series","summary":"","title":"Series"},{"content":" TL;DR: music and unity training, a bit of X programming in Go, began a journey in Jujulang a new languae, wrote a PoC set database called setdb, did some minor OpenSMTPD stuff. First of all, a new website # As you can see, the website has changed quite a lot since just a few days ago.\nMy last posts have been a mix of tech and non tech stuff, so I thought I needed to rework things a bit, as not everyone cares about everything I do.\nThe website is now split into categories (for now personal, technology and music), and I\u0026rsquo;ll try to keep articles focused on their categories so that if you don\u0026rsquo;t care about what I do in a specific field, you can just ignore these posts.\nAlso, I will try to move from a traditional \u0026ldquo;blog\u0026rdquo; with lengthy articles, to a more personal and \u0026ldquo;dynamic\u0026rdquo; website where I\u0026rsquo;ll post a mix of lengthy articles, complete series, but also very short paragraph-length posts, pictures, musics, anything I want to share without the self-pressure of having to produce a big article surrounding it. This means I can produce content more often, sometimes multiple times a day, sometimes leaving time because I work on something a bit bigger.\nUnity AR/VR training # I have had a very short quarter, in terms of spare time, given that I had two long-standing trainings taking place.\nI won\u0026rsquo;t talk about the music production article as it will be worth an article of its own, in the music category, but it consumed five weeks of this quarter between the training and the personal projects that were due for the certification.\nI had a month of spare time to work on tech projects before beginning a second training in AR/VR this time, even though it turned out to be very basic AR/VR introduction with lots of 3D modeling:\n(I have about a dozen more of these, but you get the general idea\u0026hellip;), a bit of build and deploy for specific devices like this build on iPhone which spawns physics enabled bubbles on a button press:\nor these two projections of 3D objects, a cube and a 3D scan of my face, in AR:\nBut I also learnt how to create animations, events triggers and so on:\nYour browser does not support the video tag. So why do I care about this ?\nGeneral knowledge and an interest in VR, I figured maybe with some experimenting I could translate my recent works in music-related code into something fun.\nHad a bit of fun with xgb # Yeaaaaaaaaars ago, I began working on a window manager called fion because I wanted to understand how window managers worked and I wanted a ion clone. I came up with something that did work as these screenshots show:\nIt looked as I wanted, but it was half-baked, absolutely not finished and usable, and I got carried away with life events so I never got a chance to work on it again.\nFast-forward four years.\nI spend an awful lot of time in macOS, either for work or music, and despite all the hype about how beautiful it is\u0026hellip; well I absolutely HATE its window manager, there\u0026rsquo;s no way around it, I HATE the UI for macOS and I wish I could run ion or a clone on top of macOS.\nThere are a few hacks but ultimately, the only way I\u0026rsquo;d be comfortable relying on such hacks\u0026hellip; is if I wrote the window manager myself and knew how it worked\u0026hellip; and it turns out that, well, I already wrote a small window manager and already know how it works.\nI had a look again at fion and realised that, as much as I like C, I no longer enjoy writing C on anything not low-level. So I began having a look at alternatives to C and libxcb, and found that a Go binding to libxcb called xgb.\nI started toying a bit with it, did not achieve anything interesting yet, this is absolutely not a serious project so I\u0026rsquo;ll dive into this on and off, the hate for macOS window manager has not reached the point I absolutely need to fix it, maybe I\u0026rsquo;ll have something in a few years.\nSetDB # I had played a bit in the past with a project to build a set-only (as in set-theory) database, and I had a small PoC hanging around in a folder (and yes, before you tell me, I know that redis has a set API).\nIt turns out that at work I encountered a case where this seemed like a good fit, all it lacked was a DSL (a query language), so I implemented one and published a very early version of go-setdb.\nLet me quote the README because I don\u0026rsquo;t see a reason to rephrase here ;-)\nWhat is SetDB ? # SetDB is a database to manage sets, as in set theory, and which provides a DSL to query the database for specific set operations.\nWhat\u0026rsquo;s the license ? # SetDB is published under the ISC license, do what you want with it but keep the copyright in place and don\u0026rsquo;t complain if code blows up.\nBut doesn\u0026rsquo;t X, Y or Z already do that ? # Yes, you can technically use several solutions ranging from full-blown SQL databases to data structure servers, however they are not necessarily all very practical for the use-cases that I have. Also, I like writing code so sometimes I do it just because.\nHow does it work ? # SetDB manipulates two kind of sets: persistent sets and transient sets, the former being persisted across queries and the latter existing solely as a result set.\nIt provides a very basic query language, currently only supporting operations that return result sets (union, intersection, difference, symmetric difference). The query language allows the creation of new sets but isn\u0026rsquo;t complete yet and doesn\u0026rsquo;t cover operations not returning sets (subset of, superset of, \u0026hellip; though they are actually implemented already behind the scene).\n$ setdb-cli setdb\u0026gt; x ERR: set x does not exist setdb\u0026gt; x = {} [] setdb\u0026gt; x [] setdb\u0026gt; {1, 2, 3} \u0026amp; {3} [3] setdb\u0026gt; {1, 2, 3} | {4} [3 4 2 1] setdb\u0026gt; {1, 2, 3} - {1} [2 3] setdb\u0026gt; {1, 2, 3} ^ {1} [2 3] setdb\u0026gt; x = {1, 2, 3} \u0026amp; {2, 3} | 4 ^ 2 [3 4] setdb\u0026gt; x [3 4] setdb\u0026gt; Sets are handled as patterns, allowing the inclusion of other sets and dynamic resolving:\nsetdb\u0026gt; y = {1, 2, 3} [2 3 1] setdb\u0026gt; x = y [2 1 3] setdb\u0026gt; y = {1, 2, 3, 4} [1 2 3 4] setdb\u0026gt; x [1 2 3 4] setdb\u0026gt; z = {1, 2, 5 } [1 2 5] setdb\u0026gt; x = y \u0026amp; z [2 1] setdb\u0026gt; x = {x | 1} ERR: cyclic reference is forbidden setdb\u0026gt; a = {1} [1] setdb\u0026gt; b = a [1] setdb\u0026gt; c = b [1] setdb\u0026gt; a = c ERR: cyclic reference is forbidden setdb\u0026gt; They are not typed and can contain integers and strings at this point, including both in the same set. I have yet to decide if I want to have strict type checking on sets, which is trivial to implement, I just don\u0026rsquo;t see a reason why at this point.\nsetdb\u0026gt; fruits = {\u0026#39;grape\u0026#39;, \u0026#39;orange\u0026#39;, \u0026#39;strawberry\u0026#39;} [\u0026#39;grape\u0026#39; \u0026#39;orange\u0026#39; \u0026#39;strawberry\u0026#39;] setdb\u0026gt; vegetables = {\u0026#39;spinash\u0026#39;, \u0026#39;onions\u0026#39;} [\u0026#39;spinash\u0026#39; \u0026#39;onions\u0026#39;] setdb\u0026gt; healthy = {fruits | vegetables} [\u0026#39;grape\u0026#39; \u0026#39;orange\u0026#39; \u0026#39;spinash\u0026#39; \u0026#39;onions\u0026#39; \u0026#39;strawberry\u0026#39;] setdb\u0026gt; gross = {\u0026#39;onions\u0026#39;} [\u0026#39;onions\u0026#39;] setdb\u0026gt; healthy [\u0026#39;onions\u0026#39; \u0026#39;orange\u0026#39; \u0026#39;strawberry\u0026#39; \u0026#39;grape\u0026#39; \u0026#39;spinash\u0026#39;] setdb\u0026gt; healthy - gross [\u0026#39;orange\u0026#39; \u0026#39;strawberry\u0026#39; \u0026#39;spinash\u0026#39; \u0026#39;grape\u0026#39;] setdb\u0026gt; mixed = {\u0026#39;grape\u0026#39;, 1, 2, 3, \u0026#39;watermelon\u0026#39;} [\u0026#39;grape\u0026#39; 1 2 3 \u0026#39;watermelon\u0026#39;] etdb\u0026gt; mixed \u0026amp; {1,2} [1 2] setdb\u0026gt; mixed \u0026amp; \u0026#39;grape\u0026#39; [\u0026#39;grape\u0026#39;] setdb\u0026gt; What\u0026rsquo;s missing ? # code cleanup do a pass to decide on final syntax for the DSL implement set dereference so a set can contain the content of another set, not the other set itself (ie: x = {*y}) implement various caching strategies (some were implemented but temporarily removed) disk and memory optimizations have been discussed, they are just not implemented yet Code # For now, the code is a moving target, so unless you know what you\u0026rsquo;re doing, don\u0026rsquo;t import it.\nOtherwise, look at the example implementations in cmd/, one implements a server and the other a command line tool that also ships a client.\nCode is available in my Github repository for go-setdb.\npoolpOrg/go-setdb Go 0 0 Jujulang # I\u0026rsquo;m working on a language.\nWhy ? Well because I like writing code no one cares about :-)\nSo I wrote Jujulang, an interpreted language that\u0026rsquo;s the bastard child of C, Golang and Python. It supported functions, expressions, flow control, loops, etc\u0026hellip; and it was all fun\u0026hellip;\nfn main() { println(\u0026#39;hello world !\u0026#39;) } until I decided that it was not so fun to rely only on an AST evaluator, but I should rather make this language compiled. So I managed to have it generate LLVM IR, then I managed to build a small native executable for a simple main with just a return value, then I realized it was quite hard and I needed some help so here I am now.\nI bought myself two books ( see the books section), one which told me how to do what I already knew how to do, and one teaching me something I used to know twenty years ago but completely forgot how to do. Joke aside, they are excellent books and I highly recommend them.\nI\u0026rsquo;m currently halfway through generating bytecode for jujulang, with a VM evaluating that bytecode, which will lead me to a better interpreter than what I had\u0026hellip; at which point I\u0026rsquo;ll start looking back at LLVM again for native executables.\nI KNOW no one will use juju, it serves no purpose, it\u0026rsquo;s just personal interest, leave me alone.\nI might start a serie on this website, going through all the steps of this, I don\u0026rsquo;t know. Depends if there\u0026rsquo;s demand.\nOpenSMTPD-related stuff # OpenSMTPD 7.3.0p0 (portable) released # It\u0026rsquo;s been a while since OpenSMTPD portable was left behind, no one doing synchronizations with upstream since I left.\nA discussion had even been opened, asking if the project was dead and if the repository shouldn\u0026rsquo;t be archived, despite me saying I\u0026rsquo;d help anyone get on track if they pick up from where I left.\nThen Omar Polo came out of nowhere, synchronized the repository with upstream, pushed some of the portable diffs to OpenBSD, made sure it built on various distros \u0026hellip; and asked me if we could make a release.\nHe essentially took the dying portable project, revived it, brought it back to date and crafted a release in less than two weeks\u0026hellip; this was awesome work.\nI didn\u0026rsquo;t do much but provide some guidance, history context behind choices and signed the release, so this was quite refreshing and made me want to help more :-)\nHe released OpenSMTPD 7.3.0p0, the first portable release in years, which made a lot of people happy.\nWhich leads us to\u0026hellip;\nOpenSMTPD 7.3.0p1 (portable) about to be released # Obviously, as with each portable release, someone pops his/her head after the release to notify of something not working.\nIssue was investigated and solved by\u0026hellip; Omar again, who fixed and crafted a new release which should be published very shortly as I just signed it and pushed it to the website (which I still maintain).\nfilter-rspamd release # Two weeks ago, I was doing some freelance work and was very focused on what I was doing. I worked for hours on end, which is unusual because I often get distracted by mails.\nAt the end of the day, I realise that I haven\u0026rsquo;t received mails for hours which is VERY unusual, so I quickly log to the mail server and see it\u0026rsquo;s crashed. I restart it and it starts right away, I look at the logs and I realise there\u0026rsquo;s a crash in filter-rspamd due to a failed login attempt.\nAs I start looking at the code to understand the issue, mails buffered by my backup MX start flowing and one mentions a crash in OpenSMTPD, which seems to be caused by filter-rspamd. As mails continue flowing, I see that someone had reported the issue, and it was investigated and fixed by Omar while my server was down.\nThe issue was a parsing error in filter-rspamd, caused by a special case which could be avoided with a small change in the filter protocol. The change had been made for all filter messages\u0026hellip; except this one, obviously.\nOmar submitted a change to the protocol to OpenBSD, but also submitted a PR for filter-rspamd to fix parsing with current filter protocol AND work with the next version of the protocol.\nAll I had to do was \u0026hellip; read his diff and release a new filter-rspamd, easy peasy.\nOther stuff\u0026hellip; # To be honest, I also did other stuff, I just don\u0026rsquo;t feel like talking about it yet.\nWhat\u0026rsquo;s next ? # Vacaciones en Malaga. Sol, playa y tapas.\n","date":"30 June 2023","permalink":"/posts/2023-06-30/2023-q2-music-unity-xgb-jujulang-setdb-and-opensmtpd/","section":"Posts","summary":"TL;DR: music and unity training, a bit of X programming in Go, began a journey in Jujulang a new languae, wrote a PoC set database called setdb, did some minor OpenSMTPD stuff.","title":"2023-Q2: music, unity, xgb, jujulang, setdb and opensmtpd"},{"content":" TL;DR: Received the neck\u0026hellip; too bad it\u0026rsquo;s not exactly the one I ordered :-) The neck # For the neck, I wanted a 24 frets neck and found one that I liked online.\nUpon delivery today, I realized that they didn\u0026rsquo;t ship the correct one, but a 22 frets neck instead (see picture above). So while it still looks very nice, it\u0026rsquo;s not exacrly what I wanted, and it also doesn\u0026rsquo;t fit quite right in the body despite an announced compatible size: turns out that in guitar making, millimeters do matter.\nI decided to go with a more traditional PRS neck and ordered this one:\nwhich I will try to customize it with other inlays if I manage to. I should receive it in a few days, I\u0026rsquo;ll post an update then.\nIn the mean time I\u0026rsquo;ve started looking into three things:\nthe tuners the bridge the pickups I want the tuners to be locking tuners, I\u0026rsquo;ve read good things about the Schaller M6, they are currently at the top of my list but I\u0026rsquo;m still looking for alternatives. When I make a decision and receive them, I\u0026rsquo;ll post a more lengthy article on why I picked these.\nFor the bridge, I want a floating bridge that is NOT a floyd rose because I already have one, I like them but the drawbacks makes me want a different design for this additional guitar. I currently am looking into Gotoh 1502 and 1802 tremolo bridges, haven\u0026rsquo;t decided yet if I\u0026rsquo;ll stick to this idea but Iwill also post a more lenghty article once I settle.\nFinally, for the pickups, I have not decided yet, but I\u0026rsquo;d like passive pickups and would like them mismatched: the neck pickup should be suited to jazz / blues, the bridge pickup should be suited to metal. Ideally if they are split coils, all the better. I came up with a list of pickups to investigate, but since they are the most costly part of the guitar, I\u0026rsquo;ll defer the decision for later in the process depending on how the project is going and if the build quality is decent enough to invest in good pickups.\nWhat\u0026rsquo;s next ? # Waiting for the new neck to arrive, more investigation on tuners and bridge, next post should have more details :-)\n","date":"29 June 2023","permalink":"/posts/2023-06-29/building-my-own-guitar-part-2/","section":"Posts","summary":"TL;DR: Received the neck\u0026hellip; too bad it\u0026rsquo;s not exactly the one I ordered :-) The neck # For the neck, I wanted a 24 frets neck and found one that I liked online.","title":"Building my own guitar, part 2"},{"content":" TL;DR: I\u0026rsquo;m building a guitar from scratch, picking up every component but skipping the tricky cut and glue bits :-) About this project # The primary goal of this project is to build a custom guitar, picking each and every component one by one, so that it\u0026rsquo;s 100% what I want it to be.\nThe second goal is to set myself a reasonnable objective, by not having to cut and glue wood myself, so that I can see if I like building guitars at all.\nThe serie will follow this little project, either to completion or to utter failure, whichever comes first :-)\nThe body # For the body, I want a traditional shape and since I like PRS SE guitars a lot, I decided to go with something similar or close.\nI wish I knew how to pick wood, but given that I won\u0026rsquo;t be cutting it myself, this limited my searches to whatever I could get my hands on conveniently.\nI settled on this body which is solid Mahogany and Maple veneer. Ordered it two days ago from The Guitar Fabric and it was delivered today.\nThe body looks very nice (as far as I\u0026rsquo;m concerned), but in practice it\u0026rsquo;s very rough and needs sanding, it will definitely also need varnishing and maybe some subtle dying too.\nWhat\u0026rsquo;s next ? # I found a nice neck to pair with it, ordered from a different vendor, which should arrive tomorrow and which I\u0026rsquo;ll show you once I make sure it\u0026rsquo;s usable.\n","date":"28 June 2023","permalink":"/posts/2023-06-28/building-my-own-guitar-part-1/","section":"Posts","summary":"TL;DR: I\u0026rsquo;m building a guitar from scratch, picking up every component but skipping the tricky cut and glue bits :-) About this project # The primary goal of this project is to build a custom guitar, picking each and every component one by one, so that it\u0026rsquo;s 100% what I want it to be.","title":"Building my own guitar, part 1"},{"content":" TL;DR: I will be discussing music on this blog from now on. This blog is a musical blog too # As some of you know, I have a deep interest for music.\nThis blog section will focus on music-related topics, from a listener, amateur musician and/or producer point of view.\nI will sometimes post links to music I want to share, either because I did it, or because I like it, but I will also write articles and share hardware and software reviews.\n","date":"25 June 2023","permalink":"/posts/2023-06-25/a-new-section-appeared/","section":"Posts","summary":"TL;DR: I will be discussing music on this blog from now on. This blog is a musical blog too # As some of you know, I have a deep interest for music.","title":"A new section appeared !"},{"content":" TL;DR: I played with MIDI and ChatGPT. Code-unrelated work # I began a few week ago a certified training in music production which I\u0026rsquo;ll be attending for a couple weeks still, and I\u0026rsquo;ll be sharing here some of the things I work on.\nFeel free to subscribe to my Youtube channel where I will not only publish my works in progress but also document my journey in that area.\nI have not been very active these last three months # I have not been very active these last couple months. Partly because of life events and grieving, partly because of general demotivation and partly because the training is draining what\u0026rsquo;s left of my brain. I\u0026rsquo;m slowly but surely stepping back in the game.\nMHL: MIDI to human language # For a couple of projects, I needed a way to express MIDI in a human-readable format.\nThose who have followed my work on earmuff may assume that it can serve this purpose, but it doesn\u0026rsquo;t because it expresses things programatically and at a higher-level: where earmuff works with bars and beats and durations, MIDI works with a set of events triggering at absolute ticks, where earmuff has repeats blocks to provide \u0026ldquo;loops\u0026rdquo;, MIDI \u0026hellip; well, actually repeats events sequentially.\nI wrote a tiny compiler called mhl that allows building an SMF, Standard MIDI File, from the following input and the other way around to generate MHL output from an SMF:\ntrack=1 ticks=16080 message=NoteOff channel=0 key=40 velocity=64 track=1 ticks=16080 message=NoteOn channel=0 key=59 velocity=89 track=1 ticks=16080 message=NoteOn channel=0 key=55 velocity=89 track=1 ticks=16080 message=NoteOn channel=0 key=52 velocity=89 track=1 ticks=16320 message=NoteOff channel=0 key=59 velocity=64 track=1 ticks=16320 message=NoteOff channel=0 key=55 velocity=64 track=1 ticks=16320 message=NoteOff channel=0 key=52 velocity=64 Awesome idea, right ? nope, wrong.\ngo-midicsv: MIDICSV implementation in Golang # It turns out that after I was done writing mhl, I ran into MIDICSV.\nMIDICSV is a specification to represent MIDI as CSV, a human-readable comma-separated format. That\u0026rsquo;s exactly what I wanted, except that it already existed, already covered what I wanted, and there were already reference implementations in Perl and Python. I did some experimenting with the Python version, and since it proved to cover my functional needs, I decided to toss MHL and switch to using MIDICSV.\nI could not use the python version itself, partly because I needed to call it from Golang which was painful, but also because py_midicsv insists on using files whereas I need to work on bytes\u0026hellip; so I wrote a Golang implementation for my needs. This is a work in progress, do not use for anything serious, but the initial code is already commited and available in a Github repository.\nLong story short, the package comes with two commands midi2csv and csv2midi, both built using the go-midicsv/encoding package which provides an Encoder and Decoder. The Encoder reads bytes from an SMF and converts them into a MIDICSV output, whereas the Decoder reads a MIDICSV input and converts it into an SMF output.\nThe code currently only works for valid SMF files and will likely panic on any invalidly crafted SMF, I will make it error out nicely in the upcoming weeks. Good enough for my pocs.\nMelodya: a melody and music assistant # Melodya is a project I\u0026rsquo;ll be working on and off for the next few months: it is a melody and music assistant, helping you write, understand and improve music through the help of AI. Unless you\u0026rsquo;ve been hiding under a rock, you have probably already seen ChatGPT at work, it\u0026rsquo;s impressive af, and it can prove really useful in this use-case of assisting music creation.\nUnfortunately, it only understands text so it\u0026rsquo;s hard to have it work with music: it can\u0026rsquo;t take sound or sheet music as input. I had already played with this in December, teaching ChatGPT about earmuff and having it generate new earmuff source based on instructions. It worked, it worked insanely well, but had some shortcomings because earmuff was invented after the cut-off date for its corpus and doesn\u0026rsquo;t come with enough examples. Unless my prompts were very precise, it would generate instruments names that aren\u0026rsquo;t recognized, sometimes even constructs that it thought existed but didn\u0026rsquo;t, and such\u0026hellip;\nLuckily, between earmuff, go-harmony and go-midicsv, I have written enough code that help deal with music programatically and in human-readable formats that I can use ChatGPT reliably enough for my purpose\u0026hellip; so I played tetris. I assembled blocks, and here\u0026rsquo;s the initial result after a few hours of work. A chatbot that understands what I want to do on a musical project, that keeps track of the state of my project and that can provide explanations while generating MIDI output:\nThis was the very first step.\nI would like to implement speech-to-text so I can dictate my needs, text-to-speech so it can hear back its answers, but I would also like to accept input coming from a MIDI device so I can ask it to analyze, correct me or make suggestions, just as I\u0026rsquo;d like to allow it to output to a MIDI device so I can plug it to a synthesizer or DAW so I can use it more directly.\nIn addition, because MIDI can be (indirectly) converted to sheet music, I\u0026rsquo;d like to be able to display the sheet music as it\u0026rsquo;s being worked on and adapted.\nWhat\u0026rsquo;s next ? # April will be essentially spent on my training, so I don\u0026rsquo;t expect much code to be written.\nIf I manage to, it\u0026rsquo;ll likely be melodya related.\nStay tuned !\n","date":"2 April 2023","permalink":"/posts/2023-04-02/march-2023-melodya-mhl-midi-csv-and-more.../","section":"Posts","summary":"TL;DR: I played with MIDI and ChatGPT. Code-unrelated work # I began a few week ago a certified training in music production which I\u0026rsquo;ll be attending for a couple weeks still, and I\u0026rsquo;ll be sharing here some of the things I work on.","title":"March 2023: Melodya, MHL, MIDI-csv and more..."},{"content":"Almost four years ago, your mom and I left home and rushed to the maternity hospital to give birth to your brother. As I closed the door to our home, I told your mom with a huge smile that the next time we\u0026rsquo;d open the door we\u0026rsquo;d no longer be only the two of us. This was the best day of our lives.\nIn December, we got the news that your mom was pregnant of you. We had been waiting you for so long, almost to the point of giving up. We couldn\u0026rsquo;t be happier but miscarriages taught us caution and we decided to wait the three months milestone before telling your brother, family and friends that you were expected. At every appointment, I held my breath until I could hear your heartbeats, your lovely heartbeats putting a smile on my face and tears in my eyes. I love you so much.\nThe third month\u0026rsquo; appointment arrived mid-February, I was anxious as usual but everything was fine, your heartbeats sounded as lovely as the previous times and you jumped all over the place during the echography, already playing around and making us laugh. I was relieved and I began planning your arrival. We told your brother that he\u0026rsquo;d soon be sharing his toys, it was hard negociations but he eventually agreed and started asking questions about you, what games you\u0026rsquo;d be playing together and what cartoons you\u0026rsquo;d be allowed to watch together.\nTwo weeks ago, we received a phone call after your mom took a usual blood sample, further examinations were required to make sure everything was fine with you. Nothing too alarming, after all everything was fine on all other examinations, it was just to be safe. The same had happened with your brother, and it turned out to be fine. We were very stressed but also knew it didn\u0026rsquo;t necessarily mean bad news. We were just very stressed because we didn\u0026rsquo;t want anything bad to happen to you, love. We waited a week for the examination to happen, the results would come up after a few days.\nLast week, we received a phone call from the lab. You had a genetic disease without a cure, an illness that would make your life painful. We had already discussed about this with your mom a long time ago, before she was even pregnant of your brother, we both knew what it meant because we both agreed: we didn\u0026rsquo;t want you to go through this, no matter how much we wanted you, no matter how much we loved you already. Your brother was with us when we received the phone call. I took care of him while your mom went away to cry, I waited until he was asleep to take my turn, we didn\u0026rsquo;t want him to understand yet, we didn\u0026rsquo;t want to break down in front of him.\nDuring that sleepless night, I tried to convince myself that we could make it work, that as long as we loved you and took care of you everything would be fine, but as much as I tried, I could not pretend that this would be an act of love. It would not be love to knowingly force you into a life of suffering, deprivement and reliance just because we wanted you so much. It would be unfair to both your brother and you to let our desire of a child take precedence over the well-being of both of you. It would be unfair to let you develop further knowing that you had an illness, that you\u0026rsquo;d suffer from it and that your brother would have to take care of you when we\u0026rsquo;re too old or no longer around to do it ourselves. I\u0026rsquo;m sure your mom had the same train of thoughts that night. We decided to do what was in the best interest for the both of you, regardless of how painful it was to us, we knew it wasn\u0026rsquo;t to you at this point.\nIt\u0026rsquo;s been almost a week now. It\u0026rsquo;s been almost a week since we left home and drove to the maternity hospital. We didn\u0026rsquo;t want to go, we knew that this time we\u0026rsquo;d come back alone, just the two of us. A few hours later, on the 24/02/2023, your mom gave birth to you, my little girl, and then your heart stopped beating. Mine too.\nYesterday, we built the courage to tell your brother that you were no longer inside mom. We told him you had to return to the stars and that you\u0026rsquo;ll be watching over him from there, he\u0026rsquo;ll be watching you too.\nYou existed and mattered to us, we won\u0026rsquo;t forget you, I love you 💜\n","date":"1 March 2023","permalink":"/posts/2023-03-01/i-love-you/","section":"Posts","summary":"Almost four years ago, your mom and I left home and rushed to the maternity hospital to give birth to your brother. As I closed the door to our home, I told your mom with a huge smile that the next time we\u0026rsquo;d open the door we\u0026rsquo;d no longer be only the two of us.","title":"I love you"},{"content":"","date":"1 March 2023","permalink":"/categories/personal/","section":"Categories","summary":"","title":"personal"},{"content":" TL;DR: I worked on earmuff and go-harmony, I played with ChatGPT. Re-focusing on go-harmony # As you may have read from my previous post, I have spent some time porting go-harmony to the Dart language and playing with Flutter to build apps that used dart-harmony.\nIt was fun and promising, particularly because I was able to build useful tools right away, but I decided to put this on hold and re-focus on go-harmony as I had bigger plans for it than just a bunch of tools.\nUltimately, when it reaches the state I want, I can create a dart-harmony that is really a binding and resume making tools but using a far more advanced engine.\nMade various improvements to go-harmony # Generic tuning interfacev # First of all, I added support for a generic Tuner.\nA Tuner is an object which, given a tuning system (Equal Temperament for now) and a tuning (A440, A432, \u0026hellip;):\ntuner := tunings.NewTuner(tunings.EqualTemperament, tunings.A440) return tuner.Frequency(note.Position()) can compute the sound frequency for any given note:\n$ harmony -frequency C C4 = 261.63 $ harmony -frequency D D4 = 293.66 $ harmony -frequency C3 C3 = 130.81 It isn\u0026rsquo;t used much within go-harmony but exposing it will allowing building tools that require access to frequencies, such as building\u0026hellip; a tuner, or a synthesizer or pitch detectors, etc\u0026hellip; without resorting to the use of a static table which is what I had previously and want to avoid in the engine.\nnaturals package # I introduced a naturals package to deal with naturals (C, D, E, F, G, A and B).\nThe package exposes constants, for example naturals.C, all of which provide methods to obtain the natural\u0026rsquo;s position and semitones delta relative to C, or to cycle through the naturals set with .Next() and .Previous() (ie: naturals.C.Next() will return a naturals.D and naturals.C.Previous() will return a naturals.B).\nWow, modular arithmetics, how impressive ! Yeah, I know it doesn\u0026rsquo;t seem like much but this is particularly important because it simplifies intervals computations, distances computations, chords and scales construction, and more\u0026hellip; This logic was already implemented in the notes package, but having a dedicated package makes this usable more easily in various places of the engine and allowed me to simplify a lot of code.\noctaves package # The same was done to deal with octaves (C0, C1, C2, C3, C4, C5, C6, C7, C8 and C9).\nThe package also exposes constants, similarly to naturals, with methods to compute .Next() and .Previous() but also to .Add(), .Substract() or obtain an octave constant from an octave position (ie: octaves.FromPosition(4) will return an octaves.C4).\nAgain, this was mostly due to avoid implementing the logic in the notes package but making it easier to use octaves anywhere in the engine outside of the contxt of a specific note. For example, coupling this with the Tuner interface, it could be possible to generate a table of frequencies for a specific octave without having to build each note individually to have access to octaves.\nAdapted notes to make use of naturals and octaves # I adapted the notes package to remove all naturals and octaves computations and rely on the new packages.\nThis simplified the logic, allowed to shrink the file by removing code that just wasn\u0026rsquo;t in the right place, with absolutely no visible change on the outside.\nIntroduced Note.MIDI() # The engine only implements harmony rules and will not be MIDI-aware as it\u0026rsquo;s out of its scope, however I decided to implement a MIDI() method on Note that returns its MIDI note value as this can still be VERY useful and is just an offset-ed semitones computation. I will likely also implement a Piano(keys uint8) method to return the piano key on keyboards of different sizes because that\u0026rsquo;s also just an offset-ed semitones computation.\nWhy add MIDI() ?\nWell, when writing code that deals with MIDI, you usually know the midi note (the pitch) that\u0026rsquo;s been emitted but you don\u0026rsquo;t necessarily know what note it was meant to be. For instance, in a MIDI stream I will receive a NoteOn message with a pitch of 60 which will produce the sound of a C4\u0026hellip; but it may have actually been a B#3 or a Dbb4 if you look at it from a harmonic point of view.\nDepending on the use-case, you will not care or you will want to detect the correct note name, which will result in having to map notes from go-harmony to notes that share the same pitch in whatever tool is being written. Having the engine be able to produce the same .MIDI() value for all enharmonic notes allows to simplify things by a great deal.\nIt turns out I made use of this and could shove a lot of code from earmuff :-)\nWhen Piano keys are also implemented, this means that you can convert a stream of Note into their corresponding keyboard keys without having to worry about enharmomic equivalences either.\nExtended intervals # Finally, I extended the intervals package to support all intervals up to AugmentedFifteenth, making it possible to build all extended chords which were not possible before.\nTons of changes in earmuff # Grammar updates # Until today, the grammar for earmuff source files looked like this:\nproject { bpm 80; time 4 4; instrument guitar { bar { 8th note C# on 3/1; 8th note D on 3/2; 8th note A on 3/3; } bar { half note F on 1/1; 8th note E on 4/2; } } instrument piano { bar {} bar { whole chord Eb9 on 1/1; } } } This worked for experimenting, but the grammar was very limiting and made it harder to implement some concepts.\nI decided to make some slight adjustments, it now looks like this:\nproject \u0026#34;my project\u0026#34; { bpm 80; time 4 4; track \u0026#34;lead guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { on beat 1 play 8th note C#; on beat 2 play 8th note D; on beat 3 play 8th note A; on beat 4 play 8th note F#; } } track \u0026#34;rythm piano\u0026#34; { instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play half chord C#; on beat 2 play half chord D; on beat 3 play quarter chord A; on beat 4 play quarter chord F#; } } } It doesn\u0026rsquo;t show yet, however this makes some upcoming changes MUCH easier to implement, and also simplifies readability as lines read plain english now.\nFirst, it allows naming sections which was not possible before as project didn\u0026rsquo;t take a parameter and instrument was used in place of track to list the MIDI instrument in use rather than being a free field.\nThen, the previous grammar used on beat/subdivision for placement and this was so confusing that I even got confused myself after not touching an earmuff file for a couple month. Furthermore, it came at the end of lines which tied it to the notion of Playable events (sounds, chords) when other events could also require placement (ie: comments, lyrics, cues, \u0026hellip;). By moving it upfront, parsing gets simplified and it makes it possible to implement Tickable events, events that trigger on a specific tick. Instead of a fraction, on beat accepts either an integer for on-beat events and floats for off-beat events.\nClosing the gap with MIDI # earmuff is really tied to MIDI as it compiles its source code to MIDI for both streaming to a synthesizer or generating an SMF file. Ideally, it should support ALL of MIDI features, so that any MIDI can be generated from an earmuff source\u0026hellip; so I started closing the gap by implementing missing features.\nFirst of all, I implemented velocity so that any note or chord may be given a velocity, which will be reflected in the MIDI output. It is now possible to lower or raise the intensity of individual notes and chords:\nbar { on beat 1 play 8th note C# velocity 10; on beat 2 play 8th note D velocity 25; on beat 3 play 8th note A; on beat 4 play 8th note F#; } I also implemented various MIDI Metas, allowing to add copyright, text and others which are emitted in the MIDI output.\nproject \u0026#34;my project\u0026#34; { copyright \u0026#34;Gilles Chehade\u0026#34;; text \u0026#34;this project sounds like shit\u0026#34;; bpm 80; time 4 4; track \u0026#34;lead guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { on beat 1 play 8th note C#; on beat 2 play 8th note D; on beat 3 play 8th note A; on beat 4 play 8th note F#; } } track \u0026#34;rythm piano\u0026#34; { copyright \u0026#34;Jules Chehade\u0026#34; text \u0026#34;this track however ...\u0026#34;; instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play half chord C#; on beat 2 play half chord D; on beat 3 play quarter chord A; on beat 4 play quarter chord F#; } } } Finally, I added support for single and multi-line comments:\nproject \u0026#34;my project\u0026#34; { copyright \u0026#34;Gilles Chehade\u0026#34;; text \u0026#34;this project sounds like shit\u0026#34;; bpm 80; time 4 4; track \u0026#34;lead guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { // skipping first beat //on beat 1 play 8th note C#; on beat 2 play 8th note D; on beat 3 play 8th note A; on beat 4 play 8th note F#; } } /* skipping a whole track */ /* track \u0026#34;rythm piano\u0026#34; { copyright \u0026#34;Jules Chehade\u0026#34; text \u0026#34;this track however ...\u0026#34;; instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play half chord C#; on beat 2 play half chord D; on beat 3 play quarter chord A; on beat 4 play quarter chord F#; } } */ } Lots of internal refactor # In anticipation for upcoming features, I did a pass of refactor to make it possible to plug things the way I want.\nFor instance, I\u0026rsquo;d like to be able to emit Meta\u0026rsquo;s on specific Ticks, which required reworking the way a project is structured, the way a track is structured, the way bars are structured, the logic to compile MIDI from these structures, the logic to playback a MIDI to a synthesizer, etc\u0026hellip;\nLots of small invasive changes.\nInitial attempt at muffear, MIDI-to-earmuff converter # I started working on a small utility to read a MIDI file and generate an earmuff source from it.\nI had a basic version working yesterday, it doesn\u0026rsquo;t cover all aspects obviously and I made grammar changes after so it now generates broken earmuff, but I wanted to see if this could be achieved easily.\nUltimately, this will be merged in the earmuff program, but while I work on it I\u0026rsquo;d rather have things split apart.\nThe idea is that there are various programs than can produce MIDI, either from crafting tracks or recording a MIDI source (like any DAW), or from a partition expressed in various formats (Guitar Pro, ASCII, MusicXML, \u0026hellip;). By allowing to convert MIDI to earmuff, I essentially allow converting any of these inputs to earmuff to work on it as code.\nGiven that earmuff already exports to MIDI and can produce input for these programs, this gives some symmetry.\nA demo ? # Here\u0026rsquo;s a simple 12-bars blues written in earmuff, consisting of a lead piano, a rythm guitar and a bass:\nproject \u0026#34;12 bars blues\u0026#34; { bpm 120; time 4 4; track \u0026#34;lead piano\u0026#34; { instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } } track \u0026#34;rythm guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord F7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord F7; } bar { on beat 1 play whole chord F7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord F7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } } track \u0026#34;bass\u0026#34; { instrument \u0026#34;bass\u0026#34;; bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note F2; on beat 3 play quarter note A2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note F2; on beat 3 play quarter note G2; } bar { on beat 1 play quarter note F2; on beat 3 play quarter note G2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note F2; on beat 3 play quarter note G2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } } } This can be played through a synthesizer by earmuff, producing the following output (encoded in mp3 for this blog):\nYour browser does not support the audio element. Or imported into other tools, such as Guitar Pro:\nor Logic Pro X:\nChatGPT # I tried talking to GPT a bit about earmuff \u0026hellip; it was fun :p\nBecause earmuff is a language, ChatGPT can learn its grammar, understand an input and produce outputs.\nI had it learn how earmuff works, had it transpose the 12 bars blues to a perfect fourth, make some changes to it and add a tuba for a bar.\nJust imagine when I add speech-to-text to the mix, and can dictate what changes I want to apply to a project, check that they sound as expected and generate a new sheet from them \u0026lt;3\n[...] The resulting project is:\nproject \u0026#34;12 bars blues\u0026#34; { bpm 120; time 4 4; track \u0026#34;lead piano\u0026#34; { instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } } track \u0026#34;rythm guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord D7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord D7; } } track \u0026#34;bass\u0026#34; { instrument \u0026#34;bass\u0026#34;; bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note C3; on beat 3 play quarter note E3; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note C3; on beat 3 play quarter note E3; } bar { on beat 1 play quarter note C3; on beat 3 play quarter note E3; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note D3; on beat 3 play quarter note F3; } bar { on beat 1 play quarter note C3; on beat 3 play quarter note E3; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note D3; on beat 3 play quarter note F3; } } track \u0026#34;tuba\u0026#34; { instrument \u0026#34;tuba\u0026#34;; bar { on beat 1 play whole chord C7; on beat 2 play whole chord G7; on beat 3 play whole chord D7; } } } Which resulted in transposed tracks and a new tuba track:\nwhich sounds like this:\nYour browser does not support the audio element. I seemingly got a wrong mapping for instruments as this doesn\u0026rsquo;t sound like a tuba at all, but it\u0026rsquo;s\u0026hellip; interesting ? :-)\nWhat\u0026rsquo;s next ? # A break for New Year, then I\u0026rsquo;ll be working most exclusively on go-harmony and earmuff in 2023.\nI\u0026rsquo;ll write a personal retrospective of this year shortly, if I don\u0026rsquo;t get to it before the end of the year\u0026hellip;\nHAPPY NEW YEAR TO YOU ALL.\n","date":"30 December 2022","permalink":"/posts/2022-12-30/december-2022-some-more-earmuff-and-go-harmony/","section":"Posts","summary":"TL;DR: I worked on earmuff and go-harmony, I played with ChatGPT. Re-focusing on go-harmony # As you may have read from my previous post, I have spent some time porting go-harmony to the Dart language and playing with Flutter to build apps that used dart-harmony.","title":"December 2022: some more earmuff and go-harmony"},{"content":" TL;DR: busy month, did a lot of non tech stuff, but also worked on a new project called feedchain, fixed issues and added new features to go-harmony, and started two projects called dart-harmony and harmonee, both related to music. Music-related stuff # No music productions this month, I lacked time, but I have recorded the result of some guitare practice sessions which amounts to hours of struggle :-)\nHypnosis-related stuff # I gave a two-hours talk in French, along with two other hypnotists, to fifty-ish psychology students at the faculty of Nantes on the topic of hypnosis and myths surrounding it.\nIt was very nice yet a bit stressful as I had roughly a week to prepare the slides while I had a ton of other things to do, and luckily for me it\u0026rsquo;s a topic I\u0026rsquo;m comfortable enough to go without rehearsing otherwise the anxiety disorder would have kicked in ;-)\nAs a result, I had no time to work on my hypnosis podcast for this month but I\u0026rsquo;ll be writing the scenario and recording next week as I wish to complete the fourth episode in early December. This will leave me with four episodes on the topic of dreams, hypnosis, traumas and hypnotherapy before I complete the serie and move on to another project.\nI have a book under writing with the intent to self-publish. It\u0026rsquo;s about ~150 pages at this point but still needs a lot of work and I\u0026rsquo;ll try to focus a bit on it as I\u0026rsquo;d like to avoid letting it linger for too long. It covers the topic of hypnosis from a mechanical point of view, essentially following the podcast but with much greater details. When it\u0026rsquo;s finished and published, I\u0026rsquo;ll likely translate from French to English.\nI worked a lot on the feedchain project # As you may have seen from my previous article this month, I worked a lot on an experimental project called Feedchain.\nI designed the general structure in a few hours but made iterative changes here and there for days, while constantly adapting the proof-of-concept Golang implementation to reflect these. Since I already had the Golang implementation, I also wrote a publish server that undertands how to to verify feedchains and serve them efficiently. The next step was to write a proper client for feedchain, but I decided to take a step back and not implement that in Golang but in Dart instead.\nWhy Dart, you ask ?\nIt turns out that for this project and others I keep telling myself I should make a nice UI, as a native application or a web interface, but I don\u0026rsquo;t have the motivation and time to learn too many frameworks to handle too many targets. I\u0026rsquo;ve grown curiosity for the Flutter framework which allows building native applications for multiple targets using a single codebase in Dart, and so it made sense to learn myself some Dart if only to give it a try.\nSo I implemented the feedchain parser in Dart, as well as classes to both produce and consume feedchains, which surprisingly took me only a couple hours with no prior experience with Dart. I then tried building an application in Flutter that would play nice with feedchains I generated from my Golang proof-of-concept, and this is where the struggle began because\u0026hellip; believe it or not, after \u0026gt;20 years of writing code that executes in a console, it turns out I\u0026rsquo;m not an UI guy and I suck at making anything look nice.\nI\u0026rsquo;ll continue working on this, but I needed to also work on other projects.\ngo-harmony # I wrote a bit about my go-harmony package back in June, and not much had changed since then.\nSo I dived back into it and fixed a few issues, related to interval arithmetics for the most part, but also added a lot of new features like support for progressions, for all missing intervals, for various new chord structures (7sus2, 7sus4, M7sus2, M7sus4, sus2sus4, b5, \u0026hellip;) or improvements on the handling of chord inversions.\nI also implemented a scale scanner that, given a chord, will output all scales sharing more than two notes with the chord and sorted by most notes matched first:\nKeep in mind that the project is the engine, not the useless CLI :-)\nAt this point, though not exempt of glitches, go-harmony can already:\nparse note names, chord names, scale names determine interval between two notes determine note at a specific interval from a root note transpose note or chords to a specific interval on the fly build a ton of different chords by applying a chord structure to a root note build several scales by applying a scale structure to a root note compute diatonic triads and seventh chords for a specific scale compute relative minor / major for a chord compute inverse intervals recognize a chord from a set of notes if the chord structure is already implemented and so I did what had to be done \u0026hellip; I threw some MIDI at it :-)\nI wrote a tool that could ingest some MIDI and use go-harmony to try to make sense to what was happening in the track, try to detect scales, chords and what not, and\u0026hellip; the result was both interesting and unimpressive at the same time.\nIt was interesting because on some tracks it did manage to understand what was happening, what chords and scales were being played. At one point, I thought there was a bug in the code because it was not displaying the same chord name as the score but go-harmony was actually right, it provided a more accurate name than what was written. Another interesting thing is that I tested on-the-fly transposition and it worked lovely, both to convert a track from minor to Major, but also to transpose all notes a perfect fourth upper as they were read. That\u0026rsquo;s not a huge feature but I was happy it worked flawlessly as I have plans for it.\nIt was unimpressive because it still lacks some key features (pun intended), such as add / omit, which causes a lot of chords to go undetected or misdetected. A LOT. Too many actually that it prevents me from moving forward with some of the more interesting features I have in mind, so I really need to tackle these detection failures before anything else\u0026hellip; and it\u0026rsquo;s not trivial :-)\ndart-harmony, wait what ? # Yeah, I\u0026rsquo;m spending time on a Golang implementation of a music theory engine\u0026hellip; and I\u0026rsquo;m reimplementing it in Dart at the same time.\nBear with me.\nI\u0026rsquo;m not really doing the work twice because they are essentially implementing the same calls and structures, just implementing in two different languages because I have two different uses for this.\nThe go-harmony implementation is meant to be used both by a REST API that I\u0026rsquo;ll probably make public next year, and by other stuff I have in mind but that I\u0026rsquo;m not willing to disclose yet.\nThe dart-harmony implementation is meant to be used by Flutter apps so that I can start building useful stuff, tools that people can actually install on their devices without having to understand how to \u0026ldquo;compile\u0026rdquo; something, but more importantly tools that I need to have on me frequently even if not in front of a computer :-)\nThe dart-harmomy project was started 8 days ago and it has already caught up with go-harmony, it is even slightly ahead as I have started building stuff with it and had to add missing features. It supports pretty much all of what go-harmony supports\u0026hellip; but code keeps getting improved as I learn new Dart tricks.\nHarmonee (the name might change) # With dart-harmony it became possible to start implementing tools in Flutter, so I gave a try at various stuff to get familiar with Flutter programming.\nThe first tool I wrote is a circle of fifths implementation which consists in three nested circles, the outter circle being the actual circle of fifths, the middle circle being the minor chords that map to the outer circle, and the inner circle being the diminished chords. The circle is dynamic, clicking on a specific section rotates the circle so the relevant key is exposed on top.\nMy main issue is that this doesn\u0026rsquo;t scale well on screen resizes, so I really need to work out a better way to display it:\nThanks to dart-harmony there\u0026rsquo;s a TON of informations that could be associated to each block of the circle, like related chords, related scales, substitutions, etc\u0026hellip; but the issue is really how to display information effectively and I still need a lot of work to be more effective at designing nice stuff.\nSo I decided to temporarily split into multiple different pages so I can experiment with different ways to display data, and I implemented a search engine which will dynamically display cards as you type your query. On the examples below, I started typing A which resulted in the natural A, the note A in the default octave, the A major chord and the A major scale, but as I continued typing A7 it refined to the note A in the 7th octave and the A7 chord, when I typed Amyxolydian it diplayed the A myxolydian scale:\nJust as I said for the circle of fifths, I have plenty of additional informations that could be displayed on each card, I just haven\u0026rsquo;t figure out the nicest way to display them yet.\nUltimately I decided to give another try at displaying things in a circle, and came up with something nice as a starting point to implement a chord dictionnary:\nand a scale dictionnary:\n\u0026hellip; again, I have access to more informations, like interval below each note or diatonic chord for each degree, but it quickly makes the interface unreadable\u0026hellip;\nSo the bottom line is, I need more time to undertand how to make this really useful as what I really want, ultimately, is a tool to rule them all: a complete assistant for all music theory related topics.\nThe nice thing is that it allowed me to deep dive into Flutter, I did test building and deploying on real hardware as I could test Harmonee on my phone, and the entry cost to start making interfaces was very low: I came up with usable tools, even though limited, in just a few hours of playing with it. I hope I can come up with very nice things as I get more experienced.\nWhat\u0026rsquo;s next ? # December will be a very short month for me as my kid has two weeks of holidays and one of them falls right in my free-time week, meaning I won\u0026rsquo;t have much time to work on anything beyond my noon breaks and sleepless nights ;-)\nI\u0026rsquo;ll probably keep working a bit on dart-harmony as time allows and will write my yearly retrospective late December, don\u0026rsquo;t expect too much done before 2023 !\n","date":"30 November 2022","permalink":"/posts/2022-11-30/november-2022-feedchain-go-harmony-dart-harmony-and-harmonee/","section":"Posts","summary":"TL;DR: busy month, did a lot of non tech stuff, but also worked on a new project called feedchain, fixed issues and added new features to go-harmony, and started two projects called dart-harmony and harmonee, both related to music.","title":"November 2022: feedchain, go-harmony, dart-harmony and harmonee"},{"content":" TL;DR: I implemented a file format to support a standalone feed of news similar to what I have on Twitter, as well as a proof of concept code for a reader, a writer, a publishing node, and some more. oh and fuck you, elon, you\u0026rsquo;re a dick. This is not my activity report # This is not my activity report which I\u0026rsquo;ll publish later this month or early December. I\u0026rsquo;ve worked on several projects but I thought this one would be better off discussed in a dedicated article.\nThe problem\u0026hellip; # In just a month, two services that I was heavily relying upon turned into shit.\nThe first one is Signal, a secure messaging system, which removed a key feature to the point I no longer have any use for it as most of my contacts uninstalled. It sucks and I can\u0026rsquo;t really do anything about it, there are alternative just not ones that would embark non-techies like my mother or my wife as easily as Signal did.\nThe second one is Twitter, which is taking a very dramatic turn as the, hum, how do I put it\u0026hellip; \u0026ldquo;Genius\u0026rdquo; who bought it managed to get most employees fired or resigned in less that two weeks, bringing the service to a point where it\u0026rsquo;s unclear if there are even enough people left to care for the machines that host it (according to comments from former engineers there).\nUnfortunately for me, I don\u0026rsquo;t use it only for the fun of it but also to communicate about stuff while I work on them, which comes with a ton of benefits for me, the most obvious one being the freelance gigs that occasionally come as a result from sharing information there.\nA solution adopted by many has been to move to Mastodon, which I also may end up doing, but I dislike the idea that I\u0026rsquo;ll have to trust yet another service with keeping my data and not shutting down in a year or two putting me in a similar position. There are also many other reasons which are also causing me to think twice before making a move there, including as an operator of my own instance.\nSo what do I do with that :-/\nI worked on a new little project\u0026hellip; # Feedchain is a project to maintain a feed of news that can be published as a standalone file, distributed from various locations, while allowing to fetch specific portions of it, guaranteeing the authenticity and integrity of the content and expecting as little requirements as possible for both the publisher (me) and the readers (you).\nThink of it as a kind of RSS feed\u0026hellip; but an RSS feed that\u0026rsquo;s not XML, that contains an index for random access to any information chunk contained in the feed, and that has a signature for the whole feed as well as for each individual chunk. And like an RSS feed, it will not require a specific server to work but any service that can expose the file and preferably support range queries to avoid full fetches.\nI have a working PoC # I have split the PoC into small applications to make it easier for me to experiment with it, however it\u0026rsquo;ll obviously have to be merged into a single useable application so it\u0026rsquo;s not a pain in the ass to use for others.\nThe first application is a feedchain writer which can generate an empty feedchain, update its content or metadata, and publish it somewhere.\nThe writer generates an ed25519 keypair, the feedchain is identified by the public key stored in the header and a name stored in the metadatas to ease referencing it. For instance, my feedchain has the public key nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 and is identified by the name @poolpOrg, the header, index and metadata chunks are signed with my private key so it can be verified by a feedchain reader.\n% writer -create -name poolpOrg created feedchain nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 % From this point the feedchain exists on my machine and because I only have one, I don\u0026rsquo;t have to provide the name when adding data to it, it\u0026rsquo;ll figure out where to write:\n% writer -write \u0026#34;just set up my fdchn\u0026#34; % writer -write \u0026#34;let\u0026#39;s see how this little project goes :-)\u0026#34; % The feedchain that results from this can be uploaded directly to any web server, for example using scp, but because not everyone knows how to configure and run a web server\u0026hellip; I implemented feedchain node servers that handle publishing and distribution, and which anyone can run at the cost of providing disk space and some memory, and I deployed one at https://feeds.poolp.org to bootstrap the project.\n% writer -publish % From this point, the feedchain is public and distributed by https://feeds.poolp.org at the following address https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4:\n% curl https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 \u0026gt;/dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1245 100 1245 0 0 9867 0 --:--:-- --:--:-- --:--:-- 10462 The second application is a feedchain reader, which knows how to parse the feedchain file format, verify signatures and use the index to access specific chunks.\nIt supports reading the content from a local file or from a remote HTTP(S) server as long as it supports range queries. The example reader outputs all content from the feedchain, but a smarter reader could apply filters to extract the last n chunks, restrict to a specific timeframe, match hashtags, etc\u0026hellip;\n% reader ~/.feedchain/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 [2022-11-19T01:52:16+01:00] (sha256:fbb1...4414): just set up my fdchn [2022-11-19T02:03:59+01:00] (sha256:fbdf...c88e): let\u0026#39;s see how this little project goes :-) % reader https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 [2022-11-19T01:52:16+01:00] (sha256:fbb1...4414): just set up my fdchn [2022-11-19T02:03:59+01:00] (sha256:fbdf...c88e): let\u0026#39;s see how this little project goes :-) % The third application is a feedchain watcher, which knows how to follow/unfollow feeds by URL (no dependency to a feedchain node) or by name (requires contacting a feedchain node):\n% watch -follow poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 % % watch -follow poolpOrg % \u0026hellip; then monitor them for changes to produce a live output:\n% watch [2022-11-19T01:52:16+01:00] @poolpOrg (nTGU...ZGZ4): just set up my fdchn [2022-11-19T02:03:59+01:00] @poolpOrg (nTGU...ZGZ4): let\u0026#39;s see how this little project goes :-) [output updated as feeds are updated...] The feedchain nodes which I mentioned briefly are standalone servers that makes it easier to publish feedchains.\nThey have an endpoint for posting a feedchain which makes it public if the feedchain is correctly formatted and has a valid signature:\n% curl -XPOST --data-binary @/etc/group https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 header signature verification failed % curl -XPOST --data-binary @/Users/gilles/.feedchain/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 % Then, they allow fetching the raw file from that same endpoint while allowing range requests, making it possible for the reader to fetch the header, the index then seek and read from specific offsets.\nBecause a feedchain is basically an RSS on steroids, nodes also provide an RSS view of the feedchain though it still needs a bit of work to be rendered more nicely by an RSS reader:\nThey also provide a REST API to allow fetching informations from a specific feedchain. The API exposes the header, the index, metadatas and individual blocks which may be indexed either by their digest or by their offset in the feed:\nBy contacting a node and using its API, it becomes possible to easily consume a feedchain from client-side code to embed portions of a feedchain on a website, or to implement a more advanced application without having to reimplement a feedchain parse but relying only on HTTP requests.\nFinally, I learnt myself a bit of Dart so I could start playing with Flutter and see what it meant to create a native mobile application, but I will need a bit of time to have something usable (screenshots taken from an earlier version).\nAbout the format # The feedchain file is structured in the following way:\nIt begins with a signature of the header\u0026rsquo;s digest, allowing to check that it changed by reading the first 64 bytes, and verifying that the header was not altered by reading the first 320 bytes.\nThe header contains the offset, size, digest and signature for both the index and metadata chunks, it is followed by all data chunks and ends with the index and metadata chunks. The index contains the creation time, offset, size, digest and signature for each individual chunks, the chunks themselves contain the digest of their predecessor chunk (similarly to a blockchain), and as a result the headers digest protects the structure of the entire chain.\nNot much more to be said, it is really straightforward, to read a feedchain you read the header, locate the index and iterate over the index records to locate the offsets and lengths of blocks, validating digests and signatures as you iterate.\nLimitations and rationale # First, AND VERY OBVIOUSLY, this isn\u0026rsquo;t a Twitter replacement and does not / can not cover a lot of the features you\u0026rsquo;d expect from Twitter or Mastodon. It barely tackles the issue of having to provide a third-party the ownership on data and it allows decoupling a feed from a specific hosting provider, whereas features like searching or user interactions are completely out of the scope.\nCan these even be implemented at all ?\nYes, however they would be implemented as services around feedchains with specific server-side code provided by nodes. For instance, to be able to follow a feed by name rather than URL, I had to implement a service to perform a name lookup on feeds hosted on a specific node. This is not a feature of the feedchain itself but a feature of the hosting node. This means that, by default, likes and private messages and whatnot do not exist in feedchain, they MAY only exist if server-side code supports them as optional services on the side.\nI want the feedchains to remain standalone files usable from a static web server, not requiring the use of specific server-side code BUT being able to take advantage of server-side code IF desired. By limiting the features of a feedchain to its standalone file, and adding optional features around, I can guarantee that feeds can be moved around easily without causing too much disruption, they can be republished anywhere easily and nodes can operate as a directory.\nI can decide to provide a direct link to my feedchain hosted on my static web server and not rely on any feedchain node, or I can decide to publish my feedchain to a node because I don\u0026rsquo;t have resources for self-hosting and if it goes down, I\u0026rsquo;ll publish elsewhere. There\u0026rsquo;s no requirement to rely on nodes and, as a matter of fact, I could publish on my website and someone else could find my feed interesting enough to track it from my static web server and mirror it on a node.\nThis is something that by design can\u0026rsquo;t be prevented, the way feedchain works means that I can\u0026rsquo;t impose hosting on a node of mine.\nBut it means you keep rebuilding and uploading the whole feed ? # YES \u0026hellip; and NO, it depends.\nWhat\u0026rsquo;s for sure is that using a standalone file means that adding, removing or updating can mean having to rewrite portions of the file, and that publishing may mean having to upload the whole file. So does that mean that whenever I add a small status update to my feed, I HAVE to push the whole feed again because the file was rewritten and no longer has the same checksum and signatures on header and index ?\nWell, in the proof-of-concept implementation YES, but that\u0026rsquo;s because publishing must support uploading a full chain and it was easier to start from that point\u0026hellip; however in practice, there are MANY ways that this can be avoided because of how the file is structured.\nFirst of all, the file is structured in such a way that the most common operation of editing metadata or adding/updating/deleting from the end of the feed is cheaper. In such cases, only the end of the file needs to be rewritten as it may cause offset shifts, the header being fixed-sized can have signatures and checksums overwritten in place.\nWhen publishing to a static website, the full chain would need to be re-uploaded, but when publishing to a node exposing an API, partial updates through diffs and in-place overwrites would be possible.\nThen, locally, it\u0026rsquo;s not because the end result is a standalone file that the client HAS TO to work from that file. It could store header, chunks, index and metadata separately and build the feedchain from them, so shifting, deleting, updating would not be so complex.\nWhat\u0026rsquo;s for sure is that feedchain is not an immutable feed, it allows updating older entries\u0026hellip; but at the cost of recomputing the full chain beyond edit, altering chunk identifiers (digests) on the way. This is a design choice as it is technically possible to avoid this by removing the blockchain structure (blocks containing the previous block digest), however by enforcing this it discourages the rewrite of history and ensures that people referencing a chunk do not get tricked when the chunk suddenly changes to something else. I think it\u0026rsquo;s better to support edits through the use of new chunks referencing previous ones than by rewriting old ones.\nIf this is a topic of interest, let me know and I\u0026rsquo;ll write more in a future article about how all operations work in feedchains.\nNext steps # I don\u0026rsquo;t have clear steps as I\u0026rsquo;m mostly experimenting around at this point, but among the next steps are:\n1- extending the node server code to support p2p distribution of indexes, so you could run your node and bootstrap it from mine, they would share indexes of feeds they know so that when someone asks my node if it knows where a feed is located, it could point to yours transparently.\n2- move away from proof of concept to higher-quality code for clients. Having to rely on a reader app, a writer app and a watcher app is nice to help me debug and experiment, but it\u0026rsquo;s not so nice for regular use.\n3- implement an application in Flutter so non-techies can use feedchain on their mobile phones or computers without having to understand a thing about what I wrote today.\nWant to help and contribute ? # I have only played with this for a few cumulated hours to be fair, so there\u0026rsquo;s plenty of iterations to do on it to make it nicer.\nThere\u0026rsquo;s code to write in Golang, there\u0026rsquo;s code to write in JavaScript, there\u0026rsquo;s code to write in Dart, and you could implement feedchain in your favorite language using the examples from the proof-of-concept and asking question on my discord.\nIs it tricky ? Nope. I implemented the initial proof of concept in Golang, I knew nothing about Dart and wrote a full implementation for it in just a few hours, so anyone could probably implement feedchain in whatever language of their choice in a really short timeframe.\n","date":"20 November 2022","permalink":"/posts/2022-11-20/feedchain-is-a-standalone-news-feed-project/","section":"Posts","summary":"TL;DR: I implemented a file format to support a standalone feed of news similar to what I have on Twitter, as well as a proof of concept code for a reader, a writer, a publishing node, and some more.","title":"Feedchain is a standalone news feed project"},{"content":" TL;DR: added comments support to the blog, did some music and hypnosis projects, fixed a few bugs in plakar and began a new toy project. Learnt myself some Swift # Got myself a book on the Swift programming language, spent a couple hours reading it and I really enjoyed it.\nI was mostly curious, I don\u0026rsquo;t have anything I want to do with it at the moment, but it seemed really nice as it kind of looks like a mix between C, Golang and Python, taking bits I like from the three of them.\nWe\u0026rsquo;ll see if something comes out of it for me in the future :-)\nAdded table of content and comments support to the blog # For the last few years, this blog has used a static generator so that I could write the posts in Markdown and publish by committing to a repository. This is nice because the website is standalone and I can easily move it around from machine to machine, but it has prevented me from providing comments and I ended up relying on Github discussions and adding a link to the proper discussion below each article I write.\nI do get feedback from people, but at the exception of some articles that were referenced on popular websites, these feedbacks tend to happen through other means like private messages on Twitter or discord. Maybe these people wouldn\u0026rsquo;t leave a comment on the blog if it was possible, but maybe it\u0026rsquo;s just because having to follow a link and comment elsewhere is not practical and does not keep the article available with a scroll.\nSo I found a nice javascript library that allows interfacing with the Github discussions API, and now each article has\u0026hellip; its own comment section. It is now possible to comment directly at the bottom of each article, the comments reflect what was discussed directly on Github and are published there.\nWhile at it, I figured it was possible to add a table of content to my articles so you can jump to specific sections. I have not added it to previous articles, though I might for some, but from now on I\u0026rsquo;ll make sure to always include one as I had lengthy articles in the past and it\u0026rsquo;ll ease reading.\nFeel free to comment this article and test :-)\nMusic-related work # I enrolled in a course on music production that will start in March 2023 at a local studio in Nantes, so I spent some time playing with Logic Pro X to be more familiar with it and not begin the course as a complete newbie with regard to tools.\nThese are four LoFi remixes I worked on by importing midi tracks, altering instruments, scores and tempo, and adding some effects. Definitely not fantastic but I do like the result for some so I\u0026rsquo;m sharing:\nLet me know if you like them, I have created a specific playlist on my youtube channel for these experiments, you can subscribe, comment and give thumbs up :-)\nHypnosis-related work # I haven\u0026rsquo;t spent much time at the hypnosis office since I came back from holidays, only performing a few sessions here and there, mostly due to dedicating time to my son as he entered kindergarten and I need to adjust to his schedule.\nSo I decided to spend some time working on a podcast in French to explain how hypnosis works at low-level. I have recorded the first three episodes of 8 and I intend to release roughly 1 per month, with next episode due in late November or early December as I\u0026rsquo;ll present a conference in November and won\u0026rsquo;t have time to record.\nAs a side-note, I found it enjoyable to record so I might create a podcast related to computer stuff in the future, when I\u0026rsquo;m done with this one.\nA bit of plakar work # First, I fixed a crash that happened when trying to list content of a path that consisted in a single directory and not starting with /. Basically, there is no such thing as a \u0026ldquo;relative\u0026rdquo; path in plakar as they are always relative to the snapshot and therefore to the root of a snapshot, so def:etc is really def:/etc as far as plakar is concerned. Because of how paths are handled, by splitting pathnames into atoms and recreating a filesystem view, this confused plakar about what was the root directory and caused it to crash. I fixed the function that splits a snapshot path (ie: \u0026lt;snapshot\u0026gt;:\u0026lt;pathname\u0026gt;) into prepending a slash to the pathname portion whenever there isn\u0026rsquo;t, which should not only fix the crash I observed but also all commands that accept snapshot paths.\nThen, I spotted a more annoying bug where restoring a snapshot results in a partial restore with missing files. I was concerned at first because the last thing I want in a backup tool is for a snapshot to miss files, however after a bit of testing I could verify that they weren\u0026rsquo;t really missing as they could all be restored individually, this pinpointed to a bug in the pull primitive which seems to miss some restorable files. I spent several hours trying to understand the issue but couldn\u0026rsquo;t figure out why it happens, except that it happens mostly with snapshots containing a lot of files and when pull is parallelized. It doesn\u0026rsquo;t happen, or at least I could not reproduce, when working with a small snapshot or when pull is sequential. I removed concurrency for the time being but this is not practical, sequential restore of a large snapshot is dead slow, I\u0026rsquo;ll continue tracking the issue.\nI also removed the gzip support that I added in plakar cat around May, and moved it to a dedicated plakar gzcat command. Back then it seemed like a good idea to me that plakar cat transparently inflated compressed output at it allowed to plakar cat a compressed log, but after some thinking I\u0026rsquo;d rather provide that through a dedicated command as it allows using plakar cat to redirect a compressed stream to a file, something that was no longer doable.\nFinally, I changed the way plakar checksum works and implemented a -fast option. The command allows printing the sha256 checksum of a file contained in a snapshot, and since I already had the checksum in the snapshot INDEX, the command didn\u0026rsquo;t recompute it but used the recorded checksum instead. I changed it so that it recomputes the checksum by default, reading the file chunk by chunk, and only resorting to the recorded checksum when the -fast option is used. In practice, this should always produce the same result unless the plakar store has corrupted chunks.\nAnd that\u0026rsquo;s all for plakar !\nA new project: streamchain # So this is a TOY project, nothing serious, I\u0026rsquo;m just experimenting with an idea.\nBasically, I wrote a tool which lets me create a \u0026ldquo;twitter-like\u0026rdquo; stream where I can post short messages, however instead of being centralized on a server they are stored in a standalone structured file.\nThe file is structured in a way that allows verifying both authenticity and integrity, but also allows for fast random access to any message without having to read the entire file. A streamchain reader can validate that the streamchain file originates from its owner, it can validate that it has not been tampered with, and can seek to any message in constant time regardless of the streamchain size.\nAs a result, it can be hosted anywhere from an s3 object storage to a static web server, and if the storage supports range queries, like pretty much all HTTP servers\u0026hellip; then all the better. A streamchain client reader that can talk HTTP can either mirror a streamchain locally and perform range queries to synchronize updates, or it can consume the streamchain remotely without downloading it entirely.\nI have a working PoC but I\u0026rsquo;m not ready to publish the code yet, so I\u0026rsquo;ll likely write an article about it when I open the repository publically.\nWhat\u0026rsquo;s next ? # A conference on hypnosis for psychology students in November, so probably not much happening in the next two weeks as I prepare for it.\nThen, I\u0026rsquo;ll try to get streamchain in shape to open the repository.\nTake care, stay tuned, I\u0026rsquo;ll post as soon as I resume my work !\n","date":"18 October 2022","permalink":"/posts/2022-10-18/october-2022-blog-comments-a-bit-of-plakar-and-the-streamchain-project/","section":"Posts","summary":"TL;DR: added comments support to the blog, did some music and hypnosis projects, fixed a few bugs in plakar and began a new toy project.","title":"October 2022: blog comments, a bit of plakar and the streamchain project"},{"content":" TL;DR: did a lot of music, even while writing code. Code-unrelated work # As last month, I\u0026rsquo;ll start with code-unrelated work !\nFirst, here\u0026rsquo;s my progress on learning Hyunsoo Lee\u0026rsquo;s adaptation of Bach\u0026rsquo;s Air on G string, still not quite right and with some missing parts but\u0026hellip; slowly getting there:\nWhile at it, I started learning Hyunsoo Lee\u0026rsquo;s adaptation of Beethovens\u0026rsquo; 5th\u0026rsquo;s Symphony, also not quite right and with missing parts but also making progress:\nIt was a long time since I made a LoFi track, here\u0026rsquo;s two that I thought were not too bad:\nThey\u0026rsquo;re mostly an exercise to get familiar with Logic Pro X.\ngo-harmony # go-harmony is a package written in Golang that intends to implement a music theory engine to build other tools with. It\u0026rsquo;s only a month old so there\u0026rsquo;s not much to it yet, however there\u0026rsquo;s enough that I could use it to build the project I\u0026rsquo;ll talk about in the next section.\nMy intent was not only to build the engine but also to refresh my memories and fill some gaps\u0026hellip; so I decided not to be lazy and copy-paste a bunch of values in static tables. Instead, I first implemented notes, then intervals as semitones distances to a root note, then chords as a pattern of intervals from a root note (taking into account chord inversions), then also scales also as a pattern of intervals, then derived triads and sevenths chords using the chords implementation, and so on\u0026hellip;\nI made it so it would be very easy to extend for unsupported cases, like chords which are defined as follows making it simple to add new families by simply describing the general structure:\nMajorTriad Structure = Structure{ intervals.PerfectUnison, intervals.MajorThird, intervals.PerfectFifth, } MinorTriad Structure = Structure{ intervals.PerfectUnison, intervals.MinorThird, intervals.PerfectFifth, } AugmentedTriad Structure = Structure{ intervals.PerfectUnison, intervals.MajorThird, intervals.AugmentedFifth, } DiminishedTriad Structure = Structure{ intervals.PerfectUnison, intervals.MinorThird, intervals.DiminishedFifth, } [...] AddNinth Structure = append(MajorTriad, intervals.MajorNinth) AddEleventh Structure = append(MajorTriad, intervals.PerfectEleventh) AddThirteenth Structure = append(MajorTriad, intervals.MajorThirteenth) SusSecond Structure = Structure{ intervals.PerfectUnison, intervals.MajorSecond, intervals.PerfectFifth, } SusFourth Structure = Structure{ intervals.PerfectUnison, intervals.PerfectFourth, intervals.PerfectFifth, } The same applies for scales:\nvar Ionian = []intervals.Interval{ intervals.PerfectUnison, intervals.MajorSecond, intervals.MajorThird, intervals.PerfectFourth, intervals.PerfectFifth, intervals.MajorSixth, intervals.MajorSeventh, intervals.Octave, } var Dorian = []intervals.Interval{ intervals.PerfectUnison, intervals.MajorSecond, intervals.MinorThird, intervals.PerfectFourth, intervals.PerfectFifth, intervals.MajorSixth, intervals.MinorSeventh, intervals.Octave, } [...] var BluesMajor = []intervals.Interval{ intervals.PerfectUnison, intervals.MajorSecond, intervals.PerfectFourth, intervals.PerfectFifth, intervals.MajorSixth, intervals.Octave, } var MinorPentatonic = []intervals.Interval{ intervals.PerfectUnison, intervals.MinorThird, intervals.PerfectFourth, intervals.PerfectFifth, intervals.MinorSeventh, intervals.Octave, } Then go-harmony uses artihmetics to compute everything from a root note and its intervals. For example the scale.Notes() method computes the notes for a scale by applying the intervals that match the scale pattern for a given root note:\nfunc (scale *Scale) Notes() []notes.Note { ret := make([]notes.Note, 0) for _, interval := range scale.structure { ret = append(ret, *scale.root.Interval(interval)) } return ret } No hard-coded scales for each notes and once a scale structure is implemented, it works for all notes.\nThe package by itself doesn\u0026rsquo;t do much more, it\u0026rsquo;s meant to provide APIs and structures for other projects to rely upon, however I built a small utility that\u0026rsquo;s shipped with it to showcase some of it\u0026rsquo;s features.\nFor example, it can be used to validate note names and obtain their frequencies relative to a particular tuning (only A440 for now):\n% harmony -note \u0026#39;C\u0026#39; C 261.63 % harmony -note \u0026#39;C5\u0026#39; C 523.25 % harmony -note \u0026#39;A\u0026#39; A 440 % harmony -note \u0026#39;Cb\u0026#39; Cb 493.88 % harmony -note \u0026#39;C#\u0026#39; C# 277.18 % harmony -note \u0026#39;Cbb\u0026#39; Cbb 466.16 % harmony -note Z 2022/06/14 13:25:36 bad note (Z): should be \u0026#39;C\u0026#39;, \u0026#39;D\u0026#39;, \u0026#39;E\u0026#39;, \u0026#39;F\u0026#39;, \u0026#39;G\u0026#39;, \u0026#39;A\u0026#39; or \u0026#39;B\u0026#39; Just as with notes, it can be used to validate chord names (it supports multiple notations) and decompose them into their building intervals:\n% harmony -chord \u0026#39;C\u0026#39; Cmaj 1: C 261.63 3maj: E 329.63 5: G 392.00 % harmony -chord \u0026#39;C5\u0026#39; C5 1: C 261.63 5: G 392.00 % harmony -chord \u0026#39;C7\u0026#39; C7 1: C 261.63 3maj: E 329.63 5: G 392.00 7min: Bb 466.16 % harmony -chord \u0026#39;C7b5\u0026#39; C7dim5 1: C 261.63 3maj: E 329.63 5dim: Gb 369.99 7min: Bb 466.16 % harmony -chord \u0026#39;Csus2\u0026#39; Csus2 1: C 261.63 2maj: D 293.66 5: G 392.00 % harmony -chord \u0026#39;Cadd9\u0026#39; Cadd9 1: C 261.63 3maj: E 329.63 5: G 392.00 9maj: D 587.33 % harmony -chord \u0026#39;Cadd9/E\u0026#39; Cadd9/E 3maj: E 329.63 1: C 261.63 5: G 392.00 9maj: D 587.33 % harmony -chord \u0026#39;Cadd2\u0026#39; 2022/06/14 13:24:43 unknown chord name: add2 It knows of a few scales and modes and can decompose them into notes, triads and sevenths chords for each degree of the scale:\n% harmony -scale Caeolian C C 261.63 D 293.66 Eb 311.13 F 349.23 G 392 Ab 415.3 Bb 466.16 C 523.25 Triads: Cmin Ddim Ebmaj Fmin Gmin Abmaj Bbmaj Sevenths: Cmin7 Dm7b5 Ebmaj7 Fmin7 Gmin7 Abmaj7 Bb7 % harmony -scale Cphrygian C C 261.63 Db 277.18 Eb 311.13 F 349.23 G 392 Ab 415.3 Bb 466.16 C 523.25 Triads: Cmin Dbmaj Ebmaj Fmin Gdim Abmaj Bbmin Sevenths: Cmin7 Dbmaj7 Eb7 Fmin7 Gm7b5 Abmaj7 Bbmin7 And finally, someone asked if it could build chords from a given set of notes, so I implemented it which took only a couple minutes:\n% harmony -notes \u0026#39;C,E,G\u0026#39; Cmaj C 261.63 E 329.63 G 392 % harmony -notes \u0026#39;C,Eb,G,B\u0026#39; C-M7 C 261.63 Eb 311.13 G 392 B 493.88 % harmony -notes \u0026#39;C,Eb,G,Bb\u0026#39; Cmin7 C 261.63 Eb 311.13 G 392 Bb 466.16 % harmony -notes \u0026#39;C,F,G\u0026#39; Csus4 C 261.63 F 349.23 G 392 That\u0026rsquo;s about all it can do for now, but I have many ideas I want to implement as I have big plans for it :-)\nearmuff # earmuff is a proof-of-concept compiler and interpreter for a programming language to write music.\nAfter I showed the first iteration of the interpreter, two people pointed me to Sonic Pi and it kinda is a similar concept but executed a bit differently. Whereas Sonic Pi\u0026rsquo;s is an advanced scriptable synthesizer with its own IDE, earmuff is simply a language to express music sheet as code and doesn\u0026rsquo;t come with anything but the interpreter and compiler for it.\nMy intent was to be able to write music as code in my usual programming environment, using diff, patch or even git for versionning my changes and applying new changes, and basically work with music in the same way I work with code when I don\u0026rsquo;t have instruments at hands.\nSo earmuff reads .muff source files, and either compiles them to SMF (Standard Midi File) files that can be played with a standard MIDI player, or interprets them into MIDI messages that are sent to a synthesizer to play the source code in real-time.\nIt relies on go-harmony to make sense of notes and chord names, spotting errors as it converts the source code to its internal format, and validating that the tracks are structurally consistent (ie: time signature violations) just as if it was checking for syntax or grammar errors.\nA .muff file may look like this for a single-track 12-bars blues:\nproject { bpm 120; time 4 4; instrument guitar { bar { whole chord C7 on 1/1; } bar { whole chord F7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord F7 on 1/1; } bar { whole chord F7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord G7 on 1/1; } bar { whole chord F7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord G7 on 1/1; } } } \u0026hellip; or may be more complex for multi-tracks projects, like these few bars from Django\u0026rsquo;s Nuages (I know, it\u0026rsquo;s not quite right but hey, it was late and I\u0026rsquo;m still learning the language 😅):\nproject { bpm 80; time 4 4; instrument guitar { bar { 8th note C# on 3/1; 8th note D on 3/2; 8th note A on 3/3; 8th note G# on 3/4; 8th note G on 4/1; 8th note F# on 4/2; } bar { half note F on 1/1; 8th note E on 4/2; } bar { half note Fbb on 1/1; quarter note Eb on 2/2; 8th note D on 4/2; } bar { whole note D on 1/1; } } instrument piano { bar {} bar { whole chord Eb9 on 1/1; } bar { half chord Am7b5 on 1/1; half chord D7b9 on 3/1; } bar { half chord Gmaj7 on 1/1; quarter chord Am7 on 3/1; quarter chord Bm7b5 on 4/1; } } instrument percussive { bar {} bar { half cymbal on 1/1; } bar { half cymbal on 1/1; } bar { half cymbal on 1/1; } } } By default, earmuff operates as an interpreter so running the following command will playback the source if a synthesizer is detected (currently only FluidSynth is):\n% earmuff nuages.muff Your browser does not support the audio element. The -verbose option may be passed, in which case earmuff outputs MIDI messages being sent to the synthesizer in real-time for debugging:\nsynth \u0026lt;- 0 ProgramChange channel: 0 program: 25 synth \u0026lt;- 2 ProgramChange channel: 10 program: 113 synth \u0026lt;- 1 ProgramChange channel: 1 program: 1 synth \u0026lt;- 0 NoteOn channel: 0 key: 49 velocity: 120 synth \u0026lt;- 0 NoteOn channel: 0 key: 50 velocity: 120 synth \u0026lt;- 0 NoteOn channel: 0 key: 57 velocity: 120 synth \u0026lt;- 0 NoteOff channel: 0 key: 49 synth \u0026lt;- 0 NoteOn channel: 0 key: 56 velocity: 120 synth \u0026lt;- 0 NoteOff channel: 0 key: 50 synth \u0026lt;- 0 NoteOff channel: 0 key: 57 synth \u0026lt;- 0 NoteOn channel: 0 key: 55 velocity: 120 synth \u0026lt;- 0 NoteOff channel: 0 key: 56 synth \u0026lt;- 0 NoteOn channel: 0 key: 54 velocity: 120 synth \u0026lt;- 0 NoteOff channel: 0 key: 55 synth \u0026lt;- 0 NoteOff channel: 0 key: 54 synth \u0026lt;- 2 NoteOn channel: 10 key: 51 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 51 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 49 velocity: 120 synth \u0026lt;- 0 NoteOn channel: 0 key: 53 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 55 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 65 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 58 velocity: 120 Passing the option -out will cause it to compile a .mid file:\n% earmuff -out nuages.mid nuages.muff \u0026hellip; which can either be played by a standart MIDI player, imported in tools that can ingest MIDI files such as a DAWs (here Logic Pro X):\n\u0026hellip; or tools like Guitar Pro which can render sheet music from a MIDI file, play it back after altering speed, instruments, etc\u0026hellip;\nThis is a toy project at a very early stage, I don\u0026rsquo;t have very serious plans for it but I\u0026rsquo;ll keep working on it as it improves my overall knowledge, and I kinda like the idea of being able to write music as code from my bed while the kid sleeps by my side (also I\u0026rsquo;m far faster as writing code than editing sheets) :-)\nThere are still bugs and glitches in the MIDI generation but the MP3 I linked above was converted from the .mid built by the source code just above it, so it may not be perfect yet but it works pretty decently in my opinion.\nMy current plans are to cleanup the proof-of-concept, make it work on at least macOS, Windows, Linux and OpenBSD, and then extend it with new features such as loops and functions. A very nice feature would be to allow it to read from MIDI and generate the corresponding .muff code, allowing me to auto-generate code from a MIDI input (like a keyboard for example), or simply allowing me to export a SMF from another tool to work on it with earmuff and export it back to the other tool: being able to switch back and forth between VScode and Guitar Pro or Logic Pro X would be awesome.\nWhat\u0026rsquo;s next ? # Next is A DAMN BREAK cause I said last month I\u0026rsquo;d take a couple months and couldn\u0026rsquo;t even hold my word for a single one. I\u0026rsquo;ll be marrying in a couple weeks, going on vacations a few weeks later, and got plenty of code-unrelated things to finish.\nTake care, stay tuned, I\u0026rsquo;ll post as soon as I resume my work !\n","date":"14 June 2022","permalink":"/posts/2022-06-14/june-2022-go-harmony-and-earmuff/","section":"Posts","summary":"TL;DR: did a lot of music, even while writing code. Code-unrelated work # As last month, I\u0026rsquo;ll start with code-unrelated work !\nFirst, here\u0026rsquo;s my progress on learning Hyunsoo Lee\u0026rsquo;s adaptation of Bach\u0026rsquo;s Air on G string, still not quite right and with some missing parts but\u0026hellip; slowly getting there:","title":"June 2022: go-harmony and earmuff"},{"content":" TL;DR: tons of plakar work, most notably on indexes, performances, clone \u0026amp; sync and fuse. Code-unrelated work # I\u0026rsquo;ll start with code unrelated work !\nI\u0026rsquo;ve picked up playing electric guitar and metal after years of playing mostly jazz and classical on nylon-string guitars. After a couple weeks of enduring finger pain while re-conditionning myself to playing on steel strings, I decided to give myself a challenge and learn some shredding, a topic I conveniently always dodged :-)\nI took this track which I absolutely love, Hyunsoo Lee\u0026rsquo;s adaptation of Bach\u0026rsquo;s Air on G string, and started practicing because it uses some techniques I\u0026rsquo;m not too good with such as sweeping and fast runs:\nI still have a lot of work to do, particularly when playing at full speed, but I\u0026rsquo;m very happy with the progress I\u0026rsquo;m making on this so I\u0026rsquo;m sharing :-)\nI\u0026rsquo;m also working on a couple other covers, I\u0026rsquo;ll post when I\u0026rsquo;m happy with them too.\nSplit indexes # As a follow up to refactor from previous month, I continued to work on improving the data structures used by plakar.\nJust as I had split Index and Metadata, I have now also split the Index and Filesystem view of a snapshot so a plakar snapshot now has three indexes: Metadata contains, well, metadatas for lookups of general purpose informations regarding a snapshot, Index contains everything necessary to figure out the structure of files and what\u0026rsquo;s needed to reconstruct it, and Filesystem contains everything necessary to figure otu the structure of snapshotted filesystems.\nBasically, if you want to know the size of a snapshot, Metadata is the only index necessary. If you want to figure out which chunks constitute /etc/passwd, Index is the only index necessary. If you want to figure out which files are in /etc, or what were the permissions of /etc/passwd, Filesystem is the only index necessary. As most command only rely on one of these simultaneously, this speeds up things as commands do not have to wait for the three indexes to be fetched before being able to work.\nNot all commands benefit from this split yet as they need to be adapted, but this will be a transparent change where some commands get a performance boost out of nowhere as I reimplement them properly.\nBinary indexes # Up until last month, I had used json as the storage format for indexes, and there were two reasons for that:\nFirst, it allowed me to not worry too much about serialization. I could simply json-marshal the Metadata, Index and Filesystem indexes which would generate a set of dictionnaries, both usable with a simple json-unmarshal and no state \u0026ldquo;rebuilding\u0026rdquo; (that\u0026rsquo;s converting from a disk format to different structures in memory, imposing a conversion cost).\nThen, it also helped me debug things as I could always use an unencrypted plakar and rely on the jq command to read the indexes, check that they contained the expected data and generally play with them with json-aware tools.\nThe problem is that json serialization and deserialization comes at a cost, both in terms of space consumption as it\u0026rsquo;s far from being a very compact format and snapshot may contain dozens of thousands of objects leading to a loooooooot of waste, but also in terms of performances as the format itself is more costly to serialize and deserialize than binary formats.\nI have ideas where I want to go with serialization of indexes but a good first step was to switch from json to a decent binary format, one that would be more efficient to start with but which would also allow me to ensure that nothing relied on json anymore.\nAfter some investigations, I ran into msgpack which seemed to be a good candidate, was already fairly popular in other projects, but more importantly didn\u0026rsquo;t require too much work to plug in.\nIt claimed to provide much better performances than json serialization, which I didn\u0026rsquo;t observe myself, but helped cut the index sizes considerably as in average it shrank them by 25% and up to 50% in some of my tests.\nCompact indexes # With work being done to split indexes and switch them to a binary format, I decided to stay focused on index improvements and trying to make them as compact as I could.\nThe reason is that indexes weight \u0026ldquo;two\u0026rdquo; sizes: the storage size, when they\u0026rsquo;re written to disk, and the runtime size, when they\u0026rsquo;re loaded into memory. Both these sizes have benefits being optimized, the former allowing faster deserialization, the latter providing a smaller footprint and faster serialization.\nIn an ideal world, the data structures stored on disk are compressed to the smallest possible size to reduce loading time, but the data structures loaded in memory should be as close as possible to their disk counterparts to avoid conversion costs. The optimal case would be that indexes are compressed in such a way that they can be loaded compressed in memory, and usable without decompressing them.\nSo first of all, I did a pass to compress data types as I knew there would be an instant gain from a (seemingly) simple change. The project uses uuid (16 bytes) and sha256 sums (32 bytes) all over the place, but for convenience I used human-readable representations of these formats, that is 36-bytes strings for uuid and 64-bytes strings for sha256. These landed in many data structures in their human-readable representation, meaning that 20 bytes were wasted every time a uuid was stored and 32 bytes were wasted every time a sum was stored. This doesn\u0026rsquo;t look much but that\u0026rsquo;s until you realize that most of what\u0026rsquo;s stored in the indexes are mappings that associate sums to each others, so when a snapshot contains dozens of thousands of files the waste builds up: gain was already quite nice at this point as it halved the size of indexes on the snapshots of my work directory.\nThen, I did another pass to compress index size further. As I wrote above, most of what\u0026rsquo;s stored in indexes are mappings that associate sums to each others, and to achieve that there are several mappings and reverse mappings which tend to repeat the same information in reverse order. To answer questions such as what are the chunks in object X ? or what objects contain the chunk Y ?, you need to have a mapping associating the checksum of object X to a serie of chunk checksums, but also to have a mapping associating the checksum of chunk Y to a serie of object checksums. This means that a ton of 32-byte checksums are repeated duplicated and appear at least twice in indexes. By using a symbol table and an indirection, it is possible to reduce the index sizes by replacing 32-byte checksums with a much smaller and unique 8-byte identifier. Because each checksums appear at least twice, the index shrinks by roughly 50% in the worse case and is effectively compressed in memory. at the cost of a O(1) indirection in the symbol table.\nThese were cumulative so by the end of this round of optimizations, indexes were about 4x times smaller than what they were to start with, and because of how the compression works it also means they don\u0026rsquo;t grow as much as they used to.\nI still have ideas to improve further on indexes compression, but at this point I\u0026rsquo;m satisfied with what I have and the current trade-off ratio between size and convenience. I know I have some improvement margin still but will only use it if absolutely necessary.\nDisplay index, filesystem and metadata # Because the changes above made it harder to investigate the content of indexes, as they are no longer stored in a human-readable format, I implemented a helper command to json-dump their content allowing to inspect them with:\n% plakar info -metadata 39| jq .|grep IndexID \u0026#34;IndexID\u0026#34;: \u0026#34;39d10f01-992f-471c-88f7-c730b003c8b6\u0026#34;, % plakar info -filesystem 39| jq .| grep cmd_pull.go \u0026#34;cmd_pull.go\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;cmd_pull.go\u0026#34;, \u0026#34;/Users/gilles/wip/github.com/poolpOrg/plakar/cmd/plakar/cmd_pull.go\u0026#34;: 16, % plakar info -index 39 | jq . | less [...] \u0026#34;12\u0026#34;: { \u0026#34;Start\u0026#34;: 0, \u0026#34;Length\u0026#34;: 8423 }, \u0026#34;13\u0026#34;: { \u0026#34;Start\u0026#34;: 0, \u0026#34;Length\u0026#34;: 1054 }, \u0026#34;14\u0026#34;: { \u0026#34;Start\u0026#34;: 0, \u0026#34;Length\u0026#34;: 1889 }, [...] They will load the binary format of indexes and dump them to stdout after encoding them to json. It\u0026rsquo;s a bit hard to write about this feature, so feel free to look at the output of these commands by yourself.\nPerformances, profiling and cache # I did something I had not done before: try an alternate solution to plakar and compare runtime execution timings.\nI expected plakar to be a bit slower because it lacks maturity and I didn\u0026rsquo;t spend much time optimizing yet, but I intuitively thought it would be roughly 25% slower with an easily achievable margin of improvement.\nI installed restic and after a few minutes of reading documentation to figure out the command line interface, I ran a couple snapshots\u0026hellip; and was left speechless. It was considerably faster, not by 25% but more like 25x faster. To be more precise, the initial snapshot was about 25% faster than with plakar, which was what I expected it to be\u0026hellip; but subsequent snapshots took only a couple seconds when they could take a minute with plakar. I honestly didn\u0026rsquo;t think they could be so fast.\nAfter a few seconds of contemplation and investigation to make sure that I didn\u0026rsquo;t miss something on the command line and that it actually did the subsequent snapshots, I decided to do a first pass of optimization right away to bring my plakar timings at least in the 25% range. I didn\u0026rsquo;t want to work on optimization right away, because it would be premature with all of the changes still happening, and I don\u0026rsquo;t want them to go in the way of design at this point\u0026hellip; but I also didn\u0026rsquo;t want to create a situation were the design would lock me too far from alternatives performance-wise. I mainly needed reassurance that suboptimal performances remained in the realm of achievable improvements when I\u0026rsquo;ll be tackling them.\nI wrote a runtime profiling mechanism, that you can enable with the -profiling, and which allowed me to track what were the main areas that needed improvements:\n% plakar -profiling push [profiling]: snapshot.PutMetadata: calls=1, min=325.25µs, avg=325.25µs, max=325.25µs, total=325.25µs [profiling]: cache.Commit: calls=1, min=270.75µs, avg=270.75µs, max=270.75µs, total=270.75µs [profiling]: cache.New: calls=1, min=66.272166ms, avg=66.272166ms, max=66.272166ms, total=66.272166ms [profiling]: snapshot.GetCachedObject: calls=26, min=208.584µs, avg=864.285µs, max=1.508667ms, total=22.471418ms [profiling]: snapshot.PutIndex: calls=1, min=1.446208ms, avg=1.446208ms, max=1.446208ms, total=1.446208ms [profiling]: storage.tx.PutIndex: calls=1, min=141.375µs, avg=141.375µs, max=141.375µs, total=141.375µs [profiling]: storage.tx.PutMetadata: calls=1, min=111.917µs, avg=111.917µs, max=111.917µs, total=111.917µs [profiling]: storage.Close: calls=1, min=83ns, avg=83ns, max=83ns, total=83ns [profiling]: storage.CheckChunk: calls=114, min=4.459µs, avg=8.217µs, max=44.708µs, total=936.743µs [profiling]: snapshot.CheckChunk: calls=114, min=5.125µs, avg=9.433µs, max=79.417µs, total=1.075417ms [profiling]: snapshot.CheckObject: calls=26, min=6.291µs, avg=10.828µs, max=46.709µs, total=281.541µs [profiling]: snapshot.Create: calls=1, min=1.63425ms, avg=1.63425ms, max=1.63425ms, total=1.63425ms [profiling]: storage.GetObject: calls=26, min=5.583µs, avg=9.753µs, max=44.709µs, total=253.584µs [profiling]: cache.PutFilesystem: calls=1, min=46.542µs, avg=46.542µs, max=46.542µs, total=46.542µs [profiling]: storage.tx.PutFilesystem: calls=1, min=154.917µs, avg=154.917µs, max=154.917µs, total=154.917µs [profiling]: snapshot.PutFilesystem: calls=1, min=1.157125ms, avg=1.157125ms, max=1.157125ms, total=1.157125ms [profiling]: cache.Create: calls=1, min=12.5µs, avg=12.5µs, max=12.5µs, total=12.5µs [profiling]: storage.Open: calls=1, min=131.042µs, avg=131.042µs, max=131.042µs, total=131.042µs [profiling]: storage.Transaction: calls=1, min=1.625625ms, avg=1.625625ms, max=1.625625ms, total=1.625625ms [profiling]: cache.PutMetadata: calls=1, min=12.667µs, avg=12.667µs, max=12.667µs, total=12.667µs [profiling]: snapshot.Commit: calls=1, min=3.517208ms, avg=3.517208ms, max=3.517208ms, total=3.517208ms [profiling]: cache.GetPath: calls=26, min=175.792µs, avg=778.532µs, max=1.490375ms, total=20.241836ms [profiling]: cache.PutIndex: calls=1, min=113.416µs, avg=113.416µs, max=113.416µs, total=113.416µs [profiling]: storage.tx.Commit: calls=1, min=132.042µs, avg=132.042µs, max=132.042µs, total=132.042µs % This made me realize that this very high performance gap was caused by a major difference in what it did compared to restic. It attempted to maintain repository consistency with every operation, using complex sequences of operations to maintain correct reference counts, ensuring that objects were deleted as soon as they were no longer referenced or when a push was aborted, etc\u0026hellip; while restic didn\u0026rsquo;t take care of that with every operation but used a prune command to do the cleanup.\nI really thought it would be neat to keep the repository consistency the way plakar did, but it became obvious that the cost was too prohibitive and no amount of optimizations I could think of would counterweight the cost of this. All operations were made more complex and the cost of consistency was paid with every call, when restic benefited from simpler operations and the cost of consistency was paid only when actually requested.\nI backtracked on my ideas and reimplemented plakar\u0026rsquo;s push using a much simpler strategy, removing all attempts at playing hard-links and reference counts games to track and cleanup, but implementing a cleanup operation like restic did.\nThis immediately brought a huge improvement as plakar remained in the range of 25% for the first push (with some gain though), but immediately fell to the same performances as restic for subsequent pushes. This looks promising because while it has roughly the same timings, unlike restic, plakar uses compression by default and didn\u0026rsquo;t go through actual optimization yet. This means that performances will likely put it below restic in terms of push times, while repository consumption is already much better when snapshots are compressible. I played a lot with pushing my ~2GB work directory and for the same push time, it would weight ~2GB on restic and only 350MB on plakar.\nWhile I was there, I had a look at the repository structure by poking at the created directories and realized that despite a few differences both were very, very, very close. This gave me confidence that I was not completely off with how I tackled this project :-)\nThe profiler also helped me realize that there was a bug in the implementation of the local cache. It is implemented using SQLite and I didn\u0026rsquo;t create the proper indexes, which caused cache accesses to be very costly and negated the benefits of using one as soon as there were a lot of data in it. After fixing, things were much better but I came to the conclusion that SQLite was not the proper tool for my problem, what I really need is a very simple key-value database. I temporarily replaced SQLite with LevelDB for the sake experimenting, but am not too happy as it imposes a lock that makes the cache unusable for reads when a push is in progress. For the time being I made it so the cache is skipped while a push is in progress, I\u0026rsquo;ll be looking at a different solution that allows reads of older versions of the data when a write is in progress.\nParallelized pull # The plakar pull operation was not parallelized so while a push could benefit from multiple cores, pull would restore snapshot content sequentially and one file at at time.\nI rewrote it so that it would both take advantage of the snapshot Reader interface I wrote about last month, simplifying code, but also so that it could parallelize restoring and speed things up.\nThere are other things that can be done to speed up things, like not restoring files which are already there in their last version in the target directory, or looking at stat structure to figure out if some files are hard links to others. Work still needs to be done but it\u0026rsquo;s already a better version so\u0026hellip; meh, no rush.\nAdded a progress bar # That\u0026rsquo;s a minor change but one that I really wanted to do and which someone asked me for, prompting me to start working on it.\nI implemented the -progress option for plakar push, plakar pull and plakar check, but I\u0026rsquo;m not too happy with the way I achieved it. I\u0026rsquo;ll revisit later but thought I\u0026rsquo;d leave it as is for now as it\u0026rsquo;s still more helpful than not having it.\nReworked clone and sync # OK, this is by far the most interesting feature in my opinion.\nThe goal of plakar clone is to create an exact copy of an existing repository, sharing its repository ID and configuration, including encryption key.\nThe goal of plakar sync is to synchronize two repositories by transfering the chunks and objects that are missing in the target repository, effectively making an exact copy of a snapshot available in a different repository.\nI had already implemented plakar clone and plakar sync but they both had shortcomings. First, they both only supported working with local repositories, ones available on the local filesystem. Then, sync only supported synchronizing to clone repositories or between unencrypted repositories, but it didn\u0026rsquo;t support synchronizing an unencrypted repository with an encrypted one for example. This was nice already but what I really wanted was for this to work regardless of where a repository is, and regardless of the configuration of both repositories.\nI reworked both commands so that they know support working with arbitrary repositories, and doing the necessary work so there\u0026rsquo;s no issue when synchronizing repositories not sharing the same configuration.\nWhy is this the most interesting feature in my opinion ?\nWell what I really want with backups is the ability to have them at multiple locations, preferably in different regions. I\u0026rsquo;ve already lost a few servers at cheap hosts where no disk recovery was offered, and I know of at least one DC that burnt and lost data of all local servers in the process, killing backups that were located on different machines of the same DC. I want to be able to spread backups at multiple locations but I don\u0026rsquo;t want to have to think too hard about that.\nFurthermore, I\u0026rsquo;m not necessarily interested in having ALL backups encrypted, nor synchronized at the same rate. For instance, I like having backups on my local machine readily available in case I mistakenly trash something, I like having unencrypted backups on my NAS at home in case a machine is unrecoverable, but I\u0026rsquo;d also like having off-site encrypted backups at one or two cheap hosts in case there was a fire at home and I lost everything. Not all these backups need to happen at the same frequency and I don\u0026rsquo;t want to have complex strategies here and there, depending on the machine. I can do that with several solutions and a bunch of scripting, but I\u0026rsquo;ve been there, done that, and it was never fun.\nWhat clone and sync allow me to do is to have exactly what I described in just a few commands, I can create an unencrypted repository on my laptop:\n% plakar on ~/plakar create -no-encryption Then create a clone on my NAS:\n% plakar on ~/plakar clone to ssh://nas.poolp.org Both repositories are absolutely identical at this point.\nMy laptop can do a plakar push multiple times a day from a cron, then every few hours I can synchronize both repositories:\n% plakar on ~/plakar sync to ssh://nas.poolp.org which brings the NAS repository up-to-date with the local repository. I could also synchronize the other way. Say I had multiple machines pushing to the NAS and I wanted my local repository to grab all snapshots, then I\u0026rsquo;d just:\n% plakar on ~/plakar sync from ssh://nas.poolp.org which brings the local repository up-to-date with the NAS repository. Just like I could clone the NAS repository to bring back an exact copy if I had reinstalled my machine:\n% plakar on ~/plakar clone from ssh://nas.poolp.org But where I find it very useful is that I could create encrypted repositories remotely:\n% plakar on ssh://backups.poolp.org create password: and synchronize either my NAS or laptop with them:\n% plakar on ~/plakar sync to ssh://backups.poolp.org password: % plakar on ssh://nas.local sync to ssh://backups.poolp.org password: The -keyfile option can be provided to point to a file containing the password so it doesn\u0026rsquo;t need to be typed, and allows calling sync from a cron job.\nWith all of this, it makes it trivial to have multi-site backups all synchronized one with another at different rates.\nReworked snapshot Reader # Last month, I wrote about how I implemented a snapshot Reader interface and how it helped unlock features and simplify code.\nThe next feature I\u0026rsquo;ll write about required me to rework the Reader interface. A Reader interface is one that allows a Read() operation, but I really needed a ReadSeekCloser interface, one that allows Read(), Close() and more importantly Seek() so that I can move the read cursor to arbitrary locations in a stream.\nI wrote the interface to implement Seek() on an object so that it would locate the appropriate chunk and maintain an offset to point at the proper position. This didn\u0026rsquo;t take too long as I did it in an hour or so, but it was painful and I lost my soul in the process.\nFUSE proof-of-concept # FUSE is a \u0026ldquo;Filesystem in USEerspace\u0026rdquo; mechanism which allows creating mountpoints that expose informations as if they were files on the filesystem. It\u0026rsquo;s not available on all systems, but when it is, fun stuff are available to play with.\nI implemented experimental FUSE support in plakar, making it possible to mount a repository on the filesystem and browse it as if all snapshots had been restored there.\nWhat this means is that I can do:\n% plakar on ssh://nas.local mount /mnt/nas Then I can browse just as if all the repository had been restored below /mnt/nas:\n% ls -l /mnt/nas total 5012576 drwx------ 1 gilles staff 41865948 22 May 22:21 94c815c7-15c6-4a8a-af3a-a83ca620fb55 drwx------ 1 gilles staff 41865948 22 May 22:22 a961b301-e331-4fd1-859e-7ae1d6fca17f drwx------ 1 gilles staff 2482699293 22 May 22:22 ade656c9-56e7-4e84-b90f-90b3826b08e2 % cd /mnt/nas/94c815c7-15c6-4a8a-af3a-a83ca620fb5 % ls Users % cd Users/gilles/Wip/github.com/poolpOrg/plakar/cmd/plakar % ls -l total 51216 -rw-r--r-- 1 gilles staff 1426 7 May 13:58 cmd_browser.go -rw-r--r-- 1 gilles staff 2274 11 May 21:53 cmd_cat.go -rw-r--r-- 1 gilles staff 1748 6 May 23:31 cmd_check.go -rw-r--r-- 1 gilles staff 1930 10 May 09:09 cmd_checksum.go -rw-r--r-- 1 gilles staff 1973 6 May 23:31 cmd_cleanup.go -rw-r--r-- 1 gilles staff 4590 6 May 23:31 cmd_clone.go -rw-r--r-- 1 gilles staff 2637 6 May 23:31 cmd_create.go -rw-r--r-- 1 gilles staff 7124 6 May 23:31 cmd_diff.go -rw-r--r-- 1 gilles staff 2582 6 May 23:31 cmd_exec.go -rw-r--r-- 1 gilles staff 2818 6 May 23:31 cmd_find.go -rw-r--r-- 1 gilles staff 8423 6 May 23:31 cmd_info.go -rw-r--r-- 1 gilles staff 1889 6 May 23:31 cmd_keep.go -rw-r--r-- 1 gilles staff 7776 6 May 23:31 cmd_ls.go -rw-r--r-- 1 gilles staff 1784 7 May 13:55 cmd_mount.go -rw-r--r-- 1 gilles staff 2393 6 May 23:31 cmd_pull.go -rw-r--r-- 1 gilles staff 1992 6 May 23:31 cmd_push.go -rw-r--r-- 1 gilles staff 1679 6 May 23:31 cmd_rm.go -rw-r--r-- 1 gilles staff 1228 6 May 23:31 cmd_server.go -rw-r--r-- 1 gilles staff 1684 6 May 23:31 cmd_shell.go -rw-r--r-- 1 gilles staff 1054 6 May 23:31 cmd_stdio.go -rw-r--r-- 1 gilles staff 5993 6 May 23:31 cmd_sync.go -rw-r--r-- 1 gilles staff 3096 6 May 23:31 cmd_tarball.go -rw-r--r-- 1 gilles staff 1147 6 May 23:31 cmd_version.go -rwxr-xr-x 1 gilles staff 26080594 12 May 23:14 plakar -rw-r--r-- 1 gilles staff 7562 8 May 21:59 plakar.go -rw-r--r-- 1 gilles staff 9113 6 May 23:31 utils.go % cat cmd_version.go /* * Copyright (c) 2021 Gilles Chehade \u0026lt;gilles@poolp.org\u0026gt; * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED \u0026#34;AS IS\u0026#34; AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ package main import ( \u0026#34;flag\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;github.com/poolpOrg/plakar/storage\u0026#34; ) const VERSION = \u0026#34;0.0.1\u0026#34; func init() { registerCommand(\u0026#34;version\u0026#34;, cmd_version) } func cmd_version(ctx Plakar, repository *storage.Repository, args []string) int { flags := flag.NewFlagSet(\u0026#34;version\u0026#34;, flag.ExitOnError) flags.Parse(args) fmt.Println(VERSION) return 0 } % Note that despite cat showing the content of a file here, files aren\u0026rsquo;t really on the filesystem but streamed from the repository using the ReadCloseSeeker interface. As such, even though I can browse all the snapshots of the repository and access any file content, they don\u0026rsquo;t consume disk space locally, my 256GB laptop could very well mount a repository storing far more than that and allow me to browse and cat any file.\nAlso note that I mounted a remote repository located at ssh://nas.local which could very well be encrypted as well, this would work just as fine and transparently. This is experimental as I had no prior experience playing with FUSE drivers, there can be a few glitches here and there, I fixed all the ones I found but still.\nConditional builds # I added support for conditional builds.\nExperimental FUSE support only works on Linux and macOS as far as I know, and doesn\u0026rsquo;t work for sure on OpenBSD for reasons that I\u0026rsquo;ve started investigating but which are beyond plakar, so I wanted to make sure that I could still build plakar with the experimental feature removed where it was known not to work\u0026hellip; rather than break the entire build.\nWhile at it, the browser feature prevented plakar from building on older Go version, such as the one shipped on some Ubuntu machines I administer for friends. I made sure that this conditionally built depending on the version of Go.\nFull text search experiment # I quickly experimented with FTS in plakar, allowing me to search for all snapshots and files containing a particular content:\n% plakar search PutIndex 9 matches, showing 1 through 9, took 80.125µs 1. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/storage.go (0.406516) 2. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/cache/cache.go (0.362482) 3. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/network/server.go (0.277187) 4. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/snapshot/snapshot.go (0.264234) 5. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/client/client.go (0.262951) 6. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/database/database.go (0.254370) 7. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/fs/fs.go (0.238110) 8. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/poolp.org/content/posts/2021-10-01/index.md (0.211895) 9. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/poolp.org/docs/posts/2021-10-26/october-2021-mostly-plakar/index.html (0.143937) % plakar pull 46a6777c:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/storage.go % ls Users/gilles/wip/github.com/poolpOrg/plakar/storage/storage.go storage.go % This was not committed and I have no plans to commit it for the time being, a lot of work still needs to be done before I focus on such features, but it shows how plakar can be improved in ways that my current rsync/tar backups will never match.\nS3 experiment # I also did a small experiment writing an s3 backend for plakar, allowing it to store repositories on Amazon s3 or on a minio server.\nI\u0026rsquo;m not too comfortable with the API as I never used s3 before, surely I\u0026rsquo;m not using it correctly, but someone asked me if this would be supported and so I gave it a try.\nAfter about half an hour, I had a plakar repository hosted on a local s3:\n% plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar create -no-encryption % plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar push /private/etc 2\u0026gt;/dev/null % plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar ls 2022-05-08T21:19:14Z 21388086 3.1 MB 0s /private/etc % plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar ls 21:/private/etc/passwd 2022-03-26T07:21:13Z -rw-r--r-- root wheel 7.9 kB passwd % plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar cat 21:/private/etc/passwd | tail -3 _darwindaemon:*:284:284:Darwin Daemon:/var/db/darwindaemon:/usr/bin/false _notification_proxy:*:285:285:Notification Proxy:/var/empty:/usr/bin/false _oahd:*:441:441:OAH Daemon:/var/empty:/usr/bin/false % This will likely need more work but it was a successful proof of concept :-)\nWhat\u0026rsquo;s next ? # Next is a couple months of rest.\nI\u0026rsquo;ll be marrying at the end of June and there are still a lot of things to do, I doubt I\u0026rsquo;ll find much spare time to write code this month\u0026hellip; just as I think the month of July will be needed to recover as a wedding is a lot to deal with for someone with an anxiety disorder :-)\nStay tuned, I\u0026rsquo;ll post as soon as I resume my work !\n","date":"22 May 2022","permalink":"/posts/2022-05-22/may-2022-yet-again-tons-of-plakar-stuff/","section":"Posts","summary":"TL;DR: tons of plakar work, most notably on indexes, performances, clone \u0026amp; sync and fuse. Code-unrelated work # I\u0026rsquo;ll start with code unrelated work !","title":"May 2022: yet again, tons of plakar stuff "},{"content":" TL;DR: I refactored internal structures to split metadata from the index, implemented an stdio server and finally added SSH support. The plakar.io website # A project needs a website so\u0026hellip; I published the plakar.io website.\nThis will centralize instructions and documentation for the plakar project, but it is still a work in progress as development still happens at a fast pace that\u0026rsquo;s hard to keep up with.\nThe website is published from commits to the plakar.io Github repository, so feel free to submit pull requests to improve it.\nReworked internal structures # A snapshot generates an index that contains everything needed to map chunks and objects into a filesystem hierarchy. The index is very small comparatively to the snapshot size, but it may still be relatively large as a multi-gigabyte snapshot may produce a multi-megabyte index, which still requires time to fetch, decompress and deserialize. In many cases, this is fast enough that you don\u0026rsquo;t really feel it, but the bigger the snapshot the more laggy it feels when browsing it through the UI or when listing content in a snapshot.\nFor many commands there\u0026rsquo;s no need to access the index because it\u0026rsquo;s not so much the content of files that is needed, but really some metadata or statistics. Good examples of these are plakar ls in the repository or plakar info:\n% plakar ls 2022-04-19T22:07:03Z a9d01365 2.4 GB /Users/gilles/Wip 2022-04-19T22:08:11Z f87bd54e 2.4 GB /Users/gilles/Wip 2022-04-19T22:11:46Z 70cc55f6 2.4 GB /Users/gilles/Wip 2022-04-20T22:48:55Z 1c537551 2.4 GB /Users/gilles/Wip % plakar info a9 Uuid: a9d01365-4f97-42e1-8b19-59e4fe2f630a CreationTime: 2022-04-20 00:07:03.728327 +0200 CEST Version: 0.1.0 Hostname: desktop.local Username: gilles CommandLine: ./plakar -no-cache push /Users/gilles/Wip MachineID: 3657f7dd-c012-53ba-b8e6-73e08d311a6a PublicKey: Directories: 7589 Files: 77206 NonRegular: 84 Pathnames: 77206 Objects: 63306 Chunks: 65702 Size: 2.4 GB (2445114798 bytes) Index Size (uncompressed): 113 MB (2445114798 bytes) Here, the index for snapshot a9d01365 is 113MB uncompressed (really 21MB on disk), which means that the plakar ls above required decompressing and deserializing roughly 4 times that size to list the 4 snapshots, then decompressing and deserializing that size once for the plakar info above, just to display some informations that didn\u0026rsquo;t rely on the filesystem hierarchy and content.\nI have split snapshots into two main structures: Metadata and Index.\nThe first structure contains general informations and statistics regarding the index, just enough that the structure fits in a few KiloBytes, but can avoid relying on the much larger index for commands that don\u0026rsquo;t involve diving into the content of a snapshot. The second structure contains the mappings themselves, and is only accessed when data needs to be reconstructured somehow.\nThis allowed plakar ls to get a serious boost when listing a large number of large snapshots, but it is really a first step as I\u0026rsquo;m evaluating solutions to shard the index and allow fetching subsets of it.\nImplemented a Reader interface # When a snapshot is created, plakar will consider each file as an object consisting of one or many chunks. The repository stores the object structure and chunks as separate resources, and so when a file is being recovered this is done by first fetching the structure, then fetching each chunk.\nFor example, to implement plakar cat, the code would look something like this (error handling omitted for simplicity):\n[...] object := snapshot.LookupObjectForPathname(pathname) for _, checksum := range object.Chunks { data, _ := snapshot.GetChunk(checksum) os.Stdout.Write(data) } [...] As every call may fail and needs to be checked, the code is actually far more verbose as can be seen below:\n[...] object := snapshot.LookupObjectForPathname(pathname) if object == nil { logger.Error(\u0026#34;%s: could not open file \u0026#39;%s\u0026#39;\u0026#34;, flags.Name(), pathname) continue } for _, checksum := range object.Chunks { data, err := snapshot.GetChunk(checksum) if err != nil { logger.Error(\u0026#34;%s: %s: could not obtain chunk \u0026#39;%s\u0026#39;: %s\u0026#34;, flags.Name(), pathname, checksum, err) continue } _, err = os.Stdout.Write(data) if err != nil { logger.Error(\u0026#34;%s: %s: could not write chunk \u0026#39;%s\u0026#39; to stdout: %s\u0026#34;, flags.Name(), pathname, checksum, err) break } } [...] and this construct needs to be replicated everywhere a file is accessed which is\u0026hellip; pretty much everywhere you do something with a snapshot.\nI thought it would be nice to have a Reader interface abstracting the details of reconstructing a file. This way, I could simply obtain a reader to a file and call Read() to obtain the next bytes without worrying about fetching chunks as the cursor advances:\nrd, _ := snapshot.NewReader(pathname) buf := make([]byte, 16*1024) for { nbytes, err := rd.Read(buf) if err == io.EOF { break } os.Stdout.Write(data[:nbytes]) } In the example above, snapshot.NewReader(pathname) fetches the object structure for file located at pathname. As rd.Read() is called, it takes care of fetching chunks as needed to fill buf with the next 16k bytes.\nHow is that simpler than calling GetObject() and GetChunk() ?\nWell, first and foremost, there\u0026rsquo;s a large range of functions already available in Go to do what I want but which expect a Reader interface. So rather than rolling my own version of functions, by implementing the Reader interface, I can rely on existing stuff and reduce the amount of code I have to maintain in plakar.\nRather than re-implementing a loop to read chunks and write them to stdout, plakar cat could take advantage of the new Reader interface and the fact that os.Stdout is a Writer, allowing it to use the io.Copy() function part of the standard library to copy from a Reader to a Writer:\nrd, err := snapshot.NewReader(pathname) if err != nil { logger.Error(\u0026#34;%s: %s: %s\u0026#34;, flags.Name(), pathname, err) continue } _, err = io.Copy(os.Stdout, rd) if err != nil { logger.Error(\u0026#34;%s: %s: %s\u0026#34;, flags.Name(), pathname, err) continue } This is also the case for plakar tarball, which creates a tarball from a snapshort, and relies on the standard libraries\u0026rsquo; archive/tar. The code instantiates a tar.NewWriter() and I previously had something like:\nobj := snapshot.LookupObjectForChecksum(checksum) for _, chunkChecksum := range obj.Chunks { data, err := snapshot.GetChunk(chunkChecksum) if err != nil { logger.Error(\u0026#34;corrupted file %s\u0026#34;, file) continue } _, err = io.WriteString(tarWriter, string(data)) if err != nil { logger.Error(\u0026#34;could not write file %s\u0026#34;, file) continue } } which could be rewritten as follows, leaving it up to the standard library to loop and handle do the read and writes:\nrd, err := snapshot.NewReader(file) if err != nil { logger.Error(\u0026#34;could not find file %s\u0026#34;, file) continue } _, err = io.Copy(tarWriter, rd) if err != nil { logger.Error(\u0026#34;could not write file %s: %s\u0026#34;, file, err) continue } In commands such as plakar diff which require copying the full data to a buffer, it also makes it possible to rely on existing functions like ioutil.ReadAll():\nbuf := make([]byte, 0) rd, err := snapshot.NewReader(filename) if err == nil { buf, err = ioutil.ReadAll(rd) if err != nil { return } } In the UI, this is even nicer as it uses the standard libraries\u0026rsquo; http server which uses a Writer for the response to the client. Implementing a download endpoint boils down to using the new Reader interface, then copying it to the Writer rather than doing the whole fetch object then chunks and copy them in a loop logic.\nBasically, this simplifies a TON of areas in plakar and allows me to remove a lot of custom code for things that exist in the standard library.\nFixed handling of max concurrent files # During the last few months, a lot of work was poured into parallelizing operations in plakar to utilize resources as efficiently as possible and speed things up.\nThis led to some issues when processing large directories as you could end up opening a very large amount of files, sometimes exceeding the number of descriptors available to the process and resulting in EMFILE errors.\nMy initial thought was that I could simply implement a backoff mechanism, but this is complex and in the case of plakar it doesn\u0026rsquo;t really make sense: a backoff mechanism is useful as a mean to recover from bursts that cause resources exhaustion. If plakar is going to parallelize and constantly hit the backoff mechanism, it hints that the solution should be located elsewhere.\nContrarily to a network daemon which could accept a very large number of clients and take advantage of idle times from some to process others, plakar only opens files when it will actively read or write them, so it makes no sense opening files when it can\u0026rsquo;t process them. If it opens 1024 files but can only chunk concurrently 32 files before it uses all of the CPU time available, then having 992 opened files waiting to be processed is pointless. I decided to tie the number of descriptors to a factor of the number of cores available, currently 2 * n + 1, and this ensures that the number of descriptors is low enough not to provoke an EMFILE while high enough to keep cores busy. The factor may be adjusted in the future but this feels like the proper way to handle this.\nThis doesn\u0026rsquo;t solve the ENFILE errors, where the system descriptor table is full while the process itself has not hit a limit, but I\u0026rsquo;m wondering if it\u0026rsquo;s even worth having a backoff mechanism to retry as I don\u0026rsquo;t see how a backup tool can really recover from a system not being able to reliably allocate descriptors. I\u0026rsquo;d personally rather have the backup fail and restart it when the system is in better shape, than have a backup take a lot more time to complete and then not trust that backup and do another one just in case some failures where not handled properly by plakar. I\u0026rsquo;m still not decided on this, maybe I\u0026rsquo;m wrong about it, but I need more time to think about it.\nAnyways, with this change I was able to produce large snapshots that previously hit a too many open files error, without observing any significant performances degradation and while still using cores efficiently.\nplakar stdio server # I had mentionned my work on remote repositories in this post, but didn\u0026rsquo;t communicate much about it because it was mainly a proof of concept to validate that no repository primitive expected locality of the repository, not a serious attempt at writing the final network mode.\nRemote repositories work thanks to a TCP server accepting the repository primitives and mapping them to a local repository, the client is basically a proxy which doesn\u0026rsquo;t run the primitives locally but passes them to the server and waits for the result.\nBecause the server logic is isolated in a client handler function, I thought it would be nice if I could abstract the network layer and consider that the client handler could work using stdio rather than a network connection. Instead of passing the client handler a descriptor to a network connection:\nfunc handleConnection(conn *net.Conn) { [...] } func Server(repository *storage.Repository, addr string) { [...] go handleConnection(c) [...] } I converted it to using io.Reader and io.Writer interfaces and passed the same network connection for both:\nfunc handleConnection(rd io.Reader, wr io.Writer) { [...] } func Server(repository *storage.Repository, addr string) { [...] go handleConnection(c, c) [...] } This worked nicely, so I took it a step further and created a new entry point plakar stdio which would call the client handler but passing os.Stdin as the Reader and os.Stdout as the Writer.\nfunc handleConnection(rd io.Reader, wr io.Writer) { [...] } func Stdio() error { ProtocolRegister() handleConnection(os.Stdin, os.Stdout) return nil } It meant that I could now fork plakar stdio from a parent process and communicate with it through pipes or socketpairs. I did a few tests and it worked fine which made me very happy.\nHow is this any useful, you ask ?\nplakar over SSH # I\u0026rsquo;ve been willing to add plakar over SSH support for a while but couldn\u0026rsquo;t set my mind on the simplest way to do it. I did various experiments which were all either too complex or that led to unsatisfying experiences: if I use SSH, I want the full SSH experience with known hosts, authorized keys, public key authentication, etc\u0026hellip; so all attempts that involved implementing an SSH client myself either led to missing features or to a TON of code unrelated to plakar.\nHowever\u0026hellip;\nWith plakar stdio things are much simpler because I can simply use the ssh client to spawn a remote plakar stdio, then take advantage of the fact that the client will do the stdio mapping and expose the ssh transport through stdin and stdout for me:\n% ssh gilles@backups.poolp.org \u0026#39;plakar stdio\u0026#39; Using this method, plakar does not need to have a daemon running on the remote end, it just needs the command installed.\nSo I implemented the ssh:// protocol for plakar which simply forks a process for ssh and emits packets to its stdin while reading responses from its stdout. Because it uses the ssh command, I no longer have to worry about known hosts \u0026amp; such, they are already handled, and I can even benefit from having used ssh-agent so I don\u0026rsquo;t have to keep typing my passphrase:\n% plakar on ssh://backups.poolp.org ls 2022-04-24T08:18:04Z 9bca77d9 3.1 MB /private/etc % plakar on ssh://backups.poolp.org push /bin % plakar on ssh://backups.poolp.org ls 2022-04-24T08:18:04Z 9bca77d9 3.1 MB /private/etc 2022-04-24T14:54:12Z 02895414 13 MB /bin % It took me hours to implement various SSH proof of concepts, people on the discord channel have endured me quite a lot, but this last version\u0026hellip; took only two minutes to implement using a few lines of code.\nIt was both very satisfying and very frustrating :-)\nRemote creation of repositories # Until yesterday, using a remote plakar expected you to create the repository on the remote end before starting to use it.\nI implemented the creation primitives both on the server and client and so it is now possible to do as follows:\n% plakar on ssh://backups.poolp.org create -no-encryption % plakar on ssh://backups.poolp.org push /bin % plakar on ssh://backups.poolp.org ls 2022-04-24T15:02:48Z 181702cd 13 MB /bin % % plakar on ssh://backups.poolp.org/tmp/plakar create -no-encryption % plakar on ssh://backups.poolp.org/tmp/plakar push /bin % plakar on ssh://backups.poolp.org/tmp/plakar ls 2022-04-24T15:03:37Z 8a2ab608 13 MB /bin % Note that -no-encryption is only used here to simplify the examples, dropping the flag will allow creating encrypted repositories remotely.\nWhat\u0026rsquo;s next ? # Not really decided on my next tasks as I have multiple areas that need to be improve, but a good candidate is the plakar sync command which currently only synchronizes clone repositories (same configuration, either unencrypted or both encrypted with same key). I\u0026rsquo;d like to improve it so it can synchronize snapshots with arbitrary repositories, allowing to synchronize an unencrypted snapshot with an encrypted repository and encrypting chunks on the fly.\nI\u0026rsquo;m also working on two new projects, which I won\u0026rsquo;t disclose for now, so I may be going back and forth depending on the mood.\n","date":"24 April 2022","permalink":"/posts/2022-04-24/april-2022-plakar.io-plakar-refactor-and-ssh-support/","section":"Posts","summary":"TL;DR: I refactored internal structures to split metadata from the index, implemented an stdio server and finally added SSH support. The plakar.io website # A project needs a website so\u0026hellip; I published the plakar.","title":"April 2022: plakar.io, plakar refactor and ssh support"},{"content":" TL;DR: I did key and UI stuff, mostly search related. There\u0026rsquo;s going to be plenty of images in this post. Reworked plakar keys # I won\u0026rsquo;t expand much on that as I\u0026rsquo;m not done yet, but I spent a couple days reworking the way plakar handles keys for encrypted repositories.\nWhen a user creates its first encrypted repository, plakar will complain that a key needs to be generated before encryption can be used. The key is really a bundle, identified by a UUID, and containing both a key-pair and a master key, the former being used for signing purposes and the latter to do the actual encryption within a repository. Once generated, if an encrypted repository is created, it will record the bundle UUID in its configuration so that a client can immediately know if it possesses the proper bundle to work with that repository.\nThis was fine to bootstrap the project, but there are some issues with that approach.\nFirst, because the master key is in the same bundle as the key-pair, they become too tied and a user can\u0026rsquo;t use the same key-pair to work with repositories encrypted with different master keys. This meant that either all repositories had to be encrypted using the same master key, or that a user had to generate as many key bundles as repositories encrypted with different keys.\nThen, for the reasons above, it was difficult to allow multiple users to work over the same repository despite the fact that it\u0026rsquo;s been designed for that. If I wanted to give two SREs access to the same repository, then they\u0026rsquo;d need to share the same bundle which would prevent proper auditing and making it very painful to revoke an access\u0026hellip; unless different bundles would share the same master key and the repository configuration recorded a master key UUID rather than a bundle UUID, which would still tie key pairs to a specific master key.\nThere are many ways to improve this but they all share the same initial step: breaking the relation between the key pairs and the master keys used by repositories, which is what I did. The master key for a repository is now generated when the encrypted repository is created, with no ties whatsoever to the key pair, and different users may share the same master key while having different key pairs.\nI\u0026rsquo;ll detail this in a future post as there\u0026rsquo;s still some work to do, however I have successfully created multiple repositories relying on different master keys and worked with them transparently, just as I have shared them between two different users and worked with them transparently as well despite different key pairs. The first step towards a proper solution is done.\nTONS of UI improvements # I work a lot with the CLI and I find it very easy to use for creating or restoring snapshots, as well as performing a lot of small operations when I know what I want to do, but sometimes what I really want to do is browse around a bunch of old snapshots and it\u0026rsquo;s not necessarily as user-friendly.\nA while back, I wrote the plakar ui command which launches an interface that allows browsing through snapshots, obtaining a lot of informations regarding metadata or objects structures, as well as providing a basic viewer.\nAs time passes, the UI is increasingly usable and friendly, but it still lacks some interesting features.\nSearchable repository # A feature that I believe is very important is the ability to search through a repository or snapshot.\nThere was already a bit of commented code to do that, but it was very limited as it only allowed searching for path names that contained a particular string, however snapshot indexes contain a lot of useful informations that I could make use of.\nSo the first step was to uncomment and bring back the basic search capability, which allows to lookup a directory of file by name:\nI then added the ability to search files matching a particular kind of data, like for instances searching only files that are applications, images, videos, audio, text, or other kinds:\nWhich I then extended to allowing the search of files matching an exact content type, like for instance searching specifically for PDF or JPEG files:\nAnd finally, I implemented a filter on file extensions, because quite often what I\u0026rsquo;m looking for is really a .go or .py file which I forgot the name of:\nAt this point, the search is global to a repository and looks into every snapshot, but the next step is to implement a context-aware search so that it is possible to search from within a snapshot or even within a directory within a snapshot. I\u0026rsquo;ll also be adding a few more useful search criteria, like being able to filter on users or dates.\nAt some point, I\u0026rsquo;ll be working on a far more advanced full-text search, I know how to do it so that it can work with plakar regardless of encryption and locality of the repository, but this heavy work so I won\u0026rsquo;t start before I\u0026rsquo;m already happy with the basic features.\nImproved object viewer # The UI has a page for each file that\u0026rsquo;s part of a snapshot, and that page not only lists metadata regarding the object but also provides a small viewer for text and images:\nThe viewer has a special case so that it only tries to display raw text/* and image/*, as I don\u0026rsquo;t really want the content of a binary file displayed. It worked nicely but I thought it was a bit sad that other types, like PDF for example, didn\u0026rsquo;t get a chance be properly rendered when the browser knows how to do it. I made a few changes to the code and I now have this rendering properly done for PDF files\u0026hellip;\n\u0026hellip; which is rendered just as if it had been downloaded from a website, since that\u0026rsquo;s exactly what it does :-)\nHTML files are very interesting as they resolve local links within the snapshot itself. I have pushed a copy of the OpenBSD\u0026rsquo;s website and the page below renders the index.html from the openssh/ subdirectory, which uses the CSS and image that are part of the snapshot and renders just as fine as from a regular directory.\nThis makes it very nice to browse backups of websites and see their different versions :-)\nSince I\u0026rsquo;m a developer and rely a lot on syntax highlighting, I took an extra step and added support for that too:\nIt doesn\u0026rsquo;t show above because I took these screenshots before, but I made a change to the UI so that I can decide if I want a file rendered raw or highlighted, when that makes sense.\nFor instance, the HTML example above had the page rendered raw but a button allows toggling between raw and displaying the HTML source code highlighted.\nfailed experiment with statistics\u0026hellip; # I did a failed experiment with statistics, trying to display doughnut charts about file types, repartition, duplication and other interesting informations, but it turned out that with large snapshots the amount of data makes these unreadable\u0026hellip;\n\u0026hellip; so I gave up on that idea and took a different approach.\nInstead, I added columns to tables containing categories of objects and display proportions there. It is less sexy than charts but is readable and I can think about displaying small charts that showcase a specific category vs all others to make it more visual:\n\u0026hellip; that led to a refactored landing page # The landing page used to only display a table with a row for each snapshot.\nI reworked it so that it encompasses all the changes described in this post: it contains a set of tabs that allow switching from snapshots listing to file kinds, file types or file extensions listings, providing easy access to filtered searches as well as statistics by kinds, types and extensions:\nWhat\u0026rsquo;s next ? # Working on the UI is a priority as it is the easiest way to browse through snapshots, but it is also particularly painful as I suck at making non-CLI stuff and because Go\u0026rsquo;s html/template is orders of magnitude higher in the pain-in-the-ass scale compared to Python\u0026rsquo;s jinja2 or bottlepy template engines which I\u0026rsquo;m familiar with. Stuff that could be done in a few minutes easily take me an hour to achieve when I don\u0026rsquo;t give up for a few days.\nI\u0026rsquo;ll continue working on the search capabilities, but there are two features that I really want done next month too: being able to find all duplicates of a file on all snapshots, regardless of their path names, and being able to find all files that have changed between snapshots and provide diff in the UI. Basically, I should be able to find all copies of a single file anywhere in the repository, but also be able to tell that /etc/passwd is identical in four snapshots but has changed in the fifth and display the change. This is already doable in the CLI but it needs some user-friendliness in the UI.\nVoila.\n","date":"1 April 2022","permalink":"/posts/2022-04-01/april-2022-plakar-keys-and-ui-stuff/","section":"Posts","summary":"TL;DR: I did key and UI stuff, mostly search related. There\u0026rsquo;s going to be plenty of images in this post. Reworked plakar keys # I won\u0026rsquo;t expand much on that as I\u0026rsquo;m not done yet, but I spent a couple days reworking the way plakar handles keys for encrypted repositories.","title":"April 2022: plakar keys and UI stuff"},{"content":" TL;DR: implemented cloning and synchronization between plakar repositories Slacked a bit # Shortly after I published my yearly retrospective, I was hit with two highly annoying personal issues that kept me very busy these last two months. I finally got everything back under control, my mind is mostly free again and I resumed playing with plakar.\nplakar clone # I implemented a clone operation in plakar allowing the creation of a new repository that is an exact copy of a source repository. This is a bit trickier than it reads, repositories rely on a specific structure as well as hard links magic to properly represent snapshots and reference counts on files. Performing a recursive cp of the plakar storage directory would lead to a broken repository unable to properly handle de-duplication.\nTo implement cloning, plakar creates a new repository from scratch using the exact same configuration as the source repository. It retrieves the list of chunks, objects and snapshots from the source repository, then for each chunk, object and snapshot it retrieves the data from the source repository and writes it to the destination repository. Once this is done, it loads the index file for each snapshot and performs the hard link magic to recreate the references required for snapshots to be complete. When done, both repositories are identical and the destination one can be used just as if it was the source one.\nThis required adding a few primitives to the storage layer and it was quite frustrating as mistakes didn\u0026rsquo;t show up right away. A lot of tasks can rely on the snapshot index, so when a chunk or an object are missing or incorrectly referenced for example, the cloned repository can still handle a lot of commands as if there was no issue. This is where plakar check came in handy, letting me know right away that something was off.\n$ plakar create -no-encryption $ plakar push /bin $ plakar ls 2022-03-05T22:30:58Z 3a5769ff-3353-4ae9-ae64-13fc0fc0b6ac 13 MB /bin $ plakar clone /tmp/plakar-copy $ plakar on /tmp/plakar-copy ls 2022-03-05T22:30:58Z 3a5769ff-3353-4ae9-ae64-13fc0fc0b6ac 13 MB /bin $ As you can see in the example above, the clone retains snapshots UUID as this is not a new snapshot being copied from another one in a new transaction, but really the exact same snapshot copied using lower level primitives.\nAs of today, plakar clone is not fully finished as it only supports filesystem-based repository. I need to implement the new storage primitives in the SQLite and network backends, so that cloning can happen seamlessly over the network or between different backends. When this is done, it\u0026rsquo;ll be possible to convert a filesystem plakar to an SQLite plakar by cloning it (right now, it\u0026rsquo;s only doable the other way), or to clone a plakar repository on a different machine.\nplakar sync # With plakar clone, all the primitives required to synchronize two repositories became available.\nSimilarly to cloning, synchronizing allows performing a low level copy of snapshots without initiating new transactions, thus retaining the same snapshots UUID. To achieve synchronization, plakar opens a source repository and a destination repository, then compares which chunks, objects and indexes are available in the source one but missing from the destination one to compute a delta. It then performs a cloning of that delta, only exchanging the missing bits and doing the hard links magic.\n$ plakar create -no-encryption $ plakar clone /tmp/bleh.t1 $ plakar ls $ plakar on /tmp/bleh.t1 ls $ plakar push /bin $ plakar ls 2022-03-05T23:24:08Z 480e6781-4f51-4289-9983-17f0e5962cc9 13 MB /bin $ plakar sync /tmp/bleh.t1 $ plakar on /tmp/bleh.t1 ls 2022-03-05T23:24:08Z 480e6781-4f51-4289-9983-17f0e5962cc9 13 MB /bin $ In the example above, I create a new empty repository and clone it. Then I push /bin to the source plakar, perform a plakar sync to the destination plakar, which ends up having the same snapshot.\nLike for cloning, this is a work in progress and not finished.\nThe idea behind sync is that when the primitives are ported to network plakar, it becomes possible to have a remote plakar centralize the backups of a local plakar, but it also becomes possible for multiple remote plakars to synchronize themselves one with another and provide multiple copies of the same backups very easily.\nWhat\u0026rsquo;s next ? # I\u0026rsquo;ll be working on and off as I have a wedding that\u0026rsquo;s coming soon and a lot of things to prepare, but this should not prevent me from finding time to relax with some code.\nI\u0026rsquo;ll rework clone and sync because functionality set aside I don\u0026rsquo;t like the way it\u0026rsquo;s implemented in the CLI.\nI\u0026rsquo;m currently reading the QuickCDC paper and wondering if it\u0026rsquo;s worth implementing on top of my go-fastcdc implementation, it is unclear if there will be any gain at this point.\nI\u0026rsquo;m considering a few more repository-level operations, similar to clone/sync, to help make plakar as friendly as possible to operators.\nIt will be time to start working on a website too :-)\n","date":"6 March 2022","permalink":"/posts/2022-03-06/march-2022-plakar-clone-and-plakar-sync/","section":"Posts","summary":"TL;DR: implemented cloning and synchronization between plakar repositories Slacked a bit # Shortly after I published my yearly retrospective, I was hit with two highly annoying personal issues that kept me very busy these last two months.","title":"March 2022: plakar clone and plakar sync"},{"content":" TL;DR: A retrospective of 2021 2021 was a good year overall # Pandemic and lock-downs set aside, 2021 was a good year overall.\nI\u0026rsquo;ve really enjoyed my daytime job, my physical health was fine, my mental health was\u0026hellip; meh\u0026hellip; but ok, I guess.\nLearnt about my neuro-divergences # I took a psychological assessment which pinpointed what\u0026rsquo;s \u0026ldquo;wrong\u0026rdquo; with me, the source of my alexithymia, generalized anxiety disorder and life-long uneasy feeling.\nIt was painful (a.f.) to cope with the result, and it\u0026rsquo;ll still take some time for me to get over it, but at least I now have a much better understanding of how I\u0026rsquo;m wired, what works and doesn\u0026rsquo;t work for me, and some actions I need to take to effectively improve my life.\nIt sucks to discover neuro-divergences past 40 years old, with ~half of my life already passed, but also a career and life decisions taken on partial informations. There\u0026rsquo;s a bit of grieving required, but it\u0026rsquo;s all for the best: I can acknowledge and move forward.\nI won\u0026rsquo;t disclose the details as I\u0026rsquo;m not comfortable sharing them publicly, but if I decided to write this, it\u0026rsquo;s because I hope it can be helpful to others in similar situations: taking a psychological assessment is nothing to be ashamed of, it doesn\u0026rsquo;t make you weak, and if you ever feel psychological suffering that you fail to understand, it can help put some perspective where it\u0026rsquo;s lacking. I wish I had someone tell me this twenty years ago, so I\u0026rsquo;m doing my part in case someone needs to read this now.\nWork psychology studies # Because of the pandemic I had to put my studies of work psychology on hold, which is a shame considering that I\u0026rsquo;m only a year away from starting my master thesis, but finding a company that would welcome me for a field intervention has proven impossible\u0026hellip; two years in a row\u0026hellip;\nI\u0026rsquo;ll take a break until the pandemic is behind and life is back to normal rather than spend a third year going to evening classes for nothing, I\u0026rsquo;d rather spend these hours playing with my son and learning new stuff. It\u0026rsquo;ll take considerably longer than expected to obtain the diploma but it\u0026rsquo;s ok, I\u0026rsquo;m not in a rush and there are other fields I want to study in 2022.\nHypnosis office # Despite suffering from the consecutive lock-downs and curfews, work at the hypnosis office remained active throughout the year, which allowed me to reach break-even. Considering that hypnosis is not my main activity, that I do not depend on it for a living, and that I\u0026rsquo;m only doing it because I enjoy doing it and need it in my life\u0026hellip; I\u0026rsquo;m happy that it breaks even and that I don\u0026rsquo;t have to stop that activity like so many people around me.\nHere\u0026rsquo;s a picture of my new office which looks lovely:\nStarting January 2022, I\u0026rsquo;ll be doing self-hypnosis workshops and I\u0026rsquo;m considering doing a few hypnotherapy workshops too, though I need to evaluate how much effort I can pour into this first.\nResumed doing some open-source work # For the last two years, I had lost motivation to write code and work on opensource projects: I\u0026rsquo;d sit in front of my computer and find a thousand creative ways to procrastinate. Then it struck me: I lost touch with why I enjoyed writing code in my spare time and this is why I no longer could.\nI loved writing code because I loved experimenting, learning new tricks, writing broken throw-away experiments, testing stuff and seeing what comes out of it. I loved writing code because I primarily wrote it for myself, not for others, and because I had no other goal but to enjoy the puzzle solving\u0026hellip; or giving up when interest decreased. I loved it because it was a hobby and I could do whatever I want.\nThis last decade, my spare time coding was spent for others, not myself. I wrote tons of code that I had no interest in whatsoever, just because it was what others expected to see happening, and everything I did was done thinking about how useful it would be to others and how they\u0026rsquo;d view my work. No one forced me, it just happened and I did not see how this affected me.\nIt\u0026rsquo;s hard to switch that mindset but I decided to try no longer giving a damn.\nI\u0026rsquo;ll work on projects I want, regardless of how useful they are, and I will try not to care how people feel about these projects. I\u0026rsquo;ll keep sharing them because that\u0026rsquo;s how I learnt and I want to give back, but also because that\u0026rsquo;s how people teach me new tricks by pointing at things I\u0026rsquo;ve done wrong or inefficiently.\nIf people find these projects useful, I\u0026rsquo;ll be happy, but if a project starts putting pressure on me I\u0026rsquo;ll step back, let people who care fork and maintain it, then move on to work on something else.\nThese realization and decision unlocked me and, as a result, I worked on many projects as can be seen from my last few monthly reports. Some of them, like plakar, are useful and entertaining to me while others served no other purpose but entertainment. This is how it\u0026rsquo;s going to work for me from now on ;-)\nLearnt myself some arabic speaking # I spent 2021 learning how to speak Arabic with weekly classes and I could not be happier as I can now hold basic kid-level discussions but still feel like I could be dropped in the Middle-East and manage to find my way and not starve. When I go back to Lebanon, I\u0026rsquo;ll be able to order my shawarma, manaïche and baklawa without relying on French, English or someone to translate for me. I won\u0026rsquo;t be fluent before a while but I only care about being understood, so that\u0026rsquo;s ok.\nI just began learning how to read, hopefully by the end of 2022 I can read a simple book, or better, one of my aunt\u0026rsquo;s novels \u0026lt;3\nHopefully I\u0026rsquo;ll also be making better use of my US-arabic keyboard which currently only proves useful at extending the character set of my passwords ;-)\nBegan learning how to draw # In September, I began learning how to draw from scratch (because I suck) and I actually enjoyed it.\nI wasn\u0026rsquo;t very constant as I got context switched a lot and didn\u0026rsquo;t draw every day, but I have prepared myself a schedule so that I have dedicated time starting in 2022.\nAt this point, I can\u0026rsquo;t really say I learnt much but I got a grasp at some key concepts:\nI\u0026rsquo;ll keep this blog updated with my progress as a mean to keep motivated ;-)\nStarted learning the piano # Late 2020 I took a try at creating LoFi tracks because I wanted to know how to \u0026hellip;\n\u0026hellip; and this year as I was having fun with a piano VST and my MIDI controller keyboard plugged, I got hooked and spent several evenings improvising single-hand melodies on top of songs I liked. It was fun and all but I realized that I really needed a piano in my life (and also to learn how to play with both hands).\nThat\u0026rsquo;s how I ended getting myself a Yamaha YDP S54 piano this summer, after successfully convincing my second-half that it would benefit the kid and be plenty nice in the appartment (don\u0026rsquo;t pull this at home kids, it was a professional stunt)\u0026hellip;\n\u0026hellip; but I couldn\u0026rsquo;t play it right away as we left for holidays right after I brought it home, then I got carried away in September by all the kid / work things you have to deal with when coming back from summer vacations.\nI started taking courses in November and I absolutely loved it from the first session, so much in fact that I have not touched any other instrument since then (yes, that\u0026rsquo;s bad\u0026hellip;), and that I actually play with it every single day.\nI took five courses so far so there\u0026rsquo;s a looooooong road ahead, but I can definitely picture myself sticking with that instrument in the long run.\nSatie was the main reason I wanted to learn the piano so\u0026hellip; yay, I can play a few measures :-)\nProbably forgot a few things # I probably forgot a few things, I kept myself busy all year long and didn\u0026rsquo;t keep a log of every little thing I did.\nThese were my most important achievements this year and I\u0026rsquo;m happy about all of them. I wish I could say that I\u0026rsquo;d take some rest in 2022, but\u0026hellip; I have a stack of books sitting next to me (literally on a chair) and the one on top is already keeping me busy.\nMy biggest achievement in 2021 was to realize that I STILL can do anything I want, nothing is out of reach and if something actually is, then\u0026hellip; heck, I tried and I\u0026rsquo;m free to give up any time, I\u0026rsquo;ll still have made progress on the way.\nHAPPY NEW YEAR # I\u0026rsquo;ll close this retrospective with a HAPPY NEW YEAR EVERYONE !\nI wish you success at everything you attempt and may a ton of good things happen to you in 2022.\nPeace.\n","date":"30 December 2021","permalink":"/posts/2021-12-30/farewell-2021-welcome-2022-a-personal-post/","section":"Posts","summary":"TL;DR: A retrospective of 2021 2021 was a good year overall # Pandemic and lock-downs set aside, 2021 was a good year overall.\nI\u0026rsquo;ve really enjoyed my daytime job, my physical health was fine, my mental health was\u0026hellip; meh\u0026hellip; but ok, I guess.","title":"Farewell 2021, Welcome 2022: a personal post"},{"content":" TL;DR: I worked on plakar, go-fastcdc a FastCDC implementation for plakar, and some useless stuff. This activity report will be short # This activity report will be short because I spent the last ten days caring for my kid who got sick\u0026hellip; before catching his stomach bug myself and spending the last few days in a somewhat familiar and unpleasant hell, unable to do everything I intended for this month.\nI didn\u0026rsquo;t want to skip this report as it\u0026rsquo;s the last of the year, but I\u0026rsquo;m exhausted and don\u0026rsquo;t have the energy to spend hours writing and rephrasing, I\u0026rsquo;ll make it up to you next year.\nFirst, plakar # The last few months, I worked on and off on the plakar project and while it\u0026rsquo;s not at a stage where I want to release it, it\u0026rsquo;s most certainly at a stage where feedback from early testers would be useful.\nIf you have followed it through the last two or three reports and are interested in where this is going, feel free to join my discord where a room is dedicated to this project, where you can provide feedback as you test, and where I can fix things live as you break:\n- DO NOT USE PLAKAR + DO NOT USE PLAKAR (UNLESS YOU\u0026#39;RE HELPING ME MAKING IT USABLE) If I get enough feedback and assurance that it\u0026rsquo;s not too buggy and can\u0026rsquo;t collapse into a singularity, this might even get a first release sometime early 2022 if it keeps on at the pace it was going this last quarter.\nSince we\u0026rsquo;re on the subject of plakar\u0026hellip; # I did a ton of optimizations, not just code-wise but in terms of index format and such, storage representation is not fully stable yet but barely changes anymore and plakar snapshots made this week are likely to be restorable with a build from next week. Work these days is mainly focused on improving the command line interface and features built on top of snapshots themselves, not on how data is dealt with.\nOne of the things I really wanted to improve, and which is the main thing I did this month, was to change the chunker\u0026hellip; so let\u0026rsquo;s talk about that.\nWhat\u0026rsquo;s a chunker ? # To save space and avoid storing multiple copies of a same file content, plakar checks if that content is already in the store and only records a reference if it is.\nInstead of checking for an exact copy of the file, which would no longer match if a single byte was appended, it slices the content into chunks and checks for them individually so as to only write chunks that are not already present.\nA very naive approach to this slicing is that of fixed chunking where the file is sliced into chunks of fixed size, say 16KB, which works nicely for any file that\u0026rsquo;s not supposed to change (hello /bin, you\u0026rsquo;re awesome) or that only gets appends (I see you /var/log/maillog). This approach is naive because it works perfectly\u0026hellip; as long as nothing causes a shift in chunk alignment: prepend a single byte in front of a 1GB log file that\u0026rsquo;s already in the store, and all chunks are now shifted, none can be found in store and the whole file needs to be stored again.\nA solution to this problem is the use of content-defined chunking, an approach that uses the content itself to dynamically determine the location of chunk boundaries, rather than using fixed boundaries.\nThe basic idea is that if you use a rolling buffer of data to somehow determine the chunk boundaries, then even if you insert a byte at the beginning or the middle of the stream and process the stream again again, after a while the rolling buffer will contain the same content as before the byte was inserted and therefore end up in the same chunking cycle\u0026hellip; so it should be possible to compute a rolling hash of some kind and use specific outputs of that rolling hash as a condition to delimit chunks.\nThis is nice and all but it doesn\u0026rsquo;t come for free. If chunking is not of a fixed size, then it\u0026rsquo;s no longer a simple fixed cursor increment but the result of computation done on content\u0026hellip; which means that the more content needs to be seen to determine chunks, the slower chunking becomes. ORDERS OF MAGNITUDE slower.\nLuckily for me, there\u0026rsquo;s a research community focused on content-defined chunking which has smarter people (ORDERS OF MAGNITUDE smarter) coming up with ways to speed things up, and while there\u0026rsquo;s no way it\u0026rsquo;ll ever reach the speed of fixed-length chunking, they made so much progress these last decades that it\u0026rsquo;s actually fast enough to be usable. Not only that, but through improvements on resources consumptions like for instance not pegging a CPU during the whole process of chunking, some of the time lost on chunking can be regained on parallelization of tasks for example.\nAnyways, I digressed\u0026hellip;\nIn my very early Python PoC of plakar years ago, I started with fixed-length chunking to bootstrap the code then implemented a Rabin-Karp rolling hash to provide contend-defined chunking. It worked and it was slow (a.f.), probably because it was Python, probably because it was my code, probably because Rabin-Karp isn\u0026rsquo;t\u0026hellip; too fast.\nWhen I revisited this project from scratch in Golang a few months ago, I thought I\u0026rsquo;d just use any existing chunker available to me while bootstrapping the project, then take serious time investigating what\u0026rsquo;s the chunker algorithm I really want to use and if I really need to implement it myself.\nFor a few months, I relied on restic\u0026rsquo;s chunker which\u0026hellip; is a rolling Rabin hash (hooray, no surprises), but this month I decided to reconsider as it doesn\u0026rsquo;t perform as fast as I\u0026rsquo;d like and having plakar depend on another backup utility was a bit weird and came with its own issues.\nI did some quick search and found a paper titled FastCDC: A Fast and Efficient Content-Defined Chunking Approach for Data Deduplication, claiming:\nwe propose FastCDC, a much faster CDC approach for data deduplication than the state-of-the-art CDC approaches while achieving a comparable deduplication ratio. [\u0026hellip;] Our experimental evaluation demonstrates that FastCDC obtains a chunking speed that is about 10× higher than that of the Rabin-based CDC and about 3× that of the Gear- and AE-based CDC while achieving nearly the same deduplication ratio as the Rabin-based CDC.\nSAY NO MORE, SCIENTIST, YOU CONVINCED ME WITH YOUR WISE WORDS.\nI did a bit of github search, found implementations, tested them but they didn\u0026rsquo;t work too well for me as one crashed, the other had an interface optimized for streaming, and both copied optimizations that I didn\u0026rsquo;t understand from a different implementation in a different language.\nThe paper was nicely written, came with pseudo-code displaying a very simple algorithm, so\u0026hellip;\ngo-fastcdc # The go-fastcdc package hosts my implementation of the FastCDC paper with no bells and whistles\u0026hellip; except that it takes special care to avoid unnecessary buffer allocations and has an implementation-specific optimization of my own, unrelated to how FastCDC itself works.\nBasically, a chunking loop looks like this:\nfor { chunk, err := chunker.Next() if err != nil { if err == io.EOF { // no more chunks to read break } log.Fatal(err) } // do something with the chunk } In other implementations that I read, the call to Next() is what computes the chunk boundaries so that in this loop the chunker only works during Next(), then remains idle while the caller does something with the chunk.\nMy optimization was to have a goroutine part of the chunker to keep ensuring that the next chunk is available as soon as possible, and make Next() return the next chunk as soon as it is made available by the goroutine. This doesn\u0026rsquo;t seem like much but it actually ensures that the chunker doesn\u0026rsquo;t remain idle while the caller works with a chunk, making the calls to Next() almost \u0026ldquo;free\u0026rdquo; as the cost of chunking is paid concurrently to whatever work the caller is doing (if caller isn\u0026rsquo;t pegging the CPU that is) The chunker goroutine only ensures ONE chunk ahead and will sleep as long as that chunk isn\u0026rsquo;t consumed.\nOn a loop similar to the one above with no work done on the chunk, this will bring absolutely no benefit, but for plakar which does a lot of chunking but also does a lot of stuff with chunks before requesting the next one, this saves several seconds on the snapshot of a 1GB directory.\nThere\u0026rsquo;s not much more I can say about it, it takes an input reader, produces content-defined chunks out of it following the FastCDC algorithm, not much to see, chop chop.\nIt is ISC-licensed, use it for whatever you see fit, or don\u0026rsquo;t use it and read what follows.\nplakar uses go-fastcdc # With go-fastcdc available to use, I broke ties with the restic chunker and switched plakar to go-fastcdc as its chunker, it was a very tricky change that involved a ton of I wish I could write about the challenges I faced but it was a trivial change that took two minutes as I used the same interface.\nThe performance improvement was not as great as advertised by other fastcdc implementation but it is slighly faster for my use-case, I fully know the chunker code and am the maintainer of it so if anything needs tweaking, it\u0026rsquo;s a big win.\nDoes that mean that plakar will rely on FastCDC ? I dunno.\nSee, it took me a day to switch from the previous chunker to my own FastCDC implementation, removing the downsides I had, so it was worth doing that work but this doesn\u0026rsquo;t mean that FastCDC is the one I want to go with in the end.\nThe FastCDC paper was published in 2016 claiming to be up to 10x faster than CDC, but then RapidCDC was published in 2019 claiming to be up to 33x faster than CDC, and now QuickCDC was published this year claiming to be even faster. Speed isn\u0026rsquo;t everything but\u0026hellip; in my case it kinda is.\nUnless the trade-offs (cpu cost / ram cost) are unbearable, there\u0026rsquo;s no reason to stick with FastCDC if anything else is faster. I only had the time to read and implement FastCDC before my son tripped me into a hell of body fluids, but I\u0026rsquo;ll be carefully reading (and probably implementing) RapidCDC and QuickCDC so I can do tests and pick what\u0026rsquo;s the best for plakar.\nTHAT\u0026rsquo;S ALL FOR THE USEFUL STUFF, BELOW IS JUNK # Assorted experiments to reduce my mental backlog # Implemented my first genetic algorithm # I learnt the theory behind (some) genetic algorithms reading a book when I was a student approximately a century ago, but there was no school project involving one and since I was already drowning in work (and really wanted to graduate), I decided not to disperse myself and plain forgot about the topic.\nSomehow, something reminded me of genetic algorithms this month and since I never wrote one and didn\u0026rsquo;t want to remain ignorant, I searched for a tutorial and ran into this presentation:\nI spent an hour implementing both examples, another hour trying to come up with smart optimizations to speed things up as it was painfully slow, then yet another hour to experiment with them further until I understood what they were useful for and what they were unusable for.\nIt was lovely, I\u0026rsquo;m not sure I\u0026rsquo;ll be using one again before the next decade but who knows.\nImplemented a custom DHT # Many years ago, I worked at a company that sells services built on top of a distributed storage powered by a chord-inspired DHT. I had never heard about DHT before so I studied the paper in anticipation of the interview, wrote a very basic network-less implementation, fell in love with the idea, and enjoyed my time working there with this technology even though I didn\u0026rsquo;t work on the DHT itself but on a project built on top of it.\nLife went on, I switched job and city, but I kind of kept a regret that I never worked on the DHT layer myself \u0026hellip;\n\u0026hellip; and since this month was already screwed in terms of usefulness, I spent a couple evenings experimenting with a DHT implementation of my own that has some properties I\u0026rsquo;m interested in. Some ideas were nice, others sucked or didn\u0026rsquo;t produce the result I expected. It\u0026rsquo;s an ongoing work to understand what I want to retain and what should be thrown away.\nI have a working implementation, with a memory-based key/value store built on top of it, networking and all, but there\u0026rsquo;s nothing really interesting to show so I\u0026rsquo;ll continue experimenting as a low-priority exercise and I\u0026rsquo;ll write about it here if anything nice ever comes out of it.\nHello, VR ! # If 2020 and 2021 have taught me anything, it\u0026rsquo;s that there\u0026rsquo;s an overwhelming amount of people that I wish would live on another planet. A number so high, in fact, that even if they all agreed to board a shuttle right away and head to the sun, I\u0026rsquo;d still consider moving away from this planet while they board because oh my I wish they lived on another planet.\nI had huge plans for relocating on Mars ever since I read Ray Bradbury\u0026rsquo;s Martian Chronicles as a teenager, but now there\u0026rsquo;s this electric car guy who wants to ruin there too so\u0026hellip; nu-uh, I need another place, preferably the opposite direction.\nVenus seemed very promising but the NASA won\u0026rsquo;t be floating outposts there before my time is over, I\u0026rsquo;m screwed.\nThis only leaves me with VR.\nI picked up a book and started learning about VR development. This way, when I have a rough day and I just can\u0026rsquo;t anymore with the human race, I can wear a mask and instantly put a few million kilometers between others and myself, even if just for a few deep breathes and the sight of an asteroid putting this place out of its misery.\nI\u0026rsquo;m not quite there yet but it seems achievable in a more realistic timeframe than my travel to Venus.\nThat\u0026rsquo;s all folks! # As I do every year, I\u0026rsquo;ll write a more personal retrospective of this year in a few days, maybe late 2021 or early 2022, it\u0026rsquo;s been an intense year and I need to pour it out so my head is empty again.\nHope you enjoyed reading this post, happy Xmas, happy new year and may life bring you joy and happiness.\n","date":"13 December 2021","permalink":"/posts/2021-12-13/december-2021-a-bit-of-plakar-a-bit-of-go-fastcdc-and-some-useless-stuff/","section":"Posts","summary":"TL;DR: I worked on plakar, go-fastcdc a FastCDC implementation for plakar, and some useless stuff. This activity report will be short # This activity report will be short because I spent the last ten days caring for my kid who got sick\u0026hellip; before catching his stomach bug myself and spending the last few days in a somewhat familiar and unpleasant hell, unable to do everything I intended for this month.","title":"December 2021: a bit of plakar, a bit of go-fastcdc and some useless stuff"},{"content":" TL;DR: I still have a discord, feel free to join. I worked on go-ipcmsg to make it nicer, go-privsep to make it more useable, and A LOT on plakar to make it plakar. Go-ipcmsg # In April, I wrote about go-ipcmsg, a package to bring an imsg(3)-like API to Golang and help me write code involving message and fd-passing between processes.\nIt took me a couple days of work and I was happy with the result so I left it there, letting it rest in its Github repository, until earlier this month when I figured the API could be simplified by a great deal\u0026hellip; so I decided to revisit and make some changes..\nBack then, each peer of an IPCMSG channel received two Golang channels, and could simply read or write to the other process through them:\nfunc parent() { // just a basic function that sets up a socketpair, forks a child, // and returns the child\u0026#39;s pid and the parent\u0026#39;s side of the socketpair. pid, fd := fork_child() // read \u0026amp; write channels obtained here child_r, child_w := ipcmsg.Channel(pid, fd) // write to child, last parameter == fd to pass to the other process child_w \u0026lt;- ipcmsg.MessageWithFD(IPCMSG_PING, []byte(\u0026#34;foobar\u0026#34;), -1) // read from child msg := \u0026lt;- child_r } func child() { // read \u0026amp; write channels obtained here, // fd=3 is the fork-inherited side of the socketpair. parent_r, parent_w := ipcmsg.Channel(os.Getppid(), 3) // read from parent msg := \u0026lt;- parent_r // write to parent parent_w \u0026lt;- ipcmsg.Message(IPCMSG_PONG, []byte(\u0026#34;barbaz\u0026#34;)) } In practice, instead of inlining calls like this, you\u0026rsquo;d declare a handler function which would process all messages on the read channel and reply through the write channel as such:\nfunc dispatcher(r chan ipcmsg.IPCMessage, w chan ipcmsg.IPCMessage) { for msg := range r { switch msg.Hdr.Type { case IPCMSG_PING: w \u0026lt;- ipcmsg.MessageWithFD(IPCMSG_PONG, []byte(\u0026#34;barbaz\u0026#34;), -1) } } } func child() { parent_r, parent_w := ipcmsg.Channel(os.Getppid(), 3) dispatcher(parent_r, parent_w) } However, I realized that pretty much every single use of the package would require writing that very similar dispatcher function looping over the read channel, and I also disliked the pattern of letting the user access the Golang channels directly as it made it easier to introduce mistakes in code for no benefit whatsoever.\nI decided to rewrite the ipcmsg.Channel() function to return an opaque structure hiding the underlying channels, and to provide a channel.Dispatch() function to start reading from the channel. With this change, the sample of code above now looks like:\nfunc handlePING(channel *ipcmsg.Channel, msg ipcmsg.IPCMessage) { channel.Message(IPCMSG_PONG, []byte(\u0026#34;barbaz\u0026#34;), -1) } func child() { channel := ipcmsg.NewChannel(\u0026#34;child\u0026lt;-\u0026gt;parent\u0026#34;, os.Getppid(), 3) channel.Handler(IPCMSG_PING, handlePING) \u0026lt;-channel.Dispatch() // Run() and wait until it returns due to peer close } As you can see, the message handling function receives an IPCMessage when one is available, without having to loop on the read channel, and the function setting up the channel can declare handlers for specific message types before calling the dispatcher.\nIn addition to this simplification, I introduced a new mechanism: synchronized messages.\nIn the imsg(3) API, because it is written in C which doesn\u0026rsquo;t provide coroutines, messages are sent and code can\u0026rsquo;t wait for an answer as it would block the entire process meanwhile. To avoid this, code to emit a message and code to handle answers are decoupled: a function sends and stops caring about the message\u0026hellip; until another function gets an answer and resumes what the first intended to do if it could have waited for an answer. Splitting code like this is always doable, but it can be more or less tricky with some patterns where you\u0026rsquo;d benefit from having a very sequential read.\nBecause I can use Goroutines here and have them interrupt their execution in place without blocking the entire process, I introduced two additional calls:\n// channel.Query() blocks the goroutine and resumes execution // only when the other end has used channel.Reply() // msgtype, data, fd := channel.Query(IPCMSG_PING, []byte(\u0026#34;data\u0026#34;), -1) if msgtype != IPCMSG_PONG { panic(\u0026#34;THE WORLD IS COMING TO AN END.\u0026#34;) } fmt.Println(\u0026#34;got appropriate response to my query\u0026#34;) This pattern is not necessarily better than the split pattern, they both have their use-cases which is why go-ipcmsg provides both, but in some situations it really makes code much less complex than interrupting execution to resume in another function.\nThe code is already available in a Github repository, and it works enough that I use it in various proof of concepts I write, but it still isn\u0026rsquo;t ready to use in production and was only tested by me.\nFeel free to test and report issues ;-)\nGo-privsep # In April, I also wrote about go-privsep, a package to ease the writing of OpenBSD-style daemons in Golang.\nThe idea behind it is that most OpenBSD-style daemons use the same multi-process design, yet every project has to have a different setup logic because they differ in the number of processes, which processes communicate with each other, and which messages they sent one to another.\nI disliked having to bootstrap a new daemon but at least I could copy chunks from a previous one and adapt, but this doesn\u0026rsquo;t work with Golang as I have not found any OpenBSD-like daemon to copy from. So I thought maybe I should solve the main issue which is making the part that annoys me\u0026hellip; less annoying, this way I don\u0026rsquo;t have to bother much setting up a new daemon architecture and I don\u0026rsquo;t have to copy code from another daemon either.\nThe result is a package that allows to declare how the daemon is supposed to look like when it enters its event loop, and the package takes care of creating the processes, the communication channels between them, dropping privileges and chrooting where appropriate, before having each process enter its own entry point.\nIn the following example, I have created a daemon with a design similar to OpenSMTPD\u0026hellip; except that all entry points are idle because it\u0026rsquo;s not a real daemon 😁\nfunc main() { privsep.Init() privsep.Parent(\u0026#34;parent\u0026#34;, parent.Run) privsep.Child(\u0026#34;crypto\u0026#34;, crypto.Run) privsep.Child(\u0026#34;control\u0026#34;, control.Run) privsep.Child(\u0026#34;lookup\u0026#34;, lookup.Run) privsep.Child(\u0026#34;dispatcher\u0026#34;, dispatcher.Run) privsep.Child(\u0026#34;queue\u0026#34;, queue.Run) privsep.Child(\u0026#34;scheduler\u0026#34;, scheduler.Run) privsep.GetParent().Username = \u0026#34;root\u0026#34; privsep.GetParent().TalksTo(\u0026#34;dispatcher\u0026#34;) privsep.GetProcess(\u0026#34;crypto\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;crypto\u0026#34;).Chrootpath = \u0026#34;/var/empty\u0026#34; privsep.GetProcess(\u0026#34;control\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;control\u0026#34;).Chrootpath = \u0026#34;/var/empty\u0026#34; privsep.GetProcess(\u0026#34;lookup\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;lookup\u0026#34;).TalksTo(\u0026#34;dispatcher\u0026#34;, \u0026#34;queue\u0026#34;) privsep.GetProcess(\u0026#34;dispatcher\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;dispatcher\u0026#34;).Chrootpath = \u0026#34;/var/empty\u0026#34; privsep.GetProcess(\u0026#34;dispatcher\u0026#34;).TalksTo(\u0026#34;parent\u0026#34;, \u0026#34;lookup\u0026#34;, \u0026#34;queue\u0026#34;, \u0026#34;scheduler\u0026#34;) privsep.GetProcess(\u0026#34;scheduler\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;scheduler\u0026#34;).Chrootpath = \u0026#34;/var/empty\u0026#34; privsep.GetProcess(\u0026#34;scheduler\u0026#34;).TalksTo(\u0026#34;dispatcher\u0026#34;, \u0026#34;queue\u0026#34;) privsep.GetProcess(\u0026#34;queue\u0026#34;).Username = \u0026#34;_smtpq\u0026#34; privsep.GetProcess(\u0026#34;queue\u0026#34;).Chrootpath = \u0026#34;/var/spool/smtpd\u0026#34; privsep.GetProcess(\u0026#34;queue\u0026#34;).PreChrootHandler(queue.PreChrootHandler) privsep.GetProcess(\u0026#34;queue\u0026#34;).TalksTo(\u0026#34;dispatcher\u0026#34;, \u0026#34;lookup\u0026#34;, \u0026#34;scheduler\u0026#34;) if os.Geteuid() != 0 { log.Fatal(\u0026#34;privileges separation requires root privileges\u0026#34;) } privsep.Start() } This should be self-explanatory but to summarize: you declare a list of processes with their names and entry points, then for each process you declare some properties such as the username it should run as or a chroot directory, and the list of processes it is allowed to talk to.\nWhen privsep.Start() is reached, the daemon bootstraps itself from the declarations and you end up with all processes setup:\n$ sudo ./smtpd $ ps au|grep smtpd _smtpd 70584 0.0 0.0 409218992 8112 s003 SN 12:17AM 0:00.01 ./smtpd: scheduler _smtpq 70583 0.0 0.1 409238336 9072 s003 SN 12:17AM 0:00.01 ./smtpd: queue _smtpd 70582 0.0 0.1 409233888 11120 s003 SN 12:17AM 0:00.02 ./smtpd: dispatcher _smtpd 70581 0.0 0.0 409218608 7264 s003 SN 12:17AM 0:00.01 ./smtpd: lookup _smtpd 70580 0.0 0.0 409217440 6304 s003 SN 12:17AM 0:00.00 ./smtpd: control _smtpd 70579 0.0 0.0 409225328 6320 s003 SN 12:17AM 0:00.01 ./smtpd: crypto root 70578 0.0 0.0 409219376 7936 s003 SN 12:17AM 0:00.03 ./smtpd root 70577 0.0 0.0 408638640 7296 s003 SN 12:17AM 0:00.03 sudo ./smtpd $ The go-privsep package relies on go-ipcmsg so that each processes that have called TalksTo() in the declaration have IPCMSG channels setup between them, and can declare message handlers or rely on Message() and Query() to communicate one with another.\nHere\u0026rsquo;s an example of a small daemon forking two children exchanging PING/PONG messages over and over:\npackage main import ( \u0026#34;log\u0026#34; \u0026#34;time\u0026#34; \u0026#34;github.com/poolpOrg/go-ipcmsg\u0026#34; \u0026#34;github.com/poolpOrg/go-privsep\u0026#34; ) const ( IPCMSG_PING ipcmsg.IPCMsgType = iota IPCMSG_PONG ipcmsg.IPCMsgType = iota ) func parent_main() { \u0026lt;-make(chan bool) // sleep forever } func main_foobar() { \u0026lt;-make(chan bool) } func main_barbaz() { foobar := privsep.GetProcess(\u0026#34;foobar\u0026#34;) foobar.Message(IPCMSG_PING, []byte(\u0026#34;test\u0026#34;), -1) \u0026lt;-make(chan bool) } func ping_handler(channel *ipcmsg.Channel, msg ipcmsg.IPCMessage) { log.Printf(\u0026#34;[%s] received PING\\n\u0026#34;, privsep.GetCurrentProcess().Name()) time.Sleep(1 * time.Second) channel.Reply(msg, IPCMSG_PONG, []byte(\u0026#34;test\u0026#34;), -1) } func pong_handler(channel *ipcmsg.Channel, msg ipcmsg.IPCMessage) { log.Printf(\u0026#34;[%s] received PONG\\n\u0026#34;, privsep.GetCurrentProcess().Name()) time.Sleep(1 * time.Second) channel.Reply(msg, IPCMSG_PING, []byte(\u0026#34;test\u0026#34;), -1) } func main() { privsep.Init() privsep.Parent(\u0026#34;parent\u0026#34;, parent_main) privsep.Child(\u0026#34;foobar\u0026#34;, main_foobar).TalksTo(\u0026#34;barbaz\u0026#34;) privsep.Child(\u0026#34;barbaz\u0026#34;, main_barbaz).TalksTo(\u0026#34;foobar\u0026#34;) privsep.GetProcess(\u0026#34;foobar\u0026#34;).PreStartHandler(func() error { barbaz := privsep.GetProcess(\u0026#34;barbaz\u0026#34;) barbaz.SetHandler(IPCMSG_PING, ping_handler) return nil }) privsep.GetProcess(\u0026#34;barbaz\u0026#34;).PreStartHandler(func() error { foobar := privsep.GetProcess(\u0026#34;foobar\u0026#34;) foobar.SetHandler(IPCMSG_PONG, pong_handler) return nil }) privsep.Start() } The code is also already available in a Github repository, and it works enough that I use it in various proof of concepts, though the API still evolves and I would discourage you from using it for anything but experimenting at this point.\nFeel free to test and report issues ;-)\nGo-parsey # The last bit that I really miss in Golang when writing stuff is the OpenBSD-style configuration handling.\nIn OpenBSD, all daemons share a parse.y file containing their specific (yet very similar) grammar built on top of the same lexer. After a few years of making changes in it, I grew to enjoy parse.y and the simplicity of configuration files produced by that common piece of code.\nIt already took me a while to get used to ini files in Python but when I realized that Golang developers tend to use json, toml or yaml, I just couldn\u0026rsquo;t.\nI have started working on go-parsey, a small package that allows declaring a configuration grammar that more resembles what I like. It\u0026rsquo;s nowhere near done, at this point I wrote the lexer and am playing with the API to find what is the most usable way to obtain what I want.\nIt\u0026rsquo;s at a so early stage that I can\u0026rsquo;t even show something, except maybe the following example:\n// instantiate a lexer and teach it how to recognize tokens lexer := parsey.NewLexer() lexer.RegisterToken(\u0026#34;listen\u0026#34;) lexer.RegisterToken(\u0026#34;on\u0026#34;) lexer.RegisterToken(\u0026#34;match\u0026#34;) lexer.RegisterToken(\u0026#34;=\u0026gt;\u0026#34;) lexer.RegisterTokenMatch(\u0026#34;FOOBAR\u0026#34;, go func(v string) bool { v == \u0026#34;foobar?\u0026#34; }) lexer.RegisterTokenMatch(\u0026#34;STRING\u0026#34;, lexer.IsString) lexer.RegisterTokenMatch(\u0026#34;NUMBER\u0026#34;, lexer.IsNumber) lexer.RegisterTokenMatch(\u0026#34;FLOAT\u0026#34;, lexer.IsFloat) // instanciate a grammar and teach it how to recognize token sequences grammar := parsey.NewGrammar() grammar.RegisterRule(configBuilderRule1, \u0026#34;listen\u0026#34;, \u0026#34;on\u0026#34;, \u0026#34;STRING\u0026#34;) grammar.RegisterRule(configBuilderRule2, \u0026#34;match\u0026#34;, \u0026#34;=\u0026gt;\u0026#34;, \u0026#34;STRING\u0026#34;) // instanciate a config parser using the configured lexer and grammar config := parsey.NewConfiguration(lexer, grammar) // parse a file using that parser success, err := config.ParseFile(configFile) if err != nil { log.Fatal(err) } which will parse the following format, supporting line breaks, comments, quoting and free whitespaces:\nlisten on fxp0 match =\u0026gt; fxp0 listen on \\ fxp0 match =\u0026gt; fxp0 Of course the sample above isn\u0026rsquo;t how this is going to work because recognizing token sequences is not enough to produce usable configuration files, there needs to be support for expressions and conditionals, both of which I already have code for on my laptop. I\u0026rsquo;m currently more focused on what the API should look like so it isn\u0026rsquo;t too difficult to use.\nOH, and why don\u0026rsquo;t I use Goyacc, you ask ?\nIt boils down to three reasons:\nFirst, I\u0026rsquo;d like to have a declarative setup similar to what I did in go-privsep and the ability to either freeze a config format or \u0026ldquo;extend\u0026rdquo; it dynamically if I wish.\nThen, I haven\u0026rsquo;t hand-written a parser from scratch since I was a student and this will remove some dust in my brain, even if I end up not using it\u0026hellip; it\u0026rsquo;s the journey, not the destination.\nFinally, because it seems that I\u0026rsquo;m not smart enough to get Goyacc to do what I want otherwise I would have used it to start with and not bother with this 😃\nAnyways, don\u0026rsquo;t hold your breath, this isn\u0026rsquo;t a serious project and I\u0026rsquo;ll work on it on and off throughout 2022.\nPlakar # I did a TON of work on plakar with over 150 public commits and some more in a private branch.\nThe following subsections will describe the most interesting changes, in no particular order, though many more are interesting for the project but not so much for an article.\nTried (failed) to fix fuse support build on OpenBSD\u0026hellip; # But it turns out that after fixing my code, then fixing the code in the dependency, I hit a problem in Golang\u0026rsquo;s runtime as the mount system call is not implemented for OpenBSD yet. I have a diff but I haven\u0026rsquo;t been able to test it yet as I don\u0026rsquo;t feel like breaking my runtime right now.\nThe rabbit hole was deep and I fell in it.\nRewrote the network code # I did a lot of work in both client and server code to clean them from the disgusting proof of concept to something decent, and allowing more parallelism of operations. I still have work to do in that area but the foundations are clean now.\nThe network code now uses gob to produce a binary protocol that I don\u0026rsquo;t have to parse myself thanks to gob encoders/decoders working directly on a connection handler, what a pleasure to not have to do that myself.\nThe server can now handle concurrent requests from the same client: for example it is capable to process multiple requests at once contrarily to before, speeding up considerably transfers.\nThe client also benefited from concurrency improvements.\nReworked the storage interface # I reworked the storage interface to ease the writing of custom storage backends, it is now relatively easy to write a backend and plug it in plakar without having to make changes outside of the implementation.\nThis sounds like nothing but interface implementation in Golang was puzzling me and I\u0026rsquo;m happy I worked it out.\nAdded support for an SQLite storage backend # With the storage layer reworked, I implemented an SQLite storage allowing plakar to use an SQLite database instead of the filesystem:\n$ plakar on sqlite:///tmp/plakar.db create -no-encryption $ plakar on sqlite:///tmp/plakar.db push /bin $ plakar on sqlite:///tmp/plakar.db ls 2021-11-27T00:12:04Z 99e53a9c-e503-4c32-8622-924bf966f15b 11 MB /bin $ This has advantages and disadvantages.\nSQLite backend is MUCH faster than using the filesystem on a plakar push / plakar pull, mainly because operations are reduced to reads \u0026amp; writes in SQLite when the FS has to open / close files too for each chunk and object accesses. The commit of a snapshot in SQLite is very efficient and fast, whereas on the FS it can take a long while due to a lot of hard links tricks. But\u0026hellip; on the other end, I trust my filesystem more than SQLite and I don\u0026rsquo;t have to worry about multiple concurrent writers there.\nEdit: After thinking more about it, my irrational fear of storing backups in a single structured binary file doesn\u0026rsquo;t hold: I already do it with tarballs.\nI also had a concern with concurrent writes because, many years ago, a writer would lock the entire database. This would for instance prevent plakar from pushing a small snapshot while a very big one was holding a transaction for an extended period of time. But I was stuck in the past, as SQLite introduced a WAL \u0026hellip; 11 years ago.\nThanks to Glandos @ Github for questionning my SQLite fear which had me revisit the concurrent writers concern and realize it no longer held ground, this will be very useful as even the SQLite code I wrote still assumed this old behaviour:\nFor example, the local cache opens the database in read-only and keeps track of all writes it wants to do, then when a snapshot is committed, it reopens the database in write mode to flush changes and reduce the time it had to retain the database open. This whole contorted way of working is no longer relevant and code can be simplified a lot.\nWhat\u0026rsquo;s interesting is that SQLite support uses a generic SQL database connector, so I will be able to implement support for any SQL database by rewriting some queries\u0026hellip; and obviously I have in mind databases that allow replication.\nOptimized local cache logic and replaced with an SQLite implementation # The local cache implementation was suboptimal, only useful to test the logic but not really to improve performances, partly because it\u0026rsquo;s way of working was stepping in the way of the push algorithm\u0026hellip; so I reworked it to improve the situation.\nWhile at it, I switched from the previous FS-based local cache to an SQLite local cache, mainly to avoid the cost of multiple system calls whenever checking if something was in cache.\nImplemented a small plakar shell # Nothing too fancy, I just needed to be able to run multiple commands on the same plakar session, the shell just accepts commands and applies them to the same opened plakar.\n$ plakar on sqlite:///tmp/plakar.db shell plakar@sqlite:///tmp/plakar.db\u0026gt; ls 2021-11-27T00:12:04Z 99e53a9c-e503-4c32-8622-924bf966f15b 11 MB /bin plakar@sqlite:///tmp/plakar.db\u0026gt; ls 2021-11-27T00:12:04Z 99e53a9c-e503-4c32-8622-924bf966f15b 11 MB /bin plakar@sqlite:///tmp/plakar.db\u0026gt; ^C $ That doesn\u0026rsquo;t look too useful but that\u0026rsquo;s because in my articles I mainly provide examples on an -no-encryption plakar, so you don\u0026rsquo;t see me prompted for a passphrase every command I type. When you have to type multiple commands, using the plakar shell will only prompt at opening saving sanity.\nImplemented a filesystem view # The snapshot indexes consisted mainly of maps mapping checksums to pathnames or pathnames to file informations, etc\u0026hellip;\nThis was efficient for direct access to a resource by its name, like searching what\u0026rsquo;s the file informations for /etc/passwd, but not so much for operations involving partial pathnames, such as finding what files are in a directory. Unfortunately such operations are numerous as plakar supports a ls command to browse the snapshot, but also cat-ing, pull-ing or even creating tarballs from subdirectories within a snapshot.\nTo work around, I had implemented helpers that would do things like comparing full pathnames and determining if a path was within another, but this was confusing and inefficient as it required looking at the entire set of entries in the snapshot, and playing games with string prefixes.\nI implemented a filesystem within the index in the form a tree of file informations, as well as a set of functions to lookup pathnames in that tree. This has considerably simplified a lot of code and allows to lookup resources in the index similarly to how I would on the real filesystem.\nI kept the maps as indexes for direct lookups so that whenever searching for a precise resource, there is no need to go through the filesystem and traverse nodes, but so that whenever searching for a set of resources or trying to discover what\u0026rsquo;s in a directory hierarchy, the tree can be used instead.\nImplement CPU throttling # Until now, plakar would use all of the available resources to perform operations as fast as possible.\nWhile I think the default should be to run as close as possible to max limit and let the machine throttle, it\u0026rsquo;s also not a good idea to not leave at least one core for the OS to work correctly.\nI implemented CPU throttling so that plakar can be told to limit itself to a certain number of cores with the -cpu option, and I set the default to be between 1 and cpus-1 cpus, whichever is the most. On a 1 core VPS, it will consume up to a full CPU, while on my 8 core laptop it will consume 7 by default.\nThe effect of passing a -cpu option are quite visible as you can see with this plakar -cpu 4 push ...\nSimplify and optimize plakar push # I reworked the push logic to simplify and optimize the strategy, leading to less store calls and much improved performances with much more readable code.\nWith the help of discord user Spire, improvements were made leading to performances boosts due to code patterns that caused too many allocations.\nThere are still improvements to make but things are looking bright 😃\nParallelized plakar keep, plakar rm, plakar ls # Both plakar keep and plakar rm worked by removing snapshots sequentially from the store, I updated them so that they could run the removals in concurrency, allowing for faster execution.\nThe plakar ls case was similar in that to display the snapshots listing, it had to fetch the snapshot indexes, and it did that sequentially. When multiple big snapshot indexes were in the store, this could lead to a delay before the listing was displayed as each snapshot index would be fetched and deserialized sequentially, not exploiting the availability of multiple cores. I converted plakar ls to use concurrency for the fetch and deserialize, synchronizing the routines before the sorting and display.\nFixed UI # The UI was badly broken due to the many changes I made in the API, so I spent a while fixing it and taking advantage of the new snapshot filesystem view to provide directory browsing:\nI also improved slightly the view for individual resources, added the ability to download files or tarballs.\nImplemented plakar clone # I wrote a plakar clone command to fully replicate a repository:\n$ plakar on sqlite:///tmp/plakar.db clone /tmp/cloned $ plakar on /tmp/cloned ls 2021-11-27T00:12:04Z 99e53a9c-e503-4c32-8622-924bf966f15b 11 MB /bin $ This allows creating an exact replica of an existing plakar repository, regardless of where it\u0026rsquo;s located (filesystem, network or database) into a local directory, and respecting the internal structure of the original repository: it is not just a dumb file copy but really the cloning of the internal state, preserving reference counts and such.\nThe fact that it can only clone to a local directory is a temporary technical limitation that will be lifted in the future as my ultimate goal is to be able to:\n$ plakar on plakar://192.168.1.2 clone plakar://192.168.1.1 $ \u0026hellip; and have a plakar be reconstructed from a remote machine to another remote machine.\nThe idea is that you can clone an off-site plakar on a regular basis and know that if something happened to the main one, you\u0026rsquo;d still have a version of it that has a coherent internal state. When the local filesystem technical limitation is lifted, I\u0026rsquo;ll working on a synchronization mechanism so that two remote plakar may exchange chunks, objects and snapshots in an efficient way, without having to clone the entire repository.\nImplemented plakar check and fast checking # The plakar check \u0026lt;snapshot\u0026gt; command already existed to check the integrity of a snapshot by requesting the store to return each of its chunks and objects, allowing plakar to both validate that it could perform a full restore if needed and that checksums matched.\nAfter a discussion on Discord, I implemented fast checking which is a relaxed check requesting only that the store acknowledges existence of chunks and objects. This allows speeding up considerably the snapshot checks when you trust the store to respond honestly on the availability of resources.\nI finally implemented plakar check (no snapshot parameter) to perform a full plakar repository coherency check, validating that the store doesn\u0026rsquo;t contain orphan chunks and objects, but also that they have a reference count matching the number of snapshots relying on them, and that no snapshot lacks a reference to a resource that\u0026rsquo;s still in the store\u0026hellip; only because it is held by another snapshot.\nAs long as a resource is available in the store, even if the snapshot lost its reference somehow, plakar will manage to restore the resource: this is nice because you don\u0026rsquo;t want to fail recovering something for which you do have the data, but you also don\u0026rsquo;t want that to hide the fact that there was a corruption in the store somehow.\nIn theory, unless you poke in the store and remove resources yourself, these checks are never meant to fail but I\u0026rsquo;ll sleep better knowing that I can have daily snapshots fast checks, weekly snapshot checks and monthly repository checks\u0026hellip; and that if something goes wrong, I can pull a backup and investigate what went wrong.\nA nice improvement would be to provide the ability to repair some corruptions when doable, but since this is not supposed to happen in the first place, it\u0026rsquo;s not very high in my list 😁\nAssorted other work # After a month of learning, I can now play Ode to Joy, Autumn Leaves, and And I love her on the piano, that required some assorted work.\n","date":"26 October 2021","permalink":"/posts/2021-10-26/november-2021-a-bit-of-go-ipcmsg-a-bit-of-go-privsep-and-a-ton-of-plakar/","section":"Posts","summary":"TL;DR: I still have a discord, feel free to join. I worked on go-ipcmsg to make it nicer, go-privsep to make it more useable, and A LOT on plakar to make it plakar.","title":"November 2021: a bit of go-ipcmsg, a bit of go-privsep and a ton of plakar"},{"content":" TL;DR: I have a discord now, feel free to join. I refactored plakar, implemented a local cache, improved parallelism, modified the push strategy, played with fuse and networking. I also did other stuff but let\u0026rsquo;s keep that out of this article. I have a discord now # I have a rubber duck sitting on my desk\u0026hellip;\n\u0026hellip;but I also like explaining to my peers what I\u0026rsquo;m doing, as I\u0026rsquo;m doing it, to help me get new ideas or spot shortcomings in my reasoning. I used to do it a lot on IRC as I worked on OpenSMTPD, with other developers and an active community, but I miss that a lot now that I work mostly alone on low-profile projects. I do share ideas and progress on Twitter but the tweet size limitation makes it hard to expand much and incite discussion.\nI created a Discord where I\u0026rsquo;ll hang out and discuss my projects as I work on them. Feel free to hop in if you want, and feel free to do just like me and share thoughts as you work on your own projects there: this is a virtual hack room.\nIt may not be restricted strictly to code as I have other unrelated projects sometimes :-)\nPlakar refactor # I have been working on plakar on and off for a few months now, with approximately two weeks of cumulative work, and because it was experimental and my first real project in Go, I made many mistakes both in design and implementation details. I tested a lot of ideas, some turned out to be good and I pushed them further whereas others turned out to be crap and I abandoned them.\nFor instance, I wasn\u0026rsquo;t sure if snapshots should be part of the storage engine or built on top, I wasn\u0026rsquo;t sure if encryption/compression should be at the snapshot level or at the storage level, or even if it should be possible to encrypt some snapshots and not others within a store. All of these decisions are now settled, but I had to experiment a bit before understanding why I was taking a decision and why I believed it was the right one, and this left the code with some weird parts as things weren\u0026rsquo;t always done in the right place.\nNow that I\u0026rsquo;m a bit more comfortable with Go and with how the project will move forward, it was time to scrap the draft and rewrite it properly. I spent an evening in a new branch, rewriting all layers by bringing just the bits of code necessary and making sure that each package was not doing anything that should be done elsewhere. This allowed me to kill a lot of dead code from older experiments, simplifying the logic in all layers and clarifying some boundaries.\nThis is still the very early stages of the project with a lot of room for improvement on the code base, but at least the foundation is now clean: when I work on a feature, it is clear where I should add the feature and I don\u0026rsquo;t risk breaking everything.\nCode made available on Github # Since I\u0026rsquo;m no longer ashamed of what I produced, I decided to work in the open and make the code available in a github repository so I can reference a commit when I write about plakar here instead of leaving everything to your imagination.\nDon\u0026rsquo;t use the project for anything serious:\nFirst of all, there are still issues and you don\u0026rsquo;t want to use something that has issues for your backups. Then, the utility still evolves a lot with regard to output or CLI, and I don\u0026rsquo;t want to provide support regarding last weeks\u0026rsquo; command line or output format when I have a different version on my laptop. Finally, the storage format hasn\u0026rsquo;t fully stabilized yet as I make minor changes here and there to optimize some operations: the snapshots you make today may not be restorable next week if I make use of a new metadata or change a data structure.\nOf course, I\u0026rsquo;d be very happy to have some people test and help me improve the tool, so feel free to test and report issues\u0026hellip; as long as this is only for testing and not for your real backups.\nImproved parallelism # Because of how plakar works, many operations involve splitting data into chunks or fetching data from the store to reconstruct objects.\nFor example, plakar cat deadbeef:/etc/passwd will:\nfetch the index for snapshot deadbeef from the store (or from cache if available) resolve /etc/passwd into an object identifier within the store fetch the object index from the store (or from cache if available) resolve the object into one or many chunks within the store fetch the chunks from store and output them in sequence to stdout Whereas plakar push /bin will iterate through /bin and:\nopen and split every file into chunks query the store for which chunks need to be written or have their reference count incremented write the missing chunks write an object index for every individual object write the snapshot index which maps the path names to their backing objects, among other things In my initial work on plakar, all commands were implemented with a very sequential approach, operations being performed one after another regardless if they could be parallelized or not. There was a tiny bit of parallelism, as plakar could work on multiple files at once, but it would take a sequential approach for each of these files.\nIn many cases, this caused it to operate at much slower speed than was possible to reach given the available resources on the host system. For instance, in the following screenshots, it took roughly 55s to push my 2.5GB ~/Downloads directory to an empty repository on my ~1000MB/s/writes SSD despite the 8 cores being underused.\nI reworked the storage and the snapshot layers to improve parallelism and synchronization so that plakar could safely parallelize every possible operation. This allowed it to fully exploit all available cores and avoid idling when it could be doing something, like writing multiple chunks in parallel for the same file while processing multiple files, and letting disk I/O limit the performances.\nIn the screenshots below, it took only 14.4s to push the same directory to an empty store while utilizing all cores to their fullest. It could certainly be optimized further because the commit of a snapshot has not gone through optimization yet and probably consumes over half of this time, but the difference is already quite impressive.\nThis parallelism improvement is not restricted to pushing snapshots, but benefits all plakar operations. For instance restoring snapshots became significantly faster as it used to restore all files sequentially when it will now parallelize the whole process of rebuilding the file hierachy, creating the files and fetching the chunks.\nI was afraid that the amount of goroutines would be unmanageable as I hit a couple panics when working on directories \u0026gt; 10GB with tens of thousands of files and even more chunks, but I came up with a code pattern that ensured I could both control the amount of concurrent routines executing a specific task, and that they were all done before beginning a dependent task. I\u0026rsquo;ll probably write about that in a future post as I find this solution very elegant and suitable for many use-cases.\nI will likely be adding an option to specify the amount of parallelism allowed to prevent plakar from hogging all resources at the cost of slower operations, but it is a very simple feature that I\u0026rsquo;ll keep for when I want to pretend I worked hard on something :-)\nSnapshots and objects caching # When plakar creates a snapshot, it has to scan file hierarchies and read every files to split them into chunks that will be checked against the store. This isn\u0026rsquo;t something that has to be avoided, it is really what I want it to do, but it is not always desirable as it is only necessary for files that have changed. Unless plakar is able to detect that, it has to pay the price of reading all files including the ones that haven\u0026rsquo;t changed.\nI implemented a local cache which recalls, among other things, the structure of objects that were part of previous snapshots and the inode informations associated to path names.\nWhenever scanning a directory, the inode informations for the current file are compared to the last known informations stored in the cache. If it is determined that the file has not changed since the last version, plakar will reuse the chunk information from the cache and increase the chunks reference count in store. This avoids a full read during the chunking and a potential full write of all chunks to the store.\n$ plakar -time push ~/Downloads time: 13.787579792s $ plakar -time push ~/Downloads time: 1.41720375s $ plakar -time -no-cache push ~/Downloads time: 7.432093292s $ plakar ls 2021-10-26T11:51:01Z e5c24e03-7f72-4a9b-81d4-2b891951f965 2.5 GB (files: 1225, dirs: 282) 2021-10-26T11:51:20Z 784a2a28-0af5-45e7-b440-1246c5691080 2.5 GB (files: 1225, dirs: 282) 2021-10-26T11:51:26Z e6a24d44-f257-4540-8ff7-04f251acf522 2.5 GB (files: 1225, dirs: 282) In the example above, I pushed my 2.5GB ~/Downloads directory in an empty plakar, then pushed it again once with caching and once without caching. Here, the time difference for a push with or without cache is very significant, but it is highly dependant on the number of directories, of files, of their redundancy and their sizes.\nJust like for parallelism, the local cache doesn\u0026rsquo;t only benefit the creation of snapshots but also other operations, though the boost is not as beneficial due to its current implementation: it requires small disk reads for each lookup and while this can save a lot in the snapshot creation code path by avoiding big file reads and tons of writes, it is not that interesting in code paths involving mostly the reading of chunks.\nThere is still a lot of room for improvement though:\nFirst of all, the structure of informations in cache is not optimal and makes it hard to use the cache for some operations that could benefit from it. This is something simple to fix, it\u0026rsquo;s just that I figured what I should have done differently while I was writing this article :-)\nThen, I implemented the cache using individual files to represent snapshots and objects, which means each cache query involves opening, reading and closing a file. This was acceptable to bootstrap the API and start using it but has a lot of overhead which would not exist if the cache was stored in an SQLite database or similar.\nI will continue working on local cache improvements but am already happy with the results of that first naive approach.\nFor console pr0n, here\u0026rsquo;s a run of a push with an empty cache and empty store:\n$ plakar -trace push . 332752a6-c220-4d7f-ab22-290a1c67862f: New() 332752a6-c220-4d7f-ab22-290a1c67862f: cache.GetPath(./group): KO 332752a6-c220-4d7f-ab22-290a1c67862f: cache.PutPath(./group) 332752a6-c220-4d7f-ab22-290a1c67862f: cache.GetPath(./services): KO 332752a6-c220-4d7f-ab22-290a1c67862f: PutChunk(4910bfe2b7e551c4e2085b12c36941d1e1063491b7292cb0dbca7c5fe0854be5) 332752a6-c220-4d7f-ab22-290a1c67862f: cache.PutPath(./services) 332752a6-c220-4d7f-ab22-290a1c67862f: cache.GetPath(./passwd): KO 332752a6-c220-4d7f-ab22-290a1c67862f: cache.PutPath(./passwd) 332752a6-c220-4d7f-ab22-290a1c67862f: PutChunk(e45b72f5c0c0b572db4d8d3ab7e97f368ff74e62347a824decb67a84e5224d75) 332752a6-c220-4d7f-ab22-290a1c67862f: PutChunk(8c6e2a2647ee854f469a3bb798e02ba5a8b1812cab229ff129f073e7a80c1202) 332752a6-c220-4d7f-ab22-290a1c67862f: PutObject(8c6e2a2647ee854f469a3bb798e02ba5a8b1812cab229ff129f073e7a80c1202) 332752a6-c220-4d7f-ab22-290a1c67862f: PutObject(4910bfe2b7e551c4e2085b12c36941d1e1063491b7292cb0dbca7c5fe0854be5) 332752a6-c220-4d7f-ab22-290a1c67862f: PutObject(e45b72f5c0c0b572db4d8d3ab7e97f368ff74e62347a824decb67a84e5224d75) 332752a6-c220-4d7f-ab22-290a1c67862f: PutIndex() snapshot: cache.PutIndex(332752a6-c220-4d7f-ab22-290a1c67862f) 332752a6-c220-4d7f-ab22-290a1c67862f: Commit() And here\u0026rsquo;s a run of a push using the cache:\n$ plakar -trace push . 0c8ab5bf-add9-4e20-9790-97d43cde35cc: New() 0c8ab5bf-add9-4e20-9790-97d43cde35cc: cache.GetPath(./group): OK 0c8ab5bf-add9-4e20-9790-97d43cde35cc: cache.GetPath(./services): OK 0c8ab5bf-add9-4e20-9790-97d43cde35cc: cache.GetPath(./passwd): OK 0c8ab5bf-add9-4e20-9790-97d43cde35cc: PutIndex() snapshot: cache.PutIndex(0c8ab5bf-add9-4e20-9790-97d43cde35cc) 0c8ab5bf-add9-4e20-9790-97d43cde35cc: Commit() Push strategy # When plakar wasn\u0026rsquo;t parallelized and didn\u0026rsquo;t have a cache, it was easier to interact with the store on an object-after-object basis: it would open a file, parse it into chunks, write the chunks, write the object, then move to the next object. This ensured that the full handling of a file could be isolated in a small process: everything that needed to be done with the file would be done in the window of time that it was opened, all in a very readable and sequential logic.\nWith parallelization and caching, it became more interesting to work in a different way: processing chunks at full speed first, THEN processing object indexes at full speed when all chunks are already recorded in the transaction. This allowed to simplify the logic by a great deal, replacing the sequential pattern of read chunk / write chunk / read next chunk / write next chunk, with a channel where chunks are pushed out of order as they are read and goroutines concurrently pop from the channel to write to store as fast as they can. This also had the benefit to ease deduplication of chunks and objects within the snapshot itself.\nThe current push implementation is already much more interesting than the previous one but I\u0026rsquo;m not done yet and the good thing is that, thanks to the refactor, snapshots are built on top of the storage and I\u0026rsquo;m able to experiment knowing that all my changes are isolated in that layer.\nFUSE experiment # Another feature I\u0026rsquo;m experimenting with is FUSE, an API to implement user-space filesystems.\nI\u0026rsquo;m only beginning to play with it but have implemented plakarfs, a read-only user-space filesystem exposing a plakar repository, and plakar mount which allows mounting a plakarfs. This makes it possible to browse snapshots and read files, as if all snapshots were restored, but\u0026hellip; without consuming the disk space:\n$ plakar ls 2021-10-26T10:43:12Z 9aed1ded-0e0d-4d6c-946a-6757178ec2f3 3.2 MB (files: 248, dirs: 42) 2021-10-26T10:43:13Z ecfef47f-24fc-4c6a-9dbc-fdf918f40689 3.2 MB (files: 248, dirs: 42) 2021-10-26T10:43:13Z 561edc6a-e531-4770-872a-610432d0a2b9 3.2 MB (files: 248, dirs: 42) $ plakar mount /tmp/plakar \u0026amp; [1] 19253 $ ls -l /tmp/plakar total 18912 dr-xr-xr-x 1 root wheel 3224317 26 Oct 12:43 561edc6a-e531-4770-872a-610432d0a2b9 dr-xr-xr-x 1 root wheel 3224317 26 Oct 12:43 9aed1ded-0e0d-4d6c-946a-6757178ec2f3 dr-xr-xr-x 1 root wheel 3224317 26 Oct 12:43 ecfef47f-24fc-4c6a-9dbc-fdf918f40689 $ tail /tmp/plakar/ecfef47f-24fc-4c6a-9dbc-fdf918f40689/etc/passwd _logd:*:272:272:Log Daemon:/var/db/diagnostics:/usr/bin/false _appinstalld:*:273:273:App Install Daemon:/var/db/appinstalld:/usr/bin/false _installcoordinationd:*:274:274:Install Coordination Daemon:/var/db/installcoordinationd:/usr/bin/false _demod:*:275:275:Demo Daemon:/var/empty:/usr/bin/false _rmd:*:277:277:Remote Management Daemon:/var/db/rmd:/usr/bin/false _fud:*:278:278:Firmware Update Daemon:/var/db/fud:/usr/bin/false _knowledgegraphd:*:279:279:Knowledge Graph Daemon:/var/db/knowledgegraphd:/usr/bin/false _coreml:*:280:280:CoreML Services:/var/empty:/usr/bin/false _trustd:*:282:282:trustd:/var/empty:/usr/bin/false _oahd:*:441:441:OAH Daemon:/var/empty:/usr/bin/false $ The filesystem hierarchy within the mount is built using snapshot indexes, including inode informations such as creation time, uid/gid or file size, while file reading is implemented by fetching the object index and just the chunks needed to service the read, so that chunks will be fetched as needed based on the current offset within the file.\nI have a hard time getting around with the proper way to implement plakar as a filesystem, so there are still a lot of glitches and a few limitations, but this looks very promising as far as I\u0026rsquo;m concerned. In particular because this is built on top of snapshots, themselves built on top of the store, making this work transparently with encrypted or remote stores.\nRemote plakar # I had already written that I implemented server and client support for plakar in this post, but the idea behind it was only to validate that the storage primitives could be used over the network, not to be actually usable. The client and server were implemented in a very ugly way, hacked as custom storage backends, lacking error checking and not designed to handle concurrency within a single snapshot transaction.\nI\u0026rsquo;m now at the point were I want to implement them correctly, so I have started looking at the transport and working on the protocol, experimenting with different things. I tested three PoCs at this point but am not satisfied yet.\nAnyhow, I expect this network mode to be working by the end of December as I\u0026rsquo;d like to have a plakar running on my NAS by then :-)\nAssorted other work # I also worked on other stuff but I didn\u0026rsquo;t want to pollute this post so I will just summarize:\nI submitted a diff to OpenBSD to fix regex support in OpenSMTPD\u0026rsquo;s table_db backend ( committed), helped eric@ fix and review a diff for a bug in SRS ( committed) and am currently reviewing a diff to integrate my table_procexec as the backend to run external tables in OpenSMTPD.\nI worked on a new project that I\u0026rsquo;ll write about in a few months as it is not high priority and I\u0026rsquo;d like to get plakar mostly out of my way first, but it involves mobile development, something I haven\u0026rsquo;t done in over a decade.\nI also played a bit with Arduino to learn myself some new skills, but I didn\u0026rsquo;t do anything newsworthy.\n","date":"26 October 2021","permalink":"/posts/2021-10-26/october-2021-mostly-plakar/","section":"Posts","summary":"TL;DR: I have a discord now, feel free to join. I refactored plakar, implemented a local cache, improved parallelism, modified the push strategy, played with fuse and networking.","title":"October 2021: mostly Plakar"},{"content":" TL;DR: this is a short article, no TL;DR. Why so calm ? # As some of you know, I was diagnosed a few years ago with generalized anxiety disorder, panic disorder and alexithymia, a trait that makes it particularly difficult for me to cope with the disorders.\nAs I wrote in 2019, I didn\u0026rsquo;t take the alexithymia diagnosis too well. It took me a long time to stop ruminating, move forward, and start trying to find a fix for myself with virtually no help. It turns out that I have made huge progress thanks to hypnosis and efforts at trying to decode sensations into emotions, but the road is still long and full of ups and downs.\nThis brings us to 2021, two years later.\nToo many things hinted me that a neurodivergence was at the root of my problems, I could no longer pretend not to see it. I took a psychological assessment which confirmed my intuition, then took the report to a different psychologist for a second opinion and got a confirmation.\nI\u0026rsquo;m neurodivergent.\nI don\u0026rsquo;t want to share the details because it doesn\u0026rsquo;t really matter how divergent I am, but just like in 2019, I don\u0026rsquo;t take this news too well and I need some time to accept it.\nI\u0026rsquo;m ok, no worries, nothing a summer alone with my wife and kid can\u0026rsquo;t fix.\nA quick word about sponsorship and this blog # If you are sponsoring me, please take a time to reconsider if you still want to support me.\nI have 5 days each month that I can dedicate however I want and, out of these 5 days, 1 is almost fully consumed by the monthly report because I\u0026rsquo;m a slow writer. I only have 4 days I can use to work on stuff and, because the monthly report can\u0026rsquo;t be empty, within these 4 days I must find things to do that I can write about.\nThis doesn\u0026rsquo;t leave much room for learning new skills and doing useless experiments. I find myself working on interesting stuff (to me), then watching the clock because I want to leave enough time to work on something I can write about. Who wants to read about my experiments converting EEG metrics to midi so I can listen to my brain waves ? Obviously, people prefer when I write code that brings new features in OpenSMTPD.\nIn June and July, I spent my days unsuccessfully trying to do some VR, and this didn\u0026rsquo;t leave me time to write the monthly reports. I felt bad for my sponsors as I couldn\u0026rsquo;t deliver reports, I had 0 things to write.\nSince the beginning of August, I\u0026rsquo;m experimenting with Dart/Flutter to write a mobile app using a Golang backend and a cockroach database. It\u0026rsquo;s a fun project that I\u0026rsquo;ll eventually publish and write about but if I want to produce a monthly report, I need to stop right away and dive into code I\u0026rsquo;m familiar with so I know I can produce something fast enough, rather than continue working on a fun new project that I can\u0026rsquo;t write about yet.\nI find this counter-productive, the monthly reports are preventing me from working on what I want, I think it is time to stop this format which is putting too much pressure on me and limiting future projects.\nInstead, I will write reports when I have valuable informations to share, and continue to commit everything to Github. Sometimes you will see a lot of activity on consecutive months, sometimes you will not see anything happening for a quarter.\nIf you\u0026rsquo;re not ok with that, I understand and am grateful for your support until this point \u0026lt;3\nI sent a notice to all my patrons so they are aware that I will change my way of working.\nLove \u0026lt;3\n","date":"13 August 2021","permalink":"/posts/2021-08-13/august-2021-taking-a-break/","section":"Posts","summary":"TL;DR: this is a short article, no TL;DR. Why so calm ? # As some of you know, I was diagnosed a few years ago with generalized anxiety disorder, panic disorder and alexithymia, a trait that makes it particularly difficult for me to cope with the disorders.","title":"August 2021: taking a break"},{"content":" TL;DR: J\u0026rsquo;explique superficiellement comment fonctionne notre esprit, en partant du corps, puis de notre sens de la réalité, avant de finir par les états de conscience et personnalités. Je plonge brièvement dans les rêves et l\u0026rsquo;hypnose, préparant le terrain pour un article à venir. Avertissement # Mon intérêt pour les rêves lucides a débuté aux alentours de 2013, j\u0026rsquo;ai étudié seul puis suivi différentes formations de différent type d\u0026rsquo;hypnose entre 2015 et 2020, et je pratique l\u0026rsquo;hypnose en cabinet depuis 2016 jusqu\u0026rsquo;à maintenant. Sans lien avec cette activité, j\u0026rsquo;étudie la psychologie du travail depuis 2016 et ai, dans ce cadre, suivi des cours d\u0026rsquo;introduction à la psychologie clinique et à la psychologie cognitive.\nJe ne suis ni psychologue, ni psychiatre, ni un chat, alors prenez cet article avec le recul adéquat.\nÀ propos de cet article # Cet article est une traduction de mon article Mind hacking: understanding how our mind works publié sur ce blog\nJe tente d\u0026rsquo;expliquer, d\u0026rsquo;un point de vue macroscopique, ma compréhension de comment fonctionne l\u0026rsquo;esprit, en me basant sur ce que j\u0026rsquo;ai étudié, ainsi que mes observations et expériences personnelles au travers du rêve lucide et de l\u0026rsquo;hypnose.\nCet article ne contient pas de nouvelles idées révolutionnaires, vous y reconnaîtrez peut être quelques éléments des travaux de Charcot, Janet, Breuer et même Freud, même si la façon dont je les présente peut varier car je n\u0026rsquo;adhère pas nécessairement à tout et adapte donc à MA compréhension.\nIl s\u0026rsquo;agit d\u0026rsquo;une vulgarisation qui prends des raccourcis volontaires pour véhiculer les idées principales le plus simplement, il ne s\u0026rsquo;agit absolument pas d\u0026rsquo;une explication exhaustive et chaque concept pourrait être détaillé sur des pages entières. Je serais ravi de développer dans les commentaires ou des articles suivants.\nEn résumé: ne partez pas du principe que cet article s\u0026rsquo;applique tel quel à tous, particulièrement aux personnes avec des troubles qui affectent leur sens de la réalité ou leur/s personnalité/s, il ne s\u0026rsquo;agit pas d\u0026rsquo;une description exacte de comment l\u0026rsquo;esprit fonctionne pour tous.\nNous sommes des ordinateurs biologiques # Nous sommes des ordinateurs biologiques qui résultent des spécifications combinées de deux autres ordinateurs biologiques. Ces spécifications déterminent comment nous sommes construits, à quoi nous ressemblons, quelles sont nos aptitudes et nos limites, mais pas comment nous traitons le monde, pas comment fonctionne notre système.\nNous nous interfaçons avec le monde au travers de nombreux capteurs, appelés récepteurs, répartis dans nos yeux, oreilles, langue, nez et sous notre peau pour capturer des signaux visuels, auditifs, gustatifs, olfactifs et kinesthésiques qui sont ensuite interprétés et nous permettent de voir, entendre, goûter, sentir et toucher les éléments de notre environnement. Ces récepteurs forment notre appareil sensoriel, ce que nous appelons nos sens.\nCes récepteurs font partie de ce que des développeurs pourraient considérer comme notre API. Ils sont les seules interfaces qui permettent de percevoir des signaux d\u0026rsquo;entrée depuis notre environnement, permettant à notre système de les traduire en sensations et d\u0026rsquo;encoder des souvenirs. Sans récepteurs, aucun signal n\u0026rsquo;entrerai dans notre système, nous vivrions alors une vie très terne de néant absolu. Par chance, nous avons des récepteurs dans nos yeux pour voir une tablette de chocolat blanc sur la table, sur nos mains pour sentir son contact lorsque nous l\u0026rsquo;attrapons, dans notre nez et notre langue pour sentir son odeur et son goût, et dans nos oreilles pour entendre quelqu\u0026rsquo;un demander où est passée la tablette. Oups.\nParce que nos sens sont notre seul interface d\u0026rsquo;entrée, tout ce que nous savons de notre environnement provient de nos récepteurs. Notre connaissance n\u0026rsquo;est pas apparue en nous, elle a été insérée au travers de nos sens, soit parce qu\u0026rsquo;un autre ordinateur biologique nous as transmis des signaux pour partager l\u0026rsquo;information, soit parce que nous avons interagis avec notre environnement et reçus des signaux en retour.\nLa plupart des signaux sont ignorés, sauf si l\u0026rsquo;on y prête attention, parce qu\u0026rsquo;ils sont sans importances pour nos activités courantes, comme c\u0026rsquo;est le cas pour le bruit d\u0026rsquo;une horloge que l\u0026rsquo;on entend mais oublie comme si elle n\u0026rsquo;était plus là\u0026hellip; alors qu\u0026rsquo;elle l\u0026rsquo;est toujours. Beaucoup de signaux sont traités à un bas-niveau inconscient et enregistrés pour un usage ultérieur, comme la pensée passée d\u0026rsquo;une tablette de chocolat blanc dont on a un peu envie là maintenant. Quelques signaux sont traités et rendus disponibles immédiatement à notre conscience pour que nous puissions agir dessus, comme ce texte surligné qui attire immédiatement notre attention.\nCes récepteurs sont ce qui nous aide à développer et customiser notre système. Ils nous rendent disponible de l\u0026rsquo;information concernant notre réalité, ce qui change la façon dont on traite notre environnement et interprète de nouveaux signaux. Notre système construit une compréhension du monde au travers d\u0026rsquo;une boucle de rétroaction sensorielle.\nLa réalité # Il peut sembler étrange de discuter du concept de réalité dans un article sur le fonctionnement de notre esprit mais ce n\u0026rsquo;est pas le cas. Nous vivons dans une réalité et tout ce nous faisons lui est relative: on y vient à la vie, on en apprend les règles au travers d\u0026rsquo;essais et d\u0026rsquo;erreurs, puis on développe des pensées et des comportements qui nous semblent y être adaptés. On ne peut pas commencer à comprendre comment notre esprit fonctionne sans comprendre comment il perçoit la réalité\u0026hellip; parce que la façon dont fonctionne notre esprit est dérivé de cette perception.\nQuand on pense à la réalité, on pense généralement à une réalité unique, LA réalité, et on sépare ensuite le monde en deux: d\u0026rsquo;un côté se trouvent ceux qui partagent la même réalité et que l\u0026rsquo;on considère sains, et de l\u0026rsquo;autre se trouvent ceux qui ne la partagent pas et qui sont fous. Si quelqu\u0026rsquo;un nous confie voir des cochons volants au dessus de nos têtes et qu\u0026rsquo;on ne les voit pas en levant les yeux, on reconnaît que sa perception de la réalité est inexacte parce que les cochons volants ne sont pas là. On s\u0026rsquo;attend à ce que les personnes autour de nous fassent l\u0026rsquo;expérience de la réalité de la même manière que nous parce que, s\u0026rsquo;il n\u0026rsquo;y a que LA réalité, alors il ne peut pas y avoir deux évènements contradictoires qui surviennent simultanément.\nMais ce n\u0026rsquo;est pas si simple.\nNous avons tous des aperçus différents de la réalité, et en développons un sens très personnel basé sur nos perceptions, même si elle est en grande partie partagée avec les autres être vivants. Nous pouvons vivre dans la même réalité, mais la mienne reste différente de celle d\u0026rsquo;une personne daltonienne dont les signaux visuels sont traduits différemment, ou d\u0026rsquo;une personne aveugle dont les signaux visuels sont absents. Nous pouvons vivre dans la même réalité mais en avoir une expérience différente\u0026hellip; parce que la réalité a de multiples dimensions.\nLa réalité objective # La réalité dans laquelle nous existons est la réalité objective, factuelle et contrainte par les lois de la physique, où les choses existent et se déroulent même si nous n\u0026rsquo;en avons pas conscience. Il s\u0026rsquo;agit de LA réalité dont la plupart d\u0026rsquo;entre nous pense faire l\u0026rsquo;expérience, une réalité unique qui est la même pour tous. Elle comprends tout ce qui a été par le passé, tout ce qui est dans le présent et tout ce qui sera dans le futur.\nTout ce qui existe dans l\u0026rsquo;univers, depuis chaque étoile dans chaque galaxie jusqu\u0026rsquo;à chaque grain de sable sur notre planète, fait partie de la réalité objective et existe que l\u0026rsquo;ont en ai connaissance ou non, et que nous puissions interagir avec ou non. Tous les évènements qui surviennent, n\u0026rsquo;importe où dans l\u0026rsquo;univers, font partie de cette réalité objective que l\u0026rsquo;on puisse les observer ou non. Ils existent et surviennent factuellement, que l\u0026rsquo;on le sache ou non.\nDans la réalité objective, si je vous dit que j\u0026rsquo;ai un jour mangé 1kg de chocolat blanc, cela n\u0026rsquo;a pas d\u0026rsquo;importance que vous me croyez ou non, il y a une vérité factuelle: c\u0026rsquo;est arrivé ou ce n\u0026rsquo;est pas arrivé. Si l\u0026rsquo;on est en désaccord, alors l\u0026rsquo;un de nous à tort. Il n\u0026rsquo;y a pas d\u0026rsquo;espace pour le débat dans cette réalité.\nLes signaux que perçoivent nos sens de notre environnement trouvent leur origine dans cette réalité objective, ils sont factuels et n\u0026rsquo;embarquent pas de signification subjective. Dans la réalité objective, le chocolat blanc a une odeur qui peut être perçue par nos récepteurs, mais cette odeur n\u0026rsquo;est ni plaisante, ni déplaisante, elle n\u0026rsquo;existe qu\u0026rsquo;en tant que signal qui peut être traduit et interprété par quelqu\u0026rsquo;un.\nLa réalité subjective # La réalité objective est infinie et complexe, et nous n\u0026rsquo;en sommes exposés qu\u0026rsquo;à une portion infinitésimale, ce qui nous empêche d\u0026rsquo;en faire l\u0026rsquo;expérience telle qu\u0026rsquo;elle est réellement. À sa place, nous faisons l\u0026rsquo;expérience d\u0026rsquo;une version dégradée qui ne contient que ce à quoi nous sommes exposés, et ce que nos sens parviennent à y percevoir\u0026hellip; une version différente pour chacun.\nC\u0026rsquo;est la réalité subjective, une représentation interne de la réalité objective qui est affectée par nos perceptions, et qui tente de remplir les trous des informations manquantes par des interprétations, des suppositions et l\u0026rsquo;extrapolation d\u0026rsquo;expériences passées. Elle est subjective parce qu\u0026rsquo;elle diffère d\u0026rsquo;une personne à une autre, ce n\u0026rsquo;est pas une réalité factuelle.\nSi je vous disais que j\u0026rsquo;ai mangé 1kg de chocolat blanc, peut être me croiriez vous parce que vous l\u0026rsquo;avez aussi fait une fois, et peut être quelqu\u0026rsquo;un d\u0026rsquo;autre ne me croirait pas parce que ça semble beaucoup. Dans la réalité objective, je l\u0026rsquo;ai fait ou non, mais il y a deux réalités subjectives pour deux personnes différentes: une où je l\u0026rsquo;ai fait et une ou je ne l\u0026rsquo;ai pas fait. Dans chacune de ces réalités existe une version différente de moi, une qui dit la vérité et une qui ment, ce qui entraîne un traitement différent de l\u0026rsquo;information quand elle provient de moi et selon qui la reçoit. Des personnes peuvent vivre dans la même réalité objective et la voir de façon conflictuelle.\nÇa ne veut pas dire que tout le monde a une réalité radicalement différente, nous vivons dans LA réalité objective où les lois de la physique s\u0026rsquo;appliquent, nous recevons les mêmes signaux. Si l\u0026rsquo;on regarde la même série, nous verrons les mêmes images et nous entendrons les mêmes dialogues, nous aurons peut être des différences parce que l\u0026rsquo;on prête plus ou moins attention aux mêmes détails, que les signaux auront été traduits un peu différemment ou même qu\u0026rsquo;une partie des signaux sera inaccessible à l\u0026rsquo;un ou l\u0026rsquo;autre, mais nous aurons à peu près la même expérience. Les divergences apparaissent lors qu\u0026rsquo;il manque de l\u0026rsquo;information et que l\u0026rsquo;on est forcé de remplir les trous avec des interprétations, des suppositions et des extrapolations, plutôt qu\u0026rsquo;avec nos perceptions.\nAu long de cet article, lorsque j\u0026rsquo;utilise le terme réalité sans plus de précisions, alors c\u0026rsquo;est de la réalité subjective dont je parle car c\u0026rsquo;est d\u0026rsquo;elle dont nous parlons en général.\nLes réalités alternatives # Bien que les réalités peuvent varier d\u0026rsquo;une personne à une autre, on peut avoir des attentes raisonnables concernant les réalités des autres parce que nous partageons la réalité objective et sommes soumis à ses règles.\nSi une autre personne et vous me regardiez jeter une pierre, je peux m\u0026rsquo;attendre à ce que vous ayez tous les deux vu la pierre tomber et non disparaître en l\u0026rsquo;air, flotter au dessus de ma tête ou décoller vers la Lune. Peut être n\u0026rsquo;avez-vous pas vu tous les détails, peut être vous pensez m\u0026rsquo;avoir vu jeter une canette et l\u0026rsquo;autre personne m\u0026rsquo;a vu jeter une pièce. Vous pouvez être en désaccord sur les détails mais, si vous avez observé le même évènement, vous devriez avoir vu plus ou moins la même chose car vous ne faites qu\u0026rsquo;interpréter, et non transformer, la réalité objective. Si ce n\u0026rsquo;était pas le cas, l\u0026rsquo;industrie du pari s\u0026rsquo;effondrerait.\nUne réalité alternative est une réalité qui est incompatible avec la réalité objective et, par extension, la réalité de la plupart des personnes. Si quelqu\u0026rsquo;un perçoit réellement des images d\u0026rsquo;une pierre en train de léviter lors de mon jet, alors sa réalité serait incompatible avec la réalité objective car elle défierait les lois de la physique, mais aussi avec celle des autres personnes qui ont tous vu quelque chose tomber.\nLes réalités alternatives ont leurs origines dans les autres dimensions de la physique quan\u0026hellip; euh, non, ça c\u0026rsquo;est le pilote de Sliders.\nUne réalité alternative c\u0026rsquo;est si quelqu\u0026rsquo;un me dit qu\u0026rsquo;elle est Cléopatre, souveraine du Royaume d\u0026rsquo;Égypte, et que nous sommes actuellement assis au bord du Nile en l\u0026rsquo;an -40 avant JC (passons sur l\u0026rsquo;anachronisme). On peut être en désaccord sur le fait qu\u0026rsquo;elle soit Cléopatre, souveraine du Royaume d\u0026rsquo;Égypte, mais il y a de multiples incompatibilités entre nos réalités qui les rendent impossible à coéxister dans la réalité objective: soit nous sommes en -40, soit en 2021, ça ne peut pas être les deux; soit nous sommes en France, soit en Égypte, ça ne peut pas être les deux. Au moins l\u0026rsquo;une de nos réalité est incompatible avec la réalité objective et est, de facto, une réalité alternative. Pour chacun de nous, notre réalité subjective est celle qui est construite sur la réalité objective, c\u0026rsquo;est l\u0026rsquo;autre qui vit une réalité alternative.\nÇa peut aussi être plus subtile que la caricature du dessus. Les conspirationnistes, tels que les Platistes ou une partie des anti-vax, ont des réalités compatibles avec la réalité objective\u0026hellip; sauf dans les parties qui tournent autour de la conspiration. C\u0026rsquo;est seulement lorsque ces sujets sont évoqués que les conflits et incompatibilités émergent, les contraignant à travailler dur pour préserver leurs réalités et empêcher qu\u0026rsquo;elles ne s\u0026rsquo;effondrent.\nLes réalités alternatives ne sont pas que l\u0026rsquo;expérience d\u0026rsquo;une pathologie, nous en faisons tous l\u0026rsquo;expérience fréquemment.\nTous les jours, nous allons nous coucher et notre personnage onirique plonge dans des réalités alternatives rêvées, qui n\u0026rsquo;ont aucun lien avec la réalité objective. Elles peuvent être très similaires ou différentes, réalistes ou étranges, mais du point de vue de notre esprit, ce sont des réalités alternatives qui remplacent notre réalité jusqu\u0026rsquo;à ce que l\u0026rsquo;on soit de nouveau conscient lors de notre réveil.\nC\u0026rsquo;est également ce qui arrive aux personnes en état de transe somnambulique en hypnose, qui hallucinent des objets et se comportent comme s\u0026rsquo;ils étaient quelqu\u0026rsquo;un d\u0026rsquo;autre. Ils sont entraînés à remplacer leur réalité par une réalité alternative, similaire à un monde onirique, jusqu\u0026rsquo;à être réveillés de l\u0026rsquo;état d\u0026rsquo;hypnose. Du point de vue de leur esprit, ils sont dans une réalité alternative le temps de l\u0026rsquo;expérience.\nCe qui différencie les rêveurs et les hypnotisés de la pathologie est que leur réalité alternative ne persiste pas. Lorsque l\u0026rsquo;on se réveille de nos rêves ou de nos états d\u0026rsquo;hypnose, on reconnaît la réalité alternative comme telle et on se reconnecte à notre réalité, elle ne persiste pas en tant que réalité subjective.\nL\u0026rsquo;état de conscience ordinaire (ECO) # Tous les jours, nous nous réveillons et commençons à reprendre contact avec notre réalité.\nEn nous y reconnectant, nous entrons dans notre état de conscience ordinaire (ECO), un état dans lequel nous sommes conscients de l\u0026rsquo;environnement et capables d\u0026rsquo;interagir avec lui de manière compréhensible. L\u0026rsquo;ECO est un état dans lequel on se sent\u0026hellip; normal, ou du moins familier. C\u0026rsquo;est l\u0026rsquo;état dans lequel nous fonctionnons le plus souvent pour interagir avec l\u0026rsquo;environnement, celui dans lequel nous passons le plus de temps lors que nous sommes éveillés.\nCertaines personnes n\u0026rsquo;aiment pas cette notion d\u0026rsquo;ECO qui semble impliquer que nous serions la même personne tous les jours, mais c\u0026rsquo;est une façon inexacte de comprendre les choses. l\u0026rsquo;ECO n\u0026rsquo;est pas un état statique, c\u0026rsquo;est un état dynamique qui englobe plusieurs niveaux de conscience depuis la fatigue jusqu\u0026rsquo;à une grande activité, c\u0026rsquo;est juste un état dans lequel notre conscience de l\u0026rsquo;environnement prends la plupart de notre espace mental. On est éveillé, conscient et alerte.\nUn ECO implique qu\u0026rsquo;il y a au moins un état non ordinaire de conscience\u0026hellip; mais il y a en réalité plusieurs états de conscience modifiée.\nLes états de conscience modifiée (ECM) # Lors que l\u0026rsquo;on mentionne les états de conscience modifiée (ECM), de nombreuses personnes pensent que l\u0026rsquo;on bascule dans le domaine des théories hippies new-age, mais nous entrons tous dans des ECM observables et reconnaissables tous les jours.\nUn ECM est simplement un état qui n\u0026rsquo;est pas notre ECO, rien de plus et rien de moins: pas de chakras, de forces énergétiques, rien d\u0026rsquo;autre.\nCertains de ces états sont déclenchés volontairement par nos actions, comme se saouler en absorbant de l\u0026rsquo;alcool ou se défoncer en consommant des drogues. L\u0026rsquo;expression \u0026ldquo;sous influence\u0026rdquo; implique elle-même que notre état de conscience est altéré, les actions faites \u0026ldquo;sous influence\u0026rdquo; ne sont pas celles de quelqu\u0026rsquo;un en ECO.\nLes ECM peuvent aussi être déclenchés par des évènements qui provoquent une courte sortie de notre normalité, comme par exemple lorsque l\u0026rsquo;on entre en état de choc après avoir été impliqué dans un accident, ou que l\u0026rsquo;on est euphorique après avoir accompli quelque chose d\u0026rsquo;extraordinaire. Ces évènements nous font temporairement basculer dans un ECM plus ou moins éloigné de notre ECO.\nPlus fréquent que d\u0026rsquo;être saoul ou dans un état de choc, le rythme circadien de la vigilance déclenche un ECM en moyenne toutes les 90 minutes, provoquant une légère perte de conscience de l\u0026rsquo;environnement et une absorption dans les pensées, alors que le rythme circadien de la veille-sommeil déclenche plusieurs ECM, environ toutes les 24 heures, quand nous nous endormons.\nLa plupart de ces états sont mesurables et observables, soit par des mesures EEG, soit par des changements physiologiques visibles. Par exemple, personne ne peut nier que les ECM liés au cycle du sommeil sont similaires à l\u0026rsquo;ECO, ils présentent des différences aussi bien dans les mesures EEG que dans les changements physiologiques visibles.\nLa personnalité principale # Depuis notre plus jeune âge, nous touchons à tout et observons les conséquences de nos actions, on nous dit de faire des choses et de ne pas en faire d\u0026rsquo;autres, on obéit et on désobéit, on réussit, on rate, on se fait mal, on aime, on déteste, on ressent des choses plaisantes et déplaisantes, et tout ça affecte la manière dont on fait face à des situations similaires et celle dont on approche les nouvelles.\nCes expériences sont internalisées et deviennent partie de notre identité, on construit une personnalité dont les traits en sont hérités, une personnalité qui est celle associée à notre ECO, notre personnalité principale. Il s\u0026rsquo;agit de celle qui nous décrit, et même si l\u0026rsquo;on feint certains de ses aspects lorsque l\u0026rsquo;on fait face à d\u0026rsquo;autres personnes, cette feinte fait partie de ses traits: on sait lorsque l\u0026rsquo;on fait semblant d\u0026rsquo;être quelqu\u0026rsquo;un d\u0026rsquo;autre.\nCette personnalité principale est ce qui nous caractérise, ce qui nous décrit en tant que personne si l\u0026rsquo;on exclut les traits physiques. On peut être timides ou extravertis, joyeux, pessimistes, aventuriers ou anxieux. Elle dirige nos décisions et nos actions, et les personnes qui la connaissent peuvent inférer certains de nos comportements de par leur alignement avec notre personnalité. Si vous connaissez ma personnalité et reconnaissez une angoisse générale, vous inférerez probablement qu\u0026rsquo;il y a peu de chances que je prenne une décision impulsive sans la réfléchir.\nCette personnalité principale est ce qui rend notre réalité subjective: les interprétations de notre réalité sont influencées par notre personnalité. C\u0026rsquo;est parce qu\u0026rsquo;elle a certains traits que l\u0026rsquo;on interprète les évènements d\u0026rsquo;une certaine façon, que l\u0026rsquo;on fait peu attention à certains détails et beaucoup plus à d\u0026rsquo;autres.\nLes personnalités alternatives (ou secondaires) # Parmi toutes les expériences de la réalité, certaines sont intégrées dans notre personnalité principale et d\u0026rsquo;autres ne le sont pas. Des fois, elles ne sont pas importantes pour nous ou nous n\u0026rsquo;en n\u0026rsquo;avons pas conscience, ou des fois elles sont incompatibles avec notre réalité et risqueraient sa stabilité, alors elles sont niées et refoulées consciemment ou non. Dans tous les cas, nous en faisons l\u0026rsquo;expérience, accumulons et enregistrons des informations, mais nous ne les intégrons pas dans notre personnalité principale.\nDans les ECM, lorsque l\u0026rsquo;on rêve ou que l\u0026rsquo;on est sous hypnose, des personnalités alternatives inconscientes peuvent être créées et prendre en compte ces informations. Elles ne mettent pas en péril notre réalité ou notre personnalité principale, nous sommes temporairement dans une réalité alternative, un endroit sûr où le pire qui puisse arriver est que l\u0026rsquo;on se réveille en réalisant que ce que l\u0026rsquo;on vient de vivre n\u0026rsquo;était pas réel.\nNotre inconscient peut se servir de ces personnalités pour organiser, classifier ou même supporter des informations qui ne peuvent pas être intégrées à notre personnalité principale. Cependant, parce qu\u0026rsquo;elles ont accès à des informations inhabituelles, les personnalités alternatives peuvent avoir des traits et des comportements amusants, étranges ou erratiques\u0026hellip; des fois tout à fait plausibles mais souvent très différents de notre personnalité principale. Le personnage qui nous incarne dans les rêves ou dans notre corps hypnotisé n\u0026rsquo;est pas notre personnalité principale, c\u0026rsquo;est pour ça que lorsque l\u0026rsquo;on regarde comment il s\u0026rsquo;est comporté, c\u0026rsquo;est comme s\u0026rsquo;il avait été contrôlé par quelqu\u0026rsquo;un d\u0026rsquo;autre et que nous étions spectateurs.\nÀ moins de souffrir d\u0026rsquo;un trouble de la personnalité, ces personnalités alternative subconscientes ne partagent généralement pas l\u0026rsquo;existence de notre personnalité principale. Elles apparaissent dans un rêve ou sous hypnose, lorsque l\u0026rsquo;on est dans un ECM et que notre personnalité principale s\u0026rsquo;en va avec notre conscience, puis disparaissent lorsque l\u0026rsquo;on se réveille et que l\u0026rsquo;on entre de nouveau dans notre ECO avec notre personnalité principale.\nLes phases du sommeil # De tous les ECM, les phases du cycle du sommeil sont particulièrement intéressants parce que vécus par tous, tous les jours, et donc nous pouvons tous nous y reconnaître. Personne ne peut douter que l\u0026rsquo;on s\u0026rsquo;endort, ou qu\u0026rsquo;une fois endormi on ne se trouve plus dans le même état que lorsque l\u0026rsquo;on est éveillé. C\u0026rsquo;est un ensemble d\u0026rsquo;ECM que tout le monde partage, indiscutablement.\nJe ne vais pas m\u0026rsquo;étendre en profondeur sur les phases du sommeil, elles sont déjà très largement documentées, mais je vais expliquer brièvement leur relation aux ECM.\nL\u0026rsquo;endormissement # Tout d\u0026rsquo;abord, le rythme circadien de la veille-sommeil nous envoie le signal que l\u0026rsquo;on a besoin de dormir environ toutes les 24h.\nLors de l\u0026rsquo;endormissement, on perds progressivement conscience de notre environnement et on débute une transition vers un ECM. A ce stade, nous ne sommes pas complètement détachés de l\u0026rsquo;environnement, nous perdons juste connaissance.\nC\u0026rsquo;est un ECM parce que nous perdons conscience de l\u0026rsquo;environnement, et que cette conscience est précisément ce qui définit un ECO.\nSommeil lent léger # Ensuite, durant la phase de sommeil lent léger, on bascule vers un ECM différent dans lequel nous avons perdu connaissance et sommes détachés de l\u0026rsquo;environnement, mais toujours sujets à certaines perceptions qui peuvent facilement nous ramener à l\u0026rsquo;éveil, comme le bruit, les changements lumineux ou quelqu\u0026rsquo;un qui nous touche.\nC\u0026rsquo;est un ECM différent parce qu\u0026rsquo;il y a des changements visibles de la physiologie, et que l\u0026rsquo;on peut mesurer des ondes cérébrales spécifiques à cette phase.\nSommeil lent profond # Plus tard, durant la phase de sommeil lent profond, on bascule de nouveau vers un ECM différent dans lequel nous avons non seulement perdu connaissance et sommes détachés de l\u0026rsquo;environnement, mais également de la plupart des perceptions, ce qui demande des signaux insistants de l\u0026rsquo;environnement pour nous y ramener.\nC\u0026rsquo;est là encore un ECM différent avec ses propres changements physiologiques, et ses ondes cérébrales spécifiques.\nSommeil paradoxal ou sommeil REM (rapid eye-movements) # Enfin, durant la phase de sommeil paradoxal, on bascule vers un nouvel ECM dans lequel nous sommes inconscients et détachés de l\u0026rsquo;environnement, comme dans une phase de sommeil lent profond, mais où une version alternative de nous est projetée dans la réalité alternative d\u0026rsquo;un monde onirique.\nCette phase est qualifiée de paradoxale parce qu\u0026rsquo;elle possède des caractéristiques d\u0026rsquo;un état d\u0026rsquo;éveil superposées à celles d\u0026rsquo;un état de sommeil. Les ondes cérébrales montrent l\u0026rsquo;activité d\u0026rsquo;un cerveau à l\u0026rsquo;éveil, et s\u0026rsquo;il n\u0026rsquo;y avait pas de paralysie musculaire, notre corps mimerait les actions du rêve comme c\u0026rsquo;est observé chez les personnes souffrant de troubles du sommeil affectant la paralysie.\nNous sommes en apparence éveillés et endormis simultanément.\nPourquoi les phases du sommeil sont intéressantes ? # D\u0026rsquo;abord, la phase de sommeil paradoxal est quelque chose dont nous faisons tous l\u0026rsquo;expérience, et elle expose toutes les notions présentées dans cet article. Durant cette phase, nous faisons l\u0026rsquo;expérience d\u0026rsquo;un ECM qui remplace notre ECO, de réalités alternatives qui remplacent notre réalité, et de personnalités alternatives qui remplacent notre personnalité principale.\nApprocher la compréhension de notre esprit par le sommeil paradoxal est le chemin de la moindre résistance, les concepts peuvent être expliqués sans se baser sur des théories difficiles à comprendre mais en pointant du doigt ce que les gens connaissent déjà, tout le monde sait comment fonctionne un rêve même s\u0026rsquo;ils ne se rappellent pas des leurs.\nEnsuite, ils partagent ÉNORMÉMENT de similarités avec les états hypnotiques que je décris dans la section suivante, tellement en fait que l\u0026rsquo;on pourrait dire que ce sont essentiellement les mêmes états déclenchés par des chemins différents.\nC\u0026rsquo;est important parce que la porte la plus efficace vers le subconscient est l\u0026rsquo;hypnose, et lorsque l\u0026rsquo;on comprend qu\u0026rsquo;elle fonctionne au travers de mécanismes similaires au sommeil, on comprend également que ne pas être réceptif à l\u0026rsquo;hypnose est aussi dénué de sens que de ne pas être réceptif au sommeil. La résistance à l\u0026rsquo;hypnose n\u0026rsquo;est pas une question de réceptivité mais trouve sa cause ailleurs.\nLes états hypnotiques (transes) # Les états hypnotiques sont des ECM qui sont entre l\u0026rsquo;ECO, lorsque l\u0026rsquo;on est conscient, et l\u0026rsquo;état de sommeil, lorsque l\u0026rsquo;on a perdu connaissance. Ils impliquent que notre niveau de conscience est réduit par rapport à notre ECO. Il y a trois catégories d\u0026rsquo;états hypnotiques, ou du moins trois qui soient significativement différents et observables, chacun ayant ses propres caractéristiques et ses changements physiologiques.\nLa transe commune de la vie quotidienne (ou rêverie) # Nous sommes dans une transe commune, ou rêverie, lorsque l\u0026rsquo;on se déconnecte temporairement de notre réalité pour entrer dans une réalité alternative intérieure. C\u0026rsquo;est ce qu\u0026rsquo;il se produit lorsque l\u0026rsquo;on réalise que l\u0026rsquo;on a tourné plusieurs pages d\u0026rsquo;un livre sans les lire\u0026hellip; parce que l\u0026rsquo;on était absorbés dans nos pensées sans lien avec notre environnement et ce que nous étions en train de faire. C\u0026rsquo;est une transe très légère qui survient naturellement de par le rythme circadien de la vigilance.\nLorsque l\u0026rsquo;on est en rêverie, nous sommes un peu plus suggestibles parce que nous continuons à percevoir les signaux de la réalité, mais notre conscience est ailleurs et ne fait pas son filtrage habituel. Ça ne veut pas dire que toutes les suggestions sont acceptées, mais qu\u0026rsquo;il y a moins de résistance à certaines suggestions qui seraient autrement rejetées. Elles peuvent être enregistrées et affecter nos décisions et pensées ultérieures.\nSi j\u0026rsquo;entends une publicité pour acheter des barres de chocolat pendant que je suis en rêverie, je suis plus suggestible à cette idée que si je prête attention et ai conscience que l\u0026rsquo;on tente de m\u0026rsquo;en faire acheter. Ce petit surcroît de suggestibilité peut jouer un rôle dans ma décision plus tard, et peut être me faire basculer vers l\u0026rsquo;achat si je n\u0026rsquo;étais pas opposé à l\u0026rsquo;idée, même si je n\u0026rsquo;en avait pas l\u0026rsquo;intention à l\u0026rsquo;origine.\nLa transe stuporeuse # Dans une transe stuporeuse, nous sommes progressivement déconnectés de la réalité, de manière très similaire aux phases de sommeil lent léger et profond. Nous passons par les même changements physiologiques, aussi bien au niveau de la respiration que du rythme cardiaque.\nDans une transe stuporeuse légère, nous sommes relaxés et nous faisons moins attention à l\u0026rsquo;environnement, nous détachant progressivement de la réalité mais restant capable de se reconnecter rapidement si les sens sont stimulés.\nAlors que la transe s\u0026rsquo;approfondit, nous nous détachons de plus en plus et perdons l\u0026rsquo;intérêt pour la réalité pour être absorbés par la réalité alternative intérieure. Cette perte d\u0026rsquo;intérêt inclue ce qui peut arriver au corps physique, on ne se soucie plus de l\u0026rsquo;inconfort, les signaux de douleur n\u0026rsquo;arrivent pas jusqu\u0026rsquo;à la réalité alternative intérieure, c\u0026rsquo;est pourquoi les techniques de gestions de la douleur emploient souvent une transe stuporeuse poussant à imaginer un lieu sûr où l\u0026rsquo;on se sentirait bien.\nLa transe peut s\u0026rsquo;approfondir jusqu\u0026rsquo;au sommeil ou coma hypnotique, un état dans lequel nous perdons temporairement tout intérêt pour la réalité, complètement absorbés dans notre réalité alternative intérieure.\nLorsque l\u0026rsquo;on lit ou entend parler de chirurgie sous hypnose, de dentistes utilisant l\u0026rsquo;hypnose, ou encore lorsque l\u0026rsquo;on voit un hypnotiseur endormir quelqu\u0026rsquo;un dans un spectacle, il s\u0026rsquo;agit de cette transe approfondie à différents degrés.\nLa transe somnambulique # La transe somnambulique ressemble au sommeil paradoxal. On y glisse vers une réalité alternative où des pieds peuvent être collés au sol d\u0026rsquo;un claquement de doigts, où des choses peuvent apparaître spontanément depuis nulle part, ou encore où notre inconscient peut être invoqué pour prendre le contrôle de notre corps comme s\u0026rsquo;il ne s\u0026rsquo;agissait plus du notre. C\u0026rsquo;est similaire à un rêve éveillé où les choses surviennent et on les accepte sans se poser de question, notre réalité alternative intérieure remplace simplement notre réalité.\nDans une transe somnambulique légère, la personnalité principale partage l\u0026rsquo;espace mental avec une personnalité alternative. Elle accepte qu\u0026rsquo;il peut y avoir deux personnalités dans le corps, elle-même et une personnalité alternative que l\u0026rsquo;on personnifie souvent en \u0026ldquo;l\u0026rsquo;inconscient\u0026rdquo;, et qu\u0026rsquo;elles peuvent toutes deux avoir un degré de contrôle.\nAlors que la transe somnambulique s\u0026rsquo;intensifie, la personnalité alternative prends de plus en plus de contrôle, jusqu\u0026rsquo;à avoir l\u0026rsquo;ascendant sur la personnalité principale, et finalement prendre le contrôle complet du corps en provoquant la disparition temporaire de la personnalité principale.\nC\u0026rsquo;est ce qui arrive lorsque l\u0026rsquo;on voit un hypnotiseur dire à quelqu\u0026rsquo;un de faire des choses surréalistes, que la personne s\u0026rsquo;exécute, avant de revenir à elle sans se souvenir de ce qu\u0026rsquo;elle a fait. La personnalité principale était ailleurs, une personnalité alternative inconsciente à fait les choses à sa place, et c\u0026rsquo;est pourquoi elle ne s\u0026rsquo;en rappelle pas.\nLe rêve # Le rêve est une fonction essentielle qui aide à la gestion des émotions et du stress, mais aussi à la mémorisation et la catégorisation des souvenirs, comme ce fut démontré lors d\u0026rsquo;expériences de privation du sommeil paradoxal. Tout le monde rêve, que l\u0026rsquo;on s\u0026rsquo;en rappelle ou non, parce que les rêves sont le produit de ces activités que nous traversons quotidiennement.\nFreud voyait le rêve comme une psychose sans danger, a harmless dream psychosis, qui nous soustrait du monde extérieur temporairement puis disparaît. Et si je ne suis pas un grand fan de tout son travail, c\u0026rsquo;est une façon très logique de comprendre les rêves à la lumière de ce que j\u0026rsquo;ai écrit:\nDurant le rêve, une personnalité alternative prends le contrôle de notre espace mental dans une réalité alternative qui n\u0026rsquo;a pas de lien avec la réalité objective, jusqu\u0026rsquo;à ce que le rêve se termine, que personnalité et réalité alternatives disparaissent, et que notre personnalité principale se reconnecte à notre réalité.\nBeaucoup de choses ont été dites au sujet des rêves, de leur sens et interprétations, mais je ne crois pas que l\u0026rsquo;on puisse les interpréter. Ils embarquent de l\u0026rsquo;information, mais elle est encodée d\u0026rsquo;une manière qui ne fait sens qu\u0026rsquo;à la personnalité alternative qui vit le rêve. L\u0026rsquo;information ne fait aucun sens pour notre personnalité principale ou celle d\u0026rsquo;autres personnes. C\u0026rsquo;est comme si quelqu\u0026rsquo;un nous parlait dans un nouveau langage inventé spécialement à chaque rêve, on peut tenter d\u0026rsquo;interprêter, on peut tenter de deviner et peut être même des fois avoir de la chance et tomber juste\u0026hellip; mais généralement on ne comprendra rien du sens parce qu\u0026rsquo;il n\u0026rsquo;est pas dans notre langage, et que l\u0026rsquo;on ne peut pas apprendre à comprendre quelque chose qui change radicalement à chaque fois.\nSous hypnose, les personnalités alternatives utilisent souvent des discours symboliques qui ne font absolument aucun sens pour nous, jusqu\u0026rsquo;à ce qu\u0026rsquo;elles expliquent comment ça fait sens pour elles\u0026hellip; de manière similaire à comment une personne psychotique peut avoir une explication très personnelle du monde qui l\u0026rsquo;entoure, que l\u0026rsquo;on ne peut pas comprendre avec notre logique, mais qui fait sens pour elle et qu\u0026rsquo;elle est capable d\u0026rsquo;expliquer.\nCe n\u0026rsquo;est pas différent pour les personnalités alternatives dans nos rêves.\nCe qu\u0026rsquo;il est important de bien comprendre est que les rêves, dont nous faisons tous l\u0026rsquo;expérience, englobent tous les concepts: un ECM qui diffère de notre ECO, une réalité alternative disjointe de notre réalité et de la réalité objective, des personnalités alternatives qui peuvent varier énormément de notre personnalité principale, jusqu\u0026rsquo;au point que l\u0026rsquo;on puisse considérer notre expérience du rêve comme celle d\u0026rsquo;une psychose sans danger.\nCette compréhension est à la base de la compréhension du rêve lucide et de l\u0026rsquo;hypnose, mais aussi de comment sont créés les traumas, ce qui cause l\u0026rsquo;hystérie, et comment les médicaments, le rêve lucide ou l\u0026rsquo;hypnose peuvent les affecter.\nLe rêve lucide # Le rêve lucide est un rêve un peu particulier, une sorte de hack si vous préférez, où notre personnalité principale est amenée dans la réalité alternative d\u0026rsquo;un rêve et lui fait partager l\u0026rsquo;espace mental avec les personnalités alternatives.\nTout le monde peut faire des rêves lucide mais ça demande de l\u0026rsquo;entraînement et du conditionnement, d\u0026rsquo;abord pour pouvoir se souvenir des rêves, et ensuite pour pouvoir mettre en place des déclencheurs qui provoqueront le réveil de la personnalité principale dans la réalité alternative. Certaines personnes le font très simplement, voire naturellement, d\u0026rsquo;autres auront besoin de longues périodes d\u0026rsquo;entraînement avant de réussir. Il n\u0026rsquo;y a pas de recette magique pour provoquer un rêve lucide, seulement des recettes pour augmenter les probabilités d\u0026rsquo;y arriver.\nQuand un rêve lucide est déclenché, notre personnalité principale est réveillée dans une réalité alternative construite par des personnalités secondaires\u0026hellip; mais elle est consciente que ce n\u0026rsquo;est pas sa réalité, et que le corps physique dort à l\u0026rsquo;extérieur du monde du rêve. Les personnalités alternatives tentent de préserver leur ascendant, et leur réalité, en tentant de faire perdre cette conscience par l\u0026rsquo;effondrement du rêve courant pour provoquer un réveil, ou en démarrant une nouvelle trame de rêve où nous ne sommes plus conscients. Il existe cependant des techniques simples pour empêcher ces tentatives de réussir.\nLorsque le rêve lucide est stabilisé, quelque chose de très intéressant survient: un canal de communication entre la personnalité principale et les personnalités alternatives se crée, alors que notre personnalité principale contrôle notre personnage dans le rêve mais que tous les autres personnages et éléments de décor sont des constructions subconscientes. Il devient possible de poser des questions aux personnages du rêve concernant nos comportements inconscients et d\u0026rsquo;obtenir des réponses désinhibées, même si elles entrent en conflit avec nos points de vue et notre réalité.\nDans notre ECO, on ne ressent jamais vraiment la dualité entre nos esprits conscient et inconscients, notre personnalité principale ne communique pas avec les personnalités alternatives. Dans le rêve lucide, cette dualité est omniprésente, on est conscient que ce n\u0026rsquo;est pas notre réalité et malgré notre degré de contrôle, la plupart des éléments échappent à notre contrôle et entrent en conflit avec nous: des personnages peuvent refuser de nous parler, nous contredire, ou même ramener des souvenirs qui entrent en conflit avec notre réalité, \u0026hellip;\nHypnose # L\u0026rsquo;hypnose est également une forme spéciale de rêve. Elle se déroule pendant notre éveil et des personnalités alternatives sont amenées dans notre réalité, provoquant leur partage de l\u0026rsquo;espace mental avec notre personnalité principale. C\u0026rsquo;est très similaire au rêve lucide, dans le sens où finalement notre réalité est remplacée par une réalité alternative où nous percevons des choses inexistantes pour les personnes autour, et où la personnalité principale et les personnalités alternatives sont présentes ensemble et capables de communiquer les unes avec les autres. La différence principale est que l\u0026rsquo;on développe cette réalité alternative depuis l\u0026rsquo;état d\u0026rsquo;éveil, et que les personnes extérieures, telles que l\u0026rsquo;hypnotiseur, peuvent interagir avec cette réalité alternative.\nLà aussi, l\u0026rsquo;hypnose est accessible à tous mais demande entraînement et adhésion au procédé.\nContrairement à une croyance populaire, il n\u0026rsquo;est pas nécessaire de croire à l\u0026rsquo;hypnose ou même croire qu\u0026rsquo;elle fonctionne sur nous, mais il faut vraiment vouloir qu\u0026rsquo;elle fonctionne pour ne pas (in)volontairement y résister: pour être hypnotisé, il faut accepter de temporairement abandonner notre réalité et notre personnalité principale. C\u0026rsquo;est très simple pour certains qui peuvent être hypnotisés en quelques secondes, très dur pour d\u0026rsquo;autres qui nécessitent des douzaines ou des centaines d\u0026rsquo;heures pour accepter cet abandon, et contrairement à ce que l\u0026rsquo;on pourrait penser, la résistance est généralement signe de faiblesses sous-jacentes, comme des traumas ou des peurs, plutôt que le signe d\u0026rsquo;un fort esprit \u0026ldquo;cartésien\u0026rdquo;.\nLes techniques en elles-mêmes sont simples à apprendre et lorsque l\u0026rsquo;on entre en transe somnambulique, il se passe la même chose que dans les rêves lucide: il devient possible pour notre personnalité principale de communiquer avec les personnalités alternatives. Il devient possible de poser des questions et comprendre ce qu\u0026rsquo;il se passe au niveau subconscient.\nL\u0026rsquo;hypnose est beaucoup plus efficace que le rêve lucide car durant le rêve, nous entrons dans la réalité alternative de nos personnalités alternatives et, si elles arrivent à nous en faire sortir, l\u0026rsquo;aventure s\u0026rsquo;arrête jusqu\u0026rsquo;à la prochaine tentative fructueuse de rêve lucide. Avec l\u0026rsquo;hypnose, nous faisons entrer les personnalités alternatives dans la réalité de notre personnalité principale, une réalité qu\u0026rsquo;elles ne peuvent pas interrompre. Si l\u0026rsquo;on sort de l\u0026rsquo;état d\u0026rsquo;hypnose pour une raison ou une autre, il est possible de le ré-induire autant de fois que souhaité.\nLa suite ? # Cet article visait à donner une vue macroscopique de concepts qui seront récurrents dans cette catégorie, un peu pour mettre tout le monde au même niveau de compréhension, les articles futurs pousseront ces notions plus loin.\nDans mon prochain article de cette catégorie, je présenterai un modèle sur lequel j\u0026rsquo;ai travaillé et qui décrit une organisation dynamique de l\u0026rsquo;appareil psychique, prenant en compte ses réorganisations dans les différents états de conscience. J\u0026rsquo;expliquerai comment je le construit, quelle est la logique et le raisonnement derrière chacun de ses composants, et je l\u0026rsquo;appliquerai à différents états de conscience pour comparer ce que l\u0026rsquo;on observe avec ce que le modèle prédit.\n","date":"17 May 2021","permalink":"/posts/2021-05-02/","section":"Posts","summary":"TL;DR: J\u0026rsquo;explique superficiellement comment fonctionne notre esprit, en partant du corps, puis de notre sens de la réalité, avant de finir par les états de conscience et personnalités.","title":"Mind hacking: comprendre comment fonctionne notre esprit"},{"content":" TL;DR: I explain very superficially how our minds work, starting from our body, following with our sense of reality, and ending with our states of consciousness and personalities. I dive briefly into dreams and hypnosis, paving the road for a future article. Disclaimer # I\u0026rsquo;ve started taking an interest in lucid dreaming around 2013, self-studying and training in various kinds of hypnosis from 2015 to 2020, and been working as a hypnotherapist since 2016 up to now. I started and am still studying work psychology since 2016, which had me take introductory classes in clinical and cognitive psychology.\nI\u0026rsquo;m not a psychologist, a psychiatrist or a cat, so take this article with a grain of salt.\nAbout this article # I attempt to explain, from a very macroscopic perspective, my understanding of how the mind works, based on my studies, observations or personal experiences through lucid dreaming and hypnosis sessions.\nThis article doesn\u0026rsquo;t contain revolutionizing new ideas, you may in fact recognize bits of work from Charcot, Janet, Breuer or even Freud, though these bits may not exactly reflect their work as I don\u0026rsquo;t necessarily share all their views, and adapted them to how I understand the mind works.\nThis is a vulgarization that takes voluntary shortcuts to convey its main ideas in their simplest form, it is by no mean an exhaustive explanation as each single concept could easily take pages of detailing. I\u0026rsquo;ll be more than happy to expand upon each of these concepts in the comments or as follow-up articles.\nBottom line: do not assume all of this article to apply as is to everyone, particularly to people with disorders that affect their sense of reality or personality/ies, this is not an exact depiction of how things work for everyone.\nWe are organic computers # We are organic computers resulting from the combined specifications of two other organic computers. The specifications determine how we\u0026rsquo;re built, how we look like, and some of our aptitudes and limitations, but not how we process the world, not how our operating system works.\nWe interface with the world through many sensors, called receptors, that are spread in our eyes, ears, tongue, nose and below our skin to capture visual, auditory, gustatory, olfactory and kinesthesic signals that are then interpreted and allow us to see, hear, taste, smell and feel elements of our environment. These receptors all compose our sensory apparatus, what we call our senses.\nThese receptors are part of what developers would consider an API. They are the only interfaces that can perceive input signals from the environment, allowing our operating system to translate them into sensations and encode memories. Without receptors, no signal would enter our system, and we\u0026rsquo;d live a very dull life of nothingness. Luckily for us, we have receptors in our eyes to see a white chocolate bar on the table, in our hands to feel when we grab it, in our nose and tongue to smell and taste it, and in our ears to hear someone ask where did the chocolate bar disappear. Oopsy.\nBecause our senses are our only input interfaces, everything that we ever know about our environment comes from them. Our knowledge didn\u0026rsquo;t just appear in us, it was inserted through our senses, either because another organic computer sent us a signal to share information, or because we interacted with our environment and received a signal as a feedback.\nMost signals are discarded unless we pay extra attention as they are irrelevant to our current activities, like the tick of a clock that we hear but forget as if it was no longer there\u0026hellip; even though it still really is. Many signals are processed at a low-level subconscious layer and recorded for future use, like the earlier thought of a white chocolate bar that maybe we kinda want now. A few signals are processed and made available to our immediate knowledge so that we can act upon them, such as this highlighted text which immediately catches our attention.\nThese receptors are what help us develop and customize our operating system from scratch. They make information available to us about our reality, which changes the way we process our environment and interpret new signals. Our operating system basically builds an understanding of the world through a sensory feedback loop.\nReality # It may seem strange to discuss the concept of reality in an article about how the mind works but it isn\u0026rsquo;t. We live in a reality and everything we do is relative to it: we come to life in it, learn its rules through trials and errors, then develop thoughts and behaviors that seem adapted to it. We can\u0026rsquo;t begin to comprehend how our mind works if we don\u0026rsquo;t understand how we perceive reality\u0026hellip; because how our mind works is derived from that perception.\nWhen we think of reality, we usually think of a unique single reality, THE reality, then we split the world in two: on a side are those that share the same reality as us and are considered sane, and on the other side are those that don\u0026rsquo;t and are considered insane. If someone mentions seeing flying pigs over our heads and we don\u0026rsquo;t see them when we look up, we recognize that their perception of reality is inaccurate because the flying pigs aren\u0026rsquo;t there. We expect people around us to experience reality as we do because, if there\u0026rsquo;s only THE reality, then there can\u0026rsquo;t be two contradictory events happening simultaneously.\nHowever, it\u0026rsquo;s not always that simple.\nWe all get different glimpses of reality and develop a very unique sense of it based on our own perceptions, even if we still share a large part of it with other living beings. We may live in the same reality, however mine is different from that of a color blind person whose visual signals are translated differently, and even more different from that of a blind person whose visual signals are lacking. We may live in the same reality but we experience a different reality\u0026hellip; because reality has multiple dimensions.\nThe objective reality # The reality within which we exist is the objective reality, one that is factual and bound by the laws of physics, one where things exist and happen even if we\u0026rsquo;re not aware of them. This is THE reality most people think they experience, a unique reality that is the same to all of us. It encompasses all that\u0026rsquo;s ever been, all that is and all that will ever be.\nAll that exists anywhere in the universe, from every star in every galaxy to every grain of sand on our planet, is part of the objective reality and exists regardless of our ability to interact with it or knowledge that they exist. All events that happen anywhere in the universe are part of the objective reality and exist regardless of our ability to observe them. They exist and happen factually, wether we know it or not.\nIn the objective reality, if I tell you that I once ate 1kg of white chocolate, it doesn\u0026rsquo;t matter that you believe me or not, there\u0026rsquo;s a factual truth: either it happened or it didn\u0026rsquo;t. If we disagree, then one of us is wrong. There\u0026rsquo;s no room for debate in this reality.\nThe signals that our senses perceive from the environment originate from the objective reality, they are factual and don\u0026rsquo;t carry subjective meaning. In the objective reality, white chocolate has a smell that can be perceived by our receptors, but that smell is neither pleasant or unpleasant, it just exists as a signal which may be translated and interpreted by someone.\nThe subjective reality # The objective reality is infinite and complex, and we are only exposed to an infinitesimal portion of it so we never experience it as it really is. Instead, we experience a dumbed down version that only contains what we were exposed to and what our senses can perceive out of it\u0026hellip; a version that\u0026rsquo;s different for everyone.\nThis is the subjective reality, an inner representation of the objective reality that\u0026rsquo;s affected by our perceptions, and which tries to fill the gap of missing informations by interpreting, guessing or extrapolating from past individual experiences. It is subjective because that reality differs from a person to another, it is not a factual reality.\nIf I told you that I ate 1kg of white chocolate, maybe you would believe me because you once did too or maybe someone else wouldn\u0026rsquo;t because that seems a lot. In the objective reality, either I did or I didn\u0026rsquo;t, but we have two subjective realities for two different persons: one where I did and one where I didn\u0026rsquo;t. Within each of these realities exists a different version of me, one that tells the truth and one that lies, which leads to a different treatment of information when it originates from me and depending on who receives it. People live in the same objective reality but can see it in conflicting ways.\nThis does not mean that everyone has a radically different reality, we still live in THE objective reality where the laws of physics apply and where we receive the same signals. If we watch the same show, we will see the same images and hear the same dialogues, we may have subtle differences because we pay more or less attention to details, but we should have about the same experience. The divergences appear where we lack information and are forced to fill the gaps with interpretations, guesses or extrapolations, rather than actual perceptions.\nFurther in this article, whenever I use the term reality without more precisions then it means subjective reality, as it is the one we tend to refer to when we talk about reality.\nAlternate realities # While realities can vary from a person to another, we can have reasonable expectations regarding realities of others because we all share the objective reality, we\u0026rsquo;re all bound to the laws of physics.\nIf you and another person watched me throw a rock, I can expect that you both saw it fall down and not disappear mid-air, float above my head or fly to the moon. Maybe you didn\u0026rsquo;t pay attention and missed details, maybe you think you saw me throw a can instead and the other person thinks I threw a coin. You may both disagree about details but, if you observed the same event, you should have seen more or less the same thing as you should only interpret, not transform, the objective reality. If that wasn\u0026rsquo;t the case, the gambling industry would fall apart.\nAn alternate reality is one that is incompatible with the objective reality and, by extension, the realities of most people. If someone really perceived images of the rock levitating when I threw it, then his reality would be incompatible with the objective reality by defying the laws of physics, and also with the realities of others as everyone else saw something falling.\nAlternate realities originate from other dimensions in quantum phy\u0026hellip; no, wait no, that\u0026rsquo;s the synopsis of Sliders.\nAn alternate reality is if someone tells me that she\u0026rsquo;s Cleopatra, ruler of the Kingdom of Egypt, and that we\u0026rsquo;re currently sitting by the Nile in 40 BC. We may disagree or not over the fact that she\u0026rsquo;s Cleopatra, ruler of the Kingdom of Egypt, but there are multiple incompatibilities between our realities that makes them impossible to coexist in the objective reality: either we\u0026rsquo;re in 40 BC or in 2021 AC but it can\u0026rsquo;t be both, either I\u0026rsquo;m in France or in Egypt but it can\u0026rsquo;t be both. At least one of our realities is incompatible with the objective reality and is de facto an alternate reality. To both of us, our subjective realities are the ones grounded in the objective reality, it is the other that is living in an alternate reality.\nThis can be much more subtle than the caricature above. Conspiracy theorists, such as flat earthers or some of the anti-vax crowd, have a most of their realities compatible with the objective reality\u0026hellip; except when it comes to parts revolving around the conspiracies. It is only when these topics are discussed that conflicts and incompatibilities come to light, in which case they must work very hard to preserve their alternate realities from shattering.\nAlternate realities are not only experienced through pathology, they are something we all experience frequently.\nEvery day, we go to sleep and our dream character dives into alternate dream realities, ones that have no grounds in the objective reality. They may be very similar or very different, realistic or strange but, as far as our mind is concerned, they are alternate realities that replace our reality until we wake up and are conscious again.\nThis is also what happens to people under sombnambulic states of hypnosis, hallucinating objects or acting as if they were someone else. They are instructed to switch their reality with an alternate one, similar to a dream world, until they are waken up from hypnosis. As far as their mind is concerned, they are in an alternate reality during that experience.\nWhat sets apart dreamers and hypnotized people from pathology is that their alternate realities don\u0026rsquo;t persist. When we wake up from our dreams or states of hypnosis, we recognize the alternate reality as being unreal and reconnect to our actual reality, the alternate reality doesn\u0026rsquo;t remain our subjective reality.\nThe ordinary state of consciousness (OSC) # Every day, we wake up and start getting back in touch with our reality.\nAs we connect back with it, we enter our ordinary state of consciousness (OSC), a state in which we\u0026rsquo;re aware of the environment and able to interact with it in ways we understand. The OSC is the state in which we feel\u0026hellip; normal, or at least familiar. It is the state in which we most usually function and interact with the environment, the one which we\u0026rsquo;re the most used to spend time in when we\u0026rsquo;re awake.\nSome people dislike this notion of OSC as it seems to imply that we\u0026rsquo;re the exact same person every day, but this is an inaccurate way of understanding things. The OSC is not a static state, it is a dynamic one that encompasses various levels of consciousness from being tired to being extremely active, it is just a state within which our conscience of our environment takes most of the mental space. We\u0026rsquo;re awake, aware and alert.\nAn OSC implies that there\u0026rsquo;s at least one non-ordinary state of consciousness, an understatement as there are really many altered states of consciousness.\nAltered states of consciousness (ASC) # When we mention altered states of consciousness (ASC), many people assume that we are in the realm of hippy new age theories, but we all enter observable and recognizable ASC on a daily basis.\nAn ASC is simply a state in which we\u0026rsquo;re not in our OSC, nothing more: no chakras, no energetic forces, no nothing.\nSome of these states are triggered voluntarily by our own actions, such as when we get drunk by absorbing alcohol or when we get high by consuming drugs. The term \u0026ldquo;under influence\u0026rdquo; itself implies that our state of consciousness was altered, that the actions that we did \u0026ldquo;under influence\u0026rdquo; were not that of someone under an OSC.\nASC can also be triggered by events causing us to shortly exit our normality, like for example when we enter a state of shock after being involved in an accident, or when we\u0026rsquo;re euphoric after achieving something extraordinary. These events cause us to temporily switch to an ASC that\u0026rsquo;s more or less distant from our OSC.\nMore frequently than getting drunk or in a state of shock, the circadian rythm of alertness causes us to enter an ASC on average every 90 minutes, lightly losing awareness of the environment and being absorbed in our thoughts, while the circadian rythm of sleep causes us to enter several ASC, roughly every ~24 hours, as we fall asleep.\nMost of these states are measurable or observable, either through EEG measurements or through visible physiological changes. No one could argue the ASC tied to the sleep cycle are similar to the OSC, they exhibit differences in both EEG measurements and visible changes.\nThe main personality # From early age, we start experimenting with everything and observe the outcome of our actions, we get told to do things and not do others, we abide or disobey, we succeed, fail, get hurt, love, hate, experience pleasant and unpleasant feelings, and all of this affects how we face similar situations or how we approach new ones.\nThese experiences are internalized and becomes part of our identity, we build a personality whose traits are inherited from all of them, a personality that is our natural one when we\u0026rsquo;re in our OSC, it is our main personality. It is the one that describes us, and even if we fake aspects of it when confronting other people, that faking is also one of its traits: we know when we\u0026rsquo;re pretending to be someone else.\nThis main personality is what characterizes us, what describes us as a person when excluding our physical traits. We may be shy or extravert, joyful, pessimistic, adventurous or anxious. It drives our decisions and actions, and people who know our personality can infer some behaviors based on how they align with it. If you know my personality and recognize that I\u0026rsquo;m generally anxious, you will probably infer that I\u0026rsquo;m unlikely to do something impulsive without thinking it through.\nThis main personality is what makes our reality subjective: the interpretations of our reality are shaped by our personality. It is because it has certain traits that we interpret things in a certain way, that we don\u0026rsquo;t pay attention to some details and extra attention to others.\nAlternate personalities # Among all our experiences of reality, some are integrated to our main personality and some aren\u0026rsquo;t. Sometimes, they aren\u0026rsquo;t relevant to us or we weren\u0026rsquo;t conscious of them, or sometimes they\u0026rsquo;re incompatible with our reality and would risk its stability, so we deny and repress them consciously or not. We still experience them, accumulate and record informations, we just don\u0026rsquo;t integrate that to our main personality.\nIn ASC, when we dream or when we\u0026rsquo;re under hypnosis, alternate subconscious personalities can be created that take these informations into account. They don\u0026rsquo;t risk shattering our reality or main personality, we\u0026rsquo;re temporarily in an alternate reality, a safe place where the worse that could happen is that we wake up realizing it wasn\u0026rsquo;t real.\nOur subconscious mind can make use of these personalities to organize, classify or even cope with informations that couldn\u0026rsquo;t be integrated in our main personality. However, because of this access to unusual informations, the alternate personalities exhibit all kinds of traits and behaviors that make them behave funny, strange or erratic\u0026hellip; ways that are not always implausible but that are often really different from our main personality. Our dream character or our hypnotized body is not directed by our main personality, so when we look back at how it behaved, it is as if it was controlled by someone else and we were spectators.\nUnless we suffer from a personality disorder, these alternate subconscious personalities don\u0026rsquo;t usually share existence with our main personality. They would instead appear in dreams or under hypnosis, when we\u0026rsquo;re in an ASC and our main personality goes away with our consciousness, and would disappear when we wake up and enter our OSC again with our main personality.\nStages of sleep # Out of all ASC, stages of sleep are particularly interesting because we all experience them daily, and we can all relate to them. No one doubts falling asleep or that when they\u0026rsquo;re asleep, they\u0026rsquo;re no longer in the same state as when they\u0026rsquo;re awake. It is a set of ASC that we undisputedly all share in common.\nI won\u0026rsquo;t deep-dive into the stages of sleep as they are already heavily documented, however I will lightly explain their relation to ASC.\nFalling asleep # First of all, the circadian rythm of sleep-wake sends us the signal that we need to sleep roughly every ~24h.\nAs we fall asleep, we progressively lose awareness of our environment and start transitioning to an ASC. At this stage, we\u0026rsquo;re not completely detached from our environment, we\u0026rsquo;re just losing consciousness.\nIt is an ASC because we lose awareness of the environment, which is what defines the OSC.\nLight sleep # Then, during the stage of light sleep, we switch to a different ASC within which we are unconscious and detached from the environment, but still subjected to perceptions that can easily bring us back, such as noise, light changes or even someone touching us.\nWe know it is a different ASC because there are visible changes to the physiology, and we can measure specific brain waves.\nDeep sleep # Later, during the stage of deep sleep, we switch to yet another ASC within which we are not only unconscious and detached from the environment, but also detached from most perceptions, requiring insisting signals to bring us back.\nWe know it is another ASC because, just like for light sleep, there are visible changes to the physiology, and we can measure specific brain waves.\nREM (rapid eye-movements) or paradoxical sleep # Finally, during the stage of REM sleep, we switch to another ASC within which we are not only unconscious and detached from the environment, similarly to the stage of deep, but an alternate version of us is projected in the alternate reality of a dream world.\nThe stage is called paradoxical because we exhibit characteristics from a waken state while being in a sleep state. Brain measurements show the activity of a waken state, and if it weren\u0026rsquo;t for muscle paralysis, our body would mimic our dream actions as can be observed on people with sleep disorders affecting paralysis.\nWe are seemingly both awake and asleep at the same time.\nWhy are these stages of sleep interesting ? # First of all, the paradoxical sleep state is something we all experience, and it is a showcase of all the notions discussed in this article. Within it, we experience ASC that replace our OSC, alternate realities that replace our realities, and alternate personalities that replace our main personalities.\nApproaching an understanding of how the mind works through paradoxical sleep state is the way of least resistance, the concepts can be explained without relying on theories by pointing to what people experience, everyone knows how dreams work even if they don\u0026rsquo;t often recall theirs.\nThen, they share a LOT of similarities with the hypnotic states that I\u0026rsquo;ll describe in the next section, so much actually that we could say they\u0026rsquo;re essentially the same states triggered through a different path.\nThis is important because the most efficient gate to the subconscious is hypnosis, and when we understand that it works through the same mechanisms as sleep, we also understand that not being receptive to hypnosis makes as little sense as not being receptive to sleep, and resistance is caused by other reasons.\nHypnotic states (trances) # Hypnotic states are ASC that are between the OSC, when we\u0026rsquo;re conscious, and the the state of sleep, when we\u0026rsquo;re unconscious. They imply that our level of consciousness is reduced in comparison to our OSC. There are three categories of hypnotic states, or at least three that are significantly different and observable, each with their own characteristics and physiological changes.\nThe ordinary trance, or daydreaming # We\u0026rsquo;re in an ordinary trance, or daydreaming, when we temporarily disconnect from our reality to enter an inner alternate reality. This is what happens when we realize that we\u0026rsquo;ve flipped pages of a book without actually reading\u0026hellip; because we were absorbed on inner thoughts unrelated to our environment and what we were doing. This very light trance happens naturally as a result of the circadian rythm of alertness.\nWhen daydreaming, we are slightly more suggestible as we continue perceiving signals from reality, but our conscious mind is not around to do the filtering it usually does. This doesn\u0026rsquo;t mean that all suggestions are accepted, but that there\u0026rsquo;s least resistance to suggestions that would otherwise be reject right away. They may be recorded and affect later thoughts.\nIf I hear a commercial suggesting to buy snack bars while I\u0026rsquo;m daydreaming, I\u0026rsquo;m more suggestible to this idea than if I\u0026rsquo;m paying attention and careful not to be tricked into it. This slight suggestibility increase may play a role in my decision making later, getting me to buy one because I\u0026rsquo;m not opposed to the idea, even though I wasn\u0026rsquo;t initially planning to.\nThe stuporous trance # In the stuporous trance, we\u0026rsquo;re progressively disconnected from reality, very similarly to the light and deep sleep stages. We undergo similar physiological changes, including calmer respiratory and cardiac rythm changes.\nIn a light stuporous trance, we are relaxed and pay less attention to our environment, detaching ourselves from reality but still able to reconnect easily if our senses are stimulated.\nAs the trance intensifies, we detach more and more as we lose interest in reality to focus on an inner alternate reality. This loss of interest for reality includes what happens to the physical body, we don\u0026rsquo;t care if there\u0026rsquo;s discomfort, pain signals don\u0026rsquo;t make it to the inner alternate reality, which is why pain management techniques often uses a \u0026ldquo;safe place\u0026rdquo; stuporous trance.\nThe trance can be intensified until hypnotic sleep or hypnotic coma is reached, a state where we temporarily lose all interest for reality as we\u0026rsquo;re fully absorbed in our inner alternate reality.\nWhen we read about surgeries under hypnosis, or dentists using hypnosis, or even when we see a hypnotist put someone to sleep in a show, they all exploit this trance to varying degrees.\nThe somnambulic trance # The somnambulic trance resembles paradoxical sleep. We slide into an alternate reality where feet can be glued to the ground when fingers are snapped, where things can appear out of thin air, or even where our subconscious can be summoned to take over and control our body it as if it wasn\u0026rsquo;t ours. It is similar to an awaken dream where things happen and we don\u0026rsquo;t really see a reason why they wouldn\u0026rsquo;t, the inner alternate reality simply replaces our reality.\nIn a light somnambulic transe, the main personality shares the mental space with an alternate personality. It accepts that there are two personalities within the body, itself and the alternate personality which we often personify as the \u0026ldquo;subconscious mind\u0026rdquo;, and that they both have a degree of control.\nAs the somnambulic trance intensifies, the alternate personality takes more and more control, up to dominating the main personality, until it eventually takes over control of the entire body and causes the main personality to disappear.\nThis is what happens when you see a hypnotist tell people to do surreal things, which they do, before they wak up and don\u0026rsquo;t seem to remember what happened. Their main personality was away and an alternate subconcious personality did the things on their behalf, which is why they don\u0026rsquo;t remember.\nDreaming # Dreaming is an essential function as it helps emotions and stress, as well as the recording and categorization of memories, as has been shown in experiments involving paradoxical sleep deprivation. Everyone dreams, wether they recall it or not, because dreams are a byproduct of these activities we all go through daily.\nFreud viewed the dream as a harmless psychosis which withdraws you from the external world temporarily then disappears, and while I\u0026rsquo;m not a huge fan of all of his work, this is a very logical way to understand dreams in light of what I wrote above:\nDuring the dream, an alternate personality takes over your mental space in an alternate reality that has no grounds in the objective reality, until the dream ends and both the alternate personality and alternate reality disappear, as your main personality connects back to your reality.\nA lot of things have been said about dreams, their meanings and their interpretations, but I don\u0026rsquo;t think we can interpret them. They do carry information, but one that is encoded in a way that only makes sense to the current alternate personality experiencing it. The information makes no sense to our main personality or that of other people. It is as if someone talked to us in a newly invented different language every time we dream, we can try to interpret, we can even throw guesses at what they mean and sometimes be lucky and correct\u0026hellip; but we\u0026rsquo;ll generally have no clue because it\u0026rsquo;s not our language, and we can\u0026rsquo;t build an understanding of something that keeps changing in radical ways every time.\nUnder hypnosis, alternate personalities often rely on symbolic discourses that make absolutely no sense to us, until they explain how this makes sense to them\u0026hellip; similarly to how psychotic people have their own explanation to how their reality works, one that we don\u0026rsquo;t necessarily understand but which makes sense to them, and that they can explain.\nIt isn\u0026rsquo;t different with the alternate personalities of our dreams.\nWhat is important to understand is that the dreams, which we all experience, encompass all concepts: an ASC that differs from our OSC, an alternate reality that is disjoint from both our reality and the objective reality, and alternate personalities that may diverge greatly from our main personality, up to a point we could consider our experience of a dream to be the experience of a harmless psychosis.\nThis understanding lays the foundation to understanding how lucid dreaming or hypnosis work, as well as how traumas are created, how they cause hysteria, how medications affect them and how lucid dreaming and hypnosis affect them too.\nLucid dreaming # Lucid dreaming is a special kind of dream, a hack of some sort if you will, where our main personality gets pulled into the alternate reality of a dream and causes it to share the mental space with alternate personalities.\nIt is available to all but requires training and conditioning, first to be able to recall dreams, then to be able to setup triggers that will cause the main personality to emerge in the alternate reality. Some people do it very easily, others require extended periods of training before first succeeding, there is no magical recipe that makes it an instant win but only recipes to optimize the odds of it happening.\nWhen a lucid dream is triggered, our main personality is awaken in an alternate reality built by alternate personalities\u0026hellip; while being aware that it is not its own reality, and that the physical body lies sleeping outside of the dream world. The alternate personalities try to preserve their reality by tricking us into losing consciousness again, collapsing the dream to wake us up or even begin a completely new dream sequence where we\u0026rsquo;re not conscious anymore. However simple techniques exist that help undermine these attempts.\nWhen we succeed stabilizing a lucid dream, something very interesting happens: a communication channel between our main personality and alternate personalities emerges, as our main personality inhabits our dream character but all other characters are subconscious constructs. We become free to ask questions to surrounding characters about our subconscious behaviors and get unfiltered answers, even if they conflict with our views of the actual reality.\nIn our OSC, we never really feel the duality between our conscious and subconscious minds, our main personality never gets to communicate with our alternate personalities. In lucid dreams, this duality is omnipresent, we\u0026rsquo;re conscious that we\u0026rsquo;re not in our reality and while we have a degree of control over it, most elements of that reality are outside of our control and conflict with us: dream characters may refuse to talk to us, contradict what we say, or bring back memories that conflict with our own reality, \u0026hellip;\nHypnosis # Hypnosis is also a special kind of dream, one that happens while we\u0026rsquo;re awake and where alternate personalities are pulled into our reality, causing them to share mental space with our main personality. This is very similar to lucid dreaming, in that ultimately this turns our reality into an alternate one, where we experience and perceive things that are inexistant for other people, and where main and alternate personalities are both around and able to communicate one with the other. The difference is that we develop that alternate reality from a wake state, and that external people such as the hypnotist can interact with that alternate reality.\nIt is also available to all but requires training and adhesion to the process.\nContrarily to a popular belief, we don\u0026rsquo;t need to believe in it or believe that it works, but we need to truly want it to work so that we don\u0026rsquo;t (un)voluntarily resist it: to be hypnotized, we must accept to temporarily let go of our reality and main personality. This is very easy to some who get hypnotized in seconds, very hard to others who require dozens or hundreds of hours of work to accept letting go, and contrarily to what we could think, resistance is usually the sign of underlying weaknesses, such as traumas and fears, rather than the sign of a strong \u0026ldquo;cartesian\u0026rdquo; mind.\nThe techniques by themselves are easy to learn and when we enter the somnambulic state of hypnosis, the same happens as in lucid dreams: it becomes possible for our main personality to communicate with alternate personalities. It becomes possible to ask questions that helps understand what happens subconsciously.\nHypnosis is much more efficient than lucid dreaming because during a dream, we enter the alternate reality of alternate personalities and, if they manage to kick us out, the adventure ends until our next successful lucid dream attempt in a later dream or a later night. With hypnosis, we make alternate personalities join the reality of our main personality, one that they can\u0026rsquo;t end. If we somehow exit the state of hypnosis, we can reinduce it as many times as we want.\nWhat\u0026rsquo;s next ? # This article was meant to give a macroscopic view of concepts that will be recurrent in this category, as a way to put all readers on the same step, but future articles will take these notions further.\nIn my next article of this category, I will present a model I worked on to describe the dynamic organization of the mind, taking into account how it reorganizes itself in different states of consciousness. I will explain how it was built, what was the rationale behind each component of the model, and I will apply it to various states of consciousness to compare what we observe to what the model predicts.\n","date":"12 May 2021","permalink":"/posts/2021-05-01/","section":"Posts","summary":"TL;DR: I explain very superficially how our minds work, starting from our body, following with our sense of reality, and ending with our states of consciousness and personalities.","title":"Mind hacking: understanding how our mind works"},{"content":" TL;DR: I worked on OpenSMTPD-portable, did a lot of plakar, a lot of Go and gave a technical talk on hypnosis. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !)\nThis one caused me a copyright strike so I hope it was worth it :-)\nOpenSMTPD-portable # I did a bit of review and test for diffs sent to me, helped test a diff for a reproducible crash in the new libtls code on my machines, and shared some of my nooSMTPD diffs with eric@ so he could decide to reuse them or not in OpenSMTPD.\nWith OpenBSD 6.9 coming out, OpenSMTPD 6.9.0 was tagged and eric@ asked me if I could synchronize the OpenSMTPD portable repository so it matches OpenBSD. I spent a few hours bringing back every individual commit, fixing conflicts and ended up with a libtls-powered OpenSMTPD-portable which \u0026hellip; didn\u0026rsquo;t build anywhere because the world still uses OpenSSL.\nSince I had already dealt with this in nooSMTPD, I brought back my libtls compat layer so that on systems with OpenSSL the compat layer allows using the libtls interface on top of OpenSSL. This fixed the CI for Ubuntu and Fedora, unfortunately the CI for Alpine and ArchLinux uses LibreSSL and remain broken until the latest LibreSSL is packaged there.\nBecause the libtls conversion is such a major change, there will be a delay between the time OpenSMTPD is released for OpenBSD users and the time it is released for other systems: the change to the libtls interface has been heavily tested and I\u0026rsquo;ve been running it for a while now, but the portable adaptation of it has been virtually untested. I\u0026rsquo;ll send a mail this week-end to call for testing before we can tag a release.\nPlakar # I wrote about plakar last month and made a lot of progress since then.\nI\u0026rsquo;m not publishing the code yet as there are many things I want to finish first before the first people start commenting and bikeshedding.\nrestructured the project # I\u0026rsquo;m not too familiar with Go so I didn\u0026rsquo;t structure the project very nicely at first. I spent a few hours creating specific modules and reworking things so that they are properly split.\nI\u0026rsquo;m not done but it is starting to look less shameful :-)\nremoved namespaces # I initially thought namespaces within a plakar was a nice idea, but the more I played with them the more I realized it wasn\u0026rsquo;t and didn\u0026rsquo;t bring any benefit over creating multiple plakars. I decided to remove namespaces to simplify things.\nplakar-level encryption # I brought support for encryption and tried two different approaches: snapshot-level encryption and plakar-level encryption.\nWith the first approach, a plakar repository doesn\u0026rsquo;t care about encryption. It is the snapshots themselves that are encrypted on an individual basis and the same plakar can host both encrypted and cleartext snapshots.\nWith the second approach, a plakar repository is initialized as encrypted or cleartext. The snapshots are automatically encrypted if needed and the same plakar ony hosts all encrypted or all cleartext snapshots.\nI played with both but decided to go with the second approach because the first one came with additional unnecessary complexity.\nencryption macro-details # A user generates a P384 keypair as well as a random master key, the bundle is protected by a pbkdf2-derived passphrase.\nWhen pushing to an encrypted plakar, the chunks and objects are aes256-gcm encrypted using a subkey encrypted itself by the master key.\nThe snapshot index containing all the checksums for all objects and chunks is encrypted and signed.\nUpon restore, the index signature is verified and the chunks are decrypted.\nkeypair and master key generation # The P384 keypair and master key bundle is generated using the plakar keygen command:\n% plakar keygen passphrase: passphrase (confirm): keypair saved to local store % This results in the passphrase-encrypted bundle being saved to ~/.plakar/plakar.key.\nplakar initialization # To be able to create snapshots, an initialized plakar repository must be available. A local cleartext plakar is initialized by default in ~/.plakar so that the command will work out of the box for local snapshots.\nHowever it can also be initialized elsewhere, defaulting to an encrypted plakar:\n% plakar init /tmp/plakar passphrase: /tmp/plakar: store initialized % or can be made cleartext by passing the -cleartext option:\n% plakar init -cleartext /tmp/plakar.ct /tmp/plakar.ct: store initialized % Reworked the command line interface # Last month, to use an alternative plakar instead of ~/.plakar, it was necessary to use the -store option which I disliked\u0026hellip; so I reworked the command line to introduce a notion of direction.\nWhen using the default plakar, no change is required:\n% plakar push /private/etc % plakar ls 2021-04-30T22:35:35Z 702b5b48-15dc-41cf-bfc1-1c8b94d1e985 3.1 MB (files: 242, dirs: 41) % plakar pull 702b5b48 % But when using an alternate plakar, instead of providing the -store option it is now possible to push to a plakar:\n% plakar push /private/etc to /tmp/plakar.ct % .. and run commands from a plakar:\n% plakar ls from /tmp/plakar.ct 2021-04-30T22:30:51Z d499fd92-01cb-4eb3-bb60-b147417b68a1 3.1 MB (files: 242, dirs: 41) % plakar pull d499fd92 from /tmp/plakar.ct % All commands support the direction option.\nGenerate a tarball # I thought it would be nice if I could restore a snapshot or part of it into a tarball, as I often want to extract a bit of a snapshot on a machine to send elsewhere.\nI introduced a tarball command which allows generating a tarball:\n% plakar tarball d499fd92 \u0026gt; /tmp/d499df92.tar.gz It also supports partial restore:\n% plakar tarball d499fd92:/etc \u0026gt; /tmp/d499df92_etc.tar.gz plakar server and client # I implemented a proof of concept for a plakar server and client protocol, allowing the use of plakar over the network.\nOn one end, I initialize a plakar repository and run a server from it:\nnas% plakar init /tmp/plakar passphrase: /tmp/plakar: store initialized nas% plakar server 192.168.0.2:2222 from /tmp/plakar passphrase: On the other end, I simply push to a remote plakar:\nlaptop% plakar push /etc to plakar://192.168.0.2:2222 There is absolutely no difference or limitation from the client point of view, any command that works locally will work remotely just as well.\nIt is even possible to chain servers in order to proxify a plakar server:\nnas% plakar server 192.168.0.2:2222 from /tmp/plakar passphrase: gate% plakar server 192.168.0.1:2222 from plakar://192.168.0.2:2222 laptop% plakar push /etc to plakar://192.168.0.1:2222 Does it serve a purpose ? nope, it just works by accident.\nplakar ui # Finally, I implemented a basic web UI so that I could browse the snapshots easily.\n% plakar ui Launched UI on port 40717 This opens a web browser which lets me browse the snapshots as a filesystem:\nallows me to inspect individual files:\nget a preview:\nincluding of images, pdf, videos or sound:\nand even search for files matching a pattern in every snapshot:\nBecause the web UI is a plakar client that doesn\u0026rsquo;t do anything but plakar commands, it can be launched from any plakar store, cleartext or encrypted, local or remote.\nSome work in Go # I first wrote Go code with filter-rspamd and filter-senderscore two years ago, but never really dived into the language more than these two small projects because I still have some love-hate issues with some aspects of it.\nI decided this month to get more familiar with it and started looking at what it would take to write a tiny daemon, not just a program that runs an endless loop and does all work in the same process, but one that does things the OpenBSD style with privileges separation, message passing and fd passing.\nI was not disappointed: there\u0026rsquo;s not much out there to do that.\ngo-ipcmsg # The first thing that is missing is a package that provides something like the imsg(3) framework.\nFor those not familiar with it, the imsg(3) framework provides functions that allow two processes to exchange messages, including file descriptors, while guaranteeing that messages are always received whole.\nTypically, you will create a socketpair(2) before fork(2)-ing a process. The parent and the child will both use one end of the pair to communicate with each other, using the imsg(3) framework that will take care of buffering I/O and exposing full messages to receiving end.\nThere are a lot of gory details on how it achieves this and requires understanding iovec, the semantic of sendmsg(2) and recvmsg(2), how control messages and SCM_RIGHTS works, as well as some of the side effets of cmsgbuf alignments. I did a bit of work related to resources exhaustion there a few years ago and, while the interface is lovely, I can\u0026rsquo;t say I really missed diving in that code.\nWhen I figured that there was nothing similar in Go, I started reproducing the imsg(3) API but realized that while the API was fine in C, it could be made much simpler in Go using the language-provided channels and goroutines.\nSo I wrote a Channel struct which creates two channels, one for reads and one for writes, and associates them to a socketpair(2) end:\n// parent process main routine, forks a child then sets up an ipcmsg // Channel on the socketpair, returning read and write channels. The // channels can be used to emit messages to the child process. // func parent() { pid, fd := fork_child() child_r, child_w := ipcmsg.Channel(pid, fd) // send a message to child child_w \u0026lt;- ipcmsg.Message(42, []byte(\u0026#34;foobar\u0026#34;)) // read a message from child msg := \u0026lt;- child_r log.Printf(\u0026#34;Received message: %s\\n\u0026#34;, string(msg.Data)) } // child process main routine, sets up an ipcmsg Channel on fd 3, // the socketpair end inherited from parent, // read and write channels can be used to communicate with parent // process. // func child() { parent_r, parent_w := ipcmsg.Channel(os.Getppid(), 3) // receive a message from parent msg := \u0026lt;- parent_r log.Printf(\u0026#34;Received message: %s\\n\u0026#34;, string(msg.Data)) // write a message back parent_w \u0026lt;- ipcmsg.Message(42, []byte(\u0026#34;barbaz\u0026#34;)) } In practice, you wouldn\u0026rsquo;t just inline the calls but the parent and the child would call a dispatch function to handle messages and act upon them, something similar to below:\nconst ( IPCMSG_PING = 1 IPCMSG_PONG = 2 ) func dispatcher(r chan ipcmsg.IPCMessage, w chan ipcmsg.IPCMessage) { for msg := range r { switch msg.Hdr.Type { case IPCMSG_PING: log.Printf(\u0026#34;data: %s\\n\u0026#34;, string(msg.Data)) w \u0026lt;- ipcmsg.Message(IPCMSG_PONG, []byte(\u0026#34;barbaz\u0026#34;)) } } } func parent() { pid, fd := fork_child() child_r, child_w := ipcmsg.Channel(pid, fd) dispatcher(child_r, child_w) } The file descriptors passing is handled with the function MessageWithFd() which takes an additional parameter:\nfd, err := syscall.Open(os.Args[0], 0700, 0) if err != nil { log.Fatal(err) } w \u0026lt;- ipcmsg.MessageWithFd(IPCMSG_PING, []byte(\u0026#34;barbaz\u0026#34;), fd) The sending end will have the descriptor closed upon sending, the receiving end will be receive a message with a HasFd option set in its header and an open descriptor:\nmsg := \u0026lt;- r if msg.Hdr.HasFd == 1 { if msg.Fd == -1 { // expected a descriptor but got none, handle this } else { syscall.Close(msg.Fd) } } The package is already commited on Github, however it hasn\u0026rsquo;t been heavily tested and I would not recommend using it for anything serious before it has matured a bit. I would LOVE to receive testing feedbacks though !\ngo-privsep # The second thing missing is the ability to easily do privileges separation, that is creating multiple processes that are inherited from a parent process with different privileges, and have the ability to communicate one with another.\nOn OpenBSD, daemons follow the fork+reexec pattern which boils down to the following:\nThe daemon is started, it forks all of its child processes after setting up the plumbing for communicating with them, and each child process re-executes itself so that it benefits from ASLR and doesn\u0026rsquo;t retain the memory layout of the parent process.\nThis is an improvement over the more widely used fork pattern, where each child inherits from parent, because the reexec causes all inherited data to be lost. It prevents inheriting sensitive information by accident at the cost of forcing the parent to send back the necessary information to each process. And again, it makes all child processes benefit from ASLR too which is nice.\nThis pattern was popularized in OpenBSD a few years ago, long after many of the daemons were already in place, so each one did what it could to make it happen. There was no attempt at unifying this through a framework like was done for IPC and imsg(3).\nSince I had a good understanding of how to do that and no framework to inspire myself, I worked on a privsep package that would make it easier to write such daemons. The idea is that a daemon will describe the different processes that compose it, as well as the communication channels between them, and the privsep package will then handle all the plumbing so that processes are created according to expectations.\nHere is a simple example of how it works:\npackage main import ( \u0026#34;log\u0026#34; \u0026#34;github.com/poolpOrg/ipcmsg\u0026#34; \u0026#34;github.com/poolpOrg/privsep\u0026#34; ) const ( IPCMSG_PING = 1 IPCMSG_PONG = 2 ) func parent_main() { \u0026lt;-make(chan bool) // sleep forever } func foobar_main() { parent := privsep.GetParent() parent.Write(ipcmsg.Message(IPCMSG_PING, []byte(\u0026#34;abcdef\u0026#34;))) \u0026lt;-make(chan bool) } func parent_dispatcher(r chan ipcmsg.IPCMessage, w chan ipcmsg.IPCMessage) { for msg := range r { if msg.Hdr.Type == IPCMSG_PING { log.Printf(\u0026#34;[parent] received PING, sending PONG\\n\u0026#34;) w \u0026lt;- ipcmsg.Message(IPCMSG_PONG, []byte(\u0026#34;abcdef\u0026#34;)) } } } func foobar_dispatcher(r chan ipcmsg.IPCMessage, w chan ipcmsg.IPCMessage) { for msg := range r { if msg.Hdr.Type == IPCMSG_PONG { log.Printf(\u0026#34;[foobar] received PONG, sending PING\\n\u0026#34;) w \u0026lt;- ipcmsg.Message(IPCMSG_PING, []byte(\u0026#34;abcdef\u0026#34;)) } } } func main() { privsep.Init() parent := privsep.Parent(parent_main) foobar := privsep.Child(\u0026#34;foobar\u0026#34;, foobar_main) parent.Channel(foobar, parent_dispatcher) foobar.Channel(parent, foobar_dispatcher) privsep.Start() } As you can see, it uses the ipcmsg package to handle IPC between the two processes, allowing my pingpong daemon to play ping pong indefinitely. Each process can have more specific settings set so that it runs under a specific user, a specific group or from a chroot(2) jail.\nI still have a lot of work to do on this, but I commited the current state on Github anyways.\nsmall talk on hypnosis # This is not tech related but since my other main activity is hypnosis and hypnotherapy, I spent a bit of time this month working on a 3-hours talk I gave to a community of street hypnotists and hypnotherapists in Nantes.\nI\u0026rsquo;ve been working these last three months on a model to describe the organization of the psychic apparatus, based on what I learnt, experienced and observed these last six years from hundreds of hypnosis sessions, tons of psychology readings and many experiments in lucid dreaming.\nThe model describes how the psychic apparatus organization changes with the altered state of consciousness of a subject and what it means in terms of access to the subconscious and repressed psychic content.\nThis is not limited to hypnosis as it can be used to evaluate if a particular kind of therapy even makes sense, but applied to hypnosis it is particularly useful to understand resistance, what is happening in a subject when a technique is used, what is happening when a technique isn\u0026rsquo;t working, and how to bring the subject to the proper state of consciousness.\nI don\u0026rsquo;t think this post is the right place to dive into details and, well, the talk lasted 3 hours and it was very superficial so I will stop there. If people are interested in this topic, let me know and I\u0026rsquo;ll think of a way to present this as the slides I presented are not really informative without me talking over them.\nWhat\u0026rsquo;s next ? # Not much.\nMay will be a calm month for me here as the sponsoring has decreased and I need to do some freelance to buy myself spare time in June/July. Furthermore, I have had to deal with some personal issues in April and I\u0026rsquo;m not in the mood to do a lot of stuff at the moment.\nI\u0026rsquo;ll probably do a few things, as usual, but I\u0026rsquo;m not sure what at this point as it depends on how much time I\u0026rsquo;ll have available and how my mood evolves.\n","date":"30 April 2021","permalink":"/posts/2021-04-30/april-2021-opensmtpd-plakar-ipcmsg-privsep-and-a-small-hypnosis-talk/","section":"Posts","summary":"TL;DR: I worked on OpenSMTPD-portable, did a lot of plakar, a lot of Go and gave a technical talk on hypnosis. Let\u0026rsquo;s start with some LoFi # Relax.","title":"April 2021: OpenSMTPD, plakar, ipcmsg, privsep and a small hypnosis talk"},{"content":" TL;DR: I wrote a backup utility called plakar. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !)\nPlakar: Yet another backup utility ? # Yes, I wrote another backup utility.\nI backup many hosts on a daily basis and while there are many different tools, I always fallback to using scripts built on top of rsync to do incremental backups, which always results in customization for this or that host.\nI want backups that are easy to perform and restore, that do not require me to remember a ton of options, that lets me work with snapshots independant one of another instead of increments, that allow me to browse the content easily and compare snapshots without restoring them, that store data efficiently (compressed and deduped), and finally that lets me back things up locally or remotely to machines I own.\nI\u0026rsquo;ve looked into various solutions and I figured that the best option for me was to write a tool that did exactly what I wanted. Also, it\u0026rsquo;s fun to write code.\nWhat is plakar ? # The plakar utility is a tool that performs backups as snapshots of a tree structure, scanning subdirectories and files, and storing them efficiently in a repository while deduplicating redundant data. Once a snapshot has been taken of a directory, it can be inspected and restored at will.\nHow does it work ? # To perform a backup, you only need to call plakar with the push subcommand and a set of directories that should be part of the snapshot. This will cause plakar to scan the directories, collect inode information regarding every subdirectory and files, split files into chunks and push them to the repository.\n$ plakar push /bin 82711aee-6485-445b-98f4-6cae51c83035: OK $ The ls subcommand allows listing snapshots present in the repository:\n$ plakar ls 82711aee-6485-445b-98f4-6cae51c83035 [2021-03-27T22:05:00Z] (size: 10 MB, files: 36, dirs: 1) $ The pull subcommand allows restoring a particular snapshot:\n$ plakar pull 82711aee-6485-445b-98f4-6cae51c83035 82711aee-6485-445b-98f4-6cae51c83035: OK $ ls -l bin total 9336 -rwxr-xr-x 1 root wheel 121104 1 Jan 2020 [ -r-xr-xr-x 1 root wheel 1296640 1 Jan 2020 bash -rwxr-xr-x 1 root wheel 121968 1 Jan 2020 cat -rwxr-xr-x 1 root wheel 107552 1 Jan 2020 chmod -rwxr-xr-x 1 root wheel 123232 1 Jan 2020 cp -rwxr-xr-x 1 root wheel 1104640 1 Jan 2020 csh -rwxr-xr-x 1 root wheel 277408 1 Jan 2020 dash -rwxr-xr-x 1 root wheel 139264 1 Jan 2020 date -rwxr-xr-x 1 root wheel 122160 1 Jan 2020 dd -rwxr-xr-x 1 root wheel 121840 1 Jan 2020 df -rwxr-xr-x 1 root wheel 120848 1 Jan 2020 echo -rwxr-xr-x 1 root wheel 205648 1 Jan 2020 ed -rwxr-xr-x 1 root wheel 121504 1 Jan 2020 expr -rwxr-xr-x 1 root wheel 120864 1 Jan 2020 hostname -rwxr-xr-x 1 root wheel 121232 1 Jan 2020 kill -r-xr-xr-x 1 root wheel 2552352 1 Jan 2020 ksh -rwxr-xr-x 1 root wheel 329344 1 Jan 2020 launchctl -rwxr-xr-x 1 root wheel 105136 1 Jan 2020 link -rwxr-xr-x 1 root wheel 105136 1 Jan 2020 ln -rwxr-xr-x 1 root wheel 157360 1 Jan 2020 ls -rwxr-xr-x 1 root wheel 104752 1 Jan 2020 mkdir -rwxr-xr-x 1 root wheel 106176 1 Jan 2020 mv -rwxr-xr-x 1 root wheel 291152 1 Jan 2020 pax -rwsr-xr-x 1 root wheel 173568 1 Jan 2020 ps -rwxr-xr-x 1 root wheel 120832 1 Jan 2020 pwd -rwxr-xr-x 1 root wheel 106000 1 Jan 2020 rm -rwxr-xr-x 1 root wheel 104368 1 Jan 2020 rmdir -rwxr-xr-x 1 root wheel 120912 1 Jan 2020 sh -rwxr-xr-x 1 root wheel 120784 1 Jan 2020 sleep -rwxr-xr-x 1 root wheel 138656 1 Jan 2020 stty -rwxr-xr-x 1 root wheel 120464 1 Jan 2020 sync -rwxr-xr-x 1 root wheel 1104640 1 Jan 2020 tcsh -rwxr-xr-x 1 root wheel 121104 1 Jan 2020 test -rwxr-xr-x 1 root wheel 106000 1 Jan 2020 unlink -rwxr-xr-x 1 root wheel 120752 1 Jan 2020 wait4path -rwxr-xr-x 1 root wheel 1331248 1 Jan 2020 zsh $ Compression and deduplication # When plakar performs a snapshot, it splits every file into content-defined chunks and checks if they are available in the repository before pushing them, in a way similar to what rsync does.\nEach chunk is compressed before being written to the repository, so that between compression and deduplication, a saved snapshot may take less space than the original directory:\n$ du -sh /bin 4.6M /bin $ du -sh ~/.plakar 3.8M /Users/gilles/.plakar Furthermore, since this deduplication takes place globally in the repository, pushing multiple times the same content doesn\u0026rsquo;t cause the plakar to grow much as it only grows by the size of the snapshot index:\n$ plakar push /bin f2645591-4b06-4512-89e2-dae8ad1e2360: OK $ plakar push /bin c0aa514b-452f-4122-a53f-99e984cb0548: OK $ plakar push /bin 3630358c-c634-4ad9-960d-f864d1eb56f5: OK $ plakar push /bin e521e447-5c2a-429f-b637-41561daa3826: OK $ plakar push /bin a83fe7bb-341f-4cf1-bad6-c2b1933422f8: OK $ plakar ls 82711aee-6485-445b-98f4-6cae51c83035 [2021-03-27T22:05:00Z] (size: 10 MB, files: 36, dirs: 1) f2645591-4b06-4512-89e2-dae8ad1e2360 [2021-03-27T22:06:17Z] (size: 10 MB, files: 36, dirs: 1) c0aa514b-452f-4122-a53f-99e984cb0548 [2021-03-27T22:06:17Z] (size: 10 MB, files: 36, dirs: 1) 3630358c-c634-4ad9-960d-f864d1eb56f5 [2021-03-27T22:06:18Z] (size: 10 MB, files: 36, dirs: 1) e521e447-5c2a-429f-b637-41561daa3826 [2021-03-27T22:06:18Z] (size: 10 MB, files: 36, dirs: 1) a83fe7bb-341f-4cf1-bad6-c2b1933422f8 [2021-03-27T22:06:19Z] (size: 10 MB, files: 36, dirs: 1) $ du -sh ~/.plakar 3.8M /Users/gilles/.plakar Snapshots UUID prefix-based lookup # Because using an UUID as parameter to subcommands is painful, plakar performs prefix-based lookup and allows using the beginning of an UUID as long as it is not ambiguous:\n$ plakar pull 3 3630358c-c634-4ad9-960d-f864d1eb56f5: OK In case of ambiguity, it warns so that a less ambiguous prefix can be provided:\n$ plakar ls|grep ^4 45de4fd3-d177-4378-84cb-077b2ef297fb [2021-03-27T23:09:11Z] (size: 10 MB, files: 36, dirs: 1) 4c467893-f959-43cb-858f-c50b8c46a86d [2021-03-27T23:17:05Z] (size: 10 MB, files: 36, dirs: 1) $ plakar pull 4 2021/03/28 00:17:15 plakar: snapshot ID is ambigous: 4 (matches 2 snapshots) $ plakar pull 45 45de4fd3-d177-4378-84cb-077b2ef297fb: OK Snapshots health check # Backups are not useful if they can\u0026rsquo;t be restored, but it\u0026rsquo;s also painful to create a backup and try restoring just to know if it would work. To cope with this, plakar supports health checking snapshots without actually restoring them:\n$ plakar check 3 3630358c-c634-4ad9-960d-f864d1eb56f5: OK What it does is that it fetches the snapshot index, checks that everything referenced in the snapshot exists in the repository, and does a few sanity checks to verify if it could restore the entirety of the snapshot if requested.\nSnapshots restoration # As shown in the example above, restoring a snapshot is as simple as typing:\n$ plakar pull 3 3630358c-c634-4ad9-960d-f864d1eb56f5: OK But this performs a full snapshot restore, whereas sometimes what you really want is a partial restore. For this, plakar supports providing a path within a snapshot pointing either to a subdirectory or to a file:\n$ plakar pull 3:/bin/ls 3630358c-c634-4ad9-960d-f864d1eb56f5:/bin/ls: OK $ ls -l bin total 312 -rwxr-xr-x 1 gilles staff 157360 27 Mar 23:18 ls While doing the restore, plakar will check every chunk and their checksums to detect any corruption:\n$ rm -rf ~/.plakar/default/chunks/00/008d4087e3e99c8f8c2c1ad76e9c6bc3d614e580e30ed45da06f93f912995538 $ plakar pull 3 /bin/ksh: missing chunk 008d4087e3e99c8f8c2c1ad76e9c6bc3d614e580e30ed45da06f93f912995538 /bin/ksh: corrupt file: checksum mismatch 3630358c-c634-4ad9-960d-f864d1eb56f5: KO $ ls -l bin total 19504 -rwxr-xr-x 1 gilles staff 121104 27 Mar 23:22 [ -r-xr-xr-x 1 gilles staff 743708 27 Mar 23:22 bash -rwxr-xr-x 1 gilles staff 121968 27 Mar 23:22 cat -rwxr-xr-x 1 gilles staff 107552 27 Mar 23:22 chmod -rwxr-xr-x 1 gilles staff 123232 27 Mar 23:22 cp -rwxr-xr-x 1 gilles staff 1104640 27 Mar 23:22 csh -rwxr-xr-x 1 gilles staff 277408 27 Mar 23:22 dash -rwxr-xr-x 1 gilles staff 139264 27 Mar 23:22 date -rwxr-xr-x 1 gilles staff 122160 27 Mar 23:22 dd -rwxr-xr-x 1 gilles staff 121840 27 Mar 23:22 df -rwxr-xr-x 1 gilles staff 120848 27 Mar 23:22 echo -rwxr-xr-x 1 gilles staff 205648 27 Mar 23:22 ed -rwxr-xr-x 1 gilles staff 121504 27 Mar 23:22 expr -rwxr-xr-x 1 gilles staff 120864 27 Mar 23:22 hostname -rwxr-xr-x 1 gilles staff 121232 27 Mar 23:22 kill -r-xr-xr-x 1 gilles staff 1258618 27 Mar 23:22 ksh -rwxr-xr-x 1 gilles staff 329344 27 Mar 23:22 launchctl -rwxr-xr-x 1 gilles staff 105136 27 Mar 23:22 link -rwxr-xr-x 1 gilles staff 105136 27 Mar 23:22 ln -rwxr-xr-x 1 gilles staff 157360 27 Mar 23:22 ls -rwxr-xr-x 1 gilles staff 104752 27 Mar 23:22 mkdir -rwxr-xr-x 1 gilles staff 106176 27 Mar 23:22 mv -rwxr-xr-x 1 gilles staff 291152 27 Mar 23:22 pax -rwsr-xr-x 1 gilles staff 173568 27 Mar 23:22 ps -rwxr-xr-x 1 gilles staff 120832 27 Mar 23:22 pwd -rwxr-xr-x 1 gilles staff 106000 27 Mar 23:22 rm -rwxr-xr-x 1 gilles staff 104368 27 Mar 23:22 rmdir -rwxr-xr-x 1 gilles staff 120912 27 Mar 23:22 sh -rwxr-xr-x 1 gilles staff 120784 27 Mar 23:22 sleep -rwxr-xr-x 1 gilles staff 138656 27 Mar 23:22 stty -rwxr-xr-x 1 gilles staff 120464 27 Mar 23:22 sync -rwxr-xr-x 1 gilles staff 1104640 27 Mar 23:22 tcsh -rwxr-xr-x 1 gilles staff 121104 27 Mar 23:22 test -rwxr-xr-x 1 gilles staff 106000 27 Mar 23:22 unlink -rwxr-xr-x 1 gilles staff 120752 27 Mar 23:22 wait4path -rwxr-xr-x 1 gilles staff 1331248 27 Mar 23:22 zsh The snapshot is still restored as a \u0026ldquo;best effort\u0026rdquo;, but here a warning informs that ksh is not the same as the one that was initially backed up. This is not supposed to happen in practice as long as I don\u0026rsquo;t fiddle with the repository, but it is a nice sanity check.\nOne that would be caught by plakar check:\n$ plakar check 3 3630358c-c634-4ad9-960d-f864d1eb56f5: KO Other plakar subcommands # In addition to pull, push, ls and check, plakar supports various other subcommands.\nThey are more interesting when playing with text files, so let\u0026rsquo;s push /private/etc:\n$ plakar push /private/etc open /private/etc/krb5.keytab: permission denied open /private/etc/aliases.db: permission denied open /private/etc/racoon/psk.txt: permission denied open /private/etc/security/audit_user: permission denied open /private/etc/security/audit_control: permission denied open /private/etc/sudoers: permission denied open /private/etc/sudo_lecture: permission denied open /private/etc/master.passwd: permission denied open /private/etc/openldap/slapd.conf.default: permission denied open /private/etc/openldap/DB_CONFIG.example: permission denied 98b6658b-a975-47c4-a38f-f958d8d7359f: OK Here plakar has warned about files that couldn\u0026rsquo;t make it into the snapshot, but lets inspect what files are part of the snapshot with ls:\n$ plakar ls 9:/ acc57694b78a1ea669535547ac310e7682f141b75f6ba23658078b9000dbd9ac -rw-r--r-- root wheel 20 kB /private/etc/php-fpm.d/www.conf.default a4dda57401575878b78c5b0bc4a6bc020675d7f285dc8a836d07f1fda0938715 -rw-r--r-- root wheel 12 kB /private/etc/postfix/LICENSE 863f779b43680f81799688e91c18a164047a1a8e9dfff650881e50ba35530ed1 -r--r--r-- root wheel 141 B /private/etc/uucp/port 75a3d05049424cc17623d04c5d84038ca703ae673ad4c1bcf62d8293462d90b0 -rw-r--r-- root wheel 152 B /private/etc/pam.d/login.term 591161270b2fbe74d7f1b96625c4b60551f33eb6fa28992c8ccc71dff9b7c70b -rw-r--r-- root wheel 6.2 kB /private/etc/postfix/master.cf.proto 056514aa4105c96e0aad6a144d76f63461db7157123d43fa7c4121297a6c9200 -rw-r--r-- root wheel 190 B /private/etc/asl/com.apple.authd 05f4215652c68201fd6aee81fc4eeaee4d5c395f3f4a3a155b986cb06ca0f768 -rw-r--r-- root wheel 216 B /private/etc/asl/com.apple.cdscheduler 3c4aaff1783756368c28022658ebb238d6f9f32beeb0d04d79962654e0a0ae26 -rw-r--r-- root wheel 3.3 kB /private/etc/ssh/sshd_config 23198c58755cd991531259cd290f62ea99cd9596595001ab303a15306fd03ec1 -rw-r--r-- root wheel 127 B /private/etc/pam.d/authorization_ctk 48edf5460c6ad64cc681209902d2cfef856f4b68400e34cac41ec6704f45b580 -rwxr-xr-x root wheel 687 B /private/etc/periodic/daily/430.status-rwho c998ee1b26049571ed20dc89e7dd9d3e87e85846bdf0d9ff1aae9f760774bf5b -rw-r--r-- root wheel 1.1 kB /private/etc/apache2/original/extra/httpd-info.conf b822424c1eba4ae12efa0bad98c6288f9111d5da88f43a847c69e660dc3c8c80 -r--r--r-- root wheel 6.2 kB /private/etc/openldap/schema/collective.schema c90d596b58cb588b3ae77312f9fe8b0511ea8a842caf058d130138a32cca8419 -r--r--r-- root wheel 2.1 kB /private/etc/openldap/schema/fmserver.schema 5475edebf371c8c4771d6951a51a80e4c40871a01355122cac4146966d6aa58c -rw-r--r-- root wheel 4.5 kB /private/etc/apache2/extra/httpd-mpm.conf 61b95be8351545f888956f733db55f810e3b5f8b4b221ad219428b95c55bdbc9 -rw-r--r-- root wheel 1.1 kB /private/etc/asl.conf b9611702dbb3c2d02e060d82cff7a542f6ec6de29aabaa0b4c8482e5ed1f78d0 -rw-r--r-- root wheel 113 B /private/etc/pam.d/authorization_aks 9a31f3b43190281ce1320ac62b3d7672c0bd5bb8bf607c82767994bfa815068e -rw-r--r-- root wheel 153 B /private/etc/asl/com.apple.eventmonitor cc58ac4627390ef04037b7b77ed0359b267a8e8ceb99bdc3aa156dae6367094a -rw-r--r-- root wheel 607 B /private/etc/apache2/original/extra/httpd-userdir.conf 17294d602f2d28944e6517a6a8a432548351d1eaf468062b8da6d84bbf7c5440 -r--r--r-- root wheel 133 B /private/etc/uucp/passwd 27b21d21df689f2f097f56b907d2a936acbeb943fc98bfeb56d2c62cba33c451 -rw-r--r-- root wheel 13 B /private/etc/paths.d/40-XQuartz 1f07bc400e932a1f36c65634b9f5e7d7a6249c58195f903fe21d4bf83df43c5b -rw-r--r-- root wheel 3.4 kB /private/etc/wfs/httpd_webdavsharing_template.conf 444c716ac2ccd9e1e3347858cb08a00d2ea38e8c12fdc5798380dc261e32e9ef -r--r--r-- root wheel 265 B /private/etc/bashrc fa115d33bdc964acfe0b0241c511c2bed643820f8c574e45f443a17a331cf138 -rw-r--r-- root wheel 318 B /private/etc/pam.d/screensaver_ctk eab8fa9f3d43e099731db74d17733fb46c5578206f4dbb0db204bc2cef68664a -rw-r--r-- root wheel 7.6 kB /private/etc/passwd 99c7c05d6e8690b32ea58d6fcda64a090eaae782026551d2d9e347694c626ce7 -r--r--r-- root wheel 7.8 kB /private/etc/openldap/schema/nis.schema 025f5e31cd7b2248a0a661d8d346452c31d98e280f3420c3f6017c74c829ae73 -rw-r--r-- root wheel 745 B /private/etc/ssl/openssl.cnf 9d70873703af389018ccc7b57a503d2692ba1d6b71271bcf00473d40f5095486 -rw-r--r-- root wheel 3.2 kB /private/etc/apache2/original/extra/proxy-html.conf 69c9044d7bdcdd195249b13b8893abf7c60092391975408468ac5e8969fcd79a -rw-r--r-- root _lp 6.5 kB /private/etc/cups/cupsd.conf d7c000b62ab236b5c5c4db8ad7edec232b2afe95d85c8e064a5c9f6d5308620f -rw-r--r-- root wheel 1.6 kB /private/etc/postfix/TLS_LICENSE ed9d05e8ec15f676263a184a7c30858f2d5cd0258da168d3b262b76567bc7bae -rw-r--r-- root wheel 2.9 kB /private/etc/apache2/original/extra/httpd-default.conf bb3d6775b7fc8042b2b68232384694ac3bee6d1100b87f9dcd0840ee935dd2b5 -rw-r--r-- root wheel 12 kB /private/etc/postfix/canonical 96618e0c2ba27c318f77262c22a99f039ecf2e1d83754fcb1a0f208e961da2f7 -r--r--r-- root wheel 1.5 kB /private/etc/openldap/schema/openldap.schema cf6e6fe1497d5b3983529174674b8a09941d67244727157e208d37b24b247bc1 -rw-r--r-- root wheel 166 B /private/etc/pam.d/authorization_la aa464696f1c0282b1728b79e5da52c6f44d130b93c6e3011ea4e67407461c191 -rw-r--r-- root wheel 3.6 kB /private/etc/racoon/racoon.conf 1dc9a5dec35592b043715e6b5a1796df15540ebfe97b6f25fb4960183655eec9 -rw-r--r-- root wheel 9.3 kB /private/etc/zshrc_Apple_Terminal e2dd2ce55d57c625d241b8ddbc834fbab3e9c2061955b094326201dfd083d051 -rw-r--r-- root wheel 202 B /private/etc/asl/com.apple.clouddocs eb33ba8357c7166620b5c33fc8436a0081ba093f7cbf60fef7ddb35b491a4012 -rw-r--r-- root _lp 128 B /private/etc/cups/snmp.conf d7cbf00c6fb63b14e2314cb5aff696e6d01e6911e0bcebf2c540ddb28b614340 -r--r--r-- root wheel 6.8 kB /private/etc/openldap/schema/nis.ldif 91d53776cbd13b565b93dfae32ceb7236c09df3f4629329b521fe1b523460613 -r--r--r-- root wheel 3.3 kB /private/etc/openldap/schema/dyngroup.ldif 4a054f138fbed4a755b351fb51dd83fe3005e07fc7d075c3880e10ec8950873a -r--r--r-- root wheel 123 kB /private/etc/openldap/schema/microsoft.schema 8e920f9fb045a6dfe53fd18bb6450c5b3051bd88c4c7eac88b7c41deee02e6a8 -r--r--r-- root wheel 246 B /private/etc/pam.d/sudo 9d5aa72f807091b481820d12e693093293ba33c73854909ad7b0fb192c2db193 -rw-r--r-- root wheel 189 B /private/etc/shells 81bac019ddc3523e67726f6159c180bd128b9ef1872d5359fe40e3fa78690694 -rw-r--r-- root wheel 387 B /private/etc/asl/com.apple.AddressBookLegacy 309c09ccf0f826e2e753472a01fa32d7f1f326879992e8e925ef3ced57a6252b -rw-r--r-- root wheel 11 B /private/etc/nanorc a27af5a5be37a725abe8587a803e601e243739c2ea4abfde9707199b66369deb -rw-r--r-- root wheel 1.3 kB /private/etc/newsyslog.conf 810dbd036a97217460ec5f56e543621dd2a735bb809fca45327400e2afc272ad -rw-r--r-- root wheel 27 kB /private/etc/postfix/main.cf.proto 8aa44a4856b41d4f1dd7799b975eb68472df1d6b8f8268a0cc6aaa9975f55349 -rw-r--r-- root wheel 891 B /private/etc/rtadvd.conf 49422ce34f9e3b5134cecdb73f5f54fa34480fc424279321dab18240b4bf8cde -r--r--r-- root wheel 2.4 kB /private/etc/openldap/schema/misc.schema eff9ba3a711059ac519e1914477a28db07ce3ed8c70eb36479e0da39cd41fc4e -rw-r--r-- root wheel 197 B /private/etc/pam.d/passwd e523e826b0192c39b42d71597caa8b3e5e166d1e6af98528f046f1ebe6727925 -rw-r--r-- root wheel 24 kB /private/etc/postfix/header_checks f63a90bd20d0baea6f3b7f5c0679d02e58dc294d5a37bd0c44f2046609d74b4a -r--r--r-- root wheel 16 kB /private/etc/snmp/snmpd.conf.default c7e32a8ad7f390bc508c245dfd929cccf18cf222889ca7eed281619495b3dc5f -rw-r--r-- root wheel 346 kB /private/etc/ssl/cert.pem 49fde98b0963f27c0630072414be0ee0205d71eba529bbaf86b65d1e1603eead -rw-r--r-- root wheel 2.9 kB /private/etc/apache2/extra/httpd-autoindex.conf f918b0dec7a783521bd7efbe2e5aef28e2d3250bec1cb524752bb31d18ddbe0b -rw-r--r-- root wheel 149 B /private/etc/asl/com.apple.performance b2dc9fa1938c981ce201f1458afb373d97caf860d9bfba0ded9c450a64f8e65d -rw-r--r-- root wheel 27 B /private/etc/ntp.conf 9e8dffed375e348782c5bc8e15b6a33ea002c96588b7686bd230693dd86898eb -rw-r--r-- root wheel 181 B /private/etc/pam.d/checkpw 8a8a65b24957cbf2c5376a2a7a4dccb3e02f8cc3cfb338caa58cb93345c6a23c -r--r--r-- root wheel 203 B /private/etc/pam.d/cups c74b98809cf175d83699a095bf9f20689dcc66daeedc7e48119026e88103fb09 -rw-r--r-- root wheel 44 B /private/etc/postfix/custom_header_checks 891232253c3bb2789de7cb01b8d313e3c3ed6820434457634775867cc50c657b -rw-r--r-- root wheel 5.4 kB /private/etc/postfix/makedefs.out ba4bb2c9087c046186ef306d4bd5046694e7bc0bc715503ba210e6f96e417642 -rw-r--r-- root wheel 13 kB /private/etc/apache2/original/extra/httpd-ssl.conf 4348468f1850b16e8ad3f7be8963bde559cfd8dd7aed8ffd655a40aaa551e073 -rw-r--r-- root wheel 156 B /private/etc/asl/com.apple.MessageTracer b2aac03248e8f229c703561f5bb059f9be491e5db9f447692398d775a9fb12a5 -rw-r--r-- root wheel 195 B /private/etc/auto_master bf41a1579c3d12a4b8466fad91794157fcca4b2c611fb178ddce232cd9a4e0e5 -r--r--r-- root wheel 8.1 kB /private/etc/openldap/schema/corba.schema c78f581bf6c453c6cf4e4ba241fe081d7de69bbfa42d41fc65a475827c8bd627 -rwxr-xr-x root wheel 695 B /private/etc/periodic/daily/130.clean-msgs 7e94a499c84675258a9734b150a28950fa22cbaad5f1eedfff507541d9f7d9ef -rw-r--r-- root wheel 13 kB /private/etc/apache2/magic 66dfdc46b6b66f0830252f20e1c74aa865656223e30f95fbef6b05051fdc3cde -rw-r--r-- root wheel 9.3 kB /private/etc/bashrc_Apple_Terminal f19f881084f599fa261243918d922373eab14623e78d23c41fcc031aa21ca7b6 -rw-r--r-- root wheel 941 B /private/etc/emond.d/emond.plist b19baf7d26dad9163a2e87c5b9731943e8dad7243c81aacfacf6bdcdcae03c61 -rw-r--r-- root wheel 200 B /private/etc/pam.d/chkpasswd effcfce27485ef1ca9142df4fe810d518e720092b00dce0a9504e2ef378cc42c -rw-r--r-- root wheel 27 kB /private/etc/postfix/main.cf d73737159f9e3bedf75645dd7c2254331beea55567da92f3b974d4ba8194ff81 -rw-r--r-- root wheel 13 kB /private/etc/postfix/virtual 1b3f45ae268583561d2e9baa82474dd0035d974cf687991a20f885f107dd943d -r--r--r-- root wheel 2.0 kB /private/etc/openldap/schema/collective.ldif f99a4803d9f2f8c48c33866487067fbd3a0d02080e10749e70445813b4d60954 -rwxr-xr-x root wheel 1.6 kB /private/etc/periodic/daily/110.clean-tmps 7ac5924452faaf32bbfbd41f816947fccc3ff2fc17554be7bc5fff99d71ebe2e -rw-r--r-- root wheel 6.9 kB /private/etc/postfix/relocated 7cb50dc544e9050e398ebc0844081e928ca52c09a0d304228489962a93b63692 -rw-r--r-- root wheel 365 B /private/etc/hosts fde802d853379ae3724693f05be7f273b92429ab500a7806683e7a21975f743e -r--r--r-- root wheel 3.0 kB /private/etc/openldap/schema/java.ldif dbec41507d3e3c390b0418d96c9ea5f61d3e9ac969bd025b6d23b78af25adaeb -rw-r--r-- root wheel 18 B /private/etc/paths.d/go 2012882e055c2a2502cd7d3dd373c1804f0fce782225e90aca491abad720b53a -rw-r--r-- root wheel 1.5 kB /private/etc/apache2/original/extra/httpd-vhosts.conf dd89ec314b4e26aa2481a315c7e71404ccff929ead511269e934406be0e61143 -rwxr-xr-x root wheel 888 B /private/etc/periodic/monthly/199.rotate-fax 3efeec1e339725bfadd53138dde7822ebc57ba73275a1c0fc05e0d7d8abafc4e -rw-r--r-- root wheel 624 B /private/etc/wfs/httpd_webdavsharing_proxy.conf.inactive ede47b49ba30fbd0e02f10006fa967d2887f7e4c676e6b657ae0a20285ef1d4c -rw-r--r-- root wheel 2.2 kB /private/etc/apache2/extra/httpd-multilang-errordoc.conf ebbeeaa6c956e56727c1391e4443a7da9da70e5e9201fcf69194e9ce397f9a2d -rw-r--r-- root wheel 339 B /private/etc/asl/com.apple.coreduetd 2a0ba79339b7112dbd24b5fdd4b54f32573971a0ad9906240bbeb6bad3d382bc -rw-r--r-- root wheel 5.3 kB /private/etc/rc.netboot 6fb5b260918922ca5ca4dfb296967d8edb9c12f2c043f26b64590758441d682d -rw-r--r-- root wheel 1.0 kB /private/etc/pf.conf 011fd8d7180df2a60de23826192aab50889f73488d01d6c35d186fcb7f64640d -rw-r--r-- root wheel 61 kB /private/etc/apache2/mime.types 7c1f46f8dca762135990bdf0e3aa0395d4867858e35a3f2583549fa6cd5b082a -rw-r--r-- root wheel 189 B /private/etc/csh.cshrc cdfc5a48233b2f44bc18da0cf5e26df47e9424820793d53886aa175dfbca7896 -rw-r--r-- root wheel 45 B /private/etc/paths 64d9fef3a2825ba4d770b38b5ccd12ad4df477535f345fa8649d65fea940fcbf -rw-r--r-- root wheel 512 B /private/etc/pam.d/login 38b6d46f6924364dc3013ffad83a12b7bc10bf3a878b5ae0f53ef117851900b8 -rw-r--r-- root wheel 1.7 kB /private/etc/apache2/extra/httpd-dav.conf ba4bb2c9087c046186ef306d4bd5046694e7bc0bc715503ba210e6f96e417642 -rw-r--r-- root wheel 13 kB /private/etc/apache2/extra/httpd-ssl.conf ede47b49ba30fbd0e02f10006fa967d2887f7e4c676e6b657ae0a20285ef1d4c -rw-r--r-- root wheel 2.2 kB /private/etc/apache2/original/extra/httpd-multilang-errordoc.conf 5475edebf371c8c4771d6951a51a80e4c40871a01355122cac4146966d6aa58c -rw-r--r-- root wheel 4.5 kB /private/etc/apache2/original/extra/httpd-mpm.conf e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/hosts.equiv e05e1b6f966477644849720be93b0707029323b732b4105b63475443fa740aa2 -rw-r--r-- root wheel 376 B /private/etc/asl/com.apple.networking.boringssl 39bc7222686b37aea251957d5b0658600711bc91815a792459395188ace48e33 -rw-r--r-- root wheel 23 B /private/etc/ntp_opendirectory.conf ad2902787f6a2cc2728320d9bf5387371d88a8226c32bac08df9f6d7886bfee4 -r--r--r-- root wheel 20 kB /private/etc/openldap/schema/ppolicy.schema 69a61ae495580e0e144792c9a809bffe707a6a13f292095f43a7a88c92b11ef0 -rwxr-xr-x root wheel 1.2 kB /private/etc/periodic/daily/310.accounting 49fde98b0963f27c0630072414be0ee0205d71eba529bbaf86b65d1e1603eead -rw-r--r-- root wheel 2.9 kB /private/etc/apache2/original/extra/httpd-autoindex.conf f1e89e3a029a3b72a30f82fc7346c13030b4de16b77ecc4c58f4a4de6f99c242 -rw-r--r-- root wheel 82 B /private/etc/com.apple.screensharing.agent.launchd a18ff587cf7d16939acf5a969da49a393ce503b1c4b83ad42e4b902991329b4f -r--r--r-- root wheel 48 kB /private/etc/openldap/schema/apple.schema 6293f4cc430a3d8648fa60f26546d16e55f84d05b71f996ca930a95aa3ce312c -r--r--r-- root wheel 3.3 kB /private/etc/openldap/schema/dyngroup.schema c0dbae0bb16ea7cbc26c0d49286946ef8713335564f5c0a2f4ab7339b8f948f2 -r--r--r-- root wheel 2.1 kB /private/etc/openldap/schema/misc.ldif fc1166d771ec0092cb6280cadf54e9df3fa4eff6be0132f9064f87a7b0b6416a -r--r--r-- root wheel 3.3 kB /private/etc/openldap/schema/openldap.ldif b232f6b78ffdcfdf34697f1432b6cd9bcbad27da384d493d9285b02f398c13ea -rw-r--r-- root wheel 1.7 kB /private/etc/rpc 9a40f8365adb49f0a63930316c475251631150f0f58bfad692a633c8e7486fc9 -rw-r--r-- root wheel 96 B /private/etc/syslog.conf b99ca2f2de327fd9a0374e1eacd3ccbc0f6fce17e0372c42d370ce4852bb75f4 -rw-r--r-- root wheel 22 kB /private/etc/apache2/original/httpd.conf b45fd6ac093bcb4aec44b366d14300cdb1a65f65d7a630630af0f89c9c937894 -rw-r--r-- root wheel 265 B /private/etc/openldap/ldap.conf 36ff03121c003e884d67372d060c83886d73429fc62df37c831f604ecb59cb56 -r--r--r-- root wheel 717 B /private/etc/openldap/schema/apple_auxillary.schema b01b61f6b1486186b84ac009ccfff409fc43dd66e1b780a60de94090e8459dc2 -rw-r--r-- root wheel 154 B /private/etc/newsyslog.d/wifi.conf 334c18bfe6d007a0bcb6f1e7b3619b9592acea6ce60775d6f8d3d56f7ca8d50b -r--r--r-- root wheel 72 kB /private/etc/php.ini.default 1b3a1b45deb322db6df7d39dd52f67d1d49cb89e77388c41dccf640dcd83cd37 -rw-r--r-- root wheel 10 kB /private/etc/postfix/aliases 8923cf7346a5536cc5fadf73e1f92681cf31e73d06548f8abd1e7854f5694290 -rw-r--r-- root wheel 351 B /private/etc/notify.conf a377ed0d15192ea50c9de076583030fe246bec85bd37a9fe1f3cd792e84231ab -r--r--r-- root wheel 20 kB /private/etc/openldap/schema/pmi.schema fbbd9eac3bb65ba340cda731e596fc52e76c49339e7c7dd6653f06d17edb36ad -r--r--r-- root wheel 527 B /private/etc/pam.d/sshd aab9982ea2af8b86b1fde1e8c199de6f12f45db1a831c1804c33ee65e53d6d25 -rw-r--r-- root wheel 264 B /private/etc/pam.d/authorization d0f8d2287d44031b79d9798dc6c68ea7d789ffc129ec2056b3630c938cc54068 -rw-r--r-- root wheel 3.3 kB /private/etc/wfs/httpd_webdavsharing.conf 405101681e712d2727fd6131a84b0eb1b89ed3b72ad3722916368b5ea3cdf6f6 -rw-r--r-- root wheel 943 B /private/etc/wfs/wfs.plist be95a05eafeae5a7450eef61d6477d0b1080f314c3338b5f81c5d5fc65681fdd -r--r--r-- root wheel 1.3 kB /private/etc/irbrc f33710cbbb38b977b7ed1a10038f6f1911d0ce8dd82f65e1a54db13fbecc429f -r--r--r-- root wheel 12 kB /private/etc/openldap/schema/cosine.ldif 422bfa0619b344ec6248e589333df261c4901feca2ce816fd23ef99b5229b7a0 -r--r--r-- root wheel 5.7 kB /private/etc/openldap/schema/samba.schema 950f16a5623699af1fa0e03f9f286374229bbc84d7e887efa3984a1902e48d4b -rw-r--r-- root wheel 175 B /private/etc/newsyslog.d/com.apple.slapconfig.conf 994263a50efad8a1b416754ea4ed463d5e1a7524ed2b3105b780dfe480dbfe94 -r--r--r-- root wheel 145 B /private/etc/php-NOTICE-PLANNED-REMOVAL.txt 97a0a5a23df5f84d6c4396e5e1c4c6b10c690300ee392cfbf633002e300ce2cb -rw-r--r-- root wheel 10 kB /private/etc/postfix/generic b86accf42ae92d4e00b67c50010b7f278bc76c50755bdc9638e2096cbd596890 -rw-r--r-- root wheel 6.4 kB /private/etc/protocols 24240ece7cf8aa30d242f178d7defd3a47d631bbbcf897d2f170e28b863564fa -rw-r--r-- root wheel 178 B /private/etc/asl/com.apple.mail 71be406f45c35429621a43970b0a10debcfd319f0e0ecb1f9b5c5c0ce4cfc3bd -rw-r--r-- root wheel 121 B /private/etc/csh.login 2e5298d1432df0d5f3b741a1e052cfdb73cc4d800db34651f083f36b3c64dcb8 -r--r--r-- root wheel 10 kB /private/etc/openldap/schema/duaconf.schema 36816c4a31b3fbbc86312073f65ba5b514823f89864ad2f3ede7b6b456a8ed4d -r--r--r-- root wheel 27 kB /private/etc/security/audit_event d4a58e12a2e3a9aa4ca72cbe4c63b786b20d6a212a87759f6c93db594be3e3f0 -r-xr-xr-x root wheel 1.3 kB /private/etc/security/audit_warn a744a313d0b95f6f15768b78d15cf3168ffd18abde4e4801a55b0fbb1c37ae86 -rw-r--r-- root wheel 233 B /private/etc/asl/com.apple.install a1b83027e0b929e389bde2984078b7debf7f885051d9f9be18545aa07bebac21 -rw-r--r-- root wheel 409 B /private/etc/asl/com.apple.mkb.internal 533ad90f9c16d3d000a929513d72050bdef7e75bc8ea1790694227c20ab97f2e -rw-r--r-- root wheel 176 B /private/etc/newsyslog.d/com.apple.xscertd.conf b17912cf8ac845d1098829a3f876bdc7597bc8360bb8f260e49d1e3ca26d5029 -rw-r--r-- root wheel 1.4 kB /private/etc/apache2/extra/httpd-manual.conf 28048538f6a15661bdefcf706f39d0003fb9107bdc5206df295db67b66b8345a -rw-r--r-- root wheel 365 B /private/etc/pam.d/screensaver_la 16c13e23b179e6b325102eb285e78b7678ee64231f9bbb2ff6bd5311f38f89ae -rw-r--r-- root wheel 7.4 kB /private/etc/postfix/master.cf 61b42416ea3c5c9d5850364a23ec3aa5b6ddc6e2055b0402d6435b8fed46f3c5 -r--r--r-- root wheel 652 B /private/etc/security/audit_class e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/xtab fb5827cb4712b7e7932d438067ec4852c8955a9ff0f55e282473684623ebdfa1 -r--r--r-- root wheel 3.1 kB /private/etc/zshrc 1aac36c9a80ecab24b5d4346ac5c605a9614f89590e6d0957b82f781a8d40a49 -rwxr-xr-x root wheel 1.1 kB /private/etc/periodic/daily/140.clean-rwho ca2ae7cf01205f3b961c70c607a31c5ed7cc4434dc94d5a5152e18844c5ecffe -rwxr-xr-x root wheel 378 B /private/etc/periodic/daily/199.clean-fax e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/rmtab e02803e7d3435412c397478205b16c95486fde05ecde74959f9799c49c7ce009 -rw-r--r-- root wheel 149 B /private/etc/auto_home bd0ec71eab6515a902703163e65839498429e7d31860c9584b30b635f7423367 -r--r--r-- root wheel 14 kB /private/etc/openldap/schema/java.schema 11eddeb0e0d55ea1bf43c180f04a53533d5cc71d2a69fbbcf7ca0488c8450d79 -rwxr-xr-x root wheel 23 B /private/etc/paths.d/100-rvictl 20909c75c14c9f5360a48c889d06a0d6cfbfa28080348940fc077761744f2aa5 -rw-r--r-- root wheel 822 B /private/etc/emond.d/rules/SampleRules.plist 11ae0d388aed5d193821d33d060da5ed9119d19689ce6c220b407a4c5da3553f -rwxr-xr-x root wheel 1.0 kB /private/etc/periodic/monthly/200.accounting 8c6e2a2647ee854f469a3bb798e02ba5a8b1812cab229ff129f073e7a80c1202 -rw-r--r-- root wheel 678 kB /private/etc/services 063a5972c7b72eda797684b2c034f37a000aaf7bd3b91b01f7cdd3ff74f170e7 -rw-r--r-- root wheel 5.7 kB /private/etc/gettytab b45fd6ac093bcb4aec44b366d14300cdb1a65f65d7a630630af0f89c9c937894 -rw-r--r-- root wheel 265 B /private/etc/openldap/ldap.conf.default bbe18692eb80dc6e27643a5392b358be548edaec210206eeb1a67c99892fc33e -rw-r--r-- root wheel 39 B /private/etc/csh.logout 7de66c7adb93cb1e0da88874d27668c1d78488a9578268e8234460d9083b2e01 -rw-r--r-- root _lp 6.5 kB /private/etc/cups/cupsd.conf.default ee479a7a0dd839c4a08e73d5a6fcffbe750bd47b82cf7c633ed20c5e63b8ed03 -rw-r--r-- root wheel 5.0 kB /private/etc/defaults/periodic.conf e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/kern_loader.conf 7bf0e7399139a3a478b9f447ceb042dee1137261b39dace27db18a93242ebdc2 -r--r--r-- root wheel 1.8 kB /private/etc/openldap/schema/corba.ldif c536e94effbd1e5890f6fb084590a091afc8369a65a9a8b3d71edbfde290da45 -r--r--r-- root wheel 20 kB /private/etc/openldap/schema/core.schema aea837b88e320abcd476f5839952d5e1a612c0e4a225cb9205d20071e6c37745 -rw-r--r-- root wheel 106 B /private/etc/mail.rc b14dbb949b10a0635b7303ff96d86406d8d6c64d414a1e0f91cdf8ba2a62c74a -r--r--r-- root wheel 191 B /private/etc/pam.d/other 9d6f5bffb6fda39e61e1dd90109a42b7a1abf8a5995f3e81e87b7bf4f312b309 -rw-r--r-- root wheel 119 B /private/etc/ftpusers 0cb5352ac33727fd7979e454f6ac1a56b7795a8ab5a25d7cc955133fb47cf9c4 -rwxr-xr-x root wheel 522 B /private/etc/periodic/daily/400.status-disks 16c13e23b179e6b325102eb285e78b7678ee64231f9bbb2ff6bd5311f38f89ae -rw-r--r-- root wheel 7.4 kB /private/etc/postfix/master.cf.default d9013690a0652573167ef63917f6ed2ad63a5e15b1f82d40010c75f33ec85da1 -r--r--r-- root wheel 205 B /private/etc/apache2/other/mpm.conf 14ecbb4a93f277771924c3a9aa870ecd67a742fd6164572af329feef40061325 -rw-r--r-- root wheel 318 B /private/etc/asl/com.apple.login.guest b22a5ff409482104f804f04b0035f2fd1f6801974645f79c87e9025729a50286 -rw-r--r-- root _lp 3.1 kB /private/etc/cups/cups-files.conf.default 18dc169426b3130e43cd1a5919f95dd58ccee6d49a9182a4b2ecdf5c5bb563bc -rw-r--r-- root wheel 577 kB /private/etc/ssh/moduli c64beac4afa42e602286728c1e0158fc90195768e97d5f6bc0d917208faeb3b3 -r--r--r-- root wheel 21 kB /private/etc/openldap/schema/core.ldif 8c716d8131c6f1ae290b49beed057f06bd5cd18b5227cfe3b50556b35e21ce3b -rw-r--r-- root wheel 5.3 kB /private/etc/php-fpm.conf.default a3fe9f414586c0d3cacbe3b6920a09d8718e503bca22e23fef882203bf765065 -r--r--r-- root wheel 189 B /private/etc/profile f63a90bd20d0baea6f3b7f5c0679d02e58dc294d5a37bd0c44f2046609d74b4a -rw-r--r-- root wheel 16 kB /private/etc/snmp/snmpd.conf 6e72e6f9366198e19491ccc785a0cdfb5a05510609c03fcf388b02aac43fec0b -rw-r--r-- root wheel 1.0 kB /private/etc/ssl/x509v3.cnf b86ff58053e9e930a0324f7fb5e0458213567feca9e97de92da20bff82f17e06 -r--r--r-- root wheel 2.4 kB /private/etc/uucp/sys 146f0e625f4a1b8b11fabab45b66d989e85d0408348403678074b3db881e3587 -rw-r--r-- root wheel 165 B /private/etc/newsyslog.d/com.apple.slapd.conf 726813f6f77f7b566a7c004d79cac51dfdc3b28ff3265179e7fbb032b706ba6d -r--r--r-- root wheel 4.0 kB /private/etc/openldap/schema/ppolicy.ldif 8475720c1288108fa6c55c028a26e7aa12155009095dd8c04a547969b65a95b3 -rwxr-xr-x root wheel 548 B /private/etc/periodic/daily/420.status-network abb7d93ea2ef352db65c05101a112264334e7e531522f576ee397371e13fe5d8 -rw-r--r-- root wheel 183 B /private/etc/pam.d/authorization_lacont 4ae766c32277f60bba1b505e3837ccc29587c0cb4fbf65d20b17baa9463b0fb6 -rw-r--r-- root wheel 13 kB /private/etc/postfix/transport a9419086fc2b70f69130c3ee9f8761b0a12b0c7da47d3b779b04ab3827081cf0 -rw-r--r-- root wheel 5.1 kB /private/etc/apache2/extra/httpd-languages.conf e8c63560d75c0c459666f2d8c69bd23b83151080d571148b9a30667f680d6f3c -rw-r--r-- root wheel 3.2 kB /private/etc/group 7ca5887133958e0ac30c92ecec70fe753977cfcc0137cfa93311aeed8fa38e24 -rw-r--r-- root wheel 117 kB /private/etc/openldap/AppleOpenLDAP.plist 4277bb97ba7b51577a0d38151d3e08b40bdf946753f5b5bdeb814d6ff57a8a5e -rw-r--r-- root wheel 515 B /private/etc/afpovertcp.cfg f047c7a23d830f221f142861d2ae10eb4a27d74beff3fcaf599b4c441ae44f5e -r--r--r-- root wheel 13 kB /private/etc/openldap/schema/microsoft.std.schema 94139c5762d820ac8a6d567ca5a5e9753af95e1c8560f23e65bc169be11f4042 -rwxr-xr-x root wheel 712 B /private/etc/periodic/daily/999.local 69f2f70c4d02fe6a0d4ecc99b8cfb676090aae0435c74eb45b2d79c72cb39a09 -rwxr-xr-x root wheel 606 B /private/etc/periodic/monthly/999.local b99ca2f2de327fd9a0374e1eacd3ccbc0f6fce17e0372c42d370ce4852bb75f4 -rw-r--r-- root wheel 22 kB /private/etc/apache2/httpd.conf db16889d677588787af70881d7e0af827e847a5660e9a78c955da53d8edd614b -rw-r--r-- root wheel 105 B /private/etc/asl/com.apple.iokit.power bea9ba8f43b879adff12ba844aad204ae6be074b356a4ff917305725b4341eb5 -r--r--r-- root wheel 154 B /private/etc/newsyslog.d/files.conf 0d5d7c3f51196286c17c894bf77ab0ba057256aa04a455a137c3860fbb46d98b -r--r--r-- root wheel 3.5 kB /private/etc/openldap/schema/README 6c2621756bae333c3af6429407f949f6220bd75df956606ed911bf6b5c8348fd -rw-r--r-- root wheel 140 B /private/etc/pam.d/smbd 38b19f16f2cfe9cb398e59544cbbeeb4998718dbf231251a41eb31893aed223b -rw-r--r-- root wheel 3.5 kB /private/etc/postfix/bounce.cf.default 6dcf0856acd475d75f7d4f2cb5e6a743b2df97ddd394b78e07ae172ab7f04363 -rw-r--r-- root wheel 20 kB /private/etc/postfix/postfix-files 05e4653d532915fca72e4352ade5040c3361340920eee047ac4dd313bc9d8e19 -rw-r--r-- root wheel 1.6 kB /private/etc/ssh/ssh_config 49dede3f9e65fd6ea036bc1598e07f1f73a3ad9c70c4744de954d6d0a8fdbdb5 -r--r--r-- root wheel 194 B /private/etc/apache2/other/php7.conf a5ff0b83be70bdb0107e8e6f4d2bc8e3608e7e9b15e15145b5f24ca4c559f504 -rw-r--r-- root wheel 1.9 kB /private/etc/autofs.conf 1c350ccbbf5e4e09923f5bd1d93fd1578e0a5b6b7d4b85c631d09e1a1a71dbd3 -r--r--r-- root wheel 6.3 kB /private/etc/openldap/schema/inetorgperson.schema 3ada6707f5fd265c70176f24ff988acb15983df6b2b99a5b3cdb833c8c38ea9f -r--r--r-- root wheel 74 kB /private/etc/openldap/schema/cosine.schema c3a8ab17dd75c874d9da896e729c516273900f8e5b80538daf60e8e95ff86660 -rw-r--r-- root wheel 22 kB /private/etc/postfix/access effcfce27485ef1ca9142df4fe810d518e720092b00dce0a9504e2ef378cc42c -rw-r--r-- root wheel 27 kB /private/etc/postfix/main.cf.default 768ff09154f6aacda857cb175ef29cf9d23ef9c38c69efdbf20354dbfd7875b1 -rw-r--r-- root wheel 1.6 kB /private/etc/rc.common 0235d3c1b6cf21e7043fbc98e239ee4bc648048aafaf6be1a94a576300584ef2 -r--r--r-- root wheel 255 B /private/etc/zprofile a9419086fc2b70f69130c3ee9f8761b0a12b0c7da47d3b779b04ab3827081cf0 -rw-r--r-- root wheel 5.1 kB /private/etc/apache2/original/extra/httpd-languages.conf 7c982d301cf583c578abb470f5f3782d1d4451e43f5b90c6555ae75be6060d1a -rw-r--r-- root wheel 53 B /private/etc/networks d8b2b88b7cdec55c5150ced01d4e0a90b275bb11483086ea74cfcd6687309bae -r--r--r-- root wheel 177 kB /private/etc/openldap/schema/microsoft.ext.schema 13e0a0a4991f31f4864096846b379a1d96808c2f0a735b6661aae29481f4bb94 -rw-r--r-- root wheel 356 B /private/etc/pam.d/su 0fd7f2c4f02e68615f137ed090cbba2d1facb40da6bf269e1e326ed6a10aa618 -rw-r--r-- root wheel 329 B /private/etc/pf.anchors/com.apple 847e127803905f596f8c2553ddab8e2d0bca21b4c4ebc79e4be19dcb554df9cb -rw-r--r-- root wheel 1.3 kB /private/etc/bootpd.plist e93775ee26b46f1bfc6b5eab5eab05cee15a1718e87a4a01cfbd474b5409d872 -rw-r--r-- root wheel 621 B /private/etc/locate.rc 0c4f41f80e397939b4a9ce2137a1d725efb4d373e0587c8551b9f1ac12ed0a56 -rw-r--r-- root wheel 312 B /private/etc/pam.d/screensaver_aks c9acb4b8a3e8fd249e672516e37a88d37a9bc8f224b0ab29e76d302c4179de04 -rw-r--r-- root wheel 28 kB /private/etc/pf.os 38b6d46f6924364dc3013ffad83a12b7bc10bf3a878b5ae0f53ef117851900b8 -rw-r--r-- root wheel 1.7 kB /private/etc/apache2/original/extra/httpd-dav.conf e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/find.codes dbeff46b11216fc8b39dd3160f5085309c031044afdc77f394c94baa558abb60 -rw-r--r-- root wheel 19 B /private/etc/manpaths.d/40-XQuartz c998ee1b26049571ed20dc89e7dd9d3e87e85846bdf0d9ff1aae9f760774bf5b -rw-r--r-- root wheel 1.1 kB /private/etc/apache2/extra/httpd-info.conf b22a5ff409482104f804f04b0035f2fd1f6801974645f79c87e9025729a50286 -rw-r--r-- root _lp 3.1 kB /private/etc/cups/cups-files.conf 411408289c874dfe0cb497c7d09e0e69601183ad7c9cbedcb9eae7543ff849c4 -r--r--r-- root wheel 4.8 kB /private/etc/openldap/schema/duaconf.ldif 3c4f529ed0ba941121c3bb29d6e654299d9d9c77a98c2570465f15b0cce05aef -rw-r--r-- root wheel 408 B /private/etc/pam.d/screensaver 081c0787da77e326dcf46a6d955cc807c42ad2362e99876c1c71257b38ff0c84 -rw-r--r-- root wheel 1.3 kB /private/etc/ttys 2012882e055c2a2502cd7d3dd373c1804f0fce782225e90aca491abad720b53a -rw-r--r-- root wheel 1.5 kB /private/etc/apache2/extra/httpd-vhosts.conf 43b85836499d3227d242c8ed7a9f33ee61a0b6be1df7bc41f962eff46b547ef1 -rw-r--r-- root wheel 1.4 kB /private/etc/asl/com.apple.contacts.ContactsAutocomplete 41cdd0a28b1e13f3a475fafd45dec6d335cddecd6d2872a71450d5ab3336ee67 -rw-r--r-- root wheel 4.6 kB /private/etc/man.conf 933a32202a6ca9d47c1a0af25f8577369b0134ce749002de1228f2b23a909142 -rw-r--r-- root wheel 43 B /private/etc/nfs.conf 9b5ee3e68d2668a06a26bc29488c891a0182b4f2858f8bab97835d4238c8e892 -r--r--r-- root wheel 3.5 kB /private/etc/openldap/schema/inetorgperson.ldif 6191e98d0228c084ec94f80af3639912b53c316db2cbf29a2123eab64491f709 -r--r--r-- root wheel 4.1 kB /private/etc/openldap/schema/krb5-kdc.schema 7ba16d87de833e0b81898b544f981cd8b7dedcb8e9ad285bd954af560e089816 -rwxr-xr-x root wheel 620 B /private/etc/periodic/weekly/999.local ed9d05e8ec15f676263a184a7c30858f2d5cd0258da168d3b262b76567bc7bae -rw-r--r-- root wheel 2.9 kB /private/etc/apache2/extra/httpd-default.conf b17912cf8ac845d1098829a3f876bdc7597bc8360bb8f260e49d1e3ca26d5029 -rw-r--r-- root wheel 1.4 kB /private/etc/apache2/original/extra/httpd-manual.conf eb33ba8357c7166620b5c33fc8436a0081ba093f7cbf60fef7ddb35b491a4012 -rw-r--r-- root _lp 128 B /private/etc/cups/snmp.conf.default 9d70873703af389018ccc7b57a503d2692ba1d6b71271bcf00473d40f5095486 -rw-r--r-- root wheel 3.2 kB /private/etc/apache2/extra/proxy-html.conf b5847002ceeb4b141a882ba230eb84c186e2d59c74fbb718cc323c6b157ebde8 -r--r--r-- root wheel 8.5 kB /private/etc/openldap/schema/netinfo.schema cc58ac4627390ef04037b7b77ed0359b267a8e8ceb99bdc3aa156dae6367094a -rw-r--r-- root wheel 607 B /private/etc/apache2/extra/httpd-userdir.conf 7ed4a5bae35e87e8815cb79dd3be70ac34a3056cb91216a22224a3c99b4c9720 -rw-r--r-- root wheel 350 B /private/etc/asl/com.apple.mkb 125333636796ce03bacf4b079a93b90d3d458a81159c9593339183be5bf51080 -r--r--r-- root wheel 6.9 kB /private/etc/openldap/schema/pmi.ldif 8a8b640738d46484ecc24c689f70702c5ca35dcd4868f7f413bd8727cd474f10 -rw-r--r-- root wheel 36 B /private/etc/manpaths Or list files within a specific subdirectory:\n$ plakar ls 9:/private/etc/uucp b86ff58053e9e930a0324f7fb5e0458213567feca9e97de92da20bff82f17e06 -r--r--r-- root wheel 2.4 kB /private/etc/uucp/sys 17294d602f2d28944e6517a6a8a432548351d1eaf468062b8da6d84bbf7c5440 -r--r--r-- root wheel 133 B /private/etc/uucp/passwd 863f779b43680f81799688e91c18a164047a1a8e9dfff650881e50ba35530ed1 -r--r--r-- root wheel 141 B /private/etc/uucp/port These listings rely on the snapshot index and do not fetch chunks from the repository, however it is possible to actually look at the file content with the cat subcommand:\n$ plakar cat 9:/etc/uucp/passwd # # The passwd configuration file # # # Specify a login name and password that a system can use when logging in. # uucp_tst sekret Now let\u0026rsquo;s make a change to that file and push it again:\n$ sudo sed -i \u0026#39;\u0026#39; -e \u0026#39;s/sekret/notsosecret/\u0026#39; /etc/uucp/passwd $ plakar push /private/etc open /private/etc/krb5.keytab: permission denied open /private/etc/aliases.db: permission denied open /private/etc/racoon/psk.txt: permission denied open /private/etc/security/audit_user: permission denied open /private/etc/security/audit_control: permission denied open /private/etc/sudoers: permission denied open /private/etc/sudo_lecture: permission denied open /private/etc/master.passwd: permission denied open /private/etc/openldap/slapd.conf.default: permission denied open /private/etc/openldap/DB_CONFIG.example: permission denied b41802cb-9424-47eb-8186-2847678a8646: OK From there, it is possible to perform an inode diff:\n$ plakar diff 9 b - drwxr-xr-x root wheel 160 B 2020-01-01 08:00:00 +0000 UTC /private/etc/uucp + drwxr-xr-x root wheel 160 B 2021-03-27 22:38:17.685871971 +0000 UTC /private/etc/uucp - -r--r--r-- root wheel 133 B 2020-01-01 08:00:00 +0000 UTC /private/etc/uucp/passwd + -r--r--r-- root wheel 138 B 2021-03-27 22:38:17.685791098 +0000 UTC /private/etc/uucp/passwd This shows that the /private/etc/uucp directory has had its date change, and that /private/etc/uccp/passwd has had its date and size change, and from there it is possible to request a diff of the file with the diff subcommand:\n$ plakar diff 9 b /private/etc/uucp/passwd --- 98b6658b-a975-47c4-a38f-f958d8d7359f:/private/etc/uucp/passwd +++ b41802cb-9424-47eb-8186-2847678a8646:/private/etc/uucp/passwd @@ -6,5 +6,5 @@ # Specify a login name and password that a system can use when logging in. # -uucp_tst sekret +uucp_tst notsosecret The snapshots are not incremental so it is possible to generate diffs between any of them.\nRemoving old snapshots # Even though snapshots of similar content do not consume much space, there\u0026rsquo;s no use in keeping a lot of old snapshots hanging around. Snapshots can be removed from the repository with the rm subcommand:\n$ plakar ls 98b6658b-a975-47c4-a38f-f958d8d7359f [2021-03-27T22:29:09Z] (size: 3.0 MB, files: 230, dirs: 39) b41802cb-9424-47eb-8186-2847678a8646 [2021-03-27T22:38:22Z] (size: 3.0 MB, files: 230, dirs: 39) $ du -sh ~/.plakar 2.2M /Users/gilles/.plakar $ plakar rm 9 98b6658b-a975-47c4-a38f-f958d8d7359f: OK $ du -sh ~/.plakar 2.1M /Users/gilles/.plakar $ plakar rm b b41802cb-9424-47eb-8186-2847678a8646: OK $ du -sh ~/.plakar 0B /Users/gilles/.plakar The repository keeps chunks as long as they are referenced by at least one snapshot, but removes them when they become orphans which is why the repository becomes empty after the last snapshot is removed.\nNamespaces # Sometimes I want to make sure that my backups are fully disjoint one from another, not sharing their snapshots and deduplication. The plakar repository supports namespaces so that I can keep related snapshots together and apart from others.\nFor example, I have backups for poolp.org and for hypno.cat and I want to manage the snapshots separately:\n$ plakar -namespace poolp.org push data 351348b3-8dc3-4caa-b0be-334c43f4fd13: OK $ plakar -namespace hypno.cat push data 1fa2dac2-8096-483a-a2d3-2a78ce3706ce: OK $ plakar -namespace poolp.org ls 351348b3-8dc3-4caa-b0be-334c43f4fd13 [2021-03-27T22:49:58Z] (size: 896 kB, files: 79, dirs: 4) $ plakar -namespace hypno.cat ls 1fa2dac2-8096-483a-a2d3-2a78ce3706ce [2021-03-27T22:50:40Z] (size: 896 kB, files: 79, dirs: 4) $ Stores # In addition, I may want to provide an alternate path to the repository because ~/.plakar is not ideal, plakar supports providing alternate repository path with the -store option:\n$ plakar -store /var/backups/gilles push data f6cc2508-ea59-4552-aa4b-27900f0a9781: OK $ plakar -store /var/backups/gilles ls f6cc2508-ea59-4552-aa4b-27900f0a9781 [2021-03-27T22:52:47Z] (size: 896 kB, files: 79, dirs: 4) But this doesn\u0026rsquo;t stop here \u0026hellip;\nplakar server # So far I only showed plakar working on a local repository, it however supports a network mode.\nIt is possible to start a plakar server on a machine with:\n$ plakar server 192.168.0.1:2222 $ and on another machine, point the repository to that server:\n$ plakar -store plakar://192.168.0.1:2222 push /bin 3ac8f3ff-e723-46c3-9ae7-4b11bb27f3b2: OK $ The plakar server can use -store itself, so it is possible to configure it to store at at particular location:\n$ plakar -store /var/backups server 192.168.0.1:2222 $ and because I made a monster, it is possible that this location is a different plakar server, effectively turning a plakar instance into a proxy:\nbox1$ plakar server 192.168.0.1:2222 box1$ box2$ plakar -store plakar://192.168.0.1:2222 server 192.168.0.2:2222 box2$ box3$ plakar -store plakar://192.168.0.2:2222 push /bin 998964ac-d3f6-4bf3-b784-50ba616e0776: OK $ From the local user point of view, pushing to a local plakar, to a remote plakar, or even to a remote plakar proxying to another plakar work the same.\nIs it released ? # Nope, not yet.\nThis is a PoC and my first real project in Golang, so I need a bit of time to make sure it is in a state that I\u0026rsquo;m not ashamed to release.\nIf you are a sponsor and want to give it a try, just mail me and I\u0026rsquo;ll arrange that.\nWhat\u0026rsquo;s next ? # A lot of things really\u0026hellip;\nFirst of all, every single command needs a bit of polishing to have a nice feel because the output is not always the best to work with.\nThen, I have not implemented any caching whatsoever so even though it\u0026rsquo;s not dead slow, a lot of operations that could be skipped are always performed making it slower than it should.\nAlso, the server mode doesn\u0026rsquo;t use encryption at this point and while it is ok on a personal network, I want to be able to deploy a plakar server on a dedicated server at some provider and use it safely.\nFinally, I began building an UI to browse snapshots and this requires building a global index of snapshots, which is something I haven\u0026rsquo;t tackled yet. At this point, each snapshot has its own index but I\u0026rsquo;d like to be able to have a search box which looks up for stuff through all snapshots.\nI also have other private plans with this which I\u0026rsquo;ll discuss later ;-)\n","date":"26 March 2021","permalink":"/posts/2021-03-26/march-2021-backups-with-plakar/","section":"Posts","summary":"TL;DR: I wrote a backup utility called plakar. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !)","title":"March 2021: backups with Plakar"},{"content":" TL;DR: I converted nooSMTPD to libtls and implemented SMTP ciphers, curves and protocols selection. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !)\nNothing new in OpenSMTPD-portable # eric@ sent a libtls-conversion diff to tech@ last month but there hasn\u0026rsquo;t been much progress since then.\nI tested the diff extensively and I\u0026rsquo;m currently waiting for it to be merged to OpenBSD so I can bring it to OpenSMTPD-portable.\nnooSMTPD is libtls-enabled # In nooSMTPD, I did the libtls-conversion by applying the diff and making the appropriate changes to the libtls compat layer.\nI switched my machines from my custom FrankenOpenSMTPD to nooSMTPD and been running it since late January, with no regressions.\nIt\u0026rsquo;s interesting to note that while I ran OpenSMTPD from upstream repository without the autoconf build, I\u0026rsquo;m running nooSMTPD from autoconf which led to a lot of swearing as it was never meant to be used on OpenBSD. I fixed a few unpleasant things which I\u0026rsquo;ll be able to push to OpenSMTPD-portable as well.\nThe smtp tls global configuration # OpenSMTPD allows setting up SMTP global configuration with the smtp global keyword:\nsmtp max-message-size \u0026#34;30M\u0026#34; smtp sub-addr-delim \u0026#34;+\u0026#34; Because it may be desireable to setup an alternate list of ciphers, the ciphers keyword was introduced a long time ago and a global option allows overriding the default ciphers:\nsmtp ciphers \u0026#34;HIGH:!aNULL:!MD5\u0026#34; In nooSMTPD, I decided to introduce the tls keyword for TLS related options.\nThe libtls conversion unlocks several TLS-related features that may be confusing below smtp, without any reference to TLS. For instance, I have implemented TLS protocol selection and unless tls appears in the configuration this will result in:\nsmtp protocols [...] which is much more confusing than:\nsmtp tls protocols [...] as it is unclear that the protocols are TLS related.\nAs a result, I moved ciphers below smtp tls so in nooSMTPD it reads as:\nsmtp tls ciphers \u0026#34;HIGH:!aNULL:!MD5\u0026#34; I\u0026rsquo;ll suggest this change to OpenSMTPD too.\nTLS ciphers selection # As mentionned above, I have changed the grammar to introduce a tls namespace to the smtp keyword:\nsmtp tls ciphers \u0026#34;HIGH:!aNULL:!MD5\u0026#34; Because libtls supports cipher suite groups, the secure, compat, legacy or insecure groups can be used in place of a cipher suite, and since secure is the default, the global cipher selection only needs to be specified for other cases:\nsmtp tls ciphers \u0026#34;compat\u0026#34; With this done, I also added the ability to setup the ciphers at the listener level:\nlisten on 0.0.0.0 tls ciphers \u0026#34;compat\u0026#34; listen on 192.168.0.1 tls ciphers \u0026#34;legacy\u0026#34; # use on legacy subnet This was committed to nooSMTPD and I\u0026rsquo;ll submit a diff to OpenSMPTD when libtls-conversion is done upstream.\nTLS curves selection # The selection of curves was never merged in OpenSMTPD, as a result it is not possible to select specific curves at either one of the global or listener level. The default curves (X25519,P-256,P-384) in libtls are sensible choices but some use-cases require restricting to a specific curve, or even an alternate one.\nI introduced a curves option which allows specifying curves to override the default:\nsmtp tls curves \u0026#34;X25519\u0026#34; While at it, I added the ability to setup the curves at the listener level:\nlisten on 0.0.0.0 tls ciphers \u0026#34;X25519\u0026#34; listen on 192.168.0.1 tls curves \u0026#34;P-256,P-384\u0026#34; This was committed to nooSMTPD and I\u0026rsquo;ll also submit a diff to OpenSMPTD when libtls-conversion is done upstream.\nTLS protocols selection # Just like for curves, the selection of a TLS protocol was never merged in OpenSMTPD.\nI introduce a protocols option which allows specifiying protocols to override the default:\nsmtp tls protocols \u0026#34;tlsv1.3:tlsv1.2\u0026#34; And again, I added the ability to setup the protocol at the listener level:\nlisten on 0.0.0.0 tls protocols \u0026#34;tlsv1.3\u0026#34; Guess what\u0026hellip; Yes, this was also committed to nooSMTPD and I\u0026rsquo;ll submit a diff to OpenSMPTD when libtls-conversion is done upstream.\nGlobal smtp tls grammar needs to be reworked # The smtp tls keyword was implemented as a placeholder for the ciphers, curves and protocols options, but it was not implemented the way it should.\nBasically, ciphers, curves and protocols should be options to tls so that it is possible to write:\nsmtp tls ciphers \u0026#34;secure\u0026#34; protocols \u0026#34;secure\u0026#34; At the time being, this is not possible and it must be expressed as follows:\nsmtp tls ciphers \u0026#34;secure\u0026#34; smtp tls protocols \u0026#34;secure\u0026#34; This requires a bit of grammar refactor which I\u0026rsquo;ll be working on in March.\nListener tls grammar was reworked # The listener tls keyword suffered from the same issue as the global smtp tls keyword.\nInitially, it was implemented in such a way that ciphers, curves and protocols were standalone options. This allowed avoiding the need to repeat tls before every option, you needed one tls on the listener so it was flagged as TLS-enabled and the options would check for the flag. This worked but it was not the right way to do it.\nI reworked the grammar for tls in listeners so that ciphers, curves and protocols are tls options. This avoids having to repeat tls before every keyword, and it allows relying on the grammar rather than on a flag to determine if the configuration is valid:\nlisten on 0.0.0.0 tls ciphers \u0026#34;secure\u0026#34; curves \u0026#34;P-256,P-384\u0026#34; protocols \u0026#34;secure\u0026#34; I\u0026rsquo;m happy with the result.\nWhat\u0026rsquo;s next ? # I will rework the grammar for global settings and will implement these options for the MTA layer, as these options were done only for the incoming path.\nI intend to publish a technical article in March regarding software design and data.\nI might unveil a project that\u0026rsquo;s unrelated to computer programming too :-)\n","date":"28 February 2021","permalink":"/posts/2021-02-28/february-2021-noosmtpd-libtls-conversion-ciphers-curves-and-protocols/","section":"Posts","summary":"TL;DR: I converted nooSMTPD to libtls and implemented SMTP ciphers, curves and protocols selection. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe !","title":"February 2021: nooSMTPD libtls-conversion, ciphers, curves and protocols"},{"content":" TL;DR: I do LoFi now, eric@ revived some libtls conversion work I did a while back, I worked on UNIX-domain sockets support in OpenSMTPD, a few words about nooSMTPD Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !) with a playlist of LoFi tracks that I mix.\nI\u0026rsquo;ll try to insert a new one in every monthly report\u0026hellip; if time allows ;-)\nlibtls-enabled OpenSMTPD # In January 2020, I worked on converting OpenSMTPD to the libtls API so that I could simplify TLS support and bring new features that I didn\u0026rsquo;t want to implement with the libssl API.\nThe diff was never merged into OpenBSD and ended up rotting in a private branch. This was unfortunate because I spent time making this work for portable too, building an OpenSSL-enabled libtls that would allow distributions not shipping LibreSSL to still benefit from this TLS simplification.\nEric Faurot (eric@) has recently picked up my work and pushed it even further, managing to make OpenSMTPD completely libssl-agnostic while I still needed bits for the privsep crypto engines. He submitted a diff to tech@ which is pending review. Once his diff gets committed in OpenBSD, I\u0026rsquo;ll rework the portable code to make it work again so this benefits users on other systems.\nI\u0026rsquo;ve been suggested by eric@ and a tech@ reader to look at libretls. The problem is that libretls is not yet available in the package repositories of all distributions supported by OpenSMTPD, and eric@ also brought changes in LibreSSL that libretls will need to catch up with before I can rely on it. I think the best approach is to rely on the standalone code I already wrote and which is shipped with OpenSMTPD, then once libretls is widely available I can reassess the situation.\nlibtls-related features # I have three features related to the libtls change that are already written, rotting in their own branches, and that I will submit once the TLS diff is committed. They make it possible to configure the ciphers, curves and TLS versions supported for each listeners.\nThe rationale behind this is that some setups require stronger (or specific) crypto constraints for specific networks. This change would allow declaring that a specific interface requires use of a different set of ciphers or TLS version, and allow for example to have an interface with the default settings for Internet and one with extra-paranoid settings for a specific MX.\nI also had started working on other features, like certificates fingerprinting or OCSP stapling, but I stopped as I didn\u0026rsquo;t want to have too much code building upon an uncommitted diff.\nI\u0026rsquo;ll resume working on these once eric@ commits the libtls conversion.\nsmtpd(8) now binds UNIX-domain sockets # At the exception of the local enqueuer, OpenSMTPD only supports SMTP sessions over network connections. This shows in the configuration file as it is only possible to use listen directives with a network interface parameter\u0026hellip; or with the socket keyword which is handled as a special case for the local enqueuer.\nWhen the daemon starts, it creates a UNIX-domain socket to use as its control socket for the smtpctl(8) utility. This was initially meant to accept control requests from root to interact with smtpd(8) at runtime, but later came the need for local enqueueing and the control socket was used for that purpose too.\nA special \u0026ldquo;enqueue\u0026rdquo; control command was implemented in smtpd(8). That command allows smtpctl(8) to request from smtpd(8) that it establishes an internal SMTP session and returns the file descriptor. The smtpctl(8) utility then uses that descriptor to pretend that the socket was actually connected directly to the SMTP server, and it toggles into an enqueue mode where it operates as an SMTP client.\nThis has three side-effects:\nBecause everyone is supposed to be able to enqueue a local mail, the control socket cannot have strict permissions and has to be world-writeable. Since it is also used for privileged control commands, the daemon needs to explicitly check that the client is only allowed to access the enqueue mode if the user is not privileged:\n$ ls -l /var/run/smtpd.sock srw-rw-rw- 1 root wheel 0 Dec 28 06:38 /var/run/smtpd.sock $ smtpctl show queue smtpctl: need root privileges Then, because there is some magic involved to request the file descriptor and toggle in enqueue mode, the control socket is not really connected directly to the SMTP server. It is not possible to connect to it with netcat and talk SMTP as it expects a control command sent with the imsg(3) framework. The server will just ignore the client as it doesn\u0026rsquo;t speak its protocol, and the connection must be established by smtpctl(8) requesting the enqueue mode (or a clone that knows how to do so):\n$ nc -U /var/run/smtpd.sock HELO ?????? ^C $ Finally, users expect to be able to enqueue mail even if the daemon is not available. This requires the local enqueuer to support an offline queue, which itself relies on having an executable setgid to the same group as the offline directory. Because smtpctl(8) is used for both control and enqueuing, it ends up having the setgid bit itself:\n-r-xr-sr-x 1 root _smtpq 217696 Dec 23 06:42 /usr/sbin/smtpctl A year ago, I told eric@ that I believed this was a poor decision made in the early days. Both the control and the enqueuing code could be made simpler and stricter if they were split apart.\nIf local enqueuing had its own dedicated socket which established a connection to an SMTP session, then any SMTP client that knew how to connect to a UNIX-domain socket could be used as the local enqueuer. OpenSMTPD even ships with one\u0026hellip; smtp(1).\nThis would simplify smtpctl(8) as it would no longer need an enqueue mode and a builtin SMTP client. It would also simplify smtpd(8) as it would no longer need to implement the internal SMTP session and fd passing logic, but would also no longer need to check if a user can or can\u0026rsquo;t run a command: if it\u0026rsquo;s not root, it can\u0026rsquo;t. The control socket and smtpctl(8) could both be restricted to root as they would only be used for control requests.\nIf you don\u0026rsquo;t see the benefit behind this, many years ago there were two bugs that allowed local users to crash smtpd(8) from smtpctl(8). One affected the control command counter, which a user could not have messed up with if control commands were restricted to root, and the other affected the fd passing for the enqueue mode, which would not even exist if enqueuing had its own dedicated socket instead of being a control command.\nThis sounds nice but converting local enqueuing to use its own dedicated UNIX-domain socket is trickier than it seems. The technical aspect is simple, you just bind a UNIX-domain socket instead of a TCP socket, but it raises a lot of other questions regarding the behavior of the daemon. I will discuss that in a future post as this is still being sorted out.\nWhat could be done right now and that wouldn\u0026rsquo;t raise questions was to teach smtpd(8) how to bind a UNIX-domain listener. It makes it possible to declare listeners that have UNIX-domain sockets as their endpoints and which plain SMTP clients can connect to:\n$ cat /etc/mail/smtpd.conf |grep listen listen on socket \u0026#34;/tmp/foobar.sock\u0026#34; listen on socket \u0026#34;/tmp/barbaz.sock\u0026#34; $ nc -U /tmp/foobar.sock 220 debug.poolp.org ESMTP OpenSMTPD ^C $ printf \u0026#34;subject: test\\n\\ntest\u0026#34; | smtp -s /tmp/barbaz.sock gilles@poolp.org gilles@poolp.org: EOM: 250 2.0.0 593c830c Message accepted for delivery $ It doesn\u0026rsquo;t solve the local enqueuer issues as it still uses the control socket, but it allows regular SMTP clients to submit mail over a UNIX-domain socket without relying on control commands. Moving forward in this direction, a new local enqueuer can be written that doesn\u0026rsquo;t rely on the enqueue control command, which ultimately leads to enqueuing being removed from smtpctl(8) and more restrictive permissions on the control socket.\nI already showed the diff to eric@ but I will send it to OpenBSD next week.\nsmtp(1) now talks Unix-domain socket # Following the UNIX-domain sockets listeners idea, I thought it would be nice to do the client side too.\nA while back, eric@ wrote smtp(1) which is a simple utility to submit mail to SMTP servers on the command line:\n$ cat\u0026lt;\u0026lt;EOF | smtp -s localhost gilles@poolp.org Subject: foo bar EOF gilles@poolp.org: EOM: 250 2.0.0 5cc0b0d8 Message accepted for delivery $ The smtp(1) client didn\u0026rsquo;t know how to connect to a UNIX-domain socket so I fixed that:\n$ cat\u0026lt;\u0026lt;EOF | smtp -s /tmp/smtpd.sock gilles@poolp.org Subject: foo bar EOF gilles@poolp.org: EOM: 250 2.0.0 d73542e2 Message accepted for delivery $ This allowed me to test my diff on the server side, but it also made me realize that it could be used as the base for a new local enqueuer outside of smtpctl(8).\nThe current local enqueuer is based on the femail MUA, which was kind of hacked here and there to fit in smtpctl(8) and work with its enqueue mode. It reads the mail from its standard input, does some sanitizing and crafting, then submits it using the SMTP protocol on a file descriptor which points to an SMTP session.\nIn a world where local enqueuing doesn\u0026rsquo;t require smtpctl(8) entering a special enqueue mode, it would be fairly easy to use smtp(1) as a base to write a new enqueuer. It already reads mail from standard input, it already knows how to speak SMTP and with my diff it knows how to connect to a UNIX-domain socket. All that would be missing is adding the sanitizing and crafting bits which we already have.\nI also already showed the diff to eric@ and will send it to OpenBSD next week with the listeners one.\nnooSMTPD: Not OpenBSD\u0026rsquo;s OpenSMTPD # I didn\u0026rsquo;t want to talk about this yet but since people have spotted the repository and are making assumptions, I need to explain what that is.\nIn December, I started working on an MTA. It is based on OpenSMTPD, but it takes a different direction and has different goals.\nOpenSMTPD is a general purpose MTA, written primarily for OpenBSD, which needs to fit the base system and care about legacy and how changes affect the user base.\nnooSMTPD doesn\u0026rsquo;t have these constraints. I\u0026rsquo;m writing it primarily for myself and will happily break legacy behaviors and change the configuration file every two months if it makes things easier for me. I intend to support some advanced features found in commercial MTA and that are unlikely to be accepted in OpenBSD because\u0026hellip; OpenSMTPD is a general purpose MTA.\nOpenSMTPD benefits from the work I do there as I share all diffs, but in some cases they are unsuitable for OpenBSD and this is where things diverge between the two, it will contain diffs that don\u0026rsquo;t make it into OpenSMTPD.\nI\u0026rsquo;ll write about it more in details in the future, I just wanted to clarify that this isn\u0026rsquo;t the fork some people think it is. I have exchanged multiple diffs with eric@, millert@, martijn@ and even tech@ since December, and I continue to talk about OpenSMTPD changes with eric@ every week.\nI just have ideas for a different project :-)\nbreaking changes in nooSMTPD # I have killed the dead.letter feature which allows OpenSMTPD to save a copy of a mail when it fails to submit it to the enqueuer. Modern MUA do not make use of it, it is solely used by legacy MUA, and I always thought this was not the job of the MTA to handle it. It was also a vector of attack in the past.\nI have killed delivery to the root user for all mail delivery agents, the only way root can receive a mail is through an alias to an unprivileged account.\nI have merged all of the diffs from last month, including the safety net that detects injection of a custom MDA for dispatchers not allowing exec.\nassorted portability improvements and cleanups in nooSMTPD # I reworked some code patterns confused compilers and led to false positives in warnings. For example, the following construct was found in multiple places, and led compilers not knowing that fatal() never returned into assuming that p might be uninitialized in the call to barbaz():\nvoid foobar(x) { char *p; switch (x) { case 0: p = \u0026#34;a\u0026#34;; break; case 1: p = \u0026#34;b\u0026#34;; break; default: fatal(\u0026#34;die.\u0026#34;) } barbaz(p); } I also replaced some functions, such as ctime() and localtime(), with their reentrant versions ctime_r() and localtime_r(). nooSMTPD is not threaded but this raises alerts from analysis tools and, while doing something just so a tool would shut up is not a good rationale, in this case there\u0026rsquo;s no real downside and it saves me from having to keep flagging stuff as false positives.\nWhat\u0026rsquo;s next ? # Moving forward with all of the above.\n","date":"31 January 2021","permalink":"/posts/2021-01-31/january-2021-opensmtpd-libtls-conversion-and-unix-domain-sockets-support-noosmtpd/","section":"Posts","summary":"TL;DR: I do LoFi now, eric@ revived some libtls conversion work I did a while back, I worked on UNIX-domain sockets support in OpenSMTPD, a few words about nooSMTPD Let\u0026rsquo;s start with some LoFi # Relax.","title":"January 2021: OpenSMTPD libtls conversion and UNIX-domain sockets support, nooSMTPD"},{"content":" TL;DR: In this article, I explain what is an MDA and how to write a custom one from scratch using only shell scripting. What is a Mail Delivery Agent (MDA) ? # When a mail enters an SMTP server it is initially stored in the server queue before being moved to its next destination.\nIf the destination is a remote machine, the mail goes back to a Mail Transfer Agent (MTA) and is transferred elsewhere, but if the destination is a local user then it is passed to an MDA for delivery.\nAn MDA is a program that is in charge of delivering a mail to a local user and reporting to the SMTP server if the delivery was successful or not.\nThe MDA interface is somewhat standard so that if you write an MDA that doesn\u0026rsquo;t take advantage of features specific to a particular SMTP server, it will work with others fine. This is why tools such as procmail or fdm can be used with OpenSMTPD, Postfix, Sendmail, notQmail or Exim indistinctly.\nAn MDA can be written in any language but to keep the examples simple and understandable by all, I decided to illustrate with shell scripting which is the poorest choice of a \u0026ldquo;language\u0026rdquo; you could make to write an MDA, be warned.\nHow does it work ? # An MDA works in a straightforward way.\nIt is a program that reads its input from stdin and reports success or failure through its exit(3) status.\nBased on this description, the following shell script is a valid MDA as far as any SMTP server is concerned:\n#! /bin/sh cat exit $? What it does is call the cat command, which consumes its input from stdin and writes it back to stdout, then propagate its exit value as that\u0026rsquo;s how cat reports its success or failure.\nThis MDA does a poor job at MDA-ing because all it does is consume the input without writing it anywhere useful. As a result, the MDA will report success when cat is done reading the mail and echo-ing it back to stdout but the mail content will be gone forever.\nThis MDA could be made more useful by redirecting stdout to a file so that the input does not get lost:\n#! /bin/sh cat \u0026gt;\u0026gt; /tmp/mail-archives exit $? In this version, the /tmp/mail-archives file is opened for append and cat has its stdout redirected to it. Every time the MDA gets called for a mail, the mail gets appended to that file.\nThis does a good job at explaining what is expected from an MDA, unfortunately it is a broken example.\nFirstly because an MDA is executed for each mail being delivered, possibly simultaneously, and in the absence of a locking mechanism this might result in concurrent writes mangling its content.\nSecondly because an MDA is executed with the privileges of the recipient user which means that every possible recipient needs to have write access to that file, with all the downsides you can imagine.\nLet\u0026rsquo;s see how to write something better.\nThe stdin stream # The first thing to understand is that the mail is received on the standard input, stdin, as a stream of lines which are interrupted by an EOF.\nThere is no back and forth protocol of any kind, the MDA is the recipient of a unidirectional stream of input and does not have a side-channel to let the server know of what happens during the processing of this stream.\nIt consumes stdin, does something which the server doesn\u0026rsquo;t know the details of, and tells the server when its done by exiting with a value indicating success or failure.\nIt doesn\u0026rsquo;t get much simpler.\nsysexits(3) # To report success or failure, the MDA operates like a standard unix program: it exits with 0 to indicate success and anything else to report failure.\nSince that is the only way to report information to the server, MDA can make use of sysexits(3) status code to let the server interpret the reason behind a failure \u0026hellip; or at least goes the theory.\nIn practice these codes are not used much and they don\u0026rsquo;t impact the SMTP server behaviour much. Using the wrong code will result in nothing bad happening except maybe an inaccurate error message if the SMTP server does try to interpret them.\nMDA privileges # An MDA is not a long-running process that gets started once and runs forever. Every time a delivery happens for a recipient, a process is fork(3)-ed with the recipient privileges in order to execute an instance of the MDA.\nIf gilles@poolp.org receives a mail and jules@poolp.org receives a mail, and they use the same MDA, then there will be one process owned by gilles and one process owned by jules, both running concurrently (or at least they might).\nTherefore, the MDA must not assume access to resources that require specific privileges. It must access resources that are available to the recipient user it was executed for. In other words, the MDA for gilles may access resources that gilles could access using a shell if he had one.\nThe example MDA above was bad because the /tmp/mail-archives must be shared between all users for it to work and this implies open permissions (770 if users share the same group, 777 otherwise). A simple change to create a recipient-specific file would have been enough to avoid this issue and allow a more restrictive permission of 700:\n#! /bin/sh cat \u0026gt;\u0026gt; /tmp/mail-archives.`whoami` exit $? Note that the use of whoami here is not a good idea, it is here to back my point, but we\u0026rsquo;ll discuss that in the next section.\nAlternatively, the file could be moved to the recipient user home directory to achieve the same result:\n#! /bin/sh cat \u0026gt;\u0026gt; ~/mail-archives exit $? In that case, we\u0026rsquo;d be ok permission-wise but not with regard to concurrency as lack of locking would mean concurrent deliveries to the same user might mangle the file.\nThe MDA environment # An MDA inherits an environment from the SMTP server allowing it to determine who it is running for, what was the e-mail address of the sender and recipient, and other useful information.\nFirst comes the shell environment variables, the ones you get when logging in to an account on Unix systems:\nThe PATH variable contains the default PATH allowing the MDA to execute standard programs on the host system. In my example MDA above, I didn\u0026rsquo;t write /bin/cat because /bin was exposed in the MDA PATH.\nThe SHELL variable contains the shell that was executed to run the MDA.\nThe HOME variable contains, well, the path to the current recipient user home directory.\nThe LOGNAME and USER variables contain the recipient user. This is the owner of the current process, the one whose HOME is set for.\nThen comes the MDA specific ones:\nThe DOMAIN variable contains the domain name of the recipient at the time of the SMTP session (that is before any aliasing).\nThe LOCAL variable contains the user-part of the e-mail address that was used during the SMTP session (that is before any aliasing). ${LOCAL}@${DOMAIN} is the e-mail address that was used in the SMTP session.\nThe RECIPIENT variable contains the recipient e-mail address following all aliasing.\nThe SENDER variable is set to the e-mail address of the sender or to an empty string in case of a MAILER-DAEMON bounce.\nThe EXTENSION is set to the extension if there\u0026rsquo;s any used with the recipient e-mail address. The extension is what follows + and precedes @ in e-mail addresses such as gilles+1@poolp.org\nAs you can see, there\u0026rsquo;s a few of them covering about all session informations needed to properly execute an MDA. If we were to take a silly broken example from above:\n#! /bin/sh cat \u0026gt;\u0026gt; /tmp/mail-archives.`whoami` exit $? We could replace it as follows to make use of the environment:\n#! /bin/sh cat \u0026gt;\u0026gt; /tmp/mail-archives.${USER} exit $? It would be better, but what if we wrote a better MDA, one that is usable without concurrency or permission issues ?\nWriting a maildir-like MDA # In this section, we will write an MDA that delivers mail to a maildir-like structure.\nContrarily to the previous examples, this one will ensure that there are no issues if multiple instances run concurrently for one or many users.\nFirst, we\u0026rsquo;ll write the base for our MDA, the stdin reading loop:\n#! /bin/sh cat exit $? Then, we need to figure where the maildir should be located and create the directory structure if needed, applying restrictive permissions. Because we know that the MDA is provided with a HOME environment variable, we can make use of that:\n#! /bin/sh MAILDIR=${HOME}/Maildir umask 077 test -d ${MAILDIR} || mkdir ${MAILDIR} test -d ${MAILDIR}/cur || mkdir ${MAILDIR}/cur test -d ${MAILDIR}/new || mkdir ${MAILDIR}/new test -d ${MAILDIR}/tmp || mkdir ${MAILDIR}/tmp cat exit $? At this point, we have dodged issues with two different users running the same MDA since we deliver to a recipients home directory but we have to worry about the same MDA being executed concurrently for the same user.\nThis could be handled with a lock, as is done for mbox-style mailboxes that use a single file per user, but maildir has a more elegant solution to solve this: generating unique filenames in a temporary directory, then atomically renaming to the destination directory.\nBy writing to a temporary directory, the file is not yet visible by the user until it is finished being written, at which point the atomic rename makes it visible. If any error happens in between, the file remains in the temporary directory and can be purged after a while, but there is no risk of exposing a partial file.\nWe\u0026rsquo;ll adapt the MDA to save a copy of the mail from stdin to a temporary file with a unique name. To construct the name, I decided to go with a unix timestamp, followed by a 64-bits random value and the local hostname. This ensures that there can\u0026rsquo;t be collisions if generated on two machines sharing the same network storage due to the hostname, and that collisions are unlikely on the same host as they would require a 64-bits collision of a random value happening within a second. Once the temporary file has been written and cat reported success, it is moved to the ~/Maildir/new which makes it visible to clients:\n#! /bin/sh MAILDIR=${HOME}/Maildir umask 077 test -d ${MAILDIR} || mkdir ${MAILDIR} test -d ${MAILDIR}/new || mkdir ${MAILDIR}/new test -d ${MAILDIR}/tmp || mkdir ${MAILDIR}/tmp # ie: 1609171573.deb626c88c2da87f.laptop FILENAME=`date +%s`.`openssl rand -hex 8`.`hostname` cat \u0026gt; ${MAILDIR}/tmp/${FILENAME} \u0026amp;\u0026amp; \\ mv ${MAILDIR}/tmp/${FILENAME} ${MAILDIR}/new/${FILENAME} exit $? We can test that it works by running the command outside the SMTP server:\ngilles@mba ~ % ls -l ~/Maildir ls: /Users/gilles/Maildir: No such file or directory gilles@mba ~ % sh mda.sh HOLA ! gilles@mba ~ % ls -l ~/Maildir/new total 8 -rw------- 1 gilles staff 7 Dec 29 00:52 1609199557.db7be737b9423c87.mba gilles@mba ~ % cat ~/Maildir/new/1609199557.db7be737b9423c87.mba HOLA ! gilles@mba ~ % This MDA does not perform any check on the mail content and copies the stream as is but nothing prevents your MDA from doing arbitrary work.\nThe MDA could parse the headers to extract information, collect statistics, store the content in a database, or do anything including apply stupid rules like not allowing a delivery on rainy days.\nAs much as the SMTP server knows, this is a blackbox that will acknowledge having delivered, but how it did is left to the MDA.\nConclusion # I wrote this because many people have had questions regarding custom MDA, how to write one and what they do.\nI might have missed stuff, if you have questions feel free to ask and I will expand this article.\n","date":"29 December 2020","permalink":"/posts/2020-12-29/writing-a-custom-mail-delivery-agent/","section":"Posts","summary":"TL;DR: In this article, I explain what is an MDA and how to write a custom one from scratch using only shell scripting. What is a Mail Delivery Agent (MDA) ?","title":"Writing a custom Mail Delivery Agent"},{"content":" TL;DR: Crafted the OpenSMTPD 6.8.0p1 release Fixed several bugs in the way Proposed a few OpenSMTPD improvements to OpenBSD Working on my OpenSMTPD book Let\u0026rsquo;s start with some LoFi # Relax.\nAbout sponsorship # First of all, I must say how grateful and happy I am that people value my work enough to trust and sponsor me, I appreciate it a lot and it is a huge motivation booster.\nI initially thought that after a few months I might have about 10 sponsors, but here we are after a year or so with over 50 sponsors !\nPlease let me know what you want me to work on, I may or may not do it (based on my interest and skills) but at least I\u0026rsquo;ll have an idea how to attract more sponsors as I\u0026rsquo;m clueless.\nThank you all and hopefully when December 2021 hits, I can give a three-digits number and spend more time on coding and writing for a hobby :-)\nPreparing the OpenSMTPD 6.8.0p1 release # I spent a lot of time preparing the OpenSMTPD 6.8.0p1 release that SHOULD have been released in November shortly after OpenBSD 6.8 was out, but which I couldn\u0026rsquo;t find time to work on as I was contracted for some private work.\nAs I prepared the release, I had to fix several issues, some specific to the portable version and some that have been fixed upstream at OpenBSD.\nThe 6.8.0p1 release was published today and contains the fixes for the issues mentioned below.\nFixed macOS build # The build on macOS was broken due to a missing OpenSSL include.\nA user submitted a pull request that fixed the build but another user reported that this was not enough for older macOS that lack the clock_gettime() call which we use in the profiling code.\nI\u0026rsquo;m not familiar enough (at all) with macOS to figure out which versions are \u0026ldquo;old\u0026rdquo; but this seems to fall into the \u0026ldquo;legacy\u0026rdquo; bucket and as far as the latest releases are concerned there is no known problems.\nI don\u0026rsquo;t intend to support all legacy systems out there but since fixing the clock_gettime() issue might be beneficial to portability outside of macOS, I started tackling the issue and am almost done.\nFixed get_progname() bug # A bug was introduced over a release ago which caused OpenSMTPD to improperly log the process name in maillog on some systems.\nIt was weird because the process names were correctly reported in the ps output, they just weren\u0026rsquo;t in the logs.\nI traced it back to a broken feature detection in configure.ac leading to the setprotctitle() workaround not working correctly.\nIssue was fixed and committed to the portable repository.\nImplemented an ECDSA engine for OpenSSL (sponsored development) # When OpenSMTPD 6.6.0 was released a year ago, it brought support for server-side ECDSA certificates but with the caveat that it only worked with LibreSSL. OpenSMTPD would build with OpenSSL but disable server-side ECDSA support.\nPeople assumed that this was done because I wanted to push LibreSSL as otherwise I could have just added a few #ifdef and handled both, but this was of course not the reason.\nAs I wrote in June 2019, the crypto operations in OpenSMTPD are done in a special way compared to what\u0026rsquo;s commonly seen. As a result, adding support to ECDSA was not just about slapping a few #ifdef and calling the proper function but required writing a custom crypto engine which, unluckily for me, uses an API that diverged between LibreSSL and OpenSSL.\nI didn\u0026rsquo;t have interest in doing that work because it\u0026rsquo;s not very pleasant, but people kept asking for it and a user contacted me in private to ask if I was willing to accept a sponsored development to do that work.\nI accepted and worked on it for a few days until I could start OpenSMTPD on an Ubuntu with OpenSSL and use ECDSA certificates. A few people tested the diff and reported success so it was committed to the portable repository.\nI\u0026rsquo;d like to thank again that user who wishes to remain anonymous, this sponsored development helped me replace my laptop with a new MacBook Air which is lovely, and the result benefits all users of OpenSMTPD portable.\nPlugged multiple memory leaks # As I requested feedback on the release candidate for OpenSMTPD 6.8.0p1, another user reported a memory leak which sent me on a big hunt.\nThanks to a top output I knew it was the lookup process\u0026hellip; but that process covers several scopes as it is in charge of DNS lookups, table lookups and passing information back and forth between an SMTP session and filters. I didn\u0026rsquo;t have an immediate intuition where the leak could come from as is often the case with other processes.\nI was suspicious of the filters layer since filters are still somewhat experimental and recent, however there are very few runtime allocations there and I could not find any unbalanced allocation except for a minor one that could not explain a noticeable leak. This was confirmed when the user applied the diff and saw no improvement whatsoever.\nThen I started being suspicious of DNS lookups and ended up diving into the resolver.c interface to asr, the asynchronous resolver. This led to another small leak found in OpenSMTPD and another one in asr itself prompting a diff from Eric Faurot in OpenBSD\u0026rsquo;s libc. This was slightly more noticeable than the previous one but still not very significant, it was barely visible on my own MX that had been running for a year without a restart.\nFinally, I had a eureka moment and convinced myself to look at the handling of regex lookup in tables. It turns out that a regfree() call was missing which caused a regex_t to leak for each regex lookup. Depending on the OpenSMTPD configuration, the use of regex, the size of tables used by regex and the filters in place, the leak could grow from fully inexistant to very significant.\nI submitted a diff to OpenBSD that was committed today by martijn@.\nTracked and found a possible crash in the filters state machine # As I was tracking the memory leak above, I came up by accident with a script that caused the OpenSMTPD instance on my laptop to crash\u0026hellip; just once. I tried to reproduce multiple times without success and decided to stop tracking the leak and focus on finding what caused the crash as it was more concerning.\nI spent hours adapting my script to play different kinds of sessions with varying number of recipients, concurrency levels and amounts of data sent. Eventually, I managed to get a kill script that would reliably crash my instance after a few seconds of running. It highly suggested a race condition.\nI tested the script on various instances with different configurations and realised that this did not affect all OpenSMTPD setups, but required a specific configuration and a specific client pattern to trigger. I could not crash a default install or my main MX which makes use of a lot of features, including filters, but I could easily crash my submission MX.\nThe bug was tracked for two days and it ended up being a logic issue in the filters state machine which could cause the io channel between the SMTP engine and the filters layer to be released while filters were still processing a request. Upon return from a filter, the channel would be expected to be writeable but it would in turn be NULL.\nThis led to a fix being committed to OpenBSD-current and an errata published today.\nWork on OpenSMTPD post-release # With the 6.8.0p1 release out of the way, I worked on a few diffs for future releases.\nSuggested the rename of OpenSMTPD processes (diff sent upstream) # Many years ago, OpenSMTPD had dedicated processes for the MDA and the MTA processes. OpenBSD hackers requested that I factor things a bit and merge these processes because they were both unprivileged and chrooted to /var/empty, making it pointless to over-isolate them.\nWe couldn\u0026rsquo;t find a proper name for the new process in charge of both MDA and MTA, but since an OpenBSD hacker had repeatedly requested that I offer him a pony I decided to temporarily name the process pony express (if you don\u0026rsquo;t understand why it\u0026rsquo;s funny, I can\u0026rsquo;t help). Later, the joke was improved by another developer when he named the new privsep crypto agent klondike.\nAs years passed, I considered renaming multiple times but could not convince myself the alternative names were better, and when I found better names they would not convince others. When a user said that these names were useless and unprofessional, I decided to let the joke run longer.\nYears later, I think the joke has been utilized to its fullest and is now bugging me for both technical (a space in the name of a process is annoying) and non-technical reasons.\nWhen the configuration file changed a while back to split rules into action and match sets, the concept of dispatchers was introduced in the code: an envelope is matched and based on the action, it is scheduled to a local or a remote dispatcher.\nI sent a diff to OpenBSD suggesting that the pony express process be renamed dispatcher, and that the klondike process be renamed crypto. I don\u0026rsquo;t know if/when this will be committed but the feedback was positive.\nSuggested explicitly requesting forward files (diff sent upstream) # Whenever a local dispatcher is used (actions mbox, maildir, mda or lmtp), OpenSMTPD will unconditionally look for a ~/.forward file in the recipients home directory. This is a mechanism that today can\u0026rsquo;t be turned off by an admin.\nI sent a diff to OpenBSD suggesting the introduction of a forward-file option.\nThe idea is to have OpenSMTPD consider it must NOT try to process ~/.forward files unless the admin explicitly requests it by adding the forward-file keyword to a local action. In the example below, I configure two domains with maildir, one not supporting forward files and the other supporting it:\naction \u0026#34;local_users_1\u0026#34; maildir action \u0026#34;local_users_2\u0026#34; maildir forward-file match from any for domain \u0026#34;foobar.org\u0026#34; action \u0026#34;local_users_1\u0026#34; match from any for domain \u0026#34;barbaz.org\u0026#34; action \u0026#34;local_users_2\u0026#34; The benefit is that this reduces the attack surface on setups where ~/.forward files are not desired. Processing these files require an indirection through the parent process to open the file on behalf of the daemon, but if we know we don\u0026rsquo;t want to process them the indirection can be bypassed.\nI don\u0026rsquo;t know if/when this will be committed.\nSuggested explicitly requesting custom MDA execution in forward files (diff sent upstream) # Built on top of the previous diff, I suggested the introduction of the allow-exec option for forward-file.\nThe ~/.forward files allow users to redirect their mails to another e-mail address, local or not, but they can also be used to plug a custom mail delivery agent such as fdm or procmail.\nThis is a very nice and interesting feature but it is also a vector of attack as if any bug allowed an attacker to write that file, including a bug unrelated to OpenSMTPD, it would allow them to execute commands with the recipient privileges.\nWe can\u0026rsquo;t really remove the exec feature of ~/.forward files because being able to execute a custom MDA is a very important feature. However, there are setups were you want to support users redirecting their mails elsewhere but not allow them to run their own custom MDA.\nMy proposal was to disallow execution of a custom MDA in ~/.forward files unless the admin explicitly allows it. In the example below, I configure two domains that support forward files but one does not allow execution of a custom MDA while the other does:\naction \u0026#34;local_users_1\u0026#34; maildir forward-file action \u0026#34;local_users_2\u0026#34; maildir forward-file allow-exec match from any for domain \u0026#34;foobar.org\u0026#34; action \u0026#34;local_users_1\u0026#34; match from any for domain \u0026#34;barbaz.org\u0026#34; action \u0026#34;local_users_2\u0026#34; With my diff, if a custom MDA command is placed in a ~/.forward file then the SMTP session temporarily rejects the session until the recipient fixes the file.\nI don\u0026rsquo;t know if/when this will be committed.\nSuggested explicitly requesting custom MDA execution in aliases and virtual mappings (diff sent upstream) # Built on top of the previous diff, I suggested the introduction of the allow-exec option for alias and virtual.\nHistorically, plugging a mailing list or similar tools was done by creating an alias to a command:\nlists: |/usr/local/bin/mlmmj ... This is however a very bad practice that should be discouraged because attaching a command to an alias means there\u0026rsquo;s no recipient user to run the command as, and they end up executed as the daemon\u0026rsquo;s user. In every single use-case I ever met, creating dedicated users and using their ~/.forward files to execute the commands achieves the same result while properly isolating the commands from the daemon.\nKilling support for execution of a command in aliases is a battle that I wanted to fight but that will meet a lot of resistance, so my proposal was to do the same as above for forward files and disable execution by default unless an admin explicitly requests it.\nIn the example below, I configure two domains with different aliases mappings, one not supporting execution of commands and the other supporting it:\naction \u0026#34;local_users_1\u0026#34; maildir alias \u0026lt;aliases_1\u0026gt; action \u0026#34;local_users_2\u0026#34; maildir alias \u0026lt;aliases_2\u0026gt; allow-exec match from any for domain \u0026#34;foobar.org\u0026#34; action \u0026#34;local_users_1\u0026#34; match from any for domain \u0026#34;barbaz.org\u0026#34; action \u0026#34;local_users_2\u0026#34; Note that virtual uses the same expansion loop as alias so it works the same:\naction \u0026#34;local_users_1\u0026#34; maildir virtual \u0026lt;aliases_1\u0026gt; action \u0026#34;local_users_2\u0026#34; maildir virtual \u0026lt;aliases_2\u0026gt; allow-exec match from any for domain \u0026#34;foobar.org\u0026#34; action \u0026#34;local_users_1\u0026#34; match from any for domain \u0026#34;barbaz.org\u0026#34; action \u0026#34;local_users_2\u0026#34; Similarly to the behavior of allow-exec with ~/.forward files, if a command is found on an alias that\u0026rsquo;s not allowed to execute a custom command then the SMTP session temporarily rejects the recipient until the alias is fixed.\nI don\u0026rsquo;t know if/when this will be committed.\nPrivileged-process safety net for allow-exec # I have a diff which is done but that I haven\u0026rsquo;t submitted as it depends on the diffs above and is pointless to submit until (if ?) they are committed.\nWhat it does is add a check in forkmda() so that before executing a custom MDA command from an envelope, it checks again that the dispatcher was configured to allow custom MDA commands through allow-exec.\nExcept for configuration changes, this should NEVER happen because a dispatcher that doesn\u0026rsquo;t allow-exec would reject the recipient during the SMTP session when it expands to a command. However, if an attacker managed to find a bug which allows injection of a custom MDA command into an envelope, then this safety net would prevent execution of the command by spotting the fishy situation.\nOn setups without allow-exec this would have prevented one of the security issues that Qualys disclosed earlier this year.\nI\u0026rsquo;m trying to find a similar way to provide a safety net for setups with allow-exec but it\u0026rsquo;s much harder as they explicitly allow custom MDA commands, and it\u0026rsquo;s not possible for OpenSMTPD to know from a custom MDA command if it was legitimate or not.\nReleased new version of filter-rspamd # I did a minor release of filter-rspamd to change the versioning format and make it easier to package on OpenBSD.\nNothing fancy.\nStarted releasing chapters from the OpenSMTPD book # As I\u0026rsquo;ve mentioned in the past, I work on an OpenSMTPD book which is 120ish pages as of today.\nMy plan to release it when done does not work because I keep rewriting the same chapters over and over until I\u0026rsquo;m happy\u0026hellip; and I never am. Meanwhile the project moves forward so things get outdated and I need to rewrite them again.\nI decided to take a more radical approach and work on it publicly on Github. This way, I don\u0026rsquo;t have to worry about people seeing an imperfect phrasing or pages not rendering the way I want in PDF.\nYou\u0026rsquo;ll find the repository here.\nWhat\u0026rsquo;s next ? # I have great plans for 2021 but for now I don\u0026rsquo;t want to disclose them.\nHave a merry Xmas, a happy new year, and see you in January !\n","date":"24 December 2020","permalink":"/posts/2020-12-24/december-2020-opensmtpd-6.8.0p1-released-fixed-several-bugs-proposed-several-diffs-book-is-on-github/","section":"Posts","summary":"TL;DR: Crafted the OpenSMTPD 6.8.0p1 release Fixed several bugs in the way Proposed a few OpenSMTPD improvements to OpenBSD Working on my OpenSMTPD book Let\u0026rsquo;s start with some LoFi # Relax.","title":"December 2020: OpenSMTPD 6.8.0p1 released, fixed several bugs, proposed several diffs, book is on Github"},{"content":" TL;DR: Haven\u0026rsquo;t posted since July but I wasn\u0026rsquo;t slacking, got temporarily context-switched out of opensource stuff. I tanked a year in work psychology # As some of you know, I study work psychology and have been attending evening classes for the last four years, sometimes up to 9 hours / week and more often between 3 and 6 hours / week. I\u0026rsquo;m pretty invested as you can tell, it\u0026rsquo;s a lot of time that I could spend in bed but that I spend in class instead :-)\nLast school year, I was supposed to do a pair work field intervention which would then be used as the material for my upcoming master thesis dissertation. This intervention was meant to take place physically with groups of employees and span over two or three months. We had started interviewing with a company interested in our intervention, but the lockdown put this to a hold and was renewed over and over until the school year was over.\nUnlike other courses where it was possible to substitute the evaluation method, like in psychometric testing course where we ended up analyzing a case study rather than having someone undergo interviews and tests, there\u0026rsquo;s no way to substitute this intervention without compromising the master thesis material so\u0026hellip; many of us plain tanked the course.\nBecause of the extraordinary situation, we were told that if we could begin an intervention in September and be done by end of November, the school year could be salvaged and we could integrate the thesis preparation course in December, losing just a trimester instead of a year. Since we were close to having a field, there was a slight hope that we could pull this out and I made this my priority.\nUnderstandably, the company we were in discussions with had to shift its priorities post-lockdown and summer and ended up declining the intervention, so I tanked for real and began taking the same course again for a year\u0026hellip;\nA year which begins with another lockdown.\nI opened my hypnosis office # I started learning hypnosis sometime in 2015 and eventually ended up attending multiple trainings and certifications.\nAfter a year, I began doing sessions for volunteers with specific problems so that I could practice these. I only requested that they would donate an amount of their choice to a charity of their choice, and that they would give me a honest feedback about what worked and not. It lasted a while but then I gradually stopped asking for volunteers because it was no longer necessary in my opinion.\nTwo years ago, I started getting enough people referred to me that it made sense to create a legal entity. I began working as a mobile hypnotherapy practitioner during my time off work, on evenings and week-ends. When I switched to a part-time schedule, I often plugged one or two sessions per day so that it could help me recover a bit from the financial loss.\nThis was nice and all but after two years of driving North and South of the city, sometimes rushing from a client to another because a session ended late, I started considering opening an office so it would be more confortable for me. I now receive people in two wellness centers, on Thursday and Friday.\nSince I tanked a year in work psychology, I thought I might as well make another project go through :-)\nI won\u0026rsquo;t be talking much about that activity here as it\u0026rsquo;s so remote from tech, however feel free to checkout my dedicated website, hypno.cat, and feel free to ask me anything on twitter as I\u0026rsquo;m at least as passionate about this topic than about tech.\nI brought poolp.org back home, literally # I have not had a static IP in a long time because the choice of ISP was limited in this new neighborhood, but recently an option became available allowing me to have FTTH with a static IP, IPv6 and outgoing SMTP trafic.\nI\u0026rsquo;ve been renting a ton of VPS at vultr (affiliation link) and am happy with them, but I need to reduce my expenses and they cost me around 100 EUR / month. I decided to stop hosting people for free as I\u0026rsquo;ve been doing for the last 20 years or so, and cut on the number of VPS because poolp.org doesn\u0026rsquo;t have to be such a high-availability service. I can live with a few downtimes myself if any.\nI got myself a refurbished machine from backmarket (affiliation link), installed the latest OpenBSD on it and started moving everything there so I could progressively shutdown a few VPS.\nYou\u0026rsquo;re reading this blog post from a machine hosted in my appartment, plugged right behind a 10GB theoretical connection, though the device only has a 1GB network interface :-)\nI automated poolp.org publishing in the Github CI # The posts on this blog are generated from markdown files that are available in a Github repository.\nI initially didn\u0026rsquo;t automate publishing so the repository was just used to version the posts. The action of publishing was manual and this explains why sometimes people would submit pull requests to fix a typo and the changes would not show up before days: I could merge the pull request but until I booted the proper laptop to rebuild the website and rsync the directory to the server, the change would not be visible.\nIt was annoying because it meant I could not fix anything quickly, I always had to wait until I was in front of my OpenBSD laptop which is the only one allowed to auth to my server. In one instance, I messed up and published an uncommitted draft while doing an rsync from my laptop to fix a typo, I realized because the unfinished article had been linked from hackernews before I had a chance to proofread and I could not unlink it right away.\nSo I took some time to rework this and now the blog is generated from the repository itself, I no longer do rsyncs all over the place. If you submit a pull request to fix a typo or improve an article ,the blog will be updated shortly after I merge it.\nThis is technically the first article ever that I will not rsync.\nNot that you care, but I also did the same for hypno.cat while I was at it ;-)\nBegan preparing the OpenSMTPD portable release # I have brought all upstream commits from OpenBSD to the portable version of OpenSMTPD, fixed a couple shortcomings in the way and I have a few issues I need to tackle before I can craft the release.\nI should be done in the next couple weeks so the OpenSMTPD portable release can happen in early December.\nWhat\u0026rsquo;s next ? # Not finding the time to produce useful work to write about since July has been irritating and getting back on track was not effortless. I wrote this, even though it\u0026rsquo;s not technical, to break the cycle of silence and pick up where I left.\nThere are several pull requests pending on many of my repositories, I merged some and need to review others.\nI intend to write some tutorials too.\nSee you in December !\n","date":"22 November 2020","permalink":"/posts/2020-11-22/november-2020-i-wasnt-slacking-but-no-opensource-work/","section":"Posts","summary":"TL;DR: Haven\u0026rsquo;t posted since July but I wasn\u0026rsquo;t slacking, got temporarily context-switched out of opensource stuff. I tanked a year in work psychology # As some of you know, I study work psychology and have been attending evening classes for the last four years, sometimes up to 9 hours / week and more often between 3 and 6 hours / week.","title":"November 2020: I wasn't slacking but no opensource work"},{"content":" TL;DR: worked on my webmail, on a custom MDA and on a python framework for API development. Worked on my webmail # Again, I\u0026rsquo;d like to emphasize that this is something that\u0026rsquo;s going to span over many months so\u0026hellip; don\u0026rsquo;t hold your breath.\nLast month I wrote that the webmail didn\u0026rsquo;t allow reading messages yet as the /fetch backend operation only fetched headers. This was because I focused on navigation and the fetching of messages in IMAP is a bit tricky to do right so I wanted to be focused. A naive approach that works is to simply get the full message and parse it in the client, something that I\u0026rsquo;ve done multiple time in the past when performances weren\u0026rsquo;t a big deal. With a webmail, this approach does not work: the time it takes to open a page with the mail content is unbearable when there\u0026rsquo;s a big file attached to it, it\u0026rsquo;s just not acceptable.\nLuckily, imap supports fetching a BODYSTRUCTURE which results in a response describing the MIME structure of the message. With this, a client can inspect the structure and determine which parts it wants to fetch them specifically. This is a nice solution, at the cost of an additional fetch operation, and in the case of a webmail it is exactly what I need to avoid passing huge amount of unnecessary data between the imap server, the backend and the frontend. This took me a while to complete because parsing the BODYSTRUCTURE is nightmarish: it describes a MIME structure \u0026hellip; and MIME supports nesting \u0026hellip; and the imap protocol is \u0026hellip; how do I put it \u0026hellip; not the most beautiful thing.\nWith this done, I could fetch multipart messages and allows displaying a specific part, without the performance overhead of a full fetch:\nThe above was essentially backend work with just a bit of frontend work, but the remaining of my work on the webmail was essentially on the frontend side:\nI reworked a bit the navigation to simplify, splitting between \u0026ldquo;system folders\u0026rdquo; and \u0026ldquo;custom\u0026rdquo; folders, but also allowing to click on a progressbar to skip to a specific portion of a large folder, getting rid of \u0026ldquo;pagination\u0026rdquo; links. In addition, I reworked the folder listing so that the frontend now adapts to the screen size: it determines the window height and fetches just enough entries to fill the window without generating a scrollbar. The scrollbar is unnecessary because pagination uses either the pageup/pagedown button, or is automatic when using arrow up/down while on the first or last entry of the listing. If you use mutt, you\u0026rsquo;ll be in a very familiar ground.\nA bit I\u0026rsquo;d like to improve is the display of subfolders which are currently displayed inlined in the custom folder tab.\nI also worked on mail composing, because being able to read a mail is nice but being able to answer is even better.\nI\u0026rsquo;m not a big fan of HTML emails for a variety of reasons, some of which will be explained in an upcoming article, but I\u0026rsquo;d be deluded if I thought HTML emails were going anywhere. Still, I want to avoid them as much as possible, so I decided to make the webmail allow composing HTML messages while emitting them in plain by default, if possible.\nBasically, the webmail will display a compose page which allows to customize the appearance of text, however it will:\ndefault to generating a text/plain part if no customization was used generate a text/plain alternative part stripped of customization if any was used Of course, the idea is not just to compose an email but also to send it. I wrote the backend bits to craft a MIME-message from the frontend submission and send it, which allowed me to mail myself:\nWork is still needed to allow attachments when composing a message, but there is not technical difficulty there.\nAt this point, I know the backend API is about right and I can focus on improving the frontend to make it more usable.\nWorked on a custom MDA # Last month, I talked about my custom MDA and script script to perform folder pinning. I\u0026rsquo;m happy with the idea but not so much with the implementation so I decided to revisit.\nThe issue is that folder pinning is far from being the only thing I want to do with incoming mails at delivery time, and shoving everything in the mda executable is not ideal. I rewrote the MDA to have it handle delivery only and call an API to determine where it should do it, this let me play with a ton of ideas on a custom API server without tweaking the working MDA. At the end of the day, I had incoming mails processed by various text analyzers, attachments automatically extracted and put in an s3 backing store, and mails indexed for fast lookups.\nI will not expand much on how I did this as I think it makes a nice topic for a dedicated article on custom MDA, and fun stuff you can easily do with them to provide some awesome features on your mail setup.\nWorked on a python framework for my API/web development # I often write APIs, either for work of for my own needs, and I usually use bottlepy which is a micro-framework for python. Along the years, I worked on a lot of projects using that framework, including professional projects with tons of constraints.\nEric and I built a lot of small helpers around it to make out job easier, and we kept copying these in each and every project, improving them as needed. I thought it would be a nice idea to package this so that I could easily bootstrap a new project without copying code around. So I spent a few days extracting the code from an existing project, cleaning it up, and packaging it so it could easily be used to start a new project.\nThe work isn\u0026rsquo;t done yet, but the idea is that you end up with an executable. This executable is controlled by a configuration file which lets you describe what interface and port the API is running on, as well as what package implements its endpoints. Then, all you need to do is to create a small package that contains the endpoints and implementations, something like:\nimport openbar.routes @openbar.routes.register(\u0026#34;0.1\u0026#34;, \u0026#34;backend\u0026#34;) def setup(app): app.get(\u0026#34;/\u0026#34;)(get_root) app.post(\u0026#34;/\u0026#34;)(post_foobar) def get_root(): return {\u0026#34;foobar\u0026#34;: \u0026#34;barbaz\u0026#34;} def post_foobar(): return {\u0026#34;barbaz\u0026#34;: \u0026#34;bazqux\u0026#34;} What\u0026rsquo;s interesting here is not that this is short, because you\u0026rsquo;d get about the same if you just dropped bottlepy in the same folder, but it\u0026rsquo;s that you don\u0026rsquo;t need to do all the plumbing to daemonize and drop privileges, log to syslog, etc\u0026hellip; and that it comes with a lot of helpers.\nI will also not expand much on this as I\u0026rsquo;m almost done with the packaging and I might as well demo it rather than spoil it. There is nothing super impressive, no big innovation, but just a wrapping of common stuff in interfaces that makes them pleasant to deal with.\nThe webmail backend is an obvious candidate for this, but so is the MDA API server. I keep needing this over and over, it made sense stepping into this project to cut time on the others.\nno OpenSMTPD work ? # Not much this month.\nI spent a day playing with a queue-procexec but the code was throw-away, only meant to help me detect if something was utterly broken with the idea.\nI also worked a bit on the go-opensmtpd package and made progress on the filter API, the goal being to allow writing a filter without worrying about the low-level protocol. Instead of reading the protocol, parsing it and determining what should be called or not, a filter can register callbacks for specific events and implement these callbacks.\nI will finish this in September and adapt my filter-rspamd accordingly.\nWhat\u0026rsquo;s next ? # I will be taking a break from all projects in August to enjoy some long-awaited vacations, so\u0026hellip; don\u0026rsquo;t expect any code to be written before September.\nHave a nice month !\n","date":"31 July 2020","permalink":"/posts/2020-07-31/july-2020-webmail-custom-mda-and-python-framework-work/","section":"Posts","summary":"TL;DR: worked on my webmail, on a custom MDA and on a python framework for API development. Worked on my webmail # Again, I\u0026rsquo;d like to emphasize that this is something that\u0026rsquo;s going to span over many months so\u0026hellip; don\u0026rsquo;t hold your breath.","title":"July 2020: webmail, custom MDA and python framework work"},{"content":" TL;DR: Reworked my infrastructure at poolp.org, implemented folder pinning, worked on my webmail. poolp.org infrastructure rework # I decided to move my servers from online.net to vultr.com (affiliation link).\nI have been at online.net for hosting since 2007 and have always been happy with their services, however OpenBSD has never been supported there and I always had to use custom install procedures and hope things go well.\nNowadays, I\u0026rsquo;m testing a lot of stuff and I create a lot of short-lived machines so going through these procedures each time is painful. I\u0026rsquo;ve been happy with vultr.com for a little while now so I decided to just move everything there, only keeping a backup server at online.net.\nThis forced me to rework my setup but also allowed me to improve it as the cost of running multiple VPS is considerably lower than running multiple dedicated servers, I could split multiple services into their dedicated VPS and tidy things a bit. Instead of three servers, I now operate 15 VPS. As time allows, I\u0026rsquo;ll start documenting on this blog the different parts and their configuration.\nIt was a bit painful but everything is migrated at this point and the old servers can be shutdown.\nThe idea behind this infrastructure change is also to start considering the services that I might provide my sponsors in the future :-)\nMail classification heuristics and folder pinning # A downside of mail commonly shared by users is the problem of mail classification and inbox overload.\nHistorically mails were dropped in an mbox which is a mail format where all mails are appended to a common mailbox. This meant that all mails were considered equivalent in priority and were just sorted in delivery time order. The user had to navigate through them all mixed together and decide which ones they wanted to deal with.\nThe mbox format was mostly deprecated by the Maildir format which comes with many advantages and uses a directory structure, rather than a unique mailbox file. The directory structure allows mail folders to exist as sub-directories making it possible to create dedicated folders, and use post-processing tools to classify mails in these folders. For example, at poolp.org the fdm utility was installed and allowed users to create classification rules which, in my case, were used to classify mails from OpenBSD mailing lists into their own folders and out of my Inbox. This is extra powerful and flexible but not very generic as the rules set by a user may not work for another.\nThe problem is that even if you can create folders and create rules to put mails in them, an Inbox essentially replicates the historical behavior of mbox and ends up accumulating all kinds of mails. If we keep spam and mailing lists aside for now, the Inbox is still where all commercial mails, social network notifications, password recovery, undelivered mails notifications and other land until you create a rule matching each of them.\nGmail, despite an interface I dislike, tackled the issue in a very sound way with their folder tabs: Primary, Social, Promotions, Updates and Forums. The Primary folder is the Inbox, Social is where all social networks notifications land, Promotions is where bulk mails land, Updates is where transactional mails land, and Forums is where mailing lists land.\nI wanted to replicate this on my server and it turned out to be very simple as this classification can be done with a few heuristics. I configured dovecot to expose these additionals folders by default, then configured fdm to apply a set of rules that would categorize mail in these folders:\naction \u0026#34;inbox\u0026#34; maildir \u0026#34;%h/\u0026#34; action \u0026#34;junk\u0026#34; maildir \u0026#34;%h/.Junk\u0026#34; action \u0026#34;social\u0026#34; maildir \u0026#34;%h/.Social\u0026#34; action \u0026#34;bulk\u0026#34; maildir \u0026#34;%h/.Bulk\u0026#34; action \u0026#34;transactional\u0026#34; maildir \u0026#34;%h/.Transactional\u0026#34; action \u0026#34;lists\u0026#34; maildir \u0026#34;%h/.Lists\u0026#34; account \u0026#34;mda\u0026#34; stdin match \u0026#34;^X-Spam: [Yy]es\u0026#34; in headers action \u0026#34;junk\u0026#34; match \u0026#34;^X-Facebook-Notify: .*\u0026#34; in headers action \u0026#34;social\u0026#34; match \u0026#34;^Precedence: list\u0026#34; in headers action \u0026#34;lists\u0026#34; [...] match \u0026#34;^Precedence: bulk\u0026#34; in headers action \u0026#34;bulk\u0026#34; match \u0026#34;^From: .*no-?reply.*\u0026#34; in headers action \u0026#34;transactional\u0026#34; match all action \u0026#34;inbox\u0026#34; Every time a mail that I would expect to land in a subfolder hit my Inbox, I looked at its headers and checked what could be used as a discriminator to create a new rule or improve an existing one. This considerably improved the state of my Inbox as most mails that didn\u0026rsquo;t expect an answer landed in a subfolder, while my Inbox contained mostly mails from people expecting an answer.\nThis worked wonders for most mails but then there were some that didn\u0026rsquo;t get proper classification, because sender didn\u0026rsquo;t do things correctly and made it hard to classify: these mails kept hitting the Inbox and I kept moving them out of the way without being able to write a proper rule. Also, as someone subscribed to multiple mailing-lists, having all mailing-lists mails in a single folder was not comfortable. It occured to me that what was needed on top of these heuristics was folder pinning: if a user receives mail from a sender and moves it to a folder, then most likely future mails from that sender should be moved to that folder too. This would solve the problem of senders that don\u0026rsquo;t fit generic classification, but also the problem of mails you would like classified in custom folders.\nI decided to implement this using three components:\na mail delivery agent, similar to fdm, but with extra logic that would take into account the folder pinning; a key-value store to keep track of folder pinning state; a sieve script for learning new pinning states from mail moves in IMAP; Basically, when a mail arrives at poolp.org the mail delivery agent extracts the sender information, checks in the key-value store if the recipient has a pinning rule for the sender and proceeds to delivery. If there\u0026rsquo;s a pinning rule then that folder is used, otherwise heuristics are used to find the most likely folder the mail should land in.\nIf a user is not satisfied with where a mail landed, moving it to a new folder causes dovecot to pass the mail to a sieve script which will extract the sender information, and push a new rule automatically to the key-value store so that next time the mail delivery agent will know where that sender should land.\nThe sender extraction is done on the Return-Path address, not the From header, which ensures that someone contacting me directly or through a mailing-list is classified in the proper folder, and the fact it is done at the protocol level ensures that this is works regardless of the client used to read mail.\nThis works very nicely and is particularly pleasant as you quickly forget that it is here, mail just lands where you most expect it. The way I implemented it makes it possible to share in a generic way so that everyone can benefit from this feature, regardless of which MTA they are using, however the code is not ready yet as I wrote it as an experiment for my own setup with hardcoded paths and such. I\u0026rsquo;ll keep working on it and, when it is ready, I\u0026rsquo;ll publish my mail delivery agent, the sieve scripts and the dovecot configuration in a single repository.\nWebmail work # As I mentionned in April, I\u0026rsquo;m currently working on a webmail of my own. Most of the work I did this month was around that project which is evolving at a good pace considering that I had no experience in React, and that it is the end of my scholar year which comes with assignments consuming all of my spare time.\nThe webmail comes in two parts:\nthe backend; the frontend; Backend work # The backend is just an API that exposes REST endpoints to the underlying IMAP server. It doesn\u0026rsquo;t just map a route to an IMAP command but does it in a way that\u0026rsquo;s optimized for a webmail, for instance /select is not just selecting a mailbox but also an offset and a limit to lookup envelopes for that mailbox, avoiding a round-trip for a /fetch. I wrote a version of the backend in Python and another in Go but eventually decided to stick to Python, the email parsing library is much more featureful and the performances are just there.\nIt currently implements:\n/login, to authenticate against the IMAP server /noop, which is used by the webmail to keep a connection alive /select, to select a folder and fetch up to limit envelopes from offset /fetch, to fetch a specific message (currently only headers) /move, to move a set of messages from a folder to another /expunge, to delete a set of messages permanently This backend can be used as the building block of any mail client so I will publish it as a standalone project, if people are interested in building their webmails they can then focus on the interface and not the lower-level bits.\nFrontend work # The frontend is a React client to that API, plain and simple. I only focused on the folder view at the moment so it is not usable yet to read mails, but it is possible to authenticate, navigate between folders, select and delete or junk mails, etc\u0026hellip;\nRemember that this is a work in progress at an early stage, not the final interface, but it allowed me to test the behaviors I wanted from the webmail and I\u0026rsquo;m happy with the result.\nYou\u0026rsquo;ll just have to be patient until I put up a live demo so you can test for yourself\u0026hellip;\nAt this point, all I can say is that it is progressing and that some aspects of it are impressive. It is very fast, and this is despite me testing it from my laptop on a remote IMAP server with no caching and no pre-fetching, no optimizations whatsoever.\nPatience !\nWhat\u0026rsquo;s next ? # My main focus in July will be the webmail, the sooner I get a PoC out the better.\nI have some ideas to improve mail classification further in more \u0026ldquo;creative\u0026rdquo; ways, I\u0026rsquo;ll continue exploring how to make mail management nicer.\nI have some pending work for OpenSMTPD, both on the filter side and on the queue side.\n","date":"27 June 2020","permalink":"/posts/2020-06-27/june-2020-poolp.org-folder-pinning-and-webmail-work/","section":"Posts","summary":"TL;DR: Reworked my infrastructure at poolp.org, implemented folder pinning, worked on my webmail. poolp.org infrastructure rework # I decided to move my servers from online.","title":"June 2020: poolp.org, folder pinning and webmail work"},{"content":" TL;DR: Worked on the OpenSMTPD 6.7 release; Did a lot of work on the new table API; Wrote several PoCs; WARNING:\nExamples of code and configuration that appear in this article are here to help illustrate and explain development stages of my work.\nThey are subject to changes and must not be considered as user documentation. By the time you\u0026rsquo;re reading this, they will likely no longer work or reflect reality.\nOpenSMTPD 6.7.1p1 release # On May 19, 2020 was released OpenBSD 6.7 which ships OpenSMTPD 6.7, the latest stable version of OpenSMTPD.\nI released the portable version of OpenSMTPD on the same day, followed a few days later by a minor update to fix a packaging issue spotted by Debian maintainer Ryan Kavanagh and a possible crash when relaying over IPv6 spotted post-release by Void Linux maintainer Leah Neukirchen.\nI did a lot of work on portable, extending the CI to new targets and such, I won\u0026rsquo;t talk about this work because this article contains a fair amount of more interesting topics.\nSome highlights in the 6.7.1p1 release # Before all, one of the most visible change for package maintainers is that libasr is no longer a dependency. It is shipped in the compat layer and built conditionally if the host system doesn\u0026rsquo;t have the library installed, this will solve a few headaches.\nThis release brings various improvements to the filtering protocol, not visible to users for the most part, but most notably three features deserve a special mention.\nFirst, the new bypass action which was already discussed here. It allows writing exclusion rules for certain sessions in builtin filters whereas, prior to that, all sessions connecting to a listener with filters would always have all filters applied to them:\nfilter trusted phase mail-from match src \u0026lt;trusted_sources\u0026gt; bypass filter no_rdns phase mail-from match !rdns reject \u0026#34;550 go away\u0026#34; filter no_fcrdns phase mail-from match !fcrdns \u0026#34;550 go away\u0026#34; listen on all filter { trusted, no_rdns, no_fcrdns } Then, the smtp-out reporting stream which is very similar to smtp-in but reports SMTP events for outgoing sessions. This allows writing filters that do all kind of accounting on outgoing trafic and, as will be shown in this article, can be used to make interesting filters.\nFinally, all built-in filters can now match authenticated sessions and specific authenticated users. It becomes possible to apply some filters to non-authenticated sessions only, or to avoid applying some filters to specific authenticated users.\nfilter auth_only phase mail-from match !auth reject \u0026#34;550 you must be authenticated to send mail\u0026#34; filter gilles_only phase mail-from match !auth gilles \u0026#34;550 only gilles is allowed to send mail\u0026#34; And because auth takes a table parameter, we can also do table-based filter matching using a credentials table:\ntable poolp_users { gilles = XXX, eric = XXX } filter poolp_only phase mail-from match !auth \u0026lt;poolp_users\u0026gt; reject \u0026#34;550 must be a poolp.org user\u0026#34; Outside of the filtering area, other improvements were made to smtpd.conf that makes it possible to simplify existing configurations as well as express configurations that were not possible before.\nThere are two notable improvements in my opinion:\nIn previous releases, it was possible to have rules match authenticated session as follows:\nmatch from any auth for any action \u0026#34;relay\u0026#34; but this was an all or nothing mechanism, either you wanted to match all authenticated sessions or none.\nThe mechanism was extended to support matching specific authenticated users so that you can write:\nmatch from any auth gilles for any action \u0026#34;relay_gilles\u0026#34; match from any auth eric for any action \u0026#34;relay_eric\u0026#34; Similarly to filters, the auth parameter is a table so that it is possible to pass a list of specific users and create rules targeting specific users:\nmatch from any auth { gilles, eric } for any action \u0026#34;relay_custom\u0026#34; match from any auth for any action \u0026#34;relay\u0026#34; This is very powerful as auth accepts credentials tables and allows setups serving multiple domains to have rules match their specific users, something that was not doable previously:\ntable users_poolp_org file:/etc/mail/users_poolp_org table users_opensmtpd_org file:/etc/mail/users_opensmtpd_org listen on $ip_poolp [...] auth \u0026lt;users_poolp_org\u0026gt; listen on $ip_opensmtpd [...] auth \u0026lt;users_opensmtpd_org\u0026gt; [...] match from any auth \u0026lt;users_poolp_org\u0026gt; for any action \u0026#34;relay_poolp\u0026#34; match from any auth \u0026lt;users_opensmtpd_org\u0026gt; for any action \u0026#34;relay_opensmtpd\u0026#34; Finally, a rework of the from and for logic in smtpd.conf was done to better express rules. Instead of assuming that from and for always attempted to match a source address and a destination, they now match the user intent:\nmatch from auth [...] match from mail-from gilles@poolp.org [...] match [...] for rcpt-to gilles@poolp.org This change allows expressing many configurations in a much more compact and user-friendly way:\nFor example, the following:\nmatch from any auth for domain poolp.org rcpt-to gilles@poolp.org action \u0026#34;deliver\u0026#34; becomes:\nmatch from auth for rcpt-to gilles@poolp.org action \u0026#34;deliver\u0026#34; They are both functionally equivalent but the change is semantic, the first rule requires that the envelope matches both from any and auth for origin and both for domain poolp.org and rcpt-to gilles@poolp.org for destination, whereas in the second rule it only requires that auth and rcpt-to gilles@poolp.org be matched regardless of the source address and destination domain. This semantic shift is backwards compatible so existing configuration are not impacted, they just have some rules that could be simplified in some cases.\nMost of the other work is either not visible to users, but feel free to browse the commit history for more as there\u0026rsquo;s been a LOT of work poured in the release polishing and cleaning stuff.\nfilter-prometheus # As I mentionned above, the new release comes with an smtp-out events reporting stream that makes it possible to write filters that are aware of outgoing trafic, so I decided to write a PoC filter that would highlight how this can be used in a useful way.\nPrometheus is an open-source monitoring system that\u0026rsquo;s fairly popular and that seems to have the favours of highly competent people at ${DAYJOB}. Since I need to get familiar with it at work, I thought I\u0026rsquo;d get my hands on it on my spare time too.\nBasically the idea is that you run an instance (or a cluster, but lets keep it simple) that will contact configured endpoints that expose various metrics. The metrics are then available in Prometheus through a query language, PromQL, and through graphs in a web user interface.\nI wrote a filter-prometheus which registers smtp-in and smtp-out events, creates some metrics based on them, and exposes these metrics on an HTTP endpoint.\nPrometheus is configured to hit that endpoint and\u0026hellip; voila, that\u0026rsquo;s all, they are available through prometheus.\nOn the configuration standpoint, this is straightforward. The filter itself doesn\u0026rsquo;t have configuration, it only supports a command line flag to specify which IP and port are used to exposed the metrics.\nPrometheus has a simple configuration:\nglobal: scrape_interval: 15s scrape_configs: - job_name: \u0026#34;prometheus\u0026#34; static_configs: - targets: [\u0026#34;localhost:9090\u0026#34;] - job_name: \u0026#34;OpenSMTPD: in.mailbrix.mx\u0026#34; static_configs: - targets: [\u0026#34;in.mailbrix.mx:31333\u0026#34;] OpenSMTPD has an even simpler configuration:\nfilter prometheus proc-exec \u0026#34;filter-prometheus -exporter 0.0.0.0:31333\u0026#34; # attach filter to a listener for smtp-in metrics listen on all filter prometheus # attach filter to a relay action for smtp-out metrics action \u0026#34;outgoing\u0026#34; relay filter prometheus [...] The filter is already available in a Github repository but note that this is a work in progress and that it was mostly done to play with prometheus and exporting metrics, not intended to be used for real as I don\u0026rsquo;t have any significant experience to design metrics properly at this point.\nIf you are familiar with prometheus and want to give me a hand in making this filter production-ready, I\u0026rsquo;ll be more than happy to make this happen with your help.\ntable-procexec # OK, let\u0026rsquo;s get to the hairy part now, but first a bit of a refresher.\nOpenSMTPD uses a mechanism called tables for all kinds of internal lookups. The table API support three operations, check, lookup and fetch. It uses these three operations to check if an envelope criteria is met (ie: does this user exists ?), lookup informations based on a specific key (ie: what are the aliases for root ?), or fetch a value from a set (ie: what\u0026rsquo;s a valid source IP to use for this outgoing session ?).\nIn addition to this, the table API supports various lookup services (ie: credentials, aliases, mailaddr, \u0026hellip;) which determine what kind of data the table is supposed to return. If you lookup a mailaddr in a table, the expected return value is an e-mail address that can be validated by the daemon. The three table operations are aware of this, so OpenSMTPD will for example \u0026ldquo;check if table has a mailaddr matching key gilles@poolp.org\u0026rdquo;.\nThere are multiple table backends available, both builtin to OpenSMTPD (memory, file and db) and third-party (sqlite, postgres, mysql, \u0026hellip;). Some backends, like table-passwd, only support specific lookup services (ie: credentials, userinfo) while others support all. As long as you have a data source upon which you can implement the three operations, you can implement a backend for some or all of the lookup services. The good part is that this happens outside of the daemon, so there\u0026rsquo;s no need for cooperation from OpenSMTPD, no spill of dependencies and such, table backends are fully independant standalone programs.\nThe API is very simple so in theory anyone can write a backend for their own custom use-cases, but in practice the communication with OpenSMTPD happens through the imsg API which limits the interface to C developers. Eric and I wrote the necessary bits a long time ago to abstract the details so that C developers only need to focus on implementing the operations and not deal with the IPC. It helped a lot but any backend doing something a bit tricky, ldap for example, still requires being comfortable with C programming and a sysadmin without that knowledge can\u0026rsquo;t really get anything done. A bit sad given that sysadmins are the primary users of this\u0026hellip;\nI later implemented a table-python backend which acted as a bridge. It allowed to implement the three operations in Python functions and have the backend call the functions, taking care of converting back and forth between the two languages. This made table-backends much more user-friendly but then came the Perl people, then the Lua people, then the Go people, \u0026hellip; and a bridge would have to be written and maintained for all of them, something I don\u0026rsquo;t have the time or motivation to do.\nWe had already agreed with eric@, for other reasons, that the API should be switched to a line-based protocol, like was done for filters, but this is a very tricky task in the daemon at this point and requires solving non-trivial limitations first. I\u0026rsquo;ll skip on this because this article would grow considerably, let\u0026rsquo;s just say that my first two attempts at this were unsuccessful.\nIt still annoyed me that we couldn\u0026rsquo;t move forward with this so I decided to tackle the issue differently. I wrote table-procexec a table backend which translate the imsg protocol into a filter-like query/response line-based protocol:\ntable|0.1|1590631921.2944137|table-name|check|deadbeefdeadbeef|mailaddr|gilles@poolp.org table-response|deadbeefdeadbeef|found table|0.1|1590632034.2909038|table-name|lookup|deadbeefdeadbeef|alias|root table-response|deadbeefdeadbeef|found|gilles The table-procexec backend communicates with OpenSMTPD through imsg, forks a child table backend talking the new protocol and takes care of proxying queries and responses translating imsg to this new protocol.\nOpenSMTPD \u0026lt;-- imsg --\u0026gt; table-procexec \u0026lt;-- line-protocol --\u0026gt; table-foobar The child table backend can be any program written in any language, just like with filters, and all it has to do is to implement the three operations expected from a backend, read the queries from stdin and respond to stdout.\nThis isn\u0026rsquo;t as elegant as if OpenSMTPD was producing the line-based protocol itself, allowing us to skip the table-procexec layer, but it allows moving forward in a way that\u0026rsquo;s transparent to the daemon. We can already test the API, improve it, make sure it works, implement new table backends on top of it and use them. When the limitations in OpenSMTPD are solved, the table-procexec can be thrown away transparently:\nOpenSMTPD \u0026lt;-- line-protocol --\u0026gt; table-foobar This is a work in progress and while it can be plugged to OpenSMTPD today using nothing but the existing framework, it doesn\u0026rsquo;t allow passing a configuration file to the child backend (yet).\nIn terms of configuration, it works similarly to any existing backend:\n# table foobar uses procexec backend to fork table-foobar table foobar procexec:table-foobar Ideally, we should switch to a filter like syntax, but we\u0026rsquo;re not there yet:\ntable foobar proc-exec \u0026#34;table-foobar -f /etc/mail/foobar.conf\u0026#34; Testing this require building the table-procexec backend from the branch of the same name in the OpenSMTPD-extras repository, I\u0026rsquo;ll leave this as an exercise to the reader.\ngo-opensmtpd # Of course, testing the new table API required a PoC so I decided to start with a Go implementation.\nThe go-opensmtpd/table package provides a work-in-progress implementation of the protocol parser, as well as a table API to ease development. Writing a table backend in Go using this package is simple, here\u0026rsquo;s a simple dummy example of a table that only contains my e-mail address:\npackage main import ( \u0026#34;github.com/poolpOrg/go-opensmtpd/table\u0026#34; ) func check(token string, service table.LookupService, key string) { if key == \u0026#34;gilles@poolp.org\u0026#34; { // found in table table.Boolean(token, true) } else { // not found table.Boolean(token, false) } } func lookup(token string, service table.LookupService, key string) { if service == \u0026#34;alias\u0026#34; \u0026amp;\u0026amp; key == \u0026#34;root\u0026#34; { // if looking up alias for root, return my address table.Result(token, \u0026#34;gilles@poolp.org\u0026#34;) } else { // empty result, not-found table.Result(token) } } func fetch(token string, service table.LookupService) { // only my address is in the set table.Result(token, \u0026#34;gilles@poolp.org\u0026#34;) } func main() { table.OnCheck(check) table.OnLookup(lookup) table.OnFetch(fetch) table.Dispatch() } Note that the Dispatch() function never returns as it is the event loop for the protocol parser. I don\u0026rsquo;t have much to add here, this is trivial to understand.\nThe package is a work in progress, it is already available in a Github repository but expect it to change every few days at this point, don\u0026rsquo;t use it unless you are ready to suffer.\npy-opensmtpd # The goal not being to have a Go centric API, testing with a different language required another PoC so I decided to go with Python this time.\nThe py-opensmtpd/table module also provides a work-in-progress implementation of the protocol parser, as well as a table API to ease development. Writing a table backend in Python using this module is no more complex than in Go, here\u0026rsquo;s the exact same dummy example of a table that only contains my e-mail address:\nfrom opensmtpd import table def check(token, tableName, service, key): if key == \u0026#34;gilles@poolp.org\u0026#34;: return table.boolean(token, true) return table.boolean(token, false) def lookup(token, tableName, service, key): if service == \u0026#34;alias\u0026#34; and key == \u0026#34;root\u0026#34;: # if looking up alias for root, return my address return table.result(token, \u0026#34;gilles@poolp.org\u0026#34;) return table.result(token) def fetch(token, tableName, service): # only my address is in the set table.result(token, \u0026#34;gilles@poolp.org\u0026#34;) def main(): table.on_check(check) table.on_lookup(lookup) table.on_fetch(fetch) table.dispatch() if __name__ == \u0026#34;__main__\u0026#34;: main() Again, the dispatch() function never returns as it is the event loop for the protocol parser. The interface is the exact same as what I did with go-opensmtpd.\nThe package is also a work in progress, it is also already available in a Github repository but expect it to change every few days, don\u0026rsquo;t use it unless you are also ready to suffer.\npy-consul # And because these PoC were dummy and do not show how the API can be used for real cases, I wrote a PoC for a table-backend that would actually do something more useful.\nConsul is a service to \u0026ldquo;automate network configurations, discover services, and enable secure connectivity across any cloud or runtime\u0026rdquo;. It is another tool highly valued at work and it conveniently comes with a key-value store hidden behind a REST API, something that we can work with to implement the three table backend operations.\nI implemented a table-consul python backend using the key-value store to perform its lookups. The code is a work in progress, not meant to be used in production, but it highlights how simple it is to integrate with existing tools. It took about 20 minutes to write the backend from scratch with no prior experience in Consul. The backend itself is ~70 lines including the license and empty lines. Using table-consul, one can setup a consul instance and use the key-value store to hold table informations used in OpenSMTPD lookups.\nIn the following example, I will setup and OpenSMTPD to use table-consul for aliases lookups:\ntable consul procexec:table-consul listen on all action \u0026#34;local_users\u0026#34; maildir alias \u0026lt;consul\u0026gt; [...] I then start a brand new empty consul instance and send a mail to root from my OpenSMTPD:\nThen, after adding a key root with value gilles in the foobar/alias bucket and sending a mail again, without restarting the daemon:\nThe alias lookup went to table-consul through table-procexec, table-consul did an HTTP request to consul to retrieve the alias, passed it back to an unsuspecting OpenSMTPD just as if it came from a regular table.\nBecause table-procexec can\u0026rsquo;t pass configuration to child backend yet, I decided to use \u0026lt;table\u0026gt;/\u0026lt;service\u0026gt; as the bucket for key-values which is not always very practical as it doesn\u0026rsquo;t allow sharing same keys between multiple service lookups but this was good enough for a PoC.\nLike for filter-prometheus, the table is already available in a Github repository, but this is a work in progress and was mostly to play with the new API, not intended to be used for real as I don\u0026rsquo;t have any significant experience with consul to make sound choices.\nIf you are familiar with consul and want to give me a hand in making this table production-ready, I\u0026rsquo;ll be more than happy to make this happen with your help.\nWhat\u0026rsquo;s next ? # I started this article at ~2:00AM and spent four hours writing, I can\u0026rsquo;t even remember my name at this point so I have no idea what I\u0026rsquo;ll prioritize next month, I just need some sleep.\n","date":"28 May 2020","permalink":"/posts/2020-05-28/may-2020-opensmtpd-6.7.1p1-release-table-procexec-and-many-pocs/","section":"Posts","summary":"TL;DR: Worked on the OpenSMTPD 6.7 release; Did a lot of work on the new table API; Wrote several PoCs; WARNING:\nExamples of code and configuration that appear in this article are here to help illustrate and explain development stages of my work.","title":"May 2020: OpenSMTPD 6.7.1p1 release, table-procexec and many PoCs"},{"content":" TL;DR: Worked on my webmail; Did a bit of OpenSMTPD work; Webmail vs mutt # I started using console clients to read my mail back when I was a student in early 2000s, and I\u0026rsquo;ve been using mutt for as long as I can recall now. I installed various mail clients along the years, some with graphical interfaces such as thunderbird or sylpheed, others with a web interface such as nullWebmail, SquirrelMail, Roundcube, Mailpile or currently Rainloop. No matter which, I always end up falling back to mutt for most tasks.\nThen why don\u0026rsquo;t I stick with mutt and stop trying to use other ones ?\nWether I like it or not, a lot of mails that are important to me contain MIME-parts that are just not comfortable to deal with in console. There are ways to work-around, like having lynx render HTML mails, mupdf be launched for PDF attachements, and so on\u0026hellip; but this is not comfortable enough (to me). I keep using mutt for a large part of my mails and keep some unread so that I can read them from another client.\nOk, so why don\u0026rsquo;t I stick with other ones and stop using mutt ?\nI want to be able to read my mail from various places and not rely on a heavy client. This leaves me with console-based clients that I can ssh to, and webmails that I can hit from any browser. Unfortunately for me, most webmails have interfaces that are inspired by heavy clients with framing and pagination, something that\u0026rsquo;s not practical with my use of mails.\nIf I want to cleanup the List folder which has accumulated too many mails, I\u0026rsquo;d rather use mutt which lets me scroll through the entire folder and delete mails with a key press, rather than rely on this user interface to paginate through 81 pages. This particular webmail has some keyboard shortcuts which makes it less painful than some others, but still, I have to scroll down to see the list of mails for this page and skip to the next pages to see other mails in the folder. Because of this, I will simply never use that interface to do this task and fall back to mutt which lets me do in a couple minutes.\nMy 2012 attempt at a webmail # In 2012, I wrote a small webmail for my use in python with the bottlepy mini framework, with the intent to release it opensource.\nI had not yet understood what caused me not to use them on the long run, but I knew I didn\u0026rsquo;t work along with traditional webmail interfaces. So it was mostly keyboard based and had a different look than others:\nIt also had some features I thought were interesting for my own use-case, like for example the use of robohash to immediately spot mails originating from the same sender, the detection of diffs and their highlighting to make them more readable, or a show analysis button which allowed me to display a ton of technical details about a particular sender and mail:\nUnfortunately, I got carried by ${DAYTIMEJOB} and gave up work on it before it became usable for real. It is a mail viewer with limited writing abilities.\nMy 2020 attempt at a webmail # We\u0026rsquo;re now 8 years later and I still want to scratch that itch that annoys me since then. I think my past effort had value but it didn\u0026rsquo;t tackle the issue correctly, and with 8 additional years of mail experience and nuisance, I think I can come up with something that suits me better.\nMy knowledge and the technologies available are not the same than in 2012. I\u0026rsquo;m sure I could have achieved something very usable back then but definitely not something comfortable. Part of what makes mutt comfortable is its interface, but the other part is the fluidity of being able to scroll a folder of thousands of mails. My 2012 webmail didn\u0026rsquo;t prefetch mails, didn\u0026rsquo;t do any kind of caching and did everything synchronously. Sure I could skip from a page to another with the press of an arrow key, but I then had to wait for a command to be sent to the IMAP server and for the answer to come back before the page actually changed. It was not really slow but it was definitely not really fluid.\nI decided to rewrite it from scratch, still with the intent of publishing it opensource.\nBelow is a screenshot from my initial work, start this week:\nThe webmail is now split into a backend and a frontend. The backend is written in python + flask and exposes a REST API to the IMAP server, whereas the frontend is written in react (which I had never used before this Monday) and only implements the rendering. The reason for this split is that I intend to write custom frontends too. This is too birds with one stone.\nThe screenshot is for an early work in progress so this is a \u0026ldquo;developer\u0026rdquo; interface more than a user interface. The idea here is that I have a mail listing with a cursor controled by my arrow keys, scrolling down and up will paginate mail by mail whereas left and right will paginate by full pages. Delete or d keys will move a mail to Trash, while Enter will display the mail viewer. Space will select the mail that\u0026rsquo;s below the cursor and add it to a selected list so that Delete will allow bulk deleting for example.\nOf course, I will make it easier to use for non-techies but my main goal is that I\u0026rsquo;m able to find the same comfort as mutt, THEN add buttons so that people who are more into using a mouse can find their ways.\nI\u0026rsquo;ll continue working on this every now and then until I have something that I can put up a demo for, then when I\u0026rsquo;m happy enough I will release the code for backend and frontend. Right now, having no experience with react causes me to do a lot of write and throw-away code, there\u0026rsquo;s no point in sharing code at this point.\nPreparing OpenSMTPD 6.7.0 # OpenBSD 6.7 is around the corner and OpenSMTPD 6.7.0 by the same occasion.\nAs I explained last month, I no longer contribute directly to OpenBSD as a committer but I didn\u0026rsquo;t ghost and helped review several diffs.\nBug fixes # As with any release cycle, people wake up right before a release with annoying bugs.\nThis time it was three bugs. One was a bug in the aliases expansion code and the handling of virtual aliases catchalls where the @ catch-all incorrectly caught recipients in some setups. Another was a shortcoming in the order of fields of two events in the filtering protocol that required a protocol change and version bump. The last one was a buggy root-restricted command causing the server to halt due to a fatal().\nAliasing code is particularly tricky because it doesn\u0026rsquo;t just map a recipient address to a user, it can map a recipient address to multiple users spanning over multiple domains that may or may not use aliases. It\u0026rsquo;s not a key =\u0026gt; value but a key =\u0026gt; tree resolution where each node has a different expansion context. To make things harder, the resolution can\u0026rsquo;t block as it would prevent concurrent sessions from working, so the whole expansion code is an interruptible-resumable loop which makes code simple to read but complex to follow. The first reaction to a bug affecting aliases is always denial because even simple fixes require a lot of thinking. eric@ and I confirmed and discussed the issue, we found a fix very fast that we were both sure worked, but we had to make sure that it didn\u0026rsquo;t have side-effects by testing multiple scenarios.\nThe shortcoming in the filtering protocol was easily fixed by swapping two fields but required a protocol bump, so this also needed discussions to make sure the fix was really the one we wanted and not come up with another one a month later. This also required planning for how existing filters would need to be adapted to support both 6.6.0 and 6.7.0 transparently.\nThe last issue eric@ fixed alone, I only reviewed the diff.\nAll the fixes were committed to OpenBSD and synchronized in portable, so they will not be present in the 6.7.0 release.\nPortable # The portable version of OpenSMTPD also needed a specific fix after Denis Fateyev, package maintainer for Fedora, and Ryan Kavanagh, package maintainer for Debian, reported that it failed to build with newer gcc-10. I worked on a fix which was committed to the portable version allowing it to build again and, while at it, added a gcc-10 target to our CI so we\u0026rsquo;ll detect breakages there too.\nGiven that I will spend more time working in the portable branch from now on, I also took time to set myself a proper docker-based development environment to test build and fixes on various systems.\nGithub history # Finally, I decided to tackle an issue that\u0026rsquo;s been around since forever.\nThe history for the project in our Github mirror was shit. OpenBSD is the main repository and uses cvs, we synchronize code in git by diffing the checkout from git and the checkout from git\u0026hellip; then applying the diff to git with a commit log saying sync.\nAs a result, our git repository contained detailed commit logs for portable-specific commits, but whenever trying to track a specific commit originating from OpenBSD\u0026hellip; you could not find it by commit log and you had to find the sync commit which contained the relevant diff.\nI managed to bring back the entire OpenSMTPD history from cvs to our mirror, so you can now track individual OpenBSD commits there instead of sync.\nHere\u0026rsquo;s the first import commit of OpenSMTPD into OpenBSD, which we didn\u0026rsquo;t have before.\nMore interestingly, here\u0026rsquo;s the sync commit for the fix to the aliasing bug discussed above\u0026hellip; which also contains various other commits to unrelated stuff, and here\u0026rsquo;s the actual commit to OpenBSD.\nWhat\u0026rsquo;s next ? # I have switched my part-time scheduling so that instead of a week every month, I have a day each week + 1 day each month to work on my projects. This will be the same amount of time but spread over the month.\nI\u0026rsquo;ll spend most of my spare time in May preparing the actual release of portable OpenSMTPD 6.7.0, but also to work on my webmail.\nI also have various branches I want to submit to OpenBSD for review after the 6.7.0 release, and work to add support to ECDSA in portable OpenSMTPD when OpenSSL is used instead of LibreSSL.\nIf you\u0026rsquo;re interested in my work, consider supporting me on patreon or github.\n","date":"1 May 2020","permalink":"/posts/2020-05-01/april-2020-worked-on-a-webmail-and-a-bit-of-opensmtpd-too/","section":"Posts","summary":"TL;DR: Worked on my webmail; Did a bit of OpenSMTPD work; Webmail vs mutt # I started using console clients to read my mail back when I was a student in early 2000s, and I\u0026rsquo;ve been using mutt for as long as I can recall now.","title":"April 2020: worked on a webmail and a bit of OpenSMTPD too"},{"content":" TL;DR: - I took a break to deal with personal stuff - I'm taking a long break from OpenBSD for personal reasons - I may or may not have experienced COVID-19, who knows - resumed work on OpenSMTPD and other projects This is a weird report # This is a weird report.\nI\u0026rsquo;ll mix a bit of personal info to provide some context as to why I decided to leave the OpenBSD project, explain why this doesn\u0026rsquo;t mean I won\u0026rsquo;t be an active contributor, and give some small insight into my upcoming work.\nA bit of personal context # As some may have noticed, I didn\u0026rsquo;t write a report in February: I took a small break.\nIn 2013, my body started failing me with constant headaches and nausea. I had weekly visits at my doctor to adapt non-working treatments, which then led to weekly exams to rule out a serious illness and eventually led me to being admitted in hospital for biopsy, endoscopy and other unpleasant tortures. All of this resulted in the observation that there was no organic reason to my illness, it was purely psychological. I couldn\u0026rsquo;t wrap my head around this because everything was apparently fine in my life, but I was treated accordingly for a few weeks and my physical health got much better in no time. The whole episode became a thing of the past after a while.\nLate 2018, I was diagnosed with generalised anxiety disorder and alexithymia. The general idea is that I\u0026rsquo;m pretty much constantly anxious but I\u0026rsquo;m also unaware of it most of the time: I have trouble decoding my own emotions and the related physical sensations. It is not uncommon for my better half to ask me if I\u0026rsquo;m feeling ok as she realizes I\u0026rsquo;m having an anxiety attack before/if I do. This puts a lot of perspective on the events of 2013: I don\u0026rsquo;t realise my anxiety is becoming unsustainable until my body starts to dysfunction somehow for extended periods of time.\nI\u0026rsquo;ve grown used to it and came up with fairly effective coping mechanisms. In regular times, most people don\u0026rsquo;t know about this disorder even after knowing me for years, but these are no regular times for me. I had major changes in my life with the arrival of a baby, the learning of a bad news regarding a close family member, and the doubts about staying at my daytime job.\nMy mind hasn\u0026rsquo;t been at rest in a while and keeping things under control has a cost, sadly I\u0026rsquo;m unable to properly assess when that cost is becoming too high\u0026hellip; which brings me to the next topic.\nTaking a break from OpenBSD # I\u0026rsquo;ve been an OpenBSD user since 1999 and an OpenBSD developer since 2008. OpenBSD has literally been around me for half of my life now.\nI\u0026rsquo;ve attended multiple hackathons, met plenty of skilled hackers from across the globe and been part of a project I love. A project that drove me into studying computer science and whose philosophy I\u0026rsquo;m still perfectly aligned with after all these years. I spent wonderful years and I\u0026rsquo;m a bit sad that I have to put an end to it.\nMy anxiety disorder causes me to obsess a lot about my work, and while I can easily put distance between my daytime job work and myself, it is much harder for an opensource project I\u0026rsquo;m emotionally attached to. I\u0026rsquo;m unable to disconnect and take a break, I feel a disproportionate duty towards hackers and users. I must not let people down and so I keep obsessing about what needs to be done and how it should be done. I can never find enough time to do all of it, so it leads to more anxiety and more obsession, in a vicious circle.\nThe almost two months break I just took without booting an OpenBSD laptop was a first in +10 years. I can\u0026rsquo;t recall a single vacation or week-end where I didn\u0026rsquo;t boot my laptop at least an hour to work on something.\nI decided to quit because I don\u0026rsquo;t believe I can be part of the project and work a few hours here and there, disconnecting to take a break every now and then. It\u0026rsquo;s just not in my nature. As long as I\u0026rsquo;ll be part of the team, I\u0026rsquo;ll feel a duty towards it and will be unable to put distance when needed. I have to deal with a lot of personal stuff these days and not being able to assess emotions correctly makes it hard to know my limits. The last thing I want is to push myself too far and back into hospital for being too obsessed\u0026hellip; about a hobby.\nI told the OpenBSD people I was leaving in a lengthy mail late January IIRC, and I received nothing but friendly replies. I may not be an OpenBSD hacker anymore but that doesn\u0026rsquo;t matter much, I met the coolest people.\nDoes it mean that I will no longer contribute to OpenBSD/OpenSMTPD ? # Not at all, I don\u0026rsquo;t intend to ghost OpenBSD/OpenSMTPD.\nWhat it means is that I\u0026rsquo;ll be contributing as any other user through diffs sent to specific hackers or to mailing lists. This will make it easier for me to contribute on my own schedule and will leave to others the decision of what gets in and when. Other hackers have done that in the past for diverse reasons, I wouldn\u0026rsquo;t be the first former hacker to send diffs to tech@.\nIt\u0026rsquo;ll also make it easier for me to work on other stuff. I have many other interests than but could never convince myself that my time was better spent elsewhere. You know, duty and all.\nAs far as OpenSMTPD is concerned, I\u0026rsquo;m currently sitting on diffs to convert it to libtls as well as other related features that are waiting for the 6.7 release. I will also continue my work to improve the portable version which is outside of OpenBSD\u0026rsquo;s scope and doesn\u0026rsquo;t really attract developers besides me.\nI\u0026rsquo;ll still be a main contributor, just not the lead on that project.\nSuspected COVID-19: achievement unlocked. # My wife and I were suspected COVID-19. Symptoms were mild but between the fever, the weakness, the coughing and the caring for a baby, it was not a pleasant experience.\nI\u0026rsquo;ll give it 1 star, do not recommend.\nResuming work # As I\u0026rsquo;m back from my break I don\u0026rsquo;t have much work to talk about in this report, but I\u0026rsquo;m working on my personal projects this week so expect stuff in the next one.\nI have already synchronized OpenSMTPD portable with the OpenBSD tree so it carries commits that happened by other developers during my break.\nI\u0026rsquo;m catching up on my libtls work in OpenSMTPD portable as it got interrupted by the advisory in January. I need to dive back into that code and test that it works as expected. I will also continue my work of cleanup in openbsd-compat which is bringing OpenSMTPD portable closer to a clean compat layer.\nI\u0026rsquo;m currently working on client-side mail stuff. I did a major fuckup with my mailbox and reinjected years of Trash and Junk to my Inbox. This was +30k mails to classify and it inspired me to work on a tool which I\u0026rsquo;ll continue working on and disclose when it is useable.\nFinally, I\u0026rsquo;ll catch up on the MANY notifications I received on github and the tons of mails that I accumulated in my mailbox during my break. I know there are pull requests on some filters and questions regarding OpenSMTPD. I need to catch up on all of this.\nTake care, next report will be in the usual format.\n","date":"30 March 2020","permalink":"/posts/2020-03-30/anxiety-openbsd-break-covid-19-and-resuming-work/","section":"Posts","summary":"TL;DR: - I took a break to deal with personal stuff - I'm taking a long break from OpenBSD for personal reasons - I may or may not have experienced COVID-19, who knows - resumed work on OpenSMTPD and other projects This is a weird report # This is a weird report.","title":"Anxiety, OpenBSD break, COVID-19 and resuming work"},{"content":" TL;DR: - Qualys released an advisory for a bad, bad vulnerability - an MTA is a very bad software to have a vulnerability in - hole was plugged but that's not enough, similar bugs should be mitigated in the future - article discusses what could have prevented escalation despite the bug What happened ? # Qualys contacted by e-mail to tell me they found a vulnerability in OpenSMTPD and would send me the encrypted draft for advisory.\nReceiving this kind of e-mail when working on a daemon that can\u0026rsquo;t revoke completely privileges is not a thing you want to read, particularly when you know how efficient they are at spotting a small bug and leveraging into a full-fledged clusterfuck.\nWhile I was waiting for them to mail me back the draft, I immediately jumped to the privileged process which is what worries me the most, and started reading all the entry points to that process at runtime.\nThis isn\u0026rsquo;t a big task, the privileged process doesn\u0026rsquo;t do many operations at runtime:\nit opens ~/.forward files located in home directories it authenticates username/password pair against bsd_auth(3) it creates user-owned mda processes for mail delivery agents In the case of ~/.forward files, it checks if the user has a ~/.forward file with correct permissions and mode, opens it then passes the file descriptor back to an unprivileged process without doing anything with it. I didn\u0026rsquo;t see a way this could be exploited.\nIn the case of authentication, which was my main worry given the recent vulnerabilities they found in the bsd_auth(3) layer, it received a username and password and passed them directly to auth_userokay(3), returning the result to unprivileged process. I was highly suspicious of a bug here but given how simple the code is, I didn\u0026rsquo;t see a way this could be exploited either.\nFinally, the user-owned mda processes creation became my main worry. It doesn\u0026rsquo;t do much as it basically calls fork() and executes the mda command using recipient user privileges, but it also has a special case for mbox which can\u0026rsquo;t work without privileges.\nI had hope that the bug they found would be elsewhere and not so bad as the privileged process is fairly isolated. It can only be reached by going through another unprivileged process and its input validation. The worst-case scenario would be finding an exploitable bug either in the mbox delivery special case, or in the small portion of code before privileges dropping for other mail delivery agents. I focused efforts on reading the code path again and again but didn\u0026rsquo;t find a bug. I gave up after a few hours and decided to wait for the draft.\nI woke up to an advisory draft exploiting the worst-case scenario\u0026hellip; through a logic bug in the input validation code of an unprivileged process.\nPerfect, just perfect.\nThe problem # OpenSMTPD uses an input validation function, smtp_mailaddr(), to ensure that e-mail addresses that enter the queue are valid.\nWhat the function does is that it checks that the e-mail address parses into a struct mailaddr, then validates the local part with valid_localpart() and the domain part with valid_domainpart(). The valid_localpart() function checks that the local part only contains characters from an allowed set. The valid_domainpart() either checks that the domain is a valid IP address using inet_pton(), or that it is a valid hostname using res_hnok().\nThe smtp_mailaddr() function also has to handle two cases where an invalid address is still acceptable and has to go through:\nThe first one is that of bounces (MAILER-DAEMON) with empty local and domain parts, MAIL FROM:\u0026lt;\u0026gt;, which is only acceptable during the MAIL FROM phase of SMTP.\nThe second one is that of addresses with empty domains, MAIL FROM:\u0026lt;gilles\u0026gt;, which is generated by some MUA when used locally and which will use the local domain attached to the listener, MAIL FROM:\u0026lt;gilles@laptop.poolp.org\u0026gt; on my machine. This was initially not accepted by the daemon but had to be implemented for interoperability, which is when the bug was introduced.\nUnfortunately for us, Qualys identified a logic error in smtp_mailaddr(). If one of the checks on local or domain parts fails, smtp_mailaddr() checks if it is due to one of the two cases above. In the case of failed domain, it mistakenly does an erroneous check that cancels out the previous call valid_localpart() and, as a result, an address that was rejected due to an invalid local part gets accepted again if there is no domain.\nMAIL FROM:\u0026lt;gilles;test@laptop.poolp.org\u0026gt; 553 5.1.0 Sender address syntax error MAIL FROM:\u0026lt;gilles;test\u0026gt; 250 2.0.0 Ok This mistake was introduced five years ago with the following diff:\n=================================================================== RCS file: /cvs/src/usr.sbin/smtpd/smtp_session.c,v retrieving revision 1.215 retrieving revision 1.216 diff -u -r1.215 -r1.216 --- src/usr.sbin/smtpd/smtp_session.c\t2014/07/09 12:44:54\t1.215 +++ src/usr.sbin/smtpd/smtp_session.c\t2014/10/02 21:27:54\t1.216 @@ -1,4 +1,4 @@ -/*\t$OpenBSD: smtp_session.c,v 1.215 2014/07/09 12:44:54 eric Exp $\t*/ +/*\t$OpenBSD: smtp_session.c,v 1.216 2014/10/02 21:27:54 gilles Exp $\t*/ /* * Copyright (c) 2008 Gilles Chehade \u0026lt;gilles@poolp.org\u0026gt; @@ -1758,21 +1758,20 @@ if (!valid_localpart(maddr-\u0026gt;user) || !valid_domainpart(maddr-\u0026gt;domain)) { -\t/* We accept empty sender for MAIL FROM */ -\tif (mailfrom \u0026amp;\u0026amp; -\tmaddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; \u0026amp;\u0026amp; -\tmaddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) +\t/* accept empty return-path in MAIL FROM, required for bounces */ +\tif (mailfrom \u0026amp;\u0026amp; maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; \u0026amp;\u0026amp; maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) return (1); -\t/* We accept empty domain for RCPT TO if user is postmaster */ -\tif (!mailfrom \u0026amp;\u0026amp; -\tstrcasecmp(maddr-\u0026gt;user, \u0026#34;postmaster\u0026#34;) == 0 \u0026amp;\u0026amp; -\tmaddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { +\t/* no user-part, reject */ +\tif (maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39;) +\treturn (0); + +\t/* no domain, local user */ +\tif (maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { (void)strlcpy(maddr-\u0026gt;domain, domain, sizeof(maddr-\u0026gt;domain)); return (1); } -\treturn (0); } Back then, the mistake had no consequences as OpenSMTPD had a very different implementation for delivery. It didn\u0026rsquo;t handle MDA through external utilities, like is done in other MTA, but used builtin implementations for all methods (maildir, lmtp, file, mda, \u0026hellip;). The mbox delivery was handled as a special case through a simple execle().\nWith this way of dealing with MDA, there was no shell involved and the incorrectly validated e-mail address wasn\u0026rsquo;t in the path of delivery, the bug had no consequences despite being there. This way of dealing with MDA wasn\u0026rsquo;t right though. It came with its share of complexity and short-comings, most notably a very complex mda layer handling all methods differently with many special cases, and a more complex forkmda() function which is code running in the privileged process and that we want as simple as possible.\nThe bug became exploitable with another unrelated commit three years and a half later, in a different area, when OpenSMTPD switched to the new grammar and brought support for MDA through external utilities, ditching its builtin delivery methods.\nBasically, instead of each delivery method coming with its own structure and custom checks in the entire code path, they were reimplemented as standalone executables that could be used by any MTA. From this point, OpenSMTPD didn\u0026rsquo;t know the difference between maildir and lmtp, they were just regular mail delivery agents. The MDA execution code path got simplified by removing all delivery-specific cases and unified so it would craft a command line, using what we assumed to be correctly validated input, that could be executed after the forkmda()function dropped its privileges.\nBecause the delivery agents are standalone executables that expect their options on the command line with quoting and escaping support, we decided to use the shell to execute the command rather than rewriting our own command-line parser that might contain parsing bugs. This is something done in other MTA, we didn\u0026rsquo;t invent a clever way to be lazy. The shell is executed with the recipient privileges and passed the mda command, similarly to if the recipient had executed the command themselves in their own shell. Since the command is either crafted from smtpd.conf which is controlled by postmaster, or .forward which is controlled by the recipient, there is supposedly no problem as long as the input is properly validated.\nWe could jump to the conclusion that using system() instead of execle() is what allowed the escalation but this is inaccurate. Incorrectly validated input and system() made the exploit possible, not the escalation. All delivery methods use the exact same code path and they are not all subject to the escalation. For instance, lmtp allowed unprivileged execution of commands whereas maildir is unimpacted.\nWhat made the escalation possible is a special case that OpenSMTPD has to handle specifically for mbox, and which opens the door to escalation if a bug manages to be exploited in that (relatively small) code path.\nThe fix # When I received the advisory draft, I hurried to produce a fix as soon as possible then ended with two diffs that I showed Qualys: a quick fix that just plugged the hole, and a better fix that plugged the hole by rewriting the smtp_mailaddr() function differently.\nThe quick fix is straightforward and can be implemented as a one-liner, it just tested valid_localpart() again when handling the missing domain case:\n--- src/usr.sbin/smtpd/smtp_session.c\t2019/10/04 08:34:29\t1.415 +++ smtp_session.c\t2020-01-30 08:51:31.000000000 +0100 @@ -2247,7 +2247,7 @@ smtp_mailaddr(struct mailaddr *maddr, ch return (0); /* no domain, local user */ -\tif (maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { +\tif (valid_localpart(maddr-\u0026gt;user) \u0026amp;\u0026amp; maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { (void)strlcpy(maddr-\u0026gt;domain, domain, sizeof(maddr-\u0026gt;domain)); return (1); The better fix however rearranged the function in a different way that is less error-prone, and less likely to reintroduce similar mistakes. It did so by not grouping validators into a common block, but handling each case separately:\n=================================================================== RCS file: /cvs/src/usr.sbin/smtpd/smtp_session.c,v retrieving revision 1.415 retrieving revision 1.415.2.1 diff -u -r1.415 -r1.415.2.1 --- src/usr.sbin/smtpd/smtp_session.c\t2019/10/04 08:34:29\t1.415 +++ src/usr.sbin/smtpd/smtp_session.c\t2020/01/28 21:39:20\t1.415.2.1 @@ -1,4 +1,4 @@ -/*\t$OpenBSD: smtp_session.c,v 1.415 2019/10/04 08:34:29 gilles Exp $\t*/ +/*\t$OpenBSD: smtp_session.c,v 1.415.2.1 2020/01/28 21:39:20 gilles Exp $\t*/ /* * Copyright (c) 2008 Gilles Chehade \u0026lt;gilles@poolp.org\u0026gt; @@ -2215,24 +2215,22 @@ memmove(maddr-\u0026gt;user, p, strlen(p) + 1); } -\tif (!valid_localpart(maddr-\u0026gt;user) || -\t!valid_domainpart(maddr-\u0026gt;domain)) { -\t/* accept empty return-path in MAIL FROM, required for bounces */ -\tif (mailfrom \u0026amp;\u0026amp; maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; \u0026amp;\u0026amp; maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) -\treturn (1); +\t/* accept empty return-path in MAIL FROM, required for bounces */ +\tif (mailfrom \u0026amp;\u0026amp; maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; \u0026amp;\u0026amp; maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) +\treturn (1); -\t/* no user-part, reject */ -\tif (maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39;) -\treturn (0); - -\t/* no domain, local user */ -\tif (maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { -\t(void)strlcpy(maddr-\u0026gt;domain, domain, -\tsizeof(maddr-\u0026gt;domain)); -\treturn (1); -\t} +\t/* no or invalid user-part, reject */ +\tif (maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; || !valid_localpart(maddr-\u0026gt;user)) return (0); + +\t/* no domain part, local user */ +\tif (maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { +\t(void)strlcpy(maddr-\u0026gt;domain, domain, +\tsizeof(maddr-\u0026gt;domain)); } + +\tif (!valid_domainpart(maddr-\u0026gt;domain)) +\treturn (0); return (1); } There is room for improvement and I think this function should be rewritten in a different style to make it harder to introduce bugs. But this version works, it was tested, proof-read by multiple people (including the auditors), and since it is unlikely to change before a long time (last change was in 2014), I will tackle it with a clear mind, not right now while still stressed.\nFixing the bug like I did plugs the hole, but that\u0026rsquo;s not enough for me to be relaxed, it closes the door to the escalation but doesn\u0026rsquo;t build a wall of concrete in front of it. Today, there\u0026rsquo;s no known bug that affects this code path and we may be fine, but unless we actually kill the potential for escalation, we\u0026rsquo;ll never be sure that researches won\u0026rsquo;t find another bug and another creative way to exploit it.\nIn this article, I\u0026rsquo;ll discuss mitigation mechanisms that I think should be put in place to avoid this from happening again, regardless of what bugs are found in processes exposed to attackers. Their goal is to deal with the potential for escalation, not with specific bugs.\nMy post-mortem thoughts # Even though fixing smtp_mailaddr() plugs the hole, a bug in the smtp layer should not have been able to result in such a catastrophic outcome, regardless of the bug. Today, with the combination of pledge(), privileges separation (smtp engine runs as unprivileged user _smtpd), filesystem isolation (smtp engine is chroot()-ed to /var/empty), and several internal consistency checks that will fatal() if something fishy is detected, misbehaving code is constrained and we expect limited consequences. My take has always been that I can\u0026rsquo;t code bug-free so I expect that bugs will be found, and I want OpenSMTPD to crash in such cases so that I can fix the bug and roll a release. A crash in SMTP land is really not a big deal, protocol is fault-tolerant and other mail exchangers have retry logic, I can live with that.\nThe incident here was not caused by misbehaving code, it was caused by a logic error and the code did what it was written to do: it didn\u0026rsquo;t corrupt memory and dodged sanitisation to craft an IPC message all the way to an exploit in another process. This is not a problem caused by the language, likes some people imply, and not a problem caused by a memory corruption of some sort. The sanitisation function incorrectly validated the input and from that point, the input followed the exact same code path as any other valid input until it reached the mda layer. This is not a failure of the mechanisms put in place to mitigate misbehaving code, and I think it is important to consider it for what it is: a correct implementation of an incorrect logic.\nSeeing it as such isn\u0026rsquo;t a way to shrug it off, quite the opposite. If we consider the incident without the human factor and try to address this with misbehaving code detection, it is bound to fail because\u0026hellip; the code did exactly what it was written to do. We may slap a few sanity checks here and there but if they\u0026rsquo;re supposed to check the same thing, they will have the same mistake. We can\u0026rsquo;t fix my stupid with checks.\nWe can\u0026rsquo;t prevent human mistakes, they will happen because tools won\u0026rsquo;t help spot that a human-described logic is flawed. What we need is to make changes so that OpenSMTPD becomes more resistant to human errors. In other words, we need safe-guards that are not dependant on sanity checks and input, we need safe-guards that will guarantee that even if OpenSMTPD lets completely untrusted input pass through, this will have the most limited consequences\u0026hellip; then we ensure that it doesn\u0026rsquo;t let untrusted input pass through.\nWhat made the exploit possible is the logic mistake in validation code and the use of system() with that invalid input, but what made the escalation possible is the mbox delivery method.\nThe problem(s) with mbox delivery # When a delivery takes places, OpenSMTPD does the delivery on behalf of a recipient user. Regardless of the delivery method, it will create a new mda process that will handle the delivery and drop its privileges to that of the recipient user. This is meant to ensure that a user adding a dangerous command in their .forward files can only impact themselves.\nFurthermore, OpenSMTPD strictly forbids execution of an mda command with a privileged account: root is not allowed to receive mail with maildir or lmtp, the delivery will be aborted with an error message that delivery to root is forbidden without an alias to an unprivileged user. The mda command is not processed. This is what saves maildir and lmtp from being subjected to the escalation: they cannot be ran from a privileged process, bugs will at most impact the recipient user which can never be root. Not that it\u0026rsquo;s good, but it\u0026rsquo;s an order of magnitude less bad.\nIn the case of mbox however we can\u0026rsquo;t do that.\nFirst of all, if we take a fresh install, we can\u0026rsquo;t assume that an unprivileged user exists since a user might not have created one at install. In this situation, if mbox refuses delivery to root then mails regarding security or daily/weekly/monthly will not be deliverable. Forbidding delivery to root implies requiring the creation of a user that root can be aliased to. For this reason, we can\u0026rsquo;t easily switch from mbox to another delivery method either since they would enforce the restriction, breaking mail out of the box.\nThen, unlike maildir which writes to a subdirectory of the recipient user and can therefore do it with the recipient user privileges, mbox writes to a common directory, usually /var/mail or /var/spool/mail. The mailbox is\tuser-owned, so that /var/mail/gilles can be written or read by gilles, but since gilles cannot be allowed to create his own mailbox in the common directory, the creation has to be done by a privileged process that can both create the mailbox AND change its ownership.\nSince the mbox format doesn\u0026rsquo;t support concurrent deliveries, it also requires some locking to avoid concurrent writes. This locking is historically done by creating a gilles.lock file in the common directory. Mail user agents expect this. This whole delivery logic is handled outside of the daemon by mail.local, a delivery agent that preexists OpenSMTPD and which we inherited from the previous era. The downside is that mail.local requires being called with privileges to do the work and handle privileges dropping itself.\nThis is where the potential for escalation resides.\nWhen mbox delivery is set, OpenSMTPD HAS to create a privileged process to execute mail.local no matter what user will receive the mail. It therefore has an explicit special case that forking a privileged mda is forbidden\u0026hellip; except for mbox. Even if root had a ~/.forward explicitely calling mail.local, it would be be forbidden.\nThe code path is relatively small: the mda process is created with privileges, mail.local is executed and will internally lower its privileges. Unfortunately for us, the exploit basically managed to replace the execution of mail.local with the execution of an arbitrary command, sitting in between the privileged process being fork()-ed and mail.local dropping privileges.\nThe source of the problem is that unlike other delivery methods, when using the mbox delivery method OpenSMTPD is required to create this privileged process. In this case the exploit targeted the way we executed the mail delivery agent but, thinking further, if an exploit was found that managed to abuse anything between the fork() and the execution of mail.local, we could be in a similar situation using a different attack vector: during a small period of time, we have a privileged process with a potential for being exploited somehow.\nThis is why I believe plugging the hole is not enough. It solves the immediate problem but not the underlying issue, it doesn\u0026rsquo;t kill the potential for escalation: if root is allowed to receive mail in a root-owned mailbox then at some point a privileged process had to do the delivery.\nThe only way to kill this potential for escalation is to get rid of the privileged mda process. No special case, no nothing. If we don\u0026rsquo;t have a privileged mda process, then a bug may exist and an exploit may find its way into the process, but it won\u0026rsquo;t allow escalation.\nThis is where we must be heading towards and it\u0026rsquo;ll require some creative solutions to work around mbox limitations.\nSeveral ideas # Since this blew up to my face, I had several ideas to tackle this. Some were already discussed and not retained because they had potential for other issues. The current ideas are these:\nswitching back mail delivery agents to execle() disallowing delivery to root managing to somehow run mail.local without privileges mbox creation and locking in OpenSMTPD to allow user execution of mail.local Not each of them is straight-forward but each of them would have avoided the escalation, regardless of the bug being present, so I think these are critical to implement in order to avoid future bugs from having the kind of impact we experienced.\nSwitching back mail delivery agents to execle() # This doesn\u0026rsquo;t solve the underlying issue of having a privileged process for mbox, but it would have prevented this specific attack.\nMoving all mail delivery agents to execle() is tricky but we\u0026rsquo;re going to got that way. I have already written a diff to switch mbox and it will be committed shortly. The maildir and lmtp will be similarly converted.\nThe only issue with that move is that some mail delivery agents expect a command-line interface that\u0026rsquo;s more complex than ours:\naction foobar mda \u0026#34;/usr/local/bin/procmail -a -ton -of -options \\\u0026#34;with quotes and escaping\\\u0026#34;\u0026#34; or it\u0026rsquo;s ~/.forward variant:\n|/usr/local/bin/procmail -a -ton -of -options \u0026#34;with quotes and \\\u0026#34;escaping\\\u0026#34;\u0026#34; or even it\u0026rsquo;s /etc/mail/aliases variant:\nlists:\t|/usr/local/bin/mailing-list.py -a -ton -of -options I doubt anyone wants to see these go away, and switching them to execle() implies reimplementing a command-line parser with shell-like semantics, then using the parsed command line to call the execle() function. This is the literal definition of rewriting a shell and I\u0026rsquo;d rather rely on a standard shell than rewriting my dumbed-down one.\nSince these commands can never run with privileges, what I suggest is that we use a wrapping mail delivery agent with a simple interface that we can easily plug through execle(), then let THAT mail delivery agent run the mda command through a shell. This allows us to have a single code path through execle() for all mail delivery agents in the daemon, and restrict the system() call to specific use-cases handled by a specific mail delivery agent outside the daemon.\nDisallowing delivery to root # No need for much explanations here: this would have effectively prevented the privileges escalation despite the logic bug.\nI think we should disallow deliveries to privileged users unconditionally. I never thought it was a good idea to accept doing it and this is why I disallowed it for everything but mbox. This is going to be inconvenient to many because it means that root HAS to be an alias, but everyone using maildir or lmtp has been in this mode for years now and I never had a complaint about it. We can\u0026rsquo;t tell people not to log in as root and use doas or sudo, then ensure that everything is convenient for them to log in as root and have their mails delivered knowing the risks. It doesn\u0026rsquo;t make sense.\nThis will put us at odds with other MTA implementations that accept delivering mail to root, and every time I suggested doing something differently from other MTA people got angry at me for not being able to keep their habits. But now that this blew to my face I don\u0026rsquo;t think there\u0026rsquo;s an excuse for keeping a time-bomb that we know can blow hard. I hope there will be no resistance to this change, but at some point, if we refuse to diverge from dangerous practices it means we are knowingly leaving potential for escalation. Someone else will have to defend why this is a good idea.\nWhile at it\u0026hellip;\nOn the somewhat related side-note of doing something I dislike because other MTA do it and we shouldn\u0026rsquo;t diverge, supporting commands in aliases is also a bad idea from a security point of view. It doesn\u0026rsquo;t come with the same level of risks, but it also opens the potential for abuse and I\u0026rsquo;ve been willing to get rid of that for years. I don\u0026rsquo;t know of ANY use-case of executing a command from aliases that can\u0026rsquo;t be solved from a ~/.forward file in a safer way. It\u0026rsquo;s less convenient and it requires a dedicated user, but it\u0026rsquo;s safer.\n/* this battle needs to be fought ... */ if (xn-\u0026gt;type == EXPAND_FILTER \u0026amp;\u0026amp; strcmp(ep-\u0026gt;mda_user, SMTPD_USER) == 0) log_warnx(\u0026#34;commands executed from aliases \u0026#34; \u0026#34;run with %s privileges\u0026#34;, SMTPD_USER); managing to somehow run mail.local without privileges # Disabling deliveries to privileged users is the right thing to do, however this is not sufficient if mail.local still requires privileges to deliver to gilles. The problem is the requirement for privileges during delivery, not the fact that a user itself is privileged.\nI\u0026rsquo;ve already explained the problem with mail.local and why it requires privileges. We\u0026rsquo;re discussing with other OpenBSD hackers the option to change permissions of /var/mail, the option to use a setgid mail.local and we\u0026rsquo;ll continue exploring similar options.\nThe idea for a group-writeable mail spool was apparently rejected in the past but it may be time to reassess. I don\u0026rsquo;t really have a good intuition about doing this, I don\u0026rsquo;t know why but I\u0026rsquo;ve been wrong before so it\u0026rsquo;s worth investigating this option with the help of other developers.\nThe bottom line is that unless mail.local can be started without privileges, no effort to reduce the potential for escalation will succeed. At the same time, we need to keep in mind that mail.local is not an OpenSMTPD specific component, and changing it must be done with care to avoid breaking other software. If managing to use mail.local without privileges requires making changes to it, we may have to implement an OpenSMTPD specific implementation.\nmbox creation and locking in OpenSMTPD to allow user execution of mail.local # Finally, a proposal I discussed with other hackers was to have OpenSMTPD handle the mailbox creation and locking.\nWe already consider mbox as a special case and we could convert that special case from \u0026ldquo;needs privileges\u0026rdquo; to \u0026ldquo;needs locking\u0026rdquo;. In this scenario, when a delivery uses mbox, the daemon can do the locking and mailbox creation itself before dropping privileges and executing mail.local unprivileged, like any other mail delivery agent. The lock would be removed upon SIGCHLD of the mail delivery agent. This would effectively allow removing the reasons for which mail.local requires privileges.\nAs for mail.local, all it would need to do is allow execution from unprivileged user by removing the explicit check for root it has today. This would still allow to use it with other MTA that expect it to be privileged, while allowing it to run unprivileged with OpenSMTPD. As for locking, no changes would be required since already supports skipping locking through the -L option.\nSome people see this as doing part of the delivery in OpenSMTPD and part in mail.local. I disagree and see this as executing a mail delivery agent in locking barriers: the daemon sets a locked environement around mail delivery agent execution, the mail delivery agent doesn\u0026rsquo;t know or care about the locks. This is word plays but seeing it this way makes more sense to me and doesn\u0026rsquo;t sound hackish at all.\nThis coupled with preventing delivery to root would effectively remove the potential for escalation in mbox.\nClosing words # I don\u0026rsquo;t know where to start, this section will not be as structured as previous ones.\nFirst of all, I\u0026rsquo;d like to thank the people at Qualys. They found a critical bug that could have remained out of sight until exploited by less friendly people. I won\u0026rsquo;t say it is a pleasure to be on my side of the advisory but as unpleasant as it is, it makes the software safer on the long run. I hope they don\u0026rsquo;t find something else, and if they do I hope it\u0026rsquo;s not something as big, but it is a good thing for users to have such competent people studying the code, as painful as it is for the developers when they find something.\nThen, people should use this as a reminder not to put too much trust in it. An MTA is a piece of software that can\u0026rsquo;t drop all privileges, otherwise it would already be done, and at the same time a software that is highly exposed to user-supplied data from untrusted sources. Having an MTA, whichever it is, exposed to the wild and not running on its very own machine will always be a risk. As careful as I can be, I\u0026rsquo;ll make errors and some of these errors will slip in. The only thing I can promise is that I always handle them the way I handled this one, not just fixing the issue but trying to kill the potential.\nThat being said, I will continue discussing the ideas from this article and possibly others to start implementing them as soon as possible. My goal is to kill the potential for escalation in forkmda() before the next major release. I have a few weeks in front of me and there may be some very quick wins to do here that would remove the existing risks.\n","date":"30 January 2020","permalink":"/posts/2020-01-30/opensmtpd-advisory-dissected/","section":"Posts","summary":"TL;DR: - Qualys released an advisory for a bad, bad vulnerability - an MTA is a very bad software to have a vulnerability in - hole was plugged but that's not enough, similar bugs should be mitigated in the future - article discusses what could have prevented escalation despite the bug What happened ?","title":"OpenSMTPD advisory dissected"},{"content":" TL;DR: - brought back libasr to OpenSMTPD, it is no longer an external dependency - libtls-enabled OpenSMTPD is now a thing - documented filters and improved reporting No shiny feature this month, ungrateful work # OpenSMTPD has had quite a few features implemented since its latest major release. As we get closer and closer to the next major release, my work on new features will slow down to focus more on getting the release in shape.\nI spent this month working on plumbing and stuff that\u0026rsquo;s not necessarily very visible to users, as well as multiple features that will NOT be part of the next release but will be committed shortly after release is tagged.\nThis month I focused on simplifying OpenSMTPD portable packaging by removing the need for libasr dependency. I also worked on making libtls-enabled OpenSMTPD possible for portable, a long awaited change that was not trivial and that will be skipped for the next release as it is very invasive.\nlibasr brought back to compat layer # For various purposes, OpenSMTPD relies a lot on DNS lookups and requires them to be non-blocking.\nIn May 2009, because we didn\u0026rsquo;t want threading in the daemon, a former developer implemented this using a fork()-based approach. In November 2010, I replaced the fork()-based non-blocking resolver with asr, a then experimental asynchronous resolver written by eric@, which uses an asynchronous-friendly interface that can be used with a multiplexer. In other words, asr allows DNS lookups to happen in parallel without multi-processing or multi-threading, and results to be notified through select(), poll() or similar interfaces (libevent in the case of OpenSMTPD).\nAs asr is not a standard interface, OpenSMTPD initially shipped it with its compat layer so it could be built on other BSD \u0026amp; Linux. Back then, the release process didn\u0026rsquo;t include major / minor / portable-only releases, and asr was still a moving target with bugs being spotted every few weeks. This led to a very annoying situation as bugs in asr required a new OpenSMTPD release, sometimes a few days apart.\nWe decided to split asr from OpenSMTPD and have it as a standalone library, with its own releases. This was a bit inconvenient because it meant package maintainers had to to maintain two packages, one for OpenSMTPD and one for asr, but it allowed us to decorellate both releases which was very helpful at that point.\nYears passed, OpenSMTPD improved its release management and asr stabilized so much that four years passed without a need for release, but this split was not reconsidered\u0026hellip; until recently.\nIn 2019, after filters were released, I started working heavily on improving our portable layer which suffered multiple shortcomings. While discussing with eric@, we came up with the conclusion that asr should now be brought back to OpenSMTPD compat layer: unlike years ago, rolling a portable-specific release to fix a bug in the compat layer is something we do, and it is now inconvenient for us to have the two split apart.\nThe compat layers had diverged a bit so I did the work to bring back asr to OpenSMTPD, conditionally compiling it if not available on the system. Starting with the next major version, 6.7.0 due sometime in April/May, the portable release of OpenSMTPD will no longer require a libasr dependency and --with-libasr configure flag. It will autodetect if it is available on the system and conditionally build the compat layer one if it isn\u0026rsquo;t.\nlibtls \u0026hellip; for systems without LibreSSL # To quote the LibreSSL home page:\nLibreSSL is a version of the TLS/crypto stack forked from OpenSSL in 2014, with goals of modernizing the codebase, improving security, and applying best practice development processes.\nAmong the improvements that LibreSSL brought was a new libtls library, described as follow:\nlibtls: a new TLS library, designed to make it easier to write foolproof applications.\nIndeed, as someone who started using OpenSSL and libssl as a student during the 2000\u0026rsquo;s, I can attest that using the library is overly complex and writing foolproof applications is a challenge. I won\u0026rsquo;t expand more because this post is not about OpenSSL bashing. Lets just say that it\u0026rsquo;s easy to write code that works using libssl, but it\u0026rsquo;s very hard to know if that code is correct or overlooking an option or a flag somewhere. Part of this is because crypto is not easy and you must know what you\u0026rsquo;re doing, but a bigger part of this is because the API doesn\u0026rsquo;t make it any easier: its attempt at being extremely flexible and allowing a developer to do pretty much anything provides multiple ways for a developer to fail and shoot their own feet.\nThe libtls interface is a wrapper around libssl which exposes a much simpler interface. It provides sane defaults, so that missing something somewhere doesn\u0026rsquo;t result in a less secure setup, but it also provides less possibilities of tweaking everything which is often better for most people. This results in TLS code that\u0026rsquo;s considerably smaller, easier to understand and less error-prone.\nI have had in my plans to convert OpenSMTPD to libtls since it was first released, but couldn\u0026rsquo;t switch to that interface because it requires LibreSSL and many systems still do not provide a package for it: a switch to libtls would essentially kill OpenSMTPD portable. Another option could be to build a standalone libtls that carries LibreSSL\u0026rsquo;s libssl, which is just what reyk@ did. His approach is a very good trade-off and I hope it catches with his standalone libtls becoming widespread.\nIn the meantime, OpenSMTPD cannot use libtls interface without breaking on systems that didn\u0026rsquo;t package his work and\u0026hellip; experience shows it might take time. A time during which we\u0026rsquo;re still forced to use libssl interface, writing complex code that no one is confident reviewing. This is highly annoying because multiple features were put on hold until the libtls switch, some have been deferred for so long that its a bit difficult to defer them some more. At some point we need to either implement them using the libssl interface, which will take time and efforts for code that we don\u0026rsquo;t intend to keep, or find a way to use the libtls interface with OpenSSL\u0026rsquo;s libssl as a \u0026ldquo;degraded\u0026rdquo; mode.\nSince libtls is a wrapper around libssl, you\u0026rsquo;d figure that this shouldn\u0026rsquo;t be too hard BUT LibreSSL and OpenSSL diverged on a few things, the libssl interface now has a few differences in terms of structures, functions available and options to these functions. You can\u0026rsquo;t just grab libtls and build it with OpenSSL, it won\u0026rsquo;t for a variety of reasons.\nI didn\u0026rsquo;t want to spend time working on an OpenSSL version of a standalone libtls: I believe there are good reasons why LibreSSL was forked and I\u0026rsquo;d rather see more people use LibreSSL than endorsing OpenSSL myself. So\u0026hellip; I found a middle-ground which seemed the most pramatic to me.\nI have brought libtls from latest LibreSSL into the compat layer of OpenSMTPD, then did the strict minimum to allow it to build and link with OpenSSL to cover OpenSMTPD\u0026rsquo;s needs and nothing more. This allowed me to convert OpenSMTPD to the libtls interface simplifying an awesome lot of code, while being able to build on any system wether it ships LibreSSL or not.\nSo how does this differ from a standalone libtls ?\nOpenSMTPD\u0026rsquo;s configure will detect if LibreSSL is installed and use its libtls if available. If it\u0026rsquo;s not available, it will detect if a libtls is available (reyk@\u0026rsquo;s work, for example), then if neither one is available it will build a dumbed down version that contains only what OpenSMTPD needs. Using that version may not work on another project. This has the benefit that LibreSSL is always prioritized if available, while still allowing us to switch to libtls if it isn\u0026rsquo;t. This is not ideal as it adds complexity to the compat layer in OpenSMTPD, but this complexity allows unlocking the libtls conversion which will considerably simplify OpenSMTPD itself, so I think it is a fair choice.\nThe work to build that compat libtls is done and so is the conversion of OpenSMTPD to the libtls interface. It is very invasive and, like any huge work, it has a potential for introducing regressions. Given that there\u0026rsquo;s only a few weeks before the next major release, I won\u0026rsquo;t merge this for the next major release but skip it so we have a full development cycle to spot shortcomings.\nThe next major release of OpenSMTPD, version 6.7.0 due April/May, will still use the libssl interface but later releases starting with version 6.8.0, due October/November, will use the libtls interface.\nImplemented multiple feature requests # Building on top of my libtls work, I have implemented several of the features that were requested and left pending waiting for the switch. They are committed in separate branches and waiting for the libtls work to be merged:\nselectable ciphers per listener selectable curves per listener selectable TLS protocol version per listener selectable SNI per listener (they\u0026rsquo;re currently global) Other features are being worked on but not finished yet:\nOCSP check certificate fingerprint verifications Started documenting the filter API # I have also committed a smtpd-filters(7) man page, which doesn\u0026rsquo;t get installed yet, that documents the protocol and events.\nThe feedback so far has been good and resulted in a few changes, but having more people read it and comment would be nice.\nMore smtp-out reporting improvements # I discussed smtp-out reporting in my last report, but it wasn\u0026rsquo;t finished and didn\u0026rsquo;t cover all events.\nI spent time to get this done so that we have events parity between smtp-in and smtp-out. If you enable reporting for smtp-in and smtp-out then accept a mail for relaying, you will first see the events happening from smtp-in as you accept the mail (server), then see the same events from smtp-out as you relay the mail (client).\nThe message id and transaction id are preserved so that it is possible to track the path of an envelope through its entire life.\nWhat next ? # I\u0026rsquo;m contemplating mda reporting, more on that in a future report.\nI started learning Go by writing OpenSMTPD filters a few months ago and now I\u0026rsquo;m learning Rust for other purposes. Half-way through a book, I will probably work on a real project to get my hands dirty. I\u0026rsquo;m unsure what that project will be yet as I have a few ideas in my sleeves, but if I come up with anything valuable I\u0026rsquo;ll share it.\nI will write another article next week to discuss some work that could not make it into this report.\n","date":"22 January 2020","permalink":"/posts/2020-01-22/january-2020-opensmtpd-work-libasr-and-libtls/","section":"Posts","summary":"TL;DR: - brought back libasr to OpenSMTPD, it is no longer an external dependency - libtls-enabled OpenSMTPD is now a thing - documented filters and improved reporting No shiny feature this month, ungrateful work # OpenSMTPD has had quite a few features implemented since its latest major release.","title":"January 2020: OpenSMTPD work - libasr and libtls"},{"content":" TL;DR: - le greylisting est une bonne idée - ce n'est pas très pratique aujourd'hui - beaucoup de gens se passent du greylisting ou trouvent des contournements - le SPF-aware greylisting rend le greylisting utilisable à nouveau Shout out a mes sponsors ❤️ # Un GRAND MERCI à mes sponsors sur github et patreon: votre soutien est grandement apprécié !\nOù est-ce que j\u0026rsquo;ai déjà lu ça ? # Cet article est une traduction de l\u0026rsquo;article \u0026ldquo; SPF-aware greylisting and filter-greylist\u0026rdquo;, publié sur ce blog début Décembre.\nC\u0026rsquo;est mon troisième article rédigé en français depuis des années, je vous demande donc un peu d\u0026rsquo;indulgence : si vous trouvez des fautes, vous pouvez me les remonter pour que je les corrige, ou faire une pull request pour les techies.\nN\u0026rsquo;hésitez pas à partager la publication à l\u0026rsquo;aide des icônes en fin d\u0026rsquo;article et à la commenter \u0026lt;3\nBonne lecture !\nLes erreurs SMTP en quelques mots # SMTP est un protocole fail-safe qui essaie de son mieux de garantir que les messages ne se perdent pas en transit. Parmi les différents mécanismes et exigences du standard on trouve l\u0026rsquo;utilisation d\u0026rsquo;échecs temporaires.\nConcrètement, un nœud SMTP a deux états finaux pour un message : soit le message est distribué et le nœud destination le détient, soit il n\u0026rsquo;est PAS distribué et le nœud destination ne le détient pas. Dans le second cas, le protocole SMTP exige que le dernier nœud en charge notifie l\u0026rsquo;émetteur qu\u0026rsquo;une erreur est survenue. Soit en rejetant le message lors de la session, soit à l\u0026rsquo;aide d\u0026rsquo;une mail d\u0026rsquo;erreur différé (aussi connu sous le nom de MAILER-DAEMON).\nEnfin, il y a un troisième état qui n\u0026rsquo;est pas final : l\u0026rsquo;échec temporaire. Il couvre tous les cas d\u0026rsquo;erreurs qui ont pu temporairement empêcher la distribution… mais qui pourraient tout aussi bien ne plus se produire si quelque chose était corrigé sur le nœud de destination.\nContrairement aux mails distribués, que les destinataires voient, et aux mails échoués, que les émetteurs voient, les mails en échec temporaire ne sont généralement pas vus par les utilisateurs. Ils sont traités entre les Mail eXchangers (MX) qui tentent de se les retransmettre en arrière-plan. Il ne sont généralement vus par les utilisateurs que lorsque les tentatives finissent par réussir ou qu\u0026rsquo;elles échouent définitivement après que leur temps de vie a expiré.\nDans le premier cas, le destinataire va recevoir le message un peu plus tard que prévu, le plus souvent sans même avoir su que des tentatives de retransmission ont eu lieu. Dans le second cas, l\u0026rsquo;émetteur recevra une notification lui indiquant que, malgré les retransmissions, le message n\u0026rsquo;a pas pu être distribué dans le délai imparti.\nLes stratégies de retransmission en cas d\u0026rsquo;échec temporaire sont gérées différemment dans les différents logiciels, mais vous pouvez postuler qu\u0026rsquo;il y aura une retransmission dans les secondes ou minutes qui suivent un échec temporaire. Une approche populaire, l\u0026rsquo;incrément quadratique du délai, crée un délai initialement court mais qui augmente avec le nombre de tentatives en échec de sorte que plus un hôte est injoignable longtemps, plus les tentatives vers cet hôte sont espacées.\nLes échecs temporaires arrivent tout le temps, c\u0026rsquo;est quelque chose de normal dans le protocole SMTP, et en regardant les logs on verra très souvent des lignes telles que :\n4b3a6c195c1f6010 mta delivery evpid=8f26ea98359eccea from=\u0026lt;misc+bounces-4541-[redacted]=tin.it@opensmtpd.org\u0026gt; to=\u0026lt;[redacted]@tin.it\u0026gt; rcpt=\u0026lt;-\u0026gt; source=\u0026quot;45.76.46.201\u0026quot; relay=\u0026quot;62.211.72.32 (smtp.tin.it)\u0026quot; delay=0s result=\u0026quot;TempFail\u0026quot; stat=\u0026quot;421 \u0026lt;[redacted]@tin.it\u0026gt; Service not available - too busy\u0026quot; Il ne s\u0026rsquo;agit pas d\u0026rsquo;erreurs que l\u0026rsquo;on ne trouve que chez des petits serveurs personnels. Je connais deux Big Mailer Corps qui ont des machines en échec temporaire à peu près en permanence, ils ont juste suffisamment de MX pour qu\u0026rsquo;une retransmission ait peu de chances de taper le même MX deux fois de suite. Les mails sont généralement distribués à la première ou seconde tentative.\nPetit secret rien qu\u0026rsquo;entre nous : certains Big Mailer Corps se servent volontairement de ces échecs temporaires pour évaluer la réputation des émetteurs. Ils génèrent ces erreurs à des moments clefs qui leurs permettent certaines analyses sur lesquelles je reviendrai dans un futur article.\nIl y a beaucoup de choses à dire sur les erreurs SMTP, mais résumons l\u0026rsquo;essentiel pour cet article :\nIl existe un mécanisme dans le protocole SMTP qui permet à un MX destinataire de demander à un MX émetteur de retransmettre un message. La retransmission a lieu en arrière-plan, sans aucune action des utilisateurs, et elle peut provoquer un délai dans la distribution.\nLe greylisting expliqué # Le spam est une industrie du volume.\nLes spammeurs ne sont pas intéressés par cibler des destinataires spécifiques. Ils sont intéressés par cibler un grand nombre de destinataires pour que statistiquement cela augmente le nombre de destinataires qui se feront avoir.\nJe ne vais pas trop creuser ce sujet parce que c\u0026rsquo;est l\u0026rsquo;objet d\u0026rsquo;un autre article en rédaction, néanmoins vous devez avoir à l\u0026rsquo;esprit qu\u0026rsquo;il y a différents types de spammeurs. Alors que certains utilisent des MX tout ce qu\u0026rsquo;il y a de plus normaux qui honorent les demandes de retransmissions, beaucoup d\u0026rsquo;autres utilisent des outils ou des MX customisés pour NE PAS retransmettre.\nHonorer les retransmissions ralentit considérablement l\u0026rsquo;émission pour un résultat incertain. Garder trace de tous les destinataires en échec temporaire pendant des heures ou des jours pour pouvoir les retenter est loin d\u0026rsquo;être intéressant, surtout s\u0026rsquo;il n\u0026rsquo;y a pas de certitude que la retransmission n\u0026rsquo;aboutira pas en un échec permanent de type \u0026ldquo;cet utilisateur n\u0026rsquo;existe pas\u0026rdquo;. Il faut conserver un état pour se rappeller des tentatives en échec et conserver les destinataires dans la queue d\u0026rsquo;envoi. Directement ou indirectement cela impacte la capacité à envoyer en masse.\nL\u0026rsquo;idée derrière le greylisting est très simple :\nLe greylisting déclenche un échec temporaire pour les MX auquels vous ne faites pas encore confiance et garde trace des tentatives de retransmission. Si la retransmission a lieu trop tôt elle est ignorée, les spammers ne peuvent pas faire une retransmission immédiate et doivent garder un état. Si la retransmission a lieu trop tard… et bien c\u0026rsquo;est trop tard, l\u0026rsquo;implémentation peut faire un bon nombre de choses depuis l\u0026rsquo;ajout en blacklist à la dégradation de la réputation, en passant par un nouveau tour de greylisting, encore et encore et encore, … Le résultat est que pour passer le greylisting, les spammeurs sont contraints de conserver un état et de se comporter comme des implémentations correctes du point de vue de la RFC. C\u0026rsquo;est en conflit direct avec l\u0026rsquo;économie de l\u0026rsquo;envoi de masse.\nD\u0026rsquo;un autre côté, les MX qui respectent la RFC vont naturellement retenter de nombreuses fois et finir par faire une tentative dans la bonne fenêtre de temps pour être whitelistés.\nLe greylisting ne permet pas de se prémunir des spammeurs qui ont compromis des MX légitimes, ni de ceux qui envoient à l\u0026rsquo;aide d\u0026rsquo;outils qui acceptent le coût de la retransmission. Il permet par contre d\u0026rsquo;éliminer la plus grosse partie du spam : celle qui provient de machines infectées et transformées en bots à spam, ou de scripts dont le seul but est d\u0026rsquo;envoyer un maximum possible avant qu\u0026rsquo;un hébergeur ou un administrateur ne s\u0026rsquo;en rende compte.\nComment le greylisting fonctionne habituellement # Il y a différentes solutions de greylisting et elles ne fonctionnent pas toutes pareil.\nL\u0026rsquo;idée générale est que vous avez une base de donnée qui garde trace des demande de retransmissions et de la fenêtre de temps durant laquelle une tentative est considérée comme valide pour un MX. En revanche, la manière dont sont comptabilisées les tentatives d\u0026rsquo;un MX varie d\u0026rsquo;une solution à une autre. Certaines solutions ne regardent que l\u0026rsquo;adresse IP source, alors que d\u0026rsquo;autres vont également regarder le MAIL FROM et/ou le RCPT TO. Dans certains cas que j\u0026rsquo;ai pu observer, la solution regardait également l\u0026rsquo;entête Message-ID.\nLa solution spamd(8) d\u0026rsquo;OpenBSD va par exemple garder trace de l\u0026rsquo;adresse IP source, le nom d\u0026rsquo;hôte utilisé lors de la phase d\u0026rsquo;identification HELO/EHLO, l\u0026rsquo;émetteur d\u0026rsquo;envelope (MAIL FROM) et le destinataire d\u0026rsquo;envelope (RCPT TO). Pour passer le greylisting, un MX doit faire une retransmission dans la bonne fenêtre de temps, en provenance de la même IP source, en utilisant le même nom d\u0026rsquo;hôte à l\u0026rsquo;identification, depuis le même émetteur et pour le même destinataire.\nLe problème avec le greylisting et les Big Mailer Corps # Pour de petits émetteurs, le greylisting marche très bien parce que la plupart utilisent le même MX pour le trafic entrant et sortant. Le MX vous contacte, il lui est demandé de retenter plus tard, il est accepté lorsqu\u0026rsquo;il revient une minute plus tard.\nEnsuite, vous avez Gmail / Yahoo / Outlook / … qui non seulement disposent de MX différents pour le trafic entrant et sortant, mais ont également des douzaines et des douzaines de MX sortants. La situation devient la suivante : un MX vous contacte, il lui est demandé de retenter plus tard, mais vous ne voyez jamais de retransmission de ce MX puisque c\u0026rsquo;est un autre qui a retenté et à qui il a été demandé de retenter plus tard, etc…\nLe greylisting avec ces hôtes est inutilisable et résulte en des mails qui peuvent arriver avec plusieurs heures ou jours de retard, s\u0026rsquo;ils n\u0026rsquo;expirent pas tout simplement avant de vous atteindre.\nMais en même temps, le greylisting empêche une grande quantité de nuisibles de vous atteindre en tuant le trafic de presque tous les scripts et les ordinateurs compromis. Donc pour le conserver, un grand nombre d\u0026rsquo;opérateurs de mails mettent en place des bricolages pour pouvoir continuer à utiliser le greylisting pour les petits émetteurs, tout en ne l\u0026rsquo;appliquant pas pour les Big Mailer Corps.\nStratégies de contournement # J\u0026rsquo;utilise le pluriel mais en réalité toutes les stratégies reposent sur la même idée : le whitelisting.\nLe contournement consiste à trouver la liste des adresses IP des Big Mailer Corps pour leur donner un passe droit et leur permettre d\u0026rsquo;éviter le greylisting.\nEst-ce que cela met en défaut le greylisting ? Pas vraiment. Le greylisting a pour but d\u0026rsquo;empêcher les MX ne respectant pas la RFC de vous atteindre, mais les Big Mailer Corps utilisent des MX qui respectent la RFC. Lorsque la retransmission a bien lieu plusieurs fois par le même MX sortant, ils passent le test du greylisting sans souci.\nTrès bien, ajoutons-les à la whitelist alors… mais comment ?\nPendant un moment, j\u0026rsquo;ai utilisé une approche naïve consistant à whitelister le /24 de n\u0026rsquo;importe quel MX de Big Mailer Corps qui se retrouvait greylisté. Après quelques temps, ma liste était suffisamment grande pour que de nouveaux MX soient rarement hors de ma whitelist. C\u0026rsquo;était pénible, ça ne couvrait pas le cas de nouveaux MX me contactant depuis de nouvelles plages d\u0026rsquo;adresses IP, et j\u0026rsquo;ai fini par aggréger les listes de différentes personnes à la mienne pour être sûr d\u0026rsquo;en avoir le plus possible. Le résultat était une whitelist en augmentation constante.\nCe n\u0026rsquo;était pas très malin parce que la plupart des Big Mailer Corps utilisent SPF. Ils ont un enregistrement DNS qui liste avec quelles adresses IP ils nous contactent, de façons à ce que lorsque l\u0026rsquo;on reçoit un mail avec une adresse e-mail en provenance de chez eux, on puisse vérifier que c\u0026rsquo;est bien un de leurs MX autorisé qui l\u0026rsquo;a transmis. Avec ça en tête, j\u0026rsquo;ai écrit spfwalk (maintenant une sous-commande de smtpctl) en Janvier 2018, un outil qui permet de faire une énumération dans l\u0026rsquo;enregistrement SPF d\u0026rsquo;un domaine et extraire un maximum d\u0026rsquo;adresse IP possible. Il y a un article en anglais à propos de spfwalk sur ce blog si ça vous intéresse. L\u0026rsquo;outil avait pour vocation de remplacer mon approche naïve, il est loin d\u0026rsquo;être parfait, j\u0026rsquo;ai fait mon mieux.\nLa façon dont fonctionne SPF permet de vérifier facilement si une adresse IP est autorisée, mais ne permet pas d\u0026rsquo;extraire facilement une liste d\u0026rsquo;adresse pour une vérification ultérieure. Par exemple, certaines politiques SPF s\u0026rsquo;attendent à une résolution DNS inverse de l\u0026rsquo;adresse IP source pour voir si le nom d\u0026rsquo;hôte corresponds au domaine d\u0026rsquo;envoi. Dans ce type de politique SPF, il n\u0026rsquo;y a aucune adresse IP à extraire pour mettre dans une whitelist. Donc… l\u0026rsquo;énumération SPF fonctionne à peu près tant que l\u0026rsquo;on ne tombe pas sur les politiques qui ne peuvent pas être énumérées. Pas de bol pour nous, l\u0026rsquo;un des Big Mailer Corps est dans le périmètre.\nUn grand nombre de personnes sont très contentes avec l\u0026rsquo;énumération SPF. Elle améliore considérablement la situation, mais elle reste un bricolage à mon sens et je n\u0026rsquo;utilise plus cet outil parce qu\u0026rsquo;il ne me semble pas être la bonne approche.\nLa preuve de concept filter-greylist # Le mois dernier, j\u0026rsquo;ai écrit comme preuve de concept pour OpenSMTPD un filtre qui utilise une approche différente et qui est disponible sur Github.\nJe vais juste clarifier avant tout que je ne prétends pas avoir inventé cette méthode, elle m\u0026rsquo;a juste semblée être une bonne approche alors je l\u0026rsquo;ai implémentée. Il se peut qu\u0026rsquo;on trouve d\u0026rsquo;autres implémentations de cette même idée ailleurs, j\u0026rsquo;avoue que je n\u0026rsquo;ai pas vraiment cherché très fort (pour ne pas dire \u0026ldquo;du tout\u0026rdquo;). Je ne sais donc pas s\u0026rsquo;il existe d\u0026rsquo;autres implémentations similaires, et si elles existent je ne sais pas si mon implémentation exploite l\u0026rsquo;idée de la même manière.\nL\u0026rsquo;idée de base est que nous ne voulons pas vraiment que les Big Mailer Corps soient exemptés de greylisting, ils en sont exemptés parce que l\u0026rsquo;on ne sait pas considérer tous leurs MX sortants comme identiques dans un test de greylisting. En faisant une énumération SPF pour extraire leurs adresses IP, notre intention est simplement de considérer que toutes les adresses IP de tous leurs MX sortants sont identiques pour nous : elles devraient pouvoir être interchangeables dans un test de greylisting. En d\u0026rsquo;autres termes, ce que l\u0026rsquo;on veut c\u0026rsquo;est un SPF-aware greylisting (greylisting conscient de SPF).\nLa preuve de concept considère qu\u0026rsquo;il y a deux types d\u0026rsquo;émetteurs : ceux qui publient des enregistrements SPF et ceux qui n\u0026rsquo;en publient pas.\nLes MX qui ne publient pas d\u0026rsquo;enregistrement SPF sont greylistés par adresse IP source, comme cela serait le cas avec un greylisting traditionnel. Commes les Big Mailer Corps requièrent que les émetteurs fournissent un enregistrement SPF pour ne pas dégrader leur réputation, et parce que la plupart des gens veulent pouvoir émettre vers les Big Mailer Corps, la plupart des émetteurs légitimes ne devraient pas rentrer dans ce cas. Ne devraient rentrer dans ce cas que des petits émetteurs qui n\u0026rsquo;émettent pas depuis un grand nombre d\u0026rsquo;adresse, et dont le greylisting serait identique qu\u0026rsquo;il soit par adresse IP ou domaine.\nEnsuite, nous avons les MX qui publient des enregistrements SPF, englobant l\u0026rsquo;ensemble des Big Mailer Corps pour des raisons évidentes (difficile d\u0026rsquo;imposer SPF à d\u0026rsquo;autres sans se l\u0026rsquo;imposer à soi-même). Pour ceux-ci, plutôt que d\u0026rsquo;avoir recours a un greylisting par IP source, on procède à un greylisting SPF du domaine.\nEn supposant que ce soit le premier email que nous recevons d\u0026rsquo;un MX, quand une nouvelle connexion arrive, le filtre garde trace de l\u0026rsquo;adresse IP source. La session SMTP progresse jusqu\u0026rsquo;à la création d\u0026rsquo;une transaction SMTP et le client fournit l\u0026rsquo;émetteur d\u0026rsquo;envelope (MAIL FROM) :\nS: 220 in.mailbrix.mx ESMTP OpenSMTPD C: EHLO localhost S: [...] S: 250 in.mailbrix.mx Hello localhost [127.0.0.1], pleased to meet you C: MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; À ce stade, le filtre peut faire une recherche SPF pour le domaine de l\u0026rsquo;émetteur d\u0026rsquo;enveloppe, poolp.org. S\u0026rsquo;il ne trouve pas d\u0026rsquo;enregistrement SPF, il pourra garder trace de l\u0026rsquo;adresse IP source et demander une retransmission.\nS\u0026rsquo;il trouve un enregistrement SPF, il vérifie que l\u0026rsquo;adresse IP source est valide pour cet enregistrement. Si elle ne l\u0026rsquo;est pas, il peut supposer qu\u0026rsquo;elle ne provient pas du domaine émetteur, le filtre va là encore procéder à un greylisting par adresse IP source. Une approche plus stricte pourrait être de rejeter la session, mais ce n\u0026rsquo;est pas le but d\u0026rsquo;un filtre de greylisting de décider si la violation SPF est acceptable ou non.\nSi en revanche l\u0026rsquo;adresse IP source est valide pour l\u0026rsquo;enregistrement SPF, au lieu de garder trace de l\u0026rsquo;adresse IP source, le filtre garde trace du domaine et demande une retransmission.\nIgnorons dorénavant le cas du greylisting par IP source pour nous concentrer sur le greylisting par domaine.\nLors d\u0026rsquo;une seconde connexion en provenance d\u0026rsquo;un domaine dont le filtre avait gardé trace, si l\u0026rsquo;adresse IP source est valide pour l\u0026rsquo;enregistrement SPF, le filtre va pouvoir valider que le domaine était déjà en demande de retransmission depuis une autre addresse IP et considérer le test comme réussi. En d\u0026rsquo;autres termes plus simples, si Gmail vient d\u0026rsquo;une adresse IP, il est autorisé à venir d\u0026rsquo;une autre addresse IP dès lors que les deux sont déclarées dans l\u0026rsquo;enregistrement SPF.\nCette résolution SPF en live permet de garantir qu\u0026rsquo;il n\u0026rsquo;y a pas besoin de whitelister au préalable les adresses IP. À la place il est possible de whitelister des domaines indépendamment des adresses IP avec lesquelles ils nous contacterons. Cette façon de fonctionner est de plus compatible avec les domaines qui ne fournissent pas d\u0026rsquo;enregistrement SPF, et qui dégradent en un greylisting par IP source.\nIl n\u0026rsquo;est pas possible d\u0026rsquo;implémenter cela dans un daemon comme spamd(8), la décision de greylister ou de laisser la session progresser est prise au moment du MAIL FROM, là ou la redirection spamd(8) est décidée par le firewall à la connexion.\nEst-ce que ça peut être amélioré ? # Je ne sais pas trop mais je ne pense pas que l\u0026rsquo;on puisse radicalement améliorer l\u0026rsquo;idée.\nIl y a surement quelques petites améliorations à faire à la marge, mais je ne pense pas que l\u0026rsquo;on puisse faire de grosses grosses avancées : un SPF-aware greylisting résout le problème du greylisting chez les Big Mailer Corps, ni plus, ni moins. Aujourd\u0026rsquo;hui avec ce filtre, je ne vois aucune différence de traitement entre Gmail et le petit domaine du coin. Les deux passent le greylisting assez rapidement et sans que l\u0026rsquo;on ne se rende vraiment compte qu\u0026rsquo;il y a eu retransmission.\nEn gardant en tête que le spam est une industrie du volume, je pense que du travail plus intéressant peut être fait en augmentant le coût de l\u0026rsquo;envoi de masse : de la même manière que les spammeurs n\u0026rsquo;aiment pas la retransmission, ils n\u0026rsquo;aiment PAS DU TOUT les hôtes lents parce qu\u0026rsquo;ils ont un impact ÉNORME sur la capacité à distribuer et sur la taille de la queue. J\u0026rsquo;ai déjà implémenté des fonctionnalités dans ce sens au sein d\u0026rsquo;autres filtres en corrélant les temps de réponse à la réputation d\u0026rsquo;un domaine.\nSi je passe plus de temps à essayer de faire d\u0026rsquo;OpenSMTPD une cible difficile pour les spammeurs, il est certain que mes prochains travaux seront dans la lignée des sanctions par délais.\n","date":"26 December 2019","permalink":"/posts/2019-12-26/spf-aware-greylisting-et-filter-greylist/","section":"Posts","summary":"TL;DR: - le greylisting est une bonne idée - ce n'est pas très pratique aujourd'hui - beaucoup de gens se passent du greylisting ou trouvent des contournements - le SPF-aware greylisting rend le greylisting utilisable à nouveau Shout out a mes sponsors ❤️ # Un GRAND MERCI à mes sponsors sur github et patreon: votre soutien est grandement apprécié !","title":"SPF-aware greylisting et filter-greylist"},{"content":" TL;DR: - wrote, reworked and translated multiple articles this month - got some goodies ready for my patrons - lots of work in OpenSMTPD's grammar, documentation and filters protocol WARNING:\nExamples of code and configuration that appear in this article are here to help illustrate and explain development stages of my work.\nThey are subject to changes and must not be considered as user documentation. By the time you\u0026rsquo;re reading this, they will likely no longer work or reflect reality.\nLoooots of minor stuff here and there # These last two months, I could carry Jules in a baby wrap and write code while he was asleep. He then unilateraly decided that I\u0026rsquo;m his mule now and I\u0026rsquo;m no longer allowed to stop walking when carrying him. Even standing still while writing code for more than five minutes straight is strictly forbidden.\nAs you can imagine, this causes lots of context switching with code sessions interrupted every few minutes, therefore he\u0026rsquo;s now known as the interrupt storm.\nAs a result I could not finish all of what I started working on, stuff that won\u0026rsquo;t make it into this report but which I intend to finish by the end of January for the next report.\nArticles rework and translations # This month, I have written an article regarding my work on \u0026ldquo; SPF-aware greylisting and filter-greylist\u0026rdquo;.\nIn addition, I have reworked my \u0026ldquo; Setting up a mail server with OpenSMTPD, Dovecot and Rspamd\u0026rdquo; article and removed the political aspect of it to make it a standalone article, \u0026ldquo; Decentralised SMTP is for the greater good\u0026rdquo;. Both of them were translated in French, \u0026ldquo; Mettre en place un serveur de mail avec OpenSMTPD, Dovecot et Rspamd\u0026rdquo; and \u0026ldquo; Decentralisons SMTP pour le bien commun\u0026rdquo;.\nThese were my last articles for this year.\nGoodies for my sponsors # I have set up a reward system for my sponsors, nothing too impressive, but I have received the first goodies, stickers and mugs, a few days ago.\nI don\u0026rsquo;t intend to start dispatching anything before mid-January, I don\u0026rsquo;t trust shipping stuff around this period of the year. I haven\u0026rsquo;t received the shirts yet anyways ;-)\nOpenSMTPD work # I thought I\u0026rsquo;d be slacking, but a lot of work has been put into OpenSMTPD since my last report, far more than I thought I actually did. Go figure.\nReworking match rules a bit # As you probably know, OpenSMTPD uses a ruleset to match envelopes:\nmatch from local for local action \u0026#34;in\u0026#34; match from local for any action \u0026#34;out\u0026#34; These rules may have a set of criterias to refine them further:\nmatch from local mail-from \u0026#34;gilles@poolp.org\u0026#34; for local action \u0026#34;in-alternative\u0026#34; match from local for any rcpt-to \u0026#34;eric@faurot.net\u0026#34; action \u0026#34;out-alternative\u0026#34; match from local for local action \u0026#34;in\u0026#34; match from local for any action \u0026#34;out\u0026#34; Because it has an implicit \u0026ldquo;local\u0026rdquo; behavior, rules may skip from local and for local, which makes ruleset more concise:\n# match from any for local action \u0026#34;in\u0026#34; match from any action \u0026#34;in\u0026#34; # match from local for any action \u0026#34;out\u0026#34; match for any action \u0026#34;out\u0026#34; The problem with implicit \u0026ldquo;local\u0026rdquo; # When I first started working on OpenSMTPD, mail operators kept mentionning two main problems with mail servers:\nthe configuration files were crazy difficult to understand and maintain it was way too easy to accidentally create an open relay for spammers I made it a project goal to have the most concise configuration file, providing sane defaults and removing anything unnecessary, so that the configuration would be easy to understand and so that it would take explicitely typing from any for any to create an open relay. One of the \u0026ldquo;cool\u0026rdquo; features was the use of implicit local, so that as explained above, you could make your configuration shorter due to all rules assuming local by default.\nBack then, other developers were also trying to get the configuration shorter, so sometimes I would get complaints that their configuration was taking four lines instead of three, or that the lines were taking ten keywords instead of eight, and I would try to find a way to express the grammar differently. I will fully take the blame for this one, at some point if you\u0026rsquo;re competing with other MTA that are using M4 or plaintext configuration files that have hundreds of keys, trying to remove one line out of four is a pissing contest.\nWith advances in OpenSMTPD, and a ruleset that became more and more flexible with many more matching criterias, trying to be as concise as possible to save two keywords became unproductive.\nIn some cases, writing rules like this can be confusing and result in errors like this one:\nmatch auth for any action \u0026#34;out\u0026#34; where a lot of users mistakenly assume that this will match all authenticated sessions for any destination and relay\u0026hellip; but since auth is only a criteria that specifies further the rule and since rules are local by default, this really translates to:\nmatch from local auth for any action \u0026#34;out\u0026#34; a rule that really only matches authenticated sessions coming from a local interface.\nThis is not the case that confuses users, another error I saw happen multiple times is the following one:\nmatch from any rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;out\u0026#34; which due to implicit \u0026ldquo;local\u0026rdquo; translates to:\nmatch from any for local rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;out\u0026#34; unless poolp.org is the machines\u0026rsquo; hostname, this will cause mails for gilles@poolp.org to be rejected, because the for criteria didn\u0026rsquo;t match much.\nThe proper way to write the rule would be:\nmatch from any for domain \u0026#34;poolp.org\u0026#34; rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;out\u0026#34; which everyone keeps simplifying as:\nmatch from any for any rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;out\u0026#34; just because rcpt-to already acts as a whitelist of recipients, so having to maintain a list of corresponding domains is overkill.\nThis highlights an issue which is that the grammar should describe the user intent, and the intent is very clear with rules like these:\nmatch auth for any action \u0026#34;out\u0026#34; match from any rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;in\u0026#34; The fact that it doesn\u0026rsquo;t work like they think is a problem. A problem that is caused by the implicit local behavior, because if from and for always had to be specified then such errors would not be possible.\nOh noes, not another grammar change ! # Nope, don\u0026rsquo;t worry, the grammar is correct as it is.\nWhat is incorrect is the allowing of implicit behaviors, like skipping from or for. These should be explicit and mandatory, the shortcomings of saving two keywords are far more annoying than the benefits.\nFurthermore, making them mandatory actually allows for shorter rules which are not doable today, because the implicit behaviors makes them confusing.\nSo we have decided to go full explicit from now on, the default configuration file will now provide both from and for even for local uses, using implicit behaviors will result in warnings at startup, then in a future release we will make it mandatory to declare them.\nThis means that ALL matching rules will ALWAYS have both a from and for, how does that make things shorter ?\nfrom used to be for source and for for domain # Initially, the from keyword was used to declare the source of a connection, or local if it could originate from any locally bound address or the Unix socket. The for keyword was used to declare a destination domain, or local if it could be destined for any domain known locally, usually localhost and domains obtained through gethostname() or the /etc/mail/mailname file. The special keyword all could be used to encompass any address when used with from and any domain when used with for.\nAs time passed by, we started adding support for other criterias, and ruleset could be expressed with intents that were no longer considering a source address or a destination domain. The mail-from and rcpt-to criterias are perfect example of this, they are often used with the intent of providing a whitelist of e-mail addresses. A lot of people use constructs such as:\nmatch from any mail-from \u0026lt;senders\u0026gt; [...] match from any for any rcpt-to \u0026lt;rcpts\u0026gt; [...] What they really want is to match a sender or recipient e-mail address regardless of the source. If we take a step back to get a larger picture, they want to use mail-from as an origin and rcpt-to as a destination, the source address and destination domain are set to any because they are\u0026hellip; just a criteria that makes no sense in these rules and that they want to discard.\nWhen you see things this way, it makes you reevaluate how from and for should be used, they are not source address and destination domain related but origin and destination related in a wider sense. Luckily for us the shift in paradigm is retro-compatible with previous grammar. Rulesets I write today will still be valid and work the same way tomorrow, however new constructs become available that better depicts the user intent, with no more ambiguous cases and shorter syntax for most cases.\nHow so ?\nLet\u0026rsquo;s take the examples above, what the user really expresses when writing:\nmatch from any mail-from \u0026lt;senders\u0026gt; [...] is the following:\nmatch from mail-from \u0026lt;senders\u0026gt; [...] And what the user really wants to express when writing:\nmatch from any for any rcpt-to \u0026lt;rcpts\u0026gt; [...] is the following:\nmatch from any for rcpt-to \u0026lt;rcpts\u0026gt; [...] Specifying any is not invalid, it is just redundant in both of these cases, but is still valid because using a source address criteria in conjunction to a sender or recipient e-mail address is still a valid use-case:\nmatch from src 192.168.1.0/24 mail-from \u0026lt;senders\u0026gt; [...] match from local mail-from \u0026lt;senders\u0026gt; [...] With that in mind, the following rules can now be expressed to match authenticated users regardless of their source address:\nmatch from auth for any [...] match from auth gilles@poolp.org for any [...] match from auth \u0026lt;users\u0026gt; for any [...] And just like in the previous example, it is still possible to filter on specific addresses:\nmatch from src 192.168.1.0/24 auth \u0026lt;users\u0026gt; for any [...] match from src 192.168.2.0/24 auth \u0026lt;users\u0026gt; for any [...] This change took more time convincing others than it took writing, but it is really the right direction. Because it makes the syntax simpler but also because ambiguous cases translate into people mailing me for help, and all of the common mistakes people do just vanish when from and for become explicit and the new constructs are available.\nWhat is fun is that once I got enough okays, I committed the change and I\u0026rsquo;m quite sure that other OpenBSD hackers didn\u0026rsquo;t even realize it as nothing changed for existing setups.\nThis will be part of the OpenSMTPD 6.7.0 release happening sometime around April/May, but it is already available in OpenBSD -current of in the master and portable branches on Github.\nI have converted multiple setups to the next syntax and they are much nicer this way.\nImprove documentation # Various improvements were done to the manual pages of OpenSMTPD.\nI have committed bits of missing information which jmc@ reworded and fixed to get nicer. There\u0026rsquo;s still work to do, most notably documenting how to plug filters and providing examples for DKIM and spam filtering, but the documentation had a lot of redundant bits reworked to make it easier to grasp.\nI spent hours writing an initial version of smtpd-filters(7), a man page describing the filter API for people willing to write their own filters. This is a work in progress and I have not linked it to the build yet, but it explains how things work at the protocol level, the various events that can happen, etc\u0026hellip;\nFrom now on, I\u0026rsquo;ll point people to this page when asked how to get started with filters, it will help me find the parts that need improvement.\nFix a couple protocol issue in filters # There were two issues in the filter API, issues which people would not notice because of their nature.\nThe first one is related to the request/response aspect of the filtering protocol. I had a working proof-of-concept fairly fast last year but it took months to refine the order of fields. I didn\u0026rsquo;t realize that with the reordering of fields, we ended up have two fields that are common to requests and responses swapped.\nWhat this means is that the request had the fields SESSION|TOKEN while responses had the fields TOKEN|SESSION, and since the parsing code was correct the code worked correctly, but from a developer perspective reading through the code or looking at raw protocol lines, this was confusing. The order of fields in responses was swapped to match the order of fields in requests, leading to a protocol version bump.\nThe second issue was related to the smtp-out reporting feature which I was working on. Until now a filter could only be attached to a listen line so it knew for sure it could register smtp-in events. With smtp-out a filter can be attached to a relay action allowing it to receive reporting events for outgoing trafic, but nothing in the protocol would let the filter know where it was attached, so a filter could not know if it had to register for smtp-in or smtp-out events.\nThe protocol has a handshake which allows the server to provide a set of key|value to filters at startup before they register events, so I have made sure the server would let the filters know to which subsystems they were attached so they can register the proper events:\nconfig|subsystem|smtp-in config|subsystem|smtp-out The smtp-out reporting # With the subsystem attachement issue fixed I could complete my work on smtp-out reporting.\nThis resulted in reworking how the smtp and mta layer registered sessions in filters, how reporting events were sent to filters, and which event made some informations available to filters.\nThe smtp-out and smtp-in reporting events are the same, only the direction changes, so there\u0026rsquo;s no protocol change but internally the concept of a session differs between the smtp and mta layer. For instance, a session begins when a client connects for smtp, but a session begins before we connect to a remote server in mta, so the availability of rdns, fcrdns, src and dest addresses doesn\u0026rsquo;t take place at the same timing for both.\nI won\u0026rsquo;t expand much because it\u0026rsquo;s not that interesting, but it required quite a bit of rework and it took me months of work on and off to get to a state where the code was clean and stable. Not all events are generated in smtp-out yet but this will be worked on and we\u0026rsquo;ll be fine before the next release.\nAll of my mail exchangers are now producing an event log for both incoming and outgoing trafic, I\u0026rsquo;m looking forward to exploit these into nice graphs.\nIntroduce a bypass action for builtin filters # The builtin filters are filters that operates within OpenSMTPD and provide a set of simple filtering capabilities. They allow attaching to different phases, matching different criterias and taking various decisions. I won\u0026rsquo;t dive into this because it was already discussed in this blog and is documented in the smtpd.conf(5) man page. There was just one thing missing, the ability to bypass filters in a specific phase.\nFor instance, you could define multiple filters:\nfilter no_rdns phase mail-from match !rdns reject \u0026#34;550 go away\u0026#34; filter no_fcrdns phase mail-from match !fcrdns \u0026#34;550 go away\u0026#34; listen on all filter { no_rdns, no_fcrdns } but you could not have a way to bypass these filters for a set of trusted hosts for example.\nThe bypass action allows a builtin filter to take the decision to not go through the chain but accept the phase, it would allow doing the following:\nfilter trusted phase mail-from match src \u0026lt;trusted_sources\u0026gt; bypass filter no_rdns phase mail-from match !rdns reject \u0026#34;550 go away\u0026#34; filter no_fcrdns phase mail-from match !fcrdns \u0026#34;550 go away\u0026#34; listen on all filter { trusted, no_rdns, no_fcrdns } The lack of a bypass mechanism caused multiple people to ask me for work-arounds, as many use-cases require being able to skip filters for a set of hosts, and OpenSMTPD didn\u0026rsquo;t provide any way to do that.\nThe bypass mechanism was not pushed to proc filters, I\u0026rsquo;m not convinced at this point that it is necessary. Builtin filters reside in the configuration file and are crafted with knowledge of how the filter chain looks like, this is not the case for proc filters so I\u0026rsquo;m unconvinced if this is a useful feature there. I may change my mind but no rush on this anyways.\nWork on multiple filters # All of my filters were adapted to cope with the protocol change described above with the swapped fields. I had to release new versions of all of them, a version that would keep working with the previous fields order and that would work with the new fields order.\nI didn\u0026rsquo;t work on new features but filter-rspamd was contributed to by @freswa, @whataboutpereira and @lfos who also made several contributions to filter-senderscore.\nThese filters live their lives now which is very cool.\nImprovement to filter-greylist # Following a discussion on our IRC channel (#OpenSMTPD @ irc.freenode.net), I made an improvement to filter-greylist which consists in detecting that a session was initiated by a local or authenticated user, and whitelisting the destination for messages.\nNot whitelisting destination doesn\u0026rsquo;t cause huge penalty with SPF-aware greylisting, but having destinations of local users whitelisted means that there\u0026rsquo;s no penalty at all for hosts from which we expect e-mails.\nWhat next ? # No more writing until 2020.\nI will take a few days off from computers then resume writing code maybe next week, hopefully finishing some of my pending works in progress so I can disclose them :-)\n","date":"24 December 2019","permalink":"/posts/2019-12-24/december-2019-opensmtpd-and-filters-work-articles-and-goodies/","section":"Posts","summary":"TL;DR: - wrote, reworked and translated multiple articles this month - got some goodies ready for my patrons - lots of work in OpenSMTPD's grammar, documentation and filters protocol WARNING:","title":"December 2019: OpenSMTPD and filters work, articles and goodies"},{"content":" TL;DR: - Pas de résumé, j'ai passé des heures à traduire, vous allez passer des minutes à lire ;) - OK… J'ai expliqué avec BIEN TROP DE DÉTAILS comment mettre en place un serveur de mail Merci à mes sponsors ! # Un énorme merci aux gens qui me sponsorisent sur patreon ou github.\nOù est-ce que j\u0026rsquo;ai déjà lu ça ? # En Août, j\u0026rsquo;ai publié un petit article intitulé \u0026ldquo; You should not run your mail server because mail is hard\u0026rdquo; (\u0026ldquo;Vous ne devriez pas héberger votre serveur de mail parce que c\u0026rsquo;est dur\u0026rdquo;) qui était, en gros, mon opinion sur les différentes raisons qui poussent les gens à décourager l\u0026rsquo;hébergement de mails. De manière très inattendue, l\u0026rsquo;article est devenu assez populaire, atteignant 100K lectures et continuant à recevoir des visites et des commentaires plusieurs mois après sa publication.\nEn Septembre, j\u0026rsquo;ai publié un autre article beaucoup plus long intitulé \u0026ldquo; Setting up a mail server with OpenSMTPD, Dovecot and Rspamd\u0026rdquo; (\u0026ldquo;Installer un serveur de mail avec OpenSMTPD, Dovecot et Rspamd\u0026rdquo;) qui décrivait pas à pas, de façon très détaillée, les étapes pour installer un serveur de mail complet jusqu\u0026rsquo;à la livraison des e-mails en boîte de réception chez un gros hébergeur de mail américain. L\u0026rsquo;article est devenu lui aussi plutôt populaire, sans pour autant atteindre le niveau du précedent article moins technique et spécifique, mais atteignant 40K lectures et continuant également à recevoir des visites et des commentaires plusieurs mois après la publication. Le contenu que vous vous apprêtez à lire est la traduction de cet article technique.\nC\u0026rsquo;est mon second article rédigé en français depuis des années, je vous demande donc un peu d\u0026rsquo;indulgence : si vous trouvez des fautes, vous pouvez me les remonter pour que je les corrige, ou faire une pull request pour les techies.\nCet article est DENSE parce que je vais vous tenir la main à un niveau frisant l\u0026rsquo;absurdité en expliquant chacune de mes actions et le pourquoi du comment. D\u0026rsquo;un point de vue purement technique, sans explications, l\u0026rsquo;article pourrait probablement tenir en quelques paragraphes.\nN\u0026rsquo;hésitez pas à partager la publication à l\u0026rsquo;aide des icones en fin d\u0026rsquo;article et à la commenter \u0026lt;3\nAh, et tant que je vous ai sous la main, je vous recommande chaudement la lecture de mon article non technique \u0026ldquo;Décentralisons SMTP pour le bien commun\u0026rdquo; qui aborde certaines des raisons pour lesquelles il nous faut davantage de serveurs de mail.\nBonne lecture !\nCet article est pour les techies et les sysadmins # L\u0026rsquo;auto-hébergement requiert un minimum de connaissances et de motivation. Ce n\u0026rsquo;est pas un truc qui se fait en deux clics, il vous faut des connaissances minimales et avoir l\u0026rsquo;envie d\u0026rsquo;y dédier un peu de votre temps. Si vous n\u0026rsquo;avez jamais touché une zone DNS ou si vous pensez que prendre une heure pour configurer quelque chose est une tâche difficile, alors autant vous prévenir de suite : ça ne va pas être simple. Je ne vous recommande pas vous lancer dans l\u0026rsquo;aventure sauf si vous VOULEZ avoir à apprendre une tonne de trucs.\nSi vous êtes un sysadmin familier avec le travail de sysadmin, alors la plupart de votre travail de sysadmin est déjà considérablement plus dur que mettre en place une instractructure mail. Pour vous, le mail ne sera pas quelque chose de difficile.\nEHLO hypno.cat # Pour cet article, nous mettrons en place un serveur de mail pour hypno.cat, un petit site web pour mon (hypothétique) activité prospère d\u0026rsquo;hypnopraticien. J\u0026rsquo;ai enregistré ce domaine il y a plusieurs années parce que j\u0026rsquo;aimais bien le nom mais il n\u0026rsquo;a jamais été utilisé pour quoi que ce soit d\u0026rsquo;autre que l\u0026rsquo;hébergement d\u0026rsquo;un merveilleux fichier animé.\nGoogle, Bing et Yahoo le connaissent ; on ne sait pas trop comment (je vais du moins faire semblant de ne pas savoir) et ont indexé la page d\u0026rsquo;accueil, du coup ils ne considèrent pas que ce domaine vient d\u0026rsquo;être acheté par un spammer pour immédiatement envoyer du mail, mais il reste virtuellement inconnu sur internet parce qu\u0026rsquo;il n\u0026rsquo;a aucun contenu, ne fais aucun lien, n\u0026rsquo;est linké depuis nul part, et n\u0026rsquo;a jamais émis ou reçu de mail depuis ou vers personne. C\u0026rsquo;est un détail plutôt important qui montre que l\u0026rsquo;âge d\u0026rsquo;un domaine à un impact sur la réputation, mais ça reste un critère parmi d\u0026rsquo;autres, on reparlera réputation dans un futur article.\nJe vais le mettre en place au fur et à mesure que j\u0026rsquo;écris cet article, depuis le moment où j\u0026rsquo;ai démarré un nouveau VPS jusqu\u0026rsquo;au moment où j\u0026rsquo;ai pu échanger un mail dans les deux sens avec mon compte Gmail. J\u0026rsquo;ai choisi Gmail parce qu\u0026rsquo;il est sur-représenté dans la population et qu\u0026rsquo;un nombre important de personnes se sont plains de problèmes de distribution là-bas. Très personnellement, je pense que Outlook est bien pire en terme d\u0026rsquo;interopérabilité, mais gardons ça pour un futur article peut-être.\nLes services de mail pour hypno.cat vont fournir du mail entrant et sortant sécurisés par TLS. Ils vont permettre de soumettre des messages depuis l\u0026rsquo;application Gmail de mon smartphone pour ne pas changer les habitudes de mes utilisateurs. Je pourrais tout aussi bien fournir un webmail, avec Rainloop ou Roundcube, ou installer un client lourd comme Thunderbird ou mutt. N\u0026rsquo;importe quel client fonctionnera ; donc je ne vais pas couvrir cette partie. Il y a de nombreuses alternatives et plusieurs articles existants sur internet pour couvrir leur mise en place. Il y a même des techniques pour assister les clients à automatiquement découvrir leur configuration, mais tout ça reste hors du périmètre de mon article, je ne me fais pas de soucis vous allez trouver tout seul.\nMon mail sortant passera les vérifications basiques de Gmail, plus précisément SPF, DKIM et DMARC, qui sont plus que suffisant pour laisser le mail arriver en inbox. Il est possible de faire beaucoup plus, mais le but de cet article est de trouver le bon équilibre entre faire assez de travail pour avoir bonne allure face aux autres serveurs et rester simple.\nLe mail entrant sera filtré pour réduire la quantité de spam, soit en tuant les connexions de mauvais émetteurs évidents dès qu\u0026rsquo;ils sont détectés, soit en proposant une classification du Spam dans un répertoire dédié pour éviter de saturer la boite de réception.\nJe pourrais m\u0026rsquo;arrêter là parce que c\u0026rsquo;est déjà un setup assez intéressant, mais je vais rajouter un peu de configuration pour apprendre à Dovecot à entrainer le filtre antispam. Il apprendra à reconnaitre le Spam lorsque je déplacerai un message dans le répertoire Spam, et à reconnaitre le mail légitime lorsque je déplacerai un message hors du répertoire Spam. Cette partie est un peu plus complexe parce qu\u0026rsquo;elle dépend de Sieve qui est loin d\u0026rsquo;être un bijou d\u0026rsquo;ingénierie. Si vous vous moquez de l\u0026rsquo;apprentissage, vous pouvez sauter cette partie ; j\u0026rsquo;ai fait sans pendant plus de dix ans. C\u0026rsquo;est un truc sympa à avoir mais loin d\u0026rsquo;être une obligation ; c\u0026rsquo;est juste mon petit cadeau bonus dans cet article.\nPré-requis # Le premier pré-requis est de savoir où vous allez faire tourner votre serveur de mail.\nAvec la propagation des connexions permanentes type ADSL ou FTTH (fibre), une connexion à domicile peut être tentante mais ce n\u0026rsquo;est pas une bonne idée parce que les espaces d\u0026rsquo;adressage IP des FAI sont souvent blacklistés, ou souffrent d\u0026rsquo;une très mauvaise réputation de départ. De plus, de nombreux FAI bloquent le trafic SMTP sortant pour éviter que les machines personelles compromises se transforment en bot à spam. Pour moi, la meilleure option est de louer un serveur dédié ou un VPS depuis une société d\u0026rsquo;hébergement après s\u0026rsquo;être assuré que le trafic SMTP sortant est autorisé là-bas.\nJ\u0026rsquo;ai loué mes serveurs dédiés chez online.net ces douze dernières années et je suis extrêmement satisfait. Vous trouverez même sur ce blog des instructions (en anglais) pour faire tourner OpenBSD sur leurs serveurs comme ce n\u0026rsquo;est pas supporté nativement. Ils ne filtrent pas SMTP, ce qui est bien parce que vous pouvez faire tourner un serveur de mail immédiatement, mais les adresses IP peuvent avoir été utilisées préalablement par de mauvais émetteurs et il vous faudra les tester. Si l\u0026rsquo;adresse IP qui vous est automatiquement allouée n\u0026rsquo;est pas bonne, soit il va falloir bucher pour gagner en réputation, soit il vous faudra en commander une additionelle en prenant bien le soin de la choisir dans un autre range, après avoir vérifié qu\u0026rsquo;elle n\u0026rsquo;est pas dans une blacklist, ou qu\u0026rsquo;elle ne souffre pas déjà d\u0026rsquo;une mauvaise réputation.\nAlternativement, pour les besoins d\u0026rsquo;une offre commerciale que je suis en train de monter, j\u0026rsquo;ai commencé à construire une infrastructure sur vultr.com (lien de parrainage). Je ne suis là bas que depuis quelques mois et je changerai peut-être d\u0026rsquo;avis, mais pour l\u0026rsquo;instant j\u0026rsquo;en suis très satisfait. Là-bas, SMTP est filtré par défaut et il faut ouvrir un ticket et expliquer quel est le projet pour le trafic mail. J\u0026rsquo;ai expliqué que je ne cherchais pas à devenir ESP et que je n\u0026rsquo;allais pas envoyer de mail commercial de masse mais plutôt gérer de l\u0026rsquo;hébergement ; ils ont été très serviables et m\u0026rsquo;ont retiré le filtrage le jour même.\nPeu importe où le serveur de mail sera hébergé, ce que vous devez vérifier c\u0026rsquo;est que :\nl\u0026rsquo;hébergeur n\u0026rsquo;a pas un passif avec l\u0026rsquo;hébergement de spammers et qu\u0026rsquo;il autorise SMTP vous avez une adresse IP dédiée au mail vous avez le contrôle du reverse DNS pour cette adresse IP votre adresse IP n\u0026rsquo;est pas déjà sur des blacklists Tant que ces pré-requis sont respectés, vous ne devriez pas avoir trop de problèmes.\nComme personne n\u0026rsquo;aime perdre de mails, il est aussi judicieux de se préparer aux incidents. On s\u0026rsquo;en protège en utilisant un serveur secondaire pour prendre le trafic lorsque le primaire est down, ou pour re-router le trafic lorsque le primaire est en incapacité de router correctement. Je ne vais pas couvrir ce sujet parce que ce n\u0026rsquo;est pas très complexe, il s\u0026rsquo;agit d\u0026rsquo;installer un second serveur qui route son trafic vers le premier et de faire le bon enregistrement DNS, après avoir lu cet article vous devriez vous en sortir seul.\nCEPENDANT, un grand nombre de personnes ont mentionnés s\u0026rsquo;être déjà fait blacklister et avoir perdu du mail, donc je pense que c\u0026rsquo;est le bon moment pour donner ce conseil : n\u0026rsquo;hébergez pas vos différents serveurs au même endroit. Vous ne voulez pas avoir tous vos serveurs au même endroit s\u0026rsquo;il y a une panne de courant ou de réseau, et vous voulez aussi éviter que si votre range IP est blacklisté en dommage collatéral d\u0026rsquo;un voisin malveillant, l\u0026rsquo;adresse IP de votre serveur secondaire est \u0026ldquo;suffisamment éloignée\u0026rdquo; pour ne pas être blacklistée aussi. De cette façon, si votre serveur primaire est en incapacité temporaire d\u0026rsquo;émettre à cause d\u0026rsquo;un blocage, vous pouvez re-router votre trafic au-travers de votre serveur secondaire le temps que le problème soit résolu.\nJ\u0026rsquo;ai personellement eu à gérer un seul blocage sur mon propre setup (bien plus en dehors) durant ces dix dernières années. C\u0026rsquo;est un peu chiant parce que les incidents tombent toujours au mauvais moment et que ce n\u0026rsquo;est jamais fun, mais j\u0026rsquo;ai re-routé le trafic au-travers d\u0026rsquo;un serveur secondaire pour le rétablir, rempli un formulaire de délistage en apportant la preuve que j\u0026rsquo;étais un émetteur légitime, puis re-routé le trafic par mon serveur primaire une fois le problème résolu. On est très loin du problème insurmontable qu\u0026rsquo;en font un certain nombre de personnes.\nPas de stress pour ceux qui ont un plan pour les pannes avant qu\u0026rsquo;elles ne surviennent.\nLa stack technique # Je vais démarrer un nouveau VPS chez vultr.com (toujours un lien de parrainage), installer la dernière version d\u0026rsquo; OpenBSD ( snapshot) parce que c\u0026rsquo;est mon système de préférence (installation non couverte dans cet article), et construire mon système de mail par dessus. Je vais partir du principe que nous sommes sous OpenBSD tout le reste de cet article, mais à l\u0026rsquo;exception de certaines commandes spécifiques pour installer des packages, la configuration est très similaire d\u0026rsquo;un système à un autre. Si vous êtes assez grands pour faire tourner un serveur de mail, vous êtes assez grands pour adapter des chemins, remplacer doas par sudo, et sinon… installez OpenBSD !\nPour la couche SMTP, qui est en charge de transférer des messages entre deux hôtes indépendemment de la manière dont les utilisateurs vont y accéder, j\u0026rsquo;utiliserais le logiciel OpenSMTPD. Il s\u0026rsquo;agit du serveur SMTP par défaut pour le système d\u0026rsquo;exploitation OpenBSD et il y a une version portable disponible sur Github avec des instructions pour le construire. Il est également disponible dans les dépôts de packages de différents systèmes et distributions Linux, peut-être tout juste à un pkg, un rpm ou un apt de vous.\nLes instructions de cet article sont pour la version 6.6.0 ou ultérieure, elles ne fonctionneront pas sur des versions précédentes. Si votre système ne fournit pas de package à jour, vous pouvez tenter de demander au mainteneur de le mettre à jour ou alors tenter d\u0026rsquo;en devenir mainteneur. Il vous faudra construire le package manuellement à partir du dernier tag contenant un p, pour portable, dans son nom sur Github.\nPour la couche IMAP, qui permet aux utilisateurs de récupérer les messages qu\u0026rsquo;ils ont reçus et y accéder depuis leurs smartphones ou webmail, j\u0026rsquo;utiliserais la dernière version de Dovecot. J\u0026rsquo;utiliserai également le package Dovecot-Pigeonhole, qui permettera d\u0026rsquo;entraîner notre solution antispam en lui faisant apprendre ce qui est du Spam et ce qui est du Ham. Si vous souhaitez utiliser un client console, en ssh, comme mutt par exemple, pas besoin de cette couche car OpenSMTPD peut distribuer dans une boite mail locale à laquelle les clients mail peuvent accéder en direct. Dans cet article, nous configurerons IMAP parce que nous voulons que les mails puissent être consultés depuis un smartphone comme une personne lambda.\nEnfin, pour la couche de filtrage du spam, j\u0026rsquo;utiliserai l\u0026rsquo;excellent Rspamd qui est bien plus qu\u0026rsquo;une simple solution antispam. Il fournit des méthodes de filtrage moderne, mais permet également l\u0026rsquo;intégration de filtrage antivirus, de signature DKIM, et d\u0026rsquo;une tonne de modules que vous pouvez choisir d\u0026rsquo;activer ou non pour régler plus finement votre setup. Dans cet article, nous utiliserons tout simplement ses capacités de filtrage du Spam et de signature DKIM, le strict minimum dont nous avons besoin.\nMe rendre accessible # SMTP est très étroitement lié avec DNS et les autres hôtes dépendent de recherches DNS pour trouver quelle machine gère le mail de votre domaine. C\u0026rsquo;est fait au-travers de la recherche d\u0026rsquo;enregistrements MX (Mail eXchanger), donc le minimum à faire pour que SMTP fonctionne est de déclarer un enregistrement MX pour votre domaine. Pour hypno.cat, la zone contiendra ce qui suit :\n;; un enregistrement A (et AAAA pour IPv6) est déclaré pour nommer le serveur de mail mail.hypno.cat A 217.69.8.253 mail.hypno.cat AAAA 2001:19f0:6801:867:5400:01ff:fee7:7af7 ;; un enregistrement MX est déclaré pour dire au monde que mail.hypno.cat gère le ;; mail pour hypno.cat ;; 0 est la préférence maximale, si nous avions un serveur secondaire il aurait un ;; nombre supérieur hypno.cat. MX 0 mail.hypno.cat. ;;hypno.cat. MX 10 mail-backup.hypno.cat. Je peux m\u0026rsquo;assurer que tout est correct en utilisant host et dig pour vérifier que le nom résoud et que la recherche MX retourne bien le bon nom :\n$ host mail.hypno.cat mail.hypno.cat has address 217.69.8.253 mail.hypno.cat has IPv6 address 2001:19f0:6801:867:5400:1ff:fee7:7af7 $ dig -t MX hypno.cat +short 0 mail.hypno.cat. À ce stade et avec ces enregistrements DNS, les autres serveurs de mail peuvent déjà trouver quel serveur est responsable pour hypno.cat, et le contacter quand ils veulent envoyer du mail à n\u0026rsquo;importe quelle adresse mail de ce domaine.\nMe faire bien voir # Être joignable n\u0026rsquo;est pas suffisant parce qu\u0026rsquo;aujourd\u0026rsquo;hui, vous DEVEZ avoir un reverse DNS (rDNS) ET un Forward-Confirmed rDNS (FCrDNS).\nIl y a une raison derrière cela. Un grand nombre d\u0026rsquo;hôtes qui émettent du spam sont des machines compromises, avec une grande portion d\u0026rsquo;entre elles qui sont des ordinateurs personnels derrière des connexions résidentielles. Beaucoup de ces connexions n\u0026rsquo;ont pas de rDNS ou de FCrDNS, ou ont des rDNS qui ressemblent à des patterns d\u0026rsquo;IP allouées dynamiquement (ex. : 123.123.123.123.dyn.adsl.example.com).\nComme il s\u0026rsquo;agit de machines compromises individuellement, et non de machines dont la configuration est sous le contrôle total des spammers, les rDNS et FCrDNS ne peuvent pas être configurés… il en résulte que la configuration de rDNS et FCrDNS est devenu une preuve de travail : il faut un rDNS et un FCrDNS correctement configuré et ne ressemblant pas à une allocation dynamique. Chez certains Big Mailer Corps, il s\u0026rsquo;agit de règles claires, décrites dans les guidelines, pour d\u0026rsquo;autres on le découvre par la sanction. Dans tous les cas, il s\u0026rsquo;agit du STRICT MINIMUM ; assurez-vous que vos noms d\u0026rsquo;hôtes ressemblent à un VRAI nom de serveur de mail (ex. : mail.hypno.cat, et non pas www.hypno.cat).\nLe rDNS est généralement hors de votre contrôle parce que dans une zone un peu spéciale, arpa, gérée par le propriétaire de l\u0026rsquo;adresse IP (en général votre hébergeur). Les FAI ne vous laissent pas toujours les configurer ; mais les hébergeurs de serveurs n\u0026rsquo;ayant pas vraiment d\u0026rsquo;autres choix, fournissent généralement un formulaire quelque part sur leurs espaces clients pour assigner un rDNS aux adresses qu\u0026rsquo;ils vous ont assignées :\nSi je configure mon rDNS pour avoir la même valeur que l\u0026rsquo;enregistrement DNS configuré plus haut, alors je passe automatiquement le test du FCrDNS. On peut le vérifier très simplement en faisant une recherche rDNS pour l\u0026rsquo;adresse IP :\n$ host 217.69.8.253 253.8.69.217.in-addr.arpa domain name pointer mail.hypno.cat. $ host 2001:19f0:6801:867:5400:1ff:fee7:7af7 7.f.a.7.7.e.e.f.f.f.1.0.0.0.4.5.7.6.8.0.1.0.8.6.0.f.9.1.1.0.0.2.ip6.arpa domain name pointer mail.hypno.cat. Puis cherchez l\u0026rsquo;adresse IP pour ce rDNS :\n$ host mail.hypno.cat mail.hypno.cat has address 217.69.8.253 mail.hypno.cat has IPv6 address 2001:19f0:6801:867:5400:1ff:fee7:7af7 Si les valeurs se retrouvent dans les deux sens, tout est d\u0026rsquo;équerre.\nAnnoncer quelles machines sont autorisées à émettre pour mon domaine # SPF est un mécanisme qui permet pour un domaine destination de déterminer si la machine qui émet était autorisée pour le domaine émetteur.\nLe mécanisme est très simple, on ajoute un enregistrement DNS à la zone de notre domaine qui déclare quels serveurs peuvent émettre. Quand le domaine destination reçoit un mail avec un émetteur de notre domaine, il vérifie si l\u0026rsquo;adresse IP du serveur émetteur en avait le droit.\nSPF n\u0026rsquo;est pas obligatoire pour SMTP mais c\u0026rsquo;est une des choses qui nous font paraître légitimes face aux destinataires. SPF est informatif par défaut ; ne pas passer un test SPF ou ne pas avoir d\u0026rsquo;enregistrement SPF n\u0026rsquo;implique pas qu\u0026rsquo;un mail termine en Spam ou soit détruit. Il est communément accepté que l\u0026rsquo;absence d\u0026rsquo;un enregistrement SPF a un impact très négatif sur la réputation d\u0026rsquo;un domaine, c\u0026rsquo;est même explicitement dit dans les guidelines de certains Big Mailer Corps, et vu comme l\u0026rsquo;enregistrement est simple à fournir… il n\u0026rsquo;y a vraiment aucune excuse pour ne pas le faire.\nIl y a plusieurs façons de configurer SPF ; dans cet example, je vais simplement le configurer pour n\u0026rsquo;autoriser QUE mon serveur de mail à envoyer du mail en mon nom :\nhypno.cat. IN TXT \u0026#34;v=spf1 mx -all\u0026#34; Si des hôtes reçoivent un mail avec un émetteur @hypno.cat qui ne provient pas de mail.hypno.cat, ils pourront prendre la décision (ou non) de rejeter le mail. Ils pourront en tout cas constater qu\u0026rsquo;il provient d\u0026rsquo;une machine qui n\u0026rsquo;était pas autorisée et qu\u0026rsquo;ils ont le droit de le rejeter.\nProuver que j\u0026rsquo;ai autorisé le message lui même. # DKIM est un mécanisme d\u0026rsquo;authentification par lequel on peut signer cryptographiquement les mails émis par notre serveur, prouvant qu\u0026rsquo;on les a vu passer et que l\u0026rsquo;on a pris la responsabilité de les laisser transiter. Les hôtes qui recoivent ces mails peuvent vérifier qu\u0026rsquo;ils étaient bien autorisés en vérifiant la signature, permettant ainsi de détecter des mails forgés hors du système de mail.\nComme SPF, c\u0026rsquo;est informatif par défaut et l\u0026rsquo;absence de signature DKIM ou une signature invalide ne veut pas dire qu\u0026rsquo;un mail ne sera pas distribué. En revanche, il est là aussi admis et décrit dans les guidelines que l\u0026rsquo;absence de DKIM aura un effet sur la réputation d\u0026rsquo;un domaine. Et comme configurer DKIM est facile, pas vraiment d\u0026rsquo;excuse pour ne pas le faire là non plus.\nC\u0026rsquo;est un petit peu plus complexe que pour SPF parce que, même si ça dépend aussi de DNS, il faut générer une paire de clés et publier la clé publique dans l\u0026rsquo;enregistrement DNS. Rien de bien foufou non plus.\nNous sommes en 2019. Je vais générer une clé RSA de 1024 bits ; je sais !\n\u0026mdash; BEGIN DIGRESSION \u0026mdash;\nLes clés RSA 2048 bits ne rentrent pas dans un enregistrement DNS TXT et doivent être tronquées en deux enregistrements, mais tout le monde n\u0026rsquo;est pas capable de réassocier ces clés tronquées, et vous pouvez perdre les bénéfices de DKIM si l\u0026rsquo;hôte destination n\u0026rsquo;arrive pas à utiliser votre clé publique partielle et considère la signature invalide.\nEn ce qui me concerne, ces hôtes sont responsables et nous pouvons les ignorer ; mais comme le but de cet article est de permettre une interopérabilité maximale et que DKIM n\u0026rsquo;est pas présent ici pour sécuriser quoi que ce soit, juste pour vous faire paraître légitime, on va juste regarder ailleurs en justifiant cette mauvaise pratique par le risque modéré de falsification de contenu quand il est utilisé conjointement à SPF.\nCe n\u0026rsquo;est pas une bonne manière de raisonner mais DKIM a d\u0026rsquo;autres inconvénients et comme virtuellement tout le monde utilise du RSA 1024-bits. Je suggère de laisser le débat 1024 vs \u0026gt;= 2048 à plus tard, lors que nous aurons déjà réussis à échanger correctement avec les Big Mailer Corps. Je prendrais juste le temps, sans pointer de doigt nul part, d\u0026rsquo;appuyer sur le fait que certaines sociétés dans la sécurité qui travaillent dans l\u0026rsquo;industrie du mail ont (avaient ?) encore des problèmes avec les clés 2048-bits en 2019. Des sociétés qui calculent des scores de réputation. Oh, la, belle, ironie !\nSi c\u0026rsquo;est un sujet d\u0026rsquo;inquiétude pour vous, vous pouvez expérimenter la troncature des clé et voir si ça vous convient. Adapter mon exemple est trivial ; assurez vous juste d\u0026rsquo;avoir la clé tronquée sur deux enregistrements TXT.\n\u0026mdash; END DIGRESSION \u0026mdash;\nJe crée un répertoire pour contenir les clés :\n# mkdir /etc/mail/dkim Ensuite, la commande suivante va générer une paire de clé et extraire la clé publique de la clé privée:\n# openssl genrsa -out /etc/mail/dkim/hypno.cat.key 1024 Generating RSA private key, 1024 bit long modulus …………………………++++++ ……………++++++ e is 65537 (0x10001) # openssl rsa -in /etc/mail/dkim/hypno.cat.key -pubout -out /etc/mail/dkim/hypno.cat.pub writing RSA key # cat /etc/mail/dkim/hypno.cat.pub -----BEGIN PUBLIC KEY----- MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDThHqiM610nwN1nmV8OMc7PaPO uJWVGRDz5bWj4cRjTTmQYjJQd02xrydRNrRhjEKBm2mMDArNWjoM3jvN04ZifqJx DmKr7X8jsYi+MPEHz6wZxB8mayDK6glYTCyx//zl1luUdvm26PutA38K8cgnb7iT kfVP2OqK6sHAdXjnowIDAQAB -----END PUBLIC KEY----- Enfin, je génére un enregistrement DNS TXT en retirant le blindage de la clé publique et en formattant le contenu pour qu\u0026rsquo;il soit comme suit :\n20190913._domainkey.hypno.cat.\tIN TXT \u0026#34;v=DKIM1;k=rsa;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDThHqiM610nwN1nmV8OMc7PaPOuJWVGRDz5bWj4cRjTTmQYjJQd02xrydRNrRhjEKBm2mMDArNWjoM3jvN04ZifqJxDmKr7X8jsYi+MPEHz6wZxB8mayDK6glYTCyx//zl1luUdvm26PutA38K8cgnb7iTkfVP2OqK6sHAdXjnowIDAQAB;\u0026#34; Le nom de l\u0026rsquo;enregistrement est construit selon le motif : \u0026lt;selector\u0026gt;._domainkey.\u0026lt;domain\u0026gt;., où \u0026lt;selector\u0026gt; est un nom arbitraire choisi pour permettre à plusieurs clés de co-exister. J\u0026rsquo;aime bien utiliser la date à laquelle j\u0026rsquo;ai généré la paire de clés, mais peu importe ce que vous choisissez, écrivez le pour plus tard car nous en aurons besoin pour configurer la signature DKIM.\nDe plus, la clé publique n\u0026rsquo;a pas besoin de rester dans le répertoire une fois qu\u0026rsquo;elle est publiée dans l\u0026rsquo;enregistrement DNS. Elle peut toujours être dérivée de la clé privée si nous en avons besoin, mais personnellement j\u0026rsquo;aime bien la garder pour la retrouver rapidement.\nFaites bien attention à ce que la clé privée ne soit pas en lecture pour tous car, pour une raison inconnue et pas très maline, OpenSSL pense que les clés privées doivent être créées en rw-r--r--.\nDire aux autres quoi faire en cas d\u0026rsquo;erreurs SPF et DKIM avec DMARC # SPF et DKIM sont tous les deux très intéressants, mais ils ne sont qu\u0026rsquo;informatifs, alors ils n\u0026rsquo;empêchent pas d\u0026rsquo;abuser de votre nom de domaine.\nDMARC permet de dire aux serveurs mails de destination ce que vous voulez qu\u0026rsquo;ils fassent des mails en apparence de votre domaine qui ratent les tests SPF et DKIM. Il permet par exemple de leur dire qu\u0026rsquo;ils doivent rejeter ces mails.\nIl n\u0026rsquo;y a pas d\u0026rsquo;indicateur clair : fournir une politique de rejet nous fait mieux voir que de fournir une politique sans action. Des expérimentations semblent montrer qu\u0026rsquo;avoir un enregistrement a un impact positif par rapport à n\u0026rsquo;en avoir aucun, même si l\u0026rsquo;enregistrement dit que rien ne doit être fait avec les erreurs SPF et DKIM. Il serait intéressant de mener une nouvelle expérimentation à ce sujet, mais en même temps, il est plus simple de fournir une politique de rejet et de ne pas avoir à se poser la question :-)\nLe format d\u0026rsquo;un enregistrement DMARC va bien au-delà de cet article et vous trouverez de nombreux exemples dans n\u0026rsquo;importe quel moteur de cherche. Un enregistrement simple annoncera une politique (p=) de none (reject si vous voulez rejeter, quarantine si vous avez besoin de temps pour décider). Le champ pourcentage (pct=) déclare combien de ces mails devraient être sujets à la politique DMARC, et le champ Reporting URI of Aggregates (rua=) est la destination des rapports DMARC (pour analyse avant de basculer de quarantine à reject par exemple).\nJe vais ici créer un enregistrement simple, un enregistrement qui fera que les Big Mailer Corps voient que nous nous intéressons à DMARC même si nous ne savons pas encore quoi en faire :\n_dmarc.hypno.cat. IN TXT \u0026#34;v=DMARC1;p=none;pct=100;rua=mailto:postmaster@hypno.cat;\u0026#34; Mon opinion est qu\u0026rsquo;il est préférable de basculer rapidement vers p=reject, après s\u0026rsquo;être assuré que les mails sont correctement signés et que seul notre serveur de mail les émet.\nObtenir un certificat pour TLS # Il y a plusieurs façons d\u0026rsquo;obtenir un certificat TLS. Je vais partir du principe que vous êtes familiers avec l\u0026rsquo;hébergement d\u0026rsquo;autres services et que vous êtes capable d\u0026rsquo;en obtenir un par votre registrar ou depuis letsencrypt. Si ce n\u0026rsquo;est pas le cas, vous trouverez des exemples à foison sur n\u0026rsquo;importe quel moteur de recherche.\nSi vous savez gérer vos certificats, vous pouvez sauter à la section suivante.\nComme je suis réellement en train de mettre en place mail.hypno.cat, je ne serais pas en mesure de continuer cet article sans obtenir de certificat, donc en bonus pour les utilisateurs d\u0026rsquo;OpenBSD, je vais documenter la génération de mon certificat sur ma nouvelle installation.\nacme-client est un utilitaire présent dans le système de base et qui permet de demander ou renouveller des certificats depuis la console. Il dépend d\u0026rsquo;un challenge HTTP ; il doit donc être en mesure d\u0026rsquo;écrire dans un répertoire qui soit accessible par HTTP. Et comme nous sommes chanceux… OpenBSD contient un serveur httpd que l\u0026rsquo;on peut utiliser pour ce challenge. Et comme nous sommmes TRÈS chanceux… OpenBSD fournit aussi un exemple de fichier de configuration que l\u0026rsquo;on peut utiliser sans gros changement.\nOn copie simplement le fichier d\u0026rsquo;example depuis /etc/example/httpd.conf vers /etc/httpd.conf, en remplaçant example.com par mail.hypno.cat et en supprimant le bloc TLS puisque seul le challenge nous interesse. Le fichier doit ressembler à cela :\nserver \u0026#34;mail.hypno.cat\u0026#34; { listen on * port 80 location \u0026#34;/.well-known/acme-challenge/*\u0026#34; { root \u0026#34;/acme\u0026#34; request strip 2 } location * { block return 302 \u0026#34;https://$HTTP_HOST$REQUEST_URI\u0026#34; } } Le daemon httpd peut être démarré comme suit :\n# rcctl -f start httpd httpd(ok) En ce qui concerne acme-client, c\u0026rsquo;est très simple, on copie à nouveau le fichier d\u0026rsquo;exemple depuis /etc/example/acme-client.conf vers /etc/acme-client.conf, en remplaçant example.com par mail.hypno.cat et on doit se retrouver avec un bloc comme suit :\ndomain mail.hypno.cat { domain key \u0026#34;/etc/ssl/private/mail.hypno.cat.key\u0026#34; domain full chain certificate \u0026#34;/etc/ssl/mail.hypno.cat.fullchain.pem\u0026#34; sign with letsencrypt } À ce stade, on se contente de lancer acme-client pour notre domaine :\n# acme-client -v mail.hypno.cat […] acme-client: /etc/ssl/mail.hypno.cat.fullchain.pem: created Le certificat et les clés sont créés au bon endroit, il suffira par la suite d\u0026rsquo;ajouter les chemins dans OpenSMTPD et Dovecot.\nVous pouvez garder httpd en marche pour les renouvellements et appeller acme-client depuis un cron, vous pouvez l\u0026rsquo;éteindre et renouveller d\u0026rsquo;une autre façon. Je vous laisse vous débrouiller ; vous trouverez comment faire avec l\u0026rsquo;aide de la page de manuel acme-client(1).\nInstaller et configurer Rspamd # Sur OpenBSD, Rspamd est packagé et peut être installé d\u0026rsquo;une simple commande. Nous avons aussi besoin d\u0026rsquo;installer Redis qui sert à stocker des statistiques pour Rspamd et son greylisting (entre autres), et également filter-rspamd qui est un petit bout de code qui permets à OpenSMTPD de travailler avec Rspamd. Ils existent également sous forme de packages et peuvent être installés en une seule commande.\nTout d\u0026rsquo;abord, on installe les packages :\nmail$ doas pkg_add redis rspamd opensmtpd-filter-rspamd […] redis-4.0.14: ok rspamd-1.9.0: ok opensmtpd-filter-rspamd-0.1.1: ok The following new rcscripts were installed: /etc/rc.d/redis /etc/rc.d/rspamd See rcctl(8) for details. Rspamd est très configurable, si vous voulez faire des choses un peu funky je vous laisse lire la documentation. Pour cet article je me contenterai d\u0026rsquo;une utilisation très basique, comme c\u0026rsquo;est le cas sur mes propres machines :\nJe pourrais éditer la configuration dans /etc/rspamd/actions.conf pour ajuster les seuils de mise en spam, mais les valeurs par défaut me conviennent alors je vous les montre juste :\nactions { reject = 15; # Reject when reaching this score add_header = 6; # Add header when reaching this score greylist = 4; # Apply greylisting when reaching this score (will emit `soft reject action`) […] Mais ce que je veux vraiment c\u0026rsquo;est que Rspamd gère la signature DKIM parce que c\u0026rsquo;est un des pré-requis pour paraître légitime.\nOn le fait en fournissant une configuration dans /etc/rspamd/local.d/dkim_signing.conf pour faire correspondre à notre domaine la clé DKIM générée plus tôt :\n# mkdir /etc/rspamd/local.d # cat \u0026lt;\u0026lt; EOF \u0026gt;/etc/rspamd/local.d/dkim_signing.conf allow_username_mismatch = true; domain { hypno.cat { path = \u0026#34;/etc/mail/dkim/hypno.cat.key\u0026#34;; selector = \u0026#34;20190913\u0026#34;; } } EOF La clé de configuration allow_username_mismatch est nécessaire parce que Rspamd attend des utilisateurs avec des noms de domaines, mais dans cette configuration OpenSMTPD authentifie de simples utilisateurs système. De plus, faites attention à ce que le fichier contenant la clé DKIM soit autorisé en lecture au groupe _rspamd.\nÀ ce stade on est bon, Rspamd et Redis peuvent être activés pour qu\u0026rsquo;OpenBSD les démarre à chaque reboot :\n# rcctl enable redis # rcctl enable rspamd Et on peut les démarrer immédiatement pour ne pas avoir à attendre le prochain reboot :\n# rcctl start redis redis(ok) # rcctl start rspamd rspamd(ok) Configurer OpenSMTPD # OpenSMTPD est installé par défaut sur OpenBSD donc aucune phase d\u0026rsquo;installation ici. Si vous utilisez un système d\u0026rsquo;exploitation différent, soit quelqu\u0026rsquo;un l\u0026rsquo;a packagé et vous pouvez l\u0026rsquo;installer avec votre gestionnaire de paquets, ou vous pouvez le construire depuis les sources et l\u0026rsquo;installer en utilisant le code depuis le mirroir Github et en suivant les instructions de build.\nComme écrit plus haut, ces instructions ne sont valides que pour la version 6.6.0 et ultérieures, les versions précédentes ne supportent pas les filtres et certaines des fonctionnalités décrites ici.\nLa configuration par défault d\u0026rsquo;OpenSMTPD est adaptée à un serveur de mail local, n\u0026rsquo;acceptant pas de connexions de l\u0026rsquo;extérieur, mais capable de laisser les utilisateurs locaux s\u0026rsquo;échanger des messages et en émettre vers des hôtes distants.\nElle ressemble à ce qui suit :\n# $OpenBSD: smtpd.conf,v 1.11 2018/06/04 21:10:58 jmc Exp $ # This is the smtpd server system-wide configuration file. # See smtpd.conf(5) for more information. table aliases file:/etc/mail/aliases # To accept external mail, replace with: listen on all # listen on lo0 action \u0026#34;local_mail\u0026#34; mbox alias \u0026lt;aliases\u0026gt; action \u0026#34;outbound\u0026#34; relay # Uncomment the following to accept external mail for domain \u0026#34;example.org\u0026#34; # # match from any for domain \u0026#34;example.org\u0026#34; action \u0026#34;local_mail\u0026#34; match from local for local action \u0026#34;local_mail\u0026#34; match from local for any action \u0026#34;outbound\u0026#34; Je voulais initialement mettre à jour la configuration progressivement pour vous tenir par la main, mais cet article à énormément grossi depuis sa première version. Par souci de simplicité, je vais simplement mettre les 16 lignes de configuration et les commenter ensuite :\npki mail.hypno.cat cert \u0026#34;/etc/ssl/mail.hypno.cat.fullchain.pem\u0026#34; pki mail.hypno.cat key \u0026#34;/etc/ssl/private/mail.hypno.cat.key\u0026#34; filter check_dyndns phase connect match rdns regex { \u0026#39;.*\\.dyn\\..*\u0026#39;, \u0026#39;.*\\.dsl\\..*\u0026#39; } \\ disconnect \u0026#34;550 no residential connections\u0026#34; filter check_rdns phase connect match !rdns \\ disconnect \u0026#34;550 no rDNS is so 80s\u0026#34; filter check_fcrdns phase connect match !fcrdns \\ disconnect \u0026#34;550 no FCrDNS is so 80s\u0026#34; filter senderscore \\ proc-exec \u0026#34;filter-senderscore -blockBelow 10 -junkBelow 70 -slowFactor 5000\u0026#34; filter rspamd proc-exec \u0026#34;filter-rspamd\u0026#34; table aliases file:/etc/mail/aliases listen on all tls pki mail.hypno.cat \\ filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } listen on all port submission tls-require pki mail.hypno.cat auth filter rspamd action \u0026#34;local_mail\u0026#34; maildir junk alias \u0026lt;aliases\u0026gt; action \u0026#34;outbound\u0026#34; relay helo mail.hypno.cat match from any for domain \u0026#34;hypno.cat\u0026#34; action \u0026#34;local_mail\u0026#34; match from local for local action \u0026#34;local_mail\u0026#34; match from any auth for any action \u0026#34;outbound\u0026#34; match from local for any action \u0026#34;outbound\u0026#34; C\u0026rsquo;est tout ce dont nous avons besoin pour notre configuration SMTP complète.\nLes deux premières lignes pki déclarent que mail.hypno.cat utilisera le certificat et la clé que nous avons générés plus tôt pour TLS.\nLes lignes filter appliquent un ensemble de filtres sur les connexions entrantes. check_dyndns filtrera si rDNS corresponds à certains motifs. check_rdns filtrera s\u0026rsquo;il n\u0026rsquo;y a pas de rDNS. check_fcrdns filtrera si la correspondance FCrDNS échoue. Il s\u0026rsquo;agit de filtres \u0026ldquo;internes\u0026rdquo; à OpenSMTPD que vous pouvez ajuster pour filtrer différement, à d\u0026rsquo;autres phases ou avec d\u0026rsquo;autres critères.\nsenderscore est un filtre custom que vous pouvez installer soit depuis votre gestionnaire de paquets (pkg_add opensmtpd-filter-senderscore sur OpenBSD), ou obtenir depuis Github. Il n\u0026rsquo;est pas obligatoire mais je le trouve particulièrement utile et expliquerai pourquoi dans la prochaine section. Dans cette configuration, il rejettera un émetteur si son senderscore est en dessous de 10, il junk-era le message (ajoutera un entête X-Spam si le score est inférieur à 70), et pour mon plaisir personnel, ajoutera avant chaque réponse un délai inversement proportionnel à la réputation de l\u0026rsquo;émetteur pour ralentir les mauvais émetteurs.\nrspamd est également un filtre custom que nous avons installé dans la précedente section. Il n\u0026rsquo;y a aucune configuration car il est entièrement controlé depuis Rspamd, il suffit donc de le déclarer.\nSi vous voulez être plus conservateur et ne pas rejeter les mail pour éviter des faux positifs, vous pouvez assigner l\u0026rsquo;action junk plutôt que disconnect aux filtres internes et supprimer l\u0026rsquo;option -blockBelow au filtre senderscore :\nfilter check_dyndns phase connect match rdns regex { \u0026#39;.*\\.dyn\\..*\u0026#39;, \u0026#39;.*\\.dsl\\..*\u0026#39; } junk filter check_rdns phase connect match !rdns junk filter check_fcrdns phase connect match !fcrdns junk filter senderscore proc-exec \u0026#34;filter-senderscore -junkBelow 70 -slowFactor 5000\u0026#34; De cette façon, au lieu de rejeter les sessions, OpenSMTPD va simplement junk les messages pour les faire atterrir en boite à Spam. En regardant régulièrement ce qui atterri dans cette boite, vous pourrez gagner en confiance et adapter vos réglages pour trouver ceux qui vous conviennent.\nCes filtres simples, sans même compter rspamd, sont suffisants pour tuer la plus grosse partie du spam. Chez moi, ils permettent de passer de plusieurs centaines de spam quotidiens à une poignée. Le filtre rdns regex doit être adapté en fonction du pays, vous pouvez trouver des motifs qui sont récurrents chez vous et n\u0026rsquo;apparaissent jamais chez moi. Pour faciliter la maintenance, ils peuvent être stockés dans un fichier à raison d\u0026rsquo;un par ligne, en respectant la construction suivante :\ntable \u0026lt;dyndns\u0026gt; file:/etc/mail/path-to-my-regex-file filter check_dyndns phase connect match rdns regex \u0026lt;dyndns\u0026gt; junk Continuons à disséquer la configuration.\nLa table aliases pointe vers un fichier d\u0026rsquo;alias sur deux colonnes. La colonne de gauche correspond à la partie utilisateur d\u0026rsquo;une adresse e-mail que vous recevez ; la colonne de droite correspond à l\u0026rsquo;utilisateur à qui le mail doit être livré (ex. : root: gilles). Il est recommandé d\u0026rsquo;aliaser les adresses root, abuse et postmaster vers une adresse que vous lisez réellement. Je ne le fais pas ici parce que je vais simplement détruire le setup une fois l\u0026rsquo;article fini, mais en ce qui vous concerne, faites le !\nEnsuite viennent les lignes listen. La première est le point d\u0026rsquo;attache que vont contacter les autres serveurs de mail qui souhaitent envoyer du mail vers hypno.cat. Elle est configurée pour offrir tls en utilisant les certificats et clés pour mail.hypno.cat et pour filtrer les sessions entrantes avec mes filtres. La seconde est le point d\u0026rsquo;attache que vont contacter mes propres utilisateurs sur le port de submission, qui requiert TLS en utilisant les mêmes certificats et clés, mais également l\u0026rsquo;authentification des utilisateurs système locaux (le défaut peut être changé), et filtrant à-travers le filtre rspamd uniquement dans le but de faire la signature DKIM de nos messages.\nLes lignes action définissent les actions à entreprendre avec les mails qui sont acceptés dans le système. Une action locale permet de distribuer dans une maildir, en classifiant les spams dans un répertoire spécifique, et en résolvant les alias. Une seconde action permets de relayer les mails en s\u0026rsquo;annonçant en tant que mail.hypno.cat aux autres hôtes. Si le serveur disposait de plusieurs adresses IP, il serait possible d\u0026rsquo;assigner à l\u0026rsquo;aide de l\u0026rsquo;option src celle à utiliser, mais ici je n\u0026rsquo;ai qu\u0026rsquo;une seule adresse IP donc pas besoin de la spécifier.\nEnfin, les lignes match représentent l\u0026rsquo;ensemble des règles. Quand une enveloppe entre dans le serveur SMTP au travers d\u0026rsquo;un des points d\u0026rsquo;attache listen, elle est comparée séquentiellement à chaque ligne match jusqu\u0026rsquo;à trouver une correspondance ou rejetée si aucune correspondance n\u0026rsquo;est trouvée. Lorsqu\u0026rsquo;une correspondance est trouvée, l\u0026rsquo;action associée à la règle est prise en compte. Les règles de match sont très simples à comprendre, je ne vais donc pas les expliquer ; vous devriez arriver à comprendre seuls ce que signifie from any for domain \u0026quot;hypno.cat\u0026quot;, sans quoi vous ne devriez pas lire cet article en premier lieu.\nQuelques mots au sujet de SenderScore # SenderScore est une base de donnée de réputation IP qui associe un score entre 0 et 100 à une adresse IP. Le score est lié au volume et comportement observé depuis ces adresses IP, et même si l\u0026rsquo;on ne sait pas COMMENT sont calculés ces scores parce que la méthodologie n\u0026rsquo;est pas publique, un peu d\u0026rsquo;étude permet de confirmer que ces nombres ne sortent pas de nul part : ils sont corrélés au ratio de distribution et d\u0026rsquo;erreur chez différents Big Mailer Corps et il est possible d\u0026rsquo;influencer le score en changeant de comportement.\nIl est également TRÈS clair que pour construire les scores de réputation, ils ont accès à des informations relatives aux sessions SMTP dont seuls les serveurs mails de destinations peuvent disposer, comme le volume de mails ou le ratio de destinataires acceptés et refusés, observés depuis une adresse, impliquant que ces informations sont obtenues directement depuis les Big Mailer Corps. Faites ce que vous voulez de cela.\nEn me basant sur MA compréhension, il s\u0026rsquo;agit d\u0026rsquo;un indicateur intéressant pour limiter la quantité de spam qui vous atteint. Je ne fait pas confiance à cet indicateur concernant les bons scores (parce que je sais à peu près contourner) mais j\u0026rsquo;y fais ÉNORMÉMENT confiance en ce qui concerne les mauvais scores, parce qu\u0026rsquo;il faut vraiment faire un paquet de mauvaises choses en gros volume pour que le score se dégrade. Si vous ne faites pas de mailing en masse, vous êtes censés être soit inconnu de cette base de donnée ou avoir un score supérieur à 95, sinon il y a quelque chose qui déconne chez vous.\nTout le monde n\u0026rsquo;est pas convaincu par SenderScore et certains experts en déliverabilité disent que c\u0026rsquo;est du pipeau. J\u0026rsquo;ai personnellement fait tourner une expérimentation sur plusieurs mois en graphant volume et réputation quotidienne, en comparant aux graphs de distribution. Mon opinion est contraire : il y a une correlation très significative qui permet clairement de classifier les émetteurs. Je pense que la meilleure approche est d\u0026rsquo;utiliser le filtre pour junker, et non bloquer, pour déterminer par vous même si vous en êtes satisfait.\nSenderScore considère que les émetteurs devraient avoir une réputation supérieure à 70. Je pense pour ma part que les hôtes avec un score inférieur à 70 sont de bons candidats au junk, les hôtes avec un score inférieur à 10 sont des rejets évidents.\nInstaller et configurer Dovecot # OpenBSD dispose d\u0026rsquo;un paquet pour Dovecot qui peut être installé d\u0026rsquo;une simple commande :\nmail# pkg_add dovecot dovecot-2.3.7.2v0: ok The following new rcscripts were installed: /etc/rc.d/dovecot See rcctl(8) for details. New and changed readme(s): /usr/local/share/doc/pkg-readmes/dovecot […] Et, specifiquement pour OpenBSD, en suivant les instructions du fichier readme, le fichier /etc/login.conf devrait contenir une classe dovecot pour augmenter les ressources autorisées à Dovecot, gourmand en descripteurs de fichiers :\ndovecot:\\ :openfiles-cur=1024:\\ :openfiles-max=2048:\\ :tc=daemon: Cela fait, il n\u0026rsquo;y a plus grand chose à faire à part pointer Dovecot sur le certificat et la clé TLS générés plus tôt. On le fait en modifiant les clés de configuration ssl_cert et ssl_key dans le fichier /etc/dovecot/conf.d/10-ssl.conf pour qu\u0026rsquo;elles soient comme suit :\nssl_cert = \u0026lt;/etc/ssl/mail.hypno.cat.fullchain.pem ssl_key = \u0026lt;/etc/ssl/private/mail.hypno.cat.key Puis dire à Dovecot que les mails sont livrés dans le répertoire ~/Maildir de chaque utilisateur puisque c\u0026rsquo;est là que OpenSMTPD les déposera. On le fait en modifiant la clé de configuration mail_location dans le fichier /etc/dovecot/conf.d/10-mail.conf pour qu\u0026rsquo;elle soit comme suit :\nmail_location = maildir:~/Maildir On est bon.\nLe daemon peut être activé pour être démarré à chaque reboot, puis démarré immédiatement :\nmail# rcctl enable dovecot mail# rcctl start dovecot dovecot(ok) À ce stade, il est possible de configurer n\u0026rsquo;importe quel client mail comme mutt, thunderbird ou même l\u0026rsquo;application gmail sur Android, et utiliser mail.hypno.cat pour les mails entrants et sortants.\nApprendre à Dovecot à entrainer Rspamd # Vous avez lu jusqu\u0026rsquo;ici ? bien.\nCette section est une section bonus, elle n\u0026rsquo;est absolument pas nécessaire pour configurer votre serveur de mail, mais c\u0026rsquo;est un élément que j\u0026rsquo;aime ajouter parce qu\u0026rsquo;il permet aux utilisateurs d\u0026rsquo;entrainer le filtre antispam d\u0026rsquo;une manière à laquelle ils sont habitués : en déplaçant un mail vers la boite à Spam, ou en appuyant sur un bouton \u0026ldquo;signaler un Spam\u0026rdquo;, \u0026ldquo;marquer comme Spam\u0026rdquo; ou peu importe comment il s\u0026rsquo;appelle dans l\u0026rsquo;interface utilisée. Le bon côté est que cet élément de configuration s\u0026rsquo;intègre avec IMAP, donc peu importe comment les mails sont consultés et depuis quel logiciel, l\u0026rsquo;action de marquer un mail comme Spam entraînera automatiquement le filtre.\nC\u0026rsquo;est très simple en théorie : il suffit de brancher deux scripts, l\u0026rsquo;un qui gère le déplacement dans la boite à Spam, l\u0026rsquo;autre qui gère le déplacement hors de la boite à Spam. Les Big Mailer Corps utilisent des stratégies plus poussées en détectant les mails non ouverts, ceux déplacés directement en poubelle, etc… mais débutons simple, il est toujours possible d\u0026rsquo;étendre ensuite.\nPour faire ce branchement, nous dépendont de Sieve qui est… comment dire avec diplomatie… un peu sur-conçu.\nCe n\u0026rsquo;est pas un élément critique du setup, vous êtes autorisés à éteindre vos cerveaux et suivre les instructions aveuglément. Le pire qui puisse arriver ici est que l\u0026rsquo;entraînement de la détection ne marche pas.\nPour activer imap sieve, le package Pigeonhole doit être installé, encore une fois à l\u0026rsquo;aide d\u0026rsquo;une simple commande :\nmail# pkg_add dovecot-pigeonhole dovecot-pigeonhole-0.5.7.2v1: ok […] Et… voici la partie la plus complexe, il faut configurer Dovecot pour activer imap_sieve et lui apprendre quels scripts utiliser pour quelles actions.\nD\u0026rsquo;abord, on active imap_sieve dans Dovecot en l\u0026rsquo;ajoutant dans la liste des plugins mails pour IMAP. C\u0026rsquo;est fait en mofifiant la clé de configuration mail_plugins dans le fichier /etc/dovecot/conf.d/20-imap.conf :\nprotocol imap { […] mail_plugins = $mail_plugins imap_sieve […] } Ensuite pour apprendre à Dovecot à entrainer Rspamd, on ajoute la section suivante dans le fichier /etc/dovecot/conf.d/90-plugin.conf, pour dire quel script sieve déclencher quand un mail est déplacé vers ou hors du répertoire Junk :\nplugin { sieve_plugins = sieve_imapsieve sieve_extprograms sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment imapsieve_mailbox1_name = Junk imapsieve_mailbox1_causes = COPY APPEND imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve imapsieve_mailbox2_name = * imapsieve_mailbox2_from = Junk imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve imapsieve_mailbox3_name = Inbox imapsieve_mailbox3_causes = APPEND imapsieve_mailbox3_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve } Maintenant que Dovecot est prêt, on prépare la partie Sieve qui dépend de deux scripts pour entraîner le Spam et le Ham dans /usr/local/lib/dovecot/sieve :\n# cat report-ham.sieve require [\u0026#34;vnd.dovecot.pipe\u0026#34;, \u0026#34;copy\u0026#34;, \u0026#34;imapsieve\u0026#34;, \u0026#34;environment\u0026#34;, \u0026#34;variables\u0026#34;]; if environment :matches \u0026#34;imap.mailbox\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;mailbox\u0026#34; \u0026#34;${1}\u0026#34;; } if string \u0026#34;${mailbox}\u0026#34; \u0026#34;Trash\u0026#34; { stop; } if environment :matches \u0026#34;imap.user\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;username\u0026#34; \u0026#34;${1}\u0026#34;; } pipe :copy \u0026#34;sa-learn-ham.sh\u0026#34; [ \u0026#34;${username}\u0026#34; ]; # cat report-spam.sieve require [\u0026#34;vnd.dovecot.pipe\u0026#34;, \u0026#34;copy\u0026#34;, \u0026#34;imapsieve\u0026#34;, \u0026#34;environment\u0026#34;, \u0026#34;variables\u0026#34;]; if environment :matches \u0026#34;imap.user\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;username\u0026#34; \u0026#34;${1}\u0026#34;; } pipe :copy \u0026#34;sa-learn-spam.sh\u0026#34; [ \u0026#34;${username}\u0026#34; ]; Et parce que les deux scripts Sieve dépendent de scripts shell sa-learn-ham.sh et sa-learn-spam.sh, on crée aussi ces deux scripts shell dans /usr/local/lib/dovecot/sieve :\n# cat sa-learn-ham.sh #!/bin/sh exec /usr/local/bin/rspamc -d \u0026#34;${1}\u0026#34; learn_ham # cat sa-learn-spam.sh #!/bin/sh exec /usr/local/bin/rspamc -d \u0026#34;${1}\u0026#34; learn_spam Enfin, on compile les scripts Sieve et on rend les scripts shell exécutables pour que Dovecot puisse les utiliser :\n# sievec report-ham.sieve # sievec report-spam.sieve # chmod 755 sa-learn-ham.sh # chmod 755 sa-learn-spam.sh C\u0026rsquo;est tout. Dorénavant en déplaçant un mail d\u0026rsquo;un répertoire à un autre, on doit voir les lignes suivantes dans le fichier de log /var/log/rspamd/rspamd.log :\n2019-09-13 23:59:46 #18598(controller) \u0026lt;1d44bd\u0026gt;; csession; rspamd_controller_learn_fin_task: \u0026lt;127.0.0.1\u0026gt; learned message as ham: CAHPtQbOxQxBCsVd7nUCP4podu74Pa-F6k28z+4BWfNeeqWYiAg@mail.gmail.com […] 2019-09-14 00:01:57 #18598(controller) \u0026lt;b76e28\u0026gt;; csession; rspamd_controller_learn_fin_task: \u0026lt;127.0.0.1\u0026gt; learned message as spam: CAHPtQbOxQxBCsVd7nUCP4podu74Pa-F6k28z+4BWfNeeqWYiAg@mail.gmail.com […] Il est surement possible de simplifier, je vais être honnête et dire que mon intérêt pour Sieve n\u0026rsquo;est pas assez poussé pour que je me perfectionne. C\u0026rsquo;est une fonctionnalité sur le côté et je ne toucherai probablement pas les scripts avant les dix prochaines années. Je ne suis pas du tout convaincu par Sieve et quand j\u0026rsquo;aurai le temps, je ferais une alternative qui ne me rappelle pas les années m4 de Sendmail.\nTester le tout # Et maintenant, on va juste tester que l\u0026rsquo;on arrive à faire un trajet complet aller-retour de mail depuis hypno.cat vers gmail.com.\nTout d\u0026rsquo;abord, je prépare un mail depuis mon compte hypno.cat vers mon compte gmail.com :\nAprès envoi, je vérifie qu\u0026rsquo;il arrive bien chez gmail.com :\nEnsuite, je vérifie que l\u0026rsquo;émission a bien eu lieu par dessus un canal sécurisé par TLS :\nJe vérifie également que gmail.com est content avec notre déclaration SPF, notre signature DKIM et qu\u0026rsquo;il a bien vu notre enregistrement DMARC. On fait cela en sélectionnant \u0026ldquo;Voir l\u0026rsquo;original\u0026rdquo; dans le menu associé à chaque mail :\nEnfin, j\u0026rsquo;y répond pour être sur que ça fonctionne dans les deux sens :\nCool, on a fini # Cet article est dense, je voulais expliquer pourquoi on fait les choses. Si vous retirez toutes les explications et gardez juste les détails purement techniques, vous réaliserez que nous avons construit une infrastructure de mail, fournissant des services entrants et sortants sécurisés par TLS, avec un trafic sortant respectant DKIM, SPF et DMARC, et un trafic entrant filtrant le spam.\nOn a fait cela en :\najoutant deux enregistrements A, un MX et trois TXT dans notre zone DNS; faisant attention à avoir un rDNS correctement configuré installant un certificat TLS préparant un fichier de configuration d\u0026rsquo;environ 15 lignes pour un serveur SMTP modifiant 3 lignes dans la configuration par défaut d\u0026rsquo;un serveur IMAP modifiant 8 lignes dans la configuration par défaut d\u0026rsquo;une solution antispam Et parce que nous étions de tres bonne humeur et volontaires pour faire un petit pas de plus, nous avons implémenté un apprentissage Spam / Ham de Rspamd en :\nmodifiant environ 20 lignes dans la configuration par défaut d\u0026rsquo;un serveur IMAP copiant 4 scripts Sieve dans un répertoire On admettra qu\u0026rsquo;un nouvel arrivant va devoir fournir quelques efforts pour trouver tout ça sans aide, mais aucune de ces tâches n\u0026rsquo;est réellement difficile ou complexe. De plus, la plupart sont des opérations de mises en place qui ne sont réalisées qu\u0026rsquo;une fois ; on n\u0026rsquo;édite pas la zone DNS ou un reverse DNS tous les deux jours, tout comme on ne touche pas une configuration en place qui fonctionne tant qu\u0026rsquo;il n\u0026rsquo;y a pas de nouveau cas d\u0026rsquo;utilisation.\nPour citer les plus réfractaires à l\u0026rsquo;auto-hébergement, il peut y avoir des maintenances pour gérer les conséquences d\u0026rsquo;une blacklist, mais il n\u0026rsquo;y a aucun scénario dans lequel ladite maintenance impliquerait autant de travail que la mise en place que nous venons de faire… et qui au final se réplique en 10 minutes chrono une fois réalisée une première fois pour se faire la main.\nQuelle est la prochaine étape ? # Votre prochaine étape est de mettre en place de la redondance pour vous assurer qu\u0026rsquo;une panne de votre serveur de mail n\u0026rsquo;entraine pas de perte de mail.\nEn pratique, la plupart des serveurs de mails retentent une émission de multiples fois si la destination est inaccessible donc, même en cas de panne, la plupart de vos mails seront temporisés par votre émetteur et retransmis quand votre serveur sera de nouveau accessible.\nEn théorie, on veut tout de même faire les choses bien, et reposer sur la responsabilité des autres à gérer nos pannes n\u0026rsquo;est pas très poli. Vous devez mettre en place un serveur secondaire, vous arranger entre amis pour être serveurs secondaires les uns des autres, ou même vous souscrire à un service qui offre du mail secondaire. N\u0026rsquo;importe quoi qui puisse permettre de couvrir vos arrières lorsque votre serveur primaire tombe.\nEnsuite, peu importe la méthode choisie, il vous faudra :\najouter un enregistrement MX supplémentaire dans votre zone DNS configurer le serveur secondaire pour retransmettre les mails au serveur primaire On est loin de la haute voltige.\nEst-ce qu\u0026rsquo;on a vraiment fini ? # Je vais probablement écrire une série d\u0026rsquo;article pour discuter de la réputation, ainsi que des mécanismes en place chez les Big Mailer Corps qui provoquent des pannes de distribution.\nSi ce sont des sujets d\u0026rsquo;intérêt et que j\u0026rsquo;ai du feedback positif, j\u0026rsquo;écrirais plus souvent sur les concepts autour de la déliverabilité, sinon je reprendrais mes anciennes activités : écrire mes rapports mensuels sur mes contributions opensource. À vous de me dire :-)\nFinalement, dans cet article j\u0026rsquo;ai décrit un setup vraiment simple mais il y a des tonnes de choses intéressantes à faire avec les infrastructures de mail, si vous n\u0026rsquo;avez pas peur d\u0026rsquo;aller plus loin.\nJe construis actuellement une infrastructure de mail pour un future service commercial d\u0026rsquo;hébergement. L\u0026rsquo;infrastructure s\u0026rsquo;étend sur plusieurs data centers dans plusieurs pays, a des serveurs secondaires capable d\u0026rsquo;encaisser des pannes très sérieuses des serveurs primaires, peut facilement monter à l\u0026rsquo;échelle sur des pics de volume, elle décorelle le trafic entrant et sortant pour permettre de fournir des services partiels, peut facilement re-router le trafic entre des machines pour contourner des pannes, tout en fournissant des services IMAP à des comptes virtuels sur une multitude de domaines, avec des backups quotidiens. L\u0026rsquo;infrastructure me coute… moins de 75 EUR par mois. Elle est également très simple, la plupart des machines sont des installations par défaut d\u0026rsquo;OpenBSD avec très peu de changements au système de base.\nSi lire à propos de ce genre de mises en place vous intéresse, je peux écrire à ce sujet pour aider un maximum de personnes à monter un maximum d\u0026rsquo;alternatives aux Big Mailer Corps, ce qui est au final mon souhait.\nMerci de m\u0026rsquo;avoir lu, je n\u0026rsquo;ai pas de soundcloud, mais j\u0026rsquo;ai un patreon et un github si vous voulez me sponsoriser, ou utiliser les icones en dessous du lien de commentaires pour partager cet article sur les reseaux sociaux.\n","date":"23 December 2019","permalink":"/posts/2019-12-23/mettre-en-place-un-serveur-de-mail-avec-opensmtpd-dovecot-et-rspamd/","section":"Posts","summary":"TL;DR: - Pas de résumé, j'ai passé des heures à traduire, vous allez passer des minutes à lire ;) - OK… J'ai expliqué avec BIEN TROP DE DÉTAILS comment mettre en place un serveur de mail Merci à mes sponsors !","title":"Mettre en place un serveur de mail avec OpenSMTPD, Dovecot et Rspamd"},{"content":" TL;DR: - SMTP est la méthode dont les ordinateurs échangent des e-mails - il s'agit d'un protocole décentralisé, ce qui signifie que CHACUN peut héberger un nœud et être indépendant - il est en train d'être centralisé dans des sociétés qui ont un passif d'abus - il est en train d'être centralisé dans un pays qui a un passif d'abus Où est-ce que j\u0026rsquo;ai déjà lu ça ? # En Août, j\u0026rsquo;ai publié un petit article intitulé \u0026ldquo; You should not run your mail server because mail is hard\u0026rdquo; (\u0026ldquo;Vous ne devriez pas héberger votre serveur de mail parce que c\u0026rsquo;est dur\u0026rdquo;) qui était, en gros, mon opinion sur les différentes raisons qui poussent les gens à décourager l\u0026rsquo;hébergement de mails. De manière très inattendue, l\u0026rsquo;article est devenu assez populaire, atteignant 100K lectures et continuant à recevoir des visites et des commentaires plusieurs mois après publication.\nEn Septembre, j\u0026rsquo;ai publié un autre article beaucoup plus long intitulé \u0026ldquo; Setting up a mail server with OpenSMTPD, Dovecot and Rspamd\u0026rdquo; (\u0026ldquo;Installer un serveur de mail avec OpenSMTPD, Dovecot et Rspamd\u0026rdquo;) qui décrivait pas à pas, de façon très détaillée, les étapes pour installer un serveur de mail complet jusqu\u0026rsquo;à la livraison des e-mails en boîte de réception chez un gros hébergeur de mail américain. L\u0026rsquo;article est devenu lui aussi plutôt populaire, sans pour autant atteindre le niveau du précedent article moins technique et spécifique, mais atteignant 40K lectures et continuant également à recevoir des visites et des commentaires plusieurs mois après publication.\nLe contenu que vous vous apprêtez à lire est extrait de ce second article auquel il n\u0026rsquo;aurait pas dû être rattaché. Il est bien trop (géo-)politique pour faire partie d\u0026rsquo;un article technique, j\u0026rsquo;ai donc décidé de l\u0026rsquo;en retirer et d\u0026rsquo;en faire un dédié. Je ne veux pas que les spécificités d\u0026rsquo;un article sur OpenSMTPD aille à l\u0026rsquo;encontre d\u0026rsquo;un message beaucoup plus général.\nC\u0026rsquo;est mon premier article en français depuis des années, je vous demande donc un peu d\u0026rsquo;indulgence : si vous trouvez des fautes, vous pouvez me les remonter pour que je les corrige, ou faire une pull request pour les techies.\nN\u0026rsquo;hésitez pas à partager la publication à l\u0026rsquo;aide des icones en fin d\u0026rsquo;article et à la commenter \u0026lt;3\nQui sont Big Mailer Corps ? # Je ne vais pas pointer de doigts vers des directions en particulier. Concrètement, rentrent dans ma définition des Big Mailer Corps, toute grosse entreprise qui centralise l\u0026rsquo;hébergement des e-mails d\u0026rsquo;un très grand nombre d\u0026rsquo;utilisateurs :\n- Orange ? SFR ? La Poste ? - Non, plus grand, plus mondial, … Et là, normalement le doute est dissipé, pas besoin de citer de nom. Vous avez compris qui est visé parce que TOUT LE MONDE connait les Big Mailer Corps, la plupart d\u0026rsquo;entre vous avez une adresse chez l\u0026rsquo;un d\u0026rsquo;entre eux, et la plupart de vos contacts sont hébergés chez l\u0026rsquo;un d\u0026rsquo;entre eux.\nL\u0026rsquo;auto-hébergement et encourager les petits fournisseurs est meilleur pour notre bien commun # Il y a des conséquences à la centralisation des services de mail chez les Big Mailer Corps.\nIl n\u0026rsquo;y a aucun sens à ce que Jacques Lambda, qui partage des photos de chatons avec sa famille et ses amis, se construise une infrastructure d\u0026rsquo;hébergement de mails alors que plusieurs Big Mailer Corps lui offrent, \u0026ldquo;gratuitement\u0026rdquo;, une qualité de service au top. Il peut obtenir une adresse e-mail immédiatement disponible et qui marche de manière parfaitement fiable. Il n\u0026rsquo;y a absolument aucun sens à ce que Jacques Lambda n\u0026rsquo;aille pas chez l\u0026rsquo;un des Big Mailer Corps, particulièrement quand même les techies font ce choix sans hésitation, confirmant que c\u0026rsquo;est plutôt une bonne option.\nIl n\u0026rsquo;y a rien de mal à ce que tous les Jacques Lambda choisissent un service qui marche.\nCe qui est terriblement mauvais par contre, c\u0026rsquo;est le centralisation d\u0026rsquo;un protocole de communication dans les mains d\u0026rsquo;une poignée d\u0026rsquo;entreprises commerciales, CHACUNE D\u0026rsquo;ENTRE ELLES provenant du même pays (actuellement dirigé par un lunatique qui abuse de son pouvoir et souffre probablement de trouble de la personnalité narcissique), CHACUNE D\u0026rsquo;ENTRE ELLES étant apparu dans les informations et/ou dans une cour de tribunal pour tout un assortiment de comportements \u0026ldquo;déplaisants\u0026rdquo; (abus de la vie privée, espionnage, abus de monopole, harcellement professionnel ou sexuel, j\u0026rsquo;en passe… ), et CHACUNE D\u0026rsquo;ENTRE ELLES faisant croître des bases d\u0026rsquo;utilisateurs qui dépassent de loin la population totale de plusieurs pays combinés.\nMettons un peu de perspective. Le plus gros des Big Mailer Corps annonce une base d\u0026rsquo;utilisateur qui dépasse 1.4 MILLARD d\u0026rsquo;utilisateurs (Avril 2018), soit à peu près la population totale de la Chine ou de l\u0026rsquo;Inde, plus de quatre fois la population totale des USA, ou plus de vingt fois la population totale de la France. Si vous comptiez les utilisateurs au rythme d\u0026rsquo;un par seconde, il vous faudrait 44 ans pour en faire le tour.\nEnsuite, très loin derrière, il est suivi par le prochain Big Mailer Corp qui annonce une base de 400 millions d\u0026rsquo;utilisateurs (2018 également), dépassant la population totale des USA, atteignant quatre fois la population totale de l\u0026rsquo;Égypte, et cinq fois la population totale de la France.\nDans la plupart des rapports de mailing que j\u0026rsquo;ai pu surveiller, le premier des Big Mailer Corps couvre approximativement la moitié des adresse e-mails de destinataires… l\u0026rsquo;autre moitié étant également couverte en large partie par les autres Big Mailer Corps.\nCe n\u0026rsquo;est pas sain. Ni ici, ni nulle part, ni maintenant, ni jamais, ni dans aucune dimension alternative (insérer \u0026ldquo;sliiiiiiders\u0026rdquo; en chuchotement mental ici).\nPrenez un moment pour bien enregistrer ce qui suit :\nSi ces grosses entreprises étaient contraintes de couper les communications avec un pays pour appliquer des sanctions, bien au delà d\u0026rsquo;un MILLIARD de personnes seraient hors de portée pour le pays visé. Et si vous pensez la crainte exagérée, les utilisateurs de Github en Iran, en Syrie, à Cuba ou en Crimée pourraient vous donner un point de vue différent après avoir découvert qu\u0026rsquo;ils étaient mis à la porte pour appliquer les sanctions américaines sur leurs pays.\nAussi ennuyant soit-il, Git reste un outil de techies qui n\u0026rsquo;impacte principalement que des techies, une minorité d\u0026rsquo;humains qui sait trouver facilement des solutions de contournement… facilitées par le fait que Git est décentralisé et ne nécessite pas d\u0026rsquo;interconnexions et d\u0026rsquo;interopérabilité : les utilisateurs peuvent facilement bouger ailleurs et s\u0026rsquo;auto-héberger. Aucun lien coupé avec aucun pays ne peut empêcher ça.\nLe mail fonctionne de façon différente. Il impacte tout le monde, des personnes de tous âges et qui ne sont pas forcément des techies. Il est toujours possible de trouver une solution de contournement en cas de blocage, mais ce n\u0026rsquo;est pas simple pour tout le monde. Et comme il s\u0026rsquo;agit d\u0026rsquo;un protocole pour mettre en relation un émetteur A avec un destinataire B, il y a un besoin d\u0026rsquo;interconnexion, un besoin d\u0026rsquo;interopérabilité ET besoin que le lien entre A et B ne soit pas coupé. Les conséquences d\u0026rsquo;un blocage similaire à celui de Github chez les Big Mailer Corps seraient plusieurs ordres de magnitude plus durs et visibles : un pan des utilisateurs d\u0026rsquo;Internet ne serait tout bonnement plus joignable par les citoyens des pays sanctionnés.\nLes habitants des pays de merde ne sont pas tous des techies, ils sont des personnes qui ont BESOIN d\u0026rsquo;interconnexions et d\u0026rsquo;interopérabilité pour communiquer avec plus d\u0026rsquo;un MILLIARD de personnes hébergées chez les Big Mailer Corps. Le mail permet de connecter les populations mondiales de manière fiable… UNIQUEMENT si le mail reste décentralisé. Il ne marche plus s\u0026rsquo;il est centralisé dans une poignée d\u0026rsquo;entreprises toutes localisées dans le même pays.\nBien sûr cela n\u0026rsquo;arrivera peut être jamais, et je metterai même mon billet dessus parce qu' un pays connu pour avoir intercepté et épié les communications mondiales a plus d\u0026rsquo;intérêt à conserver le flot des communications qu\u0026rsquo;à le couper, mais le seul fait que ce soit TECHNIQUEMENT possible devrait être suffisant pour nous mettre mal à l\u0026rsquo;aise. J\u0026rsquo;ignore même volontairement que la raison pour laquelle ils ne le feront pas me rends également mal à l\u0026rsquo;aise. Si les Big Mailer Corps étaient hébergés en Chine, nous serions déjà en train d\u0026rsquo;envisager les conséquences d\u0026rsquo;une coupure et d\u0026rsquo;envisager un plan de sortie, mais nous ne le faisons pas parce que nous pensons que nous sommes du bon côté de la barrière, et que des sanctions à notre encontre ne pourraient pas arriver.\nMais ça, c\u0026rsquo;est jusqu\u0026rsquo;à ce que l\u0026rsquo;on soit complètement au pied du mur sans solution.\nÀ ce stade, je pourrais également faire semblant d\u0026rsquo;ignorer que le modèle économique de la plupart de certaines de ces entreprises repose sur la création d\u0026rsquo;un profil commercial de leurs utilisateurs et qu\u0026rsquo;avoir à disposition tous les e-mails est une mine d\u0026rsquo;or.\nCertains Big Mailer Corps se défendent d\u0026rsquo;exploiter le contenu des e-mails, mais… il n\u0026rsquo;y a pas besoin de connaître le contenu des e-mails si les expéditeurs et destinataires sont écrits sur l\u0026rsquo;enveloppe. Il est tout à fait possible de savoir que vous essayer de souscrire un crédit en faisant des comparatifs, mais que vous recevez également des remboursements de santé réguliers et que vous recevez des informations concernant un certain type d\u0026rsquo;appareillage médical. Un profil très précis et qui serait très intéressant pour des organismes de prêt ou des assurances peut tout à fait être dressé sans avoir à \u0026ldquo;lire\u0026rdquo; les e-mails… mais bon, ce ne sont que des spéculations basées sur quelque chose de techniquement réalisable.\nDans la même idée, je vais faire semblant d\u0026rsquo;ignorer que tous les documents partagés peuvent être analysés pour détecter des logiciels malveillants, que le document soit confidentiel ou non, qu\u0026rsquo;il soit stratégique ou non. Tout comme les liens échangés par mail, qu\u0026rsquo;ils soient stratégiques ou non, perdent tout caractère confidentiel lorsqu\u0026rsquo;ils passent par le webmail d\u0026rsquo;un Big Mailer Corp.\nAlors OUI, il faut plus de personnes qui s\u0026rsquo;auto-hébergent ou qui dépendent d\u0026rsquo;hébergeurs géolocalisés, tissant un réseau de communication le plus vaste au-travers de plusieurs opérateurs dans plusieurs pays. Je ne pense évidemment pas que tout le monde doit s\u0026rsquo;auto-héberger, mais je pense que DAVANTAGE de monde doit sortir des Big Mailer Corps pour aller n\u0026rsquo;importe où ailleurs… il faut réintroduire de la contrainte chez les les Big Mailer Corps, qu\u0026rsquo;ils soient obligés d\u0026rsquo;interagir avec des opérateurs en dehors de leur gang, et qu\u0026rsquo;il y ait un risque financier SI soudainement ils prenaient des décisions à l\u0026rsquo;encontre des populations.\nLes Big Mailer Corps ne sont pas mauvais ou méchants, j\u0026rsquo;utilise moi-même leurs services régulièrement, et la plupart de leurs employés cherchent probablement le bien commun… mais ces entreprises sont devenues trop grosses et puissantes, il faut qu\u0026rsquo;il y ait un équilibre parce que l\u0026rsquo;on ne sait pas comment elles évolueront dans les années à venir, on ne sait pas non plus comment évoluera la politique de leurs pays d\u0026rsquo;origine dans les années à venir, et les informations récentes ne décrivent pas un avenir tout rose dans la bonne direction.\nJe concluerai en vous recommandant de regarder cette excellente présentation de Bert Hubert ( @PowerDNS_Bert) de PowerDNS, à propos de problèmes similaires qui sont en train d\u0026rsquo;émerger avec le protocole DNS et les craintes qui en découlent autour de la surveillance. De très nombreux points de sa présentation sont tout aussi valides en ce qui concerne les services de mail.\n","date":"15 December 2019","permalink":"/posts/2019-12-15/decentralisons-smtp-pour-le-bien-commun/","section":"Posts","summary":"TL;DR: - SMTP est la méthode dont les ordinateurs échangent des e-mails - il s'agit d'un protocole décentralisé, ce qui signifie que CHACUN peut héberger un nœud et être indépendant - il est en train d'être centralisé dans des sociétés qui ont un passif d'abus - il est en train d'être centralisé dans un pays qui a un passif d'abus Où est-ce que j\u0026rsquo;ai déjà lu ça ?","title":"Décentralisons SMTP pour le bien commun"},{"content":" TL;DR: - SMTP is the way computers exchange e-mails - it is a decentralised protocol meaning that ANYONE can run a node and be independant - it is being centralised at companies that have a history of abuse - it is being centralised in a country that has a history of abuse Where did you read this already ? # In August, I published a small article titled \u0026ldquo; You should not run your mail server because mail is hard\u0026rdquo; which was basically my opinion on why people keep saying it is hard to run a mail server. Unexpectedly, the article became very popular, reached 100K reads and still gets hits and comments several months after publishing.\nAs a follow up to that article, I published in September a much lenghtier article titled \u0026ldquo; Setting up a mail server with OpenSMTPD, Dovecot and Rspamd\u0026rdquo; which described how you could setup a complete mail server. I went from scratch and up to inboxing at various Big Mailer Corps using an unused domain of mine with a neutral reputation and describing precisely for each step what was done and why it was done. The article became fairly popular, nowhere near the first one which wasn\u0026rsquo;t so technical, but reached 40K reads and also still gets hits and comments several months after publishing.\nThe content you\u0026rsquo;re about to read was part of the second article but it didn\u0026rsquo;t belong there, it was too (geo-)political to be part of a technical article, so I decided to remove it from there and make it a dedicated one. I don\u0026rsquo;t want the tech stack to go in the way of the message, this is not about OpenSMTPD.\nSelf-hosting and encouraging smaller providers is for the greater good # There are political consequences to centralizing mail services at Big Mailer Corps.\nIt doesn\u0026rsquo;t make sense for Random Joe, sharing kitten pictures with his family and friends, to build a personal mail infrastructure when multiple Big Mailer Corps offer \u0026ldquo;for free\u0026rdquo; an amazing quality of service. They provide him with an e-mail address that is immediately available and which will generally work reliably. It really doesn\u0026rsquo;t make sense for Random Joe not to go there, and particularly if even techies go there without hesitation, proving it is a sound choice.\nThere is nothing wrong with Random Joes using a service that works.\nWhat is terribly wrong though is the centralization of a communication protocol in the hands of a few commercial companies, EVERY SINGLE ONE OF THEM coming from the same country (currently led by a lunatic who abuses power and probably suffers from NPD), EVERY SINGLE ONE OF THEM having been in the news and/or in a court for random/assorted \u0026ldquo;unpleasant\u0026rdquo; behaviors (privacy abuses, eavesdropping, monopoly abuse, sexual or professional harassment, you just name it\u0026hellip;), and EVERY SINGLE ONE OF THEM growing user bases that far exceeds the total population of multiple countries combined.\nLet\u0026rsquo;s put a bit of perspective. The biggest one of them reports a user base that exceeds 1.4 BILLION users (April 2018), roughly the entire population of either one of China or India, exceeding by four the population of United States by itself, or by over twenty the population of my own country. If you counted its users at the rate of one each second, it would take you over 44 years to go through.\nThen, very far below, it is followed by the next one which reported 400 million active users (2018 too), also exceeds the population of the United States, being four times the population of Egypt, and five times the population of my own country.\nIn many of the mailings I have monitored, the very first one covered approximately half the recipients e-mail address space\u0026hellip; before the other half was even split in large parts between the other Big Mailer Corps.\nThis is NOT sane, not here, not anywhere, not in any alternate dimension (insert \u0026ldquo;sliiiiiiders\u0026rdquo; whisper here).\nTake a moment to let this sink in:\nIf these companies were somehow required to cut communications with a country for sanctions, well over a BILLION people could be out of reach for the targeted country. If you think that this is far-fetched, the users from Iran, Syria, Cuba or Crimea could surely provide you with an alternate point of view after discovering they were kicked out of Github to enforce sanctions on their countries. But as annoying as it is, git is still a techie thing which mostly impacts techie people, a minority of human beings\u0026hellip; and it is also decentralized. Even if Github services are stopped, the users can move to another platform or self-host easily, there\u0026rsquo;s no need for interoperability with the punishing country if it wants to cut ties.\nE-mails are much different: they affect everyone, from all age and all backgrounds, the consequences of a similar block if it was done by Big Mailer Corps are orders of magnitude harsher. The people of shithole countries are not only techies that can work around, they are people who NEED interoperability to communicate with the billion and more of people hosted at Big Mailer Corps. Mail is good for connecting people worldwide in a reliable way, but this only works if you keep it decentralized, NOT when you centralize it in the hands of a few companies all geolocalized in the same country.\nNow that might never happen, and I would bet heavy money it won\u0026rsquo;t because a country known for intercepting communications and spying on the whole world has far more interest in keeping communications flowing, but the sole fact that it is TECHNICALLY possible should be enough to make us all uneasy. I\u0026rsquo;m even voluntarily looking away from the reason why I think they won\u0026rsquo;t do it as if it didn\u0026rsquo;t matter. If Big Mailer Corps were hosted in China we\u0026rsquo;d be dead worried and already considering a plan out, but the only reason we\u0026rsquo;re not is because we think we\u0026rsquo;re currently on the \u0026ldquo;right side\u0026rdquo; of the fence.\nBut that\u0026rsquo;s until we\u0026rsquo;re screwed.\nSo YES, I\u0026rsquo;d rather see a lot more people self-host or rely on smaller geolocalized providers, spreading the SMTP network as much as possible across multiple operators in multiple countries. I don\u0026rsquo;t think everyone should self-host, I just think MORE people should self-host or move away from Big Mailer Corps into ANY other provider\u0026hellip; so there is at least a bit of constraint for them to interoperate with operators outside their gang, as well as a financial risk for them IF they suddenly went rogue.\nI\u0026rsquo;ll say it again:\nI don\u0026rsquo;t think that either one of the Big Mailer Corps are evil or bad, I use some of their services on a daily basis, and most of the people operating them are genuinely seeking the greater good\u0026hellip; however they have grown too big and there needs to be a balance in power because who knows how they\u0026rsquo;ll evolve in the next ten years, who knows how the politics of their home country will evolve in the next ten years, and recent news doesn\u0026rsquo;t paint them as heading in the right direction.\nI\u0026rsquo;ll conclude by recommending that you see this excellent presentation by Bert Hubert ( @PowerDNS_Bert) from PowerDNS, about how a similar problem is starting to happen with DNS and the privacy and tracking concerns that arise from this. Many, many, many key points are also valid for mail services.\n","date":"15 December 2019","permalink":"/posts/2019-12-15/decentralised-smtp-is-for-the-greater-good/","section":"Posts","summary":"TL;DR: - SMTP is the way computers exchange e-mails - it is a decentralised protocol meaning that ANYONE can run a node and be independant - it is being centralised at companies that have a history of abuse - it is being centralised in a country that has a history of abuse Where did you read this already ?","title":"Decentralised SMTP is for the greater good"},{"content":" TL;DR: - greylisting is a sound idea - yet it is not really practical today - people tend to disable it or find work-arounds - SPF-aware greylisting makes greylisting usable again SMTP failures in a nutshell # SMTP is a fail-safe protocol which attempts very hard to ensure that messages do not get lost once they are in transit. Among the various mechanisms and requirements in place is the use of \u0026ldquo;Temporary Failures\u0026rdquo;.\nBasically, an SMTP node has two final states for a mail. Either it is delivered, and the recipient has the mail, or it\u0026rsquo;s not, and the recipient doesn\u0026rsquo;t have the mail. For the latter, the SMTP requires that a node notifies the sender that an error occured either by rejecting during the SMTP transaction or by sending a deferred bounce (also known as MAILER-DAEMON).\nThen, there\u0026rsquo;s a third state that\u0026rsquo;s not final: temporarily failed. It covers any transient error that prevented delivery\u0026hellip; but which might result in a delivery (or a permanent failure) if the mail was retried after something was fixed by the destination node.\nUnlike delivered mail, which the recipient sees, and failed mail, which the sender sees, the temporarily failed mails are usually not seen by users. They are handled between the mail exchangers (MX) retrying behind the scene, and only ever seen by users when the situation is either resolved or the mail time-to-live has expired. In the first case, the recipient will get the mail delivered a bit later than expected, often without even knowing that a retry took place. In the second case, the sender will get a notification that, despite retries, the mail could not be delivered after an extended period of time.\nThe retry strategies are handled differently by different software, but you can expect that you\u0026rsquo;ll see retries happening seconds or minutes after a temporary failure. A popular approach, the quadratic delay increase, causes the delay between retries to start short, but then grow with the number of retries so that the longer a destination host is unable to handle a mail, the longer between the retries from a sender.\nTemporary failures happen all the time, it is a normal thing in the SMTP world, looking at log you will very often see lines such as this:\n4b3a6c195c1f6010 mta delivery evpid=8f26ea98359eccea from=\u0026lt;misc+bounces-4541-[redacted]=tin.it@opensmtpd.org\u0026gt; to=\u0026lt;[redacted]@tin.it\u0026gt; rcpt=\u0026lt;-\u0026gt; source=\u0026quot;45.76.46.201\u0026quot; relay=\u0026quot;62.211.72.32 (smtp.tin.it)\u0026quot; delay=0s result=\u0026quot;TempFail\u0026quot; stat=\u0026quot;421 \u0026lt;[redacted]@tin.it\u0026gt; Service not available - too busy\u0026quot; This is not just for small destinations, I know of two Big Mailer Corps that have temporary issues pretty much always, but they also have so many mail exchangers that a retry is unlikely to hit the same MX twice in a row, so mail usually gets delivered at first or second try.\nA lot more could be said but lets keep this article simple and summarize:\nThere exists a mechanism in SMTP which allows a destination MX to tell a sender MX that it should retry a delivery. The retries happen behind the scene, without any action from the users, and can cause a delay in delivery.\nGreylisting explained # Spam is a volume industry.\nSpammers are not interested in targeting specific recipients, they are interested in targeting a large amount of recipients so that it statistically increases the number of recipients that will fall for it.\nI won\u0026rsquo;t dig into this too much because it is the topic of another article I\u0026rsquo;m writing, however you have to keep in mind that there are different kinds of spammers, and that while some use regular MX that respect the retry requests, a lot more rely on stateless scripts or custom MX code that DOES NOT honor the retries because\u0026hellip; it slows them down considerably for uncertain result. Waiting minutes to retry a recipient just to end up resolving to an \u0026ldquo;unknown user\u0026rdquo; permanent failure is not interesting, it clutters the queue and it prevents sending other domains at full speed.\nThe idea behind greylisting is very simple:\nGreylisting causes a temporary failure to be triggered for MX you don\u0026rsquo;t trust and keeps track of these retry requests. If the retry happens too soon, it is ignored, so spammers can\u0026rsquo;t just perform immediate retries. If the retry happens too late, well\u0026hellip; it is too late and the implementation may do different things ranging from blacklisting, to degrading reputation, to forgetting the initial request and causing greylisting to start all over again, and again, and again, \u0026hellip; The bottom line is that spammers are forced to keep state and behave like an RFC-compliant implementation, which goes against the economics of sending en masse.\nIn the other hand, RFC-compliant MX naturally try again multiple times so they eventually fall in the correct window for a retry and are whitelisted.\nHow greylisting usually works # There are different greylisting solutions and they don\u0026rsquo;t all work the same.\nThe general idea is that you have a database which keeps track of the window of time between which a retry is considered as valid to pass greylisting for a given MX. However, how the MX is considered for a retry varies and while some solutions will just look at the source IP address, others will also look at the MAIL FROM, others will also look at the RCPT TO, and based on my own studying of some failure patterns, the unique Message-ID is sometimes also tracked.\nThe OpenBSD spamd(8) for example tracks the source IP address, the identification hostname provided at HELO/EHLO, the envelope-from (MAIL FROM) and the envelope-to (RCPT TO). So, to pass greylisting, an MX must retry during the proper window of time, coming from the same IP address, identifying with the same HELO/EHLO, from the same MAIL FROM and to the same RCPT TO.\nThe problem with greylisting and Big Mailer Corps # For small destinations, greylisting works nicely as they tend to use the same MX for incoming and outgoing trafic. The MX will contact you, be told to retry later, then be accepted when it comes back after a minute or so.\nThen, you have Gmail / Yahoo / Outlook / \u0026hellip; which not only have different MX for incoming and outgoing trafic, but also have dozens and dozens of outgoing mail exchangers. This leads to a situation where an initial MX contacts you, gets asked to retry, but you never see a retry because it came from another MX which was asked to retry, and so on\u0026hellip;\nGreylisting with such hosts is unusable and results in mails being delayed for hours, days, or even expire and never reach you.\nBut at the same time, greylisting keep a lot of nuisance outside too as it kills most scripts and compromised computers acting as spamming bots, so a lot of operators put in place various work-arounds to keep using greylisting for small senders and not for Big Mailer Corps.\nWork-around strategies # I used the plural but quite frankly there\u0026rsquo;s only one strategy: whitelisting.\nThe work-around is to figure out IP addresses of Big Mailer Corps, then give them a free pass and allow them to skip greylisting.\nDoes this defeat greylisting ? Not really. Greylisting kills non-RFC compliant MX, but they operate RFC-compliant MX and if they retried from the same outgoing MX they would pass greylisting hands up.\nFair enough, let\u0026rsquo;s whitelist them\u0026hellip; but how ?\nFor a while, I used a naive approach which consisted in whitelisting the /24 of any Big Mailer Corps MX that would get greylisted. After a while, my list was big enough that new ones would rarely get greylisted. It was tedious, it didn\u0026rsquo;t cover the case of new ones hitting me from new ranges, and I ended up merging lists from various people with mine to make sure I add as many as possible. This resulted in a constantly growing whitelist.\nThis was not really smart because all of the Big Mailer Corps have SPF. They actually advertise in a DNS record which IP they will contact you with, so that whenever you receive a connection claiming to be from them, you can do a lookup and check if it is legit. So, with that in mind, I wrote spfwalk (now part of smtpctl) in January 2018, a tool that would walk through the SPF record of domains and extract as many IP addresses as it could. There is a write up about spfwalk on this blog if you\u0026rsquo;re interested. This tool was intended to replace the dummy approach I used until then, but it is in no way perfect, it is best effort.\nThe way SPF records work means that it\u0026rsquo;s easy to check if an IP matches a policy but not to extract a list of IP addresses for later check. For example, some policies expect runtime reverse DNS resolution of the source IP address to check if they match a domain, so there\u0026rsquo;s literally NO IP address to extract from the record. So\u0026hellip; this works for most cases until you hit one where it doesn\u0026rsquo;t.\nA lot of people are happy with SPF walking, it does improve the situation considerably, but it is very hackish and I personally don\u0026rsquo;t use that tool anymore as I feel it\u0026rsquo;s not the right way to tackle the issue.\nThe filter-greylist proof of concept # Last month, I wrote a proof-of-concept filter for OpenSMTPD which uses a different approach and that is available on Github.\nLet me clarify first that I don\u0026rsquo;t claim to have invented this, it just occured to me that it was a good idea so I implemented it and it may already be available elsewhere. Since I haven\u0026rsquo;t looked very hard, (that\u0026rsquo;s a synonym for \u0026ldquo;at all\u0026rdquo;), not only do I not know of another implementation in another opensource software, but if such an implementation exists, I\u0026rsquo;m also unaware if my implementation tackles the idea the same way.\nThe base idea is that we don\u0026rsquo;t really want Big Mailer Corps to skip the greylisting, we resort to whitelisting SPF because what we really want is for all IP addresses from a Big Mailer Corp to be considered as equivalent when it comes to greylisting retries. And by whitelisting IP addresses from the SPF record, what we really want is to consider the IP addresses from the SPF record as equivalent\u0026hellip; In other words, what we want is SPF-aware greylisting.\nThe proof of concept considers that there are two kinds of senders: those not publishing SPF records and those publishing SPF records.\nMX that do not publish SPF are greylisted by source address, like they would with a regular greylisting. This doesn\u0026rsquo;t affect a lot of legitimate senders because Big Mailer Corps require senders to provide SPF in order not to degrade their reputation, and because Big Mailer Corps represent such a large e-mail address space, legitimate senders tend to comply.\nThen, you have the MX that do publish SPF records, which includes all Big Mailer Corps for obvious reasons. And for these, instead of greylisting by source address, we can do an SPF-aware greylisting for the domain.\nAssuming this is the first mail we receive from an MX, when a new connection arrives, the filter tracks the source IP address. The SMTP session moves forward, initiates an SMTP transaction, and then the client provides the envelope-from (MAIL FROM):\nS: 220 in.mailbrix.mx ESMTP OpenSMTPD C: EHLO localhost S: [...] S: 250 in.mailbrix.mx Hello localhost [127.0.0.1], pleased to meet you C: MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; At this point, the filter can do an SPF lookup for the envelope-from domain, poolp.org. If it doesn\u0026rsquo;t find an SPF record, it can simply keep track of the source address and issue a retry request.\nIf it finds an SPF record, it checks if the source IP address is valid. If it is not, then it supposedly doesn\u0026rsquo;t come from the sender domain, so the filter keeps track of the source address and issues a retry request. A stricter approach could be to reject the sender but I don\u0026rsquo;t think it\u0026rsquo;s the goal of a greylisting to check SPF validity.\nIf, however, the source IP address is valid for the SPF record, then instead of keeping track of the source address, the filter keeps track of the domain and issues a retry request.\nUpon a second connection, when reaching the same point in an SMTP transaction, if the source address is valid for the SPF record and because we track the domain and not the source IP address, any SPF valid IP address will be considered equivalent to pass greylisting. What this means, in simpler words, is that if Gmail comes from one IP address, it is allowed to retry from another, as long as both IP addresses are declared in SPF.\nThis runtime SPF resolution also ensures that there is no need to pre-whitelist IP addresses, instead SPF-enabled domains are whitelisted as a whole, there\u0026rsquo;s no need to run periodic cron or anything to catch up new addresses. This way of working is fully compatible with non-SPF domains which degrade to IP-greylisted.\nIt can\u0026rsquo;t be implemented in a daemon like spamd(8), the decision to greylist or let the session move forward is taken at MAIL FROM, whereas redirection to spamd(8) is decided at connect time through firewall rules.\nDoes this get any better ? # Dunno but I doubt so.\nThere may be some adaptations to slightly improve the idea, but I personally doubt there\u0026rsquo;s much more to do in that area: an SPF-aware greylisting solves the issue that breaks greylisting for Big Mailer Corps, no more, no less.\nKeeping in mind that spamming is a volume industry, more interesting work can be done in raising the cost of sending volumes: just like spammers don\u0026rsquo;t like retrying with uncertain result, they REALLY dislike hosts that are laggy, because it has a HUGE impact on throughput and queue size. In my filter-senderscore, I have already implemented something along these lines by correlating a delay to a sender reputation, achieving stateless tarpitting for low reputation senders.\nIf I\u0026rsquo;m spending more time making OpenSMTPD a hard target to spammers it will be through time-penalty for sure.\n","date":"1 December 2019","permalink":"/posts/2019-12-01/spf-aware-greylisting-and-filter-greylist/","section":"Posts","summary":"TL;DR: - greylisting is a sound idea - yet it is not really practical today - people tend to disable it or find work-arounds - SPF-aware greylisting makes greylisting usable again SMTP failures in a nutshell # SMTP is a fail-safe protocol which attempts very hard to ensure that messages do not get lost once they are in transit.","title":"SPF-aware greylisting and filter-greylist"},{"content":" TL;DR: - our CI was improved - a new OpenSMTPD release, 6.6.1p1 took place to deal with portable issues - multiple portability issues were adressed - new table API in the works - filter-rspamd and filter-senderscore were improved - filter-greylist proof-of-concept published - wrote 2 chapters for my book Got myself a pinebook pro # I got myself an arm64 pinebook pro:\nMy only goal for it was to have a convenient machine on which I could learn some arm64 assembly, and do some portability work on an architecture that\u0026rsquo;s different from amd64.\nI still aim at getting myself a proper laptop in a few months with the sponsorship money, this one is not going to be comfortable enough, but it\u0026rsquo;s really a nice to have.\nI bricked it on the first day, it took me a while to get it back in shape, but several issues I mentionned in this article were fixed on that machine !\nBetter CI # Ihor Antonov did a very neat job at improving CI infrastructure for the project.\nUntil his contributions, we would only have Ubuntu+glibc in our CI so whenever someone reported a build breakage on another system or distro, I would spin a VPS somewhere to install the same system and attempt a build there.\nThis was less than ideal because sometimes by fixing the build on a target, I could break another, and I would spend an afternoon spinning VPS with different systems until everything was back to normal on all.\nHe currently set up Arch, Alpine and Ubuntu, which provides us with a nice combination of glibc, musl libc, OpenSSL and LibreSSL build targets, and helps spot build breakages right away. This made some of the work I\u0026rsquo;ll describe below considerably less painful.\nIt greatly improved our support for other systems.\nOpenSMTPD 6.6.1p1 released and portable development branch work # The 6.6.0p1 version was released in October and, since it became easier to install for several distributions, it also led more users to test it and report issues on their distributions.\nFor most people it worked fine, but on some systems the build was broken and on others the daemon would blow up at runtime. Given where it blew, it was pretty obvious that the issues were really related to the compatibility layer for the most part. Still, it uncovered many interesting topics.\nThey were all fixed and a new version, 6.6.1p1, was released so people would have a working OpenSMTPD on all distributions, then I proceeded to work on improving the compat layer so similar issues don\u0026rsquo;t come back and haunt us in the future.\nBelow are explained the top issues I worked on for 6.6.1p1 and after.\narc4random() # On OpenBSD, arc4random() provides high quality randomness which OpenSMTPD relies heavily upon for pretty much everything. The implementation provides a stream cipher output which is sliced between consumers, with the help of the kernel and userland, so that consumers cannot backtrack or predict the stream. There\u0026rsquo;s an excellent talk by deraadt@ which explains how this works.\nIn the portable release of OpenSMTPD, we attempt to detect if arc4random() is available on the base system and provide a replacement implementation if not. The replacement is a reliable stream cipher, so it is not some low-grade best-effort hack, however we still assume a system arc4random() to be stronger and preferable if only because it could provide system-wide slicing.\nIn addition, OpenSMTPD prefers LibreSSL but can also link against OpenSSL. The former provides an arc4random() implementation if one is not available on the system, the latter doesn\u0026rsquo;t. So depending on the combination of system and TLS library, we may not have arc4random() at all, we may have it from the TLS library, we may have it from the system or we may have it from both. And we always want to use the best version available, assuming LibreSSL\u0026rsquo;s version to be preferable over ours (even if it really is the same).\nThe compat layer was not too good at catching these cases and it was also not too good at ONLY enabling bits of compat layer. In some situations where it would not export our own version of arc4random(), it would still export some internal symbols that led to an assortment of build time and runtime failures.\nThe whole arc4random() detection was reworked and works nicely now.\nIPv6 broken on some systems # During the 6.5 -\u0026gt; 6.6 development cycle, I realized that our way to represent IPv6 address in textual format was wrong and reworked it. This was not a big rework, the diff was fairly small, and I merged it to the portable branch. CI didn\u0026rsquo;t complain and my initial testing didn\u0026rsquo;t show problems because it perfectly dodged the problematic code path (more on that shortly).\nThere are some systems **cough cough** using glibc **cough cough** that provide a broken inet_net_pton() function, supporting only IPv4 and returning EAFNOTSUPPORT when passed an IPv6 address. I first spotted this in 2013 and made a temp_inet_net_pton() function which would be used temporarily as a fallback if inet_net_pton() failed, then renamed it to broken_inet_net_pton_ipv6() in 2016 when it became clear that this would never be fixed in glibc.\nThe OpenBSD code doesn\u0026rsquo;t have this fallback code, so when the merge was applied to the portable branch, it didn\u0026rsquo;t take into account that portable version needed a parameter change for broken_inet_net_pton_ipv6(). On systems with a working inet_net_pton() the code would be bypassed, so this issue was not visible, but on other systems **cough cough** using glibc **cough cough** it would lead to the function failing and broken IPv6 support.\nFix was applied and IPv6 worked again on systmes with a broken inet_net_pton().\nAlpine Linux issues (musl libc really) # There were reports of crashes from an Alpine Linux user.\nI could not reproduce and the more I read the related code, the more it read correct. Futhermore, he reported that the previous version of OpenSMTPD worked flawlessly, and I couldn\u0026rsquo;t see any relevant change that could led to his crash.\nI couldn\u0026rsquo;t reproduce so it took a long while to investigate, but eventually we managed to obtain a backtrace which pinpointed freeaddrinfo() as being the culprit.\nSo what happened ?\nOn one side we have getaddrinfo() which resolves a node into a serie of struct sockaddr * wrapped inside a struct addrinfo * providing the linked list:\nstruct addrinfo { int ai_flags; int ai_family; int ai_socktype; int ai_protocol; socklen_t ai_addrlen; struct sockaddr *ai_addr; char *ai_canonname; struct addrinfo *ai_next; }; and on the other end, we have freeaddrinfo() which releases the items from the linked list returned by getaddrinfo(). Most notably every struct addrinfo * from the list but also the ai_canonname member of each item.\nThe interface for both functions is standardized.\nA common implementation for freeaddrinfo() is the following one:\nvoid freeaddrinfo(struct addrinfo *ai) { struct addrinfo *p; do { p = ai; ai = ai-\u0026gt;ai_next; free(p-\u0026gt;ai_canonname); free(p); } while (ai); } This is straight from the OpenBSD libc but you will find a similar implementation on other BSD systems, on Android, on OSX and on Linux/glibc.\nSo why would it crash on Alpine Linux ?\nHere\u0026rsquo;s the version of it on musl libc:\nvoid freeaddrinfo(struct addrinfo *p) { size_t cnt; for (cnt=1; p-\u0026gt;ai_next; cnt++, p=p-\u0026gt;ai_next); struct aibuf *b = (void *)((char *)p - offsetof(struct aibuf, ai)); b -= b-\u0026gt;slot; LOCK(b-\u0026gt;lock); if (!(b-\u0026gt;ref -= cnt)) free(b); else UNLOCK(b-\u0026gt;lock); } This version assumes an underlying struct aibuf * where the items of struct addrinfo * are stored, not really as a linked list, but rather as contiguous items linked together. This allows musl to do pointer arithmetics to locate elements of the list, it however diverges greatly from other implementations.\nNot saying this at all as a critique to code quality, but I personally have trouble understanding this code, I\u0026rsquo;m unable to understand from reading only if this violates the standard:\nThe freeaddrinfo() function shall support the freeing of arbitrary sublists of an addrinfo list originally returned by getaddrinfo(). Intuitively the code seems to not accept freeaddrinfo() being called on the second element of the addrinfo list, but I may very well be wrong and this needs testing, maybe the next time I spin an alpine VM.\nWhat is certain however is that struct aibuf * is a musl internal thing. The consequence being that if freeaddrinfo() expects to find it, then it can only be called on a struct addrinfo * that was built through getaddrinfo().\nUnluckily for us, OpenSMTPD comes with an asynchronous resolver that crafts its own struct addrinfo *. This results in a linked list that won\u0026rsquo;t have the underlying struct aibuf * and will work everywhere\u0026hellip; but on musl.\nBut then, how comes the previous version of OpenSMTPD worked flawlessly then?\nHere\u0026rsquo;s the version of freeaddrinfo() that musl used until October 2018:\nvoid freeaddrinfo(struct addrinfo *p) { free(p); } It didn\u0026rsquo;t work flawlessly.\nWhat happened was that before October 2018 the musl code didn\u0026rsquo;t have an underlying structure, so it didn\u0026rsquo;t crash when playing pointer arithmetic tricks to locate elements from the list. It however already assumed a unique contiguous chunk of memory and released it as a whole so, when passed the linked list OpenSMTPD crafted, it would partially release the first item of the list and leak members from that item as well as the entire sublist.\nI will contact the musl maintainers to see if they have a plan for this or won\u0026rsquo;t fix, however in the meantime I have introduced a portable_freeaddrinfo() to OpenSMTPD, as a work-around, so that whenever we call getaddrinfo() we release with freeaddrinfo(), but whenever we craft a list ourselves we release with portable_freeaddrinfo(), effectively making OpenSMTPD work again correctly with musl.\nautoconf and openbsd-compat/ cleaned up # Our portable layer was released many many years ago thanks to the work of Charles Longeau, who bootstrapped the OpenSMTPD portable project by gathering bits of the compat layer from OpenSSH and OpenNTPD. This allowed us to make OpenSMTPD available to FreeBSD, NetBSD and Linux after a few weeks or work, broadening our community and helping us get more reports to improve the software.\nNeither of us was an expert in autohell and mistakes were made back then, which we worked around throughout the years, and things worked fairly fine as long as we didn\u0026rsquo;t look too much into details.\nThe compat layer was meant to provide the features we detected as missing on a target system, however it built the entire compat layer and played #ifdef games to exclude sections of code we didn\u0026rsquo;t need on a host. In some cases, the #ifdef didn\u0026rsquo;t properly exclude everything and resulted in compat layer carrying a function that it shouldn\u0026rsquo;t. For functions like strlcpy() this wouldn\u0026rsquo;t be an issue because who cares if we\u0026rsquo;re using OpenSMTPD\u0026rsquo;s version over the systems\u0026rsquo;, but for some others this was an issue.\nThe arc4random() section above described one case of where this could be an issue, because we might pick a less preferable implementation, or it might pick the proper implementation but shadow an underlying symbol with consequences ranging from nothing to a crash.\nAnother case is the OpenSMTPD portable on top of OpenBSD case. It makes sense that building OpenSMTPD portable on top of OpenBSD should result in an empty compat layer since\u0026hellip; well, there should be no missing OpenBSD function in OpenBSD.\nIn practice, the compat layer still carried things and this made me uncomfortable. For years I discouraged using OpenSMTPD portable on top of OpenBSD, claiming it was unsupported, but this was an intuitive claim and while technically there should be no issue, something felt really wrong about it.\nThen OpenBSD\u0026rsquo;s pledge() system call came to the scene and my worry became very real: building OpenSMTPD portable on OpenBSD led to an executable that would crash at startup. The compat layer didn\u0026rsquo;t properly exclude a replacement function, which relied upon a system call that we didn\u0026rsquo;t allow in our pledges. BOOM.\nI proceeded to rework the compat layer and fix a few things:\nFirst of all, I have removed pretty much EVERYTHING that\u0026rsquo;s not needed for OpenSMTPD. Among the things we carried from OpenSSH or OpenNTPD are functions which are never used in OpenSMTPD, and so I have removed their detection in configure as well as their replacement implementations in the compat layer. Anything we have in the compat layer is now needed.\nThen, I have cleaned up a bit the function detection in configure.ac to make sure that functions we need replacements for are properly detected. There is still work to do in terms of handling priorities, as in \u0026ldquo;do I pick the system\u0026rsquo;s or the one from a library\u0026rdquo;, but the logic to handle all detections THEN ULTIMATELY decide if we want a replacement is already there.\nFinally, I have switched from a mode where the compat layer is built with all of its .c files included but sprinkled with #ifdef to exclude portions, to a mode where the compat layer conditionally includes .c files based on configure detection. This has the nice side-effects that we can\u0026rsquo;t accidentally carry something that was not properly #ifdef-ed, but also that you can spot at build time what\u0026rsquo;s included or not on a target host, which can raise eyebrows right away if you see arc4random.c built on a system which provides arc4random() for instance.\nThis is already in place in the portable branch, so this is what you get on an OpenBSD system:\nlaptop$ make gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -I/usr/local/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wno-pointer-sign -fno-strict-aliasing -fno-builtin-memset -c -o empty.o empty.c rm -f libopenbsd-compat.a /usr/bin/ar cru libopenbsd-compat.a empty.o ranlib libopenbsd-compat.a laptop$ and on a Linux system (Archlinux, glibc, LibreSSL):\n$ make gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o empty.o empty.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o closefrom.o closefrom.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o crypt_checkpass.o crypt_checkpass.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o errc.o errc.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o event_asr_run.o event_asr_run.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o fgetln.o fgetln.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o fmt_scaled.o fmt_scaled.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o fparseln.o fparseln.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o freezero.o freezero.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o getpeereid.o getpeereid.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o imsg.o imsg.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o imsg-buffer.o imsg-buffer.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o pidfile.o pidfile.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o recallocarray.o recallocarray.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o setproctitle.o setproctitle.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o strlcat.o strlcat.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o strlcpy.o strlcpy.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o strmode.o strmode.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o strtonum.o strtonum.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o vis.o vis.c rm -f libopenbsd.a /usr/sbin/ar cru libopenbsd.a empty.o closefrom.o crypt_checkpass.o errc.o event_asr_run.o fgetln.o fmt_scaled.o fparseln.o freezero.o getpeereid.o imsg.o imsg-buffer.o pidfile.o recallocarray.o setproctitle.o strlcat.o strlcpy.o strmode.o strtonum.o vis.o /usr/sbin/ar: `u\u0026#39; modifier ignored since `D\u0026#39; is the default (see `U\u0026#39;) ranlib libopenbsd.a $ There is still a lot of work to do in the compat layer, the library was part of it but the handling of missing headers and such is also not too good. I have plans to keep working on this for the next few months until I\u0026rsquo;m comfident that running the portable branch on OpenBSD doesn\u0026rsquo;t have side-effects.\ntable API rework # Something I mentioned last month was that I wanted to turn the table API into something similar to filters, so people could easily write table backends in any language and rely on existing libraries.\nI have made very good progress on this front and I wrote a couple table backends using the new interface, however I\u0026rsquo;m not done yet and I will keep the details for another article.\nfilter-rspamd improved # My filter-rspamd got additional contributions from Reio Remma (@whataboutpereira), to extend the headers and have them include the scoring for the symbols they matched.\nThis results from a discussion and a previous pull request that I had rejected as I felt the header was way too verbose.\nThe 0.1.4 release of filter-rspamd would display headers as:\nX-Spam: yes X-Spam-Score: 8.789999 / 15 X-Spam-Symbols: MIME_TRACE, FROM_NEQ_ENVFROM, ARC_NA, RCPT_COUNT_ONE, URI_COUNT_ODD, DMARC_POLICY_ALLOW, R_DKIM_ALLOW, HAS_X_PRIO_THREE, MIME_GOOD, HAS_REPLYTO, R_SPF_ALLOW, FROM_HAS_DN, TO_MATCH_ENVRCPT_ALL, RCVD_TLS_ALL, FORGED_SENDER, MID_RHS_MATCH_FROM, HTML_SHORT_LINK_IMG_1, RCVD_IN_DNSWL_NONE, DKIM_TRACE, RSPAMD_URIBL, PREVIOUSLY_DELIVERED, HAS_LIST_UNSUB, BAD_REP_POLICIES, SUBJECT_ENDS_QUESTION, ASN, REPLYTO_DOM_NEQ_FROM_DOM, RCVD_COUNT_TWO, TO_DN_NONE In development, it now replaces the X-Spam-Symbols with X-Spam-Status and outputs as follows:\nX-Spam-Status: Yes, score=7.267 required=15.000 tests=[ARC_NA=0.000, BAYES_SPAM=5.100, DCC_BULK=0.667 FROM_EQ_ENVFROM=0.000, FROM_HAS_DN=0.000, MID_RHS_MATCH_FROM=0.000 MIME_BAD_ATTACHMENT=1.600, MIME_GOOD=-0.100, MIME_TRACE=0.000 PREVIOUSLY_DELIVERED=0.000, RCPT_COUNT_ONE=0.000 RCVD_COUNT_THREE=0.000, RCVD_TLS_LAST=0.000, RCVD_VIA_SMTP_AUTH=0.000 TO_DN_NONE=0.000, TO_MATCH_ENVRCPT_ALL=0.000] This is not only more compact but also avoids having to go look into /var/log/rspamd/rspamd.log for details about what scored how much.\nfilter-senderscore improved # In July, I wrote filter-senderscore which looks up the reputation for a source address in the SenderScore database, and allows blocking, junking and delaying sessions whose score falls below some threshold.\nThe filter works fairly well so I didn\u0026rsquo;t bother making any improvement to it.\nIn September, the junk action was introduced which allows a filter to tell that a session or transaction can be junked by OpenSMTPD. Since the filter was written before that feature, it did the junking itself by registering for the dataline phase and injecting a X-Spam: yes header, so I simplified by having the filter use the junk action instead.\nWhile I was there, I thought it would be nice to add an option to let user configure at which phase the client should be disconnected. Until now, if the -blockBelow option was provided and the reputation for an IP fell below the threshold, the session would be disconnected at the banner. The new -blockPhase option allows deferring the disconnect to later, making it possible to track the sender or recipient addresses before disconnecting if needed.\nfilter-greylist proof-of-concept # The SMTP protocol provides a mechanism for \u0026ldquo;temporary failures\u0026rdquo; which requests a sender to retry transfer of a message. A regular MTA like OpenSMTPD will honor that request but scripts and spam tools that are targeted at spamming en masse will not necessarily retry, as it would slow them and force them to keep track of which hosts need to be retried.\nThe idea behind greylisting is to trigger a temporary failure voluntarily when first contacted by an MX, then keep a reasonnable window of time within which the MX should perform a retry. If the MX retries in that timeframe then it is whitelisted, otherwise it is either blacklisted or kept in temporary-failure-land forever.\nMultiple greylisting solutions exist, including the spamd(8) (man page) daemon in OpenBSD. A lot of them are unusable as is with big mailers like Gmail / Yahoo / Hotmail because their retries do not come from the same MX host and they never get whitelisted.\nA few years ago, I introduced an spf walk utility which made things a bit better.\nWhat it did was to perform an SPF walk to catch as many IP addresses as it could from an SPF record to they could be whitelisted. Since all big mailers publish SPF records, it improved considerably the situation.\nIt wasn\u0026rsquo;t perfect either because some SPF records are context-dependant and can\u0026rsquo;t be looked up, others rely on hostnames which have geo-localized IP addresses preventing whitelists from being shared across countries, and finally whitelisting this way didn\u0026rsquo;t account for updates of SPF records\u0026hellip; so spf walk had to be ran frequently and there was always a possibility that an SPF change happened right after a run. A lot of people see spf walk as THE solution to the big mailer greylisting issue, but I don\u0026rsquo;t.\nI wrote filter-greylist as a proof-of-concept for an SPF-aware greylisting.\nWhat it does is consider that there are two kind of senders, those that do SPF and those that don\u0026rsquo;t. If a host doesn\u0026rsquo;t do SPF, then retries are expected to come from the same IP address. If a host does SPF, then the sending domain is remembered and ALL IP addresses part of the SPF record are considered as valid for the retry. This means that if Gmail contacts from an IP, then retries from another, this will be considered as a valid retry.\nI have been running with this filter for weeks and it works perfect, to the point where I had actually forgotten that it was running.\nThe filter has been published on Github but since this is a bit more risky than my other filters, test at your own risks.\n2 chapters of the book written # I have written two additional chapters to my OpenSMTPD book.\nI\u0026rsquo;m now focusing on the configuration examples which is probably the most interesting part for readers, but by far the most annoying part to write for me given how many features we support :-)\nWhat next ? # It\u0026rsquo;s unclear what I\u0026rsquo;ll do in December because I have multiple works in progress, the table API will be worked on for sure, the smtp-out reporting also.\nI\u0026rsquo;d like to improve my filter-greylist proof of concept but it\u0026rsquo;s not high priority for me, I\u0026rsquo;ll be happy to review pull requests ;-)\nI have started writing articles outside of my monthly reports, however they grow really big really fast, I need to find how to properly split.\n","date":"17 November 2019","permalink":"/posts/2019-11-17/november-2019-report-opensmtpd-6.6.1p1-filter-greylist-and-tons-of-portable-cleanup/","section":"Posts","summary":"TL;DR: - our CI was improved - a new OpenSMTPD release, 6.6.1p1 took place to deal with portable issues - multiple portability issues were adressed - new table API in the works - filter-rspamd and filter-senderscore were improved - filter-greylist proof-of-concept published - wrote 2 chapters for my book Got myself a pinebook pro # I got myself an arm64 pinebook pro:","title":"November 2019 report: OpenSMTPD 6.6.1p1, filter-greylist and tons of portable cleanup"},{"content":" TL;DR: - yay, surprise emergency hand surgery... - OpenSMTPD 6.6.0 was tagged and released, including portable version - Merged contributions to fix filter-rspamd bug with DKIM - Work resumed on 6.7.0 feature - An OpenSMTPD book is in the works How I landed in the emergencies # I accidentally cut my hand real bad early October and rushed to the nearest hospital. They had a look, told me it was superficial and required to be sutured, so I was out a few hours later with a few stitches and pain killers.\nA few days later, my hand was swell so I rushed to another hospital that was specialized in hand wounds, expecting a second advice and maybe some additional drugs.\nTurns out the cut was not superficial at all, it had hit joints and needed more than just a few stitches. The stitches also lacked propre protection so the wound had infected. I was told that I had to undergo immediate surgery because the infection was looking real bad, and that I was lucky I took this seriously and came right away.\nSo this is how my month of October was ruined. It took me two weeks to be able to type back on a keyboard, it is still painful and I\u0026rsquo;m still unable to practice music or martial arts as I can\u0026rsquo;t close my fingers entirely.\nLuckily, I should be able to recover fully, but\u0026hellip; it will take some time.\nMy work this month was obviously impacted, BUT I managed to pull the OpenSMTPD 6.6.0 release and do some thinking work around redesigns planned for development cycle that just started. I will discuss a bit my plan in this article.\nOpenSMTPD 6.6.0 was released # I have published and announced the OpenSMTPD 6.6.0 release today.\nIt is a very important release for us, probably one of the most important that happened in the last few years, as it brings the filter API and makes it possible to compete with other MTA by interfacing with a wide variety of third party tools.\nIt is now possible to do dnsbl, interface with antispam solutions such as Rspamd or SpamAssassin, perform DKIM signing and verifying at session time, rejecting sessions that do not behave as we expect, \u0026hellip;\nThe API was released as an undocumented experimental feature. This doesn\u0026rsquo;t mean it is unstable, the code has been used on the OpenSMTPD mail exchanger for over a year, used by developers for at least as long, and tested by users in the wild for several months. The only reason it is considered experimental is that we didn\u0026rsquo;t want to \u0026ldquo;release\u0026rdquo; a stable version of the API with easy access to documentation, without having a chance to obtain a first round of feedback over the course of a release with a limited set of filters and developers ;-)\nIn addition to filters, this is the first release that is considering LibreSSL as its target TLS library while providing support for OpenSSL. A lot of distributions remained with ancient OpenSMTPD release because of TLS library issues. This release is going to make every distro and system out there able to catch up and use the latest OpenSMTPD release.\nDKIM signing and verifying was fixed in filter-rspamd # Thanks to Reio Remma @whataboutpereira, a bug was fixed in filter-rspamd which would cause some mails to be incorrectly DKIM signed or verified.\nThe issue was that filters receive the raw SMTP lines during the DATA phase, and the filter didn\u0026rsquo;t account for the escaping of lines starting with a dot, leading to some mails being incorrectly signed or verified.\nHis fix was committed to github and prompted me to issue a new release, before updating the OpenBSD port as well.\nI have received an ok to commit the fix to stable packages and will do so this week-end.\nOpenSMTPD 6.7.x work # With my hand out of service, I spent time thinking about bits that needed a redesign.\nThe smtp-out reporting was almost completed and didn\u0026rsquo;t make it to this release, so I will make sure it is completed very soon so it doesn\u0026rsquo;t miss the next release.\nThe switch to libtls requires a standalone OpenSSL-based libtls interface to be written, I have started taking a look at it too. I already have a libtls-based OpenSMTPD in a branch, but I don\u0026rsquo;t want to switch to that until we know for sure that we\u0026rsquo;ll be able to provide a portable release, so hopefully for 6.7.x in six months and more realistically for 6.8.x in a year.\nThe MTA layer needs to be reworked and was discussed at lengths with Eric Faurot, this would be a change transparent to users but which would make the daemon MUCH simpler for developers. I hope we can pull that for 6.7.x but work has to start very soon to make this happen, so we\u0026rsquo;ll see in the next few weeks how this project evolves.\nFinally, I\u0026rsquo;d like to replace the current table API with one that\u0026rsquo;s similar to the filter API. The current API is a binary protocol relying on the imsg framework, requiring C plumbing and making the writing of table backends limited to skilled developers. I already discussed with Eric Faurot my feeling that it should be converted to a line-based protocol, similar to filters, which would allow backends to be written in any language, providing the same benefits as filters in terms of privilege separation and memory isolation. This would make the OpenSMTPD-extras repository deprecated, but given that I\u0026rsquo;m the author for pretty much every addon there, I think I can easily replace them with new interface counterparts.\nAn OpenSMTPD book is in the works # I had started writing a book about OpenSMTPD years ago but, because huge reworks were in progress in both configuration and filter areas, I kept delaying as I didn\u0026rsquo;t want to write configuration bits that would have to be rewritten from scratch, and I didn\u0026rsquo;t want the book to lack a chapter about spam filtering.\nWith all of the work that has gone through OpenSMTPD in the last two years, I feel that there\u0026rsquo;s no longer a gap to fill and a very complete and useful book can be written. I have therefore resumed work and currently have a PDF that\u0026rsquo;s well over a hundred pages when fully built, describing the history and design of OpenSMTPD, large chunks of how SMTP and DNS work, how to setup your MX, how to configure various setups, and much more.\nNeedless to say, this is a LOT of work.\nIt will take months to complete the book and publish something I\u0026rsquo;ll be happy with, so if you\u0026rsquo;re interested let me know and if you want to support me and help me get this out the sooner, then you know how to show your love and buy me time :-)\nWhat next ? # I have started writing a new article related to SMTP and reputation, I will try to complete it next week and provide early access to my patrons.\nDevelopment cycle for OpenSMTPD as resumed, I have a few tickets to tackle on Github and a few ideas that I have to dig, but the table API rework is a very high priority thing to do for me.\nI will also resume working on cleaning up the portability layer.\n","date":"26 October 2019","permalink":"/posts/2019-10-26/october-2019-report-opensmtpd-6.6.0-release-mostly/","section":"Posts","summary":"TL;DR: - yay, surprise emergency hand surgery... - OpenSMTPD 6.6.0 was tagged and released, including portable version - Merged contributions to fix filter-rspamd bug with DKIM - Work resumed on 6.","title":"October 2019 report: OpenSMTPD 6.6.0 release mostly"},{"content":" TL;DR: - I started writing this post a week ago but got interrupted by a baby, Jules - Spent MANY hours on writing OpenSMTPD-related articles - Enabled continuous integration in the OpenSMTPD portable repository - Managed to get rid of all the blocking issues for OpenSMTPD 6.6.0 release - Added some features and fixed a crash in filter-rspamd Welcome Jules, fork() completed # In my last report from August, I concluded not knowing if this report would be published on time.\nThe expected date for bringing our little monkey home was the 2nd of October, but while I started writing this article the 21st of September, he decided not to stick to the plan.\nThis is why I\u0026rsquo;m publishing today an article dated from the 21st, which is when I started writing it ;-)\nSo my biggest completed project for this September report was the little Jules, born Sunday the 22nd of September 2019 at 08:03 AM in the city of Nantes.\nHopefully a future OpenBSD hacker:\nThough he can be anything, including a unicorn if he wants. He cute af. The cutest.\nWrote two articles for the community # The first one, You should not run your mail server because mail is hard was intended to be an MTA-agnostic article, debunking some of the common claims that mail is hard.\nThe second one, Setting up a mail server with OpenSMTPD, Dovecot and Rspamd was intended both as a follow up to the previous article, to back the claims with a technical setup, but also as a mean to promote OpenSMTPD AND answer the \u0026ldquo;there\u0026rsquo;s not many OpenSMTPD tutorials out there\u0026rdquo; request.\nThe first article took me a couple hours to write, whereas the second one took me about ten hours because I wanted to explain everything I did and make it a reference tutorial for OpenSMTPD users.\nDeployed CI on our Github mirror repository # OpenSMTPD is developed on OpenBSD, so we\u0026rsquo;re guaranteed that it builds because we benefit from the HYAYFBTB1 continuous integration (CI) mechanism.\nThe portable branch is different. It is mainly maintained by myself and\u0026hellip; I only run OpenBSD and OSX daily. Most of the changes are merges from the OpenBSD branch which aren\u0026rsquo;t challenging to portability, but occasionally I need to rely on community feedback or spin a server for a specific distribution.\nSometimes, I mess up and the trivial merge from OpenBSD lacks an include or uses a function that I thought was portable, until someone reports the build is broken.\nGithub enabled CI on our account so I immediately set it up on the OpenSMTPD repository. It will still need improvements but at this point, every time we commit to the portable branch, a bootstrap, configure, make and make install are done, and members of the team all get an e-mail if the build is broken.\nWhile at it, I also added CI to the filter-rspamd and filter-senderscore repositories.\n1Hackers Yell At You For Breaking The Build\nIntroduced junk action for builtin and proc filters # Until now, builtin and proc filters had only 4 possible actions: proceed to let other filters handle the session, rewrite to man-in-the-middle rewrite parameters to the filter hook, reject to reject the phase with a custom SMTP message, and disconnect to reject AND disconnect the phase with a custom SMTP message.\nMost filters used reject or disconnect as a mean to kill spammers upfront, with builtin filters such as:\nfilter check_rdns phase connect match !rdns disconnect \u0026#34;550 GO AWAY\u0026#34; But I received a lot of feedback from people scared of rejecting legitimate mails by being too harsh, and asking if there was a way to junk the messages rather than throw them away.\nI introduced the junk action which flags a session or transaction so messages can proceed, but will get an X-Spam: yes header prepended. The following builtin filter would match the same session as above but, instead of killing the session, all messages from the session would be marked as spam:\nfilter check_rdns phase connect match !rdns junk With this new action, people can actually build confidence for a while before going stricter.\nAs a bonus, this makes it easier to write proc-filters that junk messages. Before this change, filters would have to keep state of junked sessions and register a callback for data-line phase, so they could craft the header and prepend it. With this change, they can simply junk the message right away and let OpenSMTPD prepend the header.\nImplemented Sender Rewriting Scheme support # OpenSMTPD lacked Sender Rewriting Scheme (SRS) support.\nLong story short, the Sender Policy Framework (SPF) was introduced to let a domain control which hosts can send mail on its behalf. This is very nice and all, but it doesn\u0026rsquo;t play well with mailing lists and forwarding hosts.\nFor example, if eric@faurot.net sends mail to misc@opensmtpd.org, the opensmtpd.org domain needs to send mail to all subscribers on behalf of faurot.net, but\u0026hellip; it is not allowed to do so, resulting in an SPF check failure.\nTo work around this, mailing list software encode the sender address in the protocol so it originates from the mailing list domain, and allows them to decode back upon replies:\nbb957901382a5d3f mta delivery evpid=25d3675b29721118 from=\u0026lt;misc+bounces-probe-eric=faurot.net@opensmtpd.org\u0026gt; to=\u0026lt;gilles@poolp.org\u0026gt; rcpt=\u0026lt;-\u0026gt; [...] The misc+bounces-probe-eric=faurot.net@opensmtpd.org address resolves to misc@opensmtpd.org once you remove the + subaddressing and the mailing list software handling that address uses the subaddressing value bounces-probe-eric=faurot.net to retrieve the original sender.\nSo far so good, but this only works for mailing lists. The openbsd.org domain is essentially a mail forwarder: sending mail to eric@openbsd.org really forwards to eric@faurot.net and sending mail to gilles@openbsd.orgreally forwards to gilles@poolp.org. In this case, no mailing list software encodes and decodes addresses, and this creates problems.\nIf eric@faurot.net sends mail to gilles@openbsd.org, the openbsd.org mail exchangers accepts the mail and forwards it to poolp.org. But the mail exchangers at poolp.org sees a mail originating from faurot.net coming from a mail exchanger that\u0026rsquo;s not the correct one\u0026hellip; the SPF check fails again.\nSRS comes into play and, once enabled, makes sure that any address forwarded is encoded in such a way that it passes SPF correctly and that replies can be traced back to the original sender. I won\u0026rsquo;t detail this much, there\u0026rsquo;s a specificiation that can be easily found on a search engine, but it\u0026rsquo;s now supported natively in OpenSMTPD.\nIf your mail exchanger is intended to forward mails on behalf of other domains, you can enable SRS using the following configuration:\nsrs key secret_key_to_prevent_people_from_impersonating_me [...] action \u0026#34;outgoing_mails\u0026#34; relay srs Resulting in:\nPlain and simple.\nLots of minor stuff here and there # Tons of minor stuff were committed here and there, ranging from new reporting events for developers, cleanups, stricter checks, documentation.\nI won\u0026rsquo;t list everything we did because you can check the history.\nOpenSMTPD 6.6.0 # OpenSMTPD 6.6.0 is now \u0026ldquo;feature-ready\u0026rdquo; and the only things we\u0026rsquo;ll commit until the release are bug fixes if we find any. The upcoming weeks will be stabilization only, though\u0026hellip; the last weeks were also stabilization mainly ;-)\nIf you hit issues with the portable branch, don\u0026rsquo;t worry, we tag the release when the OpenBSD version is stable and keep working on improving the portable version after, giving it its own tag. If you experience issues with portable at the time we tag 6.6.0, it doesn\u0026rsquo;t mean the issues will not be fixed before the portable version is released.\nVarious improvements to filter-rspamd # With the help of Reio Remma @whataboutpereira, filter-rspamd got a lot of new features.\nFirst of all, I fixed a concurrency-related crash experienced by two OpenBSD hackers. That was quite a mandatory feature. not crashing.\nThen, I added an X-Spam-Symbols header as we needed it to provide the same level of informations as the rspamc client, which is the alternate way of integrating Rspamd with OpenSMTPD.\nReio did a bit of cleanup, taught the filter how to notify Rspamd about the MX name, how to reuse Rspamd-provided SMTP messages in the reply to SMTP sessions, how to add / remove headers based on the Rspamd configuration. He basically made it closer to the Rspamd milter that\u0026rsquo;s available to other MTA.\nWhat next ? # I haven\u0026rsquo;t slept much this week so my mind is a bit blurry about what\u0026rsquo;s on my roadmap, but my goal is now to focus on the portable branch to improve it.\nWhen my next report is due, OpenSMTPD 6.6.0 may or may not have been released based on OpenBSD\u0026rsquo;s release plan.\nDepending on what happens, either I\u0026rsquo;ll spend the month working on the release or I\u0026rsquo;ll start working on the next big works, namely reworking the table layer to work similarly to filters, allowing tables to be written in any language.\nI may also be focusing on diapers.\n","date":"21 September 2019","permalink":"/posts/2019-09-21/september-2019-report-jules-opensmtpd-6.6.0-upcoming-release-and-related-things/","section":"Posts","summary":"TL;DR: - I started writing this post a week ago but got interrupted by a baby, Jules - Spent MANY hours on writing OpenSMTPD-related articles - Enabled continuous integration in the OpenSMTPD portable repository - Managed to get rid of all the blocking issues for OpenSMTPD 6.","title":"September 2019 report: Jules, OpenSMTPD 6.6.0 upcoming release and related things"},{"content":" TL;DR: - NO TL;DR: this time, I spent hours writing, you should spend minutes reading. - OK... I explain in WAY TOO MUCH details how to setup a mail server EDIT (2019-10-26) # OpenSMTPD 6.6.0 was released today, the article was updated to reflect that it applies to the current release and no longer a development version.\nEDIT (2019-12-15) # I have refactored the article to remove the political aspect behind self-hosting, which is now a standalone article, as well as a blurb on reputation since this will become part of a serie of articles.\nWow, that was unexpected # In a previous article, I explained why I think the \u0026ldquo;mail is hard\u0026rdquo; myth is unfounded and why I think people shouldn\u0026rsquo;t be discouraged from running their own mail services. I didn\u0026rsquo;t know this article would gather so much attention, reaching over 75k reads in three days (currently past 85k) when my next most read article reached 15k reads\u0026hellip; in over a year.\nI wish I had a soundcloud. I have a patreon though. I also wish I had entered more into details and not covered things so superficially.\nThis article is DENSE but this is because I will hold hands at an absurd level, removing all my blabber will only leave very few technical parts.\nThis is for techies and sysadmins # Self-hosting mail requires a minimum of knowledge and dedication, it is not a two-clicks thing, you need to have some basic knowledge and you have to be willing to pour some time into it. If you\u0026rsquo;ve never touched a DNS zone or if you think that taking an hour to setup something is hard, then this will not be easy and I don\u0026rsquo;t recommend you do it unless you WANT to learn a ton of stuff.\nIf you\u0026rsquo;re a sysadmin who is familiar with sysadmin work, then most of the sysadmin work you already do is considerably harder than setting up a mail instrastructure. I\u0026rsquo;ll reiterate it for you: mail is not hard.\nEHLO hypno.cat # For this article, we will setup a mail server for hypno.cat, a small website for my thriving (hypothetical) hypnotherapist activity. I registered the website years ago because I liked the name, but never done anything with it beyond hosting an awesome animated file.\nGoogle, Bing and Yahoo have somehow known about it for years and indexed that front page, so they won\u0026rsquo;t consider it as a domain that was just bought by a spammer to start sending mail, however it is virtually unknown to anyone on the internet because it has no content, doesn\u0026rsquo;t link to anywhere, isn\u0026rsquo;t linked from anywhere, and has never sent or received mails from or to anyone. It is an important detail because it shows how the age of a domain has a huge impact on your reputation.\nI\u0026rsquo;m going to set it up for real as I write the article, from the moment I have booted a brand new VPS to the moment I have received mail back and forth to my Gmail account. I have chosen that one because it is over-represented in the population and this explains why people complained the most about delivery issues there. I personally think that Outlook is worse in terms of interoperability, but lets keep that for a future article maybe.\nThe mail services for hypno.cat are going to provide TLS-secured incoming and outgoing e-mail. They are also going to allow me to submit messages from my smartphone\u0026rsquo;s IMAP client, the Gmail application, so my users don\u0026rsquo;t feel much of a change in habits. I could just as easily provide a webmail, with Rainloop or Roundcube, or setup Thunderbird or mutt. Any IMAP client will work just fine so I will not cover this part, there are many alternatives and plenty of tutorials on how to set them up already. There\u0026rsquo;s even techniques to help mail clients auto-discover their configuration, you can figure that out by yourselves.\nMy outgoing mail will pass the basic checks at Gmail, namely SPF, DKIM and DMARC, which are more than enough to get your mail through. We could do a lot more, but the goal here is to find the proper balance between doing enough work to look good and remaining simple.\nIncoming mail will be filtered to reduce the volume of spam, either by killing obvious bad senders at session time when they are detected, or by providing classification so Spam is in a dedicated folder to avoid cluttering the Inbox. I could stop here, since that would already be a fairly nice setup, but I\u0026rsquo;ll also throw in a bit of configuration to teach Dovecot how to train spam detection through moving mail from the Inbox to Spam and the other way around. That part is slighly more complex because it relies on Sieve which is not the best piece of engineering, feel free to discard if you don\u0026rsquo;t care, I did without it for over ten years. It\u0026rsquo;s a nice-to-have, not a must-have.\nRequirements # The very first requirement is to figure out where you are going to run your mail server.\nWith widepsread permanent connections through DSL or FTTH, a home connection may be tempting but it is not a good idea as the IP address spaces of ISP are often blacklisted or suffer from a bad reputation to start with. In addition, many ISP prevent outgoing SMTP traffic to avoid compromised desktops from becoming spamming bots. I find the best option to be renting a dedicated server or a VPS from a hosting company after making sure that SMTP traffic is allowed there.\nI have rented my dedicated servers at online.net for the last twelve years and am very happy with them. You\u0026rsquo;ll even find on this blog instructions to run OpenBSD on their servers as they don\u0026rsquo;t support it natively. They do not filter SMTP, which is good because you can run an SMTP service right away, however the IP addresses may have been previously used by bad senders and you will want to test them. If the IP that is automatically allocated to you isn\u0026rsquo;t good, you can still order an additional one and pick it in a different range after checking that it isn\u0026rsquo;t in a blacklist, or that it doesn\u0026rsquo;t have a bad reputation already.\nAlternatively, for the purpose of creating my own commercial mail services, I started building an infrastructure on vultr.com (that\u0026rsquo;s a referal link). I haven\u0026rsquo;t been there for long so I might still change my mind, but so far I\u0026rsquo;m very happy with it. They filter SMTP by default so you have to open a ticket to explain what you intend to do with mail. When I explained that I didn\u0026rsquo;t intend to become an ESP but rather provide hosting services, they were helpful and unfiltered the same day.\nIt does not matter much where you decide to run your mail server, but what you need to check is that:\nthe host doesn\u0026rsquo;t have a history of hosting spammers and allows you to do SMTP you have an IP address that is dedicated to your mail you have control of the reverse DNS for that IP address your IP address isn\u0026rsquo;t already in blacklists As long as these requirements are met, you should be fine.\nIt is a good plan to prepare for incidents and this is done by using a backup mail server to take over when the primary one is down. I won\u0026rsquo;t cover this as it\u0026rsquo;s not complex, just a matter of an additional DNS record and an additional mail server routing traffic to the primary one, you\u0026rsquo;ll figure this out by yourselves. HOWEVER, a lot of people mentioned that you can get blacklisted and lose mail, so I think now is the right time to recommend that you don\u0026rsquo;t host your different mail servers at the same place. You don\u0026rsquo;t want the two of them to go down if there\u0026rsquo;s a power or network shortage in your datacenter, and you also want to ensure that if your IP range gets blacklisted as collateral damage from an evil neighbor, the IP of your backup mail server is \u0026ldquo;far enough\u0026rdquo; to not be blacklisted too. This way, should your primary mail server be temporarily blocked from sending, it can reroute traffic through the backup mail server until the issue is solved.\nI have endured collateral damage on my own setup only once during the last ten years. It was annoying for sure because incidents are never fun, but I rerouted traffic to my secondary server to unbreak traffic, filled the form to delist my IP while providing proof that I was a legitimate sender, rerouted traffic back when issue was resolved. It was nowhere near the big deal people make out of it.\nNo stress for people who plan ahead.\nThe technical stack # I will spin a new VPS at vultr.com (that\u0026rsquo;s still a referal link), install the latest OpenBSD snapshot because that\u0026rsquo;s how I roll (this will not be covered in this article), and build my mail system on top of it. I will assume OpenBSD throughout the rest of my writing but besides system-specific commands to install packages the configuration should be similar from a system to another. If you\u0026rsquo;re tall enough to run a mail server, you should be tall enough to adapt pathnames accordingly or \u0026hellip; just use OpenBSD already !\nFor the SMTP layer, which is in charge of exchanging messages between hosts disregarding what users will use to access their mail, I will be using the OpenSMTPD software. It is the default SMTP daemon for the OpenBSD operating system and has a portable version which is available on Github with instructions on how to build. These instructions are for the 6.6.0 version or later, they will not work on earlier versions.\nFor the IMAP layer, which will allow users to retrieve messages they received and access them from their smartphones or webmails, I will use the latest version of Dovecot. I will also use the Dovecot-Pigeonhole package, which will allow us to train the antispam solution into learning Ham from Spam. If you intend to only use a console client over ssh, such as mutt for example, you can skip that part as OpenSMTPD can deliver in a local mailbox that mails clients can access directly. In this article, we setup IMAP because we acually want to read mails from a smartphone as that\u0026rsquo;s what the regular people do.\nFinally, for the spam filtering layer, I will use the excellent Rspamd daemon which is far more than just an antispam solution. It provides state of the art spam filtering methods, but also provides antivirus integration, dkim signing, and a ton of modules which you can chose to use or not to fine tune your setup. In this article, we\u0026rsquo;ll simply use it for its DKIM-signing and spam filtering features which is the bare minimum we need.\nMaking myself reachable # SMTP is very tighly coupled with DNS and other hosts rely on DNS lookups to find out which machine handle mail for your domain. This is done through the lookup of MX (Mail eXchanger) records, so the most minimal thing to do for SMTP to work is to declare an MX record for your domain. For hypno.cat, the zone contains the following:\n;; an A (and AAAA record for IPv6) record is declared to name your mail server mail.hypno.cat A 217.69.8.253 mail.hypno.cat AAAA 2001:19f0:6801:867:5400:01ff:fee7:7af7 ;; an MX record is declared to let the world know that mail.hypno.cat handles mail for hypno.cat ;; 0 is the highest preference, if we had a backup MX we\u0026#39;d set a higher number on it, ;; let\u0026#39;s keep that for later, shall we ? hypno.cat. MX 0 mail.hypno.cat. ;;hypno.cat. MX 10 mail-backup.hypno.cat. I can verify that everything is fine using the host and dig utilities to check that the mail server name resolves and that the MX lookup returns the mail server name:\n$ host mail.hypno.cat mail.hypno.cat has address 217.69.8.253 mail.hypno.cat has IPv6 address 2001:19f0:6801:867:5400:1ff:fee7:7af7 $ dig -t MX hypno.cat +short 0 mail.hypno.cat. At this point and with these DNS records, other mail servers can already lookup which MX is responsible for hypno.cat, and contact it when they want to send mail to any address for that domain.\nMaking myself look all fine and dandy # That\u0026rsquo;s not enough because nowadays you MUST have a reverse DNS (rDNS) AND Forward-Confirmed rDNS (FCrDNS).\nThere\u0026rsquo;s a reason for that. A large number of the hosts spamming the world are compromised machines, with a large share of them being home computers behind residential connections. Many of these residential connections don\u0026rsquo;t have rDNS, don\u0026rsquo;t have FCrDNS, or have an rDNS that matches a dynamically allocated IP pattern (ie: 123.123.123.123.dyn.adsl.example.com). Because they are individually compromised machines, and not regular servers under the spammers control, rDNS and FCrDNS can\u0026rsquo;t be configured to look nice\u0026hellip; so it became a proof-of-work to configure your rDNS and FCrDNS correctly while ensuring that it doesn\u0026rsquo;t look like a dynamically allocated IP pattern. In some of the Big Mailer Corps guidelines this is explicitely stated, for others you just discover that the hard way. Either way, it\u0026rsquo;s a BARE MINIMUM, make sure that your hostname looks like a REAL mail server (ie: mail.hypno.cat, not www.hypno.cat).\nrDNS is usually out of your control because it\u0026rsquo;s managed in a somewhat special \u0026ldquo;arpa\u0026rdquo; zone owned by the IP address owner. ISP usually don\u0026rsquo;t let you configure it but server providers can\u0026rsquo;t realistically expect their customers not to, so they generally provide a small form somewhere in their control panels to let you provide the rDNS you want for the IP address they allocated you:\nIf I setup my rDNS to be the same as the forward records I configured above, then I automatically pass the FCrDNS test. This can be easily verified by looking up the rDNS for an IP address:\n$ host 217.69.8.253 253.8.69.217.in-addr.arpa domain name pointer mail.hypno.cat. $ host 2001:19f0:6801:867:5400:1ff:fee7:7af7 7.f.a.7.7.e.e.f.f.f.1.0.0.0.4.5.7.6.8.0.1.0.8.6.0.f.9.1.1.0.0.2.ip6.arpa domain name pointer mail.hypno.cat. Then looking up the IP address for that rDNS:\n$ host mail.hypno.cat mail.hypno.cat has address 217.69.8.253 mail.hypno.cat has IPv6 address 2001:19f0:6801:867:5400:1ff:fee7:7af7 If they match both ways, then everything\u0026rsquo;s fine.\nAdvertise which machines are allowed to send mail on behalf of my domain # SPF is a mechanism which makes it possible for destination mail servers to determine if a machine was allowed to send mail on behalf of a domain.\nHow it works is very simple, basically you add a DNS record to the zone for your domain stating which servers will emit mails. When a destination mail server receives mail with a sender from your domain, it checks if the IP address of the server submitting the e-mail is part of the whitelist.\nSPF is not mandatory but it is one of the things that makes you look good. While it is only informative by default, not necessarily resulting in mails being junked if they fail the test, it does have an impact on computing the likeliness of a message being a spam when added to other factors. It is commonly accepted that lack of SPF has a very negative impact on the reputation of a domain, so given how simple it is to provide the record, there\u0026rsquo;s simply no excuse.\nThere are multiple ways to setup SPF, in this example I will simply set it up so ONLY my mail server can send mail on my behalf:\nhypno.cat. IN TXT \u0026#34;v=spf1 mx -all\u0026#34; Should hosts receive a mail with a sender @hypno.cat not originating from mail.hypno.cat, they will be able to assume, or even enforce, that it should be rejected.\nProve that I actually authorized the message # DKIM is an authentication mechanism by which you can cryptographically sign mails emitted by your mail server, proving that you saw them and took responsibility in letting them transit. Hosts receiving these mails can verify that you authorized them, by verifying the signature and detect possible forgery.\nLike SPF, this is informative by default and failing a DKIM signature or even lacking the mechanism doesn\u0026rsquo;t mean a mail can\u0026rsquo;t inbox, however it accounts in the bad points given to a sender and degrades the reputation. It is also commonly accepted that lack of DKIM has a very negative impact on the reputation of a domain, so again, just do it.\nIt is slightly more complex than SPF because, although it also relies on DNS, for DKIM you have to generate a keypair in order to publish the public key in a DNS record.\nWe\u0026rsquo;re in 2019, I will be generating a 1024 bits RSA key, I know.\n\u0026mdash; BEGIN DIGRESSION \u0026mdash;\n2048 bits RSA public keys do not fit in a DNS TXT record and must be truncated into two records, but not everyone is able to cope with these truncated keys so you may lose DKIM benefits by going 2048 if hosts fail to use your (partial) public key and assume you\u0026rsquo;ve failed signing. Given that we\u0026rsquo;re not looking into security more than trying to look good at this point, we\u0026rsquo;ll just look the other way while excusing the fact that there\u0026rsquo;s a very moderate risk of impersonation when used in conjunction with SPF. This is very bad thinking but DKIM also has other short-comings and given that virtually everyone does 1024 bits RSA, I suggest we argue about the 1024 vs \u0026gt;= 2048 key size when we\u0026rsquo;re already successful at exchanging mails with Big Mailer Corps, but I\u0026rsquo;ll just take a moment to stress out, without pointing fingers at any, that even some security companies that actually work in the mail industry have issues with 2048 bits RSA.\nCompanies that actually provide sender reputation scoring. Oh, the, irony.\n\u0026mdash; END DIGRESSION \u0026mdash;\nIf this genuinely worries you, you can still experiment truncating key into two records and see if it suits you, adapting my example is trivial, just make sure you have a TXT record with a part of the key and another TXT record with the remaining.\nI\u0026rsquo;ll create a directory to hold the keys:\n# mkdir /etc/mail/dkim Then, the following commands will generate the keypair and extract the public key out of the private key:\n# openssl genrsa -out /etc/mail/dkim/hypno.cat.key 1024 Generating RSA private key, 1024 bit long modulus ..............................++++++ ...............++++++ e is 65537 (0x10001) # openssl rsa -in /etc/mail/dkim/hypno.cat.key -pubout -out /etc/mail/dkim/hypno.cat.pub writing RSA key # cat /etc/mail/dkim/hypno.cat.pub -----BEGIN PUBLIC KEY----- MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDThHqiM610nwN1nmV8OMc7PaPO uJWVGRDz5bWj4cRjTTmQYjJQd02xrydRNrRhjEKBm2mMDArNWjoM3jvN04ZifqJx DmKr7X8jsYi+MPEHz6wZxB8mayDK6glYTCyx//zl1luUdvm26PutA38K8cgnb7iT kfVP2OqK6sHAdXjnowIDAQAB -----END PUBLIC KEY----- Finally, I can create the DNS TXT record by extracting the public key out of the armor delimiters and formatting the content so it displays as follows:\n20190913._domainkey.hypno.cat.\tIN TXT \u0026#34;v=DKIM1;k=rsa;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDThHqiM610nwN1nmV8OMc7PaPOuJWVGRDz5bWj4cRjTTmQYjJQd02xrydRNrRhjEKBm2mMDArNWjoM3jvN04ZifqJxDmKr7X8jsYi+MPEHz6wZxB8mayDK6glYTCyx//zl1luUdvm26PutA38K8cgnb7iTkfVP2OqK6sHAdXjnowIDAQAB;\u0026#34; The name of the record should be constructed using this pattern: \u0026lt;selector\u0026gt;._domainkey.\u0026lt;domain\u0026gt;., where \u0026lt;selector\u0026gt; is a name you chose and which allows multiple keys to coexist. I like to use the date at which I generated the keypair, but whatever you chose, write it down for later as we\u0026rsquo;ll need it when configuring DKIM signing.\nAlso, note that the public key doesn\u0026rsquo;t have to be kept in the directory once it\u0026rsquo;s published in DNS. It can always be retrieved from the private key should it be needed for some reason, however I like keeping it around for fast reference. And make sure the private key isn\u0026rsquo;t world-readable if you have local users on your system, for some reason OpenSSL thinks private keys should be created rw-r--r--.\nInstructing hosts what to do with SPF and DKIM failures with DMARC # SPF and DKIM are both fine, but if they\u0026rsquo;re informative then it also mean they don\u0026rsquo;t prevent abuses of your domain name.\nDMARC allows instructing a destination mail server what to do with senders that fail SPF and DKIM tests. Most notably it allows instructing them to reject such senders.\nThere\u0026rsquo;s no clear indicator that providing a reject policy brings any positive points to a domain over not doing anything, however there are multiple hints that having a DMARC record has a positive impact over having none, even if that record states that nothing should be done about SPF and DKIM failures.\nThe format of DMARC records goes beyond this article and you\u0026rsquo;ll find multiple examples in any search engine, but a very simple record will simply state a policy (p=) of none (reject if you want to be harsh, quarantine if you need more time to decide). The percentage field (pct=) declares how many of these mails should be subjected to the DMARC policy, and the Reporting URI of Aggregated field (rua=) is where you should receive DMARC reports should you want to analyze them (before switching from quarantine to reject for example).\nI\u0026rsquo;ll just setup the most dummy record, one that will make all Big Mailer Corps see that we care about DMARC even though we don\u0026rsquo;t know what to do with it:\n_dmarc.hypno.cat. IN TXT \u0026#34;v=DMARC1;p=none;pct=100;rua=mailto:postmaster@hypno.cat;\u0026#34; Getting yourself a certificate for TLS # There are multiple ways you can obtain a TLS certificate, and assuming that you are already familiar with hosting other services, I\u0026rsquo;ll just pretend that you already know how to obtain one from your registrar or from letsencrypt.\nIf you know how to deal with your certificates, you can just skip this section and go to the next one.\nSince I\u0026rsquo;m really setting up mail.hypno.cat, I won\u0026rsquo;t be able to continue this article without obtaining a certificate, so as a courtesy to OpenBSD users I\u0026rsquo;ll document how I will generate mine on a brand new OpenBSD install. acme-client is a utility that lets you request and renew certificates from the console. It relies on an HTTP challenge, so it needs to be able to write to a directory that is served over HTTP, but luckily OpenBSD also ships with an httpd daemon that we can use for the challenge. Because we\u0026rsquo;re EXTRA lucky, it also provides an example configuration file that we can use with almost no change.\nSimply copy the example configuration from /etc/example/httpd.conf to /etc/httpd.conf, replace example.com with mail.hypno.cat and remove the tls block since we only care about serving the challenge. The file should now read as follow:\nserver \u0026#34;mail.hypno.cat\u0026#34; { listen on * port 80 location \u0026#34;/.well-known/acme-challenge/*\u0026#34; { root \u0026#34;/acme\u0026#34; request strip 2 } location * { block return 302 \u0026#34;https://$HTTP_HOST$REQUEST_URI\u0026#34; } } The httpd daemon can be started with the following command:\n# rcctl -f start httpd httpd(ok) The acme-client itself is straighforward to configure, again simply copy the example configuration from /etc/example/acme-client.conf to /etc/acme-client.conf, replace example.com with mail.hypno.cat and you should end up with a block like this:\ndomain mail.hypno.cat { domain key \u0026#34;/etc/ssl/private/mail.hypno.cat.key\u0026#34; domain full chain certificate \u0026#34;/etc/ssl/mail.hypno.cat.fullchain.pem\u0026#34; sign with letsencrypt } at this point you can simply run acme-client on the domain:\n# acme-client -v mail.hypno.cat [...] acme-client: /etc/ssl/mail.hypno.cat.fullchain.pem: created The certificate and keys are created at the appropriate place, we\u0026rsquo;ll later adjust paths in OpenSMTPD and Dovecot to point to the proper files.\nYou can keep the httpd running for future renewals and call acme-client from a cron, you can shut it down and deal with renewals in another way. You sort out how you want to renew with the help of the acme-client(1) man page.\nInstalling and configuring Rspamd # On OpenBSD, Rspamd is packaged and can be installed with a single command. We also need to install Redis which is used to store Rspamd statistics and greylisting states (among other things), as well as filter-rspamd which is the piece of code that allows OpenSMTPD to work with Rspamd. Both of them are also packaged so they can be installed with a single command too.\nAs I\u0026rsquo;m writing this, filter-rspamd will only be available in the next OpenBSD release due in October, so if you want to play with it before then, you can simply obtain it from github and read the README instructing how to build. Both Redis and Rspamd are already available in OpenBSD packages so the instructions below are already valid for them.\nFirst, I\u0026rsquo;ll install these packages:\nmail$ doas pkg_add redis rspamd opensmtpd-filter-rspamd [...] redis-4.0.14: ok rspamd-1.9.0: ok opensmtpd-filter-rspamd-0.1.1: ok The following new rcscripts were installed: /etc/rc.d/redis /etc/rc.d/rspamd See rcctl(8) for details. Rspamd is highly configurable so if you want to do funky things, it\u0026rsquo;s up to you to go read the extensive documentation. For the purpose of this article I\u0026rsquo;ll keep it very basic, as is the case on my own machines.\nI could edit the configuration in /etc/rspamd/actions.conf to adjust junking thresholds to my liking, but the defaults suit me fine so this was just to show them to you:\nactions { reject = 15; # Reject when reaching this score add_header = 6; # Add header when reaching this score greylist = 4; # Apply greylisting when reaching this score (will emit `soft reject action`) [...] And what I really, really want is for Rspamd to handle my DKIM signing because this is one of the requirements to not look bad to the world.\nThis is done by providing the configuration to match the DKIM key I generated earlier in /etc/rspamd/local.d/dkim_signing.conf:\n# mkdir /etc/rspamd/local.d # cat \u0026lt;\u0026lt; EOF \u0026gt;/etc/rspamd/local.d/dkim_signing.conf allow_username_mismatch = true; domain { hypno.cat { path = \u0026#34;/etc/mail/dkim/hypno.cat.key\u0026#34;; selector = \u0026#34;20190913\u0026#34;; } } EOF The allow_username_mismatch configuration is needed here because Rspamd expects usernames to contain the domain name, but in this setup OpenSMTPD authenticates against simple usernames. Also make sure the dkim key file is readable by members of the _rspamd group.\nWe\u0026rsquo;re done here, Rspamd and Redis can be enabled so OpenBSD starts them at next reboot:\n# rcctl enable redis # rcctl enable rspamd And I can start them right away so we don\u0026rsquo;t have to wait until next reboot:\n# rcctl start redis redis(ok) # rcctl start rspamd rspamd(ok) Configuring OpenSMTPD # OpenSMTPD is installed by default on OpenBSD so there is no installing phase here. If you run a different operating system, either someone has packaged it there and you can install using your package manager, or you can build from source and install by getting code from Github mirror and following build instructions.\nAs stated earlier, these instructions are only valid for the 6.6.0 release and later, earlier versions don\u0026rsquo;t support filters and some other features described here.\nThe default configuration for OpenSMTPD is suitable for local mail server, not accepting connections from the outside, but able to let local users exchange messages between themselves and to a remote host. It looks like this:\n# $OpenBSD: smtpd.conf,v 1.11 2018/06/04 21:10:58 jmc Exp $ # This is the smtpd server system-wide configuration file. # See smtpd.conf(5) for more information. table aliases file:/etc/mail/aliases # To accept external mail, replace with: listen on all # listen on lo0 action \u0026#34;local_mail\u0026#34; mbox alias \u0026lt;aliases\u0026gt; action \u0026#34;outbound\u0026#34; relay # Uncomment the following to accept external mail for domain \u0026#34;example.org\u0026#34; # # match from any for domain \u0026#34;example.org\u0026#34; action \u0026#34;local_mail\u0026#34; match for local action \u0026#34;local_mail\u0026#34; match for any action \u0026#34;outbound\u0026#34; It uses sane defaults as to make it hard to misconfigure, so if you\u0026rsquo;re worried about the last line (match for any) turning your server into an open-relay, don\u0026rsquo;t worry too much as it has an implicit (from local). You would have to very explicitely tell your server to match from any for any in order to let spammers abuse you.\nI initially wanted to update the configuration file progressively to hold your hand, but this article grew a lot since my first version. I\u0026rsquo;ll just put the complete 16 configuration lines configuration file and comment it extensively afterwards:\npki mail.hypno.cat cert \u0026#34;/etc/ssl/mail.hypno.cat.fullchain.pem\u0026#34; pki mail.hypno.cat key \u0026#34;/etc/ssl/private/mail.hypno.cat.key\u0026#34; filter check_dyndns phase connect match rdns regex { \u0026#39;.*\\.dyn\\..*\u0026#39;, \u0026#39;.*\\.dsl\\..*\u0026#39; } \\ disconnect \u0026#34;550 no residential connections\u0026#34; filter check_rdns phase connect match !rdns \\ disconnect \u0026#34;550 no rDNS is so 80s\u0026#34; filter check_fcrdns phase connect match !fcrdns \\ disconnect \u0026#34;550 no FCrDNS is so 80s\u0026#34; filter senderscore \\ proc-exec \u0026#34;filter-senderscore -blockBelow 10 -junkBelow 70 -slowFactor 5000\u0026#34; filter rspamd proc-exec \u0026#34;filter-rspamd\u0026#34; table aliases file:/etc/mail/aliases listen on all tls pki mail.hypno.cat \\ filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } listen on all port submission tls-require pki mail.hypno.cat auth filter rspamd action \u0026#34;local_mail\u0026#34; maildir junk alias \u0026lt;aliases\u0026gt; action \u0026#34;outbound\u0026#34; relay helo mail.hypno.cat match from any for domain \u0026#34;hypno.cat\u0026#34; action \u0026#34;local_mail\u0026#34; match for local action \u0026#34;local_mail\u0026#34; match from any auth for any action \u0026#34;outbound\u0026#34; match for any action \u0026#34;outbound\u0026#34; That\u0026rsquo;s all is needed for our complete SMTP setup.\nThe first two pki lines declare that mail.hypno.cat will use the certificate and key we generated earlier for TLS.\nThe filter lines apply a set of filters to incoming connections, check_dyndns will filter if rDNS matches some patterns, check_rdns will filter if rDNS is missing, check_fcrdns will filter if FCrDNS is missing, these are builtin filters in OpenSMTPD that you can tweak to filter at different phases or using other criterias.\nsenderscore is a custom filter that you can install either through your package manager (pkg_add opensmtpd-filter-senderscore on OpenBSD), or obtain from Github. It is not mandatory but I find it particularly useful and will explain why in the next section. In this configuration, it will reject a sender if its senderscore is below 10, junk message (add X-Spam header) if score is below 70, and for my own personal pleasure, add a delay to each response that is proportional to how bad the reputation is so bad senders are slowed down.\nrspamd is also a custom filter that we installed in the previous section. There\u0026rsquo;s no configuration as it is fully controlled from the Rspamd daemon responses, all we need to do is declare it.\nIf you want to be more conservative and not reject mails to avoid false positives, you can assign a junk action rather than disconnect action to builtin filters and remove the -blockBelow option in the senderscore filter:\nfilter check_dyndns phase connect match rdns regex { \u0026#39;.*\\.dyn\\..*\u0026#39;, \u0026#39;.*\\.dsl\\..*\u0026#39; } junk filter check_rdns phase connect match !rdns junk filter check_fcrdns phase connect match !fcrdns junk filter senderscore proc-exec \u0026#34;filter-senderscore -junkBelow 70 -slowFactor 5000\u0026#34; This way instead of rejecting sessions, OpenSMTPD will simply junk them so they are in your Spam folder. By monitoring what ends in your junk folder, you can gain confidence and tune settings to match what you really want.\nThese simple filters, not even counting rspamd, are enough to cut most of my Spam turning hundreds of daily spam into a handful. The rdns regex one can be tuned as depending on your country, you may see patterns that are particularly repetitive and that would never appear in mine. To ease maintenance, you can store them in a file, one by line and use the following construct:\ntable \u0026lt;dyndns\u0026gt; file:/etc/mail/path-to-my-regex-file filter check_dyndns phase connect match rdns regex \u0026lt;dyndns\u0026gt; junk Let\u0026rsquo;s continue dissecting the configuration.\nThe table aliases points to a file with\u0026hellip; aliases in two columns. The left column has the user-part of an e-mail address you receive, the right column has the user to which it should be delivered (ie: root: gilles) It is recommended that you have aliases mapping root, abuse and postmaster to an account that you actually read. I didn\u0026rsquo;t do that because I\u0026rsquo;m going to trash the install after this article, but do it.\nThen comes the listen lines. The first one is the public endpoint that other mail servers can contact to send mails to hypno.cat, it is configured to offer tls using the certificate and key for mail.hypno.cat and filter incoming sessions through all my filters. The second one is the endpoint for my own users, on the submission port, it requires tls using the same certificates, requires auth using system users (the default, can be configured otherwise), and filters through the rspamd filter only so we can DKIM-sign our own messages.\nThe action lines define an action for local mail which is to deliver to a maildir, while classifying junk in a specific folder and resolving aliases, the other action is to relay mail and let other host know that we\u0026rsquo;re mail.hypno.cat when we greet ourselves. If a server had multiple IP addresses the relay action should also specify the source address to use (with src) to make sure the proper IP address is used, here I only have one IP address and it is used for mail so I don\u0026rsquo;t have to.\nFinally, the match lines are the ruleset. When an envelope enters the SMTP server through one of the listen endpoints, it is compared to each match line one after another until a matching action in found or the envelope is rejected. The conditions are fairly explicit so I won\u0026rsquo;t describe these four lines, you should be able to figure out by yourself what from any for domain \u0026quot;hypno.cat\u0026quot; means otherwise you shouldn\u0026rsquo;t be reading this in the first place.\nA few words about SenderScore # SenderScore is an IP reputation database which associates a score ranging between 0 and 100 to an IP address. The score is tied to the volume and behavior observed from these IP addresses, and while we can\u0026rsquo;t really know HOW these scores are computed because the methodology is not public, a bit of studying allows confirming that they didn\u0026rsquo;t make these numbers out of nowhere: they do correlate with delivery and error rates at various Big Mailer Corps and it is possible to influence the scoring with changes of behaviors that also correlate to changes in delivery and error rates.\nIt is also VERY obvious that to build their reputation score they have access to SMTP session informations that only a destination mail server can have, such as the volume of mails observed from an IP address, or the ratio of failed vs accepted recipients, and since the scoring correlates with delivery and error rates in my observations, this implies that they get this information from Big Mailer Corps. Do what you want with that bit of information.\nBased on MY understanding, it is a very interesting indicator to limit the amount of spam hitting you. I don\u0026rsquo;t trust it to be accurate when it comes to good reputations (because I kinda know how to work a bit around) but I do trust it to be HIGHLY accurate when it comes to bad reputations, because it takes a certain amount of doing bad things in high volumes to see a scoring degrade. If you don\u0026rsquo;t do bulk mailing, you should definitely either be unknown to them or have a scoring around 95 or above, otherwise you\u0026rsquo;re very likely doing something wrong.\nNot everyone is convinced about SenderScore and some delivery experts claim it\u0026rsquo;s bullshit. I personally ran months-long experiments graphing daily reputation and volumes, then comparing them to delivery graphs, and my opinion is quite the opposite: to me there is a very significant correlation that definitely helps classify senders. I think the best approach is to use the filter for junking, not blocking, and determining by yourself if you\u0026rsquo;re happy with it.\nSenderScore considers that hosts should have a reputation above 70, I would personally assume hosts below 80 to be good junking candidates, and hosts below 10 to be obvious rejects.\nInstalling and configuring Dovecot # OpenBSD also has a package for Dovecot which can be installed with a simple command:\nmail# pkg_add dovecot dovecot-2.3.7.2v0: ok The following new rcscripts were installed: /etc/rc.d/dovecot See rcctl(8) for details. New and changed readme(s): /usr/local/share/doc/pkg-readmes/dovecot [...] And, specifically for OpenBSD, following instructions from the readme, the /etc/login.conf file should contain a dovecot class to bump the resources allowed as Dovecot is very hungry for file descriptors:\ndovecot:\\ :openfiles-cur=1024:\\ :openfiles-max=2048:\\ :tc=daemon: Once that is done, there\u0026rsquo;s really not much to do on the Dovecot side except have it point to the proper TLS certificate we generated earlier.\nI do that by tweaking the ssl_cert and ssl_key configuration keys in the /etc/dovecot/conf.d/10-ssl.conf file so they read as follows:\nssl_cert = \u0026lt;/etc/ssl/mail.hypno.cat.fullchain.pem ssl_key = \u0026lt;/etc/ssl/private/mail.hypno.cat.key Then Dovecot must be told that mails should be looked up in the user ~/Maildir directory as this is where OpenSMTPD drops files. I do that by tweaking the mail_location configuration key in the /etc/dovecot/conf.d/10-mail.conf file so it reads as follows:\nmail_location = maildir:~/Maildir We\u0026rsquo;re all set, we can enable the daemon so it\u0026rsquo;s started at next boot, and start it right away:\nmail# rcctl enable dovecot mail# rcctl start dovecot dovecot(ok) At this point, you can already configure any mail client like mutt, thunderbird or even the gmail app on Android, so that it uses mail.hypno.cat both for incoming and outgoing mails.\nTeaching Dovecot to train Rspamd # You read until here, good.\nSo this is a bonus section, one that\u0026rsquo;s absolutely not mandatory in setting up your mail server, but one that I\u0026rsquo;d like to add because it lets users train the spam filtering the same way they\u0026rsquo;re used to already: moving a mail to spam, marking a mail as spam or reporting a mail as spam, whichever it is called on their interface. The good news is that it integrates with IMAP, so no matter if they use their smartphone app, a client on their desktop or a webmail, the action of reporting a spam will automatically train the filter.\nHow you do that is relatively simple in theory, it consists of plugging a couple scripts, one to handle moving a mail from Inbox to Spam and one to handle moving mail from Spam to elsewhere. In pratice, Big Mailer Corps filters also detect that you delete a mail without opening it and such, but let\u0026rsquo;s not get too excited and start with something that\u0026rsquo;s already good enough. How you do that in practice is a bit more complex, and that is because we will rely on Sieve which is\u0026hellip; how do you put it with diplomacy \u0026hellip; \u0026ldquo;overengineered\u0026rdquo;.\nGiven that this is not a critical piece of the setup, you are allowed to shut your brain off and follow blindly instructions. Worst than can happen at this point is that it will not train anything.\nTo enable imap sieve, the Pigeonhole package must be installed, and as has been the case throughout this article all I need is a simple command:\nmail# pkg_add dovecot-pigeonhole dovecot-pigeonhole-0.5.7.2v1: ok [...] And\u0026hellip; here comes the tricky part, you need to configure Dovecot to enable imap_sieve, and teach it what sieve scripts to use for what actions.\nFirst, I will need to enable imap_sieve in Dovecot by adding it to the mail plugins for IMAP. This is done by tweaking the mail_plugins configuration key in /etc/dovecot/conf.d/20-imap.conf:\nprotocol imap { [...] mail_plugins = $mail_plugins imap_sieve [...] } Then to teach Dovecot how to train, I\u0026rsquo;ll add the following bit to /etc/dovecot/conf.d/90-plugin.conf, telling it that there are sieve scripts to trigger when mail gets moved in and out of Junk mailbox:\nplugin { sieve_plugins = sieve_imapsieve sieve_extprograms sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment imapsieve_mailbox1_name = Junk imapsieve_mailbox1_causes = COPY APPEND imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve imapsieve_mailbox2_name = * imapsieve_mailbox2_from = Junk imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve imapsieve_mailbox3_name = Inbox imapsieve_mailbox3_causes = APPEND imapsieve_mailbox3_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve } Now that dovecot is ready, I\u0026rsquo;ll have to prepare the sieve part which relies on two sieve scripts to train spam and ham inside /usr/local/lib/dovecot/sieve:\n# cat report-ham.sieve require [\u0026#34;vnd.dovecot.pipe\u0026#34;, \u0026#34;copy\u0026#34;, \u0026#34;imapsieve\u0026#34;, \u0026#34;environment\u0026#34;, \u0026#34;variables\u0026#34;]; if environment :matches \u0026#34;imap.mailbox\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;mailbox\u0026#34; \u0026#34;${1}\u0026#34;; } if string \u0026#34;${mailbox}\u0026#34; \u0026#34;Trash\u0026#34; { stop; } if environment :matches \u0026#34;imap.user\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;username\u0026#34; \u0026#34;${1}\u0026#34;; } pipe :copy \u0026#34;sa-learn-ham.sh\u0026#34; [ \u0026#34;${username}\u0026#34; ]; # cat report-spam.sieve require [\u0026#34;vnd.dovecot.pipe\u0026#34;, \u0026#34;copy\u0026#34;, \u0026#34;imapsieve\u0026#34;, \u0026#34;environment\u0026#34;, \u0026#34;variables\u0026#34;]; if environment :matches \u0026#34;imap.user\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;username\u0026#34; \u0026#34;${1}\u0026#34;; } pipe :copy \u0026#34;sa-learn-spam.sh\u0026#34; [ \u0026#34;${username}\u0026#34; ]; Because both sieve scripts rely on shell scripts sa-learn-ham.sh and sa-learn-spam.sh, I also need to create these shell scripts in /usr/local/lib/dovecot/sieve:\n# cat sa-learn-ham.sh #!/bin/sh exec /usr/local/bin/rspamc -d \u0026#34;${1}\u0026#34; learn_ham # cat sa-learn-spam.sh #!/bin/sh exec /usr/local/bin/rspamc -d \u0026#34;${1}\u0026#34; learn_spam And finally, I need to compile the sieve scripts and make the shell scripts executable so that Dovecot can actually use them:\n# sievec report-ham.sieve # sievec report-spam.sieve # chmod 755 sa-learn-ham.sh # chmod 755 sa-learn-spam.sh That\u0026rsquo;s all, now moving mails from a folder to another I can see the following lines in the /var/log/rspamd/rspamd.log file:\n2019-09-13 23:59:46 #18598(controller) \u0026lt;1d44bd\u0026gt;; csession; rspamd_controller_learn_fin_task: \u0026lt;127.0.0.1\u0026gt; learned message as ham: CAHPtQbOxQxBCsVd7nUCP4podu74Pa-F6k28z+4BWfNeeqWYiAg@mail.gmail.com [...] 2019-09-14 00:01:57 #18598(controller) \u0026lt;b76e28\u0026gt;; csession; rspamd_controller_learn_fin_task: \u0026lt;127.0.0.1\u0026gt; learned message as spam: CAHPtQbOxQxBCsVd7nUCP4podu74Pa-F6k28z+4BWfNeeqWYiAg@mail.gmail.com [...] I\u0026rsquo;m sure these can be simplified, I\u0026rsquo;ll be honest and say that I don\u0026rsquo;t really care because it\u0026rsquo;s just a side-feature, I\u0026rsquo;m unlikely to touch the sieve scripts for the next decade, I don\u0026rsquo;t think anyone should touch sieve scripts, we should just come with something better that doesn\u0026rsquo;t remind me of m4 wankery.\nTesting it all # And now, we\u0026rsquo;ll just test that we can do a round-trip from hypno.cat to gmail.com.\nFirst, I\u0026rsquo;ll craft a mail from my hypno.cat account to my gmail.com account:\nAfter sending it, I\u0026rsquo;ll check it arrives at gmail.com:\nThen, I\u0026rsquo;ll check that it was sent over a TLS-secured channel:\nI\u0026rsquo;ll check that gmail.com is happy with our SPF declaration, our DKIM signature and that it saw we had a DMARC record. You do that by selecting \u0026ldquo;Show original\u0026rdquo; in the menu associated to each mail:\nAnd finally, I\u0026rsquo;ll reply to the mail just to be sure that it works both ways:\nYay, we\u0026rsquo;re done # So this article is dense, I wanted to explain why we do things, if you remove all my blabber and focus on the purely technical aspects you\u0026rsquo;ll realize that we built a working mail infra, providing TLS inbound and outbound services, with DKIM, SPF and DMARC compliant outgoing traffic, and spam-filtered incoming traffic. And we did this by:\nadding a couple A records, an MX record and three TXT records to our DNS zone making sure rDNS was properly set installing TLS certificates building a ~15 lines configuration file for an SMTP server modifying 3 lines in the default configuration of an IMAP server modifying 8 lines in the default configuration of an antispam solution And because we were in a very good mood and willing to take an extra step to help users train antispam, we implemented ham and spam training of Rspamd by:\nmodifying approximately 20 lines in the default configuration of an IMAP server copying 4 sieve scripts in a directory Agreed, it may take a new comer some efforts to come up with these without help, but none of these tasks qualify as hard or tricky. In addition, pretty much all of them are only one-time operations, you don\u0026rsquo;t edit the DNS zone or reset your rDNS every two days, just like you don\u0026rsquo;t touch your working configuration once in place unless you have a new use-case. It may happen that you have to do a maintenance, whatever that means, but I don\u0026rsquo;t see a scenario where such a maintenance would imply having to do \u0026ldquo;all\u0026rdquo; of that again.\nThat being said, after you do it a few times, it takes about ten minutes to replicate this setup from scratch as you\u0026rsquo;ll know what you\u0026rsquo;re doing, because\u0026hellip; you know, mail is not hard.\nSo what should you do next ? # What you should do next is setup redundancy to ensure that a failure of your mail server does not causes mail loss.\nIn practice, most mail exchangers will retry if their destination is down, so even if you endure a downtime, most of your mails will be buffered by your sender and resumed when your server is back.\nIn theory, you still want to do things right, and letting others buffer your own messages to cope with your downtimes is fairly impolite as they bear responsibility of your messages far longer than they should\u0026hellip; You should investigate how to setup a backup mail exchanger, arrange with friends you trust to backup each other, or even subscribe to a service that will offer backup mail to you. Anything as long as you know a secondary server will be able to cover for you if you have a downtime.\nThen, no matter which way you chose to do it, it will boil down to:\nadding an additional MX record to the DNS zone configuring a backup mail server so it forwards mail for your domains to your mail server Rocket, science.\nAre we done ? # I will write a serie of articles to discuss reputation, as well as some of the mechanisms in place at Big Mailer Corps that can cause people to fail delivery.\nIf these topics are of interest and I get positive feedback, I might just write more often about deliverability concepts, otherwise I\u0026rsquo;ll just resume what I was doing before: writing about my monthly opensource contributions. It\u0026rsquo;s up to you :-)\nFinally, in this article I have described a very simple setup but there are tons of interesting stuff you can do with mail infrastructures if you\u0026rsquo;re not scared of working a bit on it.\nI\u0026rsquo;m currently building an infrastructure for a future mail hosting service which spans over multiple data centers in multiple countries, which has a very high tolerance to failures, has backup mail servers to cope with very extreme failures of primary mail servers, can very easily scale to bursts of volumes, that decorrelates incoming and outgoing traffic so it can provide partial services to people, that can easily reroute traffic across machines to work around failures, while providing IMAP services to virtual accounts on a multitude of domains, all with daily backups of all mail directories. The infrastructure currently costs me\u0026hellip; less than 75 EUR a month. It is also fairly simple with most machines being default installs of OpenBSD with very few changes to the base system.\nIf reading about how you can build such setups is of any interest, I could also write about it to help people build MORE hosting alternatives which is ultimately what I hope for.\nThanks for reading me, I still don\u0026rsquo;t have a soundcloud, however I still have a patreon if you want to support me, or use the widgets right below the comment link to share this article on social medias.\nIf you find errors in this article, please let me know so I can fix them, I started writing this article late and continued half-way through the night so I might have let errors slip in despite multiple readings.\n","date":"14 September 2019","permalink":"/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/","section":"Posts","summary":"TL;DR: - NO TL;DR: this time, I spent hours writing, you should spend minutes reading. - OK... I explain in WAY TOO MUCH details how to setup a mail server EDIT (2019-10-26) # OpenSMTPD 6.","title":"Setting up a mail server with OpenSMTPD, Dovecot and Rspamd"},{"content":" TL;DR: - Mail is not hard: people keep repeating that because they read it, not because they tried it - Big Mailer Corps are quite happy with that myth, it keeps their userbase growing - Big Mailer Corps control a large percentage of the e-mail address space which is good for none of us - It's ok that people have their e-mails hosted at Big Mailer Corps as long as there's enough people outside too EDIT (2019-12-15) # A practical guide to set up a mail exchanger was published on this blog.\nDisclaimer # THIS IS FOR SYSADMINS WITH TECH KNOWLEDGE, WHO KNOW HOW TO HOST SERVICES. Self-hosting mail is not HARD but requires WORK, which are two different things. Setting up a mail infrastructures requires a lot of initial work, then basic long term maintenance.\nI work on an opensource SMTP server. I build both opensource and proprietary solutions related to mail. I will likely open a commercial mail service next year.\nIn this article, I will voluntarily use the term mail because it is vague enough to encompass protocols and software. This is not a very technical article and I don\u0026rsquo;t want to dive into protocols, I want people who have never worked with mail to understand all of it.\nI will also not explain how I achieve the tasks I describe as easy. I want this article to be about the \u0026ldquo;mail is hard\u0026rdquo; myth, disregarding what technical solution you use to implement it. I want people who read this to go read about Postfix, Notqmail, Exim and OpenSMTPD, and not go directly to OpenSMTPD because I provided examples.\nI will write a follow-up article, this time focusing on how I do things with OpenSMTPD. If people write similar articles for other solutions, please forward them to me and I\u0026rsquo;ll link some of them. it will be updated as time passes by to reflect changes in the ecosystem, come back and check again over time.\nFinally, the name Big Mailer Corps represents the major e-mail providers. I\u0026rsquo;m not targeting a specific one, you can basically replace Big Mailer Corps anywhere in this text with the name of any provider that holds several hundred of millions of recipient addresses. Keep in mind that some Big Mailer Corps allow hosting under your own domain name, so when I mention the e-mail address space, if you own a domain but it is hosted by a Big Mailer Corp, your domain and all e-mail addresses below your domain are part of their address space.\nOnce upon a time, the \u0026ldquo;mail is hard\u0026rdquo; myth # When you first look into becoming independant with your e-mails, as soon as you ask in a public tech medium for help setting up a \u0026ldquo;mail\u0026rdquo; server, people will invariably jump in the discussion to discourage you from attempting because \u0026ldquo;mail is hard\u0026rdquo;.\nNot only \u0026ldquo;mail is hard\u0026rdquo; but it also seems that \u0026ldquo;Big Mailer Corps have already won\u0026rdquo;, that \u0026ldquo;all mail you send will end up in your recipients\u0026rsquo; spam box\u0026rdquo;, and that \u0026ldquo;you will be flooded by spammers\u0026rdquo; who will \u0026ldquo;abuse your server to relay spam to the world\u0026rdquo;.\nWow, that\u0026rsquo;s overwhelming :-|\nYou just wanted to send and receive mail because it seemed like a good idea, and now this turned into the worst life decision. But is it ?\nSoftware is hard # Mail used to be hard. A long long time ago.\nIt was hard because software were hard to setup correctly. A mail server like Sendmail required a Ph.D in compilers theory to operate, and the elitist culture of postmasters who could read Sendmail\u0026rsquo;s so-called \u0026ldquo;configuration files\u0026rdquo; didn\u0026rsquo;t help create a user-friendly environment. Postmasters bragging about how hard Sendmail was, while disclosing their unconditional love for m4 was a pissing contest. I can easily imagine these people whipping themselves with fresh nettles as a hobby.\nPostfix is an alternative to Sendmail that has been around since 1998. While I wouldn\u0026rsquo;t consider it as \u0026ldquo;simple\u0026rdquo; by any stretch of the word, it is orders of magnitude simpler than Sendmail. With the help of a search engine, new comers will easily find tutorials to spot the three or four configuration options they need to tweak.\nOpenSMTPD is another alternative which was first released in 2013. It is also orders of magnitude simpler than Sendmail. The configuration reads almost as plain english and a usable configuration file can actually fit \u0026hellip; in a tweet.\n[full screen](/images/2019-08-30-tweet.png) I don\u0026rsquo;t have experience in other contenders, but most operating systems and distributions provide multiple alternatives, pre-packaged so you can install and run with a simple command.\nMail software is NOT hard. It was if you stopped looking in the 90\u0026rsquo;s.\nOK, software isn\u0026rsquo;t hard but dealing with SPAM is # The myth goes on by saying that mail is hard because as soon as you run your mail server, spammers are going to come in hordes and dealing with spam will become a daily nightmare. This could not be further from the truth.\nNow that we\u0026rsquo;ve established trust, I\u0026rsquo;m not going to lie to you: there are hordes of spammers.\nWhen you plug your server in to the internet, you\u0026rsquo;ll start seeing connections from random sources trying to get mail accross. You\u0026rsquo;ll see them coming from home connections, from remote countries, from IP addresses sharing the same range, there will be no end to how much amazement this will procure.\nBasically, they all fit in two categories:\nclients trying to abuse your server to use it as a relay to send spam to the world clients trying to send spam to you after having obtained your e-mail address somehow The ones from the first category are easy to deal with: IGNORE THEM. They search for misconfigured servers and try doing things that will be rejected on a properly configured server, or even try to authenticate with a dictionnary attack which is not going to succeed if you have good passwords. They are like mosquitoes on a summer evening, annoying but\u0026hellip; meh.\nc1a89cb774083905 smtp connected address=185.234.219.64 host=\u0026lt;unknown\u0026gt; c1a89cb774083905 smtp failed-command command=\u0026#34;AUTH LOGIN\u0026#34; result=\u0026#34;503 5.5.1 Invalid command: Command not supported\u0026#34; c1a89cb774083905 smtp disconnected reason=disconnect c1a89cb8c5b84cbf smtp connected address=193.32.160.143 host=\u0026lt;unknown\u0026gt; c1a89cb8c5b84cbf smtp bad-input result=\u0026#34;500 5.5.1 Invalid command: Pipelining not supported\u0026#34; c1a89cb8c5b84cbf smtp disconnected reason=quit c1a89cb9441966e7 smtp connected address=185.234.219.193 host=\u0026lt;unknown\u0026gt; c1a89cb9441966e7 smtp failed-command command=\u0026#34;AUTH LOGIN\u0026#34; result=\u0026#34;503 5.5.1 Invalid command: Command not supported\u0026#34; c1a89cb9441966e7 smtp disconnected reason=disconnect If they really, really bother you or you dislike logs full of such attempts, write a script that detects such patterns in logs and add them to your firewall. Out of pure laziness, I have never ever used scripts to deal with these in the twenty years I operated mail servers and I\u0026rsquo;m still here to talk about it. As long as you don\u0026rsquo;t see them succeeding anything, you can just disregard these.\nThe ones from the second category are slighly more annoying because if you ignore them, your mailbox becomes full of spam. Luckily, they are not so hard to filter through several simple means and it is very easy to reduce spam to a few, every now and then, properly classified in a Spam folder (also known as \u0026ldquo;junked\u0026rdquo; mails). I take absolutely no precaution hiding my e-mail address, gilles@poolp.org, and I sometimes get one or two spam e-mails per day in the junk folder. Not only is that not a daily nightmare, but it\u0026rsquo;s less than what I actually receive on my own Big Mailer Corps account, which I do not share as easily and which has an average of three to five daily junked mails.\nNote that some very simple filters you\u0026rsquo;d apply for the second category of spammers, are HIGHLY effective to also kill the spammers from the first category:\na56dece24dcac3d2 smtp failed-command command=\u0026#34;DATA\u0026#34; result=\u0026#34;550 message rejected\u0026#34; a56ded3dd6cda8c2 smtp failed-command command=\u0026#34;\u0026#34; result=\u0026#34;550 your IP reputation is too low for this MX\u0026#34; a56ded3f7db5b96c smtp failed-command command=\u0026#34;DATA\u0026#34; result=\u0026#34;550 message rejected\u0026#34; a56dec6ffdb2caef smtp failed-command command=\u0026#34;\u0026#34; result=\u0026#34;421 you must have rDNS to contact this MX\u0026#34; a56dec895475b9bf smtp failed-command command=\u0026#34;\u0026#34; result=\u0026#34;421 you must have FCrDNS to contact this MX\u0026#34; I\u0026rsquo;ll just state it as it is: You will never reach \u0026ldquo;absolute 0 spam\u0026rdquo;, it was proven mathematically in the 2000s, but the amount you\u0026rsquo;ll receive while self-hosted can be as low or lower as what you receive at Big Mailer Corps. Spam is not more of an issue self-hosted, no matter how much the marketing tries to tell you otherwise.\nTo illustrate this I emptied the Spam folder on my poolp.org account and my Big Mailer Corps account this morning, here\u0026rsquo;s the screenshot I took of both accounts tonight, poolp.org on the left and Big Mailer Corps on the right. Neither of them have spam in Inbox.\n[full screen](/images/2019-08-30-spambox.png) OK SPAM is not the issue but my mails will not reach my users at Big Mailer Corps # Another myth, with slightly more substance this time, is that sending e-mail to an address at a Big Mailer Corp will result in the e-mail being rejected or junked.\nLet me tell you a secret: Big Mailer Corps are not worried about you but are worried about big senders harassing their users. They do not care about your personal server sending a few mails, even if its in the thousands per months. What they care about is the infected computers or compromised servers flooding their users. What they care about are the marketing companies that are literally shitting over them, sending individually millions of commercial mails per day, trying to work-around spam filters, and that sometimes manage to go for a while without being rejected. Unless you are sending hundreds of thousands of mails to them on a daily basis, quite frankly and without trying to hurt your feelings, you fall waaaaaaaaaaaaaaay below the radars.\n[full screen](/images/2019-08-30-bigmailercorps.png) So why did I say this claim has more substance than the others ?\nBig Mailer Corps have introduced proof-of-work into mail exchanges, voluntarily or not.\nUnlike legitimate senders who want to reach a specific user, spammers want to reach a ton of users whoever they are, because statistically some of them are going to fall for whatever it is that they\u0026rsquo;re trying to sell. Among these spammers, we\u0026rsquo;ll also include marketing companies that buy lists of users from partners, sending indiscriminately to \u0026ldquo;activate\u0026rdquo; them in hope of reaching some percentage of openings. They don\u0026rsquo;t care who receives an e-mail as long as it opens, because you know, statistics. So the harder it is for them, the higher the chance they\u0026rsquo;ll switch to another target because it\u0026rsquo;s the only way to not fall behind statistically. Don\u0026rsquo;t think I\u0026rsquo;m making this up, you\u0026rsquo;d be amazed at what tricks and hacks are used to mechanically increase openings by a few percents, including lowering the number of recipients at a particular destination and increasing at another.\nIn opposition to this are legitimate users who can\u0026rsquo;t just switch to another target, they want to reach a specific recipient and, if work is needed to achieve that, well\u0026hellip; you got to do what you got to do. They\u0026rsquo;ll pour in the work needed to make it work.\nKnowing this, Big Mailer Corps came with sets of rules about what a Good Sender should do to be able to communicate with them. These lists are basically a proof of work: they do not guarantee that you\u0026rsquo;ll be able to send to them, they do not guarantee that you\u0026rsquo;ll hit the inbox and will not be junked, but they are the minimal set of things you should do to prove you actually care. And considering some spammers do their best to look good, if you don\u0026rsquo;t do it yourself it basically means you\u0026rsquo;re not even willing to do better than spammers.\nThese rules are not only here to annoy you, they are also very effective: some of the rules cannot be achieved for spammers who use compromised hosts, for example. This proof of work paradigm is annoying because it raises (in time) the cost of entry, but it can also be leveraged by others to kill spam. Since Good Senders most definitely want to be able to send to Big Mailer Corps, if you receive connections from clients that didn\u0026rsquo;t do the minimum to deliver there, you can shut them down yourself because they\u0026rsquo;re already cut from a big portion of the e-mail address space.\nThis is why I wrote this has more substance than the other claims. It\u0026rsquo;s true that IF you don\u0026rsquo;t even try to do the minimum work, THEN you\u0026rsquo;ll start with a penalty.\nIn practice, a notion of reputation is also into play. Some people don\u0026rsquo;t even try but they fall but fall sooooooo far below the radars\u0026hellip; that even without trying they\u0026rsquo;ll manage to send e-mails without issues. I often send mail to my Big Mailer Corp account from my development laptop, far from being properly configured, and they almost always reach inbox. A good reputation allows you to make some mistakes and go through without respecting all rules, while bad reputation increases the amount of work you need to do. Considering that a good reputation is earned from doing things right, you get the general idea: do things right.\nThe minimum set rules is VERY FAR from being hard. To quote someone on twitter:\n[full screen](/images/2019-08-30-tweet_2.png) I would add a few things to that list but this highlights that some people are already \u0026ldquo;golden\u0026rdquo;, just by setting up proper rDNS, SPF and DKIM. That last bit about handling incoming spam, we\u0026rsquo;ve already discussed it above ;-)\nOK, then why is everyone saying it\u0026rsquo;s hard ? # The first reason, is because no one claims otherwise and, since Big Mailer Corps benefits from this situation, they\u0026rsquo;re not going to contradict it either. Big Mailer Corps BENEFIT from the myth that mail is hard as this means more people rely on them, they control more of the e-mail address space, and this translates to more e-mails being analyzed for targeted advertisement. The more people are discouraged, the more people will eventually subscribe to their services, and since they already control a large share, they can make mail slighly more difficult by making their requirements higher (harder, not hard). This is not something they do in some kind of conspiracy, this is just the result of them obtaining more power because people stay away from self hosting.\nAnother reason is because it used to be hard a long time ago. People got traumatized by how hard it was to not screw up and never reevaluated the situation. Some people today genuinely discourage other people from running their mail server, citing the very real difficulties they faced over a decade ago, far before some of today\u0026rsquo;s tools even existed.\nAnd finally, another reason is that people keep repeating it without actually trying themselves. I know this for a fact because people have been telling me that mail is hard for the last ten years, and for the last ten years I asked what they found hard in order to try improving the situation. A VAST majority of these people confessed that they never actually tried: they read or heard that mail was hard, often from a source they trusted, then accepted that claim and started telling others that mail was hard. We can\u0026rsquo;t really blame them when that myth has been around for so long, if I had no previous knowledge of mail and did a quick search today to find out how to setup my server, I might just decide not to do it given how difficult it seems from reading others.\nWhat do we do from now ? # We need to reclaim mail. I\u0026rsquo;m not saying people shouldn\u0026rsquo;t be hosted at Big Mailer Corps, but these should not become the Pavlovian reaction to \u0026ldquo;where do I get an e-mail address ?\u0026rdquo;.\nAs long as there are enough mail hosts out there, Big Mailer Corps HAVE to remain friendly because their users can complain that their legitimate mail is junked, which results in a risk of them leaving for elsewhere, less e-mails, less targeted advertisement, less $$$. If the number of mail hosts shrinks to the point that trafic not coming from Big Mailer Corps is irrelevant, then it becomes their protocols and they can start making up rules that are not sustainable by anyone but them, because no one will notice when the insignificant number of e-mails not coming from there is junked. Again, not a conspiracy but a side effect of being the few relevant actors of a system: why bother abiding to standards that work for everyone and ensuring any sender can reach them\u0026hellip; if most of the trafic comes from a handful of actors.\nThis is already happening with one specifically requiring mails to be sent from another Big Mailer Corp to hit the inbox, or requiring that senders be added to the contacts for others. Any other sender will hit spambox unconditionnally for a while before being eventually upgraded to inbox.\nWe can\u0026rsquo;t let that happen: allowing e-mail to be fully controlled by a small set of cooperating multi-million users hosts is just accepting to be screwed.\nIt is therefore very important that we don\u0026rsquo;t let the myth propagate further. Our best interest is to have a WIDE variety of mail hosts and providers, small and big, commercial and not. We must not allow the number of mail hosts to shrink, they must increase so the e-mail address space out of the control of Big Mailer Corps remains significant.\nAnd by all means, we must not push everyone to use Big Mailer Corps, particularly because a lot of people simply read their e-mail from a smartphone and don\u0026rsquo;t see a difference in interface, so they could be using pretty much any provider and be just as happy.\nI hope my point gets accross, feel free to share this wherever you want and point people to this article.\nIn a few days I\u0026rsquo;ll publish a practical description of how to setup a host similar to mine, providing spam protection for incoming mail, and basic proof of work to make most Big Mailer Corps happy.\nIf you like my work, support me on patreon !\n","date":"30 August 2019","permalink":"/posts/2019-08-30/you-should-not-run-your-mail-server-because-mail-is-hard/","section":"Posts","summary":"TL;DR: - Mail is not hard: people keep repeating that because they read it, not because they tried it - Big Mailer Corps are quite happy with that myth, it keeps their userbase growing - Big Mailer Corps control a large percentage of the e-mail address space which is good for none of us - It's ok that people have their e-mails hosted at Big Mailer Corps as long as there's enough people outside too EDIT (2019-12-15) # A practical guide to set up a mail exchanger was published on this blog.","title":"You should not run your mail server because mail is hard"},{"content":" TL;DR: - small inprovements to the fion window manager - plakar is a backup utility I wrote a long time ago that I will share - tons of opensmtpd stuff, mostly filters and issues handling Shout outs to my patrons ! # As has become the habit, this report begins with a big thank you to my patrons, cited by contribution then alphabetical order.\nThis month has been sponsored by:\nJ. Derrick Wesley Mouedine Assaby Mischa Peters Diego Meseguer Edgar Pettijohn Jdelic Sean Geoff Hill Thomas Bleader Raton Nick Ryan Vegar Linge Halaand Igor Zinovik Paul Kelly Jan J C I have recently switched to a 75% part-time schedule at work so that I can spend a \u0026ldquo;free\u0026rdquo; week each month working on my own stuff, mostly opensource, without any kind of pressure: no one knows what I\u0026rsquo;ll be working on and no one but me gets to decide how I\u0026rsquo;ll spend this time. This comes at the cost of slashing a quarter of my wage, which is sustainable but not ideal, so while most of my \u0026ldquo;free\u0026rdquo; weeks will be spent on opensource, I\u0026rsquo;ll be doing some sponsored development or short contracts to cover some of the loss if needed.\nI opened a patreon account so that people who care about my work can sponsor it, allowing me to spend most (if not all) of these \u0026ldquo;free\u0026rdquo; weeks publishing code for the community. If you want me to spend more time doing this then you know what to do: become my patron !\nSo thanks again to my patrons who make this possible, I will list them in all of my monthly reports, they are the sponsors of the work I describe below.\nFLOSS Weekly #543 # This week, hosts Randal Schwartz and Jonathan Bennett invited me to discuss SMTP and OpenSMTPD in their show, FLOSS Weekly.\nThere\u0026rsquo;s not much to say besides that it was a pleasant experience, that I really enjoyed talking to them about OpenBSD and OpenSMTPD, and that you should go watch that episode and subscribe to the show :-)\nFion # I didn\u0026rsquo;t work much on the fion window manager, it\u0026rsquo;s obviously less of a priority to me than upcoming\u0026rsquo;s October OpenSMTPD release. Yet, I still managed to get a couple things in.\nI started by introducing a notion of \u0026ldquo;keyboard\u0026rdquo; mode to ease interactions with the window manager:\nFor debugging purposes, I had hardcoded a few keys as triggers for window management actions: w would create a workspace, d would destroy it, n and p would switch to next and previous ones. Then I worked on tiles, so I reassigned the n and p keys to switch between tiles. You get the idea, at some point you have to be able to use both workspaces and tiles at the same time :-)\nKeyboard modes allow some combinations of keys win+X to enter a mode specific to a fion concept. For instance, win+w enters workspace mode so the next key pressed applies to workspaces, win+t enters tile mode so the next key pressed applies to tiles. This makes it easier for me to remember the shortcuts because n becomes next in whatever mode you are, p becomes previous in whatever mode you are, etc\u0026hellip;\nThen, I worked on fixing one of the issues on the TODO because if I fix at least one issue each month, I might just be able to switch to fion by the end of the year :-p\nFion displays a status bar at the top of the screen which provides informations regarding the time, the current workspace and active tile. The status bar is updated by a call to a function called layout_update() which\u0026hellip; updates the layout on all screens. However, because I was learning XCB, I used the function xcb_wait_for_event() in my event loop, which caused fion to block between events and the layout_update() function to only be called when an event wakes up the loop. This meant that seconds in the displayed time would hang, then jump by several, it made things look laggy.\nI needed to switch to a multiplexed model and luckily, for once, it was not tricky to find how to do it as there\u0026rsquo;s a xcb_get_file_descriptor() function providing a descriptor to the connection. All it took was some rearranging of the event loop, using poll() to wake up on events happening on the connection, and a timeout to act as a \u0026rsquo;tick\u0026rsquo; so the layout can be updated disregarding if events occured or not.\nI committed these improvements, my next task will be to handle the superposition of tabs inside tiles so we can have multiple X clients sharing the same tile, then I\u0026rsquo;ll deal with killing tiles and resizing siblings, which is the main show stopper for switching to fion as far as i\u0026rsquo;m concerned.\nPlakar # I wasn\u0026rsquo;t sure if I was going to mention this yet but it\u0026rsquo;s useful enough.\nI\u0026rsquo;ve been self-hosted since ~1999 and along the years I\u0026rsquo;ve dealt with backups in various ways. I went from tar, to dump/restore, to incremental dump/restore, to rsync, to incremental rsync, and started writing my own tools for different use-cases I had.\nAs I\u0026rsquo;m doing some cleanup of my private repositories I found the following ones:\ndrwxr-xr-x 2 gilles gilles 512 Aug 3 2012 backup drwxr-xr-x 2 gilles gilles 512 Jul 23 2012 backuptool drwxr-xr-x 6 gilles gilles 512 Apr 1 2015 plakar Both backup and backuptool are going to bite the dust, they were basically experiments to come up with a one-file backup format, providing dedup and versionning of content. I won\u0026rsquo;t enter much details as they did work but were not that interesting and I didn\u0026rsquo;t use them for long, proof that they weren\u0026rsquo;t itching my scratch.\nOn another hand, plakar is far more interesting.\nIt is a utility that created a repository in ~/.plakar, and would then let you snapshot a filesystem, splitting each file into content-defined chunks and storing them into the repository. The chunks would be compressed and encrypted, hard-link games would ensure that the repository can maintain multiple snapshots of the same directories at virtually no cost, encryption would allow the repository to be pushed to a remote server since I tend to keep backups at two sites and on my google drive.\nAt this point, some of you will ask \u0026ldquo;Isn\u0026rsquo;t that what tarsnap do ?\u0026rdquo;, to which I\u0026rsquo;ll answer \u0026ldquo;It seems to but I don\u0026rsquo;t really know as I haven\u0026rsquo;t used it and tarsnap seems not to allow self-hosting\u0026rdquo;.\nAnyways, that repository contains a version written in Python which supports various operations: push to save a snapshot into the repository, pull to restore a snapshot, remove to kill a snapshot, list to list snapshots, diff to diff two snapshots and search to search for files and directories inside snapshots. In addition, operations can be done on portions of a snapshot, not requiring the extraction of an entire snapshot when only a few files or directories are needed. I had started an UI which allowed plakar to display a randomized password and launch a local webserver expecting that password, a web interface would then allow visualizing the snapshots and their contents.\nlaptop$ plakar list laptop$ plakar push ~/wip/OpenSMTPD snapshot ee30bf0b4a4f44b5bfe71c632424c4d3: dirs=47, files=426, errors=0 laptop$ plakar list 2019-08-25-05:34:01 ee30bf0b4a4f44b5bfe71c632424c4d3 dcnt=47 fcnt=426 size=28.4MB laptop$ plakar pull ee30 laptop$ ls ee30bf0b4a4f44b5bfe71c632424c4d3/home/gilles/wip/OpenSMTPD/ README THANKS regress smtpd smtpscript laptop$ ls /home/gilles/.plakar chunks pending purge resources snapshots laptop$ plakar ui To use UI, go to http://127.0.0.1:8080, and use password: 2f1f1b61350fdfbb Bottle v0.12.17 server starting up (using WSGIRefServer())... Listening on http://127.0.0.1:8080/ Hit Ctrl-C to quit. [full screen](2019-08-25-plakar_1.png) [full screen](2019-08-25-plakar_2.png) [full screen](2019-08-25-plakar_3.png) [full screen](2019-08-25-plakar_4.png) I don\u0026rsquo;t intend to release the Python code as I will not maintain it, however to learn Golang I rewrote plakar in Go this week-end and I have a version that works. It\u0026rsquo;s not yet finished but the push/pull/list/trash commands are working, I\u0026rsquo;m going to start using it on my machines and see how it goes for the next few weeks before I publish the code on Github.\nlaptop$ plakar list laptop$ plakar push ~/wip/OpenSMTPD 2b5bec66-1799-48a1-8418-42509347eac1 laptop$ plakar list 2b5bec66-1799-48a1-8418-42509347eac1 2019-08-25 05:38:58.912591955 +0200 CEST laptop$ plakar pull 2b5 Restoring directories: 48/48 Restoring files: 426/426 laptop$ plakar trash 2b5 laptop$ plakar list laptop$ laptop$ ls 2b5bec66-1799-48a1-8418-42509347eac1/home/gilles/wip/OpenSMTPD/ README THANKS regress smtpd smtpscript laptop$ ls /home/gilles/.plakar chunks objects purge snapshots transactions laptop$ OpenSMTPD # On the OpenSMTPD front, I\u0026rsquo;ve done quite a lot of work so I will only mention the big stuff.\nErrata published # A user reported a bug in OpenSMTPD causing the daemon to hit one of its own sanity checks and call fatal(), resulting in a denial of service.\nThe bug affected OpenSMTPD 6.5 and prior, so I had to investigate and prepare patches for OpenBSD 6.5, OpenBSD 6.4 as well as OpenSMTPD-portable.\nProxy-v2 support # I committed experimental support for proxy-v2 protocol, based on a diff contributed by Antoine Kaufmann and reworked a bit.\nWhat this does is allow OpenSMTPD to work with SMTP sessions proxied by haproxy, for example. The session begins with a header which contains informations regarding the source address, the proxy layer in OpenSMTPD will use that information to rewrite a struct sockaddr_storage for the SMTP engine.\nI have ran the code for my own testing but it is experimental because I have only done light testing, we need feedback from power users.\nHandled a ton of Github issues # We had been lagging a bit behind issues on Github, many of them being either easily fixable, already resolved or user mistakes.\nI took the time to go through about twenty of them, resolving half in a way or another and ensuring the other half is on the right track.\nSeveral issues are actually dependant of some projects we have, so they are stuck until we\u0026rsquo;re done with a project and will be closed in batches once this is done. This is for example the case for several tickets related to TLS which will not be handled until we complete the switch to libtls\u0026hellip; which provides out of the box support for these features or makes them trivial.\nfilter-rspamd # I have pre-released version 0.1.0 of my filter-rspamd on Github.\nIt has been running since last month on my machines without a single issue, so I guess it is fairly usable.\nI also committed a port to OpenBSD so that it\u0026rsquo;ll be possible to pkg_add filter-rspamd.\nfilter-senderscore # I have also pre-released version 0.1.0 of my filter-senderscore on Github.\nWhat it does is query the SenderScore DNS to obtain the reputation of a sender IP address, then allow blocking, flagging as Spam and/or applying a reputation-dependant delay to sessions.\nI committed a port to OpenBSD for that one too so it\u0026rsquo;ll be possible to pkg_add filter-senderscore.\nfilter-checksenderdomain # This one I don\u0026rsquo;t think I\u0026rsquo;ll release, but it\u0026rsquo;s available on Github.\nThe name should be quite obvious, it simply performs a DNS lookup on the MAIL FROM domain to check if it resolves. I wrote it to prove that some of our issues don\u0026rsquo;t belong in OpenSMTPD but can be solved as easily in a filter.\nsmtp-out reporting # I decided to postpone the smtp-out reporting because it requires a bit of rework on the IPC between SMTP engine and filtering layer, something not visible to people writing filters, but which I\u0026rsquo;m not comfortable doing at this point knowing that I may not be very available in September in case I break things.\nI will continue working on smtp-out reporting so it can get committed when OpenBSD 6.6 is out, until then I\u0026rsquo;ll work on stuff that are risk-free :-)\nWhat next ? # I will take next few weeks to stress test OpenSMTPD and ensure our next release is rock-solid, then I will enter some kind of code freeze until 6.6 is out because the changeset is already significant.\nWhen I write my next report in late September, I may or may not yet have fork()-ed a little human in my house, no guarantees that the September report will be on time !\nThat being said, I\u0026rsquo;m curious if the format of these reports is readable, if you have suggestions feel free to let me know. Should I provide more/less details, be more/less technical ?\nI intend to write some technical articles regarding the internals of Plakar, since there may be bits of interest to other projects, including a noSQL private repo I still have sleeping out there ;-)\nAm I readable ? Let me know.\nIf you like my work, support me on patreon !\n","date":"25 August 2019","permalink":"/posts/2019-08-25/august-2019-report-fion-plakar-and-opensmtpd/","section":"Posts","summary":"TL;DR: - small inprovements to the fion window manager - plakar is a backup utility I wrote a long time ago that I will share - tons of opensmtpd stuff, mostly filters and issues handling Shout outs to my patrons !","title":"August 2019 report: Fion, Plakar and OpenSMTPD"},{"content":"","date":"25 August 2019","permalink":"/tags/plakar/","section":"Tags","summary":"","title":"plakar"},{"content":" TL;DR: - not much work outside of OpenSMTPD this week - OpenSMTPD portable builds with OpenSSL 1.1.x again - smtp-out reporting is working correctly on my laptop - wrote two filters that I'm actually using as you read this Shout outs to my patrons ! # As will become the tradition hopefully, this report begins with a big thank you to my patrons, cited by contribution then alphabetical order.\nThis month has been sponsored by:\nJ. Derrick Mischa Peters Diego Meseguer Vegar Linge Halaand Bleader Raton Nick Ryan C Igor Zinovik Jan J I have recently switched to a 75% part-time schedule at work so that I can spend a \u0026ldquo;free\u0026rdquo; week each month working on my own stuff, mostly opensource, without any kind of pressure: no one knows what I\u0026rsquo;ll be working on and no one but me gets to decide how I\u0026rsquo;ll spend this time. This comes at the cost of slashing a quarter of my wage, which is sustainable but not ideal, so while most of my \u0026ldquo;free\u0026rdquo; weeks will be spent on opensource, I\u0026rsquo;ll be doing some sponsored development or short contracts to cover some of the loss if needed.\nI opened a patreon account so that people who care about my work can sponsor it, allowing me to spend most (if not all) of these \u0026ldquo;free\u0026rdquo; weeks publishing code for the community. If you want me to spend more time doing this then you know what to do: become my patron !\nSo thanks again to my patrons who make this possible, I will list them in all of my monthly reports, they are the sponsors of the work I describe below.\nWhat\u0026rsquo;s in it for you ?\nNot much beyond the monthly public shout out, the satisfaction of seeing some projects make progress thanks to you, as well as my gratefulness for helping me spend more time on something I love.\nThe adventurous may even get early beta access to some code or private repository when stated in the report ;-)\nOpenSMTPD portable builds again with OpenSSL 1.1.x # As I explained in a previous report in May, API changes that occured in OpenSSL after the LibreSSL fork caused enough divergence that it became impossible to build the same code for both. Since OpenSMTPD is developed on OpenBSD which ships with LibreSSL, this resulted in failures to build with OpenSSL 1.1.x and me trying to unfuck the situation through a maze of #ifdefs until my sanity began melting.\nIn May, I brought a few functions from OpenSSL to LibreSSL to help reduce the divergence in OpenSMTPD scope and as a result, with minimal work, I have been able to build a working OpenSMTPD on an Ubuntu system with OpenSSL 1.1.x again.\n$ uname -srm Linux 5.0.0-15-generic x86_64 $ openssl version OpenSSL 1.1.1b 26 Feb 2019 $ sudo smtpd -d [sudo] password for gilles: info: OpenSMTPD 6.5.0-portable starting ^C When I announced that I would target LibreSSL and no longer OpenSSL, a lot of people assumed that I was breaking OpenSSL support voluntarily which isn\u0026rsquo;t accurate. I just did not fix a breakage that was already there: the same code could not build on LibreSSL, OpenSSL 1.0.x and 1.1.x, but while LibreSSL was my daily target and would always build, OpenSSL 1.0.x was still very widespread and OpenSSL 1.1.x was not yet packaged everywhere.\nEven though it was pleasing to no longer deal with the differences between the three, I did not enjoy the situation as it shrank our community considerably, reduced the number of contributions we received, forced people into switching to another MTA or use old versions of OpenSMTPD with old versions of OpenSSL\u0026hellip; It was possible to build LibreSSL and build an OpenSMTPD with LibreSSL everywhere, I did it myself on various systems, but the fact is there aren\u0026rsquo;t many systems that provide an OpenSMTPD package depending on LibreSSL. Either they stopped providing OpenSMTPD, they packaged it with a ton of unsupported patches to make it build with OpenSSL, or they took the easy path and had an old OpenSMTPD depend on OpenSSL 1.0.x, causing many people to run very old versions of our code\u0026hellip;\nSo what has changed since then ?\nOpenSSL no longer supports most of 1.0.x so it\u0026rsquo;s no longer as widespread as it used to, only 1.0.2 will be supported until the end of this year, most systems have already swiched to either LibreSSL or OpenSSL 1.1.x. With that and the fact that I committed changes in LibreSSL to reduce the gap with OpenSSL 1.1.x, the cost of supporting OpenSSL 1.1.x while keeping LibreSSL my daily target has become minimal.\nWe used to have #ifdefs in the code to cope with differences because it was VERY DIFFICULT to do otherwise, but now the code can assume LibreSSL and the OpenSSL differences are handled in the compat layer outside of OpenSMTPD code itself. I can look at the smtpd code without having to think \u0026ldquo;is this for LibreSSL or OpenSSL ?\u0026rdquo;.\nThe only #ifdef that has creeped in the code is wether or not ECDSA server certificates are supported, because the API is radically different, but in this case it simply disables it if a function from LibreSSL could not be found. This is very very manageable to me, I can live with that one ifdef.\nThese changes were all committed to the portable branch, so the next OpenSMTPD release in November will support OpenSSL 1.1.x again and there will be no excuse not to run a recent version of OpenSMTPD :-)\nReporting API gets extended # The reporting API, a mechanism by which OpenSMTPD can notify an external process of internal events in real time, has been extended considerably.\nAs a reminder, the API is a building block of filters as they can rely on it to construct an internal state for sessions, but it can also be used to build a wide range of external tools that rely on analyzing events. Improvements to the API can simplify considerably how filters are written, just as it can make possible writing new tools that couldn\u0026rsquo;t be written before.\nThe first extension to the reporting API is the link-auth event. It is generated whenever an SMTP authentication is requested by a client, and it records the username as well as the result of the authentication request. By registering for this event, it is possible to determine if a session was authenticated or not, track which sessions were issued by which user, etc\u0026hellip;\nreport|1|1564213979.757908|smtp-in|link-auth|63dc9f08477410de|gilles|pass report|1|1564214009.115878|smtp-in|link-auth|63dc9f06465b1cb2|gilles|fail report|1|1564214037.439016|smtp-in|link-auth|63dc9f0732384d8d|gilles|error While working on that, I made sure that OpenSMTPD would obfuscate parameters to any event it reports when in an authentication phase. This ensures that one can\u0026rsquo;t accidentally leak credentials to a third party filter, a database or a log file:\nreport|1|1562975948.315256|smtp-in|protocol-client|4772092f35819fed|AUTH LOGIN report|1|1562975948.315572|smtp-in|protocol-server|4772092f35819fed|334 VXNlcm5hbWU6 report|1|1562975961.559697|smtp-in|protocol-client|4772092f35819fed|******** report|1|1562975961.559724|smtp-in|protocol-server|4772092f35819fed|334 UGFzc3dvcmQ6 report|1|1562975974.600669|smtp-in|protocol-client|4772092f35819fed|******** report|1|1562975975.325244|smtp-in|protocol-server|4772092f35819fed|535 Authentication failed report|1|1563215271.737448|smtp-in|protocol-client|3138586bbd3c217b|AUTH PLAIN report|1|1563215271.737705|smtp-in|protocol-server|3138586bbd3c217b|334 report|1|1563215272.493300|smtp-in|protocol-client|3138586bbd3c217b|AUTH PLAIN ******** report|1|1563215273.000374|smtp-in|protocol-server|3138586bbd3c217b|535 Authentication failed The second extension is the introduction of a tx-reset event which is generated whenever a transaction is reset. A transaction encompasses a sender, one or multiple recipients as well as a message, so filters that deal with transaction will usually register multiple tx-* events to build a state with the informations they need. Since multiple transactions can occur in a session, there needs to be a way to clear the transaction data when it is no longer relevant. Until now, to do this filters had to register for events that initiate or terminate a transaction tx-begin, tx-commit and tx-rollback, to make sure that a cleanup function was called from each of these. With tx-reset, filters no longer need to register these events if they don\u0026rsquo;t intend to do anything with them, they can register a tx-reset handler to cleanup transaction with the guarantee that when a transaction is no longer relevant the event will be generated.\nreport|1|1564191006.311117|smtp-in|tx-commit|f6369c781fea802b|fac552ff|2806 report|1|1564191006.311120|smtp-in|tx-reset|f6369c781fea802b|fac552ff report|1|1564191100.066978|smtp-in|tx-rollback|f6369c8a16ddbcb8|8a4f9aad report|1|1564191100.066982|smtp-in|tx-reset|f6369c8a16ddbcb8|8a4f9aad Finally, until now the reporting API \u0026ldquo;only\u0026rdquo; generated events for smtp-in which is incoming SMTP trafic, but I have started teaching smtp-out, which is outgoing SMTP trafic, to also report all events. This work is not fully done but it works enough that I have deployed it on my own MX:\nreport|1|1564191030.737128|smtp-out|link-connect|f6369c7b41a82d01|localhost|pass|45.76.46.201:40938|127.0.0.1:25 report|1|1564191030.738602|smtp-in|link-connect|f6369c7cc2cb4dfe|out.mailbrix.mx|fail|45.76.46.201:40938|127.0.0.1:25 report|1|1564191038.043809|smtp-in|link-connect|f6369c7d5157f9d8|mail.mailbrix.mx|pass|95.179.226.54:39341|45.76.46.201:25 report|1|1564191052.947431|smtp-out|link-connect|f6369c85f7a894d6|localhost|pass|45.76.46.201:4502|127.0.0.1:25 link-auth and tx-reset have already been committed, the smtp-out work should be committed this month when I have built enough confidence it doesn\u0026rsquo;t break anything.\nHere comes my filter-rspamd # I had already written a proof of concept rspamd filter in Python a few months ago, but it was meant to prove the API allowed such filters, not to be usable for real so I didn\u0026rsquo;t really care about it being robust or not.\nMany people have asked me if I intended to improve and release it, so I decided to actually take time and write a proper filter-rspamd that I would use myself. I wrote the first lines of the filter yesterday early in the morning and by noon I had it deployed on my own MX. I decided to write it in Golang because I didn\u0026rsquo;t (and still don\u0026rsquo;t) know that language, so if I managed to write a useful filter in a language I don\u0026rsquo;t know in just a few hours I guess the goal for a simple and useful API is reached ;-)\nSo how does it work ?\nFirst the filter is built:\n$ cd filter-rspamd $ go build Then it is installed:\n$ doas cp filter-rspamd /usr/libexec $ The smtpd.conf file needs to be updated:\n[...] filter rspamd proc-exec \u0026#34;/usr/libexec/filter-rspamd\u0026#34; listen on all filter rspamd [...] And that\u0026rsquo;s all, what\u0026rsquo;s left to do is rspamd configuration outside of my scope :-)\nSince I\u0026rsquo;m not confident in my Golang skills, I wrote the filter in such a way that ANY error happening for pretty much any reason will flush message unprocessed as if the filter had been bypassed. No matter if you forgot to start rspamd or if I made an error that causes my filter to fail parsing rspamd response in some situations, any error will just output the mail as it was received: fail-safe by default.\nAs of today, the filter already does a lot of useful things.\nAdding X-Spam{,-Action,-Score} headers:\n$ cat /var/mail/gilles From gilles@laptop.home Fri Jul 26 08:54:35 2019 Return-Path: \u0026lt;gilles@laptop.home\u0026gt; Delivered-To: gilles@laptop.home X-Spam-Action: add header X-Spam: yes X-Spam-Score: 9 / 15 Received: from localhost (laptop.home [local]) by laptop.home (OpenSMTPD) with ESMTPA id f5918d8b for \u0026lt;gilles@laptop.home\u0026gt;; Fri, 26 Jul 2019 08:54:35 +0200 (CEST) From: \u0026lt;gilles@laptop.home\u0026gt; Date: Fri, 26 Jul 2019 08:54:35 +0200 (CEST) To: gilles@laptop.home Subject: test Message-ID: \u0026lt;04eabdcea50bc883@laptop.home\u0026gt; test Rewrite Subject line:\n$ cat /var/mail/gilles From gilles@laptop.home Sat Jul 27 10:43:34 2019 Return-Path: \u0026lt;gilles@laptop.home\u0026gt; Delivered-To: gilles@laptop.home X-Spam-Action: rewrite subject Received: from localhost (laptop.home [local]) by laptop.home (OpenSMTPD) with ESMTPA id 97b0006c for \u0026lt;gilles@laptop.home\u0026gt;; Sat, 27 Jul 2019 10:43:34 +0200 (CEST) From: \u0026lt;gilles@laptop.home\u0026gt; Date: Sat, 27 Jul 2019 10:43:34 +0200 (CEST) To: gilles@laptop.home Subject: *** SPAM *** test Message-ID: \u0026lt;7e41c7e558f77a3e@laptop.home\u0026gt; test Reject content, greylist a session or soft reject a transaction:\n$ echo \u0026#39;XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X\u0026#39; | mail -s test gilles 550 message rejected $ echo greylist-test | mail -s test gilles 421 Try again later $ echo softfail-test | mail -s test gilles 451 Try again later Finally, in addition to this, filter-rspamd supports adding the DKIM-signature header provided by rspamd if any, so by properly configuring rspamd to perform DKIM signing, e-mails will be automagically DKIM signed when filter-rspamd is installed. NO MORE FUCKING DKIMPROXY.\n[...] DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=poolp.org; s=opensmtpd; t=1564217527; h=from:from:reply-to:subject📅date:message-id:message-id:to:to:cc; bh=g3zLYH4xKxcPrHOD18z9YfpQcnk/GaJedfustWU5uGs=; b=bPRnicsNRhVi+SdeF6UYQw7ZfViIloc460vRm+5qljYgNtBKmYSnda0JBhSKm1Kryn8Zde hRShh6QPHChc2RtCsh33U1il6MSE12OmC6vnI6TxDa0t2hXbYFFihx/4UNkFe7J1MYnR4i 0hilMVclvtxknWwJf2sMIuVYFijl4b8= X-Spam-Action: add header X-Spam: yes X-Spam-Score: 6.3 / 15 Received: from smtp-out.poolp.org (out.mailbrix.mx [212.83.129.132]) by out.mailbrix.mx (OpenSMTPD) with ESMTP id a1b7fac1 for \u0026lt;gilles.chehade@gmail.com\u0026gt;; Sat, 27 Jul 2019 08:52:06 +0000 (UTC) Received: from localhost (poolp.org [local]) by poolp.org (OpenSMTPD) with ESMTPA id c097ac23 for \u0026lt;gilles.chehade@gmail.com\u0026gt;; Sat, 27 Jul 2019 10:52:06 +0200 (CEST) From: gilles \u0026lt;gilles@poolp.org\u0026gt; Date: Sat, 27 Jul 2019 10:52:06 +0200 (CEST) To: gilles.chehade@gmail.com Subject: test Message-ID: \u0026lt;63dca0f89904c9b1@poolp.org\u0026gt; test All of these behaviors and thresholds are controlled from the rspamd configuration, so filter-rspamd doesn\u0026rsquo;t actually have any configuration itself.\nThe filter is still a work in progress, so the code is not released yet ( patrons may get an early copy), but it is definitely useable as not only I\u0026rsquo;m running it on two MX but another OpenBSD hacker is also using it. I\u0026rsquo;ll likely release soon.\nHere also comes my filter-jsonlog # I worked for years at a company dealing with large volumes of mails and know the importance of analytics and dashboards. Until recently, the only way to push data for analytics was to parse the mail log, extract patterns of information from lines to push them into a database. The log format was designed to make this easier but it was not optimal as it was also meant to not overwhelm humans.\nPeople have managed to create OpenSMTPD dashboard using the mail log, but also through parsing the output of smtpctl show stats which both work but are respectively unfriendly and hackish. We can do better !\nWith reporting API came a format that was designed solely for applications, so the mail log format was simplified and trimmed from redundant informations, whereas the reporting API would publish a lot of informations humans don\u0026rsquo;t really care about when looking at logs.\nI wrote a filter-eventlog last month which would simply write the reporting stream to a file:\n[...] report|1|1564217526.405901|smtp-in|protocol-client|63dca0f760bad4af|MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; report|1|1564217526.406175|smtp-in|filter-response|63dca0f760bad4af|mail-from|proceed report|1|1564217526.408265|smtp-in|tx-begin|63dca0f760bad4af|c097ac23 report|1|1564217526.408278|smtp-in|tx-mail|63dca0f760bad4af|c097ac23|\u0026lt;gilles@poolp.org\u0026gt;|ok report|1|1564217526.408292|smtp-in|protocol-server|63dca0f760bad4af|250 2.0.0: Ok report|1|1564217526.408414|smtp-in|protocol-client|63dca0f760bad4af|RCPT TO:\u0026lt;gilles.chehade@gmail.com\u0026gt; report|1|1564217526.408630|smtp-in|filter-response|63dca0f760bad4af|rcpt-to|proceed report|1|1564217526.410380|smtp-in|tx-envelope|63dca0f760bad4af|c097ac23|c097ac2360bfa130 report|1|1564217526.410400|smtp-in|tx-rcpt|63dca0f760bad4af|c097ac23|\u0026lt;gilles.chehade@gmail.com\u0026gt;|ok report|1|1564217526.410417|smtp-in|protocol-server|63dca0f760bad4af|250 2.1.5 Destination address valid: Recipient ok [...] After discussing with a friend well versed in ELK he suggested I adopt a key-value format so it could be injected more easily, so I added a command line option so the filter would output in the following format:\n[...] timestamp=1564217526.405901 subsystem=smtp-in evt=protocol-client session=63dca0f760bad4af line=\u0026#34;MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; \u0026#34; timestamp=1564217526.406175 subsystem=smtp-in evt=filter-response session=63dca0f760bad4af phase=mail-from response=proceed timestamp=1564217526.408265 subsystem=smtp-in evt=tx-begin session=63dca0f760bad4af msgid=c097ac23 timestamp=1564217526.408278 subsystem=smtp-in evt=tx-mail session=63dca0f760bad4af msgid=c097ac23 address=\u0026lt;gilles@poolp.org\u0026gt; status=ok timestamp=1564217526.408292 subsystem=smtp-in evt=protocol-server session=63dca0f760bad4af line=\u0026#34;250 2.0.0: Ok\u0026#34; timestamp=1564217526.408414 subsystem=smtp-in evt=protocol-client session=63dca0f760bad4af line=\u0026#34;RCPT TO:\u0026lt;gilles.chehade@gmail.com\u0026gt; \u0026#34; timestamp=1564217526.408630 subsystem=smtp-in evt=filter-response session=63dca0f760bad4af phase=rcpt-to response=proceed timestamp=1564217526.410380 subsystem=smtp-in evt=tx-envelope session=63dca0f760bad4af msgid=c097ac23 evpid=c097ac2360bfa130 timestamp=1564217526.410400 subsystem=smtp-in evt=tx-rcpt session=63dca0f760bad4af msgid=c097ac23 address=\u0026lt;gilles.chehade@gmail.com\u0026gt; status=ok timestamp=1564217526.410417 subsystem=smtp-in evt=protocol-server session=63dca0f760bad4af line=\u0026#34;250 2.1.5 Destination address valid: Recipient ok\u0026#34; [...] This was better as I could basically pipe this output into elasticsearch without post-processing and see these events in a Kibana dashboard, however it only eased creating dashboards to compare different records for a similar event, a much much better format would be to dump the entire session state for each event which would allow displaying ANY information for ANY event.\nI wrote filter-jsonlog which is another golang filter that is slightly more complex than filter-eventlog. Instead of just reading events from the stream and outputing them in a different format, it actually builds an internal representation of the state of SMTP sessions as it receives events, and for each event it outputs the state of the correspond SMTP session in json format since that format is easily injectable in various databases.\nIf we take the previous example, while it would be very easy to create a piechart of top recipients because you\u0026rsquo;d be comparing data associated to a similar event, it would be very hard if not impossible to create a piechart of TLS vs non-TLS sessions for a specific recipient. With filter-jsonlog, because any event has its entire session state associated, visualzation can be created about ANY session information occuring at ANY event.\nThe output is logged to syslog so that syslog configuration can be used to determine in which file it should land, and newsyslog can be used to determine the rotation policy, while I use filebeat to inject the log to an elasticsearch node. The setup for this is really trivial :-)\nHere are two screenshots: full screen\n[full screen](2019-07-27-kibana.jpeg) Unfortunately I suck at creating dashboards myself, so if you want to help me get things in shape to improve the jsonlog output and provide dashboard templates for the community, get in touch with me !\nThis filter is also still a work in progress and the code is not released yet ( patrons may get an early copy), but it does work kinda.\nWhat about the Fion window manager ? # I haven\u0026rsquo;t talked about the Fion window manager because I didn\u0026rsquo;t spend much time on it, I was stuck trying to handle keyboard shortcuts correctly and I didn\u0026rsquo;t want to waste my week on this, however I got unstuck late last night so I will resume my work on it ;-)\nWhat next ? # The smtp-out reporting is very important to me so it needs to be completed and committed. I will also try to package filter-rspamd so it\u0026rsquo;s one pkg_add away from a fresh install. Hopefully people will help with filter-jsonlog and dashboards so I can move that faster than I would alone.\nThere\u0026rsquo;s still work needed in the filters area but it\u0026rsquo;s becoming good enough that I will start documenting, making it easier for people to actually start playing with it.\nIf you like my work, support me on patreon !\n","date":"27 July 2019","permalink":"/posts/2019-07-27/july-2019-report-tons-of-smtpd-work-mostly/","section":"Posts","summary":"TL;DR: - not much work outside of OpenSMTPD this week - OpenSMTPD portable builds with OpenSSL 1.1.x again - smtp-out reporting is working correctly on my laptop - wrote two filters that I'm actually using as you read this Shout outs to my patrons !","title":"July 2019 report: tons of smtpd work mostly"},{"content":" TL;DR: - started working on FION, a static tile window manager - revived BPG, a PGP parser - converted OpenSMTPD to libtls - wrote a library to make writing of native C OpenSMTPD filters easy - started writing a filter-rspamd Thanks to my patrons ! # First of all, a huge thanks to my first patrons:\nBleader Raton Diego Meseguer Mischa Peters Vegar Linge Halaand I have recently switched to a 75% part-time schedule at work so that I can spend a \u0026ldquo;free\u0026rdquo; week each month working on my own stuff, mostly opensource, without any kind of pressure: no one knows what I\u0026rsquo;ll be working on and no one but me gets to decide how I\u0026rsquo;ll spend this time.\nThis comes at the cost of slashing a quarter of my wage, which is sustainable but not ideal, so while most of my \u0026ldquo;free\u0026rdquo; weeks will be spent on opensource, I\u0026rsquo;ll be doing some sponsored development or short contracts to cover some of the loss if needed.\nI opened a patreon account so that people who care about my work can sponsor it, allowing me to spend most (if not all) of these \u0026ldquo;free\u0026rdquo; weeks publishing code for the community. If you want me to spend more time doing this then you know what to do: become my patron !\nSo thanks again to my patrons who make this possible, I will list them in all of my monthly reports, they are the sponsors of the work I describe below.\nFION: a featureless ion # I\u0026rsquo;m not too much of a graphical interface person, so when I was shown the ion window manager in 2003 I became addicted and, ever since, it\u0026rsquo;s been very hard for me to work efficiently using anything else. This is not an elitist thing, it is really me being easily distracted by stuff like windows not being properly aligned and similar stupid concerns that actually slow me considerably. I call this the \u0026ldquo;oh-a-butterfy\u0026rdquo; effect.\nUnfortunately, the last ion release dates from 2009 and is unmaintained. This leads to multiple problems, one being the lack of modern features, like multi-screen, which is essential for me considering the amount of time I spend reading code and specs side-to-side. Another problem is the fact that it\u0026rsquo;s been removed from most repositories, and I keep having to build it from source here and there. Recently it\u0026rsquo;s been a bit more difficult because on some distro, the dependencies for ion were no longer available and I had to start patching it here and there so it could build.\nI fear the day this happens on OpenBSD because it will mean that I have to spend time maintaining a piece of code for which there\u0026rsquo;s no upstream anymore \u0026hellip; or be back to working partly in console, partly in a maximized tmux when I really need a browser visible.\nI know this will lead people to tell me \u0026ldquo;but there\u0026rsquo;s notion, ion\u0026rsquo;s fork\u0026rdquo; or \u0026ldquo;but you can use awesome/dwm/i3/wmii/\u0026rdquo; but honestly I have tried a lot of different window managers along the years to try getting away from ion as it was abandoned and they never quite fit for me. I can only work if I have a wm that does tabs within static tiles that I can h-split/v-split at will and resize however I want. Notion came short because at least on OpenBSD it was laggy.\nI might as well spend a few hours / days working on my own, learn how they work, and release something I\u0026rsquo;m willing to maintain on the long run as it scratches my own itches.\nFION\u0026rsquo;s feature # I didn\u0026rsquo;t care that much for many of ion\u0026rsquo;s feature, never used the lua binding or the floating workspaces.\nFION will simply implement what I intend to use, so at the time being, this is:\npledge()-ed multiple screens of different sizes multiple workspaces per screen workspaces may be split in multiple horizontal and vertical tiles each tile may hold multiple X clients in tabs tiles may be resized screens / workspaces / tiles / tabs may be controlled through keyboard As of today, I\u0026rsquo;d say I\u0026rsquo;m halfway through, FION will detect screens and setup the workspaces to match screen size. It can create as many workspaces as you want on every single screen. It can h-split and v-split as many tiles as you want on the current workspace. It can iterate through workspaces and tiles. It can start the X clients and attach them, properly sized, to the current tile. It can give the focus to a tile through keyboard-triggered iteration of tiles or by moving mouse over a tile.\nCan you try it yet ? nope.\nFION currently does not have the logic to remove tiles, you can create and split but not kill tiles, which is not hard to do but I just ran of time this week ;-)\nAlso, keyboard events are not properly handled yet so I had to map features to a simple key like \u0026rsquo;n\u0026rsquo; or \u0026lsquo;p\u0026rsquo; to test them. Until I have more time for figuring out how I can implement my keyboard shortcuts so that they use control sequences, it will not be usable for daily use.\nCode is already committed on Github but again \u0026hellip; not usable, so stay tuned, the first usable version will be announced on twitter when it\u0026rsquo;s available.\nObligatory screenshots # I must have spent about six hours worth of work on it and the result is quite ok so I have good hope (no promise !) to make a first beta release during my \u0026ldquo;free\u0026rdquo; week in July.\nHere are a couple screenshots:\nFION ran on my laptop a few days ago before X clients were resized to match tiles:\n[full screen](2019-06-30-fion_1.png) FION ran on a bigger screen this morning:\n[full screen](2019-06-30-fion_2.png) And because a project doesn\u0026rsquo;t exist until it has a logo:\n[full screen](2019-06-30-fion_logo.jpg) BPG: BSD Privacy Guard # While looking at old private repositories to decide if I should axe them or not, I ran into an interesting one: bpg.\nLong story short, while I was a student in 2002 or 2003, I was doing night shifts at a security company to monitor firewalls and since it was essentially a passive job because I had nothing to do until the screens went all red, I started working on BPG a BSD-licensed PGP implementation.\nThen, NetBSD came up with a similar project for a Google Summer of code and they had someone already working on it, so I gave up as I had only a few weekly hours available for this and my work would surely be less good than someone working on it full-time for NetBSD.\nAnyways, I spent a while looking at the code to see what could be done with it, and it is actually not bad at all.\nI have a parser that can read all PGP packets in both old \u0026amp; new format, map them to appropriate structures that can actually be used: it can essentially read the keyrings and packets generated from gpg2, though it will not decrypt/verify.\nThe parser code was quite clean and I didn\u0026rsquo;t make changes to it except remove an allocator to replace it with malloc_conceal(3). I did a bit of code cleanup on other parts, like armoring which was ugly, but that\u0026rsquo;s about it.\nI don\u0026rsquo;t know what to do about that code, I have no idea how usable is the NetBSD implementation and how interesting it is for people to have another PGP utility. I will release the parser for sure since the code is already here and maybe it\u0026rsquo;s going to be useful to someone but I\u0026rsquo;m unsure if I should move project forward at this point. Oh and I need a new name anyways because bpg is already taken and it is certainly not going to be OpenBPG to avoid confusion with OpenBGP :-)\nI have quickly written code to generate RSA keypairs just for fun:\nlaptop$ bpg key gen rsa 2048 \u0026gt; /tmp/rsa2048.bpg ..........................................+++...................+++. laptop$ bpg debug parse /tmp/rsa2048.bpg ## 05 : New format Secret-Key packet (920 bytes) version: 4 algorithm: RSA (encrypt or sign) keyid: e60e44654c3409be fingerprint: 7bfd297855a9c0d22a492ba2e60e44654c3409be ## 0d : New format User ID packet (16 bytes) content: test@example.com laptop$ cat /tmp/rsa2048.bpg -----BEGIN PGP PRIVATE KEY BLOCK----- Version: BPG v0.1 (OpenBSD) xcLYBF0YccABCACzay1zxLQzD9WvOaF5DZXKIx5nmwP70+ff4XceepFnylwe7iKzTUF6W4kv QxOF++z9SQ5mF0melyj0+TkJXUSs7MzNjuwb4OhDLiM9uVJ2J1wxA3KivY0pVjvJPtPTlMpZ D7Ebb1CnagP75DEcgpdzWHDSzBbgHN89ct6M7fPZ756iDxHdnhO8syr9sB4GOJyj8hC2dC4y qJvn7G7/BdQwDov23FUGcddWXIyPMs4GGbE+mpV1U6jqZbNUkGPl+Kh8CfRU15D2/60PsrgD 3OSDo3J/DwGd2/NiVRRWVSOOtpzm4sc4j1J76nwrUJvvWRlH+AMOfDxoDu03bY2Jb/NtABEB AAEAB/9Pw8ZhQYIbcV6+mBCBkNiXFSXfSbtrqbncfpBGrJcYXY628YfbzuzdSPSkXl2/o1Cp CmGsYY4JQ4qh3mrNDvoJJv2mJXQysLqRo2Fnf4x5muYRpEbCsyKezgemYJgr6GpNTfyfBc4F n8xFoB11X1mVniwKi1FgMXXOC9OFNATFTlK2EjiBCBN1g+AbO31ykqDyhuVjAisQXlrpvRSr Jlk3zwNDgyu8jd9NyqCj3wUbp5GTj9/N+JVqm4lxqRAXUJk804w9q4Vn1x5f+lxyOw70m6vV 9xKZqKB/g2CmqFNVn8nzparvZQ45FSwKFkCz8Z0XrCJG8HvxkS0fBewu2BLBBADvAzcqblpX 9ygSDsOF/JDt9nhduLcVDgFzOagfZvh9eAny0YGQZ7rue2Vd3DKbMRBkybpn3jKpRbJC8+96 6AVaaxtoaHt7jKxZMHdjlxRPe6f9rFqETgL4a0+xyGzGd72t2OZG9pXNbydrwJPo0PSybkvR hkyd7vP3886X1EkdrwQAwCupzj3qSKf06ATAy+31rKoIUgsHD7OuyrNl7I1FZIMWNttaEl1C Z0bYaKHaTZbtoNk9EsISxSIXf3HB7nisKmGDe+LfXD06q9Bk1Tpjoj+zMXhJAm/NOQfbWXgJ 3/1b6EbI63o1JcBlZDXbrGv5IOY+c7gZqbOc3hD8EThHA6MD/0fQC4TuqH9nkJVyiDp0G+48 nq2oFDiDEfJ58TiSFlws/K+nuy2uGGO91iJLA6go5B87b0ihvl6A1Z4XFXuopRMcVp87n9gH qqRLhvoBqhMdVlfPnYr8JZYu0HkzmFtCnwuQOIA8EH8FQ65ImrP5PopazIHiUwhunrTzoAWp /4JOulbNEHRlc3RAZXhhbXBsZS5jb20= =0fuz -----END PGP PRIVATE KEY BLOCK----- laptop$ They are not valid, generating a valid key that can be imported to gnupg requires adding a signature packet which means I have to write the signing code to do that.\nSo there we go, either the project ends up as a PGP parsing library that others may use freely in their PGP implementations or it can evolve into a PGP library and utility, it depends on how needed and wanted this is by the community.\nI will commit what I have during July.\nOpenSMTPD: # What should come as no surprise: I spent most my week working on OpenSMTPD.\nI have tackled two topics:\nmigration from the OpenSSL to the libtls API for TLS support filters OpenSMTPD: OpenSSL -\u0026gt; libtls migration # I have started working on the migration from the OpenSSL API to the libtls API for TLS support in OpenSMTPD.\nAnd by \u0026ldquo;I have started\u0026rdquo;, I really mean \u0026ldquo;I\u0026rsquo;m almost done\u0026rdquo;.\nThe server side is mostly finished as I just need to rewrite the SNI support in a slightly different way. The client side is also mostly finished but needs a bit of cleanup.\nThis wasn\u0026rsquo;t as straightforward as I thought it would be, mostly due to the fact that OpenSMTPD uses a low-level \u0026ldquo;io\u0026rdquo; API for all its network io and that it was tricky to work on a progressive migration. I eventually found a way that allowed me to migrate the client-side first, then focus on the server-side, then use solely libtls. When this will be committed to OpenBSD, it will have to be an atomic switch, I won\u0026rsquo;t be able to progressively bring server and client side migrations.\nI also ran into some issues that were not easy to tackle, like the fact that we used ex data in RSA / EC_KEY objects to pass pki name to the privsep crypto engine for key lookups \u0026hellip; but libtls already used ex data to pass a checksum. I could have worked-around but it made sense leaving the checksum in ex data, as it\u0026rsquo;ll be useful in the future, so a bit of refactoring was require here and there.\nSo libtls-enabled OpenSMTPD is now a thing, when its finally commited I\u0026rsquo;ll be able to close multiple feature-requests that will be trivial to implement or already solved by sife-effect of the mgiration.\nDoes this mean OpenSMTPD will become LibreSSL only ?\nNOPE. This means that instead of having OpenSMTPD detect and deal with the differences between OpenSSL and LibreSSL as well as different OpenSSL versions, it will just depend on the libtls interface. We can then work on having a portable libtls for use as a wrapper around OpenSSL. This will then solve the LibreSSL vs OpenSSL vs different versions of OpenSSL issue for good.\nOpenSMTPD: Filters # I have resumed working on filters and this resulted in a few interesting bits.\ntimestamps in filter events # Until now, filter events reported the time at which an event was generated using a time_t unix timestamp accounting seconds since epoch.\nI had already converted reporting events to use a timeval for sub-second precision, so I also converted filter events to use a timeval too.\nfilter strderr mapped to log_warn() # I okayed a diff from martijn@ to map the stderr file-descriptor in filters to an endpoint in smtpd which logs all input with log_warn().\nThis means that a filter calling warnx() or a shell script doing an echo bleh \u0026gt;\u0026amp;2 will be logged in maillog, making it simpler to see output during development and in case of issues in production.\nfilter-eventlog # I have committed to github my first native filter, filter-eventlog which is a filter written in C that reports ALL smtp-in events to an append-only file, and creating a new file every day to hold the events of the day.\nThis is useful for filter developers as it allows them to check what events are generated by the daemon and which ones they should listen to in their filters.\nWith the lines below in your config:\n[...] filter mylogs proc-exec \u0026#34;/usr/libexec/filter-eventlog /tmp/logs\u0026#34; listen on socket filter mylogs [...] Sending a mail will result in the following being written to files within /tmp/logs:\nreport|1|1561888072.457091|smtp-in|link-connect|d812eceafa0ae39e|laptop.home|pass|local:0|local:0 report|1|1561888072.457781|smtp-in|filter-response|d812eceafa0ae39e|connected|proceed report|1|1561888072.457800|smtp-in|protocol-server|d812eceafa0ae39e|220 laptop.home ESMTP OpenSMTPD report|1|1561888072.458290|smtp-in|protocol-client|d812eceafa0ae39e|EHLO localhost report|1|1561888072.458773|smtp-in|filter-response|d812eceafa0ae39e|ehlo|proceed report|1|1561888072.458778|smtp-in|link-identify|d812eceafa0ae39e|localhost report|1|1561888072.458784|smtp-in|protocol-server|d812eceafa0ae39e|250-laptop.home Hello localhost [local], pleased to meet you report|1|1561888072.458787|smtp-in|protocol-server|d812eceafa0ae39e|250-8BITMIME report|1|1561888072.458790|smtp-in|protocol-server|d812eceafa0ae39e|250-ENHANCEDSTATUSCODES report|1|1561888072.458794|smtp-in|protocol-server|d812eceafa0ae39e|250-SIZE 36700160 report|1|1561888072.458797|smtp-in|protocol-server|d812eceafa0ae39e|250-DSN report|1|1561888072.458800|smtp-in|protocol-server|d812eceafa0ae39e|250 HELP report|1|1561888072.459268|smtp-in|protocol-client|d812eceafa0ae39e|MAIL FROM:\u0026lt;gilles@laptop.home\u0026gt; report|1|1561888072.459718|smtp-in|filter-response|d812eceafa0ae39e|mail-from|proceed report|1|1561888072.460764|smtp-in|tx-begin|d812eceafa0ae39e|754cbaa4 report|1|1561888072.460770|smtp-in|tx-mail|d812eceafa0ae39e|754cbaa4|\u0026lt;gilles@laptop.home\u0026gt; |ok report|1|1561888072.460774|smtp-in|protocol-server|d812eceafa0ae39e|250 2.0.0: Ok report|1|1561888072.460995|smtp-in|protocol-client|d812eceafa0ae39e|RCPT TO:\u0026lt;gilles@laptop.home\u0026gt; report|1|1561888072.461131|smtp-in|filter-response|d812eceafa0ae39e|rcpt-to|proceed report|1|1561888072.462172|smtp-in|tx-envelope|d812eceafa0ae39e|754cbaa4|754cbaa41a1afada report|1|1561888072.462180|smtp-in|tx-rcpt|d812eceafa0ae39e|754cbaa4|\u0026lt;gilles@laptop.home\u0026gt; |ok report|1|1561888072.462184|smtp-in|protocol-server|d812eceafa0ae39e|250 2.1.5 Destination address valid: Recipient ok report|1|1561888072.462496|smtp-in|protocol-client|d812eceafa0ae39e|DATA report|1|1561888072.462638|smtp-in|filter-response|d812eceafa0ae39e|data|proceed report|1|1561888072.463142|smtp-in|tx-data|d812eceafa0ae39e|754cbaa4|ok report|1|1561888072.463147|smtp-in|protocol-server|d812eceafa0ae39e|354 Enter mail, end with \u0026#34;.\u0026#34; on a line by itself report|1|1561888072.463866|smtp-in|protocol-client|d812eceafa0ae39e|. report|1|1561888072.463997|smtp-in|filter-response|d812eceafa0ae39e|commit|proceed report|1|1561888072.464034|smtp-in|tx-commit|d812eceafa0ae39e|754cbaa4|477 report|1|1561888072.465786|smtp-in|protocol-server|d812eceafa0ae39e|250 2.0.0: 754cbaa4 Message accepted for delivery report|1|1561888072.466065|smtp-in|protocol-client|d812eceafa0ae39e|QUIT report|1|1561888072.466294|smtp-in|filter-response|d812eceafa0ae39e|quit|proceed report|1|1561888072.466300|smtp-in|protocol-server|d812eceafa0ae39e|221 2.0.0: Bye report|1|1561888072.466481|smtp-in|link-disconnect|d812eceafa0ae39e libopensmtpd # The filters protocol is a very simple line-based protocol that I already explained in previous blog posts, but while writing filters in various scripting languages is simple, writing a filter in C was still a bit tedious.\nI started writing a libopensmtpd to provide a straightforward method for writing native C filters. It is not ready yet for publishing but it is already functional and works enough to trigger all hooks.\nAll it takes is:\n$ cc -o /tmp/filter-debug debug.c -lopensmtpd $ To make the following filter usable:\n/* * Copyright (c) 2019 Gilles Chehade \u0026lt;gilles@poolp.org\u0026gt; * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED \u0026#34;AS IS\u0026#34; AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include \u0026lt;err.h\u0026gt; #include \u0026lt;inttypes.h\u0026gt; #include \u0026lt;stdlib.h\u0026gt; #include \u0026#34;opensmtpd.h\u0026#34; extern char *__progname; /* reporting */ static void report_link_connect(opensmtpd_ctx_t *ctx, const char *rdns, const char *fcrdns, const char *src, const char *dest) { warnx(\u0026#34;report_link_connect: rdns=%s, fcrdns=%s, src=%s, dest=%s\u0026#34;, rdns, fcrdns, src, dest); } static void report_link_disconnect(opensmtpd_ctx_t *ctx) { warnx(\u0026#34;report_link_disconnect\u0026#34;); } static void report_link_identify(opensmtpd_ctx_t *ctx, const char *helo_name) { warnx(\u0026#34;report_link_identify: helo_name=%s\u0026#34;, helo_name); } static void report_link_tls(opensmtpd_ctx_t *ctx, const char *tls_line) { warnx(\u0026#34;report_link_tls: tls_line=%s\u0026#34;, tls_line); } static void report_tx_begin(opensmtpd_ctx_t *ctx, uint32_t msgid) { warnx(\u0026#34;report_tx_begin: msgid=%08x\u0026#34;, msgid); } static void report_tx_mail(opensmtpd_ctx_t *ctx, uint32_t msgid, const char *mail_from, const char *result) { warnx(\u0026#34;report_tx_mail: msgid=%08x, mail_from=%s, result=%s\u0026#34;, msgid, mail_from, result); } static void report_tx_rcpt(opensmtpd_ctx_t *ctx, uint32_t msgid, const char *rcpt_to, const char *result) { warnx(\u0026#34;report_tx_rcpt: msgid=%08x, mail_from=%s, result=%s\u0026#34;, msgid, rcpt_to, result); } static void report_tx_envelope(opensmtpd_ctx_t *ctx, uint32_t msgid, uint64_t evpid) { warnx(\u0026#34;report_tx_envelope: msgid=%08x, evpid=%016\u0026#34;PRIx64\u0026#34;\u0026#34;, msgid, evpid); } static void report_tx_commit(opensmtpd_ctx_t *ctx, uint32_t msgid) { warnx(\u0026#34;report_tx_commit: msgid=%08x\u0026#34;, msgid); } static void report_tx_data(opensmtpd_ctx_t *ctx, uint32_t msgid, size_t msg_size) { warnx(\u0026#34;report_tx_data: msgid=%08x, msg_size=%zd\u0026#34;, msgid, msg_size); } static void report_tx_rollback(opensmtpd_ctx_t *ctx, uint32_t msgid) { warnx(\u0026#34;report_tx_rollback: msgid=%08x\u0026#34;, msgid); } static void report_protocol_client(opensmtpd_ctx_t *ctx, const char *line) { warnx(\u0026#34;report_protocol_client: line=%s\u0026#34;, line); } static void report_protocol_server(opensmtpd_ctx_t *ctx, const char *line) { warnx(\u0026#34;report_protocol_server: line=%s\u0026#34;, line); } static void report_filter_response(opensmtpd_ctx_t *ctx, const char *phase, const char *response) { warnx(\u0026#34;report_filter_response: phase=%s, response=%s\u0026#34;, phase, response); } static void report_timeout(opensmtpd_ctx_t *ctx) { warnx(\u0026#34;report_timeout\u0026#34;); } /* filtering */ static void filter_connect(opensmtpd_ctx_t *ctx, const char *rdns, const char *arg) { warnx(\u0026#34;filter_connect: rdns=%s, arg=%s\u0026#34;, rdns, arg); proceed(ctx); } static void filter_helo(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_helo: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_ehlo(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_ehlo: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_starttls(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_starttls: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_auth(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_auth: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_mail_from(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_mail_from: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_rcpt_to(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_rcpt_to: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_data(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_data: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_data_line(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_data_line: arg=%s\u0026#34;, arg); dataline(ctx, arg); } static void filter_rset(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_rset: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_quit(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_quit: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_noop(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_noop: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_help(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_help: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_wiz(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_wiz: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_commit(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_commit: arg=%s\u0026#34;, arg); proceed(ctx); } int main(int argc, char *argv[]) { opensmtpd_ctx_t *ctx; if (argc != 1) errx(1, \u0026#34;usage: %s\u0026#34;, __progname); smtpd_init(\u0026amp;ctx); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;link-connect\u0026#34;, report_link_connect); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;link-disconnect\u0026#34;, report_link_disconnect); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;link-identify\u0026#34;, report_link_identify); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;link-tls\u0026#34;, report_link_tls); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-begin\u0026#34;, report_tx_begin); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-mail\u0026#34;, report_tx_mail); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-rcpt\u0026#34;, report_tx_rcpt); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-envelope\u0026#34;, report_tx_envelope); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-data\u0026#34;, report_tx_data); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-commit\u0026#34;, report_tx_commit); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-rollback\u0026#34;, report_tx_rollback); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;protocol-client\u0026#34;, report_protocol_client); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;protocol-server\u0026#34;, report_protocol_server); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;filter-response\u0026#34;, report_filter_response); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;timeout\u0026#34;, report_timeout); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;connect\u0026#34;, filter_connect); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;helo\u0026#34;, filter_helo); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;ehlo\u0026#34;, filter_ehlo); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;starttls\u0026#34;, filter_starttls); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;auth\u0026#34;, filter_auth); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;mail-from\u0026#34;, filter_mail_from); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;rcpt-to\u0026#34;, filter_rcpt_to); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;data\u0026#34;, filter_data); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;rset\u0026#34;, filter_rset); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;quit\u0026#34;, filter_quit); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;noop\u0026#34;, filter_noop); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;help\u0026#34;, filter_help); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;wiz\u0026#34;, filter_wiz); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;commit\u0026#34;, filter_commit); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;data-line\u0026#34;, filter_data_line); if (! smtpd_event_loop(ctx)) return 1; return 0; } and register all hooks:\nlaptop$ /tmp/filter-debug register|report|smtp-in|filter-response register|report|smtp-in|link-connect register|report|smtp-in|link-disconnect register|report|smtp-in|link-identify register|report|smtp-in|link-tls register|report|smtp-in|protocol-client register|report|smtp-in|protocol-server register|report|smtp-in|timeout register|report|smtp-in|tx-begin register|report|smtp-in|tx-commit register|report|smtp-in|tx-data register|report|smtp-in|tx-envelope register|report|smtp-in|tx-mail register|report|smtp-in|tx-rcpt register|report|smtp-in|tx-rollback register|filter|smtp-in|auth register|filter|smtp-in|commit register|filter|smtp-in|connect register|filter|smtp-in|data register|filter|smtp-in|data-line register|filter|smtp-in|ehlo register|filter|smtp-in|helo register|filter|smtp-in|help register|filter|smtp-in|mail-from register|filter|smtp-in|noop register|filter|smtp-in|quit register|filter|smtp-in|rcpt-to register|filter|smtp-in|rset register|filter|smtp-in|starttls register|filter|smtp-in|wiz register|ready ^C laptop$ In OpenSMTPD, with the following config:\n[...] filter foobar proc-exec \u0026#34;/tmp/filter-debug\u0026#34; listen on socket filter foobar it will result in the following output:\n[...] 86c0820a13ffee17 smtp connected address=local host=laptop.home \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_link_connect: rdns=laptop.home, fcrdns=pass, src=local:0, dest=local:0 \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_connect: rdns=laptop.home, arg=local \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=connected, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=220 laptop.home ESMTP OpenSMTPD \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=EHLO localhost \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_ehlo: arg=localhost \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=ehlo, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_link_identify: helo_name=localhost \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-laptop.home Hello localhost [local], pleased to meet you \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-8BITMIME \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-ENHANCEDSTATUSCODES \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-SIZE 36700160 \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-DSN \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250 HELP \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=MAIL FROM:\u0026lt;gilles@laptop.home\u0026gt; \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_mail_from: arg=gilles@laptop.home \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=mail-from, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_begin: msgid=2b8bbe3b \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_mail: msgid=2b8bbe3b, mail_from=\u0026lt;gilles@laptop.home\u0026gt;, result=ok \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250 2.0.0: Ok \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=RCPT TO:\u0026lt;gilles@laptop.home\u0026gt; \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_rcpt_to: arg=gilles@laptop.home \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=rcpt-to, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_envelope: msgid=2b8bbe3b, evpid=2b8bbe3b60567a11 \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_rcpt: msgid=2b8bbe3b, mail_from=\u0026lt;gilles@laptop.home\u0026gt;, result=ok \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250 2.1.5 Destination address valid: Recipient ok \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=DATA \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data: arg= \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=data, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=Received: from localhost (laptop.home [local]) \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg= by laptop.home (OpenSMTPD) with ESMTPA id 2b8bbe3b \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg= for \u0026lt;gilles@laptop.home\u0026gt;; \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg= Sun, 30 Jun 2019 12:01:55 +0200 (CEST) \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_data: msgid=2b8bbe3b, msg_size=10034375646789 \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=354 Enter mail, end with \u0026#34;.\u0026#34; on a line by itself \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=From: \u0026lt;gilles@laptop.home\u0026gt; \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=Date: Sun, 30 Jun 2019 12:01:55 +0200 (CEST) \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=To: gilles \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg= \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=test \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=. \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=. \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_commit: arg= \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=commit, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_commit: msgid=2b8bbe3b \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250 2.0.0: 2b8bbe3b Message accepted for delivery \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=QUIT \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_quit: arg= [...] filter-rspamd in progress # I had written a filter-rspamd for OpenSMTPD in python already, however it was a proof-of-concept more than a real filter and I didn\u0026rsquo;t make it really robust.\nWith libopensmtpd at hands, I decided to write a native C filter-rspamd that I would officially maintain, one that would be robust enough to be used in production.\nI had most of the logic done in half an hour, unfortunately I ran out of time for this \u0026ldquo;free\u0026rdquo; week and didn\u0026rsquo;t complete the filter.\nBasically as it is right now, the filter will build everything needed for the call to rspamd, but I didn\u0026rsquo;t write the http request.\nI will surely finish that during my next \u0026ldquo;free\u0026rdquo; week.\nWhat next ? # Well, see you in a month for the July report.\nHopefully, by then I will have finished my filter-rspamd, I will have released an initial version of FION, and I will have released an initial version of libopensmtpd.\nWhen I\u0026rsquo;m done with FION and have decided what to to with BPG, I will have a look at some of my other private repositories to see what can be released :-)\nIf you like my work, support me on patreon !\n","date":"30 June 2019","permalink":"/posts/2019-06-30/june-2019-report-fion-bpg-and-smtpd/","section":"Posts","summary":"TL;DR: - started working on FION, a static tile window manager - revived BPG, a PGP parser - converted OpenSMTPD to libtls - wrote a library to make writing of native C OpenSMTPD filters easy - started writing a filter-rspamd Thanks to my patrons !","title":"June 2019 report: fion, bpg and smtpd"},{"content":" TL;DR: In this post I explain crudely how ca.c works and changes to OpenSMTPD related to ca.c I wrote an ECDSA privsep crypto engine I did some EEG work too This is the first report # I will now switch to a monthly report of my tech activities on this blog, and this is the first post in that new format.\nThe focus will be put on interesting topics, not necessarily every single commit and bug fix I do (ie: won\u0026rsquo;t mention LMTP bug fix here), this depends on the amount of slacking that ocurred during the month and how much I want to cover it up :-)\nDon\u0026rsquo;t forget to tip me on the link above if you like reading my reports, donations cover my random geek expenses and keep me motivated to write about what I do.\nAbout OpenSMTPD\u0026rsquo;s ca.c file # A few years ago, Neel Mehta blessed us with the Heartbleed bug in the OpenSSL library. A bug which ultimately resulted in the disclosure of sensitive data, including cryptographic private keys, that resided in the memory of OpenSSL\u0026rsquo;s custom allocator.\nA good approach at mitigating the issue was taken by the LibreSSL folks. Long story short, a custom allocator is a bad idea because it potentially defeats any security measure provided by the standard allocator. In OpenBSD, the multiple security mechanisms in place to detect memory corruption and invalid access were bypassed by the use of the OpenSSL custom allocator, and the first steps taken by LibreSSL were to ensure that functions such as OpenSSL_malloc and OpenSSL_free where REALLY calling malloc and free, and not something else trying to be \u0026ldquo;smart\u0026rdquo;.\nAnother approach was taken by reyk@ and consisted in assuming that private keys in a network facing process was too dangerous. To mitigate the risk, a good option would be to isolate the crypto operations that required access to the private key in a separate process, one that would be fairly isolated from untrusted users. Here comes ca.c, a process operating privileges separated crypto operations.\nHere\u0026rsquo;s how it works:\nUpon startup, OpenSMTPD declares a custom RSA engine which provides its own primitives for crypto operations. It then registers that custom engine as the default RSA engine, meaning that any use of the OpenSSL API to perform RSA operations will use the new primitives rather than the default ones.\nThe primitives provided by the RSA engine are actually of two kind: either they rely on the private key in which case they will perform an IPC to the CA (crypti agent) process which will do the operation itself through the default primitive and provide the result, or they don\u0026rsquo;t rely on the private key in which case the primitive simply falls back to the default one.\n[...] static int rsae_pub_dec(int flen,const unsigned char *from, unsigned char *to, RSA *rsa, int padding) { log_debug(\u0026#34;debug: %s: %s\u0026#34;, proc_name(smtpd_process), __func__); return (rsa_default-\u0026gt;rsa_pub_dec(flen, from, to, rsa, padding)); } static int rsae_priv_enc(int flen, const unsigned char *from, unsigned char *to, RSA *rsa, int padding) { log_debug(\u0026#34;debug: %s: %s\u0026#34;, proc_name(smtpd_process), __func__); if (RSA_get_ex_data(rsa, 0) != NULL) { return (rsae_send_imsg(flen, from, to, rsa, padding, IMSG_CA_PRIVENC)); } return (rsa_default-\u0026gt;rsa_priv_enc(flen, from, to, rsa, padding)); } [....] The idea is that if the network facing process is compromised, then no matter how bad the memory corruption and/or info leak is, the private keys are not available to that process.\nFix OpenSMTPD ca.c build with OpenSSL # That private key privsep relies on using the OpenSSL crypto engine API.\nEver since OpenSSL switched to 1.1.x, the crypto engine API changed because structures that were public and could be dereferenced would no longer be public. They would require a set of getters and setters accessors to make any use of the structure. The sad thing is that this is not the case with LibreSSL, the structures are still public and accessors are not available.\nThis results in the same code not being able to be built for both, either you use the opaque structures and build is broken on LibreSSL, or you use the public structures and build is broken on OpenSSL.\nI rewrote ca.c to pretend that the structure was opaque, not performing any direct deref, and added the missing accessors within an ifdef. This is not ideal but it reduces the delta between our native branch and portable, and as soon as accessors are available in LibreSSL we can simply remove the ifdef-ed part.\nI submitted the diff to my fellow OpenBSD hackers, it\u0026rsquo;s pending review at this point, should be committed soon.\nTurn most of OpenSMTPD\u0026rsquo;s SSL structures opaque # A lot of other structures that are also opaque did have accessors in LibreSSL, notably the EVP interface which we use for encrypted queue support.\nThe code used to rely on automatic variables and dereferences, I converted the code to heap allocated structures using the opaque structures interface.\nDiff was submitted, okayed and committed.\nAdd support for opaque RSA_METHOD engine in LibreSSL # As mentioned earlier, code in ca.c was broken due to the fact that LibreSSL considers RSA_METHOD as a public structure that can be dereferenced, whereas OpenSSL considers it as an opaque structure that requires accessors.\nEven considering that RSA_METHOD is a public structure, this doesn\u0026rsquo;t prevent us from providing accessors to allow a common idiom between LibreSSL and OpenSSL.\nI submitted a diff to the LibreSSL folks to bring the missing accessors, which will allow me to write code that builds for both LibreSSL (my target) and OpenSSL. This will not be an OpenSMTPD only thing because any OpenBSD daemon that relies or will rely on ca.c and wants to be portable will face the same issue.\nThe diff has been reviewed and okayd, it will require a LibreSSL (libcrypto, libssl and libtls) minor crank so I\u0026rsquo;m waiting for the appropriate time to do it. I\u0026rsquo;m also waiting for an ok from ports to make sure it doesn\u0026rsquo;t break ports as adding new symbols may have some configure scripts uncover other symbols we don\u0026rsquo;t have.\nWrite an ECDSA privsep engine for OpenSMTPD # The crypto privsep engine only supported RSA and this means that starting OpenSMTPD with an ECDSA results in a fatal():\ndebug: init private ssl-tree debug: SSL library error: ssl_ctx_create: error:06FFF07F:digital envelope routines:CRYPTO_internal:expecting an rsa key debug: SSL library error: ssl_ctx_create: error:140AE006:SSL routines:func(174):EVP lib pony express: ssl_ctx_create: could not fake private key: Invalid argument More and more people have complained that we didn\u0026rsquo;t support ECDSA certificates, but it was not a trivial one-liner as adding support meant writing a custom ECDSA crypto engine akin to what reyk@ did for RSA. It took me time to find the motivation, but ultimately I came around to implement such an engine this week.\ndebug: rsa_engine_init: using RSA privsep engine debug: ecdsa_engine_init: using ECDSA privsep engine The details of how I implemented the ECDSA crypto engine are not that interesting, they follow the same pattern as the RSA crypto engine:\nECDSA_SIG * ecdsae_do_sign(const unsigned char *dgst, int dgst_len, const BIGNUM *inv, const BIGNUM *rp, EC_KEY *eckey) { log_debug(\u0026#34;debug: %s: %s\u0026#34;, proc_name(smtpd_process), __func__); if (ECDSA_get_ex_data(eckey, 0) != NULL) return (ecdsae_send_enc_imsg(dgst, dgst_len, inv, rp, eckey)); return (ecdsa_default-\u0026gt;ecdsa_do_sign(dgst, dgst_len, inv, rp, eckey)); } The real challenge was to teach ca.c that multiple crypto engines could coexist, figuring out the primitives for ECDSA and doing the proper serialization in IPC.\nOn my laptop, OpenSMTPD can now load and work with both RSA and ECDSA certificates, accepting SMTP sessions on listeners presenting either one:\n9362d76af3ab2882 smtp connected address=127.0.0.1 host=localhost debug: looking up pki \u0026#34;localhost\u0026#34; debug: session_start_ssl: switching to SSL debug: pony: rsae_priv_enc 9362d76af3ab2882 smtp tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256 8c25253d498c0990 smtp connected address=127.0.0.1 host=localhost debug: looking up pki \u0026#34;localhost\u0026#34; debug: session_start_ssl: switching to SSL debug: pony: ecdsae_do_sign 8c25253d498c0990 smtp tls ciphers=TLSv1.2:ECDHE-ECDSA-AES256-GCM-SHA384:256 I have submitted the diff today, pending review, it should be committed shortly.\nTroubleshoot eegreader and write initial support for an eegviewer # A while ago, I got interested in reading electroencephalogram data to detect interesting patterns during self-hypnosis sessions.\nI got myself an OpenEEG device and realized there were no simple utilities to extract data from it, most were C++ bloated software that didn\u0026rsquo;t build on my laptop and required dependencies.\nI found out that the OpenEEG protocol was very easy to implement and wrote eegreader a utility to output an OpenEEG data stream to stdout, displaying sequence, version, the six channels and buttons values.\nThere seemed to be a bug because packet loss kept triggering my recovery code, skipping bits until finding the alignment for a new packet (eg: packets 53, 56, 58, 5a, 5c are missing in this sample):\n[...] 4f|2|512|511|505|499|493|487|7 50|2|532|511|505|500|493|487|7 51|2|513|65281|63745|62465|60673|59143|5 53|2|497|511|505|499|493|487|7 54|2|508|511|505|499|493|487|7 55|2|513|65281|63745|62465|60929|59143|5 57|2|514|511|505|500|494|487|5 59|2|506|511|505|500|494|487|5 5b|2|535|511|505|500|494|487|5 5d|2|499|511|505|500|493|487|5 [...] I investigated and realized that I didn\u0026rsquo;t have the issue on some other OSes, it seems to be an OpenBSD USB bug. Furthermore, with the latest OpenBSD snapshot, I seem to be hitting kernel panics in USB so some investigation is definitely needed. I pinged some of the USB gurus, we\u0026rsquo;ll see how I can help.\nI still managed to do something useful as I started the writing of eegviewer, which is a libxcb graphical interface displaying the eegreader input as a real-time graph. The idea is to simply pipe eegreader into eegviewer to obtain the graphs.\nThe eegreader utility has already been published a while ago on Github under the ISC license. I\u0026rsquo;ll publish eegviewer under the same license when I\u0026rsquo;m happy with the code, libxcb not being something I\u0026rsquo;m familiar with.\nFinally, I will very likely write an ISC-licensed libeeg which will provide an event-driven interface to OpenEEG data streams, allowing callbacks to be triggered upon certain events occuring on certain channels. I haven\u0026rsquo;t started that work yet but it is trivial so I might release that soon.\nOculus Quest # Thanks to my geek donations, I got myself an Oculus Quest device :-)\nI\u0026rsquo;m not a gamer but I\u0026rsquo;m genuinely curious what can be done with it, when time allows I\u0026rsquo;ll start digging the SDK.\nWhat next ? # See you in a month for the June report, very likely focusing around OpenSMTPD filters !\n","date":"2 June 2019","permalink":"/posts/2019-06-02/may-2019-report/","section":"Posts","summary":"TL;DR: In this post I explain crudely how ca.c works and changes to OpenSMTPD related to ca.c I wrote an ECDSA privsep crypto engine I did some EEG work too This is the first report # I will now switch to a monthly report of my tech activities on this blog, and this is the first post in that new format.","title":"May 2019 report"},{"content":" TL;DR: Way too many things happened in a six months timeframe. This post won't need a TL;DR as I'll keep it short. Generalized anxiety disorder and alexithymia. # Late 2018, I\u0026rsquo;ve been diagnosed with generalized anxiety disorder and alexithymia a couple weeks apart.\nThe anxiety disorder didn\u0026rsquo;t really come as a surprise as I know very well the traumatic experience that led to the disorder. It\u0026rsquo;ll still take a lot of hours of hypnosis to continue cleaning up trauma after trauma, but taking care of it and releasing some of the traumas has improved considerably my anxiety.\nI\u0026rsquo;ve also tried different approaches out of curiosity, including consumption of increasing doses of CBD in various forms, but didn\u0026rsquo;t observe noticeable improvements no matter if smoked, vaped, eaten, tinctured, \u0026hellip; it does help sleep earlier if I consume a large dose of CBD but only hypnosis relieves anxiety effectively for me.\nThe alexithymia diagnosis crushed me for a while. I didn\u0026rsquo;t know what it was but as the psychologist explained the disorder, all of the implications throughout the years hit me in the face very harshly.\nAlexithymia is an inability to identify and describe emotions in the self. I learnt at 38 that I\u0026rsquo;m mostly cut from emotions and that when people say \u0026ldquo;I feel happy\u0026rdquo; or \u0026ldquo;I feel sad\u0026rdquo; they actually feel it, whereas I will say that I feel them when I \u0026ldquo;know\u0026rdquo; I should feel them, not because I actually feel them.\nI won\u0026rsquo;t enter much more details as it still makes me uncomfortable at this point, but one of the side-effects of alexithymia is that it makes the anxiety disorder considerably worse: I cannot see attacks coming and can\u0026rsquo;t use any techniques to prevent them from unfolding. In most situations I will not even realize that I\u0026rsquo;m having an anxiety attack but will still suffer from all of its symptoms without being able to make a connection, nor understand what is happening.\nI wish I had read about this earlier and put some sense into it earlier, which is why I\u0026rsquo;m taking a few paragraphs to explain and will answer any question in the comments if you have some. I will probably write again about it in the future but I don\u0026rsquo;t think GAD or alexithymia characterize me, they haven\u0026rsquo;t prevented me from having a career or hobbies though they made it considerably harder, so I will not give them a bigger place than they already have in my every day life, it won\u0026rsquo;t be a recurrent topic.\nI\u0026rsquo;d say it\u0026rsquo;s been an emotional rollercoaster but now you know I don\u0026rsquo;t really grasp what it means :-)\nOther misc. personal stuff # In just a few months, I learnt my half was pregnant and we\u0026rsquo;re expecting a little guy in October \u0026lt;3, we moved to our own place that we\u0026rsquo;ve been waiting for two years to be built, my cat died after sharing sixteen years with me, I fully completed my final hypnotherapist training putting an end to years of training, and mostly finished my third year of studying occupational clinical psychology.\nI have also changed role at work and requested to switch to a 75% part-time job, spending three weeks at work for one week off to work on my projects, both open-source and hypno/psycho related.\nLife has been busy and kept me away from writing here.\nWhat next ? # Now that I will be able to spend one week off each month, I\u0026rsquo;ll write a monthly report of my activities on this blog, this will be an easier rythm than writing whenever I do something.\nThis post was not the May monthly report, which I\u0026rsquo;ll write right after and publish tonight, I just wanted to get the non-technical stuff out of the way.\nCheers, Gilles\n","date":"2 June 2019","permalink":"/posts/2019-06-02/happy-new-year-2019-a-personal-post/","section":"Posts","summary":"TL;DR: Way too many things happened in a six months timeframe. This post won't need a TL;DR as I'll keep it short. Generalized anxiety disorder and alexithymia.","title":"happy new year 2019, a personal post"},{"content":" TL;DR: regex table lookups were introduced for builtin filters. After a few weeks of working solely on filters, I wanted to work on something else. Using the same mechanism, all match criterias using tables can support regex. K_REGEX lookups # The table mechanism is used within OpenSMTPD to perform all kinds of lookups.\nRecently, while working on builtin filters, I introduced the K_REGEX lookup type allowing tables to serve regex(3) patterns.\nWhen a K_REGEX lookup is performed, the table API will use the lookup key as a string and iterate over entries in the table, compiling them with regcomp(3) and performing a simple regcomp(3) against the key.\nregex use in match criterias # In the new smtpd.conf, accept rules where replaced with match rules.\nThe match rules consist of matching criterias which describe how an envelope should look like to be accepted for a specific action. Among these criterias, you\u0026rsquo;ll find for example from src, for domain, mail-from or even rcpt-to, which can either take a table, or a string literal which \u0026hellip; is actually an inlined on-element table.\nThe new feature introduces a regex keyword that can be used to instruct OpenSMTPD that the table will use regex. This basically allows the following:\ntable regex_domains file:/etc/mail/list-of-regex.txt\u0026#34; match for domain regex \u0026#34;^.*\\.example.com$\u0026#34; action foobar match for domain regex \u0026lt;regex_domains\u0026gt; action foobar except that it also allows it for pretty much every criteria that can be used on a match rule and expect a parameter.\nWhat next ? # This was just committed a few minutes ago in -current and the feature is essentially complete, so there\u0026rsquo;s not much in that front.\nYou can already play with that feature which has been documented in the smtpd.conf(5) man page.\nSo \u0026hellip; NOW, I\u0026rsquo;m taking a break until next week because I did everything I wanted to do for this year ;-)\n","date":"21 December 2018","permalink":"/posts/2018-12-21/opensmtpd-now-supports-regex-in-match-rules/","section":"Posts","summary":"TL;DR: regex table lookups were introduced for builtin filters. After a few weeks of working solely on filters, I wanted to work on something else.","title":"OpenSMTPD now supports regex in match rules"},{"content":" TL;DR: Not this time, pal/gal, I took hours writing this post, you'll take a few minutes reading it all. Oh, and merry X-mas :-* A bit of short-sighted history # The filtering feature has been introduced only recently in OpenSMTPD, first presented on this blog a month ago.\nI had a working proof-of-concept running on my laptop and my plan was to start bringing the code to the OpenBSD tree, small chunks by small chunks, through a serie of diffs.\nI won\u0026rsquo;t cover the serie in details because it\u0026rsquo;s irrelevant, but the point is that we went from scratch to a working filter implementation in a few days, one that we knew would work but that had only ran on a one-interface laptop and was not polished.\nIn the last few weeks, as various proof-of-concepts for useful filters were written, the protocol and API improved and while there\u0026rsquo;s many more improvements to come before the next major release, it is safe to say that the API now allows full-featured filters.\nHow does filtering work in OpenSMTPD ? # During the lifetime of an SMTP session, whenever a client sends a command, the server checks if the command should be accepted or rejected then replies to the client.\nFirst it checks if the command is valid in terms of syntax and context (for example, AMIL FORM is an invalid command, MAIL FROM: \u0026gt;\u0026lt; is a valid command with an invalid parameter, and you can\u0026rsquo;t send RCPT TO before MAIL FROM). Then, it checks if the command is actually allowed by the ruleset. Sometimes, it may reject a valid command for issues related to lack of ressources, but let\u0026rsquo;s keep this corner case out.\nHow does filtering work then ?\nWell, right between the two types of checks, after the server has checked that the input was valid in terms of syntax and context, it pauses processing and requests the lookup process to take a decision based on the input. The lookup process passes that input to the filters sequentially and if any decides to take an action, the server will enforce that action instead of resuming its usual check.\nWhen it comes to the DATA part, things are trickier because the server doesn\u0026rsquo;t answer to the individual lines but to the whole message. Also, it is expected that the filters can actually make changes to the content of the message. As such, there are two kinds of filtering in OpenSMTPD: protocol filtering and data filtering. The former provides a simple query/reply mechanism to alter the decision taking in the SMTP engine, the latter provides a simple transformation mechanism consuming an input and producing an output.\nLet\u0026rsquo;s get a bit more technical # As mentionned in the blog post linked in my introduction, there are two mechanisms involved: reporting and filtering.\nEach of these have their own specific hooks, triggering at specific phases of the SMTP session, and allowing the SMTP engine to generate reporting events or filter queries at appropriate times.\nThe basic idea when I designed the filter API was that we don\u0026rsquo;t want complexity in these hooks. We do want filters to have as much flexibility as possible, a filter should be able to reject a session after the data phase because the source address used during the connect phase didn\u0026rsquo;t have a proper reverse DNS configured, but this should not come at the cost of having hooks with tons of parameters or huge structures encompassing each and everything.\nTo achieve this, the reporting and filtering mechanisms were designed with the following philosophy in mind:\nThe reporting mechanism should generate enough events that it is possible to replicate the state of an SMTP session, but these events should only contain the information related to the event itself. A filter that wants to reject a session after the data phase because of the reverse DNS doesn\u0026rsquo;t need to receive the reverse DNS during the data phase, instead it needs to receive it during the connect phase and keep it in a local state to check it at the data phase. This allowed making the reporting mechanism very simple and not limiting since a filter can essentially hook all reporting phases and have the exact same state as the SMTP session itself.\nThe filtering mechanism in turn should only take decisions based on the parameters of the phase itself \u0026hellip; and any local state. A filter registering a hook for the MAIL FROM phase will only receive the address of the sender. If it wants to take decisions based on anything more than that, then it needs to rely on a local state gathered from the reporting mechanism.\nThe data filtering is trickier because we want to be able to add, change or suppress lines. As a result, we can\u0026rsquo;t rely on the number of bytes we sent and the number of bytes we received, just like we can\u0026rsquo;t rely on a lines count. Furthermore, there can be multiple filters processing the data in sequence, so the first filter may receive 1 line, generate 4 headers, the second filter remove some of these headers, the third filter append a ton of data, etc\u0026hellip;\nThis is where eric@ had a very clever idea which is to consider the DATA part as a stream. The SMTP protocol ends DATA with a single . on a line by itself, so the lines are streamed to the filters which read them up to the . and output a new stream terminate by a . itself. Doing this allows the SMTP engine to stream to filters the DATA up to the client-submitted ., then read back a stream from the filters up to the filter-submitted ., not having to care if the stream was altered.\nSounds like a lot of details ?\nYou need not worry, in practice writing a filter is trivial.\nA case study: filter-rspamd # I wrote many filters to experiment and refine the API, but here\u0026rsquo;s an interesting one.\nLet\u0026rsquo;s have a look at how this whole reporting and filtering mechanisms come into play. I wrote the filter in python, it is fairly small but I will not copy paste the whole code because I wrote it as a PoC, and I\u0026rsquo;m ashamed of the quality.\nFirst of all, let\u0026rsquo;s define what we want to do with the filter.\nWe want a filter that will pass a DATA part to the rspamd daemon, with several session informations gathered from the connection up to the message itself so it can take a decision, and then alter the message to insert headers and possibly reject temporarily or permanently the message. Since I was in a good mood when I wrote the PoC, I also added DKIM signing but it\u0026rsquo;s out of scope ;-)\nRemember that this is a Python example, because I find it easier to understand for all, but filters can be written with any language really.\nsessions = {} class Rspamd(): def __init__(self): self.stream = smtp_in() self.stream.on_report(\u0026#39;link-connect\u0026#39;, link_connect, None) self.stream.on_report(\u0026#39;link-disconnect\u0026#39;, link_disconnect, None) self.stream.on_report(\u0026#39;link-identify\u0026#39;, link_identify, None) self.stream.on_report(\u0026#39;tx-begin\u0026#39;, tx_begin, None) self.stream.on_report(\u0026#39;tx-mail\u0026#39;, tx_mail, None) self.stream.on_report(\u0026#39;tx-rcpt\u0026#39;, tx_rcpt, None) self.stream.on_report(\u0026#39;tx-data\u0026#39;, tx_data, None) self.stream.on_report(\u0026#39;tx-commit\u0026#39;, tx_cleanup, None) self.stream.on_report(\u0026#39;tx-rollback\u0026#39;, tx_cleanup, None) self.stream.on_filter(\u0026#39;commit\u0026#39;, filter_commit, None) self.stream.on_filter(\u0026#39;data-line\u0026#39;, filter_data_line, None) def run(self): self.stream.run() Based on this excerpt alone, without looking at the actual implementation of the callback functions, here\u0026rsquo;s what you can understand:\nThe only times when we actually want to mess with a session somehow is when making changes to the data, which is handled by the data-line callback, and when we want to possibly reject temporarily or permanently a message, which is handled by the commit callback.\nAll of the on_report() calls are used solely to accumulate enough informations so the two filter hooks can work, and if we look at the implementation of one of these, you\u0026rsquo;ll realize that all it does is really store a particular bit of information in the local state for a session.\nFor example, the purpose of registering for the link-identify event is to store the helo name the client submitted in the local session state:\ndef link_identify(ctx, timestamp, session_id, args): helo = args[0] session = sessions[session_id] session.control[\u0026#39;Helo\u0026#39;] = helo So that the information can be sent to the rspamd daemon when the filter begins sending the message in the data-line callback.\nPer-listener filtering # In the first version of the filter feature I commited, filters were declared globally in the configuration, listeners would only enable/disable filtering, the filters would be applied in sequence:\nfilter smtp-in connect check-fcrdns reject \u0026#34;550 go away you punk\u0026#34; filter smtp-in connect check-rdns reject \u0026#34;550 go away you punk\u0026#34; listen on all filter\t# not filtered listen on socket\t# filtered This was a first step at plugging filters on and off, but real use cases rely on being able to plug different sets of filters on different interfaces, so that you can for example reject senders without a reverse DNS on an interface, while allowing them on the submission port where they authenticate.\nThis feature is now supported.\nFilter grammar changes # The filter declaration grammar I used to begin was easy to start playing right away, it couldn\u0026rsquo;t cover some of the use-cases I had in mind and planned for, but now that the plumbing has evolved enough we can move towards what\u0026rsquo;s going to look like the final grammar.\nFirst of all, let\u0026rsquo;s look how I would plug that filter-rspamd in my config:\nfilter rspamd proc-exec \u0026#34;/usr/local/bin/filter-rspamd\u0026#34; listen on all filter rspamd That\u0026rsquo;s all.\nI declared a filter named rspamd, instructed OpenSMTPD that it has to execute a proc filter from /usr/local/bin/filter-rspamd, and instructed the listener that all sessions handled through it should go through the rspamd filter.\nIf I didn\u0026rsquo;t want to use rspamd but a builtin filter to reject sessions without a forward-confirmed rDNS, I could have used:\nfilter fcrdns builtin connect check-fcrdns reject \u0026#34;550 go away you punk\u0026#34; listen on all filter fcrdns But what\u0026rsquo;s more interesting is the chaining of filters, which allows \u0026hellip; well, chaining filters:\nfilter fcrdns builtin connect check-fcrdns reject \u0026#34;550 go away you punk\u0026#34; filter rspamd proc-exec \u0026#34;/usr/local/bin/filter-rspamd\u0026#34; filter nazi_mode chain { fcrdns, rspamd } listen on all filter nazi_mode This, combined with the fact that the filters apply per-interface, allow for very flexible setups that could never be expressed on OpenSMTPD before.\nWhat next ? # The code and new grammar is working and committed in a branch, I intend to do some additional cleanup and code simplification before comitting to the OpenBSD tree hopefully this week.\nI will spend the remaining of the release cycle, until April, performing code cleanups, refining the reporting events and protocol, and maybe committing a few minor features (or maybe I have some other nice major features ready, who knows ;-)\nYou are HIGHLY encouraged to start playing with filters next week after my commit. There\u0026rsquo;s currently NO filter available, you can really be the first to do useful stuff for the community.\nStay tuned !\nOH AND MERRY X-MAS BECAUSE I WONT BE POSTING BEFORE THEN ;-)\n","date":"19 December 2018","permalink":"/posts/2018-12-19/more-on-opensmtpd-filters/","section":"Posts","summary":"TL;DR: Not this time, pal/gal, I took hours writing this post, you'll take a few minutes reading it all. Oh, and merry X-mas :-* A bit of short-sighted history # The filtering feature has been introduced only recently in OpenSMTPD, first presented on this blog a month ago.","title":"more on OpenSMTPD filters"},{"content":" TL;DR: I *FINALFUCKINGLY* commited proc filters support allowing full filtering in OpenSMTPD. eric@ implemented fc-rDNS lookups. fc-rDNS # fc-rDNS, or forward-confirmed reverse DNS, consists in performing a reverse DNS lookup to determine the hostname associated to an IP address\u0026hellip; then performing a DNS lookup on that hostname to check if it resolves back to the IP address.\nOn my request, eric@ implemented fc-rDNS lookups in our SMTP engine, causing OpenSMTPD to perform the double lookup upon clients connections.\nRight now, the fc-rDNS result is only passed to the reporting API but the idea is to allow using it in a builtin filter, something along the lines of filter smtp-in connect check-fcrdns disconnect \u0026quot;550 go away punk\u0026quot;.\nNote that this is particularly efficient at cutting spam, it essentially means that a spammer must control both the DNS and reverse DNS zone to pass the test, killing the bulk of zombified spam proxies living on infected home computers.\nThere is still some refining to do and the plugging of a builtin-filter, but the code is here and -current users will be able to test it soon\ntx-mail and tx-rcpt events # I have added two new reporting events for smtp-in and smtp-out: tx-mail and tx-rcpt.\nThey are generated whenever MAIL FROM or RCPT TO have received a result from the server:\nreport|1|1544130229|smtp-in|tx-mail|0f3004c08c82d33e|fc08ce7d|\u0026lt;owner-hackers+M85937=gilles=poolp.org@openbsd.org\u0026gt;|ok report|1|1544130229|smtp-in|tx-rcpt|0f3004c08c82d33e|fc08ce7d|\u0026lt;gilles@poolp.org\u0026gt;|ok A filter may listen for these events to determine which sender and recipients have been accepted in a transaction, something that would otherwise require keeping track of protocol-client and protocol-server events.\nproc filtering finally in # I have committed full proc filtering support today, allowing a standalone filter to perform all kind of filtering on every single phase of an SMTP session.\nThis means that with the code in -current, it is now possible to write a filter that talks to rspamd or that computes dkim signatures, something that was not doable until today without resorting to tricky setups.\nThe code is still a work in progress, we have things to clean up, improve and there are some known minor issues that we are working on. We will likely hit corner cases that will cause new issues\u0026hellip; but this is a fully working implementation, the real deal, not a PoC. Consider it as \u0026ldquo;in a process of stabilization to be production ready by April\u0026rdquo;.\nWhen we have stabilized the API, I\u0026rsquo;ll take time to write about how it works internally, for now let\u0026rsquo;s just celebrate because I FINALLY FUCKING COMMITED FILTERS.\npython bridge, proof of concept # As I wrote in a previous post, proc filters are programs reading on stdin and writing to stdout, making it possible write them in any language.\nI put my code where my mouth is:\nimport opensmtpd def link_connect(timestamp, session_id, args): rdns, fcrdns, laddr, raddr = args def link_disconnect(timestamp, session_id, args): _ = args def tx_begin(timestamp, session_id, args): tx_id = args[0] def tx_mail(timestamp, session_id, args): tx_id, address, status = args def tx_rcpt(timestamp, session_id, args): tx_id, address, status = args def tx_envelope(timestamp, session_id, args): tx_id, evp_id = args def tx_rollback(timestamp, session_id, args): tx_id = args[0] def tx_commit(timestamp, session_id, args): tx_id, nbytes = args def protocol_client(timestamp, session_id, args): line = args[0] if __name__ == \u0026#34;__main__\u0026#34; o = opensmtpd.SMTP_IN() o.on_report(\u0026#39;link-connect\u0026#39;, link_connect) o.on_report(\u0026#39;link-disconnect\u0026#39;, link_disconnect) o.on_report(\u0026#39;tx-begin\u0026#39;, tx_begin) o.on_report(\u0026#39;tx-mail\u0026#39;, tx_mail) o.on_report(\u0026#39;tx-rcpt\u0026#39;, tx_rcpt) o.on_report(\u0026#39;tx-envelope\u0026#39;, tx_envelope) o.on_report(\u0026#39;tx-commit\u0026#39;, tx_commit) o.on_report(\u0026#39;tx-rollback\u0026#39;, tx_rollback) o.on_report(\u0026#39;protocol-client\u0026#39;, protocol_client) o.on_report(\u0026#39;protocol-server\u0026#39;, protocol_server) o.run() This only covers the callback API for \u0026ldquo;report\u0026rdquo; events, but it does so thanks to an opensmtpd package that consists of less than 100 lines of code.\nI will be implementing the on_filter callback API probably next week, so I can start writing the filters I need in python.\nIf you want to discuss how to write an interface for the language of your choice, feel free to jump to our IRC channel, #opensmtpd @ freenode ;-)\nWhat next ? # Essentially code cleanup and simplification, API stabilization and writing filters to spot improvements required in the API.\nStay tuned !\n","date":"6 December 2018","permalink":"/posts/2018-12-06/opensmtpd-proc-filters-fc-rdns/","section":"Posts","summary":"TL;DR: I *FINALFUCKINGLY* commited proc filters support allowing full filtering in OpenSMTPD. eric@ implemented fc-rDNS lookups. fc-rDNS # fc-rDNS, or forward-confirmed reverse DNS, consists in performing a reverse DNS lookup to determine the hostname associated to an IP address\u0026hellip; then performing a DNS lookup on that hostname to check if it resolves back to the IP address.","title":"OpenSMTPD proc filters \u0026 fc-rDNS"},{"content":" TL;DR: The reporting mechanism has been described shortly in my previous article about both reporting and filters. Let's focus a bit more on the reporting bits this time. The format is improving further and has extended to outgoing trafic reporting. Reporting # In previous article, I described the events reporting mechanism that has been introduced in the development branch of OpenSMTPD.\nTo sum it up, you could now write an event processor as simple as a shell script reading its stdin on a loop:\n$ cat /tmp/reporting.sh #! /bin/sh # while read line; do echo $line \u0026gt;\u0026gt; /tmp/reporting.log done and configure your OpenSMTPD so it would report all incoming SMTP events:\n$ grep report /etc/mail/smtpd.conf proc reporting \u0026#34;/tmp/reporting.sh\u0026#34; report smtp on reporting which would then produce an events report log in /tmp/reporting.log containing entries similar to these:\nreport|smtp-in|link-connect|1541271219|3189ac6874354895|localhost|127.0.0.1:41564|127.0.0.1:25 report|smtp-in|protocol-server|1541271219|3189ac6874354895|220 poolp.org ESMTP OpenSMTPD report|smtp-in|protocol-client|1541271222|3189ac6874354895|helo localhost report|smtp-in|protocol-server|1541271222|3189ac6874354895|250 poolp.org Hello localhost [127.0.0.1], pleased to meet you report|smtp-in|link-disconnect|1541271224|3189ac6874354895 Improvements on the reports format # I made several improvements to the format described in the previous article.\nThe first improvement is that there is now a version embedded in each report. This allows event processors to be able to check if they know how to parse an event report, it allows them to easily support backward-compatible versions should we make changes to the format of an event report, but most importantly it allows you to just store these events somewhere then have tools post-process them months later without ambiguity with regard to the format of entries, even if there were OpenSMTPD updates in between.\nThe second improvement is that the timestamp which was appearing after the event type was moved in front of it. This doesn\u0026rsquo;t seem like an improvement but it eases reading and allows me to simplify some of the code :-)\nThe third improvement comes from adding some new events and adding information to some existing events. For instance, OpenSMTPD reported these transaction events:\nreport|smtp-in|tx-begin|1541271225|3189ac6874354895 report|smtp-in|tx-commit|1541271225|3189ac6874354895 report|smtp-in|tx-rollback|1541271225|3189ac6874354895 But reading from these, you could only obtain the session identifier, there was no way to find out the transaction identifier, how many envelopes were generated in the transaction or even the size of the message.\nTo solve these issues, the format of the events above has been extended so it would contain the transation identifier (aka. msgid), a tx-envelope event was introduced to report the generation of a new envelope in the transaction along with its envelope identifier (aka. evpid), and finally the size of the DATA part is reported on a tx-commit event.\nLast but not least, the DATA part begins with a DATA command issued by the client and ends with a single . on a line by itself. Despite the single . being sent within the DATA phase, it is not really part of the DATA itself and must be considered as a commit request.\nThis doesn\u0026rsquo;t seem like much but the devil is in the details. Not reporting that commit request as a protocol-client command means that we go straight from DATA command to a tx-commit event, without allowing a filter to actually refuse the commit request. If we generate this commit request event, a filter may decide that it wants to reject it which will then produce a tx-rollback that was not possible before.\nA pattern emerges that tx-* events should appear in between protocol-* events otherwise they cannot be filtered.\nHere is a sample curated event report log from my own server as of today:\n$ cat /tmp/reporting.log report|1|1541750432|smtp-in|link-connect|c73c0aff0dfb6250|poolp.org|local:0|local:0 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|220 poolp.org ESMTP OpenSMTPD report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|EHLO localhost report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250-poolp.org Hello localhost [local], pleased to meet you report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250-8BITMIME report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250-ENHANCEDSTATUSCODES report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250-SIZE 36700160 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250 HELP report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; report|1|1541750432|smtp-in|tx-begin|c73c0aff0dfb6250|f84306b3 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250 2.0.0: Ok report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|RCPT TO:\u0026lt;gilles@poolp.org\u0026gt; report|1|1541750432|smtp-in|tx-envelope|c73c0aff0dfb6250|f84306b3|f84306b34f388082 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250 2.1.5 Destination address valid: Recipient ok report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|DATA report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|354 Enter mail, end with \u0026#34;.\u0026#34; on a line by itself report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|. report|1|1541750432|smtp-in|tx-commit|c73c0aff0dfb6250|f84306b3|301 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250 2.0.0: f84306b3 Message accepted for delivery report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|QUIT report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|221 2.0.0: Bye report|1|1541750432|smtp-in|link-disconnect|c73c0aff0dfb6250 Introducing smtp-out # Obviously my plan is to be able to report and create dashboard for ALL trafic, not just incoming.\nI have worked on generating reports for smtp-out and I actually have something working in a branch, which I intend to commit next week.\nThe format is identical with the sole difference that smtp-in is replaced with smtp-out, the event types are the same, the parameters are the same, you just need to get your head around the fact that protocol-client is your peer when in smtp-in, whereas protocol-server is your peer when in smtp-out:\nreport|1|1541750707|smtp-out|link-connect|c73c0b206ad6c41c||:0|64.233.167.26:25 report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|220 mx.google.com ESMTP k132-v6si807155wma.16 - gsmtp report|1|1541750707|smtp-out|protocol-client|c73c0b206ad6c41c|EHLO out.mailbrix.mx report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-mx.google.com at your service, [212.83.129.132] report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-SIZE 157286400 report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-8BITMIME report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-STARTTLS report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-ENHANCEDSTATUSCODES report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-PIPELINING report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-CHUNKING report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250 SMTPUTF8 report|1|1541750707|smtp-out|protocol-client|c73c0b206ad6c41c|STARTTLS report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|220 2.0.0 Ready to start TLS report|1|1541750707|smtp-out|link-tls|c73c0b206ad6c41c|version=TLSv1.2, cipher=ECDHE-RSA-CHACHA20-POLY1305, bits=256 report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|EHLO out.mailbrix.mx report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-mx.google.com at your service, [212.83.129.132] report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-SIZE 157286400 report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-8BITMIME report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-ENHANCEDSTATUSCODES report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-PIPELINING report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-CHUNKING report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250 SMTPUTF8 report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250 2.1.0 OK k132-v6si807155wma.16 - gsmtp report|1|1541750708|smtp-out|tx-begin|c73c0b206ad6c41c|8ea46ad1 report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|RCPT TO:\u0026lt;gilles.chehade@gmail.com\u0026gt; report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250 2.1.5 OK k132-v6si807155wma.16 - gsmtp report|1|1541750708|smtp-out|tx-envelope|c73c0b206ad6c41c|8ea46ad1|8ea46ad1ccae4934 report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|DATA report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|354 Go ahead k132-v6si807155wma.16 - gsmtp report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|. report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250 2.0.0 OK 1541750708 k132-v6si807155wma.16 - gsmtp report|1|1541750708|smtp-out|tx-commit|c73c0b206ad6c41c|8ea46ad1|818 report|1|1541750718|smtp-out|protocol-client|c73c0b206ad6c41c|QUIT report|1|1541750718|smtp-out|protocol-server|c73c0b206ad6c41c|221 2.0.0 closing connection k132-v6si807155wma.16 - gsmtp report|1|1541750718|smtp-out|link-disconnect|c73c0b206ad6c41c Because not everyone needs reporting and not everyone needs both incoming and outgoing reporting, I have added the smtp-in and smtp-out keywords to the grammar so that you can:\n$ grep report /etc/mail/smtpd.conf proc reporting \u0026#34;/tmp/reporting.sh\u0026#34; report smtp-in on reporting report smtp-out on reporting I\u0026rsquo;ll probably make report smtp on a shortcut for both smtp-in and smtp-out.\nThere is still work to be done on the smtp-out path because the SMTP engine is more complex than for the smtp-in path. For instance, it is currently not possible to have any of the transaction events generated between the protocol-client and protocol-server events due to how the state machine is written. Not really as much of a big deal as for smtp-in since smtp-out isn\u0026rsquo;t filtered and the order issues are less annoying, but to be really clean and consistent, the smtp-in and smtp-out reports should be very parallel in terms of order events. I should be able to look at the smtp-out reports from my laptop and the smtp-in reports from my server and see them appear in the same order.\nFinally, there is also an rDNS lookup that needs to be added so the report is identical to smtp-in, and we should be fine.\nWhat\u0026rsquo;s so good about this ? # Reporting is not JUST about being able to write dashboards, it is not just about being able to generate state for filters eithers.\nGenerating event reports logs that can be parsed by external tools open the way for many side applications ranging from tools to replay sessions when tracking issues, tools to analyze behavior of peers and feedback into pf or OpenSMTPD tables, and more interestingly for people who will developer filters\u0026hellip; it brings the ability to write and test a filter without a running OpenSMTPD instance, piping the event log directly into the filter.\nTo be very honest, I\u0026rsquo;m personally more excited by this new feature than the filters feature which might be more visible but would be far less powerful without the event logs.\nWhat next ? # More changes should happen to the format of entries in the next few weeks and months, this is a moving target as I wrote in previous article.\nBuiltin filters already require some of these lines to provide more informations and this is being worked on.\nMy next focus is the filtering of the DATA phase which is the requirement for us to provide support for dkim and antispam stuff without the need of proxies and reenqueuing. Work has already started but I will probably not commit any code related to this before the end of November.\n","date":"9 November 2018","permalink":"/posts/2018-11-09/opensmtpd-reporting-update/","section":"Posts","summary":"TL;DR: The reporting mechanism has been described shortly in my previous article about both reporting and filters. Let's focus a bit more on the reporting bits this time.","title":"OpenSMTPD reporting update"},{"content":" TL;DR: Filters have been a (the most ?) long awaited feature in OpenSMTPD. I finally committed most of the filters code to OpenBSD. There is still a bit of work required but the trickiest parts are done. This article describes how filters are implemented and what to expect. OpenSMTPD 6.4.0 was released ! # We have released OpenSMTPD 6.4.0 last week without filters.\nI won\u0026rsquo;t expand on the features in the 6.4 release as I already wrote about the configuration file changes, the issues that required it and the refactors involved, this was the one true major feature of the release.\nOne notable aspect though is that we dropped our support for OpenSSL in favor of LibreSSL, and THAT I should expand upon ;-)\nLibreSSL FTW ! # OpenSMTPD was started long before LibreSSL and from the very first portable version, we had to include ifdefs to accomodate the differences between the different OpenSSL versions across Linux distros: some had SNI, others don\u0026rsquo;t, some had GCM, others don\u0026rsquo;t.\nWhen LibreSSL was forked from OpenSSL after a large cleanup of the code, we thought we\u0026rsquo;d just depend on that because it was cleaner and also we knew that this version had all the features we relied on. Also, there was this plan for a libtls wrapper which would be much simpler to use and less error-prone than libssl.\nIt turned out that it wasn\u0026rsquo;t possible because OpenSSL and LibreSSL couldn\u0026rsquo;t coexist on many systems. The fact that they shared the same library names (libssl, libcrypto), and shared object versionning, caused issues with the confused runtime link editor. We decided to accomodate both, keeping checks in the configure layer to detect what was being used, etc\u0026hellip; It was horrible. It kept breaking every now and then due to either one making changes. I had to deal with these breakages. It was horrible.\nBefore the 6.4 release, I did the portability build passes and yet again it broke on new OpenSSL conflicts with our grammar. Three different people using different distros had sent me three different diffs which fixed the issues for them, but which I couldn\u0026rsquo;t merge without creating a delta with the native branch which was not acceptable for us. The more I tried fixing and the more I was irritated by this OpenSSL/LibreSSL hacks.\nI did a bit of investigation and it turns out the issues that prevented them from coexisting were no longer relevant. I verified by building, linking and running OpenSMTPD against LibreSSL on FreeBSD, Ubuntu, Debian, CentOS, ArchLinux and Fedora. It is technically possible to make OpenSMTPD depend on LibreSSL without forcing distros to switch from OpenSSL to LibreSSL, so there\u0026rsquo;s no reason for us not to depend on LibreSSL anymore.\nThis allowed me to start removing ifdefs from our code, removing useless tests from the configure.ac and assume that we have the features we need for all systems.\nThis doesn\u0026rsquo;t mean that people can\u0026rsquo;t link OpenSMTPD against OpenSSL, it just means that they get to handle the diffs to their own version, the extra work of maintaining OpenSSL goes out of our hands.\nFilters # We started working on filters years ago, I even wrote an article about them in 2014.\nA first implementation was written. After a while, it became clear that the design was wrong and causing unfixable bugs in some filters. I decided to pull the plug on that attempt and we removed all of the code.\nA year ago, during EuroBSDCon 2017, I mentionned the filtering daemon we had worked on and which solved these limitations. But as OpenBSD 6.4 release was getting closer, I started to dislike that idea because I believed it was not addressing the proper problem: we had came up with a working solution, not the best solution to the problem. The more I spent thinking about this, the more the problem looked different from what we initially modeled.\nI discussed my concerns with Eric, told him that I thought we had made a mistake in modeling filters, that by thinking them differently we could come up with a much simpler solution, and somehow managed to convince him I was right :-)\nGiven we took so much time already, it made sense delaying for a release so we would be confident about our technical choice.\nI worked on a proof of concept for the new model and a week later I had filters working on my laptop, a convinced eric and\u0026hellip; many months to make it bright and shiny because it was obviously not going to make it into 6.4\nEvent reports and filter request # The new filters are modeled around the notions of event reports and filter requests.\nDuring the lifetime of a session, an SMTP engine will report various events such as the beginning of a new connection, the negotiation of ciphers during the TLS handshake or even the beginning of an SMTP transaction with MAIL FROM. These events help the SMTP engine build a state for a session and determine what is acceptable or not from a client at a given time.\nSome filters may not need a state, they may operate on the parameter to a command and that\u0026rsquo;s all. Other filters however may need to build a state for a session, not necessarily the same exact and complete state as the SMTP engine, but a state that makes sense given the work the filter will do.\nThe event reports are informative messages, which a filter may decide to process or not, and that encompass ALL of the SMTP events a session goes through. It is possible to replay an entire SMTP session based on these reports.\nThe filter requests however are not informative. A filter will be configured to handle a particular filtering phase and will receive filter requests for that phase, containing the phase and the parameter to filter, it will then be REQUIRED to answer the request with an action to take.\nAgain, simple filters may only deal with filter requests, more complex filters may process the event reports to learn about sessions before dealing with filter requests.\nEvent processors # Event processors are standalone executables which \u0026hellip; process event reports and filter requests in an infinite loop.\nOpenSMTPD executes them at startup and sets up an environment so they behave like traditional Unix-filters: they read input from stdin and write output to stdout.\nBoth event reports and filter requests will consist of single lines, each consisting of multiple fields separated by a pipe symbol |. All lines will hold a generic set of fields such as event type and session identifier, as well as a set of event-specific fields such as an ip address for connection event, an email address for mail-from event, etc\u0026hellip;\nEvent reports will expect no response, meaning that for a line read on stdin there will be no line written on stdout, whereas filter requests will expect a response, meaning that for a line read on stdin there will be one line written to stdout.\nThe simplest processor to write is one reading events to log them:\n#! /bin/sh # while read line; do echo $line \u0026gt;\u0026gt; /tmp/output.log done If you have ever parsed OpenSMTPD logs to inject them into an ELK or similar, you should now be in love with the events reporting feature.\nFilters are not much more complex, the only difference is that $line needs to be split to extract the session identifier, and the processor needs to write back its decision.\nThe simplest filtering processor to write is one that accepts every filter-request:\n#! /bin/sh # while read line; do if echo $line | grep \u0026#39;^filter-request|\u0026#39;; then SESSION=`echo $line | cut -d\\| -f4` echo \u0026#34;filter-response|${SESSION}|proceed\u0026#34; fi done Note that while this example shows a synchronous protocol where a filter-request gets an immediate answer, the implementation does not impose that limitation. There should be a filter-response for each filter-request, but they do not need to happen right away and they do not need to happen in the same sequence order.\nNow that you\u0026rsquo;re all sold on this solution, here are the bonus advantages:\nOpenSMTPD only knows that it has to execute the processor and write to its stdin. There are no dependencies and no limitations to what language a processor is written in, a postmaster does not have to learn C to write a filter but can effectively hack something fast with a few lines of shell or python.\nThe events processors are forked: they do not share the same memory space as OpenSMTPD, they do not share the same memory space as each other, and on systems that provide randomized memory layout they do not even share the same memory layout. On OpenBSD, each processor could benefit from its own pledge() and its own unveil(). A compromise of either one does not imply the compromission of the others or of the daemon.\nIn addition, they can be executed with different privileges and including chrooted() to a particular directory:\nproc filter1 \u0026#34;/tmp/filter1.sh\u0026#34; user _filter1 proc filter2 \u0026#34;/tmp/filter2.sh\u0026#34; group _filter2 proc filter3 \u0026#34;/tmp/filter3.sh\u0026#34; user _filter3 group gilles proc filter4 \u0026#34;/filter4\u0026#34; user _filter4 chroot \u0026#34;/tmp\u0026#34; Event proc # Event processors can receive event reports for each events generated by the SMTP engine (for now).\nThis allows two things. First, it allows fine-grained reporting to specialized software that can extract metrics, generate fancy dashboards and such.\nUntil now, people had to try and extract this information from our log files which were meant to be consumed by humans primarily. The event logs provides all of the useful informations in a format that can be easily parsed by scripts. Converting these entries to the proper format for injecting in a timeseries database or similar becomes trivial.\nThen, some filters may be able to work without any context by looking solely at the current command, but in many cases a decision at a particular phase may be the consequence of decisions at earlier phases, and filters can consume these event reports to build their own view of the state of sessions.\nHere is a sample of event reports:\nreport|smtp-in|link-connect|1541271219|3189ac6874354895|localhost|127.0.0.1:41564|127.0.0.1:25 report|smtp-in|protocol-server|1541271219|3189ac6874354895|220 poolp.org ESMTP OpenSMTPD report|smtp-in|protocol-client|1541271222|3189ac6874354895|helo localhost report|smtp-in|protocol-server|1541271222|3189ac6874354895|250 poolp.org Hello localhost [127.0.0.1], pleased to meet you report|smtp-in|link-disconnect|1541271224|3189ac6874354895 And this is how you define an event proc in smtpd.conf:\nproc my_event_proc \u0026#34;/usr/local/bin/script.sh\u0026#34; report smtp on my_event_proc This gets OpenSMTPD to run the script at startup and to pipe the event reports to its stdin.\nFilter proc # Filter processors are very similar to event processors in how they work, excepted that they are registered for specific filtering phases:\nproc my_filter_proc \u0026#34;/usr/local/bin/script.sh\u0026#34; # line below can be uncommented if filters needs event reports to build a state # report smtp on my_filter_proc filter smtp helo on my_filter_proc filter smtp ehlo on my_filter_proc filter smtp mail-from on my_filter_proc filter smtp rcpt-to on my_filter_proc The filters will receive phase-specific parameters:\nfilter-request|smtp-in|ehlo|e68808ff61812a6f|laptop.home|localhost filter-request|smtp-in|mail-from|e68808ff61812a6f|\u0026lt;gilles@laptop.home\u0026gt; filter-request|smtp-in|rcpt-to|e68808ff61812a6f|\u0026lt;gilles@laptop.home\u0026gt; filter-request|smtp-in|ehlo|e688090368c275a3|laptop.home|localhost filter-request|smtp-in|mail-from|e688090368c275a3|\u0026lt;gilles@laptop.home\u0026gt; filter-request|smtp-in|rcpt-to|e688090368c275a3|\u0026lt;gilles@laptop.home\u0026gt; To which they will have to reply with a decision:\nfilter-response|e68808ff61812a6f|proceed filter-response|e68808ff61812a6f|rewrite|BLEH filter-response|e68808ff61812a6f|reject|550 go away filter-response|e68808ff61812a6f|disconnect|550 go away The only possible decisions are proceed, rewrite, reject or disconnect.\nBuiltin filters # Filter proc are very useful when a decision relies on anything else than the parameter to the command being filtered, when it depends on information gathered from previous events, or even when it depends on complex logic involving multiple lookups.\nIn many cases though, a decision to filter may rely solely on the current command and a simple table lookup. For instance, one may want to reject a HELO hostname that\u0026rsquo;s not part of a table, or reject a MAIL FROM that matches a regular expression.\nFor these cases, OpenSMTPD provides a small set of builtin filters which are fairly generic to be applied to all phases.\ntable helo-reject { \u0026#34;foobar\u0026#34;, \u0026#34;barbaz\u0026#34;, \u0026#34;bazqux\u0026#34; } table helo-reject-regex { \u0026#34;^f[oO]o[bB]ar$\u0026#34; } filter smtp helo check-table \u0026lt;helo-reject\u0026gt; reject \u0026#34;550 go away\u0026#34; filter smtp helo check-regex \u0026#34;^go\\-away$\u0026#34; reject \u0026#34;550 go away\u0026#34; filter smtp helo check-regex \u0026lt;helo-reject-regex\u0026gt; reject \u0026#34;550 go away\u0026#34; We will also implement some very limited additional phase-specific builtin filters that cover common use-cases. OpenSMTPD performs a reverse DNS lookup on connect, both connect and helo/ehlo filter phases commonly check reverse DNS, so we already provided a check-rdns for these phases:\nfilter smtp connect check-rdns reject \u0026#34;550 you need a reverse DNS\u0026#34; filter smtp ehlo check-rdns reject \u0026#34;550 your HELO hostname and rDNS mismatch\u0026#34; There shouldn\u0026rsquo;t be many builtin filters, we don\u0026rsquo;t expect more than a handful of them, but they do exist so there\u0026rsquo;s that ;-)\nWhat next ? # This code has been committed to OpenBSD and will be available in OpenSMTPD 6.5.0, which should happen sometime around April 2019.\nI haven\u0026rsquo;t documented anything yet because both configuration grammar and protocol are still going through lots of changes, I would strongly suggest against playing with event reports and filter procs before March, unless you\u0026rsquo;re a developer and happy with having to make changes to your code every few days.\nThe only part that I have not committed yet is the filtering of DATA, which still requires a bit of work but will be working and committed sometime in November.\nOnce I have this part done, I\u0026rsquo;ll implement a few filters for the use-cases I have such as filter-spf and filter-rspamd, then it will be ok as far as I\u0026rsquo;m concerned.\nMy main work for the release cycle to come is finishing and polishing filters, fixing the portable layer which has grown a monster, and ultimately doing a full release cycle of\u0026hellip; cosmethic cleanup so code is pleasing to read ;-)\n","date":"3 November 2018","permalink":"/posts/2018-11-03/opensmtpd-released-and-upcoming-filters-preview/","section":"Posts","summary":"TL;DR: Filters have been a (the most ?) long awaited feature in OpenSMTPD. I finally committed most of the filters code to OpenBSD. There is still a bit of work required but the trickiest parts are done.","title":"OpenSMTPD released and upcoming filters preview"},{"content":" TL;DR: Switching to new config is not too hard and can be done in minutes. The new config is also a new queue that is not backwards compatible. The easiest way is to flush the mail queue before switching. We came up with a solution to help maintainers of more complex setups. Switching from old config to new config # The new OpenSMTPD configuration grammar is slightly different from the current one, rules are no longer stated as single lines, but the conversion from previous ruleset to new ruleset isn\u0026rsquo;t that hard.\nLet\u0026rsquo;s do the exercise with poolp.org\u0026rsquo;s smtpd.conf which is a fairly complex ruleset, making use of several features including TLS, authentication, multi-domain hosting with primary and virtual domains, different aliases mappings, backup MX, relaying through a DKIM proxy, and more\u0026hellip;\npki mx1.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mx1.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; pki mail.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mail.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; table sources { 212.83.181.8 } table helonames { 212.83.181.8 = mx1.poolp.org } table aliases \u0026#34;/etc/mail/aliases\u0026#34; table opensmtpd-aliases \u0026#34;/etc/mail/aliases-opensmtpd.org\u0026#34; table pdomains \u0026#34;/etc/mail/primary-domains\u0026#34; table vdomains \u0026#34;/etc/mail/virtual-domains\u0026#34; table vusers \u0026#34;/etc/mail/virtual-users\u0026#34; table bdomains \u0026#34;/etc/mail/backup-domains\u0026#34; table shithole\t{ \u0026#34;@qq.com\u0026#34; } listen on lo0 listen on lo0 port 10028 tag DKIM listen on egress tls-require pki mx1.poolp.org hostnames { 212.83.181.7 = mail.poolp.org, 212.83.181.8 = mx1.poolp.org } listen on egress smtps pki mail.poolp.org auth hostname mail.poolp.org listen on egress port submission tls-require pki mail.poolp.org auth hostname mail.poolp.org reject from any sender \u0026lt;shithole\u0026gt; for any accept for local alias \u0026lt;aliases\u0026gt; deliver to maildir accept from any for domain \u0026lt;pdomains\u0026gt; alias \u0026lt;aliases\u0026gt; deliver to maildir accept from any for domain opensmtpd.org alias \u0026lt;opensmtpd-aliases\u0026gt; deliver to maildir accept from any for domain \u0026lt;vdomains\u0026gt; virtual \u0026lt;vusers\u0026gt; deliver to maildir accept from any for domain \u0026lt;bdomains\u0026gt; relay backup mx1.poolp.org accept tagged DKIM for any relay source \u0026lt;sources\u0026gt; hostnames \u0026lt;helonames\u0026gt; accept for any relay via smtp://127.0.0.1:10027 Fixing pki directives # First of all, the pki directives which contain the certificates and private keys are not really affected. They work exactly as with the old grammar, we only shortened the certificate keyword to cert by popular demand. This results in:\npki mx1.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mx1.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; pki mail.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mail.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; being rewritten as:\npki mx1.poolp.org cert \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mx1.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; pki mail.poolp.org cert \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mail.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; The ca directive which allows setting a custom CA certificate, and which is not used in this configuration file, is subjected to the same change so if you have a directive such as:\nca mail.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.ca\u0026#34; all you have to do is replace certificate with cert:\nca mail.poolp.org cert \u0026#34;/etc/ssl/poolp.org.ca\u0026#34; Other pki options are unchanged and default to the sanest option.\nFixing table directives # No changes in table directives:\ntable sources { 212.83.181.8 } table helonames { 212.83.181.8 = mx1.poolp.org } table aliases \u0026#34;/etc/mail/aliases\u0026#34; table opensmtpd-aliases \u0026#34;/etc/mail/aliases-opensmtpd.org\u0026#34; table pdomains \u0026#34;/etc/mail/primary-domains\u0026#34; table vdomains \u0026#34;/etc/mail/virtual-domains\u0026#34; table vusers \u0026#34;/etc/mail/virtual-users\u0026#34; table bdomains \u0026#34;/etc/mail/backup-domains\u0026#34; table shithole\t{ \u0026#34;@qq.com\u0026#34; } Fixing listen directives # The listen directive works as before, however like for pki some keywords were shortened. For instance, mask-source, which is not used in my config, was shortened to the more compact versions mask-src.\nIn my smtpd.conf, there was no change to listen directives despite using a large subset of listen features:\nlisten on lo0 listen on lo0 port 10028 tag DKIM listen on egress tls-require pki mx1.poolp.org hostnames { 212.83.181.7 = mail.poolp.org, 212.83.181.8 = mx1.poolp.org } listen on egress smtps pki mail.poolp.org auth hostname mail.poolp.org listen on egress port submission tls-require pki mail.poolp.org auth hostname mail.poolp.org Fixing the ruleset # This leaves us with the complex part of the change, switching from the one-line rules that used to define a decision, an envelope matching pattern and an action as a whole, to the new two-line rules defining a set of valid actions and a distinct set of matching patterns referencing the actions.\nDon\u0026rsquo;t worry, be happy. The change is mechanical and doesn\u0026rsquo;t need to completely rethink how you used to do your configuration files. The rules are now split into two parts, the actions and the matching patterns. The actions must be declared before matching patterns but may be declared in any order. The matching patterns work like previous ruleset, they are listed in \u0026lsquo;first match wins\u0026rsquo; order and therefore require the most specific rules first as the ruleset is essentially cascading on mismatches. There is no change whatsoever to that logic, so converting a former ruleset to a new ruleset is basically\u0026hellip; splitting previous ruleset into action and match directives with the match directives following the exact same order as with current ruleset.\nIn my case, the following ruleset needs to be rewritten:\nreject from any sender \u0026lt;shithole\u0026gt; for any accept for local alias \u0026lt;aliases\u0026gt; deliver to maildir accept from any for domain \u0026lt;pdomains\u0026gt; alias \u0026lt;aliases\u0026gt; deliver to maildir accept from any for domain opensmtpd.org alias \u0026lt;opensmtpd-aliases\u0026gt; deliver to maildir accept from any for domain \u0026lt;vdomains\u0026gt; virtual \u0026lt;vusers\u0026gt; deliver to maildir accept from any for domain \u0026lt;bdomains\u0026gt; relay backup mx1.poolp.org accept tagged DKIM for any relay source \u0026lt;sources\u0026gt; hostnames \u0026lt;helonames\u0026gt; accept for any relay via smtp://127.0.0.1:10027 First of all, we need to identify what are the unique actions within that ruleset, and this boils down to \u0026ldquo;aliases/virtual/userbase and anything after relay, deliver to\u0026rdquo;:\nalias \u0026lt;aliases\u0026gt; deliver to maildir alias \u0026lt;opensmtpd-aliases\u0026gt; deliver to maildir virtual \u0026lt;vusers\u0026gt; deliver to maildir relay backup mx1.poolp.org relay source \u0026lt;sources\u0026gt; hostnames \u0026lt;helonames\u0026gt; relay via smtp://127.0.0.1:10027 The new grammar for these will result in:\naction act01 maildir alias \u0026lt;aliases\u0026gt; action act02 maildir alias \u0026lt;opensmtpd-aliases\u0026gt; action act03 maildir virtual \u0026lt;vusers\u0026gt; action act04 relay backup mx mx1.poolp.org action act05 relay src \u0026lt;sources\u0026gt; helo-names \u0026lt;helonames\u0026gt; action act06 relay host smtp://127.0.0.1:10027 Now that all actions have been defined, we need to identify what are the matching patterns within the former ruleset, rewrite them with new grammar and attach them to their respective action. The change is very mechanical but keywords were shortened and made less ambiguous:\nmatch from any mail-from \u0026lt;shithole\u0026gt; for any reject match for local action act01 match from any for domain \u0026lt;pdomains\u0026gt; action act01 match from any for domain opensmtpd.org action act02 match from any for domain \u0026lt;vdomains\u0026gt; action act03 match from any for domain \u0026lt;bdomains\u0026gt; action act04 match tag DKIM for any action act05 match for any action act06 There is exactly as many match rules as there used to be accept+reject rules, they are in the same order, they perform the same action, they are just expressed differently.\nPutting it all together # This is the resulting smtpd.conf for poolp.org which should be considerably more complex thant most setups, we use a large part of OpenSMTPD\u0026rsquo;s feature across all domains, most setups will have two or three actions at most:\npki mx1.poolp.org cert \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mx1.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; pki mail.poolp.org cert \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mail.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; table sources { 212.83.181.8 } table helonames { 212.83.181.8 = mx1.poolp.org } table aliases file:/etc/mail/aliases table opensmtpd-aliases file:/etc/mail/aliases-opensmtpd.org table pdomains file:/etc/mail/primary-domains table vdomains file:/etc/mail/virtual-domains table vusers file:/etc/mail/virtual-users table bdomains file:/etc/mail/backup-domains table shithole\t{ \u0026#34;@qq.com\u0026#34; } listen on lo0 listen on lo0 port 10028 tag DKIM listen on egress tls-require pki mx1.poolp.org hostnames { 212.83.181.7 = mail.poolp.org, 212.83.181.8 = mx1.poolp.org } listen on egress smtps pki mail.poolp.org auth hostname mail.poolp.org listen on egress port submission tls-require pki mail.poolp.org auth hostname mail.poolp.org action act01 maildir alias \u0026lt;aliases\u0026gt; action act02 maildir alias \u0026lt;opensmtpd-aliases\u0026gt; action act03 maildir virtual \u0026lt;vusers\u0026gt; action act04 relay backup mx mx1.poolp.org action act05 relay src \u0026lt;sources\u0026gt; helo-src \u0026lt;helonames\u0026gt; action act06 relay host smtp://127.0.0.1:10027 match from any mail-from \u0026lt;shithole\u0026gt; for any reject match for local action act01 match from any for domain \u0026lt;pdomains\u0026gt; action act01 match from any for domain opensmtpd.org action act02 match from any for domain \u0026lt;vdomains\u0026gt; action act03 match from any for domain \u0026lt;bdomains\u0026gt; action act04 match tag DKIM for any action act05 match for any action act06 Easing the conversion # The smtpd.conf.5 man page has been adapted but given that it is essentially a rewrite, it is possible that we missed stuff and we will do our best to bring them to the usual quality you expect from OpenBSD man pages.\nYou can find here a sample smtpd.conf that lists all directives the configuration parser supports. The file is not exhaustive as this is not doable given the flexibility of the pattern matching, but if you have a doubt on how to rewrite an old rule to a new rule this along the man page should be enough to help you.\nThe #OpenSMTPD channel on irc.freenode.net, as well as our mailing lists are also available to help you do the conversion.\nThe queue is not backward compatible # Just switching from old config to new config and restarting is not going to work.\nWhile the grammar change is very mechanical and seems to be very close to previous grammar, the underlying structures have drastically changed, the action resolving too and the result is that on-disk envelopes are not compatible and CAN\u0026rsquo;T be made compatible. There\u0026rsquo;s no way we can convert them because there\u0026rsquo;s runtime resolving logic that expects to find an information we can\u0026rsquo;t make up out of nowhere.\nSo there are two conversion paths.\nThe first one, the prefered of course, is to flush the mail queue before upgrading to new smtpd. You pause incoming sessions with smtpctl pause smtp so your primary MX stops accepting mail, which is never an issue because you have a secondary MX of course, and you issue a smtpctl schedule all so all pending mails are sent right away. When the mail queue is empty, which you can check with smtpctl show queue, you stop the daemon, upgrade and start the new one on your new config.\nThe second one is for busy mail hosts that can\u0026rsquo;t flush a mail queue. This is basically the case for a mailing list server which can almost never flush a queue empty due to unreachable hosts, remote host limits requiring the server to be down for an extended period of time, etc\u0026hellip; For these, we came up with the smtpd-salvage daemon.\nBasically, OpenSMTPD has envelopes versionning so a new config OpenSMTPD will not process old envelopes it can\u0026rsquo;t support, and an old config OpenSMTPD will not process the new envelopes it can\u0026rsquo;t support. If you can\u0026rsquo;t flush the queue, start the new config OpenSMTPD which will skip old envelopes and start creating new envelopes, then start smtpd-salvage with the older config. The smtpd-salvage daemon will then drain the queue from old messages. After a few days, all messages that weren\u0026rsquo;t delivered should be expired so smtpd-salvage can be stopped and deleted forever.\nUSE queue flush if you can.\n","date":"21 May 2018","permalink":"/posts/2018-05-21/switching-to-opensmtpd-new-config/","section":"Posts","summary":"TL;DR: Switching to new config is not too hard and can be done in minutes. The new config is also a new queue that is not backwards compatible.","title":"switching to OpenSMTPD new config"},{"content":" TL;DR: OpenBSD #p2k18 hackathon took place at Epitech in Nantes. I was organizing the hackathon but managed to make progress on OpenSMTPD. As mentionned at EuroBSDCon the one-line per rule config format was a design error. A new configuration grammar is almost ready and the underlying structures are simplified. Refactor removes ~750 lines of code and solves _many_ issues that were side-effects of the design error. New features are going to be unlocked thanks to this. Anatomy of a design error # OpenSMTPD started ten years ago out of dissatisfaction with other solutions, mainly because I considered them way too complex for me not to get things wrong from time to time.\nThe initial configuration format was very different, I was inspired by pyr@\u0026rsquo;s hoststated, which eventually became relayd, and designed my configuration format with blocks enclosed by brackets.\nWhen I first showed OpenSMTPD to pyr@, he convinced me that PF-like one-line rules would be awesome, and it was awesome indeed.\nIt helped us maintain our goal of simple configuration files, it helped fight feature creeping, it helped us gain popularity and become a relevant MTA, it helped us get where we are now 10 years later.\nThat being said, I believe this was a design error. A design error that could not have been predicted until we hit the wall to understand WHY this was an error. One-line rules are semantically wrong, they are SMTP wrong, they are wrong.\nOne-line rules are making the entire daemon more complex, preventing some features from being implemented, making others more complex than they should be, they no longer serve our goals.\nTo get to the point: we should move to two-line rules :-)\nSMTP is transactional protocol # SMTP is a transactional protocol which considers each message to be part of a transaction with one or many recipients.\nA message gets assigned a unique transaction identifier when committed to queue:\n250 2.0.0: 8a2e1208 Message accepted for delivery That identifier (supposedly) guarantees that the message has been written to disk, that the mail exchanger will take responsibility not to let it vanish, and that all recipients from the same transaction will share the same identifier.\nSimilarly, recipients may result in aliasing or may have ~/.forward files resulting in new recipients, and since all recipients from the same transaction must share the same identifier, this implies that all aliases and .forward expansions must be done before accepting a recipient.\nIf you must remember only one thing from this section, when the message identifier has been replied to the client, no new envelope will be generated for this transaction.\nThe problem with one-line rules # OpenSMTPD decides to accept or reject messages based on one-line rules such as:\naccept from any for domain poolp.org deliver to mbox Which can essentially be split into three units:\nthe decision: accept/reject the matching: from any for domain poolp.org the (default) action: deliver to mbox To ensure that we meet the requirements of the transactions, the matching must be performed during the SMTP transaction before we take a decision for the recipient.\nGiven that the rule is atomic, that it doesn\u0026rsquo;t have an identifier and that the action is part of it, the two only ways to make sure we can remember the action to take later on at delivery time is to either:\nsave the action in the envelope, which is what we do today evaluate the envelope again at delivery And this this where it gets tricky\u0026hellip; both solutions are NOT ok.\nThe first solution, which we\u0026rsquo;ve been using for a decade, was to save the action within the envelope and kind of carve it in stone. This works fine\u0026hellip; however it comes with the downsides that errors fixed in configuration files can\u0026rsquo;t be caught up by envelopes, that delivery action must be validated way ahead of time during the SMTP transaction which is much trickier, that the parsing of delivery methods takes place as the _smtpd user rather than the recipient user, and that envelope structures that are passed all over OpenSMTPD carry delivery-time informations, and more, and more, and more. The code becomes more complex in general, less safe in some particular places, and some areas are nightmarish to deal with because they have to deal with completely unrelated code that can\u0026rsquo;t be dealt with later in the code path.\nThe second solution can\u0026rsquo;t be done. An envelope may be the result of nested rules, for example an external client, hitting an alias, hitting a user with a .forward file resolving to a user. An envelope on disk may no longer match any rule or it may match a completely different rule If we could ensure that it matched the same rule, evaluating the ruleset may spawn new envelopes which would violate the transaction. Trying to imagine how we could work around this leads to more and more and more RFC violations, incoherent states, duplicate mails, etc\u0026hellip;\nThere is simply no way to deal with this with atomic rules, the matching and the action must be two separate units that are evaluated at two different times, failure to do so will necessarily imply that you\u0026rsquo;re either using our first solution and all its downsides, or that you are currently in a world of pain trying to figure out why everything is burning around you. The minute the action is written to an on-disk envelope, you have failed.\nA proper ruleset must define a set of matching patterns resolving to an action identifier that is carved in stone, AND a set of named action set that is resolved dynamically at delivery time.\n# this is resolved at delivery time action my_action mbox # this is resolved at SMTP transaction time match from any for local action my_action Action and Match # By splitting these, we get immediate benefits, I will only list a few of the improvements this allowed and why.\nFirst of all, the on-disk envelopes now save the name of the action, rather than the action, and resolve the action right before delivery.\nThe obvious benefit is that if I was wrong and wanted to convert to maildir, it would be enough to change the previous example to:\naction my_action maildir match from any for local action my_action After a restart, all envelopes would immediately catch up change without any need to rewrite anything. The same would apply if I got my action wrong:\naction my_action maildir \u0026ldquo;~/Maildirr\u0026rdquo;\nWhich would be unfixable today without going to edit envelopes on-disk manually, but would only require fixing smtpd.conf and restarting with the new configuration.\nThe other benefit is that code paths like aliases expansion required going down to the action to write the envelope, this would involve looking up users, expanding .forward variables, making sure that the delivery data is fully ready so that the delivery process just has to pass it to the MDA. The aliases expansion is already a complex code path by nature, throwing a ton of additional delivery-time data to process and craft into structures made some parts really hard to work with.\nThe structures involved would not only contain the envelope data, they would also contain the delivery data which were passed to all components including the SMTP server, the disk queue, the scheduler, the delivery and finally the mda, when the only processes really needing them were the deivery and mda processes.\nCode parsing user-supplied variables, which could be expanded by the user process right before executing the mda command needed to be parsed during the SMTP transaction within an OpenSMTPD owned process.\nI don\u0026rsquo;t know where to end, the refactor to move to his new grammar essentially simplified every single process within OpenSMTPD, removed over 750 lines of code from the daemon, automagically unlocked features that were previously not even doable, and makes it possible to move forward with other features which were tricky.\nNew features ? # Yes, there are new features which required the grammar change but they are not part of it and will come after.\nThe only features the new grammar bring is some matching patterns such as:\nmatch tls [...] # to match a session using TLS match auth [...] # to match an authenticated session match from socket [...] # to match sessions issued by the local enqueuer match helo foobar [...] # to match sessions that issues HELO foobar What\u0026rsquo;s the catch ? # The code is ready but I haven\u0026rsquo;t finished the man page update which is a pre-requisite to me sending this to OpenBSD developers to get their feedbak on the new grammar.\nIn addition, the change of behaviour and grammar is absolutely not backward-compatible and will make the older queue incompatible.\nBesides that, it has 100% benefits and no downside compared to previous grammar, it is cleaner, safer, saner and allows for a brighter future.\nIt is currently powering poolp.org and the misc@opensmtpd.org mailing list and shows no sign of regression.\nThis should hit OpenBSD -current in a few days/weeks and be part of the next OpenBSD (6.4) and OpenSMTPD major release.\n","date":"30 April 2018","permalink":"/posts/2018-04-30/opensmtpd-new-config/","section":"Posts","summary":"TL;DR: OpenBSD #p2k18 hackathon took place at Epitech in Nantes. I was organizing the hackathon but managed to make progress on OpenSMTPD. As mentionned at EuroBSDCon the one-line per rule config format was a design error.","title":"OpenSMTPD new config"},{"content":" TL;DR: I run several \u0026quot;dedibox\u0026quot; servers at online.net, all powered by OpenBSD. OpenBSD is not officially supported so you have to work-around. Running full-disk encrypted OpenBSD there is a piece of cake. As a bonus, my first steps within a brand new booted machine ;-) Step #0: choosing your server # OpenBSD is not officially supported, I can\u0026rsquo;t guarantee that this will work for you on any kind of server online.net provides, however I\u0026rsquo;ve been running https://poolp.org on OpenBSD there since 2008, only switching machines as they were getting a bit old and new offers came up.\nCurrently, I\u0026rsquo;m running two SC 2016 (SATA) and one XC 2016 (SSD) boxes, all three running OpenBSD reliably ever since I installed them.\nRecently I\u0026rsquo;ve been willing to reinstall the XC one after I did some experiments that turned it into a FrankenBSD, so this was the right occasion to document how I do it for future references.\nI wrote an article similar to this a few years ago relying on qemu to install to the disk, since then online.net provided access to a virtual serial console accessed within the browser, making it much more convenient to install without the qemu indirection which hid the NIC devices and disks duid and required tricks.\nThe method I currently use is a mix and adaptation from the techniques described in https://www.2f30.org/guides/openbsd-dedibox.html to boot the installer, and the technique described in https://geekyschmidt.com/2011/01/19/configuring-openbsd-softraid-fo-encryption.html to setup the crypto slice.\nStep #1: boot to rescue mode # The web console has a rescue mode which will essentially boot the server on a system running in RAM.\nThis usually allows you to unfuck a broken system by booting a Linux or FreBSD system, mounting disks, making appropriate changes to the disk, then rebooting back to the original system.\nWe will actually make use of this rescue mode to write an OpenBSD boot disk image at the beginning of the real disk, allowing us to reboot the server right into the OpenBSD intaller.\nlaptop$ ssh gilles@163.172.61.249 [...] gilles@163-172-61-249:~$ wget https://ftp.fr.openbsd.org/pub/OpenBSD/6.2/amd64/miniroot62.fs [...] gilles@163-172-61-249:~$ sudo dd if=miniroot62.fs of=/dev/sda [sudo] password for gilles: 9600+0 records in 9600+0 records out 4915200 bytes (4,9 MB, 4,7 MiB) copied, 0,116668 s, 42,1 MB/s gilles@163-172-61-249:~$ You can then reboot back to normal mode and activate the serial console from the web console.\nStep #2: boot to the installer # On the serial console, you will be greeted by the bootloader prompt. For some reason, every couple seconds a keystroke gets triggered by the interface causing the \u0026rsquo;n\u0026rsquo; character to be inserted. It takes a bit of synchronization but you should be able to set tty to com1, getting rid of the keystroke and allowing proper output to the terminal:\nboot\u0026gt; set tty com1 \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 boot\u0026gt; boot The bootloader will load a ramdisk kernel and drop you into the installer, which is our next step.\nStep #3: prepare softraid # Once the installer is started, drop immediately into a shell:\nWelcome to the OpenBSD/amd64 6.2 installation program. (I)nstall, (U)pgrade, (A)utoinstall or (S)hell? s # First of all, we need to rewrite the MBR after we messed it up with our miniroot trick:\n# fdisk -iy sd0 Writing MBR at offset 0. # Then, we can enter the disklabel to setup a RAID slice and a swap slice, keeping in mind that swap is already encrypted by default on OpenBSD. The RAID slice will be used to setup softraid with the crypto discipline.\n# disklabel -E sd0 Label editor (enter \u0026#39;?\u0026#39; for help at any prompt) \u0026gt; p M OpenBSD area: 64-500103450; size: 244191.1M; free: 244191.1M # size offset fstype [fsize bsize cpg] c: 244198.3M 0 unused \u0026gt; a partition: [a] offset: [64] size: [500103386] 240000M Rounding size to cylinder (16065 sectors): 491524676 FS type: [4.2BSD] RAID \u0026gt; a partition: [b] offset: [491524740] size: [8578710] FS type: [swap] \u0026gt; w \u0026gt; q No label changes. # Once this is done, we can use bioctl to setup the encrypted slice:\n# bioctl -c C -r auto -l /dev/sd0a softraid0 New passphrase: Re-type passphrase: sd1 at scsibus1 targ 1 lun 0: \u0026lt;OPENBSD, SR CRYPTO, 006\u0026gt; SCSI2 0/direct fixed sd1: 240002MB, 512 bytes/sector, 491524148 sectors softraid0: CRYPTO volume attached as sd1 # At this point, we\u0026rsquo;re ready to perform a regular install\u0026hellip; on sd1, not sd0, beware,:\n# install At any prompt except password prompts you can escape to a shell by typing \u0026#39;!\u0026#39;. Default answers are shown in []\u0026#39;s and are selected by pressing RETURN. You can exit this program at any time by pressing Control-C, but this can leave your system in an inconsistent state. Terminal type? [vt220] System hostname? (short form, e.g. \u0026#39;foo\u0026#39;) pocs Available network interfaces are: em0 em1 vlan0. Which network interface do you wish to configure? (or \u0026#39;done\u0026#39;) [em0] IPv4 address for em0? (or \u0026#39;dhcp\u0026#39; or \u0026#39;none\u0026#39;) [dhcp] em0: DHCPDISCOVER - interval 1 em0: DHCPOFFER from 163.172.61.1 (00:81:c4:f6:e9:17) em0: DHCPREQUEST to 255.255.255.255 em0: DHCPACK from 163.172.61.1 (00:81:c4:f6:e9:17) em0: bound to 163.172.61.249 -- renewal in 2147483647 seconds IPv6 address for em0? (or \u0026#39;autoconf\u0026#39; or \u0026#39;none\u0026#39;) [none] Available network interfaces are: em0 em1 vlan0. Which network interface do you wish to configure? (or \u0026#39;done\u0026#39;) [done] Default IPv4 route? (IPv4 address or none) [163.172.61.1] add net default: gateway 163.172.61.1 Using DNS domainname online.net Using DNS nameservers at 62.210.16.6 62.210.16.7 Password for root account? (will not echo) Password for root account? (again) Start sshd(8) by default? [yes] Change the default console to com1? [yes] Available speeds are: 9600 19200 38400 57600 115200. Which speed should com1 use? (or \u0026#39;done\u0026#39;) [9600] Setup a user? (enter a lower-case loginname, or \u0026#39;no\u0026#39;) [no] gilles Full name for user gilles? [gilles] Password for user gilles? (will not echo) Password for user gilles? (again) WARNING: root is targeted by password guessing attacks, pubkeys are safer. Allow root ssh login? (yes, no, prohibit-password) [no] What timezone are you in? (\u0026#39;?\u0026#39; for list) [Europe/Paris] I insist again, you want to write to the new softraid-backed disk:\nAvailable disks are: sd0 sd1. Which disk is the root disk? (\u0026#39;?\u0026#39; for details) [sd0] sd1 No valid MBR or GPT. Use (W)hole disk MBR, whole disk (G)PT or (E)dit? [whole] Setting OpenBSD MBR partition to whole sd1...done. The auto-allocated layout for sd1 is: # size offset fstype [fsize bsize cpg] a: 1.0G 64 4.2BSD 2048 16384 1 # / b: 16.2G 2097216 swap c: 234.4G 0 unused d: 4.0G 36067392 4.2BSD 2048 16384 1 # /tmp e: 29.5G 44455968 4.2BSD 2048 16384 1 # /var f: 2.0G 106342848 4.2BSD 2048 16384 1 # /usr g: 1.0G 110537152 4.2BSD 2048 16384 1 # /usr/X11R6 h: 10.0G 112634304 4.2BSD 2048 16384 1 # /usr/local i: 2.0G 133605824 4.2BSD 2048 16384 1 # /usr/src j: 6.0G 137800128 4.2BSD 2048 16384 1 # /usr/obj k: 162.7G 150383040 4.2BSD 4096 32768 1 # /home Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] Just for the purpose of simplifying, I will create a custom layout with only a root slice, note that this is insecure and, unless you know what you\u0026rsquo;re doing you should avoid that as it exposes your system to a denial of service.\nThe swap slice is redundant with the one we already create along the RAID slice in sd0, so the options are either to stick with the auto layout or to create a custom layout that\u0026rsquo;s similar but without swap.\nAt the very very least, you want to isolate / from /tmp, /var, /usr and /home, though isolating it from /usr/local is not a bad idea.\nUse (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] C Label editor (enter \u0026#39;?\u0026#39; for help at any prompt) \u0026gt; p OpenBSD area: 64-491508675; size: 491508611; free: 491508611 # size offset fstype [fsize bsize cpg] c: 491524148 0 unused \u0026gt; a partition: [a] offset: [64] size: [491508611] FS type: [4.2BSD] mount point: [none] / Rounding size to bsize (64 sectors): 491508608 \u0026gt; w \u0026gt; q No label changes. /dev/rsd1a: 239994.4MB in 491508608 sectors of 512 bytes 295 cylinder groups of 814.44MB, 26062 blocks, 52224 inodes each Available disks are: sd0. Which disk do you wish to initialize? (or \u0026#39;done\u0026#39;) [done] /dev/sd1a (448be85498897147.a) on /mnt type ffs (rw, asynchronous, local) Let\u0026#39;s install the sets! Location of sets? (disk http or \u0026#39;done\u0026#39;) [http] HTTP proxy URL? (e.g. \u0026#39;http://proxy:8080\u0026#39;, or \u0026#39;none\u0026#39;) [none] HTTP Server? (hostname, list#, \u0026#39;done\u0026#39; or \u0026#39;?\u0026#39;) [ftp.fr.openbsd.org] Server directory? [pub/OpenBSD/6.2/amd64] Select sets by entering a set name, a file name pattern or \u0026#39;all\u0026#39;. De-select sets by prepending a \u0026#39;-\u0026#39;, e.g.: \u0026#39;-game*\u0026#39;. Selected sets are labelled \u0026#39;[X]\u0026#39;. [X] bsd [X] base62.tgz [X] game62.tgz [X] xfont62.tgz [X] bsd.mp [X] comp62.tgz [X] xbase62.tgz [X] xserv62.tgz [X] bsd.rd [X] man62.tgz [X] xshare62.tgz Set name(s)? (or \u0026#39;abort\u0026#39; or \u0026#39;done\u0026#39;) [done] Get/Verify SHA256.sig 100% |**************************| 2152 00:00 Signature Verified Get/Verify bsd 100% |**************************| 12777 KB 00:08 Get/Verify bsd.mp 100% |**************************| 12858 KB 09:46 Get/Verify bsd.rd 100% |**************************| 9565 KB 00:05 Get/Verify base62.tgz 100% |**************************| 139 MB 01:24 Get/Verify comp62.tgz 100% |**************************| 75525 KB 50:08 Get/Verify man62.tgz 100% |**************************| 7008 KB 00:03 Get/Verify game62.tgz 100% |**************************| 2718 KB 01:56 Get/Verify xbase62.tgz 100% |**************************| 17964 KB 12:29 Get/Verify xshare62.tgz 100% |**************************| 4417 KB 02:43 Get/Verify xfont62.tgz 100% |**************************| 39342 KB 00:20 Get/Verify xserv62.tgz 100% |**************************| 12572 KB 00:06 Installing bsd 100% |**************************| 12777 KB 00:00 Installing bsd.mp 100% |**************************| 12858 KB 00:00 Installing bsd.rd 100% |**************************| 9565 KB 00:00 Installing base62.tgz 100% |**************************| 139 MB 00:10 Extracting etc.tgz 100% |**************************| 189 KB 00:00 Installing comp62.tgz 100% |**************************| 75525 KB 00:08 Installing man62.tgz 100% |**************************| 7008 KB 00:01 Installing game62.tgz 100% |**************************| 2718 KB 00:00 Installing xbase62.tgz 100% |**************************| 17964 KB 00:01 Extracting xetc.tgz 100% |**************************| 7036 00:00 Installing xshare62.tgz 100% |**************************| 4417 KB 00:01 Installing xfont62.tgz 100% |**************************| 39342 KB 00:02 Installing xserv62.tgz 100% |**************************| 12572 KB 00:01 Location of sets? (disk http or \u0026#39;done\u0026#39;) [done] Location of sets? (disk http or \u0026#39;done\u0026#39;) [done] Saving configuration files...done. Making all device nodes...done. Multiprocessor machine; using bsd.mp instead of bsd. Relinking to create unique kernel...done. CONGRATULATIONS! Your OpenBSD install has been successfully completed! To boot the new system, enter \u0026#39;reboot\u0026#39; at the command prompt. When you login to your new system the first time, please read your mail using the \u0026#39;mail\u0026#39; command. # Step #4: reboot to encrypted OpenBSD system # The root slice being encrypted, you\u0026rsquo;ll need to type your password every time you reboot.\nConnect to the serial console, you should see the boot prompt asking for a password. The \u0026rsquo;n\u0026rsquo; glitch is still around, so either you break out of password by typing enter so you can do the \u0026lsquo;set tty com1\u0026rsquo; trick, or you do as I do and synchronize with the \u0026rsquo;n\u0026rsquo; keystroke to delete it and type password really fast.\n# reboot syncing disks... done sd1 detached rebooting... [...] Using drive 0, partition 3. Loading...... probing: pc0 com0 com1 mem[632K 2009M 14336M a20=on] disk: hd0+ sr0* \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 Passphrase: boot\u0026gt; set tty com1 \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 boot\u0026gt; boot Passphrase: [...] OpenBSD/amd64 (pocs.online.net) (tty01) login: gilles Password: OpenBSD 6.2 (GENERIC.MP) #134: Tue Oct 3 21:22:29 MDT 2017 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. You have new mail. $ As suggested by semarie@, once logged in you can take opportunity to fix the \u0026rsquo;n\u0026rsquo; glitch by setting tty in boot.conf:\n$ su Password: # echo set tty com1 \u0026gt; /etc/boot.conf Bonus: further tightening your system # These are the few steps I immediately do to tighten up my systems furthers:\nenable doas # The \u0026lsquo;gilles\u0026rsquo; account I created at install is part of the \u0026lsquo;wheel\u0026rsquo; group, which turns out to be exactly what the example doas.conf allows:\n$ su Password: # cat /etc/examples/doas.conf |tail -1 permit keepenv :wheel # cp /etc/examples/doas.conf /etc # exit $ doas sh doas (gilles@pocs.online.net) password: # disable the root account # Now that \u0026lsquo;gilles\u0026rsquo; can use doas we no longer ever need to authenticate as root, so disable it by setting the password to \u0026lsquo;*\u0026rsquo;. This will prevent \u0026lsquo;root\u0026rsquo; from being usable directly or through su, yet if really needed \u0026lsquo;gilles\u0026rsquo; can still doas su to obtain a shell running as user \u0026lsquo;root\u0026rsquo;:\n# usermod -p\u0026#39;*\u0026#39; root # update system with syspatch # The brand new system may require some patches to be applied, the syspatch command written by ajacoutot@ performs a binary patching of the system, then causes the kernel to be relinked using the KARL mechanism to shuffle objects order:\n# syspatch Get/Verify syspatch62-001_tcb_inv... 100% |*************| 465 KB 00:00 Installing patch 001_tcb_invalid Get/Verify syspatch62-002_fktrace... 100% |*************| 785 KB 00:21 Installing patch 002_fktrace Get/Verify syspatch62-003_mpls.tgz 100% |***************| 837 KB 00:00 Installing patch 003_mpls Get/Verify syspatch62-004_libssl.tgz 100% |*************| 2515 KB 00:01 Installing patch 004_libssl Relinking to create unique kernel... done. add my ssh public key to my ~/.ssh/authorized_keys # Password for accessing SSH are bad, copy the SSH public key generated on my laptop with ssh-keygen to the authorized_keys file of my account on the server:\n# echo \u0026#39;ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII19dxt4hf1t4Lo05TaPZQqrtwtszHHyAF7ctPYPRIvp gilles@debug.poolp.org\u0026#39;\u0026gt;\u0026gt;~gilles/.ssh/authorized_keys # disable password authentication within ssh # No reason to allow PasswordAuthentication anymore, disable in sshd_config and restart sshd:\n# echo PasswordAuthentication no \u0026gt;\u0026gt; /etc/ssh/sshd_config # /etc/rc.d/sshd restart sshd(ok) sshd(ok) # reboot so you boot on a brand new up-to-date system with latest stable kernel # # reboot syncing disks... done sd1 detached rebooting... [...] Using drive 0, partition 3. Loading...... probing: pc0 com0 com1 mem[632K 2009M 14336M a20=on] disk: hd0+ sr0* \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 Passphrase: boot\u0026gt; set tty com1 \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 boot\u0026gt; boot Passphrase: [...] OpenBSD/amd64 (pocs.online.net) (tty01) login: Password: VOILA !\nWant to comment ? Open an issue on Github\n","date":"29 January 2018","permalink":"/posts/2018-01-29/install-openbsd-on-dedibox-with-full-disk-encryption/","section":"Posts","summary":"TL;DR: I run several \u0026quot;dedibox\u0026quot; servers at online.net, all powered by OpenBSD. OpenBSD is not officially supported so you have to work-around. Running full-disk encrypted OpenBSD there is a piece of cake.","title":"Install OpenBSD on dedibox with full-disk encryption"},{"content":" TL;DR: deraadt@ thought it would be nice to have a spf fetch utility in base. Aaron Poffenberger wrote a shell-based `spf_fetch` utility. I wrote a C-based `spfwalk` utility that's `pledge()`-ed. The `spfwalk` utility got merged to `smtpctl`. What\u0026rsquo;s SPF in a few words # SPF is the Sender Policy Framework, a standard to verify the domain name of an e-mail sender.\nLong story short, the SMTP protocol does not come with a way to authenticate a domain and, during an SMTP session, nothing really prevents a sender from pretending to come from any domain:\n$ nc localhost 25 220 poolp.org ESMTP OpenSMTPD HELO pussycat 250 poolp.org Hello pussycat [127.0.0.1], pleased to meet you MAIL FROM:\u0026lt;gilles@systemd.lol\u0026gt; 250 2.0.0: Ok ^C $ Note that this is a feature of the SMTP protocol, not a bug.\nIt turns out that the internet is a hostile place and people started abusing this so\u0026hellip; the solution, as usual when it comes to SMTP, was to shove data inside DNS records \\o/\nSPF allows the owner of a DNS zone to declare which machines are allowed to send mail on behalf of the domain in a TXT record. With this feature, whenever a spammer attempts to send mail on behalf of @google.com, the receiving MX can simply check that the client is allowed to send mail by the google.com zone.\nThis seems lovely, however it\u0026rsquo;s not a spam killer because it actually requires senders to create the record, destination nodes to actually check the TXT record and match the client address against it, and it doesn\u0026rsquo;t prevent spammers from adding SPF records to their own domains.\nWhat it does, when everything is setup correctly on both ends, is protect the destination node from spammers trying to impersonnate a sender domain\u0026hellip; as long as the spammer doesn\u0026rsquo;t control a machine declared in the TXT record [ it\u0026rsquo;s easy to whitelist a /16 ;) ].\nGreylisting # A technique that\u0026rsquo;s been used widely to reduce spam is to rely on greylisting.\nBasically, when a MX you don\u0026rsquo;t know connects to your node, it gets bounced with a temporary failure requesting a retry. All SMTP server know how to handle these retries, however in a spamming economy based on sending volumes of messages it\u0026rsquo;s often not worth it to retry on temporary failures.\nSpammers will come from many source addresses, keep being seen as new MX, keep being bounced, life is good.\nWhy am I talking about greylisting you ask ?\nWhen BIG MAILERS made it easier for spammers to annoy us # It\u0026rsquo;s impolite to point fingers so I won\u0026rsquo;t name them explicitely, you know who they are.\nGreylisting works lovely to waste spammers\u0026rsquo; time\u0026hellip; as long as you can assume legitimate hosts come from the same IP addresses.\nAt some point, BIG MAILERS decided that: nope, we\u0026rsquo;ll send from this host, then we\u0026rsquo;ll retry from this one, then another one, and then since we have hundreds of IP addresses available, well just fuck you, we\u0026rsquo;ll make sure we never hit you twice with the same.\nI don\u0026rsquo;t know if these were the exact words, but it essentially leads to that \u0026ldquo;fuck you\u0026rdquo; result so\u0026hellip;\nSince BIG MAILERS could not send from a small set of addresses and reuse the same ones upon retries, they started advertising their MX in SPF records. And by that I mean, they started whitelisting full ranges in records requiring recursive and cross-domains lookups because WHY NOT ?\nThis way, greylisting became essentially unusable to many who turned it off because they can\u0026rsquo;t easily whitelist BIG MAILERS.\nThis is how we get to spfwalk # We don\u0026rsquo;t want to NOT have greylisting, so we want a tool to actually harvest records enabled in BIG MAILERS\u0026rsquo; SPF record.\nderaadt@ ran into an issue with one of his MX which had BIG MAILERS unable to pass greylisting. He asked me if I could write a utility to fetch SPF records and insert them into a PF table to whitelist MX for particular domains.\nCoincidentally, Aaron Poffenberger announced the next day that he had been working on a tool called spf_fetch https://github.com/akpoff/spf_fetch to do just that. The idea was nice however it couldn\u0026rsquo;t be committed to OpenBSD as is because it is written as a set of shell scripts and we wanted a piece of code that could be pledge()-ed.\nI contacted Aaron and told him I was going to be working on a C version based on the asr asynchronous resolver, and told him I would appreciate if he contributed since he had already started a similar project. A few weeks later, we had a work in progress spfwalk utility commited to my repository https://github.com/poolpOrg/spfwalk.\nsmtpctl spf walk # Instead of having a new utility committed to base, we decided to make it a subcommand of the smtpctl utility.\nThe main idea is that smtpctl spf walk will read a list of domains from stdin and output a list of IP addresses and ranges to stdout, allowing it to be used in scripts, to generate file lists, or to be piped directly into pfctl to feed a table.\nThis is still a work in progress, but it allows you to cat /etc/mail/BIGMAILERS.txt | smtpctl spf walk | pfctl -t spf-white -T add -f -, which is quite an improvement over having nothing to do it to start with :-)\nFinal words # sunil@ merged my utility to smtpctl, I made sure it ran unprivileged, so now we have an spf fetching utility that runs unprivileged and pledge()-ed in the base system.\nThis was committed a few days ago to OpenBSD -current, it requires testing because I\u0026rsquo;m pretty sure it has bugs still.\nWant to comment ? Open an issue on Github\n","date":"8 January 2018","permalink":"/posts/2018-01-08/spfwalk/","section":"Posts","summary":"TL;DR: deraadt@ thought it would be nice to have a spf fetch utility in base. Aaron Poffenberger wrote a shell-based `spf_fetch` utility. I wrote a C-based `spfwalk` utility that's `pledge()`-ed.","title":"spfwalk"},{"content":" TL;DR: Came very close to a burn out late 2016, had to quit former employer. Started working at another company early 2017. Became a student again, currently halfway through second year in psychology. Haven't done much opensource since early 2017, slowly resuming. Happy new year ! # I wish you a very happy new year 2018 and hope that you succeed in whatever you attempt.\nIt feels odd to write again to this blog considering the last blog post dates from September 2016, over a year ago.\nI didn\u0026rsquo;t lack interest in writing but I almost burnt out at my former company, which killed all my motivation to touch a computer outside of dayjob whatsoever, and had to quit my former employer to begin a new job somewhere else in early 2017.\nThis change, in addition to my numerous psychology night classes and my many many many hypnosis sessions and trainings, has left me with almost no spare time to boot a laptop at home.\nI must admit that I enjoyed putting some distance between computers and myself for a change. It allowed me to focus on things a bit less virtual and to enjoy the few times I actually booted an OpenBSD machine to write some code.\nIt would be a lie to say I didn\u0026rsquo;t do a thing, I actually have plenty of code that has moved forward: 2018 will be the year I start sharing back code ;-)\nI\u0026rsquo;ll keep this blog post short, there are two posts coming shortly, so in the meantime\u0026hellip;\nHAPPY NEW YEAR, ENJOY YOURSELVES, 2018 WILL BE A GOOD YEAR !\n","date":"8 January 2018","permalink":"/posts/2018-01-08/news-from-the-front/","section":"Posts","summary":"TL;DR: Came very close to a burn out late 2016, had to quit former employer. Started working at another company early 2017. Became a student again, currently halfway through second year in psychology.","title":"News from the front !"},{"content":" TL;DR: We just released OpenSMTPD 6.0.0 and it's quite different from former releases. Turns out most of the changes are not visible. A featureless release # I managed to wrap the 6.0.0 release yesterday.\nUnlike most of our releases, it comes out with almost no new feature.\nThe changelog fits in less than 10 lines as follows:\n- new fork+reexec model so each process has its own randomized memory space - logging format has been reworked - a \u0026quot;multi-line response\u0026quot; bug in the LMTP delivery backend has been fixed - connections concurrency limits have been bumped - artificial delaying in remote sessions have been reduced - dhparams option has been removed - dhe option has been added, supporting auto and legacy modes - smtp engine has been simplified - various cosmethic changes, code cleanup and documentation improvement Seems like a very productive slacking, however some of these changes turn out to be very interesting in terms of code simplification and security.\nIn this article I\u0026rsquo;ll focus on the fork+(re)exec model change.\nMemory space randomization # OpenBSD provides each process with a randomized memory space where pretty much everything ending up in memory is going to have different locations each time you run a new instance. All memory allocations using either malloc() or mmap() are randomized, the dynamic loader is randomized and recent work was committed so that libc gets randomized at each and every system start, the goal being that not only the place where the symbols are mapped in memory changes at every run, but that the symbols themselves get shuffled so that they are randomized one relative to another.\nThis basically means that if we both run the same program, there\u0026rsquo;s about zero chances that we will have a similar memory layout and that functions will be sitting at the same addresses.\nIn terms of security this makes exploitation particularly hard as there aren\u0026rsquo;t so many things an attacker can guess regarding the memory layout. She can\u0026rsquo;t study her own system to determine where things are located on yours, and every time the instance is restarted, former knowledge has gone away so that anything learnt will no longer be true once the program restarts.\nThis coupled with the fact that OpenBSD daemons tend to exit rather than attempt to rexecute a crashed process, level up the barrier quite a bit.\nHere, a program will simply print \u0026ldquo;the same\u0026rdquo; addresses every time it is executed:\n$ ./a.out 0x7f7ffffd4514 address of a stack variable 0x1061902ffe40 address of a malloc() allocation 0x105eeb000a64 address of a function within the program 0x105eeb000a6a address of the main function 0x1060fcf67a90 address of a libc function $ ./a.out 0x7f7ffffbb714 address of a stack variable 0xf7cbac277c0 address of a malloc() allocation 0xf7a05d00a64 address of a function within the program 0xf7a05d00a6a address of the main function 0xf7c34ba9a90 address of a libc function $ ./a.out 0x7f7ffffeab74 address of a stack variable 0x15b5c11cc5c0 address of a malloc() allocation 0x15b354800a64 address of a function within the program 0x15b354800a6a address of the main function 0x15b5ab116a90 address of a libc function $ The OpenSMTPD bootstrap # The idea for fork+exec in OpenSMTPD came from a discussion between deraadt@ and eric@ during a hackathon I hosted in Nantes mid-May this year.\nThe OpenSMTPD bootstrap process was quite simple:\nUpon executation, the parent process would read configuration, build a memory representation of it and would then create a bunch of socketpair() before fork()-ing all of its child processes.\nBy the virtue of fork(), all children would inherit the configuration as well as all the sockets necessary for IPC. They would only have to zero and free the bits of configuration they don\u0026rsquo;t need and close sockets meant to be used by the other children and be set.\nThis was nice and all, but could be improved.\nWhen a process fork()s, the child process gets an exact copy of the parent memory space. Any symbol in the parent address space is present at the same address in the child and only new allocations will cause the parent and child to diverge.\nSo everytime you run OpenSMTPD, the processes in your instance are different from the last time you ran it, however within a single instance the processes share a lot of addresses together. In the following example, note how all of the symbols share the same address in parent and child, except for the malloc() call that happened after fork():\n$ ./a.out === in parent 0x7f7ffffd3c44 address of a stack variable 0x8e8a457c040 address of a malloc() allocation BEFORE fork 0x8e8181631c0 address of a malloc() allocation AFTER fork 0x8e5ce100b34 address of a function within the program 0x8e5ce100c20 address of the main function 0x8e88231aa90 address of a libc function === in child 0x7f7ffffd3c44 address of a stack variable 0x8e8a457c040 address of a malloc() allocation BEFORE fork 0x8e8ad0311c0 address of a malloc() allocation AFTER fork 0x8e5ce100b34 address of a function within the program 0x8e5ce100c20 address of the main function 0x8e88231aa90 address of a libc function $ ./a.out === in parent 0x7f7ffffc0684 address of a stack variable 0x1fc99fc64100 address of a malloc() allocation BEFORE fork 0x1fc9443a6280 address of a malloc() allocation AFTER fork 0x1fc704400b34 address of a function within the program 0x1fc704400c20 address of the main function 0x1fc97c85aa90 address of a libc function === in child 0x7f7ffffc0684 address of a stack variable 0x1fc99fc64100 address of a malloc() allocation BEFORE fork 0x1fc90bb1e280 address of a malloc() allocation AFTER fork 0x1fc704400b34 address of a function within the program 0x1fc704400c20 address of the main function 0x1fc97c85aa90 address of a libc function $ While all the OpenSMTPD processes end up diverging by a great lot when it comes to locally allocated addresses, the problem is that if a process is compromised then it gives immediate information regarding all of the other processes.\nKnowing the memory address of the configuration in a child process will immediately tell an attacker where the configuration is located in all of the other processes, and if the proper amount of bugs is found this can prove useful.\nThe new fork()+(re)exec model # The exec family function will transform an existing process into a new process by basically reconstructing it from the new program.\nThis is what happens when your shell stops being a shell and becomes a ls instance for example. It forgets (pretty much) everything it knew about the former process and starts with its own brand new randomized memory space.\nSo deraadt@ suggested that if OpenSMTPD would not just fork() children but instead fork() them and reexecute the smtpd binary, then each of the children would have its own randomized memory space.\nThe idea itself is neat, however not so trivial to implement because when we reexec the whole \u0026ldquo;inherit configuration and descriptors\u0026rdquo; part goes away. It\u0026rsquo;s not just fork and exec, it\u0026rsquo;s fork and exec and figure a way for the parent to pass back all the information and descriptors back to the new post-fork instance so it is the new instance that allocates memory and decides where the information goes.\n$ ./a.out === in parent 0x7f7fffff6984 address of a stack variable 0x10a78e9cdc0 address of a malloc() allocation 0x10873500b84 address of a function within the program 0x10873500c57 address of the main function 0x10b6b748a90 address of a libc function === in child 0x7f7fffff8bc4 address of a stack variable 0x142a9fe228c0 address of a malloc() allocation 0x14289e900b84 address of a function within the program 0x14289e900c57 address of the main function 0x142ad94e0a90 address of a libc function $ ./a.out === in parent 0x7f7ffffbdee4 address of a stack variable 0x1f673b59d700 address of a malloc() allocation 0x1f6482500b84 address of a function within the program 0x1f6482500c57 address of the main function 0x1f66b1b71a90 address of a libc function === in child 0x7f7ffffdcc74 address of a stack variable 0xe10ddc66e40 address of a malloc() allocation 0xe0ec9f00b84 address of a function within the program 0xe0ec9f00c57 address of the main function 0xe10d3a5fa90 address of a libc function $ Eric reworked the entire bootstrap process so the parent reads configuration, creates all of the sockets, forks \u0026amp; execs all children in a special mode\u0026hellip;\n\u0026hellip; then have the parent pass each children only the bit of configuration they need and the descriptors they require for IPC.\nThis was really hard work and I\u0026rsquo;m amazed he managed to wrap it between the hackathon and the release freeze, this makes OpenSMTPD far more resistant to a whole lot new range of attacks.\nI\u0026rsquo;ll write about the smtp engine improvements in a further article, stay tuned.\n","date":"12 September 2016","permalink":"/posts/2016-09-12/opensmtpd-6.0.0-is-released/","section":"Posts","summary":"TL;DR: We just released OpenSMTPD 6.0.0 and it's quite different from former releases. Turns out most of the changes are not visible. A featureless release # I managed to wrap the 6.","title":"OpenSMTPD 6.0.0 is released !"},{"content":" TL;DR: OpenSMTPD on github no longer diverges from OpenBSD. Since last episode # Sometime this year, we released OpenSMTPD 5.7.1 which was the first version that shipped with the long-awaited experimental filters support. The result of over a year and a half of very very deep refactor in our IO layer and the addition of a very elegant but also very tricky new API.\nThis refactor took place outside the OpenBSD tree because it required breaking a lot of code for extended period of times, which spanned over 3 OpenBSD releases. It was near impossible to work in tree as we could not break the base daemon and leave it in a weird state for too long, and the changes were so invasive that it was also impossible to backout parts of it when a regression was spotted.\nWe eventually reached a point where it was very stable but also diverged greatly from the daemon in base. We made a \u0026ldquo;checkpoint\u0026rdquo; by releasing 5.7.1 separately of OpenBSD and the goal was now to bring the daemon in tree up to date.\nThe release was quickly followed by 5.7.2 and 5.7.3, fixing security issues that were uncovered by Qualys during a coordinated audit and an issue spotted by an independant user.\nBringing back OpenSMTPD to the OpenBSD tree # Working off-tree came with a lot of burden, first because it cut us off from the other OpenBSD hackers but also because it came with extra work.\nFor instance, during an internal audit we spotted a way to trigger a denial with our control socket. This applied to both the version in OpenBSD and Github. Both versions diverged quite a lot so the fix to one would not apply to the other and of course it took place at worst point in time, when OpenBSD was so close to its new release that 3 stable versions were tagged forcing us to test the fix on all stable branches + current branch + github\u0026rsquo;s master + github\u0026rsquo;s portable.\nNot to mention that every time someone submitted a pull request on Github we had to adapt it to apply on the older version, and every time an OpenBSD hacker made a cleanup to the older version, we had to adapt and apply to Github. This was no fun at all, many of the contributions not applying cleanly and increasing a gap.\nOn another hand, it was very difficult to bring the changes back to OpenBSD as a HUGE part of it was a refactor that could not be split into reviewable minidiffs for other to look at it. Many small features could be split, but given that they were written on top of the refactor, this implied that even taken independantly, they had to be adapted to apply to the old code. This was considerable work that caused us to defer, defer, defer, \u0026hellip;\nFinally, we reached a point where it was no longer possible to defer and I dived back into this with help from sunil@, jung@ and millert@. I took a few days off, spent full-time on this, and we managed to make huge progress that allowed us to fully bring OpenBSD\u0026rsquo;s smtpd up-to-date after a two-months effort of spliting the delta into small reviewable chunks.\nI owe all of them a very large amount of beer.\nOpenBSD\u0026rsquo;s pledge() # One of the most notable change that has taken place in the last months is that we fully integrated the new OpenBSD pledge() system-call.\nBut what is pledge() ?\npledge() is a new system call that will appear with OpenBSD 5.9. What it does is allow a program to promise it will only use certain features and allow the system to terminate the process if it doesn\u0026rsquo;t stick to them.\nFor example, let\u0026rsquo;s imagine the following program which creates a directory:\n#include \u0026lt;sys/stat.h\u0026gt; #include \u0026lt;unistd.h\u0026gt; int main(int argc, char *argv[]) { mkdir(\u0026#34;/tmp/test\u0026#34;, 0700); return 0; } If you execute it, the process creates the directory as expected:\n$ cc test.c $ ./a.out $ ls -l /tmp|grep test drwx------ 2 gilles wheel 512 Dec 22 10:44 test $ But now, let\u0026rsquo;s imagine that the program actually makes a promise that it will only use stdio features, essentially IO on previously allocated descriptors and memory allocation so the libc basic stuff works:\n#include \u0026lt;sys/stat.h\u0026gt; #include \u0026lt;unistd.h\u0026gt; int main(int argc, char *argv[]) { pledge(\u0026#34;stdio\u0026#34;, NULL); mkdir(\u0026#34;/tmp/test\u0026#34;, 0700); return 0; } We run the program again:\n$ cc test.c $ ./a.out Abort trap (core dumped) $ But this time, when the process calls mkdir() the kernel detects that it has violated the promises made in the program and aborts execution.\nSo pledge() is a security feature ? # At first it looks like pledge() is just a security feature to mitigate risks and it DOES bring a very effective mitigation mechanism: an attacker who wants to run arbitrary code in a compromised process still needs to respect promises made by the program\u0026hellip; in that arbitrary code\u0026hellip;\nIn practice, it\u0026rsquo;s far from being JUST a security feature, it is also a quality feature.\nWhen a developer converts a program to pledge(), he needs to understand what features are being used, why they are being used, when they are being used and where they are being used. If there\u0026rsquo;s a wrong assumption, the process dies. In addition, the promises can be shortened at runtime to restrict further.\nIn a project like OpenSMTPD, this is particularly interesting because it comes with a multi-process design where different processes do specific things, each being able to declare its own list of very reduced promises.\nThe daemon starts with a huge promise list to be able to setup its sockets and processes, chroot, change privileges, do some filesystem stuff, then processes rapidly reduce their promises down to what they will really use at runtime: in many cases stdio.\nIntuitively, having worked on this code base for so long, I had a pretty clear idea of what the processes were doing at runtime so I thought it would be very straightforward and would \u0026ldquo;just work (c)\u0026rdquo;.\nIn practice, pledge() pinpointed a few cases where my promises were not held and had me investigate WHY a component needed a particular feature at runtime, when it was not part of the promises I intuitively thought off.\nMost cases were really simple to solve and a matter of doing a call just a bit too late in the code path, but in some cases it really helped refactor code to get rid of a feature that wasn\u0026rsquo;t really needed.\npledge() helps you write better code by getting a very clear feedback that a piece of code is doing more than you think it does.\nWhat\u0026rsquo;s next ? # The code in Github is now a clone of the OpenBSD CVS tree and the only delta is a version check so that it can build on OpenBSD releases that do not have the pledge() system call.\nThe portable branch has been sync-ed with the CVS tree but it currently does not build as it requires a bit of plumbing. We are currently working on this so we can resume publishing snapshots.\nWe\u0026rsquo;re going to synchronize our releases with OpenBSD again so you can expect OpenSMTPD 5.9.1 to be released when OpenBSD 5.9 is released, though there is a chance we release an intermediate version so people can run a version that benefits from all the cleanups, improvements and safer \u0026ldquo;enqueuer\u0026rdquo; before the 5.9.1 release in May.\n","date":"22 December 2015","permalink":"/posts/2015-12-22/home-sweet-home/","section":"Posts","summary":"TL;DR: OpenSMTPD on github no longer diverges from OpenBSD. Since last episode # Sometime this year, we released OpenSMTPD 5.7.1 which was the first version that shipped with the long-awaited experimental filters support.","title":"Home, sweet home"},{"content":" TL;DR: yeeeees, filters are coming. don't believe us ? here's an example. be patient. On my death bed # On my death bed, when my life flashes before my eyes and I start recalling what people have told me during my (hopefully long) lifetime, these sentences will single out:\n\u0026ldquo;When will OpenSMTPD support filters ? I need it.\u0026rdquo;\nNot that it carries a philosophical meaning that will have taken me a lifetime of thinking, but because for the last three years I have been hearing this every time I met someone IRL and discussed OpenSMTPD, I have read it on our Github issues tracker, on GTalk, on IRC, on Twitter, on Facebook, on random forums, and just this week three times in my mailbox. I can honestly say that during these three years, OpenSMTPD filters have been mentioned to me hundreds of times without exaggerating. Yup. That\u0026rsquo;s what I\u0026rsquo;ll remember :-/\nA short reminder # The SMTP server accepts connections from clients, it establishes sessions and processes SMTP transactions which then eventually result in a mail being accepted for delivery or rejected. This decision isn\u0026rsquo;t a very smart process, the server looks at a ruleset that basically lists which domains are to be accepted or rejected, and it tries to locate a user under these domains. That\u0026rsquo;s it, more or less.\nSo what are filters useful for ? # Filters are small pieces of code that can be used to modify the behavior of an SMTP session and decide to accept, reject or modify the client input based on decisions that are not related to that ruleset. For instance, a filter may decide that besides the ruleset, it also wants to check if a domain is not part of a public blacklist. It may want to decide that a particular client should not be able to send mail because according to some database, it has exceeded a quota. It may simply want to check at the content of the DATA phase and try to detect some keywords typical of spam or even alter it to remove (bad bad bad) or append some headers.\nTechnically these behavior could be implemented in the daemon itself, but providing an API for filters allows this code to remain outside of the daemon code, keeping it simple, and allowing anyone to customize it for their needs without needing us to cooperate.\nFilters goals # Filters are a feature available in various MTA, this is not an OpenSMTPD thing and that\u0026rsquo;s why so many people have requested it. They expect the software to be extensible through filters because there are many tools out there that they could plug somehow to make it fit their use case. These tools are usable with the other MTA, so they should be usable with ours.\nThere are different ways of achieving a filter API, some relying on forked processes with a common convention to read on standard input and write on standard output, others providing a set of functions to do the work. Some as standalone filters, others as shared objects or libraries importing the features inside the daemon code base.\nThe most commonly used is the milters API which was written for a competitor software and adapted to other competitors with varying degrees of success and completeness. It is mature, has a wide range of filters written for it, and allows doing pretty much anything. When we started thinking about how we\u0026rsquo;d integrate filters, this was the first API we looked at as we could have a reference implementation to compare ours to and we would benefit from the plethora of filters that were already available.\nThen, we changed our mind: it doesn\u0026rsquo;t do what we want and it doesn\u0026rsquo;t fit well in our asynchronous model, it\u0026rsquo;s going to be a pain in the ass. We don\u0026rsquo;t want to grab an existing API and modify our design to have it fit it, we want an API that just fits in our design.\nWhat we need is an API that\u0026rsquo;s fully asynchronous, that\u0026rsquo;s highly flexible while not bloated and that\u0026rsquo;s dead easy. It needs to be so flexible that we can implement bindings to other languages than C, or even a milter interface to translate milter calls to ours, and it needs to be so easy that it doesn\u0026rsquo;t require hours to get a basic filter working even if using the API for the first time.\nFilters design # So, I did a bit of research on how we would implement this API and after a while came up with these requirements:\nfilters should not be libraries or shared objects, they should be standalone executables running in their own memory space; they shouldn\u0026rsquo;t be fork()-ed at runtime because it would crush performances; they should be able to be individually chroot()-ed; they should be able to run with their very own privileges, isolated one from another if needed; whatever their dependencies, the daemon should not have to care, it should not know anything besides its filter protocol; Long story short, while it would be considerably easier for us, we don\u0026rsquo;t want filters to be plugged as shared objects. The problem with that approach is that the filter would then share the daemon memory space and that\u0026rsquo;s something we don\u0026rsquo;t want to happen. The filters should be isolated in their own memory space and they should know nothing about the server except what the server hands them. That way, if a filter suffers from a bug allowing leak of random memory areas, it will not be able to leak information that is part of the server. This concern predates by far the Heartbleed catastrophe but since some people back then thought it was not a real problem, now they have a perfect example of why it was ;-)\nAnyways, to respect these requirements and given that the daemon itself drops privileges at startup, filters had to become daemons of their own started with the server and communicating with it during its lifetime. Fortunately this is something we\u0026rsquo;re quite familiar with, it is basically OpenSMTPD\u0026rsquo;s model: parent starts, sets up communication channels with imsg framework and starts other processes.\nWe decided to build the filter API on that model, then we realized that while we were familiar with it, it would be far too complex for users to deal with this whole fork() / imsg thing. So we wrote a \u0026ldquo;filter glue\u0026rdquo; which takes care of setting everything up and running so that a user can write a filter by focusing only on its own functions.\nstatic int on_connect(uint64_t id, struct filter_connect *conn) { log_warnx(\u0026#34;filter-stub: OHAI, SOMEONE CONNECTED !\u0026#34;); return (1); } int main(int argc, char **argv) { int\tch; log_init(-1); while ((ch = getopt(argc, argv, \u0026#34;\u0026#34;)) != -1) { switch (ch) { default: log_warnx(\u0026#34;warn: filter-stub: bad option\u0026#34;); return (1); } /* NOTREACHED */ } argc -= optind; argv += optind; /* register a callback for \u0026#34;on_connect\u0026#34; event */ filter_api_on_connect(on_connect); /* start doing stuff */ filter_api_loop(); log_warnx(\u0026#34;warn: filter-stub: exiting\u0026#34;); return (1); } As shown in this example, the main function only calls filter_api_on_connect() to let it know that it wants on_connect() to be called when there\u0026rsquo;s a connection. Then it calls filter_api_loop() to actually start processing events.\nThe whole \u0026ldquo;setup a channel with the daemon, fork and maintain state\u0026rdquo; is completely hidden within the API layer and transparent to the user. The filter API exposes a unique session identifier and per-hook structures filled with the data available in that phase, the filter is free to maintain its own data structures using the identifier to pass data from a hook to another.\nIt doesn\u0026rsquo;t get much simpler.\nBut that\u0026rsquo;s C ?! # Of course, it\u0026rsquo;s C, what else ?\nThe API is written and exposed in C allowing C programmers to write native filters. We want to make it possible to write them in other languages but our language of choice and the one we will use for any filter that may be shipped with the daemon remains C.\nSo from there, if we want to allow filters to be written in different languages, we have two ways:\neither we provide a binding, which is basically a reimplementation of our API in that language; or we provide bridge filters, which are basically proxies that will convert calls in the end-user language to C calls;\nBoth are slightly different, they achieve the same, but it seemed to me a better approach to write bridge filters that keeps a binding of only the few exposed API calls, than having to reimplement the entire API glue. Also, if we provide a \u0026ldquo;kind of official\u0026rdquo; bridge, it will benefit everyone ultimately.\nSo how hard is it to write a bridge ? # It really depends on what the target language is. For instance, I had a working filter-python bridge in less than an hour without having ever used the python API before. The filter-perl bridge took a few hours. The code to make the bridge itself is rather short, it is simply a matter of mapping a callback to a translation function. For example, if we look at our filter-python and how it implements on_connect(), we would see something like this (just focusing on the interesting parts):\nstatic int on_connect(uint64_t id, struct filter_connect conn) { PyObject\tpy_args; PyObject\tpy_ret; PyObject\tpy_id; PyObject\tpy_local; PyObject\tpy_remote; PyObject *py_hostname; /* we translate our parameters to types that python can understand */ py_args = PyTuple_New(4); py_id = PyLong_FromUnsignedLongLong(id); py_local = PyString_FromString(filter_api_sockaddr_to_text((struct sockaddr *)\u0026amp;conn-\u0026gt;local)); py_remote = PyString_FromString(filter_api_sockaddr_to_text((struct sockaddr *)\u0026amp;conn-\u0026gt;remote)); py_hostname = PyString_FromString(conn-\u0026gt;hostname); /* we prepare for calling the on_connect function inside the python script */ PyTuple_SetItem(py_args, 0, py_id); PyTuple_SetItem(py_args, 1, py_local); PyTuple_SetItem(py_args, 2, py_remote); PyTuple_SetItem(py_args, 3, py_hostname); /* we let python do the work and give us back a return value */ py_ret = PyObject_CallObject(py_on_connect, py_args); Py_DECREF(py_args); if (py_ret == NULL) { PyErr_Print(); log_warnx(\u0026#34;warn: filter-python: call to on_connect handler failed\u0026#34;); exit(1); } return (1); } int main(int argc, char *argv[]) { /* we don\u0026#39;t care what\u0026#39;s above */ /* we look if there\u0026#39;s a symbol \u0026#34;on_connect\u0026#34; defined in the end-user filter */ py_on_connect = PyObject_GetAttrString(module, \u0026#34;on_connect\u0026#34;); /* if there\u0026#39;s one and it\u0026#39;s a function, we register our translating callback */ if (py_on_connect \u0026amp;\u0026amp; PyCallable_Check(py_on_connect)) filter_api_on_connect(on_connect); /* we don\u0026#39;t care what\u0026#39;s below */ } Just for the purpose of comparing with another bridge, here\u0026rsquo;s the filter-perl bridge, doing the exact same thing:\nstatic int on_connect(uint64_t id, struct filter_connect *conn) { const char\t*local; const char\t*remote; local = filter_api_sockaddr_to_text((struct sockaddr *)\u0026amp;conn-\u0026gt;local); remote = filter_api_sockaddr_to_text((struct sockaddr *)\u0026amp;conn-\u0026gt;remote); /* looks simpler than python ? I hid the gross parts in call_sub_sv() ;-) */ call_sub_sv((SV *)pl_on_connect, \u0026#34;%i%s%s%s\u0026#34;, id, local, remote, conn-\u0026gt;hostname); return filter_api_accept(id); } int main(int argc, char *argv[]) { /* we don\u0026#39;t care what\u0026#39;s above */ /* we look for a function \u0026#34;on_connect\u0026#34; defined in the end-user filter and register a callback */ if ((pl_on_connect = perl_get_cv(\u0026#34;on_connect\u0026#34;, FALSE))) filter_api_on_connect(on_connect); /* we don\u0026#39;t care what\u0026#39;s below */ } As you can see, adding support for writing filters in another language is not really a big challenge, I could write two bridges in an afternoon without having played with either one of the C/API for these languages. I\u0026rsquo;m sure someone else could have completed them faster than I did.\nNow, anyone willing to write a filter in perl or python can simply install the bridges then focus on writing the filter script itself in their language of choice. Here\u0026rsquo;s an example for Python:\nimport filter def on_connect(session, local_addr, remote_addr, hostname): print \u0026#34;on_connect:\u0026#34;, hex(session), local_addr, remote_addr, hostname return filter.accept(id) And here\u0026rsquo;s one for Perl:\nuse strict; use warnings; sub on_connect { my ($id, $l, $r, $h) = @_; return smtpd::filter_api_accept $id; } The nice part is that this comes with a very very very low overhead, the interpreter is started only once and the performances should be near native no matter which language is used.\nWhat about milters ? # Well, it should be possible to implement a milter bridge that can translate milter API calls to our API using the same technique as for translating python and perl calls to our API. There may be some changes required in our API, but we\u0026rsquo;re not opposed to that and if someone stepped up to write a milter bridge, then I would definitely help by adapting the API where needed\u0026hellip; as long as it remains simple and clean.\nNow, as far as I am concerned\u0026hellip;\nI have absolutely no interest in a milter interface myself as I will be using the native one, so clearly this is not even going to hit my todo list. In my opinion, having native filters is a much better goal than trying to make filters written for another MTA work with ours, I suspect there will always be corner cases that will turn this into a nightmare when the same effort could be put in writing quality native filters.\nIf someone wants to spend time trying to make milters run on OpenSMTPD, well, good luck and may you find entertainment in that journey !\nSo, what\u0026rsquo;s the state of filters ? # The filter API works. It is not done, it still has bugs, it still lacks features, but it\u0026rsquo;s in a state where you can actually write a filter and have it process all kinds of events and tell the SMTP server to reject or accept based on filter decision. I wrote tiny test scripts to verify that I could write filters to limit the number of connections from a same user, reject upon some keywords, alter content, skip headers, etc\u0026hellip; all of this works.\nJust as an example, here\u0026rsquo;s a filter I wrote this morning, all it does is declare callbacks for every hook and print the parameters it receives. I wrote it in Python because it\u0026rsquo;s easier for everyone to figure out what it does, but you could do the same in C or Perl:\nimport filter def on_connect(id, local_ip, remote_ip, hostname): print \u0026#34;on_connect:\u0026#34;, hex(id), local_ip, remote_ip, hostname return filter.accept(id) def on_helo(id, helo): print \u0026#34;on_helo:\u0026#34;, hex(id), helo return filter.accept(id) def on_mail(id, mail): print \u0026#34;on_mail:\u0026#34;, hex(id), mail return filter.accept(id) def on_rcpt(id, rcpt): print \u0026#34;on_rcpt:\u0026#34;, hex(id), rcpt return filter.accept(id) def on_data(id): print \u0026#34;on_data:\u0026#34;, hex(id) return filter.accept(id) def on_dataline(id, line): print \u0026#34;on_dataline:\u0026#34;, hex(id), line return filter.writeln(id, line) def on_eom(id, size): print \u0026#34;on_eom:\u0026#34;, hex(id), size return filter.accept(id) def on_commit(id): print \u0026#34;on_commit:\u0026#34;, hex(id) return filter.accept(id) def on_rollback(id): print \u0026#34;on_rollback:\u0026#34;, hex(id) return filter.accept(id) def on_disconnect(id): print \u0026#34;on_disconnect:\u0026#34;, hex(id) With this filter plugged in, the following SMTP session:\n220 debug.poolp.org ESMTP OpenSMTPD EHLO myself 250-debug.poolp.org Hello myself [127.0.0.1], pleased to meet you 250-8BITMIME 250-ENHANCEDSTATUSCODES 250-SIZE 36700160 250-DSN 250 HELP MAIL FROM:\u0026lt;gilles\u0026gt; 250 2.0.0: Ok RCPT TO:\u0026lt;gilles\u0026gt; 250 2.1.5 Destination address valid: Recipient ok DATA 354 Enter mail, end with \u0026quot;.\u0026quot; on a line by itself Subject: ohai ! test . 250 2.0.0: 2fb4c3b7 Message accepted for delivery Results in the following log on the server running in debug mode, with all filter callbacks triggered:\ndebug: smtp: new client on listener: 0x1c9bbad31000 smtp-in: New session a7bc78b6177f09b5 from host localhost [127.0.0.1] on_connect: 0xa7bc78b6177f09b5L 127.0.0.1 127.0.0.1 localhost on_helo: 0xa7bc78b6177f09b5L myself on_mail: 0xa7bc78b6177f09b5L gilles@debug.poolp.org on_rcpt: 0xa7bc78b6177f09b5L gilles@debug.poolp.org on_data: 0xa7bc78b6177f09b5L smtp: 0x1c9c37af3000: fd 5 from queue smtp: 0x1c9c37af3000: fd 7 from filter on_dataline: 0xa7bc78b6177f09b5L Received: from myself (localhost [127.0.0.1]); on_dataline: 0xa7bc78b6177f09b5L by debug.poolp.org (OpenSMTPD) with ESMTP id 2fb4c3b7; on_dataline: 0xa7bc78b6177f09b5L for \u0026lt;gilles@debug.poolp.org\u0026gt;; on_dataline: 0xa7bc78b6177f09b5L Fri, 12 Dec 2014 15:48:42 +0100 (CET) on_dataline: 0xa7bc78b6177f09b5L Subject: ohai ! on_dataline: 0xa7bc78b6177f09b5L on_dataline: 0xa7bc78b6177f09b5L test debug: smtp: 0x1c9c37af3000: data io done (195 bytes) on_eom: 0xa7bc78b6177f09b5L 195 filter: eom not received yet debug: 0x1c9c37af3000: end of message, msgflags=0x0000 debug: scheduler: evp:2fb4c3b7f8d5ee1d scheduled (mda) smtp-in: Accepted message 2fb4c3b7 on session a7bc78b6177f09b5: from=\u0026lt;gilles@debug.poolp.org\u0026gt;, to=\u0026lt;gilles@debug.poolp.org\u0026gt;, size=195, ndest=1, proto=ESMTP on_commit: 0xa7bc78b6177f09b5L CAN I HAZ IT FOR REAL !!? # Filters are not stable yet, just this morning I have fixed a bug that lead to a crash and another user has found a corner case which causes the filter glue to crash when a filter forks. They\u0026rsquo;re not just disabled, but also removed from our stable release. Unlike a while ago, there\u0026rsquo;s no way to activate them in a stable release. Gone. Bye-bye. Ma3 salam.\nFilters support is only available in our development master and portable branches which are mirrored on Github. It is also available in our snapshots, which we publish every now and then from these branches. However\u0026hellip; they are not meant for general use.\nPeople keep complaining that we didn\u0026rsquo;t document the API, but what they fail to grasp is that:\nwe don\u0026rsquo;t write documentation before actual code, we write code then document; we don\u0026rsquo;t document code that is subject to heavy changes, it\u0026rsquo;s not meant to be used, it should be ignored, you should pretend it doesn\u0026rsquo;t exist; we have a working API but it\u0026rsquo;s not done, it has issues, we don\u0026rsquo;t want people to use it;\nAt the time being, we leave it undocumented because it makes it harder for people who can\u0026rsquo;t read code to actually play with it, hit issues that aren\u0026rsquo;t bugs and confuse us about what is a real issue and what is a user bug.\nWe want people to help us squash the bugs in this API, however we can\u0026rsquo;t hold hands of everyone willing to give this API a try and write a tiny filter. So we decided to keep the API undocumented for now and only help those that can read code and figure things out on their own, we want to keep the noise ratio to the lowest. If you don\u0026rsquo;t know C, I would discourage you to play with the API yet, even if you intend to write filters using a bridge. You will be on your own and unlikely to go much far.\nI wanted to show code today because so many people keep requesting a filter API and we\u0026rsquo;ve been saying that \u0026ldquo;it\u0026rsquo;s a work in progress\u0026rdquo; for so long that it looked like vaporware. I hope this post clarifies that we actually HAVE an API, a usable one on top of that, but it\u0026rsquo;s just not ready and we\u0026rsquo;re too close to having what we want that we\u0026rsquo;re not going to ruin it all by rushing to deliver something half-baked.\nNote that I didn\u0026rsquo;t show how you could actually plug it, this was done on purpose. If you manage to find by yourself, feel free to help us move the API forward by starting to write your own filters. If you need help to activate, you\u0026rsquo;ve failed the easiest test, by all means stay away from this API until it has stabilized.\nAWWWWW :-( # Don\u0026rsquo;t be sad :-(\nWe\u0026rsquo;re working precisely on getting this API stable enough that we can ship it with our next major release. It will be considered experimental, but it will be shipped and you will be able to play with it.\nBefore you ask, no we don\u0026rsquo;t know when that release would be. Unlike other releases which we can plan on a regular schedule, this one is so invasive and the feature is so big that we will delay for as long as it takes before planning the release. The only thing delaying the major release is the filter code.\nAnd don\u0026rsquo;t get confused, there will be an OpenSMTPD-5.4.4 release soon\u0026hellip; but this is a minor release, not the one with filters.\n","date":"12 December 2014","permalink":"/posts/2014-12-12/the-state-of-filters/","section":"Posts","summary":"TL;DR: yeeeees, filters are coming. don't believe us ? here's an example. be patient. On my death bed # On my death bed, when my life flashes before my eyes and I start recalling what people have told me during my (hopefully long) lifetime, these sentences will single out:","title":"The state of filters"},{"content":"EHLO,\nThis is the third post of a series about OpenSMTPD improvements that have taken place since this summer.\nContent altering # For a long time, we have developed OpenSMTPD with a strict rule that the daemon should not alter DATA (as in the DATA SMTP phase) in any way.\nThe rationale was that by enforcing that rule, the message writing was simplified as the smtp process would simply read data from a client and write it, without any post-processing, to a file descriptor. We didn\u0026rsquo;t want to prevent the writing of data modifiers, but we wanted to move this feature out of the daemon and into the filtering API which would allow any kind of rewriting outside the daemon memory space and with different privileges if we wanted to.\nThe other rationale was that by avoiding adding hooks to alter content we were receiving, we made it harder on us and contributors to go lazy and implement every feature out there in the daemon just because it was easier through the hooks. When we wanted to implement a feature, we would immediately ask ourselves if it could be done outside the daemon scope before eventually adding it.\nBesides the occasional complaints that we did not support address masquerading (yet), this did not really cause any issue with our users running a wide range of MUA and we decided to maintain that rule, convince people that masquerading was really a filter thing and that it would become available as soon as our filter API had stabilized.\nThen we became default MTA on OpenBSD # We started getting complaints from hackers that, although without a value, the BCC header was being emitted by OpenSMTPD.\nIt seemed rather strange because clients typically strip them from the DATA phase and we never actually receive them. It turned out that some older MUA did not do that stripping and I had to add a basic check to skip BCC line(s) if any appeared in the DATA phase before the message content. I was not too happy about that change, and a few other hackers/people either as it turns out, but it was not really altering content, simply discarding content that we were not meant to receive in the first place so\u0026hellip;\nThen came another complaint: some hackers were receiving mails from some other hackers \u0026hellip; but the mail had a From header with the local domain instead of the sender domain.\nThis also seemed strange because the \u0026ldquo;local domain\u0026rdquo; is not necessarily the \u0026ldquo;sender domain\u0026rdquo;. For instance, if you mail me at @opensmtpd.org, the destination domain will be @opensmtpd.org but the local domain will be poolp.org as that\u0026rsquo;s the local domain of the MX machine. This could only mean that the domain had been inserted by the receiving MX because it was missing.\nNow, when a mail goes out, it can take two paths:\nit can go through a SMTP connection to localhost; or it can use the local enqueuer, send-mail or sendmail, which are actually both mapped to smtpctl working in enqueue mode; Many MUA support both modes, but usually rely on the second mode by default when ran locally except for a few modern ones that need to be configured explicitly to do so.\nWhat bothered me was that modern MUA usually append a domain so they could pretty much all be ruled out, and older MUA tend to rely on the local enqueuer which has code to insert the local domain if it is missing. In these two cases, the mail would have not been emitted without a domain.\nSo the logical explanation was that the sender used a MUA that both bypassed the local enqueuer AND did emit a From without a domain. After a bit of testing, this proved to be the case and it brought hell on me.\nAddresses Parsing # It became clear that if we wanted to continue supporting these older MUA, some basic rewriting was required in the daemon and were no longer just a feature but a mandatory part of the session processing. Without this, an OpenSMTPD server could send mails to another MTA such as Sendmail or Postfix and the recipient would get unexpected headers due to the local rewriting performed by them.\nAs a quick fix, I wrote a small addresses parser that would extract From, To and Cc fields. It would then attempt to parse the addresses in them, check if a domain was missing and append the local domain in that case. This proved to be a very very bad idea because unlike email addresses in the SMTP protocol, the email addresses inside a message can be written in many ways and are a nightmare to parse. Addresses can span on multiple lines, have comments inside them, they can have brackets or not, they can be surrounded by components, parts of the representation may use different encodings and we even ran into representations such as \u0026ldquo;Gilles Chehade\u0026rdquo; which goes beyond mind-fuckery.\nThe issue was not just the parsing, but also that once appended the address had to be rendered back so not only I had to find a way to parse the way-too-many fucked up representations into a structure I could work on, but I had to be able to render the fucked up representation back with the domain inserted in them.\nI took a first approach of not doing the replacing in place, but rather constructing a list of sanitized addresses that would be output as a multi-line header. The addresses would be parsed into a common structure and rendered so that (superfluous spaces intended):\nFrom: \u0026quot;Gilles Chehade\u0026quot; gilles, \u0026quot;Eric Faurot\u0026quot; eric@poolp.org, \u0026quot;Charles Longeau\u0026quot; \u0026lt; chl \u0026gt; would turn into:\nFrom: \u0026quot;Gilles Chehade\u0026quot; \u0026lt;gilles@poolp.org\u0026gt;, \u0026quot;Eric Faurot\u0026quot; \u0026lt;eric@poolp.org\u0026gt;, \u0026quot;Charles Longeau\u0026quot; \u0026lt;chl@poolp.org\u0026gt; But this lead to another round of complaints that the rewriting was altering the message structure, not only did it expand multi-line (which is RFC compliant) but \u0026lsquo;\u0026lt;\u0026rsquo; and \u0026lsquo;\u0026gt;\u0026rsquo; appeared surrounding the addresses even if they were lacking in the original header.\nIt became really clear at this point that any attempt at taking a \u0026ldquo;parse-then-render\u0026rdquo; approach was not going to work.\nMessage Parser # After a discussion with eric@, I decided to take a different approach and split the problem in two smaller problems.\nThe first one is locating the specific parts of a message that need to be altered, in this case some specific headers. The second one is to actually do the appending somehow.\nLocating the specific parts of a message is a seemingly simple issue, however it raises some concerns of its own. The DATA arrives line by line and the headers may span on multiple lines, therefore the processing can\u0026rsquo;t be done as lines arrive but requires a bit of context and knowing which line is the last one \u0026hellip; when this information is actually carried by the next one which has not yet been received.\nThe simplest way to deal with the problem is to take a full message parser, such as the one we have in the enqueuer, grab the part that deals with the headers and feed it with the full headers for the session in progress.\nTo achieve this, we could either buffer the headers in memory during the session then work on that buffer or, since we\u0026rsquo;re writing the full message to a file anyway, have the message parser work on that file before it is committed to the queue. Unfortunately, neither of these works with us:\nKeeping full headers in memory paves the road for resources starvation attacks. On a default install, a message can be up to 35MB with many people bumping that value further. Depending on your operating system, OpenSMTPD may accept several thousands of concurrent clients. Now, if we went that path, an evil human being could simply craft very large messages consisting solely of headers and have thousands of clients emitting them without sending the final \u0026ldquo;.\u0026rdquo; but waiting for the server timeout to trigger instead. You get the idea why this is not a good idea.\nWorking with the file we saved the message to is slightly better, however two issues pop-up.\nFirst, we can\u0026rsquo;t do in-place editing so we will basically be rewriting the entire content to another file. For large messages, the performances penalty will be very visible. Then, we have an atomicity requirement, we can\u0026rsquo;t acknowledge the client that we have accepted the message until it has been committed to queue, so between his final \u0026ldquo;.\u0026rdquo; and our acknowledgement, he will have to wait for the entire message to be processed, copied to the other file and committed to queue\u0026hellip; during that time, he will be consuming a session slot preventing another client from being handled. If he gets lucky, he may even have a chance to hit a timeout causing his disconnect while the server will abort processing of his message and trash it.\nSo, nope, working with full message or full headers is not the way\u0026hellip; data has to be processed as a stream.\nStream Message Parser # I discussed the idea with eric@ a bit and then came up with a prototype for a stream message parser that we would plug in the SMTP session.\nvoid rfc2822_parser_init(struct rfc2822_parser *); int rfc2822_parser_feed(struct rfc2822_parser *, const char *); void rfc2822_parser_reset(struct rfc2822_parser *); void rfc2822_parser_release(struct rfc2822_parser *); int rfc2822_header_callback(struct rfc2822_parser *, const char *, void (*)(const struct rfc2822_header *, void *), void *); void rfc2822_header_default_callback(struct rfc2822_parser *, void (*)(const struct rfc2822_header *, void *), void *); void rfc2822_body_callback(struct rfc2822_parser *, void (*)(const char *, void *), void *); Before, the workflow would basically go like \u0026ldquo;read a line from this client, write that line to this file\u0026rdquo;.\nNow, the SMTP server will associate a parser context to every session and register a set of callbacks. The default callbacks will simply receive the line as parameter and write it to the file, doing exactly what was done before. However, they now get a chance to modify it somehow before writing it.\nFor example, before the parser was integrated, the BCC remove code I had quicked-fix required adding code to the session reading function to detect if we were still in headers, if there was a continuation line, as well as explicit strcasecmp() checks to bypass the writing. It was not much, but it made it clear that the function would become a big pile of crap once we started adding other special cases.\nNow, it is as simple as registering a callback for that header:\nrfc2822_header_callback(\u0026amp;s-\u0026gt;rfc2822_parser, \u0026#34;bcc\u0026#34;, header_bcc_callback, s); And writing a callback that \u0026hellip; does nothing:\nstatic void header_bcc_callback(const struct rfc2822_header *hdr, void *arg) { } During a session, the lines read from the clients are passed to the rfc2822_parser_feed function which then takes care of all context handling, buffering and calling registered callbacks when needed. The callbacks will receive headers as a rfc2822_header structure holding a list of lines, and the lines can then either be written as is to a file or modified at will. A message crafted to exhaust resources by providing a single header with an insanely large number of lines will cause the parser to generate an error that the server will properly handle, so all cases solved.\nWith this in, the code in the SMTP session remains clear with no special cases and no \u0026ldquo;state\u0026rdquo; variables all over the place.\nSo how do we deal with addresses ? # Let\u0026rsquo;s get back to the main issue: fixing addresses.\nSo, the message parser has made it possible to extract specific headers and we could now do the following without having to kludge the session reading code:\nrfc2822_header_callback(\u0026amp;s-\u0026gt;rfc2822_parser, \u0026#34;from\u0026#34;, header_masquerade_callback, s); rfc2822_header_callback(\u0026amp;s-\u0026gt;rfc2822_parser, \u0026#34;to\u0026#34;, header_masquerade_callback, s); rfc2822_header_callback(\u0026amp;s-\u0026gt;rfc2822_parser, \u0026#34;cc\u0026#34;, header_masquerade_callback, s); But how do we implement header_masquerade_callback ?\nInstead of using the previous \u0026ldquo;parse-and-render\u0026rdquo; approach, I decided to give a try at another way: locate insert points.\nWhile it is hard to parse the header into a series of per-address structures that I can work with and render back, it is not too hard to parse the header into a series of buffers in which I can check if a domain is missing and locate where it should have been. This technique has the advantage that once the processing has been done, the buffers can be written back in sequence preserving the structure of the original message.\nFor instance:\nFrom: \u0026quot;Gilles Chehade\u0026quot; gilles, \u0026quot;Eric Faurot\u0026quot; eric@poolp.org, \u0026quot;Charles Longeau\u0026quot; \u0026lt; chl \u0026gt; Would now be parsed into three buffers containing (note that the various superfluous spaces were preserved):\n\u0026quot;Gilles Chehade\u0026quot; gilles \u0026quot;Eric Faurot\u0026quot; eric@poolp.org \u0026quot;Charles Longeau\u0026quot; \u0026lt; chl \u0026gt; The domain-appending logic would then go through the three buffers and determine:\nis a domain appending necessary ? if it is, where in the buffer should it be inserted ? The second buffer would be unchanged while the first and third buffers would result in an insert point right after gilles and right after chl. The code would then simply copy the buffer to another one up to that insert point, insert the domain, then copy the remaining. Once all buffers have been processed, they are written sequentially resulting in:\nFrom: \u0026quot;Gilles Chehade\u0026quot; gilles@poolp.org, \u0026quot;Eric Faurot\u0026quot; eric@poolp.org, \u0026quot;Charles Longeau\u0026quot; \u0026lt; chl@poolp.org \u0026gt; Again note that structure has been preserved, header has not be rendered multi-line and superfluous spaces in the original header are left as is.\nSo\u0026hellip; we\u0026rsquo;re all done ? # This has been committed several weeks ago and there\u0026rsquo;s been no complaints so I guess the solution was correct.\nWe\u0026rsquo;ve discussed about masquerading a bit and maybe we will reevaluate our position that it would be better off in a filter, after all \u0026hellip; the code to do the domain append is masquerading so if we\u0026rsquo;re going to do it for the incoming case we might as well make it handle the outgoing case too.\nHowever, our stance has not changed and while the message parser offers a lot of ease for in-daemon content altering, we still believe that the bulk of content altering is filters material. If only to benefit from privileges separation while parsing untrusted content.\n","date":"10 December 2014","permalink":"/posts/2014-12-10/some-opensmtpd-overview-part-3/","section":"Posts","summary":"EHLO,\nThis is the third post of a series about OpenSMTPD improvements that have taken place since this summer.\nContent altering # For a long time, we have developed OpenSMTPD with a strict rule that the daemon should not alter DATA (as in the DATA SMTP phase) in any way.","title":"Some OpenSMTPD overview, part 3"},{"content":" Why we killed the MFA (filter) process # For as long as I can remember, a process called MFA was created by OpenSMTPD at start time.\nMFA stood for \u0026ldquo;Mail Filter Agent\u0026rdquo; and the goal of that process was initially to take care of all filtering tasks ranging from filtering senders based on the ruleset matching to starting and controlling filters.\nAs time passed by, we figured the lookup process was better suited for ruleset matching and the MFA process became mostly idle, we renamed it to \u0026ldquo;filter\u0026rdquo; process since the only thing it had to do was take care of starting filters and making sure they had the proper environment.\nAt this point, we should have realized that this was unnecessary but this process had been with us for so long that it seemed just right. After all, we gave it a name, how do you kill something you named \u0026hellip;\nAnyways, as we improved the filter API design it became clear that the filter process kept getting in the way of us doing things in a simple way. We had all informations available at a place, yet we had to make an indirection and pass data back and forth just so the process could get a chance to acknowledge it didn\u0026rsquo;t care. If the filter process is no longer needed after starting the filters, why not have them started by another process which actually does something useful afterwards instead of staying idle ?\nEric offered to take the process down and while it was heart-breaking, I encouraged him to kill it with fire.\nThe pony process # There\u0026rsquo;s been complaints that the daemon was starting too many processes and that probably some could be merged.\nAfter a bit of brainstorming, we figured that the smtp process in charge of accepting incoming sessions, the mta process in charge of starting outgoing sessions and the delivery process in charge of delivering mail locally were a bit related: they were all unprivileged, running as _smtpd, chrooted to /var/empty and far from being busy even in scenarios where the server was busy itself.\nIn addition, they all shared a common design, they all had a notion of sessions following the same pattern and we figured that without too much efforts we could merge them into a single process.\nI did the initial work to merge them three into a single process and eric@ did another pass to cleanup the names of the IMSG we were passing to improve further.\nPretty neat, hu ? Indeed. However we faced another challenge: this change proved to be controversial.\nWhile pretty much everyone agrees that this was the way to go (actually I have no idea, I didn\u0026rsquo;t run a poll), some people were not too receptive to the name we gave that process: \u0026ldquo;pony\u0026rdquo;. It is apparently unprofessional and should we leave it that way, gosh so much $$$ are we going to lose.\nActually \u0026ldquo;pony\u0026rdquo; was not meant to stay, it had nothing to do with the pony trend on internet and the name was chosen because we couldn\u0026rsquo;t agree on a proper name for that process which did smtp, mta and delivery. Instead of spending two hours trying to find a name, I suggested we name it \u0026ldquo;pony express\u0026rdquo; as an hommage to the courageous messengers of the far west. If someone came up with a better name, we could just switch, I just didn\u0026rsquo;t want to be stuck on that meaningless task. Until now, the challenge remains unsolved.\nAlso, I promised blambert@ a pony during a hackathon in Budapest so \u0026hellip; one stone, two birds.\nThe klondike process # As you may have seen, this year was a terrible shot for the OpenSSL project. The Heartbleed vulnerability has shaken the internet and it even came out with a logo.\nA few months earlier, for unrelated reasons I had started working on reducing the amount of time a key is exposed in the memory-space of a network facing process. I didn\u0026rsquo;t solve the issue completely but tried to improve it slightly. My improvements would have helped in some cases but not with Heartbleed.\nIn the wake of the catastrophe, reyk@ implemented a privilege-separated RSA engine which allowed the RSA private parts (huhu) to remain COMPLETELY isolated from the network facing processes. His solution wasn\u0026rsquo;t a fix to Heartbleed but a proper design which would have made it inoperant.\nOf course, I immediately asked him if he could help integrate in OpenSMTPD which he did\u0026hellip; in a process named \u0026ldquo;klondike\u0026rdquo;\u0026hellip; because we\u0026rsquo;re all \u0026ldquo;pony express\u0026rdquo; and unprofesionnal :-p\nThe OpenSMTPD-extras # For a long time, we shipped countless pieces of code that didn\u0026rsquo;t really belong in our repository.\nNot because they\u0026rsquo;re not related to OpenSMTPD but because they are not really useful in the stock daemon, they came with some dependencies that we didn\u0026rsquo;t want to have or because too few people use them that we want to maintain them as part of the official release.\nSo I lobbied and lobbied and lobbied to convince eric@ and chl@ that we needed an OpenSMTPD-extras repository where we could move table_ldap, table_mysql, table_postgres, table_redis and table_whateverwecomeupwith to avoid keeping too many things in our main repository.\nSome work was done to split the code between these two repositories and make sure that the content of OpenSMTPD-extras would build by itself. This work was mainly done by chl@ and myself, and turned out not to be too painful, it only took us a couple hours to finish the split.\nThe idea behind this repository is that the OpenSMTPD repository is essentially read-only, only developers can commit and the commits must follow strict rules with regard to dependencies and what they bring to the server. We don\u0026rsquo;t want to ship mysql or posgres support with the daemon bringing their dependencies, and that\u0026rsquo;s even more relevant now that they can be standalone backends that can be packaged separately.\nBy providing a separate repository, we can remove these restrictions: the OpenSMTPD repository remains strict, the OpenSMTPD-extras repository is a set of unrequired plugins that can ship with whatever dependencies they need and we won\u0026rsquo;t be strict about that. Of course that doesn\u0026rsquo;t mean we will allow bad code to creep in, but we will be less strict as these will be optional user contributions, kind of what \u0026ldquo;ports\u0026rdquo; are for the BSD.\nThe OpenSMTPD-extras repository is now packaged in OpenBSD ports at least.\n","date":"1 December 2014","permalink":"/posts/2014-12-01/some-opensmtpd-overview-part-2/","section":"Posts","summary":"Why we killed the MFA (filter) process # For as long as I can remember, a process called MFA was created by OpenSMTPD at start time.","title":"Some OpenSMTPD overview, part 2"},{"content":" EHLO world ! # Yesterday I thought I\u0026rsquo;d write a first OpenSMTPD-related post to sum up the changes that have happened since this summer but it turned out to be painful as they amount to quite a lot. Instead, I think a better strategy is to split this into a serie of smaller posts focused on the specific changes ;-)\nSo, in June I have organized a mini hackathon at my brand new place and invited some of the French Connection at home. The goal was to work a bit and find an excuse to watch the French team take over the world cup before an eventual surrender once everyone had left. Since it was an OpenSMTPD hackathon, eric@ and chl@ were present, but we also invited miod@ who had been hosting me several times at his hackathons and jca@ who turned out to live close to my place.\nEric and I planned to finish work on the filter API which would allow plugging third-party filters to the daemon for session-time and data post-processing, Charles had a few things to cleanup in the portable branch, and we all had a few other things we wanted to deal with.\nToday, I will talk about the filter API and pluggable backends.\nThe filter API # Filters have been long awaited in OpenSMTPD, it\u0026rsquo;s been a constant feature-request for at least two years.\nIn fact, filters have existed in OpenSMTPD for a long time as an (unsatisfying) unexposed prototype, but we had strong ideas about where we wanted to go with this subsystem and this was just not it.\nWe decided to start from scratch and only retain one idea from the prototype, the fact that a filter is not a shared object. The idea behind that was that we wanted filters to run outside of the daemon memory space and be able to provide per-filter privilege separation and chrooting. Of course, we didn\u0026rsquo;t want to execute the filter for every session as the overhead would be ridiculous, so a filter would effectively be a daemon communicating with the server through IPC.\nNow, our other goal was to make filters dead simple for both users and developers and having to write a daemon with IPC was kind of raising the bar too high. So the biggest challenge was to overcome this and make sure that neither of them would need to actually see that.\nWe came up with an API which turned the filters into event machines, a filter writer would call a filter_init() function that would take care of all the daemon/imsg part, then call a set of functions to register callbacks on specific events and finally call a filter_loop() function to make the filter start processing requests. These ideas were already part of the first prototype, but had to be revamped to allow for privilege separation and chrooting to be doable.\nAt the same time, eric@ completely rethought filter chaining to allow filters to pass data one to another, it was a much more complex task than it looked as the tricky parts are not getting it to work, but getting it to fail gracefully if a filter suddenly craps itself.\nBy the end of the hackathon, we had the subsystem mostly done and functional to a point where I could write both a python bridge filter and a php bridge filter so filters could be written in these languages, and write a few dummy filters to alter content, turn it into l33tsp34k, reject mails containing viagra and what not.\nShortly after, we received our first filter contribution from a community member who wrote a DKIM signer using our API, another feature long awaited :-)\nOur next major release should come with this API enabled, the only thing blocking us from doing so at the moment is the lack of feedback and confidence that it is rock solid. People have been requesting this API for ages yet it is very very hard to actually get people to use and comment on it.\nPluggable backends # As I already mentionned in previous posts, we had done a lot of work on abstracting the queue API to make it \u0026ldquo;pluggable\u0026rdquo; in the sense that one can write a queue in a standalone file, expose it through a structure and have a pointer point to that structure to start using it instead of our stock queue. This has been done a long time ago to make it easier for us to experiment, revert and experiment again with confidence we didn\u0026rsquo;t break something else in the way.\nFrom OpenSMTPD\u0026rsquo;s perspective, a queue is basically an iterable key-value store and all it has to know is that messages and envelopes have identifiers that it can pass to the queue operations which will know what to do with them. It no longer has any knowledge of the internals and you can basically move a queue to a database, a cloud or your own filesystem-based storage and the server will not notice a difference. As a matter of fact, the first prototype we wrote to prove this worked was to store a queue on the Google cloud, and let either one of eric\u0026rsquo;s or my laptop schedule the message without having it locally.\nAt some point, I convinced eric@ that we should have a similar interface for the scheduler if only to make sure that the daemon did not make any wrong assumption on scheduling. I came up with a prototype while at an OpenBSD hackathon in Budapest two years ago and once it started working, eric@ took over, fixed some API layer violations and improved it further. All over last year, we had to do many changes to the scheduler while running OpenSMTPD in a high-volume environment and this has proved to be a very useful change.\nSo\u0026hellip; if we have to play games with pointers and rebuild, that\u0026rsquo;s not really \u0026ldquo;pluggable\u0026rdquo; is it ?\nWhen I first considered splitting the subsystems into their own isolated API, what I had in mind was not just to make it easier to test things. I wanted to be able, just like filters, to provide an API that would allow plugging custom queues and schedulers without having to modify something in the server. People have tons of different use-cases and we want to keep the OpenSMTPD code base as simple, clean and dependency-free as possible. So by providing the proper API we can ensure that people can deal with their use-cases without having us to cooperate, they\u0026rsquo;re free to use whatever dependency and we will not have to even care.\nDuring our little hackathon, I asked eric@ what was needed to avoid the pointer juggling and just make it possible to set it from our configuration file. He worked on it for a few hours and we ended up with a version that would allow me to build a queue as an external executable that could be plugged and communicate with the daemon like filters. Since I was in a pythonic mood, I wrote a queue-python which allowed me to write a queue backend in python and used it to write a prototype queue_mysql in python. This was not really intended to be a serious project, just a proof of concept that this kind of things can be achieved, but it only took a couple hours to complete these and the python implementation of a queue is so small that it paves the road to interesting experiments :-)\nEric also made the scheduler pluggable, I didn\u0026rsquo;t write one as I got context-switched to something else, but you get the idea of what can be achieved now.\nWhat\u0026rsquo;s coming in next post ? # I\u0026rsquo;ll be discussing the new OpenSMTPD-extras repository which holds all pluggable contents, as well as the smtp/mta processes merge, the mfa process removal and reyk@\u0026rsquo;s RSA privsep engine.\nEric will be talking about asr, the asynchronous resolver, and how we pushed him into making it a standalone library.\nI\u0026rsquo;ll convince Charles to talk a bit about portable :-)\n","date":"26 November 2014","permalink":"/posts/2014-11-26/some-opensmtpd-overview-part-1/","section":"Posts","summary":"EHLO world ! # Yesterday I thought I\u0026rsquo;d write a first OpenSMTPD-related post to sum up the changes that have happened since this summer but it turned out to be painful as they amount to quite a lot.","title":"Some OpenSMTPD overview, part 1"},{"content":"Good news everyone !\nWith last post dating from almost a year, we can all agree that I\u0026rsquo;ve outpassed my slacking skills by a great margin and I probably deserve an award of some kind ;-)\nAnyways, there\u0026rsquo;s been a few complaints during these 12 months because this blog was the main source of information for the OpenSMTPD project and several posts which used to describe features and configuration samples had suddenly disappeared. I\u0026rsquo;m sorry about that, I guess an explanation is in order.\nI didn\u0026rsquo;t close it out of rage, nor trip on a cable, nor lose content in a database crash or whatever. I made a bad tech decision at a very bad time.\nWhat happened is that this blog relies on another project of mine for storage, and that project needed some love, so I had taken the decision to simply put the blog in pause for the week-end until I could finish what I wanted to finish on that other project.\nWhile I had taken it down for a few days, life bit me. Family and health issues I was dealing with at that time had worsened, then while coping with that an opportunity to leave Paris and move to the city of Nantes had popped-up overflowing me with much higher priority things to deal with like preparing the moving out for one. As you can imagine, the blog has plummeted at the booooooottom of a very high list of things I had to take care of.\nNow, I\u0026rsquo;ve moved in to Nantes for a few months already, things have eventually settled down for the better or the worse, and I finally managed to find some time to work back on fixing my other project to bring this blog back up. I\u0026rsquo;m not done, but I don\u0026rsquo;t intend to take it down again so I hacked up a shit-tons of indirections here and there to let me improve the storage without taking the entire universe down with me.\nI\u0026rsquo;ll resume posting about OpenSMTPD and technical topics starting next week, I still need to finish a few unrelated things first.\nNote that this is not an official OpenSMTPD website, just the personal blog of one of the developers so unlike before I will be posting more and more stuff that are unrelated and implement categories so that you don\u0026rsquo;t have to go through my personnal / political / music / martial arts / pro-Palestinian ramblings ;-)\nBe back in a few days !\n","date":"24 November 2014","permalink":"/posts/2014-11-24/what-the-fsck-did-just-happen/","section":"Posts","summary":"Good news everyone !\nWith last post dating from almost a year, we can all agree that I\u0026rsquo;ve outpassed my slacking skills by a great margin and I probably deserve an award of some kind ;-)","title":"what the fsck did just happen"},{"content":"EHLO readers,\nThis blog post is the first since a few months, I\u0026rsquo;ve been busy and struggling with some personal health and familial issues. I won\u0026rsquo;t share them here as its not really something anyone can help with, so\u0026hellip; let\u0026rsquo;s focus on OpenSMTPD !\nWhat happened since last post\nWhen I wrote the last blog post, we had just released 5.3.2 which was a minor release that fixed a few non-critical bugs that were reported to us since the first major release a few months earlier.\nA while later, we released another minor release, 5.3.3, that also fixed minor bugs and brought some new non-invasive features to deal with common use-cases reported by our increasing user base.\nOpenSMTPD 5.3.3 was very stable, it\u0026rsquo;s been running on busy servers at work and we did not experience any bug with it while accepting and routing millions of daily messages with remote hosts on several machines.\nIt was a nice release for what it\u0026rsquo;s worth :-)\nWhat now ?\nWell, we didn\u0026rsquo;t stop hacking on OpenSMTPD and since 5.3.3 we have gone through lots of simplifications and adding new features. There are actually so many changes that a blog post can\u0026rsquo;t possibly go through all of it but I\u0026rsquo;ll discuss some of the most important and visible ones.\nWe have released new major version 5.4.1 a few days ago, and the features that are described below are all part of it. It is a very good release IMO and you should definitely take time to switch your 5.3.x setups to this new one.\nsmtpd.conf improvements\nFirst of all, despite being already very simple our configuration file has been simplified further and new features were introduced.\nFor example, certificate configuration had always been a bit tricky because we inherited an old behaviour were we would infer file names from a certificate name. This imposed certificates to be in a specific location /etc/mail/certs and to follow a naming conventions so that a certificate foobar would expect /etc/mail/certs/foobar.key and /etc/mail/certs/foobar.crt to be found.\nI was heavily bothered with this because I store my certificates and keys in /etc/ssl and /etc/ssl/private respectively and didn\u0026rsquo;t like having the /etc/mail/certs path imposed because that\u0026rsquo;s the way we had it with Sendmail. For years I used symbolic links but that was a hack around fixing this broken behavior.\nAlso, some people were confused because not only they could provide the path to their certificates but the listen directive expected parameters to be provided in a specific order which meant that:\nlisten on all tls hostname example.org certificate example would fail while:\nlisten on all tls certificate example hostname example.org would work the way they expected. We therefore went into a grammar cleanup to ensure that parameters were swappable where it made sense and to come up with a much simpler way to deal with certificates. The pki directive was introduced which was no longer tied to a listen statement but declared a database of certificates and keys associated to a hostname:\npki example.org certificate \u0026ldquo;/etc/ssl/example.org.crt\u0026rdquo; pki example.org key \u0026ldquo;/etc/ssl/private/example.org.key\u0026rdquo; With this in place, it became possible to simplify the configuration by referencing the hostname in places where it made sense:\nlisten on all tls pki example.org accept for any relay pki example.org and with parameters swappable, it was now also possible to use either one of these:\nlisten on all tls pki example.org listen on all pki example.org tls with the same expected result.\nWhile there, we decided to introduce a set of new features to our grammar so that use-cases that previously required several rules could be simplified. For example, we introduced the possibility to use negations:\naccept from ! local [\u0026hellip;] reject for ! domain We also added the possibily to filter senders and recipients at the ruleset level:\naccept from any sender \u0026ldquo;@poolp.org\u0026rdquo; [\u0026hellip;] accept from any sender \u0026ldquo; gilles@poolp.org\u0026rdquo; [\u0026hellip;] accept for domain poolp.org recipient \u0026ldquo;@poolp.org\u0026rdquo; [\u0026hellip;] accept for domain poolp.org recipient \u0026ldquo; gilles@poolp.org\u0026rdquo; [\u0026hellip;] accept from any sender [\u0026hellip;] And pushed it further by accepting wildcards in domain parts:\naccept for domain poolp.org recipient \u0026ldquo;gilles@*.poolp.org\u0026rdquo; [\u0026hellip;] Eric thought it would be a good idea to be able to accept mail for domains but require that the mail be forwarded elsewhere. We therefore introduced the forward-only keyword which allows you to:\naccept for domain poolp.org foward-only Accepting mails for local recipients ONLY if they are redirected to an external address either through the use of a ~/.forward file or by an alias.\nWith these the configuration file became quite more powerful while keeping the intended simplicity. But \u0026hellip; we did more ;-)\nUntil now, a listener had to declare it\u0026rsquo;s hostname:\nlisten on all hostname myhostname But this meant that if you had a single interface accepting mails for various domains, you had to list them all by their IP addresses:\nlisten on 192.168.1.1 hostname mx1.poolp.org listen on 192.168.1.2 hostname mx1.pool.ps Eric introduced the hostnames table to perform address to hostname lookups, allowing for a single listen line to use a hostnames table:\nlisten on all hostnames and reduce considerably the size of a multi-domain setup.\nMore features were added later but they are not part of the latest major release so I will discuss them in a post next week ;-)\nTLS improvements\nThe new pki stuff in smtpd.conf allowed us to simplify a bit of code internally and it allowed us to implement new features. Some where not mature enough to make it to the major release (I\u0026rsquo;ll talk about them next week), however there\u0026rsquo;s been a couple improvements that did make it to the release.\nWe have introduced support for TLS Perfect Forward Secrecy. We already had support for ephemeral key exchange for years, but after a suggestion in the OpenBSD hackers list to add EC support to another project, we came up with a diff to bring support for ECDHE in OpenSMTPD. It was such a simple change that it was committed the same day and we saw the result almost immediately when reading headers from mails we received.\nWe then brough back a feature I had implemented a long time ago but disabled for some reason I can\u0026rsquo;t recall. It required a bit of work but when declaring a PKI, a custom CA certificate may be provided allowing a listener to use a private CA certificate to authenticate users from an organization\u0026hellip; as simple as:\npki mx1.poolp.org ca \u0026ldquo;/etc/ssl/myca.pem\u0026rdquo; For a long time, certificates were verified but would simply alter the headers, but with this feature came the need to be able to prevent sessions from establishing if verification failed. We introduced the \u0026ldquo;verify\u0026rdquo; keyword:\na user MUST provide a certificate that can be verified with our system bundle if no \u0026ldquo;ca\u0026rdquo; is part of our pki entry, or with our \u0026ldquo;ca\u0026rdquo; if we declared one. listen on all tls-require verify pki mx1.poolp.org We pushed the feature a bit further, by allowing a relay rule to only work if it can establish a TLS session \u0026hellip; and eventually verify the remote certificate:\nONLY relay through TLS accept [\u0026hellip;] relay tls\nONLY relay through TLS \u0026hellip; if remote certificate is valid ! accept [\u0026hellip;] relay tls verify\nONLY relay through our route if remote certificate is valid ! accept [\u0026hellip;] relay via tls://mysmarthost verify To increase the security of the entire SMTP network, when relaying OpenSMTPD will always look for a pki entry matching its hostname and present a client certificate. This works transparently and takes into account hostnames table, source address and other tweaks.\nmta improvements\nOur MTA now always attempts TLS before falling back to plaintext. As stated above, it can also require remote hosts to present valid certificates and it will always attempt to present one itself.\nWe implemented support for AUTH LOGIN which was missing and made a lot of improvements to the logic used to optimize transfers. The mechanisms range from detecting hosts which we can\u0026rsquo;t establish a valid session with, to keeping a connection alive a few seconds to avoid a round-trip if an envelope gets scheduled for that same host just a few seconds later.\nTo summarize, we tried to make our MTA smarter so it takes less resources locally while not beeing seen as too agressive from the other side. There are many tricky details about this, if there\u0026rsquo;s demand we\u0026rsquo;ll post a more in-depth view of the internals ;-)\nsmtp improvements\nWe added some features that were requested by users, such as the possibily to create a listener that only listens to inet4 or inet6 rather than both.\nWe made it possible for listeners to require valid certificates, but also to select the banner name from a table dynamically based on the local address the client has connected to.\nA common feature was to allow the Received line to hide the From part which is not really a requirement for us to display and which allows hiding details from internal network. This is done by simply indicating on the listener that it should mask source:\nlisten on all mask-source Finally, we did a lot of internal rework since we are currently working on the filter API and while it is disabled, we wanted the code path to enter the filter loop to detect that there was no regressions when no filters are plugged.\nqueue improvements\nAn envelope cache was introduced to improve disk-IO pattern.\nOpenSMTPD is very strict about commiting changes to disk and keeping a coherent disk-based queue that it can start from if we were to crash or suffer a power down at the \u0026ldquo;wrong\u0026rdquo; moment.\nThe entire queue API has been designed around that constraint and this security comes with a price with regard to performances. There\u0026rsquo;s not so much we can do about disk-writes, however a common pattern is for OpenSMTPD to transmit an envelope to queue which writes it to disk and notifies the scheduler \u0026hellip; which requests the queue to load that envelope so that it can be delivered.\nBy introducing an envelope cache, the envelope is written to disk but retained in queue memory so that when the scheduler requests it back we avoid a disk read which we know will happen almost right away.\nThe queue cache will be improved with time but this first version already had a positive impact.\nWe also made many improvements that I won\u0026rsquo;t describe here as they are very technical and require a knowledge of the interactions between queue, mta, mda and scheduler. If people are interested, I\u0026rsquo;ll go into details in a more specific post.\nscheduler improvement\nLike for the queue, we made TONS of improvements and I won\u0026rsquo;t go into too many details either. But two of them are interesting enough that they need to be mentionned \u0026hellip; also they kept Eric and I working hours on investigating and fixing so just for that reason they deserve a mention.\nFirst improvement was the breaking of envelope batches into smaller batches. For optimization reasons, the scheduler tries to pack sets of envelopes to send to the mta/mda however when there are many many envelopes going to the same destination, the batches may grow very large.\nDue to our design, an event handler should never take a long time to operate otherwise it blocks the process from handling other events. In a situation, we had about 10k bounces from a major ISP and this caused the scheduler to send to the queue these 10k bounces in a row. The scheduler became hogged, it would not schedule anything until done with that task, and the queue was hogged as it would be processing this huge amount of envelopes without the scheduler letting it handle any other event. We broke the batch into small pieces allowing other events to interlace and this caused us to have much more efficient scheduling.\nThe other problem was an algorithmic issue\u0026hellip; We mainly work with trees but the scheduler had one sorted-list which it used for a single purpose. While this was no problem when working with a small set of envelopes (small being below 100K), as we grew trafic and were facing larger sets of envelopes, our O(n) inserts became so consuming that it caused the scheduler to eat all CPU for a few seconds \u0026hellip; every few minutes. In our very busy environement, this led to late that cumulated and cumulated until it was no longer possible to deliver in time. We ended up figuring the issue and replaced with a nice tree which allowed us to deal with a much larger volume and a very relaxed smtpd. OpenSMTPD has no problem working with queues that are several hundreds of thousands messages larges, but in theory we could go much higher we just don\u0026rsquo;t deal with enough volume to prove it ;-)\nsmtpcl improvements\nThe smtpctl utility which is the admin interface to the daemon has been rewritten to be much simpler and many new features were added.\nIt can now be used to show routing information, state of remote hosts, it can pause and resume individual envelopes and messages, and it transparently supports showing envelopes and messages even if the queue is encrypted and/or compressed.\nWe\u0026rsquo;re planning on many new improvements to it but it\u0026rsquo;s quite sexy as is ;-)\ndocumentation improvements\nWe introduced a table(5) man page to describe the format of tables, something that has bothered many users and someone contributed a sendmail(8) page to document our sendmail-like interface.\ntables improvements\nWe introduced table_passwd to provide user info and credentials in a passwd like table that is compatible with software such as Dovecot. Users can now store their recipients in a shared file and avoid duplicating information.\nWe made sure table_sqlite worked for all kinds of lookups, it is used in one of my side-project where an OpenSMTPD is used for a completely virtual setup where there\u0026rsquo;s not a single real user.\ntable_mysql has been improved and is pretty much production-ready, we use it in production ourselves and do not face issues with it.\nportable\nI reworked the entire autotools layout to reduce the delta between master and portable branches. Every component is being dealt as a module and while I\u0026rsquo;m not done, this is already much more clean and will allow us to reduce the inheritance of unrequired dependencies.\nI\u0026rsquo;ll keep working on it until it reaches perfection :-)\nA few words on performances and volumes\nI work at a company that handles a lot of mails, and by a lot I\u0026rsquo;m talking in matters of several millions per day both incoming and outgoing.\nWe initially relied on other opensource software deployed on many servers and as timed passed, progressively switched many servers to OpenSMTPD 5.3.3.\nAfter the many improvements done to the queue and scheduler, the number of servers could shrink and it is now technically possible to handle the same amount of mails (we\u0026rsquo;re still talking several millions a day) on a single instance of OpenSMTPD 5.4.1 that\u0026rsquo;s nowhere near busy.\nI know for a fact other companies have started using OpenSMTPD with high volumes too, so people who wonder if we will be efficient enough for their uses should simply ask themselves: do I need to send and receive several millions of mails each day. If that\u0026rsquo;s not the case, then you can\u0026rsquo;t possibly go wrong with us\u0026hellip; and if that\u0026rsquo;s the case, well, we do that and much more efficiently than with the older software, so you probably can rely on us.\nI will ask my manager if I can put out some graphs and numbers of real-life trafic as I don\u0026rsquo;t believe in oriented benchmarks ;-)\nThat\u0026rsquo;s all folks !\nThat\u0026rsquo;s all for this first post since a long time, I will make sure to post every weeks about updates since we\u0026rsquo;re very active on development.\nOh and the RSS feed will be back shortly, no need to tell me ;-)\n","date":"14 December 2013","permalink":"/posts/2013-12-14/news-from-the-front/","section":"Posts","summary":"EHLO readers,\nThis blog post is the first since a few months, I\u0026rsquo;ve been busy and struggling with some personal health and familial issues. I won\u0026rsquo;t share them here as its not really something anyone can help with, so\u0026hellip; let\u0026rsquo;s focus on OpenSMTPD !","title":"News-from-the-front"},{"content":"Hey,\nNo OpenSMTPD news since almost a month\u0026hellip; time to break the trend !\nAs far as I\u0026rsquo;m concerned, I\u0026rsquo;ve been busy with other work but still a git log shows that between Eric and I we have accumulated quite a few things during this month. Below is a short non-exhaustive summary of changes and news.\nAll of it is already pushed to our Github mirror and part of the snapshots that were published yesterday. I\u0026rsquo;d like to stress out that if you can run bleeding edge stuff, you REALLY want to test the latest snapshot as it has some very invasive improvements which we want to make sure are \u0026ldquo;bug-free\u0026rdquo; before the next big release in a few months.\nWe have also fixed some minor bugs that were reported to us since 5.3.1 and discovered/smashed some other bugs on our own. The bugfixes are part of the snapshots and were backported to the 5.3.2 stable release that should take place next week.\nMTA improvements\nOpenSMTPD has various \u0026ldquo;logical\u0026rdquo; optimizations to try and optimize its transfers. It will create several, but not too many, connections to a destination; it will detect that two domains share a same set of MX; it will spread load across MX of same priority; it will group messages on connections and group envelopes for messages; proceed to quadratic increase of delay upon temporary failures, and much more \u0026hellip;\nThis all performs pretty well with regular use-cases but we have discovered a use-case that was sub-optimal: dealing with a large list of single recipient messages going to the same destination. For example, the misc@opensmtpd.org list uses DKIM-proxy and signs all outgoing mail for each single recipient. Then, our code will optimize that out by grouping these recipients so that their mail is delivered through the same connection. So far, so good.\nProblem occurs if the other side uses grey-listing and rejects the recipients with 451 instead of dropping the connection with a 421 at first recipient: we keep reusing the connection to submit envelopes which we know are likely to fail. With a small list like ours, that\u0026rsquo;s not a problem as we will fail a few dozens envelopes and the quadratic delay will cause things to sort themselves after a few bounces but \u0026hellip;\n\u0026hellip; now imagine your list is that of a company with hundreds of thousands of recipients and the same happens. By the time you\u0026rsquo;re done submitting your list of recipients, it\u0026rsquo;s already time to try out the first of the list. The delay between your last recipient and the retry of first one causes you to fail to pass the grey-listing again, and you enter another loop of failures\u0026hellip; It eventually sorts itself out when the quadratic delay is so large that the delay between your last submitted recipient and the next retry of the first recipient respects the grey-listing imposed delay.\nMost people will never hit this case, but those who hit it will see their mails delivered after much more time than they really should (ie: your customers could receive their mail 12 hours later when they could be delivered after 4 hours.\nEric and I came up with two distinct mechanisms to improve the situation:\nFirst of all, instead of starting several concurrent connections, OpenSMTPD will first try to establish one and ensure that it is working before creating new ones progressively. If an issue prevents connections from being established and sessions from being started, the MTA layer will penalize the route and prevent it from being used for a small period. If new envelopes arrive for that destination, they are delayed until the route is considered usable again.\nThen, another mechanism kicks in. If too many recipients are rejected in a row, instead of trying the entire list OpenSMTPD will consider that they are all temporarily failed and mark the route temporarily unusable. This will cause the deliveries to be reattempted later, just like when a grey-listing mechanism is in place, without hammering the host with tons of recipients that are most likely going to fail. The ones that were rejected are penalized and their delay is slightly increased compared to those that OpenSMTPD failed. With this, in most situations people will not hit the case; in case we really face the problem, we will avoid hammering the server that kindly asked us to try again later several times; and if this was just a coincidence, legitimate envelopes are only deferred for 400 seconds.\nFinally, Eric spotted a bug that could cause a wrong error to be detected and imposing a long delay on some deliveries to hosts that advertise both IPv4 and IPv6 MX records.\nsmtpctl show routes\nWith all this routing logic added, we needed a way to be able to better understand what is happening at the MTA level.\nLog files have been extended and they provide a precise history, however it\u0026rsquo;s always nice to have a tool that provides real-time display of the state of something you\u0026rsquo;re investigating. So Eric added a \u0026ldquo;smtpctl show routes\u0026rdquo; subcommand that display the routing informations that are currently in used by MTA. It displays active routes, routes that have been penalized and when their penalty is going to expire and the route be usable again. This is a shortened output of the command right when I sent the \u0026ldquo;new snapshot published\u0026rdquo; mail to our mailing list, it displays the route in use, the state of the route, the number of connections currently active to that route, the penalty level and the timeout before a route is usable again after we detected a failure:\nsmtpctl show routes 88.190.237.114 \u0026lt;-\u0026gt; 144.76.32.53 (arati.lostca.se) \u0026mdash;- nconn=1 penalty=0 timeout=- 88.190.237.114 \u0026lt;-\u0026gt; 213.41.253.7 (gw.zefyris.com) \u0026mdash;- nconn=1 penalty=0 timeout=- 88.190.237.114 \u0026lt;-\u0026gt; 213.165.67.115 (mx01.gmx.net) \u0026ndash;Q- nconn=0 penalty=0 timeout=2s 88.190.237.114 \u0026lt;-\u0026gt; 213.165.67.97 (mx01.gmx.net) \u0026ndash;Q- nconn=0 penalty=0 timeout=1s 88.190.237.114 \u0026lt;-\u0026gt; 213.165.67.99 (mx00.gmx.net) \u0026ndash;Q- nconn=0 penalty=0 timeout=2s [\u0026hellip;] about a hundred entries [\u0026hellip;] 88.190.237.114 \u0026lt;-\u0026gt; 72.249.41.52 (lavabit.com) \u0026mdash;- nconn=1 penalty=0 timeout=- 88.190.237.114 \u0026lt;-\u0026gt; 87.98.163.156 (ns1.huynguyen.org) N\u0026mdash; nconn=1 penalty=0 timeout=- 88.190.237.114 \u0026lt;-\u0026gt; 95.130.12.24 (host1.trois-doubles.net) \u0026ndash;Q- nconn=0 penalty=0 timeout=1s\nTruly perfect for troubleshooting remote delivery issues !\nSSL issues\nEric ran into an issue where the daemon would crash when accepting a SSL connection from a client that didn\u0026rsquo;t do what we expected.\nI tried to reproduce but for some reason, it would not happen on OpenBSD, only on Linux. I spent a while troubleshooting and eventually found a subtle issue with our handling of I/O events.\nDuring a non-SSL session, OpenSMTPD enables and disables events as the session progresses since a SMTP session is a succession of command/responses where it is not expected to read a command when it has to send a response, and not expected to send a response when it is waiting for a client command. This works fine.\nDuring a SSL session, however, renegociation can take place and expect the server to write while the server is in read mode, and the other way around. To cope with this, the code wrongfully enabled both read/write events, causing the daemon to eventually end up in a weird state leading to either a hang until the client disconnects or a fatal() being hit.\nIn addition, we never set the client socket non-blocking, even though the logic around expected it, because it was assumed that we didn\u0026rsquo;t really need to do so due to the event-based approach. However, I found ways to cause SSL_accept() to be triggered BUT block with a misbehaving client that didn\u0026rsquo;t complete the SSL \u0026ldquo;handshake\u0026rdquo;. Other related issues could cause a SSL client to \u0026ldquo;block\u0026rdquo; other concurrent sessions until it timedout.\nAll these were fixed by making the client socket non-blocking and by being more strict with our event handling, none of the bugs could be reproduce on Linux after the fixes while no regressions were observed on OpenBSD.\nPortable improvements\nAn OpenBSD hacker complained that on Linux authentication didn\u0026rsquo;t work.\nIn fact, it would only work when built with PAM (which most Linux users have been doing so far) as the default authentication method in -portable is getpwnam(3) which doesn\u0026rsquo;t return the passwd field on Linux even when called as root. He suggested we implement a getspnam(3) method and provided the code which we merged.\nI spent a while tracking a crash on FreeBSD-STABLE that didn\u0026rsquo;t trigger on FreeBSD-RELEASE and which was caused by their import of NetBSD\u0026rsquo;s strnvis(3) which has \u0026hellip; swapped parameters compared to OpenBSD\u0026rsquo;s strnvis(3). This caused an invalid read that lead to a segv. We marked strnvis(3) as broken on NetBSD and FreeBSD to ensure that -portable builds with the compat glue until we deal with it differently.\nCharles has also worked on some fixes for OSX which we broke with all our new code :-)\nsmtpd -dvn improvements\nThe -d option was supposed to start it in foreground, while the -v option was supposed to start it verbose. In practice, this was not the case as -d started it in foreground AND verbose, while -v was essentially no-op. Oh and due to the log.[ch] code we inherited, it was not possible to start in foreground without verbose, and it was not possible for us to activate traces without being verbose too.\nSome people have been willing to start it in foreground without the verbose mode because they want to run it with a process manager and let it handle god knows what. Since it\u0026rsquo;s a common request and our behaviour was not consistent with the documentation, I decided to rework that and make it behave like expected.\nWhile at it, I fixed the requirement for verbose mode to enable traces so that now it is possible to:\nsmtpd -d -\u0026gt; start smtpd in foreground but not verbose smtpd -v -\u0026gt; start smtpd in background but verbose smtpd -dv -\u0026gt; foreground and verbose\nand no matter if -d and/or -v is specified, \u0026ldquo;smtpctl trace \u0026quot; will activate traces at runtime which will either appear at foreground or be logged to syslog as mail.debug.\nI also received two complaints the same day that \u0026lsquo;smtpd -n\u0026rsquo; checked the configuration file for validity but didn\u0026rsquo;t detect SSL certs errors as they are loaded after the configuration file parsing (on purpose). I slightly reworked the code so that we still load them after the configuration is parsed but to detect errors in the \u0026lsquo;smtpd -n\u0026rsquo; case.\nAssorted bugfixes and improvements\nTable lookups where not always folding key to lowercase which could lead to failed lookups for existing entries.\nThe \u0026ldquo;as\u0026rdquo; feature allows a rule to override the sender at SMTP-level (not in the message itself) and a missing check would lead it to override the sender when generating a bounce. Since the override can be partial (ie: @domain, or user without domain) and the sender for a bounce is an empty address, this could lead to invalid addresses (ie: MAIL FROM: , MAIL FROM: \u0026lt;@poolp.org\u0026gt;) causing a reject of the bounce by remote host.\nAlso, on the bounce side, the delivery_mbox backend didn\u0026rsquo;t assume that a sender could be empty in the bounce case and it would call the mailer.local third-party utility with an empty sender. That utility would write to the mbox with the usual delimiter \u0026ldquo;From\n\u0026quot; but with an empty address causing the following delimiter to be generated \u0026ldquo;From \u0026quot; (two spaces before date). Some MUA that do not check for \u0026ldquo;From \u0026quot; but for \u0026ldquo;From \u0026quot; as a delimiter would fail to detect the bounces in the mbox. As mentionned before, the table API and queue API now allow people to write standalone backends that are started by OpenSMTPD. They can have their own dependencies and be maintained out of the official tree, just as any other tool. To help with this, various APIs have been improved and changed slightly to ease this mechanism.\nA user has improved local LMTP delivery by adding support for unix sockets so that OpenSMTPD can talk LMTP to a unix socket setup by another daemon (Dovecot seems to be the trend ;-)\nAnother user reported a hang after a while and Eric traced it back to a missing check that could lead to an EOF not being detected. The mda/mta sessions would be in a broken state for a session and fail to deliver the message, causing it to stay in queue indefinitely. It was rare enough that we never hit it as it was \u0026hellip; specific to the size of a message \u0026hellip;\nA third user reported a \u0026ldquo;reject\u0026rdquo; rule not working as expected and we spotted a typo in the config file parser causing \u0026ldquo;reject\u0026rdquo; rules to fail when used with the \u0026ldquo;sender\u0026rdquo; parameter.\nA poolp user mentionned the \u0026lsquo;+\u0026rsquo; delimiter not working for one of his domains while it worked with his @poolp.org mail account. I realized that the \u0026lsquo;+\u0026rsquo; delimiter handling was not implemented for virtual domains, only for primary domains\u0026hellip; so I added support for it.\nFinally, an OpenBSD hacker reported that he had issues with his ~/.forward file causing recipient to be rejected when the file is empty. This was indeed the case and anoter user provided a diff to fix it.\nThat\u0026rsquo;s all folks !\nFor now ;-)\n","date":"14 December 2013","permalink":"/posts/2013-12-14/opensmtpd-improvements-summary/","section":"Posts","summary":"Hey,\nNo OpenSMTPD news since almost a month\u0026hellip; time to break the trend !\nAs far as I\u0026rsquo;m concerned, I\u0026rsquo;ve been busy with other work but still a git log shows that between Eric and I we have accumulated quite a few things during this month.","title":"OpenSMTPD improvements summary"},{"content":"Yop,\nLes services de poolp.org sont hébergés sur plusieurs serveurs.\nLe serveur principal à été basculé d\u0026rsquo;un petit hébergeur US \u0026ldquo;OpenBSD-friendly\u0026rdquo; vers Online.net depuis 3 ans maintenant. J\u0026rsquo;ai rusé à l\u0026rsquo;époque pour installer OpenBSD sur la Dedibox \u0026ldquo;DC\u0026rdquo; en utilisant une vieille version de Yaifo et en faisant des upgrades successives du système d\u0026rsquo;exploitation. Mis à part 3 petits downtimes dont 1 de ma responsabilité, je n\u0026rsquo;ai jamais eu à me plaindre de leurs services et de leur matériel.\nLe serveur secondaire, qui ne fournit que de l\u0026rsquo;espace disque pour les backups et un service de DNS et de SMTP secondaire en cas de panne du serveur primaire, est actuellement un serveur Kimsufi de chez OVH sur lequel j\u0026rsquo;ai installé OpenBSD via le vKVM. Le serveur est sous-utilisé, recevant un trafic anecdotique en dehors des backups journaliers en provenance du serveur primaire, mais je suis globalement content du service qui n\u0026rsquo;a pas subi de downtime depuis plus d\u0026rsquo;un an en ce qui me concerne.\nDes fois, pour les besoins d\u0026rsquo;un projet, je prends un serveur supplémentaire pour une durée indéterminée et je le laisse expirer à la fin du projet, ou bien je bascule le serveur secondaire dessus si à prix équivalent le matériel est plus confortable.\nCe week-end, je me suis rendu compte de la disponibilité des nouveaux serveurs Dedibox Classic+ Gen2 qui avaient l\u0026rsquo;air pas mal sexy. J\u0026rsquo;allais justement renouveller un serveur un peu moins confortable au même tarif, j\u0026rsquo;ai donc décidé de prendre un nouveau serveur chez Online et de laisser expirer celui que j\u0026rsquo;allais renouveller chez OVH.\nLe Classic+ Gen2 avait l\u0026rsquo;air assez sympa et la dispo d\u0026rsquo;un KVM/IP laissait présager qu\u0026rsquo;installer OpenBSD serait une partie de plaisir.\nCa, c\u0026rsquo;était avant d\u0026rsquo;essayer pour de vrai.\nLet\u0026rsquo;s go !\nMon serveur est commandé et je recoit peu après un mail m\u0026rsquo;indiquant sa disponibilité. Ni une, ni deux, je me loggue et ne vois pas de menu pour avoir accès au KVM, juste un bouton \u0026ldquo;Install\u0026rdquo;.\nComme il me faudra de toutes facons connaitre un peu le materiel avant d\u0026rsquo;installer OpenBSD par KVM, je prends le premier OS de la liste et lance une install pour pouvoir recuperer quelques infos du dmesg.\nL\u0026rsquo;install est rapide, il faut attendre un délai d\u0026rsquo;une heure avant de réussir à se connecter à la machine par SSH. Ca laisse le temps de chercher comment on accède au KVM depuis la console Online.\niDRAC\nL\u0026rsquo;Install de Debian finie, des menus supplémentaires apparaissent dans la console Online, dont le menu iDrac qui permets de lancer une console virtuelle.\nPas de bol, il faut un OS \u0026ldquo;java\u0026rdquo;-compliant, je viens de vendre mon laptop et je n\u0026rsquo;ai rien qui fasse l\u0026rsquo;affaire. Après moultes galères, je mets la main sur un Windows et je lance la console virtuelle, choisis l\u0026rsquo;ISO d\u0026rsquo;OpenBSD et tente un boot.\nPendant le boot, perte de connexion, impossible de relancer la console virtuelle, impossible de rebooter le serveur depuis la console Online ou bien simplement de faire un ssh sur la machine (normal, elle est en plein boot de l\u0026rsquo;installeur OpenBSD\u0026hellip;).\nSur le canal IRC d\u0026rsquo;#online, on m\u0026rsquo;indique que l\u0026rsquo;iDRAC est dans le caniveau, une intervention est planifiee.\nJe retente de differentes facons, mais le crash de l\u0026rsquo;iDRAC est systematique et me fait perdre la main completement sur le serveur sans aucune action possible à part quémander un redémarrage sur IRC et/ou ouvrir un ticket pour planifier une intervention.\nA la cinquième tentative, mes reves d\u0026rsquo;install par KVM/IP s\u0026rsquo;envolent, on me suggère de faire tourner une VM (ESXi est proposé comme OS), mais je suis pas trop fan donc\u0026hellip;\nQEMU à la rescousse\nJe cherche du coté de Yaifo vu qu\u0026rsquo;il m\u0026rsquo;a sauvé la vie la première fois, mais il n\u0026rsquo;est plus maintenu depuis un bail. J\u0026rsquo;envisage de le mettre à jour, mais pas trop envie de rester sur le problème 40 ans et l\u0026rsquo;idée de me retapper des heures d\u0026rsquo;attentes entre chaque intervention sur mes tentatives de tests échoué m\u0026rsquo;enchante pas des masses.\nSur IRC, Enjolras (#gcu) me fait savoir qu\u0026rsquo;il a installé DragonFlyBSD sur un Kimsufi en se servant d\u0026rsquo;un FreeBSD en mode rescue et d\u0026rsquo;unionfs pour faire l\u0026rsquo;install sur le disque physique depuis un QEMU lancé en curses.\nIdée séduisante, en rescue sur une Dedibox même pas besoin d\u0026rsquo;unionfs\u0026hellip;\nCi-dessous, voici donc la méthode pour installer le plus simplement possible un OpenBSD sur une Dedibox. Elle devrait marcher pour n\u0026rsquo;importe quel OS qui supporte le matériel avec un minimum de changements. Merci Enjolras !\nRécupérer les infos utiles\nIl faut tout d\u0026rsquo;abord booter sur le Linux installé et récupérer une copie du dmesg.\nLa raison est toute simple, QEMU va simuler le matériel, l\u0026rsquo;installation verra un disque et une interface réseau différentes de ceux du serveur physique. Lorsque l\u0026rsquo;installation sera terminée, si on reboot sur le serveur physique sans avoir fait quelques modifications, le système cherchera une interface et un disque inexistants avec le résultat attendu\u0026hellip; un ticket pour une intervention dans quelques heures ;-)\nLes infos qui nous intéressent sont éventuellement le DUID pour identifier le disque dans /etc/fstab et la description des interfaces réseau pour retrouver le nom du driver qui les prendra en charge et ainsi pouvoir créer les fichiers de conf réseau.\nSur le Classic+, j\u0026rsquo;ai décidé de ne pas me servir du DUID vu que c\u0026rsquo;est sympa mais pas obligatoire et pas d\u0026rsquo;un intérêt fou dans mon cas. En revanche, un petit grep eth a mis en évidence deux interfaces réseau \u0026ldquo;Broadcom BCM5716\u0026rdquo; et un petit coup de man bnx à confirmé que le driver bnx(4) les prends en charge sous OpenBSD. QEMU proposera une interface Realtek prise en charge par le driver re(4), donc ca n\u0026rsquo;ira pas.\nLe disque dur est monté sur /dev/sda, avec OpenBSD dans QEMU il sera visible en tant que /dev/wd0c, et avec OpenBSD sur le serveur physique il sera visible en tant que /dev/sd0c.\nDe plus, il va nous falloir les infos pour la configuration réseau:\nadresse IP broadcast netmask gateway On peut les récupérer depuis l\u0026rsquo;interface Online ou depuis ifconfig et route, en ce qui me concerne:\nadresse : 88.191.185.XXX broadcast : 88.191.185.255 netmask : 255.255.255.0 gateway : 88.191.185.1 Il faudra aussi l\u0026rsquo;adresse d\u0026rsquo;un serveur de nom pour éviter de passer 4 plombes sur le reverse lookup à la première connexion ssh, personnellement j\u0026rsquo;ai utilisé le 8.8.8.8 de Google pour mon premier boot puisque j\u0026rsquo;installe un serveur de nom local par la suite.\nLet\u0026rsquo;s Go Pour De Vrai!\nPremière étape: installer QEMU ;-) % sudo apt-get update % sudo apt-get install qemu\nSeconde étape: télécharger l\u0026rsquo;ISO d\u0026rsquo;install % wget ftp://ftp.fr.openbsd.org/pub/OpenBSD/5.3/amd64/install53.iso\nTroisième étape: lancer l\u0026rsquo;install. % sudo qemu-system-x86_64 -curses -boot -d -cdrom install53.iso -hda /dev/sda\nA ce stade, qemu boot sur l\u0026rsquo;installer et on fait face a une install tout a fait normale d\u0026rsquo;OpenBSD. Je le lance en mono-cpu, je prefere une install qui boot sur un kernel monoproc la première fois et faire une install du kernel MP après le premier boot, chacun est libre de faire comme il veut ;-)\nJe ne vais pas documenter l\u0026rsquo;installation, il y a moultes informations à ce sujet, par contre je vais juste faire une petite remarque utile pour vous éviter de perdre du temps inutilement:\nen mode \u0026ldquo;auto\u0026rdquo;, disklabel ne consommera pas tout le disque, il faut faire un découpage custom\nIl vaut mieux s\u0026rsquo;en rendre compte pendant l\u0026rsquo;install que de se galérer à faire les resize après \u0026hellip; croyez moi.\nInstallation finie ?\nAvant de quitter le mode rescue et de booter sur le serveur physique, quelques petites modifications.\nOn configure\u0026hellip;\n\u0026hellip; la gateway:\necho 88.191.185.1 \u0026gt; /etc/mygate\n\u0026hellip; l\u0026rsquo;interface réseau:\necho inet 88.191.185.XXX 255.255.255.0 88.191.185.255 \u0026gt; /etc/hostname.bnx0\n\u0026hellip; le resolver:\ncat \u0026lt; /etc/resolv.conf nameserver 8.8.8.8 search file bind EOF\n\u0026hellip; le fstab (attention à pas se rater):\nsed \u0026rsquo;s/wd0/sd0/g\u0026rsquo; \u0026lt; /etc/fstab \u0026gt; /tmp/fstab \u0026amp;\u0026amp; mv /tmp/fstab /etc/fstab\n\u0026hellip; et HOP on reboot\nreboot\nPremier reboot\u0026hellip;\nLe premier boot est long. Tres long. En fait pas si long que ca, mais l\u0026rsquo;anxiete de devoir refaire un ticket pour une intervention et attendre 4h pour refaire une tentative donne l\u0026rsquo;impression que des heures s\u0026rsquo;ecoulent avant la premiere reponse aux ping ;-)\nUne fois ce stade, on doit pouvoir se SSH sur la machine dans un OpenBSD avec un kernel monoproc. Si c\u0026rsquo;est bon, il ne reste plus qu\u0026rsquo;à installer le kernel MP:\ncp /bsd /bsd.sp ftp ftp://ftp.fr.openbsd.org/pub/OpenBSD/5.3/amd64/bsd.mp cp bsd.mp /bsd sync reboot\nAu reboot on est sur un OpenBSD 5.3 GENERIC#MP \\o/\nOpenBSD 5.3 (GENERIC.MP) #62: Tue Mar 12 18:21:20 MDT 2013 deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8557342720 (8160MB) avail mem = 8307052544 (7922MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe6da0 (57 entries) bios0: vendor Dell Inc. version \u0026ldquo;2.2.3\u0026rdquo; date 10/25/2012 bios0: Dell Inc. PowerEdge R210 II acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP SPMI DMAR ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR HEST ERST BERT EINJ acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) XHC_(S4) PXSX(S4) RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5) PXSX(S4) RP06(S5) PXSX(S4) RP07(S5) PXSX(S4) RP08(S5) PEG0(S5) PEGP(S5) PEG1(S5) PEG2(S5) PEG3(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3093.38 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV, PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL, DTES64,MWAIT,DSCPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1, SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF, PERF,ITSC,FSGSBASE,SMEP,ERMS cpu0: 256KB 64b/line 8-way L2 cache cpu0: apic clock running at 99MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3092.98 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH, DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL, VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT, DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS cpu1: 256KB 64b/line 8-way L2 cache cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3092.98 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH, DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL, VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT, DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS cpu2: 256KB 64b/line 8-way L2 cache cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3092.98 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH, DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL, VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT, DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS cpu3: 256KB 64b/line 8-way L2 cache ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 3 (P0P1) acpiprt2 at acpi0: bus 2 (RP01) acpiprt3 at acpi0: bus -1 (RP02) acpiprt4 at acpi0: bus -1 (RP03) acpiprt5 at acpi0: bus -1 (RP04) acpiprt6 at acpi0: bus -1 (RP05) acpiprt7 at acpi0: bus -1 (RP06) acpiprt8 at acpi0: bus -1 (RP07) acpiprt9 at acpi0: bus -1 (RP08) acpiprt10 at acpi0: bus 1 (PEG0) acpiprt11 at acpi0: bus -1 (PEG1) acpiprt12 at acpi0: bus -1 (PEG2) acpiprt13 at acpi0: bus -1 (PEG3) acpicpu0 at acpi0: C3, C2, C1, PSS acpicpu1 at acpi0: C3, C2, C1, PSS acpicpu2 at acpi0: C3, C2, C1, PSS acpicpu3 at acpi0: C3, C2, C1, PSS acpipwrres0 at acpi0: FN00 acpipwrres1 at acpi0: FN01 acpipwrres2 at acpi0: FN02 acpipwrres3 at acpi0: FN03 acpipwrres4 at acpi0: FN04 acpitz0 at acpi0: critical temperature is 106 degC ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 3093 MHz: speeds: 3101, 3100, 3000, 2900, 2800, 2700, 2600, 2500, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 vendor \u0026ldquo;Intel\u0026rdquo;, unknown product 0x0158 rev 0x09 ppb0 at pci0 dev 1 function 0 \u0026ldquo;Intel Xeon E3-1200v2 PCIE\u0026rdquo; rev 0x09: msi pci1 at ppb0 bus 1 mpii0 at pci1 dev 0 function 0 \u0026ldquo;Symbios Logic SAS2008\u0026rdquo; rev 0x03: msi mpii0: PERC H200A, firmware 7.15.8.0 IR, MPI 2.0 scsibus0 at mpii0: 42 targets sd0 at scsibus0 targ 1 lun 0: SCSI4 0/direct fixed naa.600508e0000000003cd970ac3a16d00c sd0: 953344MB, 512 bytes/sector, 1952448512 sectors ehci0 at pci0 dev 26 function 0 \u0026ldquo;Intel 6 Series USB\u0026rdquo; rev 0x04: apic 0 int 20 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 \u0026ldquo;Intel EHCI root hub\u0026rdquo; rev 2.00/1.00 addr 1 ppb1 at pci0 dev 28 function 0 \u0026ldquo;Intel 6 Series PCIE\u0026rdquo; rev 0xb4: msi pci2 at ppb1 bus 2 bnx0 at pci2 dev 0 function 0 \u0026ldquo;Broadcom BCM5716\u0026rdquo; rev 0x20: apic 0 int 16 bnx1 at pci2 dev 0 function 1 \u0026ldquo;Broadcom BCM5716\u0026rdquo; rev 0x20: apic 0 int 17 ehci1 at pci0 dev 29 function 0 \u0026ldquo;Intel 6 Series USB\u0026rdquo; rev 0x04: apic 0 int 23 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 \u0026ldquo;Intel EHCI root hub\u0026rdquo; rev 2.00/1.00 addr 1 ppb2 at pci0 dev 30 function 0 \u0026ldquo;Intel 82801BA Hub-to-PCI\u0026rdquo; rev 0xa4 pci3 at ppb2 bus 3 vga1 at pci3 dev 3 function 0 \u0026ldquo;Matrox MGA G200eW\u0026rdquo; rev 0x0a wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) pcib0 at pci0 dev 31 function 0 \u0026ldquo;Intel C202 LPC\u0026rdquo; rev 0x04 ahci0 at pci0 dev 31 function 2 \u0026ldquo;Intel 6 Series AHCI\u0026rdquo; rev 0x04: msi, AHCI 1.3 scsibus1 at ahci0: 32 targets ichiic0 at pci0 dev 31 function 3 \u0026ldquo;Intel 6 Series SMBus\u0026rdquo; rev 0x04: apic 0 int 19 iic0 at ichiic0 sdtemp0 at iic0 addr 0x19: mcp98243 spdmem0 at iic0 addr 0x51: 8GB DDR3 SDRAM ECC PC3-10600 with thermal sensor isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 kbc: cmd word write error pcppi0 at isa0 port 0x61 spkr0 at pcppi0 mtrr: Pentium Pro MTRR support uhub2 at uhub0 port 1 \u0026ldquo;Intel Rate Matching Hub\u0026rdquo; rev 2.00/0.00 addr 2 uhidev0 at uhub2 port 1 configuration 1 interface 0 \u0026ldquo;Avocent USB Composite Device-0\u0026rdquo; rev 1.10/0.00 addr 3 uhidev0: iclass 3/1 ukbd0 at uhidev0: 8 variable keys, 6 key codes wskbd0 at ukbd0 mux 1 wskbd0: connecting to wsdisplay0 uhidev1 at uhub2 port 1 configuration 1 interface 1 \u0026ldquo;Avocent USB Composite Device-0\u0026rdquo; rev 1.10/0.00 addr 3 uhidev1: iclass 3/1 ums0 at uhidev1: 3 buttons, Z dir wsmouse0 at ums0 mux 0 uhub3 at uhub1 port 1 \u0026ldquo;Intel Rate Matching Hub\u0026rdquo; rev 2.00/0.00 addr 2 uhub4 at uhub3 port 5 \u0026ldquo;Standard Microsystems product 0x2514\u0026rdquo; rev 2.00/0.00 addr 3 vscsi0 at root scsibus2 at vscsi0: 256 targets softraid0 at root scsibus3 at softraid0: 256 targets root on sd0a (f32d1747471a3037.a) swap on sd0b dump on sd0b bnx0: address d4:ae:52:c8:01:89 brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 bnx1: address d4:ae:52:c8:01:8a brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8\n","date":"8 May 2013","permalink":"/posts/2013-05-08/installer-openbsd-sur-une-dedibox-classic-gen2/","section":"Posts","summary":"Yop,\nLes services de poolp.org sont hébergés sur plusieurs serveurs.\nLe serveur principal à été basculé d\u0026rsquo;un petit hébergeur US \u0026ldquo;OpenBSD-friendly\u0026rdquo; vers Online.net depuis 3 ans maintenant.","title":"Installer OpenBSD sur une dedibox Classic+ Gen2"},{"content":"Yop,\nThis week has been very productive with several tickets closed and more about to be closed.\nEric has done an amazing work as you will soon realize ;-)\nI will push to our Github mirror and publish a snapshot this week-end very likely, until then don\u0026rsquo;t look for the features, only we have them !\nCompressed and Encrypted queue\nWhile working on bringing back encrypted queue support, I ran into a couple issues.\nFirst, I could use a compressed queue or an encrypted queue, but activating both would fail. This was strange because each one worked fine and their output was correct but somehow the combination led to errors. I tracked it down and figured that the reuse of the same buffer led to an overlap which didn\u0026rsquo;t happen before the switch to aes-256-gcm but would completely corrupt decryption now. Using two separate buffers for the compression and encryption was enough to solve this.\nThen, I realized that while everything was working flawlessly at runtime, OpenSMTPD would fail to load envelope at startup. After some tracking I isolated the bug to the queue_fsqueue.c backend which was violating API layers by trying to validate the envelope content while it should be done by the upper layer after decryption/decompression. The fix seemed tricky at first but was really about 10 lines to end up with.\nAs of my sandbox, I currently have an OpenSMTPD that runs with the following configuration file: listen on lo0\nqueue compression\ngenerate key with: openssl rand -hex 16 queue encryption key 95ca5f8053fc2baca7c390bea62bd9e2\naccept from any for domain poolp.org deliver to maildir accept for any relay All it lacks is better smtpctl integration so that we can \u0026ldquo;smtpctl show envelope\u0026rdquo; on an encrypted envelope :-)\ntable_proc API\nA feature I have wanted for many months is the ability to externalize table backends so that we can write very funky backends with dependencies that are never going to make it to the OpenBSD tree.\nFor example, people are going to want to store their aliases in mysql or pgsql and while the feature is easy to achieve thanks to the table API, we can\u0026rsquo;t provide it out of the box because of the dependencies AND people can\u0026rsquo;t contribute it easily because tables are part of the daemon and will require rebuilding after some patching.\nSo what I wanted was to have table backends work like our filters: they are standalone programs that are executed at startup by OpenSMTPD and communication takes place through the imsg(3) framework.\nEric took that idea, improved it and made a working implementation. So now, one can write a custom backend without having us do anything on our side, with whatever dependencies wanted, and OpenSMTPD can make use of it by providing the path in the configuration file:\ntable aliases \u0026ldquo;proc:/usr/libexec/smtpd/backend-table-sqlite -f /etc/mail/sqlite.conf /etc/mail/sqlite-aliases.db\u0026rdquo;\nlisten on lo0\naccept from any for domain poolp.org alias deliver to maildir accept for any relay\nWriting a custom backend implementation is very simple, fill in the blanks: /* * Copyright (c) 2013 Eric Faurot eric@openbsd.org * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED \u0026ldquo;AS IS\u0026rdquo; AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */\ninclude include include include \u0026ldquo;smtpd-defines.h\u0026rdquo; include \u0026ldquo;smtpd-api.h\u0026rdquo; static int table_stub_update(void); static int table_stub_check(int, const char *); static int table_stub_lookup(int, const char *, char *, size_t); static int table_stub_fetch(int, char *, size_t);\nint main(int argc, char **argv) { int ch;\nwhile ((ch = getopt(argc, argv, \u0026ldquo;f:\u0026rdquo;)) != -1) { switch (ch) { default: errx(1, \u0026ldquo;bad option\u0026rdquo;); /* NOTREACHED */ } } argc -= optind; argv += optind;\ntable_api_on_update(table_stub_update); table_api_on_check(table_stub_check); table_api_on_lookup(table_stub_lookup); table_api_on_fetch(table_stub_fetch); table_api_dispatch(); return (0); } static int table_stub_update(void) { return (-1); }\nstatic int table_stub_check(int service, const char *key) { return (-1); }\nstatic int table_stub_lookup(int service, const char *key, char *dst, size_t sz) { return (-1); }\nstatic int table_stub_fetch(int service, char *dst, size_t sz) { return (-1); }\nSexy hu ? :-)\nqueue_proc API\nSince he was already deep into the proc-ification of stuff, he also proc-ified the queue which was another feature I have wanted real bad for a very long time.\nSo it is now possible to write a queue backend with whatever dependency you want and have OpenSMTPD use it through a single configuration line.\nAnd just like for tables, writing a custom queue is pretty easy, just fill in the blanks: /* * Copyright (c) 2013 Eric Faurot eric@openbsd.org * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED \u0026ldquo;AS IS\u0026rdquo; AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */\ninclude include include include \u0026ldquo;smtpd-defines.h\u0026rdquo; include \u0026ldquo;smtpd-api.h\u0026rdquo; include \u0026ldquo;log.h\u0026rdquo; static int queue_stub_message_create(uint32_t *msgid) { return (0); }\nstatic int queue_stub_message_commit(uint32_t msgid) { return (0); }\nstatic int queue_stub_message_delete(uint32_t msgid) { return (0); }\nstatic int queue_stub_message_fd_r(uint32_t msgid) { return (-1); }\nstatic int queue_stub_message_fd_w(uint32_t msgid) { return (-1); }\nstatic int queue_stub_message_corrupt(uint32_t msgid) { return (0); }\nstatic int queue_stub_envelope_create(uint32_t msgid, const char *buf, size_t len, uint64_t *evpid) { return (0); }\nstatic int queue_stub_envelope_delete(uint64_t evpid) { return (0); }\nstatic int queue_stub_envelope_update(uint64_t evpid, const char *buf, size_t len) { return (0); }\nstatic int queue_stub_envelope_load(uint64_t evpid, char *buf, size_t len) { return (0); }\nstatic int queue_stub_envelope_walk(uint64_t *evpid) { return (0); }\nint main(int argc, char **argv) { int ch;\nwhile ((ch = getopt(argc, argv, \u0026ldquo;f:\u0026rdquo;)) != -1) { switch (ch) { default: log_warnx(\u0026ldquo;warn: backend-queue-stub: bad option\u0026rdquo;); exit(1); /* NOTREACHED */ } } argc -= optind; argv += optind;\nqueue_api_on_message_create(queue_stub_message_create); queue_api_on_message_commit(queue_stub_message_commit); queue_api_on_message_delete(queue_stub_message_delete); queue_api_on_message_fd_r(queue_stub_message_fd_r); queue_api_on_message_fd_w(queue_stub_message_fd_w); queue_api_on_message_corrupt(queue_stub_message_corrupt); queue_api_on_envelope_create(queue_stub_envelope_create); queue_api_on_envelope_delete(queue_stub_envelope_delete); queue_api_on_envelope_update(queue_stub_envelope_update); queue_api_on_envelope_load(queue_stub_envelope_load); queue_api_on_envelope_walk(queue_stub_envelope_walk); queue_api_dispatch(); return (0); } The awesome part is that this layer considers the queue as a key-value store and unless it WANTS to inspect content, it can consider it as a blob\u0026hellip; and since this layer is below the encryption and compression layer, a backend that doesn\u0026rsquo;t need to inspect content will magically support the compression and encryption feature out of the box ;-)\nIf someone is interested in writing a queue_torrent/queue_git or queue_whateverdecentralizedtechnology, I will help as I have ideas around decentralized encrypted queues\u0026hellip;\nscheduler_proc\nNo, just kidding, he didn\u0026rsquo;t do that much. scheduler_proc should be next ;-)\nassorted improvements\nThere\u0026rsquo;s been various improvements here and there on my part.\nI have improved logging so that relaying displays the source address as well as the relay session.\nI have a pending diff that is waiting for an OK and which ensures that OpenSMTPD copes with corrupted envelopes at runtime by moving them to the /corrupt queue. Currently, if an admin edits manually an envelope and fucks up, the daemon will abort.\nI also found a missing check within the imsg API that can cause OpenSMTPD and possibly other daemons to fatal() under some situations. The diff is ready and waiting for okays from other OpenBSD hackers.\nThat\u0026rsquo;s about all I think ;-)\nWant to make us happy ?\nI will take the opportunity to let you know, in case you missed it, that after a lot of hesitation and feeling bad about it, Eric has asked for a laptop or small donations to help him get one. So far he hasn\u0026rsquo;t received much which is a shame considering how shitty his laptop is and how much time he spends working on it to write tricky DNS and SMTP code ;-)\nConsider making him a small paypal donation ( eric@openbsd.org) if you like his work, he really deserves not to work on that trashpile.\nAs for me, feel free to Flattr or head your spare bitcoins my way (13tTchonwhjvLRo2NcUwyqfKrTvKMLBWRi) as it automagically converts into much needed beers ;-)\nThe OpenSMTPD book\nThe OpenSMTPD book initially planned for late May will be delayed slightly as my health issues (nothing worrisome, just annoying stuff) caused me to stop writing for weeks. I will resume by next week but I don\u0026rsquo;t want to rush just to make it to the deadline I imposed myself \u0026hellip; so it\u0026rsquo;ll probably be released sometime this Summer.\n# I hope, you\u0026rsquo;ll enjoy it because it\u0026rsquo;s proved to be much more efforts than I assumed ;)\nMore next week !\n","date":"26 April 2013","permalink":"/posts/2013-04-26/opensmtpd-table_proc-queue_proc-crypto-queue-and-other-stuff/","section":"Posts","summary":"Yop,\nThis week has been very productive with several tickets closed and more about to be closed.\nEric has done an amazing work as you will soon realize ;-)","title":"OpenSMTPD: table_proc, queue_proc, crypto queue and other stuff"},{"content":"OHAI,\nI have not updated this blog since the release of OpenSMTPD 5.3, over a month ago. In the meantime, some minor bugs have been fixed and we have released OpenSMTPD 5.3.1.\nThe main reason for the silence is that I was busy with completely unrelated day-time work, unrelated opensource code, the OpenSMTPD book in progress and some minor health issues that have caused endless side-effects keeping my mind away from this blog.\nAnyway, starting next week I should be actively blogging about developments in progress as we have resumed working on new features that I\u0026rsquo;m looking forward to see committed to master. For the time being, here\u0026rsquo;s a few features that have been committed or that are being worked on actively at the moment:\nhelo no longer requires source The \u0026ldquo;helo\u0026rdquo; features allows OpenSMTPD to advertise a particular helo hostname when relaying to another MX. It does so by looking in a table for a helo entry matching the IP address that it has used to connect to the MX. The feature required that both a source table AND a helo table be used for the helo feature to work, but this was an unrequired constraint and prevented some setups to be possible as reported by Todd Fries, so I simply removed the constraint. This has been committed to master and will be part of new snapshots.\n\u0026ldquo;error\u0026rdquo; alias type When writing code, we often have to test how be behave when facing a success, permanent failure or temporary failure. To achieve this, we usually create mail accounts that discard content, that have broken ~/.forward files (temporary failure) or we simply mail non-existent accounts (permanent failure). Todd had another use case where he wanted a virtual domain to have accounts that returned errors. I implemented a new alias type, \u0026ldquo;error\u0026rdquo;, which allows an alias to expand to an error code and message:\ntable aliases { foobar = \u0026ldquo;error:550 you\u0026rsquo;re not allowed to mail foobar\u0026rdquo;, barbaz = \u0026ldquo;error:450 woops, temporary failure\u0026rdquo; }\nThe error alias type can be used for regular aliases, virtual domains as well as ~/.forward files if a user temporarily wants to bounce mails to his account. It only supports 5xx and 4xx codes. It has been committed to master and will be part of new snapshots.\nOffline queue is broken Charles Longeau had spotted a strange issue where mail submitted while the daemon was offline would not be enqueued correctly when the daemon would start. I had never seen this but my OpenSMTPD instance is almost never down so the offline enqueuer is barely used. I tried to reproduce without success and eventually managed to hit the issue.\nIt turns out that this was only a permission problem and that mails submitted by the root user would manage to be enqueued, while mails submitted by other users would fail and remain in the offline queue. The issue has been fixed, committed and upcoming snapshots will have the fix.\nQueue spin Elbarto, a FreeBSD user, has reported that after changing permissions of files in the queue, he got the queue process to go crazy and spin. This looked strange because OpenSMTPD checks permissions at startup and should warn the admin if they are inappropriate. It turns out that the issue happened when the admin would change permissions of some directories in the spooler AFTER startup of the daemon.\nThis is an uncommon case and people should never hit in regular use, but the daemon should cope with these basic errors and not go crazy. It took some time to reproduce but I eventually found the culprit and came up with a fix that was committed. It will be part of upcoming snapshots.\nLog envelope source A user reported that we didn\u0026rsquo;t log the IP address that was used during a relaying when we log the status of the envelope. In setups that make use of multiple IP addresses, it makes it harder to track the path of an envelope while adding the source address is trivial. The log format for envelopes now contain an additional \u0026ldquo;source\u0026rdquo; field with the IP address that was used for the relaying. It will be part of new snapshots.\nIPv6 over IPv4 preference ftigeot@ from DragonFlyBSD reported strange behaviour where IPv6 was tried AFTER IPv4 which is inappropriate on that system. This behaviour is caused by OpenBSD\u0026rsquo;s default IPv4 before IPv6 search strategy, which turns out to be quite the opposite of what other systems assume. We decided to have portable OpenSMTPD use IPv6 before IPv4 strategy since that is what most people expect. Charles committed the fix to ASR with Eric\u0026rsquo;s and my approval.\nK_ADDRNAME not supported by db backend Todd Fries reported that he wasn\u0026rsquo;t able to use a db(3)-backed table as a helo table. It turns out that the db backend didn\u0026rsquo;t support the K_ADDRNAME service. The fix was trivial and committed shortly after. New snapshots will support it.\nEncrypted queue I have started working again on encrypted queue and the implementation is currently being discussed with other OpenBSD hackers. I\u0026rsquo;m expecting it to be committed within the next two weeks. It will allow people to have their mail queue transparently encrypted and protected against tampering. It has absolutely no use in a self-hosted environment, but will allow users to host their own mail servers while delegating the hosting of the queue to a company they trust for hosting but not for privacy (ie: I\u0026rsquo;d trust Google to store my queue safely, but I don\u0026rsquo;t trust them not to look at it).\nQueue encryption is compatible with queue compression and can be enabled with:\nqueue encryption key \u0026ldquo;foobarbaz\u0026rdquo;\nThe key will be expanded using 100.000 iterations of pkcs5_pbkdf2 and a random salt for each envelope and message. Each envelope and message will be encrypted with AES-256 in GCM mode, providing encryption and integrity, with a random IV for each. I\u0026rsquo;m looking forward to that feature :-)\nWhat\u0026rsquo;s coming next ? A big surprise. Eric is working on a feature that I\u0026rsquo;ve been wanting for MONTHS. It will be huge.\nStay tuned ;-)\n","date":"21 April 2013","permalink":"/posts/2013-04-21/news-from-the-front/","section":"Posts","summary":"OHAI,\nI have not updated this blog since the release of OpenSMTPD 5.3, over a month ago. In the meantime, some minor bugs have been fixed and we have released OpenSMTPD 5.","title":"News from the front"},{"content":"OHAI !\nI\u0026rsquo;ve stayed silent for the last month as we got ready for the big thing\u0026hellip;\nCharles, Eric and I are proud to announce the release of OpenSMTPD 5.3, our first official production-ready release !\nOpenSMTPD has been in use at poolp.org since 2008 and these last five years have seen a considerable amount of work. At various times we have been contemplating a release then delaying to add that one thing that seemed essential. Today, the code we release is still not perfect but we are very happy with it and it has proved to do the work pretty fine. It will continue to improve and mature from now on, and we hope you will enjoy it ;-)\nBelow is a copy of the release mail, see you soon for updates on new features ;-)\nOpenSMTPD 5.3 has just been released.\nOpenSMTPD is a FREE implementation of the SMTP protocol with some common extensions. It allows ordinary machines to exchange e-mails with systems speaking the SMTP protocol. It implements a fairly large part of RFC5321 and can already cover a large range of use-cases.\nOpenSMTPD has been under development for a long time now and many people are already using it in production, but this is our first stable release and if you were waiting for a GO! from us, here it is ;-)\nThe archives are now available from the main site at www.OpenSMTPD.org\nWe would like to thank the OpenSMTPD community for their help in testing the snapshots, reporting bugs, contributing code and packaging for other systems.\nFeatures: HUMAN READABLE CONFIGURATION IPv4 and IPv6 support STARTTLS and SMTPS support for both incoming and outgoing sessions AUTH support: bsd_auth(3) and crypt(3) SIZE support: limit the size of client-submitted messages Listener-specific banner hostname Listener-specific sessions tagging Support for global and per-domain expiry for messages Support for customizable delays for bounces Support for primary and virtual domains Support for alternate user database: db(3), file or smtpd.conf Support for aliases and ~/.forward mappings Delivery to mbox, maildir or third-party MDA Support for LMTP relaying Support for smarthost Support for sending certificate when connecting to remote host Support for backup MX Support for relay source address override Support for relay HELO override Support for SMTP-level sender override Support for connections reuse and optimization Support for queue backends: filesystem and ram Support for lookup backends: db(3), static Run-time statistics through \u0026ldquo;smtpctl show stats\u0026rdquo; Run-time tracing through \u0026ldquo;smtpctl trace \u0026quot; Run-time monitoring through \u0026ldquo;smtpctl monitor\u0026rdquo; Experimental:\nSQLite lookup backend LDAP lookup backend Portable:\nSupport for PAM authentication Known to build and work on FreeBSD, NetBSD, DragonFlyBSD and Linux Limitations:\nNo filters support yet (work in progress) No masquerading or address rewrite yet Checksums: SHA256 (opensmtpd-5.3.tar.gz) = 05efe80755e7fa01e79e6bba1a4e89244849406acb1152995d2c1da5e9e3a596\nSHA256 (opensmtpd-5.3p1.tar.gz) = 618092f1f0b5aba5f8d4c933536a76d3a5a8e45c28b599a6420321cd4478f3d9\nSupport: You are encouraged to register to our general purpose mailing-list: http://www.opensmtpd.org/list.html\nThe \u0026ldquo;Official\u0026rdquo; IRC channel for the project is at: #OpenSMTPD @ irc.freenode.net\nReporting Bugs: Please read http://www.opensmtpd.org/report.html Security bugs should be reported directly to security@opensmtpd.org Other bugs may be reported to bugs@opensmtpd.org\nOpenSMTPD is brought to you by Gilles Chehade, Eric Faurot and Charles Longeau.\n","date":"17 March 2013","permalink":"/posts/2013-03-17/opensmtpd-5.3-released/","section":"Posts","summary":"OHAI !\nI\u0026rsquo;ve stayed silent for the last month as we got ready for the big thing\u0026hellip;\nCharles, Eric and I are proud to announce the release of OpenSMTPD 5.","title":"OpenSMTPD 5.3 released"},{"content":"OHAI,\nWe had planned to slow down a bit to prepare for release: no more feature, no more invasive changes until we tag something out of which we can publish a release. I then discussed with Bret Lambert (blambert@) and he asked me if we used some kind of code scanner to detect defects in our code.\nWe do use the clang static analyzer every now and then, the last time being a couple week ago, and we spend a few minutes cleaning up the defects we believe to be real issues.\nBret then asked me if I had ever tried the scanner at Coverity and told me I should try to get our codebase scanned with it. I recalled seeing a summary of issues uncovered by their scanner on Apache a while ago and it was very impressive, so I asked to register OpenSMTPD and was happy when they confirmed I could use their scanner without having to sign any NDA, terms of services or random lawyer documents [rare and appreciable] :-)\nSo here\u0026rsquo;s a summary of this week\u0026rsquo;s work, all of it has been committed and merged in the OpenBSD tree and Github mirror. The latest snapshot is up to date.\nTRACE_LOOKUP\nA user reported on IRC that he had an issue with aliases. The issue turned out to be a user error but it prompted me into improving the aliases expansion logging.\nUntil now, to follow the aliases expansion logic, one had to run in debug mode to get to see the tons of log_debug() we have there. I introduced TRACE_LOOKUP and replaced the log_debug() with log_trace() which allow to turn the logging on or off at runtime with the command smtpctl trace lookup\nMTA logging improvement\nEric and I thought that we didn\u0026rsquo;t log enough information when relaying. The incoming path logs many details about sessions, but the outgoing path logged almost exclusively SSL-related details. So Eric spent some time adding proper logging to te MTA process, which should make it much easier for an admin to understand what happens by reading logs ;-)\nImproved memory usage\nEric introduced the mta_envelope to represent an envelope within the mta process. This allows keeping in memory ONLY the information needed by the mta and shrinks a lot the memory usage when the daemon is under a lot of stress.\nHe also introduced the mda_envelope structure to achieve the same for the mda process and raised limits to allow more concurrent local deliveries to take place.\nBug fixes\nFinally, most of the work this week has been spent on fixing bugs reported by the code scanner provided by Coverity.\nIt doesn\u0026rsquo;t run on OpenBSD but since it runs on Linux and our portable version is very close to the native version, we could run the test on Linux and fix on OpenBSD.\nWe didn\u0026rsquo;t find anything really scary, probably due to the fact we already cleaned and removed scary code detected by Clang, but we did find lots of little issues that we wouldn\u0026rsquo;t spot just from reading, even several times.\nAmongst the issues, we found a descriptor leak, several small memory leaks, possible double-free, possible double-close, possible double-fclose, use-after-free, various missing bzero, dead code and null-derefs. Many of these would lead to a crash on OpenBSD but the reason we didn\u0026rsquo;t experience them is that they happened in error code paths for the most part. We have fixed about 40 issues, there are 40 still, with several part of external code (mail.local, aldap, etc) and some we believe are false positives but require further reading with a clear mind. The scanner manages to find errors which makes you wonder if they really are, until you realize that it does a far better job at unrolling the code ;-)\nThis teaches us two lessons: find a way to better test error paths, run the scanner from Coverity regularly !\nAnyways, all related commits are available from Github in case you\u0026rsquo;re interested, as usual I\u0026rsquo;ll document funky bugs here so stay tuned.\n","date":"2 February 2013","permalink":"/posts/2013-02-02/opensmtpd-minor-fixes--preparing-release/","section":"Posts","summary":"OHAI,\nWe had planned to slow down a bit to prepare for release: no more feature, no more invasive changes until we tag something out of which we can publish a release.","title":"OpenSMTPD: minor fixes + preparing release"},{"content":"OHAI,\nLately we have stabilized and decided that we\u0026rsquo;re okay with the features we have for our first release, this time for good, so we focused on making sure that we didn\u0026rsquo;t introduce regressions and that we were rock solid.\nWe stressed incoming and outgoing path and found areas of improvements. This will be a long-standing task, but we have already improved memory usage considerably under high-load, we\u0026rsquo;re pretty happy with the result.\nAll of the following has been committed, we published snapshots and we sync-ed the OpenBSD tree with our own repository so that OpenBSD-current now has all of the features, improvements and bug fixes we came up with in the last few weeks (or months for some stuff we didn\u0026rsquo;t backport).\nAnyway, here goes a summary and I\u0026rsquo;m off to bed as I\u0026rsquo;m sick as shit.\nfix a descriptor leak\nWe were told of a crash which appeared to be coming from a descriptor leak. On Linux, lsof would display that message files were removed while the transfer process still had a descriptor opened. We tracked the issue and spotted that it would happen because of a missing condition in the transfer process where the message file would not be closed if recipients were rejected and the session was reused to send another message. Two liner fix.\nSSL code cleanup\nAn OpenBSD user reported a problem with ldapd\u0026rsquo;s ssl.c where the prime used for DH parameters was 512 bits which is short enough by today\u0026rsquo;s standards to be rejected by OpenLDAP\u0026rsquo;s client. OpenSMTPD\u0026rsquo;s ssl.c has been changed a long time ago to bump this prime to a 1024 bits prime, but ldapd\u0026rsquo;s ssl.c was actually a copy of OpenSMTPD\u0026rsquo;s ssl.c from two years ago.\nThe desynchronization of ssl.c accross OpenBSD daemons has annoyed me for a long time but it occured to me that maintaining this gap would be causing further divergences in the future as reyk@ moves OpenIKED and relayd forward. After a quick chat with him I started creating a daemon-agnostic version of ssl.c.\nThere is still work to do in that area, but OpenSMTPD now comes with a ssl_privsep.c that is equivalent to that of relayd; a ssl.c file that no longer knows of any smtpd specific structures and which can be shared with other daemons; a ssl_smtpd.c that contains the smtpd-specific bits.\nNote that ssl.c doesn\u0026rsquo;t contain new code, this was only a rework of the interfaces to allow it to be shareable with different daemons.\nRuntime tracing and profiling\nI have added a feature that I\u0026rsquo;ve been wanting for a long time: activating traces at runtime without a daemon restart.\nSay a user suddenly observes that connections to a remote host fails, he can now simply type: smtpctl trace transfer\nTo obtain a real-time view of the sessions as they take place with remote hosts.\nThere are many trace subsystems, for incoming connections, outgoing connections, msg exchanges accross processes, etc, etc, \u0026hellip; read the man page ;-)\nWhile at it, if you need to verify bottlenecks OpenSMTPD also supports real-time profiling of imsg and queue using: smtpctl profile imsg and smtpctl profile queue\nBoth tracing and profiling can be turned on and off at runtime.\nImprove memory use\nEric cleaned up some code to avoid passing envelopes when we could pass evpid instead. This prevents OpenSMTPD from accumulating data in the inter-process buffers (an evpid is 64bits, an envelope structure is several kbytes).\nFor the places where the envelope really needs to be passed, he suggested that we send a compressed version. I proposed that we use the ASCII envelope conversion as we know it works given that it\u0026rsquo;s already used for disk-based envelopes. The ascii envelopes allow compressing the envelope quite a lot as instead of passing a large datastructure with partially used large fields, we pass the ascii representation that can be as small as a hundred bytes.\nWe should now be able to cope better in very heavily loaded situations ;-)\n","date":"25 January 2013","permalink":"/posts/2013-01-25/opensmtpd-gentlemen-fasten-your-seatbelt/","section":"Posts","summary":"OHAI,\nLately we have stabilized and decided that we\u0026rsquo;re okay with the features we have for our first release, this time for good, so we focused on making sure that we didn\u0026rsquo;t introduce regressions and that we were rock solid.","title":"OpenSMTPD: gentlemen, fasten your seatbelt"},{"content":"\u0026lt;p\u0026gt;Hey,\u0026lt;/p\u0026gt; I wanted to publish something last Friday but I unfortunately got my hands on a copy of Super Nintendo's Zelda which led to an instant loss of motivation.\nHere's a quick summary of what happened recently in OpenSMTPD-land, it's not an exhaustive description and to be honest I'm keeping a lot of information private at this point and probably until release ;-)\nA snapshot should be published tomorrow.\nStressing the daemon\nFirst of all, we have done a lot of stress testing lately.\nWe have tested the incoming path, accepting hundreds of thousands of mails from hundreds of sessions, arriving in chaotic order and with random data. We're quite confident that our incoming path is rock solid now, we will be performing our final test very soon with one billion messages.\nWe have also tested our outgoing path, first in a confined environment which revealed no bugs, and then in a live environment which revealed minor bugs that where fixed in the process. There is still one \"memory usage\"-related issue but it applies to very special and stressed setups, not something people would usually experience, and we happen to have a diff for it which requires some additional work before being in a state proper for commit.\nDuring these stress tests, we have gathered a lot of information to prove some of our theories right and wrong. We now know where we stand with regard to other MTA and we have built tools which allow us to have a very precise understanding of our areas of improvements. Clearly, we have no reason to be ashamed, far from it.\noptimization\nOur queue code has a design that allows for very efficient queues to be written. We know that because we have already written several backends, we know the time spent in the API, the time spent in the backends and for disk-based backend the exact cost of the disk IO.\nYou'd assume our queue would be very fast, but the default backend we ship is sub-optimal by design because the admin that is in me wants the queue to provide some features that are incompatible with performances.\nOne of these features is to provide per-envelope files as well as a locality between envelopes and messages so that SMTP transactions can be backed up and restored easily on another machine. Stuff like that.\nThis user-friendliness means that we can't rely on tricks to avoid hitting the disk too often, we have as many open()-fsync()-close() as there are envelopes; we have as many mkdir()/rename() as there are messages; and you can add many more file-system related calls used to deal with the atomicity of our queue commits. A lot of this is unnecessary for OpenSMTPD and could be handled in a much more efficient way... but is really only done so that a human can inspect and manipulate the queue more easily.\nA queue that allowed for envelopes to be written sequentially in a single binary file that would require only one fsync() could increase drastically our performances, and we will surely do that at some point, but our default queue will always be the user-friendly one even at the cost of a slower incoming path.\nThat being said, we still want our queue to operate fast and limit the impact of our design. Ideally, our queue shouldn't be more than 10% slower than the other software with their optimized queues. So...\nI spent some time tracing our queue code and spotted that the system calls pattern was not really matching what I'd expect. Our queue logic is very simple and has a very \"linear\" pattern, several system calls should have a matching number of calls. I tracked and fixed until a kdump output displayed the optimal number of calls for each system calls according to the numbers I had on paper. I added a couple functions to help profile every single queue operation, Eric came up with a better interface for these functions and we now have an invaluable tool for queue backend development :-)\nEric also spent time improving our memory usage and removing some pressure from inter-process IO by coming up with an API that allows to encode/decode the data more efficiently. Until now we passed structures, which could contain huge buffers for which we only consumed a few bytes; now the new API not only passes only the required data but also provides type checking which allows us to make sure we don't pass the wrong data by mistake. As a bonus, the API can use the types to know the average size of the data and only reallocate in cases where we exceed that size.\nWhen we were done with this, we were very good with the outgoing path, and we were pretty much equivalent to the other MTA with regard to the incoming path. There is still a lot of room for improvement, but given the constraints we imposed ourselves I'm really glad that we're not twice as slow as the slowest MTA out there :-)\nMore resistant fs-queue\nOur default queue had a design that was very strict with \"strange\" situations.\nAny time an unexpected situation happened, the daemon would fatal. Since unexpected situations are not supposed to happen, this shouldn't be a problem right ?\nNo. Not right. On a regular setup this never happens, but sometimes a human does something as innocent as a chmod on a file or directory... and OpenSMTPD figures something is not normal and commits suicide.\nTo be honest not only did I not receive a report of a queue fatal in months, but I also don't recall ever hitting any of these on poolp.org... until the stress.\nI hit a couple fatal() which turned out to be related to an error in our use of the fts(3) API which for some reason didn't trigger until the stress. I fixed the issue then decided to go track every single fatal() in the queue code and try to convert it into a temporary failure condition so that even if a failure happened, the daemon would deal with it gracefully.\nIt turned out to be much simpler than I assumed and our fs-queue is now capable of coping with an admin messing with the queue. Of course, an admin should never tweak with the queue, but being able to not fatal() on a chmod or mv felt like essential ;-)\nPer-listener hostname\nOur smtpd.conf file had a \"hostname\" directive which allowed setting the hostname to display on the greeting banner.\nThe directive was removed and it is now possible to specify the hostname for each listener:\nlisten on lo0 listen on 192.168.1.1 hostname mx1.int.poolp.org listen on 192.168.2.1 hostname mx2.int.poolp.org Not specifying one will use the machine's hostname.\nPer-source HELO\nWhen relaying mail, smtpd.conf allows for overriding the source address using a table containing an address or a list of addresses:\ntable myaddrs { 192.168.1.2, 192.168.1.3 }\naccept for all relay source The above causes the relaying to bind one of these addresses a the source address. However, during a SMTP session, our MTA has to advertise itself at the HELO/EHLO stage and tell its hostname. The hostname is sometimes checked and if it doesn't match the source address used, the MTA is rejected.\nSo we needed a way to have our MTA provide the remote host with a HELO/EHLO parameter that corresponds with the source address used. We had ideas that involved performing a DNS lookup from MTA but it would not work due to NAT.\nI suggested that we use a new lookup service K_ADDRNAME which allows for a mapping of an IP address to a name:\ntable myaddrs { 192.168.1.2, 192.168.1.3 } table myhelo { 192.168.1.2 =\u003e mx1.poolp.org, 192.168.1.3 =\u003e mx2.poolp.org }\naccept for all relay source helo With the following, MTA will use a source from the table myaddrs and, at HELO/EHLO time, will use the name from the table myhelo that matched the address it used to connect.\nSender filtering\nAnother feature that people have been requesting very frequently is the ability to use a sender email address as a matching condition in the ruleset.\nUntil recently, the matching of a rule was done by looking at the client address and the destination domain. It was not possible to express something like \"accept all mail coming with sender gilles@poolp.org\" or \"reject all mail coming with sender @redhat.com\".\nI have introduced the \"sender\" filtering which can apply to a full email address or to a domain, both in accept and reject rules. It works as follow:\naccept from any source, if sender has domain @openbsd.org [...] accept from any sender \"@openbsd.org\" for any relay\naccept from localhost, if sender has domain @poolp.org [...] accept sender \"@poolp.org\" for any relay\naccept from any source, only if sender is gilles@poolp.org [...] accept sender gilles@poolp.org for any relay It can apply to relay or deliver rules, and allows the use of tables to apply different relay rules to different domains or users coming from different networks.\ntable hackers { \"@opensmtpd.org\", \"@poolp.org\" } table slackers { richard@foot-cheese.org, lennart@thepig.org }\naccept from 192.168.1.0/24 sender relay accept from 192.168.2.0/24 sender relay via smtp://example.org Various little fixes\nYou have no idea.\nWe have fixed various little bugs that triggered in very specific cases which you just can't hit out of a live test.\nWe also fixed/improves/rework minor things, like making the \"relay backup\" parameter optional by picking the machine hostname, changing the API for queue remove to take an evpid instead of a full envelope, removing userinfo from a structure that wasn't using it, switching the queue code to use the first two bytes of a msgid instead of the last two bytes to create the bucket, fixing a segfault with a specific configuration file, allowing authentication to fail temporarily, etc, etc, etc ...\nOh and on a completely useless and unrelated note:\nI installed OpenSMTPD on a raspberry, so here's a picture of my mailserver at home:\n","date":"18 January 2013","permalink":"/posts/2013-01-18/opensmtpd-stress-and-features/","section":"Posts","summary":"\u0026lt;p\u0026gt;Hey,\u0026lt;/p\u0026gt; I wanted to publish something last Friday but I unfortunately got my hands on a copy of Super Nintendo's Zelda which led to an instant loss of motivation.","title":"OpenSMTPD: stress and features"},{"content":"YO!\nI know I should have posted a \u0026ldquo;Happy New Year\u0026rdquo; note earlier but you know how it is: new year eve\u0026rsquo;s aftermath, days to recover, work resumes, mood slacks, bleh.\nAnyway, today\u0026rsquo;s note is simply about that: wishing you and me an awesome year 2013 and hoping you will not turn it into a fucking mess. After all, you\u0026rsquo;re in charge so we know who to blame ;-)\nAs far as I\u0026rsquo;m concerned, year 2012 wasn\u0026rsquo;t \u0026ldquo;great\u0026rdquo;, it was not the absolute suck either, it was even quite decent. I resumed music more seriously; started sponsoring two little kids in Lebanon and Palestine; moved to a nicer place in Paris; played some metal with my friends of Spinach Blast; visited my family and friend in Lebanon; hacked and shared burgers with fellow OpenBSD hackers in Hungary; ate a fair share of ice-creams in Nice, southern France; visited buddies in Nantes, western France; sadly left great co-workers and friends at a company \u0026hellip; to join a fun crew at a new company and work on my own project for a few months.\nOk, despite some down times, 2012 was a very decent year actually\u0026hellip; I probably missed a lot of things too, it was a busy year ;-)\n2013 will be an even better year I hope.\nI\u0026rsquo;m supposed to spend the next few months working full-time on OpenSMTPD to make it fit the needs of my employer; by then, I will be done with paying back my car which will effectively set me free from bank slavery; this will of course happen right before I move back to Nantes from where I\u0026rsquo;ll work remotely for my employer on new challenging tasks that are less OpenSMTPD-centric; and being away from Paris will allow me to save enough money by the end of the year to try and convince my banker to enslave me for a couple decades so I can buy an appartment with a view on the Erdre river. Pffffiou\u0026hellip;\nThat\u0026rsquo;s the plan at least and I guess we\u0026rsquo;ll see next year how it went ;-)\nMeanwhile, Poolp.org enters its 6th year of providing free services based on open-source software to its little community of users. No changes planned on that front besides maybe more services and a more active blog that\u0026rsquo;s a bit more diverse than my OpenSMTPD hacking summaries.\nFor the time being, you g{al,uy}s have fun and I\u0026rsquo;ll just zZzZz\n","date":"8 January 2013","permalink":"/posts/2013-01-08/welcome-to-you-2013/","section":"Posts","summary":"YO!\nI know I should have posted a \u0026ldquo;Happy New Year\u0026rdquo; note earlier but you know how it is: new year eve\u0026rsquo;s aftermath, days to recover, work resumes, mood slacks, bleh.","title":"Welcome to you, 2013"},{"content":"OHAI,\nI was unable to write a story last week as my desk broke into pieces for no apparent reason but my foot hitting it accidentally.\nI still don\u0026rsquo;t have a desk so I will make this short by not telling about the queue profiling and SQL log conversion I wrote, and not even writing about the filter work done by Eric, he\u0026rsquo;ll write about it if he wants to or I\u0026rsquo;ll do that next time.\nYou guys enjoy New Year\u0026rsquo;s Eve, don\u0026rsquo;t get too drunk and remember to send tons of mails to those not around using an awesome piece of software :-p\nSSL verification and separation\nOpenSMTPD has had code to deal with establishing secure channels for a very long time now, however what it didn\u0026rsquo;t do was to perform certificate chain verification in both server and client modes. In server mode, it would never request a client certificate and in client mode it would never check that the certificate handed by the server was valid.\nSo I started adding support and hit the first issue: access to the CA bundle from within the chroot. After a discussion with reyk@ over what was the best way to deal with it, he convinced me that we could really improve our design by moving certs and keys from network-facing processes to a separate process and performing on-demand requests.\nDealing with OpenSSL was as nice as usual, it almost seemed like sharing a tasty dinner with Richard Stallman, but I eventually saw the light and got it to work as expected. The good part is that the client and server modes have symmetric operations which means that the code is identical for the most part.\nWe now have the sensitive stuff isolated in the lookup process. The smtp and mta processes use the imsg framework to request them and to pass over chains for verifications. It all fits in a very few lines of code which is cool because the less OpenSSL code I have to deal with, the better.\nFor now, we don\u0026rsquo;t do a full verification of the X509 attributes so OpenSMTPD lies about the verification and pretends it didn\u0026rsquo;t do it in the Received lines, however the admin can see the verification take place in the logs. I\u0026rsquo;ll fix the Received line to tell the truth when I\u0026rsquo;m confident the verification is 100% accurate.\nNow, before I switch subject, a couple related ideas for the future:\nIf we were to provide a K_CERT table service, we could fetch the certificates and chains from custom backends (ldap, sql, you name it, \u0026hellip;), this would take approximately one hour of work. The only reason I\u0026rsquo;m not doing it is that I don\u0026rsquo;t have a need for that at the moment ;-)\nAlso, the certificate verification reply to mta and smtp is a simple status that is either success or failure. This means that implementing a certificate-based authentication is now trivial, probably an hour of work too. Guess why I didn\u0026rsquo;t implement it yet ?\nFix relaying logic\nWhile working on the SSL code, I spotted strange behaviour when relaying between my primary and secondary MX.\nAfter some testing, I realized that we lost the \u0026ldquo;backup\u0026rdquo; flag somewhere. After a quick chat with eric@, I convinced him we should introduce the backup:// schema and get rid of the flag in the envelope. This way, we could make it obvious that mx2.poolp.org is a backup MX by having backup://mx2.poolp.org as a relay URL.\nThen, I spotted some strange issues where I didn\u0026rsquo;t request TLS and it would attempt it, or where I requested TLS but it would fallback to plaintext. It became obvious there was something fishy with the semantic of our relay URL schemas and that they needed to be more clearly defined.\nAfter going through every possible schema, we defined them as follow:\nsmtp+tls:// -\u0026gt; attempt TLS, then fallback to plaintext, this is the default smtp:// -\u0026gt; plain text ONLY, no encryption tls:// -\u0026gt; STARTTLS only, encrypted channel guaranteed smtps:// -\u0026gt; SMTPS only, encrypted channel guaranteed ssl:// -\u0026gt; STARTTLS, fallback to SMTPS, encrypted channel guaranteed\nWith this, unless smtp+tls:// is specified, we never fallback to plaintext if an encrypted channel is requested and we never break the user assumption that a relaying will be secure by sending the data over a plaintext channel.\nI guess that\u0026rsquo;s all for this time, I really need to zZzZ\n","date":"28 December 2012","permalink":"/posts/2012-12-28/opensmtpd-ssl-relay-stuff/","section":"Posts","summary":"OHAI,\nI was unable to write a story last week as my desk broke into pieces for no apparent reason but my foot hitting it accidentally.","title":"OpenSMTPD: SSL \u0026 relay stuff"},{"content":"\u0026lt;p\u0026gt;OHAI,\u0026lt;/p\u0026gt; This week has been crazy, my brain melted a little as I worked on filters, it melted a little as I worked on LDAP and it melted a little more as I tried to trick Eric into working on filters with me (yes, that actually worked :-).\nAnyway, as usual, this is just going to be a summary of what we did as tons of little stuff have been committed here and there. You can keep track of the changes by checking the commit log of our Github mirror, or by joining our little gang on our IRC channel: #OpenSMTPD @ freenode.\nSnapshots containing all the following features should be published tomorrow.\nLet the fun begin !\nLDAP backend\nI had started working on LDAP support for OpenSMTPD a long time ago but for some reason the support was never finished and ended up rotting in my sandbox.\nA few months ago, I brought the bits back to a git branch so that I would keep running into it every few days as a reminder that I should not slack. But since I'm not too much of a LDAP fan, or a LDAP user for what it's worth, I made the branch public in hope someone would pick it up and move it forward.\nA poolp user had started bringing the bits up to date and getting a working support in shape for aliases lookup. Resuming from there I simplified the code further and added support for almost all kinds of lookups making OpenSMTPD capable of using LDAP as a backend for the most common use-cases.\nHere's a configuration file to authenticate local users, lookup a domain and perform aliases lookups against LDAP:\n/etc/mail/smtpd.conf table myldaptable ldap:/etc/mail/ldapd.conf\nlisten on egress tls auth accept for domain alias deliver to maildir accept for any relay and here's the table configuration:\n/etc/mail/ldapd.conf url ldap://127.0.0.1 username cn=admin,dc=opensmtpd,dc=org password thisbemypasswd basedn dc=opensmtpd,dc=org\naliases lookup alias_filter (\u0026(objectClass=courierMailAlias)(uid=%s)) alias_attributes maildrop\ncredentials lookup credentials_filter (\u0026(objectClass=posixAccount)(uid=%s)) credentials_attributes uid,userPassword\ndomains lookup domain_filter (\u0026(objectClass=rFC822localPart)(dc=%s)) domain_attributes dc The support is functional but it needs to be improved further as it currently has two drawbacks: the backend does not reconnect to the LDAP server should it lose the connection, and it uses the aldap synchronous API which means that queries that take time to complete will be heavy on the lookup process.\nAlso, I only tested with OpenBSD's ldapd(8) as it was dead simple and I didn't want to add more pain than required on my shoulders. Turns out, it did make my experiment far more enjoyable that I would have assumed ;-)\nOh, and I ordered a LDAP book to get more familiar with the service as I suspect I'll be getting questions regarding LDAP every now and then given how many times it's been requested in the past. I might as well know what I'm talking about :-)\nSource address selection\nEric has plugged the K_SOURCE lookup service to relay rules, this allows OpenSMTPD to perform a lookup of the source address it should use when the transfer process establishes an outgoing connection to a relay.\nUntil now OpenSMTPD could not force the IP address it used for outgoing trafic without relying on a hardcoded hack that was committed to the poolp branch. It was done this way on purpose and we delayed this feature until the other parts were rewritten appropriately for the puzzle to fit right.\nIt is now possible to force an address using the source keyword:\ntable myaddrs { 88.190.237.114, 91.121.164.52 }\naccept for any relay source If multiple addresses are provided, they will be cycled through, and the mta code will detect which ones are no longer usable.\nIntermediate bounces\nA feature we had a long time ago and which disappeared during a cleanup was the support of intermediate bounces.\nWhen OpenSMTPD fails to deliver a mail it has to notify the sender that the message was never delivered. It sometimes happens immediately, but sometimes the failure may be temporary and the daemon keeps the message and tries to deliver it every now and then (ok, the logic is slightly more complex, but you get the idea). In such cases, the bounce will not be sent before OpenSMTPD gives up on trying after 4 days by default.\nObviously, getting a mail 4 days later to tell you that no one read yours when you assumed it was already in the recipients mailbox for a while is quite irritating. The intermediate bounce will instead notify the sender that an error occured after a few temporarily failed deliveries, and let him know that the daemon will keep trying to deliver for a while.\nAfter discussions, Eric reimplemented intermediate bounces in OpenSMTPD but did it in a slightly different way than with other daemons. By default, an intermediate bounce will be sent after a mail has been sitting in the queue for over 4 hours without being delivered ... but in addition, a set of delays may be provided in smtpd.conf to send multiple intermediate bounces. For example, if I wanted intermediate bounces to be sent after 4 hours, after a day and after two days, I could simple use:\nbounce-warn 4h, 1d, 2d The keyword may change, but the idea and code is here and working.\nTags \u0026 DKIM example\nI've implemented tagging of sessions a very long time ago, I think it was actually already there when OpenSMTPD was not yet OpenSMTPD but still a poolp project :-)\nThe feature was hidden and undocumented, it had uses but so limited that I did not want users to start using it in random situations that I would have to cope with later. Basically, a listener may tag all sessions initiated through it; then rules may apply to specific tags allowing some rules to apply only to some sessions.\nEric realized that this was perfect to deal with one of our use-cases: DKIM signatures.\nWe want DKIM signatures but we don't necessarily want to write a filter for that as there are already tools that do the work. So we need to accept a message, pass it to a DKIM signing tool, which will in turn pass it back to us, so that we can send it where it needs to be sent.\nA tool to do this is DKIMproxy. I won't enter the details behind DKIMproxy, I'll actually post a description of how to setup OpenSMTPD with it very soon on this blog. But the idea is that using tags we can determine which sessions we want forwarded to DKIMproxy, and which sessions are coming back from it and need to be relayed to the final destination:\nlisten on all interfaces that are attached to the default route listen on egress\nlisten on loopback interface on port 10028 and tag DKIM listen on lo0 port 10029 tag DKIM\nonly accept to relay the sessions that are tagged DKIM accept tagged DKIM for any relay\nthis is reached by the sessions that are NOT tagged and will cause OpenSMTPD to relay to the DKIMproxy accept for any relay via smtp://127.0.0.1:10028 Now if I were to send mail to my gmail account, I would connect to the daemon, my session would not be tagged so it would match the last rule causing the message to be send to DKIMproxy. DKIMproxy would then relay back the mail to my loopback interface on port 10029 which would have the new session tagged DKIM causing the first rule to be matched. 4 lines. ridiculous.\nOpenSMTPD goodies\nHere it is, OpenSMTPD goodies, not for sale, limited to friends and a few spare I will give away or sell at low price depending on how many are left.\nIf you were kind enough to contribute and finish the FAQ section of the OpenSMTPD website, I might just send you a mug and a tshirt signed by Charles, Eric and I ;-)\nTime to go zZz, stay tuned for more news !\n","date":"14 December 2012","permalink":"/posts/2012-12-14/opensmtpd-ldap-support-selectable-source-dkim-and-goodies/","section":"Posts","summary":"\u0026lt;p\u0026gt;OHAI,\u0026lt;/p\u0026gt; This week has been crazy, my brain melted a little as I worked on filters, it melted a little as I worked on LDAP and it melted a little more as I tried to trick Eric into working on filters with me (yes, that actually worked :-).","title":"OpenSMTPD: LDAP support, selectable source, DKIM and Goodies"},{"content":"OHAI,\nThis week I had intended to work on filters. After 3 days of pain and swearing, even though it did move forward, I finally decided to step back for a few days and work on something else to preserve my sanity.\nAs you may have guessed, this week has not been too productive\u0026hellip; Oh wait, yes it has ! It\u0026rsquo;s actually been incredibly productive :-)\nJust as usual, I will only write a summary for this week, the complete changelog can be retrieved from our mirror on Github.\nK_SOURCEADDR table service\nOpenSMTPD has been taught how to fetch a source address from a table, but does not make use of it yet. This will, for example, allow users to force a source address for their outgoing mail when they get blacklisted by the monkeys at spamhaus.\nThe K_SOURCEADDR service performs a cyclic lookup returning each address of a table one after the other so that a table holding multiple addresses will cycle through them.\nImproved DNS API\nEric has cleaned and improved the DNS API: dnsquery*() functions now have a more logical ordering of their arguments; struct dns has been killed and we now use two small imsg-specific structures; dns_query_mx_preference() has been introduced to retrieve the preference level of a specific MX for a domain; and finally all MX addresses are looked up in parallel instead of sequentially.\nMTA improvements\nEric has also improved the MTA internal operations so that it uses better abstractions for relay domains, hosts, sources and routes. A MTA session now operates on a given route and reports errors on that route only.\nThe relay tries to open as many routes as possible, within various limits, and drains all mails dispatching them. Oh and it is ready to use the K_SOURCEADDR lookup service but doesn\u0026rsquo;t do it yet, this will probably be part of next week\u0026rsquo;s milestone.\nTable API improvements and new SQLite backend\nThe table API now provides a simple mechanism for backends to support a configuration file without having to deal with the parsing. The backend can be declared in smtpd.conf using:\ntable foobar mybackend:/etc/mail/mybackend.conf\nThen the backend may simply do:\nstatic int table_mybackend_config(struct table *t, const char *configfile) { void *cf;\ncf = table_config_create(); if (! table_config_parse(cf, configfile, T_HASH)) { table_config_destroy(cf); return 0; }\ntable_set_configuration(t, cfg); return 1; }\nTo have the /etc/mail/mybackend.conf file parsed into a key/value table. It can then fetch the values from any other handler using:\nstatic void * table_mybackend_open(struct table *t) { void *cf = table_get_configuration(t);\nif (table_config_get(cf, \u0026ldquo;key\u0026rdquo;) == NULL) { log_warnx(\u0026ldquo;table_mybackend: open: missing key from config file\u0026rdquo;); return NULL; }\nreturn t; }\nSince I needed a use case, I added support for SQLite as a table backend allowing the use of SQLite for any kind of lookup. I had already done it in the past as a proof of concept when we were still using the map API for lookups, but this time it\u0026rsquo;s the real deal.\nSQLite support is achieved using the same approach as that of Postfix where the schema is not imposed but rather the user provides the queries themselves to allow as much flexibiliy as possible.\nTo show you how it can be setup, here\u0026rsquo;s a sample smtpd.conf:\nsmtpd.conf\ntable mytbl sqlite:/etc/mail/sqlite.conf\ni could have another one configured differently\ntable mytbl2 sqlite:/etc/mail/sqlite-other.conf\nand i can have the same one serve different kinds of lookups ;-)\naccept for domain alias deliver to mbox\nand here\u0026rsquo;s the sample sqlite.conf that goes with it\nPath to database\ndbpath /tmp/sqlite.db\nAlias lookup query\nrows \u0026gt;= 0\nfields == 1 (user varchar)\nquery_alias select value from aliases where key=?;\nDomain lookup query\nrows == 1\nfields == 1 (domain varchar)\nquery_domain select value from domains where key=?;\nOf course, you may have multiple smtpd tables using sqlite backends, they may use different configuration files, you can have two aliases databases for two different domains or use the same database table to hold all information for all lookups. It\u0026rsquo;s as flexible as it gets \u0026hellip; ALL lookup services are supported by the SQLite backend so it can be used to store anything used by OpenSMTPD.\nK_USERINFO lookup service\nOpenSMTPD uses the table API for every lookups but there was still one kind of lookups that were performed using a different API: users informations.\nThe table API expects lookups to be done asynchronously but OpenSMTPD had some code that looked up users synchronously, like right before a delivery or to find the home directory of a user for a ~/.forward check.\nI introduced a new lookup service, K_USERINFO, which allows processes to lookup for informations regarding a username, such as its uid, gid and home directory. I then reworked the ~/.forward check and the delivery code to ensure the user information lookup is performed asynchronously through the K_USERINFO service rather than through the user_lookup() API which was synchronous.\nThe only backend to implement K_USERINFO was table_getpwnam which was essentially doing the same as before, just asynchronously, and at that point, user_lookup() bit the dust.\nREALLY VIRTUAL users\nA feature that has been requested for a long time and which was very heard to implement was support for virtual users.\nOpenSMTPD required that the end user be a real system-user that could be looked up using getpwnam(). The K_USERINFO lookup service changed this slightly by having OpenSMTPD require that the end user be a user that could be looked up using table_lookup().\nAnd since we can write lookup services using any backend, I wrote K_USERINFO handlers for table_static, table_db and table_sqlite. I then added a new keyword to smtpd.conf to allow rules to specify a user table:\ntable bleh1 { vuser =\u0026gt; vuser:10💯/tmp/vuser } table bleh2 { vuser =\u0026gt; vuser:20:200:/tmp2/vuser }\naccept for domain poolp.org users deliver to maildir accept for domain opensmtpd.org users deliver to maildir accept for domain pool.ps deliver to maildir\nWith this, OpenSMTPD will accept mail for domain poolp.org but will only find users if they are part of the table bleh1. The domain opensmtpd.org has a different users database, that shares a username but does not share the uid, gid and homedir. In this example, I used static tables, but it could really be sqlite, db or whatever ;-)\nThe domain pool.ps has no users table and defaults to the system database which is what most users will expect.\nRelay URL update and K_CREDENTIALS lookup service\nSince several months, smtpd.conf supports a \u0026ldquo;relay URL\u0026rdquo; format to define relays we want to route via:\ntable creds { mail.poolp.org =\u0026gt; gilles:mypasswd } accept for any relay via tls+auth://mail.poolp.org:31337 auth\nWhen sending mail, the creds table will search for an entry matching the domain name of the relay and find the credentials there. This has annoyed me for a while because it meant that it was not possible to share credentials between multiple relays, it was not possible to have different credentials for two relays operating under the same name, etc, etc \u0026hellip;\nAlso, it annoyed me that outgoing authentication would use K_CREDENTIALS while incoming authentication would not use the table API but the auth_backend API instead.\nI convinced Eric that it would be nice to provide a new mechanism in relay URL so that we could have a label, like tls+auth://label@mail.poolp.org:31337 and it would be used as the key for the credentials lookup.\nThis would allow multiple relays to refer to the same label, or different relays under the same hostname to refer to different labels. It would also allow two nice tricks: first, since the labels are looked up in a different service then we can update the creds table live and MTA will pick up the change; then, if for incoming authentication we assume the username to be the label, then K_CREDENTIALS can be use as THE mechanism to authenticate both in and out sessions and we can kill the auth_backend API.\nSo \u0026hellip; I wrote it and we can now do:\ntable in_auth { gilles =\u0026gt; gilles:encryptedpasswd } table out_auth { bleh =\u0026gt; gilles:cleartextpasswd }\nlisten on all tls auth\naccept for domain poolp.org deliver to maildir accept for any relay via tls+auth://bleh@mail.poolp.org auth\nAnd this closes the last issue with regard to assuming any locality of the users for any purpose. An OpenSMTPD instance no longer assumes users to be local, or to really exists, for any purpose whatsoever.\nNext week should see a long awaited feature, I will not say much and leave it as a surprise. Meanwhile, you guys have a nice week-end, I need a zZzZ ;-)\n","date":"7 December 2012","permalink":"/posts/2012-12-07/opensmtpd-fully-virtual-setups-updated-dns-mta-code-sqlite-support/","section":"Posts","summary":"OHAI,\nThis week I had intended to work on filters. After 3 days of pain and swearing, even though it did move forward, I finally decided to step back for a few days and work on something else to preserve my sanity.","title":"OpenSMTPD: fully virtual setups, updated DNS \u0026 MTA code, SQLite support"},{"content":"OHAI,\nFirst of all, on a completely unrelated note, I\u0026rsquo;d like to emphasize that this blog post is being written from my text editor launched by a little shell and pushed to poolp using the Ocean API. I can be web2.0 from console and emacs, and that\u0026rsquo;s definitely worth a mention ;-)\nAs has become a habit, a lot of work has been poured in OpenSMTPD this week, here is an incomplete summary of the most notable changes. For the complete changelog feel free to check the commit log on our Github mirror.\nA snapshot should be published tomorrow but meanwhile enjoy the reading, some paragraphs have guest-starred eric ;-)\nSimplify parse.y grammar\nI have spent many hours working on a simplification of the parse.y grammar which worked fine for the correct configurations but which could allow very twisted configurations to parse valid.\nWhile fixing an ambiguous case, I realized that the parsing had inherited some complex logic from a feature that was desired a long time ago but which we never used as the rules could become ambiguous. So I began shooting down various parts which were not worth implementing and ended up removing no longer required lists and structures, making the use of tables more coherent accross the daemon as sometimes they were refered by table pointer and sometimes by table id.\nThe end result is a much less bloated grammar, which is semantically more right, and which is much cleaner.\nMore virtual simplification\nBuilding on the foundations from last week and the cleaned up parse.y, I have brought a new feature which was not possible before: using \u0026ldquo;for local\u0026rdquo; or \u0026ldquo;for any\u0026rdquo; as the destination of a virtual domain. Indeed, you could \u0026ldquo;accept for any\u0026rdquo; or \u0026ldquo;accept for local\u0026rdquo; but then OpenSMTPD would assume a primary domain and would perform a system user lookup for the user part. This was actually a side-effet of a parse.y ambiguity where ANY, LOCAL, DOMAIN and VIRTUAL were at the same level and VIRTUAL being considered as a special case.\nSince last week, a virtual domain is simply a regular domain which has defined a virtual mapping. The same logic has been applied to ANY and LOCAL meaning that you can \u0026ldquo;accept for any\u0026rdquo; or \u0026ldquo;accept for any virtual [\u0026hellip;]\u0026rdquo; and it will work as expected, instead of the syntax I demonstrated last week: accept for domain \u0026ldquo;*\u0026rdquo; virtual [\u0026hellip;], which is still valid but will work in a different way internally.\nA user had opened a ticket to ask if we could turn OpenSMTPD into a sink where it would accept mail for any destination and deliver to a single account. With the \u0026ldquo;accept for any virtual [\u0026hellip;]\u0026rdquo; improvement it became simpler as it only required adding a global catch-all that isn\u0026rsquo;t domain aware.\nVirtual now support a global catch-all \u0026ldquo;@\u0026rdquo; which allows a static mapping to handle the catch-alls for multiple domains:\naccept for any virtual { \u0026ldquo;@\u0026rdquo; =\u0026gt; gilles } deliver to maildir\nThe above will have any user of any domain delivered to local account gilles.\nGeneral cleanup\nWe have spent a large amount of time working on a general cleanup of the code base. Amongst other things, we removed some fields from the global struct smtpd to statically isolate them to the specific files that were using them. This helped ensure that we didn\u0026rsquo;t violate API layers.\nThen we spent a great deal of time killing a monster structure, struct submit_status, which was used for all kind of inter-process exchanges. We came up with several lighter structures, carrying only the required information and being tied to a particular process to process exchange. This has required a bit of rework in various processes but the end result is less confusion, code that\u0026rsquo;s easier to maintain and read for new comers.\nMFA rework\nThe MFA process, in charge of filtering the different stages of a SMTP session has been considerably simplified. It no longer knows about SMTP states, which was an API layer violation, and has had a lot of code removed while providing the same service. It shrank by over 100 lines and has become a very small piece of code whose only purpose is to serve as the entry point to the filters evaluation.\nThis was a pre-requisite to the filter work which is already complex enough that we didn\u0026rsquo;t want to add unneeded complexity upfront.\nMTA rework\nA lot of work has been done in the MTA engine which was almost completely rewritten. MTA now knows how to share MX, PTR and credentials for a route. This means that a single DNS request to the lookup daemon can be performed to deal with multiple messages heading to multiple domains.\nIt also deals much better with MX problems: When too many sessions fail to establish a link with a specific MX, the MX is marked as broken and not tried any further by new sessions.\nIt is also capable of dispatching connections on various MX of same priority: When sending many messages, the MTA will spawn sessions against the different MXs with the lowest priority within the default connection limit.\nIf all MXs at a given priority have errors, it moves to the next level to reach backup MXs. If no MX can be reached for a route, a temporary error is triggered.\nI should mention that several kinds of error that used to trigger a permanent error on messages will now simply mark MX on which they occur as broken, giving the mails a chance to be routed through another MX.\nThe logs have been improved too, especially with SSL-related errors. All problems on MXs are now logged (as \u0026ldquo;smtp-out:\u0026rdquo;) to help the administrator diagnose with relaying.\nSMTP rework\nThe SMTP engine has been reworked as a pre-requisite for filters. It is now running on poolp.org and powers the OpenSMTPD mailing list.\nThe idea was to make the code generally simpler to follow and extend.\nFirst, most of the smtp specific structures and defines have been isolated into smtp_session.c, which makes the smtpd.h file a bit less bloated. It was also the opportunity to finally get rid of the horrible submit_status structure.\nThe dispatching of user command is more straightforward, and the imsg dispatching code now makes use of very specific message structures.\nThe rewrite was painful, mostly because the former code was not easy to grasp, and because we slacked too much on the regression suite, which got improved in the process.\nFurthermore, the interaction with the MFA got more complicated with the recent update. Basically we need to de-couple the forwarding of the message data to the MFA from the receveing of filtered data.\nTo sum up, the new code should be a much better ground to implement features like proper filtering and pipelining on the SMTP side.\nimsgproc and filter work\nOpenSMTPD has had filters for over a year now, but disabled as the API is not stable and there were more important stuff to deal with.\nThe design for filters has been discussed on this blog already but as a quick reminder, filters in OpenSMTPD are standalone programs that are linked against a lib we provide to demonize and provide an event-based callback mechanism.\nThe filters can be very very easy and an example of a working filter could be as simple as:\ndefine SMTPD_FILTER\ninclude \u0026ldquo;smtpd-api.h\u0026rdquo;\nvoid mail_cb(uint64_t id, struct filter_mail p, void *arg) { / block idiots */ if (! strcmp(p-\u0026gt;domain, \u0026ldquo;0pointer.net\u0026rdquo;)) { filter_api_reject(id, 530, \u0026ldquo;You\u0026rsquo;re not welcome, go away !\u0026rdquo;); return; }\nfilter_api_accept(id); }\nint main(int argc, char argv[]) { / init the lib and setup daemon and imsg framework */ filter_api_init();\n/* register callbacks */ filter_api_register_mail_callback(mail_cb, NULL);\n/* event loop */ filter_api_loop();\n/* never reached */ return 0; }\nI have spent time in the glue code to change a bit how it worked and have it rely on a new api imsgproc which generalizes the setup imsg / fork / exec operation which we\u0026rsquo;ll end up using for all filters but also possibly for other pluggable backends. It works fine and I\u0026rsquo;ve successfully compiled about a dozen different filters working at different hooks and combinations of hooks.\nThe API is broken at this point due to the late commit of the smtp rework which didn\u0026rsquo;t leave me much time to fix before the week-end but surely it will get better next week.\nMonkey Branch\nYesterday morning I was waiting for Eric to merge a blocking part for my current work so I decided to switch to something radically different.\nSo I tackled a new issue: how do we ensure that our error code path is correct for the errors that you cannot reproduce easily because they are so rare. A typical example of such an error is a getpwnam() failure because a descriptor was not available or because we received an EIO. In such cases, we want OpenSMTPD to correctly handle the error as transient and not reject it permanently which would lead to a lost mail.\nI recalled this Chaos Monkey tool at Netflix that voluntarily produced errors at random to let them ensure their high-availability really works, and I came up with a set of MONKEY_* macros to randomly produce chaos at strategic places. The monkeys will provoke latency in imsg handling and will cause some places to report a temporary failure at random.\nIn the ten minutes that followed my initial testing, they helped spot two very subtle bugs which we would have probably not run into in years. We now know for sure that these code paths are correct and we need to spread more monkeys ;-)\nThis is done in a separate branch which I mirrored on github and which is synched with master. To test, simply checkout the monkey branch (lol no ?) and setup env CFLAGS=-DUSE_MONKEY before building. Then, when sending mail to yourself you should start seeing:\n$ echo test | mail -s \u0026rsquo;test\u0026rsquo; gilles $ echo test | mail -s \u0026rsquo;test\u0026rsquo; gilles send-mail: command failed: 421 Temporary failure $ echo test | mail -s \u0026rsquo;test\u0026rsquo; gilles send-mail: command failed: 421 Temporary failure $ echo test | mail -s \u0026rsquo;test\u0026rsquo; gilles $\nIdeally, OpenSMTPD should NEVER return a 5xx error while running in monkey mode that it doesn\u0026rsquo;t return when not running in monkey mode.\nFun with Google Charts ;-)\nThat has nothing to do with OpenSMTPD itself but rather with me playing with a Google API to generate graphs based on the output of smtpctl show stats:\n\u0026lt;img src=\u0026quot;https://www.poolp.org/~gilles/graph.png\u0026quot;\u0026gt; \u0026lt;img src=\u0026quot;https://www.poolp.org/~gilles/graph2.png\u0026quot;\u0026gt; I will try to spend some time making sure the smtpctl tool can be used to build other tools upon for administrators to take full advantage of the real-time statistics, profiling, scheduler-provided mail queue info, etc\u0026hellip;\nStay tuned, more goodies next week !\n","date":"30 November 2012","permalink":"/posts/2012-11-30/opensmtpd-more-features-more-cleanup-more-more/","section":"Posts","summary":"OHAI,\nFirst of all, on a completely unrelated note, I\u0026rsquo;d like to emphasize that this blog post is being written from my text editor launched by a little shell and pushed to poolp using the Ocean API.","title":"OpenSMTPD: more features, more cleanup, more more"},{"content":"Almost a month since my last post\u0026hellip; I know, I know.\nSo I left my position at Scality two weeks ago. I had been there for a year and a half working on a very interesting and tricky project. I will miss my colleagues and the discussions we had on all kinds of topic, I hope them the best future and that we will be able to share beers again ;-)\nI should also emphasize that they spoiled me for my leave with a ton of gifts including most of the books from my amazon wishlist (Cryptography Engineering, Implementing SSL/TLS, Theory of music) as well as candies, chocolate, rubber ducks see rubber duck debugging, boxers (for when I\u0026rsquo;ll become a homeworker) and, last but not least, a book about the inner details of former president Sarkozy occupation of the Elysee. I suspect the book to be a reminder that if even criminals with disgusting ass-faces manage to succeed in life, I shouldn\u0026rsquo;t worry about new challenges. Thanks.\nObligatory OpenSMTPD section of my blog post ;-)\nThe last two weeks have been spent working very hard on improving OpenSMTPD and we\u0026rsquo;re making progress at an alarming rate. I won\u0026rsquo;t go into the details of each and every improvement we made but here\u0026rsquo;s a quick summary:\nThe logging format has been improved, we have done a lot of rewording and changes to provide the most information using a concise format that can be easily understood by humans and that can easily be parsed or grepped.\nSupport for a monitor command has been added to the smtpctl utility. It allows an administrator to easily monitor a running instance of OpenSMTPD and what it is doing in real-time, displaying states every second.\nFirst shot at a regress suite with a utility that allows the scripting of SMTP sessions scenarios. In a near future, we will be writing various scenarios which will allow us to verify that we don\u0026rsquo;t introduce regressions with new features and bugfixes.\nImproved scheduler API by removing the very annoying and tricky to implement Qwalk API and replacing it with a new queue operation Q_LEARN.\nMailq now supports an \u0026ldquo;online\u0026rdquo; mode which provides more information than the offline mode. The online mode will query the scheduler getting reliable real-time information. Offline mode is equivalent to what we had before.\nImproved format for expansion strings in smtpd.conf and ~/.forward files. We used to support one char formats like %a, %u, and it was both confusing and hard to extend to support new formats while still making sense at first sight. We now support a clearer format: %{rcpt.domain}, %{user.directory}, etc\u0026hellip; The are various supported formats documented in smtpd.conf(5) and each support partial expansion using optional begin and end positive and negative offsets: with user \u0026ldquo;gilles\u0026rdquo;, %{user.username[1:5]} will expand \u0026ldquo;ille\u0026rdquo;.\nAdded a RAM queue_backend, mostly useful for debugging at this point or if you don\u0026rsquo;t care about losing mail when you shutdown the daemon ;-)\nImproved the grammar of our configuration file by removing a lot of stuff that was no longer relevant and changing the syntax to remove all ambiguities. Sure, it breaks existing configuration files but given that none should exceed about 10 lines, it should be straightforward to fix: the poolp.org smtpd.conf which voluntarily exhibits a complex setup (aliases, primary domains, backup MX, virtual domains, relaying, \u0026hellip;) was switched in two minutes.\nReplace the map API with a new table API that provides much simpler and saner semantics. Tables are simpler to declare than maps from a smtpd.conf point of view; they have types so that OpenSMTPD can detect tables used in an inappropriate context at smtpd.conf parse time; and they provide lookup services which allows backends to support only a few kinds of lookups and smtpd to spot that at smtpd.conf parse time too. The new table API simplifies a lot of things, the backends are simpler to write and the end-result is much more reliable.\nWith new table API I could get rid of the user_backend API and replace it with a table lookup service. It is now possible to write table backends to lookup system users instead of relying on getpwnam\u0026hellip; but at the moment OpenSMTPD hardcodes the use of internal table so there\u0026rsquo;s still work to do.\nWe no longer support \u0026ldquo;file\u0026rdquo; as a lookup backend. Whenever a smtpd.conf refers to a file for a lookup, it will internally convert it to a static table which will achieve the exact same result while removing duplicate code. The change is not visible to the users.\nNew dict* API, akin to the tree* API but with char* keys will allow us to simplify a lot of code with regard to how tables are handled and managed by OpenSMTPD.\nTONS of KNF cleanup (several hundreds), removal of old defines, refactors to remove structures that are used as unnecessary indirections, simplifications of equivalent code, etc, etc \u0026hellip;\nVarious bug fixes including one causing some envelopes to possibly be skipped and triggering a start-time crash.\nThat\u0026rsquo;s a SUMMARY, the complete updates can be checked on the commit logs of our github mirror if you\u0026rsquo;re interested. Keep in mind that this is the changelog of the past two weeks only, we have the same kind of changes coming in the next few days and we\u0026rsquo;re definitely not slowing down ;-)\nStay tuned !\n","date":"17 November 2012","permalink":"/posts/2012-11-17/news-from-the-front/","section":"Posts","summary":"Almost a month since my last post\u0026hellip; I know, I know.\nSo I left my position at Scality two weeks ago. I had been there for a year and a half working on a very interesting and tricky project.","title":"News from the front"},{"content":"The crypto backend that has been committed a couple days ago started a little discussion and it was decided to switch the default cipher from Blowfish to AES-128 and make it a default.\nEncryption can now be enabled as simply as:\nqueue encryption key \u0026ldquo;foobar\u0026rdquo;\nThat being said, Charles cleaned up a bit the compression code to make it use FILE* instead of file descriptors. This allows us to ensure we don\u0026rsquo;t have to deal with any buffering and interruption handling code.\nThe fread/fwrite buffer sizes have been increased to make compression and encryption more efficient and have them spend last time in IO.\nFinally, Mathieu (naabed- on #opensmtpd) has played puzzle with some old code of mine to bring LDAP support and provide a map_ldap backend. It\u0026rsquo;s not ready yet, but we imported code in the tree so that it can be worked on more efficiently.\nStay tuned !\n","date":"31 August 2012","permalink":"/posts/2012-08-31/opensmtpd-crypto/compress-fixes-and-import-initial-stab-at-ldap/","section":"Posts","summary":"The crypto backend that has been committed a couple days ago started a little discussion and it was decided to switch the default cipher from Blowfish to AES-128 and make it a default.","title":"OpenSMTPD: crypto/compress fixes and import initial stab at LDAP"},{"content":"A few days ago, Charles committed the compress_backend API which allowed transparent deflation/inflation of envelopes and messages as they hit the queue.\nThe compression code is executed before the queue_backend gets the data so that any queue_backend can benefit transparently from compression without having to add any code to handle it. Due to our design, this also means that if the queue_backend stores envelopes and messages remotely, then the compression will take place before sending and after fetching, the remote end only ever sees compressed data.\nThe idea with the compress_backend was to not only to save disk-space on the queue, which is a very worthy improvement for hosts that deal with very large queues; but also to prove a point: the queue_backends can be made agnostic of the envelopes and messages content, effectively handling blobs.\nAnd a side-effect of that is that since queue_backends don\u0026rsquo;t need to inspect envelopes and messages, and since they can store data remotely, we could provide full queue encryption and ensure that the remote end only ever sees encrypted data.\nThe crypto_backend provides a transparent encryption/decryption service that can be used in any part of OpenSMTPD. As of now, the queue is the only consumer but we already have other use-cases and probably some ideas we haven\u0026rsquo;t come up with yet will pop-up months from now ;-)\nOf course, encryption is tricky so the configuration is hard \u0026hellip; NOT !\nqueue encryption key \u0026ldquo;foobar\u0026rdquo; This will enable transparent encryption of the queue using Blowfish in CBC mode with a random IV and will internally expand the key \u0026ldquo;foobar\u0026rdquo; using sha256. This is just to provide sane defaults, one could override the cipher and digest using the following:\nqueue encryption key \u0026ldquo;foobar\u0026rdquo; cipher aes-128-cbc digest sha512 The IV is randomly chosen for every envelope and message, it is then encrypted and prepended to the message. This ensures that the same key used to encrypt the same envelope will produce different output.\nThings will still be improved further but in my opinion it is already covering the use-cases of many who would need an encrypted queue, and to be honest with a quick look on the intertubez I couldn\u0026rsquo;t find a competitor to compare to on that particular feature so we\u0026rsquo;ll move it forward as we hit use-cases.\nOh\u0026hellip; and of course this does work along the compression support so you can have both encryption and compression enabled.\nThis should hit the OpenBSD tree pretty shortly, this evening for sure.\n","date":"29 August 2012","permalink":"/posts/2012-08-29/opensmtpd-crypto_backend-and-encrypted-queue/","section":"Posts","summary":"A few days ago, Charles committed the compress_backend API which allowed transparent deflation/inflation of envelopes and messages as they hit the queue.\nThe compression code is executed before the queue_backend gets the data so that any queue_backend can benefit transparently from compression without having to add any code to handle it.","title":"OpenSMTPD: crypto_backend and encrypted queue"},{"content":"Howdie,\nEric and I had planned a loooooong time ago to have a hackathon this week-end. However, some asshole piece of shit ran into my car while it was parked, made it unable to take the road and flee without leaving a note. Long story made short, I had to be at the repair shop this week-end to be able to get my car before end of September.\nWe did get our hackathon though and while it was probably much less productive than if we had met in the same room, it was still very very interesting with tons of improvements and many great features written :-p\nAdd support for types and improve increment/decrement in stat_backend I recently committed the stat_backend API which allows us to provide different backends for statistics storage. Until now these stats were necessarily integer counters that supported being incremented, decremented and set.\nI added support for types allowing us to store more precise/accurate statistics in places where integers weren\u0026rsquo;t the best way to represent a value. It is currently possible to store counters, unix timestamp, timeval, timespec, etc \u0026hellip; We can easily extend types as we ran into uses for them.\nWhile at it, Eric asked me if I could allow increments/decrements by values to avoid having to call stat-\u0026gt;increment() or stat-\u0026gt;decrement() multiple times when a batch of increments/decrements are happening.\nDone, committed.\nIntroduce TRACE_PROFILING and profile stats A while ago, Eric has written profiling support which we have used to pinpoint some bottlenecks and fix them pretty successfully. For some reason, it was never committed but the idea was to fetch clock value before and after an imsg callback, allowing us to know precisely the time spent in each events.\nI merged the profiling support to our -current tree since it required some rework and committed it; then I took advantage of the new stats types support to also support pushing profiling information into the stats API.\nWith the ramstat backend it has limited value, but with a persistent storage backend for statistics you can easily graph your events profiling in real-time with about no impact on performances. This doesn\u0026rsquo;t sound too sexy to users but believe me it is. If you run OpenSMTPD on a very busy machine and you think it is not working as efficiently as it should, you can get a LIVE insight at which operations are taking time and should be optimized.\nStart removing user backend We currently have a user_backend API which is supposed to allow us to bring support for virtual users. That API needed to provide getbyname() and getbyuid() handlers to allow users lookups by name AND uid, preventing us from simply using the maps API.\nTurns out that the only use for getbyuid() was in the offline enqueue code path which can safely use getpwuid() as it will necessarily come from a local user. With getbyuid() out of the way, we\u0026rsquo;re almost ready to start removing user_backend API and start providing support for completely virtual users using maps and the supported backends.\nScheduler improvement Eric did quite a bit of work to improve the scheduler, fix some bugs in it and think about some of the improvements we want to bring to it.\nWe still have a disagreement on one algorithm to use internally to boost performances further but it only affects a small part of the scheduler and we will sort that out by experimenting both as it is really nitpicking at this point and only a live test with billions of envelopes in the queue will prove either theory to be right/wrong.\nImproved logging Todd (toddf@) improved IPv6 addresses logging by also printing ports when needed. He also changed the enqueuing logging to display both server and clients exchanges, instead of the old method that only displayed server answers. While there Eric added logging for smtpd pauses and resumes as well as envelopes removals.\nEvents race condition at startup We discovered a race condition that could cause OpenSMTPD to crash at startup if some processes start sending imsg to others before they are ready to service them.\nEric fixed the following bugs by disabling some events and enabling them back when the process is ready to serve. This has been a long-standing issue fixed with a 6 liners diff :-p\nFD exhaustion handling in smtpd and control process Until now, if OpenSMTPD had reached system-imposed limits on descriptors, it would simply fatal. I spent some time making it more reliable by defining a descriptor reserve under which it would stop accepting clients on the network or through smtpctl until the exhaustion situation is resorbed.\nExperimenting showed that it works pretty fine, there is still probably some improvements to do in that area performances-wise but at least it is stable in such situations and sessions run smooth without performances degradation.\nQueue compression Charles committed his queue compression feature which allows OpenSMTPD to deflate/inflate envelopes and messages transparently when they hit or are read from the queue.\nIt worked pretty fine but when I tried to inflate an envelope, I realized that the envelopes and messages didn\u0026rsquo;t use the same compression algorithm. I reworked a bit the compression backend to have it use gzip algorithm consistently.\nThis is awesome, it works out of the box by adding to smtpd.conf:\nqueue compress\nWith this, the size required to store envelopes is halved and a huge gain can be made on messages at the cost of some CPU time \u0026hellip; and from an administration point of view, the queue can be inspected using gzcat instead of cat \u0026hellip; piece of cake !\nQwalk rewrite We received a bug report where envelopes would appear several times in a mailq output.\nEric started investigating the issue and pinpointed the qwalk() API as the culprit. He rewrote it in a saner way and got the issue fixed.\nWhile he was doing it, another OpenBSD hacker sent a different bug report which turned out to be also fixed with the new qwalk(). That\u0026rsquo;s what we call proactive bug fixing :-p\nQueue encryption Finally, one of the features I\u0026rsquo;ve been waiting for a long time to implement: encrypted queue.\nI built upon Charles\u0026rsquo; queue compression diff to provide queue encryption as well. It works in a transparent way and out of the box by simply adding to smtpd.conf:\nqueue encrypt \u0026ldquo;mysecretkey\u0026rdquo;\nWith this, envelopes and messages are encrypted using AES-128 in ECB mode with \u0026ldquo;mysecretkey\u0026rdquo; as a key. I have plans to make the algorithm and mode selectable but I wanted it to work first before I start improving it. This feature is compatible with the compression, so you can enable both and it will compress AND encrypt (in proper order ;-) transparently.\nWhat\u0026rsquo;s even nicer is that our handling of envelopes/messages is fully isolated OUT of queue backends so that if you write a queue backend you never deal with envelopes and message contents. The side effect is that you can write a queue backend to store on a remote untrusted storage \u0026hellip; and people with control of that storage can never inspect content if encryption is enabled :-p\nI didn\u0026rsquo;t commit it yet but it should hit the tree this week.\nStay tuned for more awesomeness !\n","date":"26 August 2012","permalink":"/posts/2012-08-26/opensmtpd-i-have-no-idea-for-that-title-sorry/","section":"Posts","summary":"Howdie,\nEric and I had planned a loooooong time ago to have a hackathon this week-end. However, some asshole piece of shit ran into my car while it was parked, made it unable to take the road and flee without leaving a note.","title":"OpenSMTPD: I have no idea for that title, sorry"},{"content":"It\u0026rsquo;s been over a month since my last post but I have an excuse: I was super busy, then I was super on vacation \u0026hellip; but I\u0026rsquo;m back again ;-)\nSo my life was pretty intense these last few weeks, both professionnally and on the opensource front. I\u0026rsquo;ll explain later, for now let\u0026rsquo;s just focus on the OpenSMTPD goodies.\nOpenSMTPD portable to a new system First of all, I\u0026rsquo;d like to stress out that Rune Lynge and Charles Longeau did an amazing job at porting OpenSMTPD to yet another system \u0026hellip; MacOSX. While I don\u0026rsquo;t know anyone running a mail server on MacOSX, nor do I know anyone who knows anyone doing it, the fact that we can do it fills my heart with joy. We can officially be a free and very expensive mail server at once ;)\nThe compressed/encrypted queue experiment Charles also worked on a proof-of-concept to compress the message and envelopes in the queue. It is not committed yet but there is discussion around it in private as it will very likely hit the tree soon. This is a very important move forward as it also proves that compressing/encrypting the queue does not prevent OpenSMTPD from working which in turn proves that we will be able to store an encrypted queue in an untrusted storage (ie: public cloud provider) and still be able to deliver mail while retaining privacy. Believe me, we\u0026rsquo;re not done on that front and you\u0026rsquo;ll hear about it again in the future.\nScheduler/Queue layer separations On my end, not much done since the hackathon. While there, we had discussed a scheduler and mta refactor with Eric Faurot. The design was not optimal, there was a layer violation with the scheduler taking looks at the queue during initialization, etc \u0026hellip;\nDuring the hackathon I moved the queue initialization to the queue process. I didn\u0026rsquo;t commit it but Eric cloned my repository, improved it further and committed the result a few days ago. I then realized that since the scheduler no longer had to look at the queue, it could be chrooted to /var/empty instead of /var/spool/smtpd to further restrict it in case of a catastrophe.\nTransactional scheduling At the end of the hackathon I also started working on a new scheduler logic, it started with just a few simplifications but then I realized that making the scheduler transactionnal would help us reach the design we had discussed.\nThe scheduler initially consisted of a set of trees and a list sorted by schedule time. An envelope would be part of the trees allowing fast lookups for internal purposes, and the scheduler would just have to look at the first envelope of the list to see if something could be scheduled.\nThis had a side effect that as soon as pushing an envelope in the scheduler, it could be scheduled. That is not always desireable as sometimes we have multiple recipients for a same message / destination and we start scheduling before we get a chance to check that other recipients can be part of the same SMTP transaction. This would lead to inefficient MTA sessions as we would potentially connect multiple times to the same host to deliver the same message for a handful of different recipients.\nThe transactional scheduler uses a set of tree and linear queue BUT when it receives envelopes it doesn\u0026rsquo;t add them to these structures. Instead it adds them to a temporary set of structures, the changeset, which can then be atomically committed to be merged to the scheduling structures, or rollbacked depending on errors we encounter. That ensures that whenever we commit and make envelopes scheduleable, we are able to send as many envelopes as possible in one go to optimize MTA sessions.\nWhen I left the hackathon this was partly done but still required lots of work and improvements which Eric completed by himself and finished just a few days ago. It works perfect, it is a HUGE step forward which will make OpenSMTPD far more efficient.\nNew MTA logic While he was at it, he also reworked entirely the MTA logic. I\u0026rsquo;m not familiar with the new code yet but what I can tell from it is that it\u0026rsquo;s been simplified, cleaned and that it now supports some long awaited features like connection-caching to avoid the overhead of round-trips by reusing existing connections when different messages are heading to the same host.\nActually, it is a bit more complex than that because we no longer deal with hosts but rather with \u0026ldquo;routes\u0026rdquo; which allows reusing the same connection when talking to a MX that is shared by multiple domains (ie: poolp.org / opensmptd.org share mx1.poolp.org, so if we have a connection to mx1.poolp.org already opened, mails for both domains will take advantage of it).\nThere is still work to do in that area but this is also another HUGE step forward on a bit of code that we had planned to rework for months.\nStats backends Statistics used to be stored as size_t fields of fixed structures residing in a chunk of shared mmap-ed memory.\nThis was highly annoying as adding new counters would require changing structures, while the shared memory was in complete opposition with our separated processes design. To make things more annoying even, some of the statistics were used to alter logic when they were supposed to be informative only \u0026hellip; and the statistics were necessarily non-persistent as they were kept in RAM.\nI came up with the stat_backend API to allow writing custom statistic backends and implement the ramstat backend to provide the same feature as before.\nThe main difference lies in the fact that the stats are pushed and never fetched so the logic can never use them; the keys are dynamic so we can generate very precise statistics by creating keys as dynamic strings containing msgid, evpid, etc \u0026hellip; and since they are only pushed, we can centralize them to a single process, have them pushed by imsg and remove the need for shared memory.\nI also wrote a sqlitestat backend which is not committed but that does work and that allows providing persistent statistics that survive accross restarts so that admins can generate graphs and whatnot.\nIdeas float to write stat backends to send to collecting daemons, snmpd, etc \u0026hellip; but the next plan for the stat API is to extend it to provide different type of statistics so we can have ratios, times, etc\u0026hellip;\nBounces grouping OpenSMPTD tries to avoid bounces by rejecting early at session time. Sometimes this is not doable as we accept a recipient and the delivery fails later, after a ~/.forward for example, and we have to generate a bounce for a failed recipient.\nInitially OpenSMTPD grouped all failed recipients coming from same sender as part of the same bounce message so that sender would only receive one report with all failed recipients \u0026hellip; but that feature got broken at some point in time leading to as many bounces as failed recipients.\nIt was conceptually simple to implement but it could not easily be done until queue/scheduler interaction logic was rewritten. Now the queue handles the bounce re-enqueueing. The grouping is simply achieved by delaying the actual bounce a bit in case another bounce for the same message arrives shortly after. Guess who designed and implemented it ? Yup, Eric !\nLoop detection Loop detection was performed by the scheduler, which was a layer violation and which went in the way of Charles\u0026rsquo; attempt at the compression experiment.\nWe got rid of it altogether and reimplemented it at the proper place. I added a Delivered-To loop detection at the MDA level while Eric plugged the Received loop detection at the MTA level.\nBackup MX Oh and since Eric was bored with all these \u0026ldquo;little\u0026rdquo; reworks, he implemented today the also long awaited \u0026ldquo;backup MX\u0026rdquo; features.\nUntil now, OpenSMTPD could not operate like a real backup MX, we could trick it into doing a similar job by using a set of \u0026ldquo;relay via\u0026rdquo; rules like I did on mx2.poolp.org:\naccept for domain \u0026ldquo;poolp.org\u0026rdquo; relay via \u0026ldquo;mx1.poolp.org\u0026rdquo;\nBut this would be a pain to maintain when many backup MX with different priorities are declared in the zone and it would not honour the priorities.\nIt is now possible to do:\naccept for domain \u0026ldquo;poolp.org\u0026rdquo; relay backup \u0026ldquo;mx2.poolp.org\u0026rdquo;\nwhich will declare ourselves as the mx2.poolp.org MX backup for domain poolp.org and allow OpenSMTPD to perform a MX lookup and try to find a MX with a higher priority to hand the mail over. The diff is so small that it\u0026rsquo;s a shame we didn\u0026rsquo;t have this before :-p\nGetting closer to release OpenSMTPD is now in an almost production-ready state, in a better shape than ever, and we\u0026rsquo;re looking forward to receive as many tests and bug reports as possible to ensure that it\u0026rsquo;s rock solid.\nYou can submit your bug reports on our github tracker, feel free to join us on IRC at #OpenSMTPD @ freenode and subscribe to our mailing list misc@opensmtpd.org by sending a mail with subject [misc] subscribe.\nCheers !\n","date":"21 August 2012","permalink":"/posts/2012-08-21/opensmtpd-plenty-of-news/","section":"Posts","summary":"It\u0026rsquo;s been over a month since my last post but I have an excuse: I was super busy, then I was super on vacation \u0026hellip; but I\u0026rsquo;m back again ;-)","title":"OpenSMTPD: plenty of news"},{"content":"Yesterday, I came back from Budapest and was too tired to write anything, but here\u0026rsquo;s a quick summary of all the features that have been implemented with regard to OpenSMTPD by Eric, Charles and I.\nFirst of all, Eric [eric@] has moved ASR, his asynchronous resolver, to OpenBSD\u0026rsquo;s libc. This is a huge step forward that will allow OpenSMTPD to contain less code, while providing a better coverage of the resolver code. I really hope it catches on other systems which can easily get the resolver from OpenBSD and provide saner asynchronous resolving.\nI cleaned up the scheduler code a bit by introducing a new file to separate the scheduler code from the scheduler API. This brings the scheduler code closer to the logic we have for other parts of OpenSMTPD where we support custom backends.\nWhen OpenSMTPD receives a mail for a local address with a \u0026lsquo;+\u0026rsquo; in it, it ignores the second part so that for example \u0026lsquo; gilles+foo@poolp.org\u0026rsquo; is really delivered to \u0026lsquo;gilles\u0026rsquo;. Charles [chl@] added support for Maildir tagging, a very nice feature that had been requested by a user recently and which allows OpenSMTPD to automatically create directories within a Maildir to categorize incoming mails. So now, if you setup OpenSMTPD to deliver to a maildir AND OpenSMTPD receives a mail for a local address with a \u0026lsquo;+\u0026rsquo; in it, it will extract the second part and use it as a subdirectory of the Maildir.\nI finally plugged the relay URLS in parse.y allowing simpler smtpd.conf syntax when using \u0026ldquo;relay via\u0026rdquo; rules, but also preparing the way for upcoming relay maps. I had discussed this on this blog a while ago, but the idea is that you can now express relays using URL like \u0026lsquo;smtp://mx1.poolp.org\u0026rsquo;, \u0026rsquo;tls://mail.poolp.org\u0026rsquo;, etc \u0026hellip; and soon you will be able to provide a map with multiple URL and let OpenSMTPD deliver from either.\nOpenSMTPD initially supported multiple queues and as time passed by, we removed them and came up with a simpler design that was more effective. The code to differentiate between queues was still all over the place and both Eric and Charles worked on getting rid of it. Finally, Charles managed to remove the last pieces and kill the latest bits of it.\nWith Eric, we decided to reduce the number of buckets from 0xfff to 0xff as we observed performances degradations during enqueuing when the queue had thousands of buckets. It was not a matter of envelopes in the buckets but really a degradation caused by the number of entries in the first level directory.\nI simplified the scheduler loop logic to make it much much simpler than before. It cannot be much simpler than now: check next envelope schedule time -\u0026gt; schedule / sleep until schedulable. This may sound too simple for real world, but it is actually because we moved the smartness outside of the scheduling loop and it will deserve a post by itself. I have only committed the scheduler loop simplification for now, the rest is coming soon ;-)\nThere was a problem in the scheduler logic loop which was caused by an optimization I wrote and the fact that our signals are handled asynchronously. This would not be visible most of the time, but if you ever had a very busy queue with tons of messages being schedulable, then the scheduler could not be interrupted with a signal. Sending CTRL-C to OpenSMTPD would kill all processes excepted the scheduler which would keep on as long it had a schedulable envelope before catching the signal. This has been fixed by forcing the scheduler to exit the scheduling loop after each envelope but setting a new scheduling loop call right away. It allows OpenSMTPD to handle the signal and resume its loop.\nI added a TRACE_SCHEDULER trace so that we can easily log scheduler actions using log_trace() and enable/disable them at will without rebuilding. This was prompted by Eric asking me if I still had to keep the ugly output or if I was done with debugging the scheduler ;-)\nCharles added support for literal inet addresses, allowing a user to send mail from or to a user at a specific internet address (gilles@[192.168.1.2], for example). This was requested a while ago, not to hard to accomplish but it was still sitting in our bug tracker.\nHe also restricted the character set allowed in an email address. We\u0026rsquo;re supposed to handle a shitload of characters including [!#$\u0026amp;\u0026rsquo;*/=?^`{|}] as well as any other one inside double quotes. This is so completely fucked up that we decided we don\u0026rsquo;t want to support them and deal with being scared that one could leverage a very smart attack exploiting a user part in a shell filter. If your email address contains one of the forbidden characters, either come up with a good rationale, or use another MTA. For other people, you will probably not notice because no one sane would use \u0026lsquo;#\u0026rsquo; or \u0026lsquo;{\u0026rsquo; in an email address.\nFinally, the hackathon has been much more productive as Eric and I shared the same room and got to discuss some refactors during hours after our hacking sessions at the hackroom. We have designed the new scheduler and MTA refactors which are going to be very very impressive, I can\u0026rsquo;t wait to write about it when it\u0026rsquo;s done [yes I have started working on it, and Eric cloned my branch and worked on it in the plane] ;-)\nOff to sleep, I have to recover now and wake up for my real work in a few hours !\n","date":"15 July 2012","permalink":"/posts/2012-07-15/g2k12-openbsd-hackathon-part-ii/","section":"Posts","summary":"Yesterday, I came back from Budapest and was too tired to write anything, but here\u0026rsquo;s a quick summary of all the features that have been implemented with regard to OpenSMTPD by Eric, Charles and I.","title":"g2k12: OpenBSD hackathon - part II"},{"content":"Yesterday, after nearly missing my plane by 5 minutes, I finally managed to make it to Budapest for g2k12, the OpenBSD general hackathon.\nWhile this is not my first hackathon, this is the first general one with many hackers working on all kinds of subsystems ranging from ports to network and kernel. It\u0026rsquo;s fun to see many people focused on improving different areas which you use daily, sometimes without even noticing, and doing it with as much fun as you are having fixing your own area ;-)\nThere are way too many changes happening to mention them all, and they will probably be part of an Undeadly series of article, so I'll just focus on what Eric, Charles and I did. I apologize in advance for the low redactional quality of this post, but it's late, I'm tired and it's about hotter here than on the sun so I will probably be too lazy to read myself. Anyway, this is just a summary of what happened since yesterday:\nFirst of all, I scratched the Github repository for OpenSMTPD and reimported it using 'git cvsimport' to retain the 4 years of history. Worked like a charm, there are still a few glitches in the workflow but they should be sorted soon.\nIn the past, OpenSMTPD supported multiple queues but it turned out to be a useless features which made the queue API slightly more complex. Eric had started simplifying and removing the code that was superfluous but we still had this queue_kind thing that allowed the logic to determine which queue it was dealing with. Charles got rid of it to simplify our queue code further.\nMeanwhile, Eric (finally) committed his asynchronous resolver in the libc. This allowed him to get immediate feedback from other hackers who confirmed that it was pretty impressive and improved drastically some ports. There are still some shortcomings but this is a very nice move forward and allows to remove the asynchronous resolver from OpenSMTPD to keep the code base small.\nI then plugged the text_to_relayhost() code which allows expressing relays as an URL. This means that: accept for domain foobar.org relay via mx1.poolp.org \\ port 666 tls auth becomes: accept for domain foobar.org relay via \"tls+auth://mx1.poolp.org:666\" It is nicer as when relay maps are finished, one rule will be able to refer to multiple relays that do not necessarily share the same protocol and options. Also, I think that it looks much better and is much easier to remember ;-)\nRecently, Eric and I did a quick benchmark of OpenSMTPD to figure out what were the bottlenecks. The benchmark was quite interesting and I think it deserves a post by itself so I won't go too much into the details. Point is, after a quick discussion yesterday evening I removed the /envelopes/ subdirectory of each message to store envelopes at the same level as the message they are carrying. This saves an extra mkdir() call per message, not much but saves 3 seconds on the enqueuing of 1000 mails. Also, Eric noted that the use of 24 bits for our buckets caused performances to degrade when our queue is full because of the readdir() iterations on a dir entry with 4096 entries. We reduced to 16 bits, which allowed to save a few more seconds. There's still lots of room for improvements but the performances area I will really need to write a full post about, we have fancy graphs and stuff to show how much we understand the bottlenecks and how we can be confident we will be damn fast ;-)\nCharles then adapted portable OpenSMTPD to the removal of ASR and got a build that would link against it externally. It was committed to Github, no snapshot generated yet but we will generate one tomorrow for sure.\nI started simplifying the runner code. The runner is actually a scheduler, it was called \"runner\" because it used to iterate over runqueues which we no longer have. Since it is a scheduler, I renamed it and made sure everything that refered to it would refer to it with the appropraite name. Also, since the scheduler code is tricky we need to be able to log and Eric was annoyed by the very verbose logging. I added support for a \"scheduler\" trace which allows to toggle the very verbose logging using '-T scheduler'. While at it, since the scheduler logic was tricky, I started working on making it much simpler by heavily commenting it and reducing it to about 20 lines of very very simple logic.\nThere's still code to be written for the scheduling because we discussed of a new strategy for scheduling envelopes. It will (in theory) make our lives a lot simpler and allow some very nice feature we had in the pipe for long. I've started but at this point I got too drunk to keep writing that kind of invasive code.\nMost important of all, we spent a lot of time talking, designing and thinking about what will be done this week and in the weeks and months that follow. These discussions are probably worth more in terms of productivity than all of the code we've written these last few hours...\nFinally, as I'm writing this, Charles is sitting at a table with me in the hall of the hotel working on closing one of our \"known issues\"... but I'll leave that for tomorrow ;-)\n","date":"9 July 2012","permalink":"/posts/2012-07-09/g2k12-openbsd-hackathon/","section":"Posts","summary":"Yesterday, after nearly missing my plane by 5 minutes, I finally managed to make it to Budapest for g2k12, the OpenBSD general hackathon.\nWhile this is not my first hackathon, this is the first general one with many hackers working on all kinds of subsystems ranging from ports to network and kernel.","title":"g2k12: OpenBSD hackathon"},{"content":"It\u0026rsquo;s been a while since my last post about OpenSMTPD.\nI don\u0026rsquo;t know what Eric and Charles have been up to, I was quite busy with my daytime job and couldn\u0026rsquo;t really spend much time on the hobby ;-)\nAnyways, one of the most requested feature of OpenSMTPD is to provide users with a mean to report bugs and let them contribute more easily. It\u0026rsquo;s been requested two times this week, it\u0026rsquo;s been requested countless times in the last few weeks and months, and as a matter of fact it\u0026rsquo;s also been a nuisance for us as until now we\u0026rsquo;ve kept track of bugs and feature requests in my mailbox. Bug reports have been lost, others have been forgotten, it\u0026rsquo;s not productive.\nI decided to take a shot at fixing that issue and proposed to my fellow coworkers that we create a Github project, just like what jasper@ did for ports. Charles was already a big fan of Github (\u0026ldquo;I \u0026lt;3 Github\u0026rdquo;) and Eric approved and created an account. So here we are ;-)\nWhy Github ?\nI initially thought about writing a small tool at poolp.org to keep track of bugs and features, and it wouldn\u0026rsquo;t be too hard, but it requires time and that is a very scarce resource these days.\nI\u0026rsquo;ve had very good feedback of Github, it has a nice interface, is easy to use and has the bug tracker, comments and annotations I want. I don\u0026rsquo;t see the point in reinventing the wheel (at this point) so as far as I\u0026rsquo;m concerned it will be just fine.\nWe don\u0026rsquo;t intend to use the repository there as a development repository but instead as a mean to let users know what\u0026rsquo;s in the works by exposing upcoming features.\nSee, OpenSMTPD has several repositories:\nThe OpenBSD repository, which is our primary repository and where OpenBSD users should grab the latest version of OpenSMTPD. It is as close to production-ready as it can get and no other tree (that I know of :-) holds a \u0026ldquo;better\u0026rdquo; version.\nThe portable repository, which is maintained by Charles Longeau at Github, and which is basically a checkout of the OpenBSD tree with the portability goo required to build on other operating systems.\nThen, at poolp.org, we have a private repository where Charles, Eric and I create branches to work on specific new features that can possibly break OpenSMTPD, bring their share of regressions, or that are simply not meant to hit the OpenBSD tree as-is (the so-called \u0026ldquo;experiments\u0026rdquo; :)\nThe repository at poolp.org contains at least 2 branches:\nThe master branch which is synchronized with the OpenBSD tree and from which we derive when we implement new features; the poolp branch which is an experimental branch we use to run code live and test that it is reliable before it is committed to OpenBSD; and eventually several minor branches that come and go.\nThe workflow goes more or less like this:\ncreate a branch X from master for feature X; work on feature X until reasonnably stable; merge feature X in branch poolp; test on mx1.poolp.org that branch poolp is stable; commit feature on OpenBSD; synchronize master branch with OpenBSD; The repository at Github would clone our master branch and eventually other branches, making it visible to the community.\n","date":"19 June 2012","permalink":"/posts/2012-06-19/opensmtpd-development-bug-tracker-github/","section":"Posts","summary":"It\u0026rsquo;s been a while since my last post about OpenSMTPD.\nI don\u0026rsquo;t know what Eric and Charles have been up to, I was quite busy with my daytime job and couldn\u0026rsquo;t really spend much time on the hobby ;-)","title":"OpenSMTPD: development, bug tracker \u0026 GitHub"},{"content":"This is the first post of a series to illustrate and describe a \u0026ldquo;proof of concept\u0026rdquo; code by Charles, Eric and I. I will describe the features as they are implemented.\nSince mid-November 2011, OpenSMTPD offers support for an extensible queue API. The queue_backend API allows a developer to write a custom storage driver by implementing a small set of functions that take care of storing, updating and removing envelopes and messages. The details behind these functions aren\u0026rsquo;t exposed to the daemon which only manipulates message and envelope identifiers.\nWhen the feature was committed, I said that \u0026ldquo;we should know be able to store the queue anywhere\u0026rdquo; to which an OpenBSD hacker joked about being webscale with a MongoDB backend. Little did he know that using a cloud was (one of) the real motivations to that change ;-)\nOpenSMTPD ships with a single queue backend, namely the queue_fsqueue backend, that implements a file-system store using a hashed layout. Technically, we SHOULD be able to use ANY backend as long as it allows key/value storage and linear walk (thought linear walk of a subset is preferable ;-).\nNow, what would we gain from storing the queue in a cloud ?\nSMTP servers tend to store their queue locally. When my primary MX is down, another server, called a \u0026ldquo;backup MX\u0026rdquo; is going to have to accept the mail and store it locally, so that when my primary MX is up again that backup MX sends the mail so my primary MX can store it locally. After what it can finally decide to deliver it, either locally or by relaying to another MX.\nWe actually have a hierarchy of prioritized servers, each accepting mail for a domain and keeping it locally until a server with a higher priority is available again. That sounds safe, but it is actually rather fragile as it relies on EVERY MX doing the proper work to ensure nothing gets lost. Technically, it shouldn\u0026rsquo;t receive much mail, but in practice my secondary MX never has an empty queue and if this is true for a site as small as poolp.org, I\u0026rsquo;d guess secondary MX at bigger sites should also have mails they don\u0026rsquo;t want to lose on their secondary MX.\nTo be really safe, EACH MX server be it primary or secondary should ensure queue redundancy and replication to another machine so that a mail never gets lost if the machine goes down temporarily or permanently. Of course, I don\u0026rsquo;t think I know anyone doing that. Of course, I don\u0026rsquo;t even do that. In place, we rely on statistics that it is unlikely to happen, or that it happens so rarely that it\u0026rsquo;s not worth the trouble.\nNowadays, there are plenty of solutions to store mails on distributed storages. This could be an opensource solution deployed on a set of servers you own, or a service provided by Google, Amazon or even that awesome company called Scality. By sharing a queue that provides high-availability and replication, we can ensure that the burden is no longer on the MX. As far as incoming mail is concerned, ALL MX are equivalent and can be cheap and low end machines with small disks, that don\u0026rsquo;t provide replication and backups. If a mail is accepted by a specific MX, it enters THE SAME QUEUE as the primary MX in charge of delivering the content.\nTechnically this means we can remove the priorities and create a pool of many MX with identical configuration that all accept incoming mail.\nIf we face a peak and need to scale \u0026hellip; we just add clones to the pool !\nNice theory, how long before it is reality ?\nI told Charles and Eric that it would be very nice if we had a REST backend. This would allow people to write small frontends to their storage backend, and let OpenSMTPD communicate with these frontends. With this, we could let OpenSMTPD use any cloud instead of showing a preference to one (even though SCALITY offers the best of all worlds, but that\u0026rsquo;s just a totally biased opinion ;-)).\nSince I couldn\u0026rsquo;t provide access to a scality RING to Eric and I was already familiar with Google\u0026rsquo;s AppEngine API, I installed the SDK and started writing the frontend to their datastore. The beauty of our API is that if it works for ONE, we know it will work for EITHER ONE. I thought it would take a few days to achieve the result I wanted but after about an hour I had a working REST server and had sent Eric a mail with the description of the API. Basically, the goal of the front-end is just to map an URL to a storage operation, so it was really about writing a few dozens lines of code.\nThe next day, Eric told me that he had cloned the API on his machine to ease development and that he had started writing the backend. Yes. He had rewritten a frontend just to ease development because it was much easier than using the remote frontend. That speaks a lot about how easy it is to write one for your own custom storage backends.\nHe asked me to make somes changes to the frontend API that would make things easier and more efficient for him, and after about another hour of doing things right I left him with the latest version of the REST server deployed on Google\u0026rsquo;s cloud.\nWhen I woke up the next day, I had a mail in my mailbox saying that \u0026ldquo;his queue is web-scale\u0026rdquo;.\nEric\u0026rsquo;s backend uses an asynchronous HTTP client he had written and crafts the queries that are expected by the frontend. The code is not in a committable state yet, but it does work and paves the road for many many more interesting features to come.\nWith his backend, OpenSMTPD no longer stores mails locally, it has its queue deported to Google AppEngine and fetches/commits there. I can start another instance of OpenSMTPD with the same configuration on another machine that sees the exact same queue.\nIn one word, it\u0026rsquo;s \u0026hellip; AWESOME ;-)\nWhere do I get the code ?\nYou don\u0026rsquo;t, the PoC is not finished yet, we have three steps missing\u0026hellip; each as funky as this one ;-)\nWe will very likely make the REST backends part of the official distribution as they are independent of the storage solution, however we won\u0026rsquo;t ship the frontends as they are tied to the storage solution (and commercial companies) and it\u0026rsquo;s not acceptable with regard to OpenBSD rules.\nThe frontends will be distributed unofficially, we\u0026rsquo;ll sort out how in time.\nWhat\u0026rsquo;s next ?\nWAIT AND SEE ;-)\nYou can join us on #OpenSMTPD @ irc.freenode.net to discuss and help.\nWe have a mailing list available at misc@opensmtpd.org where we can discuss OpenSMTPD related stuff that is not OpenBSD-specific. To subscribe, just send a mail with subject: [misc] subscribe\nOH, and just for the record, NO I have not been paid by scality to place their product here, I just like what we do there ;-)\n","date":"6 June 2012","permalink":"/posts/2012-06-06/opensmtpd-rest-queue/","section":"Posts","summary":"This is the first post of a series to illustrate and describe a \u0026ldquo;proof of concept\u0026rdquo; code by Charles, Eric and I. I will describe the features as they are implemented.","title":"OpenSMTPD REST queue"},{"content":"First of all, I\u0026rsquo;d like to \u0026ldquo;thank\u0026rdquo; The Spamhaus project which forced me into hacking this feature in a hurry.\nSee, I rent a server at online.net which I use for OpenSMTPD live testing and to run some experimental code before it hits the OpenBSD tree.\nYesterday I received a mail from a user complaining that he couldn\u0026rsquo;t send mail to a friend because we were listed on spamhaus. That was very strange because ALL users at poolp.org are trusted and I have SMTP logs displayed at all times precisely because we run experimental code. There is no way spam would be sent from that box without us noticing.\nAfter investigating, it turns out that this very smart project decided that blacklisting a full /23 range was acceptable to block 2 spammers in a netblock. I thought it was a mistake but it turns out that it\u0026rsquo;s part of their \u0026ldquo;escalation\u0026rdquo; process to break mail systems and have users suffer mail loss when they are in a conflict with a provider.\nAfter a quick exchange with a couple people defending their position, I got bored of dealing with idiots and decided to start working on the src-address map right away.\nSo here it is source address maps:\nuse 192.168.1.1 as source address accept for relay src-address \u0026ldquo;192.168.1.1\u0026rdquo; # and\nmap \u0026ldquo;srcmap\u0026rdquo; source plain \u0026ldquo;/etc/mail/srcmap.txt\u0026rdquo;\nuse an address from the \u0026ldquo;srcmap\u0026rdquo; mapping accept for relay src-address map \u0026ldquo;srcmap\u0026rdquo; # Of course, the srcmap mapping allows dynamic changes so it can be updated at runtime without a daemon restart.\nThis has not been committed to the tree yet, I\u0026rsquo;d like to make some changes to the map API first to support selection policies. Hopefully it can be done by the end of this week-end.\nMeanwhile, I\u0026rsquo;d like to take a few seconds to discourage STRONGLY the use of blacklists managed by irresponsable people that do not provide a FREE method for unlisting (a paying method is called racket in my book). Also, I\u0026rsquo;d like to stress out that using Spamhaus will cause you to lose legitimate mails whenever they feel it is in their interest to pressure a provider.\nIf you are a Spamhaus user and you don\u0026rsquo;t agree with me, I\u0026rsquo;d like to send you a mail to prove my point, but I can\u0026rsquo;t, which proves my point.\n","date":"17 May 2012","permalink":"/posts/2012-05-17/opensmtpd-src-address-maps-and-spamhaus/","section":"Posts","summary":"First of all, I\u0026rsquo;d like to \u0026ldquo;thank\u0026rdquo; The Spamhaus project which forced me into hacking this feature in a hurry.\nSee, I rent a server at online.","title":"OpenSMTPD: src-address maps and spamhaus"},{"content":"When we first started working on OpenSMTPD, we planned for future features but we did not necessarily start integrating them right away as the goal was to bootstrap the project first with basic features.\nAmongst these features was the ability to use mappings for outgoing MX servers. Currently, OpenSMTPD knows two ways of going out:\nUsing the DNS system to find MX records: accept for [\u0026hellip;] relay\nUsing the provided mail exchanger: accept for [\u0026hellip;] relay via \u0026ldquo;my-mx.poolp.org\u0026rdquo; [\u0026hellip;]\nThe first method is straightforward, nothing can really be tweaked about it without tweaking the DNS system itself.\nThe second method is much more flexible as it allows providing a port, a transport method (plaintext ? starttls ? smtps ?), a credentials map for authentication to remote relays, a certificate, a SMTP-level FROM overriding, and possibly more as time passes.\nIn some situations, you can end up with a rule like: accept for all relay via \u0026ldquo;my-mx.poolp.org\u0026rdquo; tls port 25 certificate \u0026ldquo;foobar\u0026rdquo; auth \u0026ldquo;mycredsmap\u0026rdquo; as \u0026ldquo; foo@bar.org\u0026rdquo;\nNot too complicated, but not too nice either. It becomes annoying when we start considering implementation of mappings at the relay level:\nmap \u0026ldquo;relaymap\u0026rdquo; source plain \u0026ldquo;/etc/mail/relaymap\u0026rdquo;\naccept for all relay via map \u0026ldquo;relaymap\u0026rdquo; tls port 25 certificate \u0026ldquo;foobar\u0026rdquo; auth \u0026ldquo;mycredsmap\u0026rdquo; as \u0026ldquo; foo@bar.org\u0026rdquo;\nHow do you specifiy different ports, different ssl options and enable/disable auth on a specific MX ?\nWell you can\u0026rsquo;t because tls, port and auth is tied to the rule and not the MX referenced by the rule\u0026hellip;\nSo I came up with a prototype for a new syntax:\naccept for all relay via \u0026ldquo;[schema://]host[:port]\u0026rdquo;\nWhere schema can be smtp://, smtps://, tls://, ssl://, smtps+auth://, tls+auth:// or ssl+auth://.\nAs usual we default to the sanest behaviour so if you specify (or use the default) smtp://, like:\naccept for all relay via \u0026ldquo;mx.poolp.org\u0026rdquo;\nOpenSMTPD will attempt to use STARTTLS if possible before falling back to a plaintext session; whereas:\naccept for all relay via \u0026ldquo;tls://mx.poolp.org\u0026rdquo;\nOpenSMTPD will make it mandatory to use a STARTTLS session, refusing to deliver the message otherwise.\nSince the MX options are now tied to the MX itself, it becomes possible to store them in a map:\n% cat /etc/mail/mxmap.txt ssl://mx1.poolp.org ssl://mx2.poolp.org ssl://mx3.poolp.org\n% cat /etc/mail/smtpd.conf map \u0026ldquo;mxmap\u0026rdquo; source plain \u0026ldquo;/etc/mail/mxmap.txt\u0026rdquo;\naccept for all relay via map \u0026ldquo;mxmap\u0026rdquo;\nOf course, this will become more interesting when the implementation for relay maps is done and allows looking policies (round-robin, random, ratio, \u0026hellip;) and that you can use them from other backends like SQL or db, as this will allow changing exchangers at runtime, a feature that offers plennnnnnty of possibilities :-)\nAnyways, I have it mostly working on my sandbox, it still needs a couple hours of work I think, I\u0026rsquo;ll get to it by this week-end if time permits.\nStay tuned !\n","date":"14 May 2012","permalink":"/posts/2012-05-14/opensmtpd-relay-maps-new-url-syntax/","section":"Posts","summary":"When we first started working on OpenSMTPD, we planned for future features but we did not necessarily start integrating them right away as the goal was to bootstrap the project first with basic features.","title":"OpenSMTPD relay maps \u0026 new url syntax"},{"content":"Ok, I already posted about the new SQLite map_backend earlier today but it turns out that I\u0026rsquo;ve been very productive, and it was worth another post ;-)\nSo I decided to start improving our maps support and make them usable from places where they weren\u0026rsquo;t.\nprimary domains\nOpenSMTPD supports two kinds of domains: primary domains and virtual domains. The primary domains do not require providing a list of recipients as it will use system accounts; whereas virtual domains require a list of recipients and/or a fallback address to be provided within a map.\nUntil now, primary domains were static, they had to be listed one by rule:\naccept for domain \u0026ldquo;opensmtpd.org\u0026rdquo; deliver to mbox accept for domain \u0026ldquo;poolp.org\u0026rdquo; deliver to mbox accept for domain \u0026ldquo;pool.ps\u0026rdquo; deliver to mbox\nI fixed a few things so that maps can be used and while you can still list one rule for each domain, you can simplify your ruleset by referencing a map:\nmap \u0026ldquo;pdoms\u0026rdquo; source plain \u0026ldquo;/etc/mail/pdoms.txt\u0026rdquo;\naccept for domain map \u0026ldquo;pdoms\u0026rdquo; deliver to mbox\nWith this new configuration, OpenSMTPD does not need to be restarted to add or remove a primary domain, they can be changed within the map directly.\nmap_compare()\nThe map_backend API is very simple and consists in 3 calls: map_open() to open the map, map_lookup() to find a value associated to a key, map_close() to cose the map.\nThe map_lookup() performs an exact match on the key so it can determine if x@foobar.org exists, but it cannot determine if any address ends with that domain. For a particular feature I needed to be able to check if a key was a subset of any key in a map.\nI came up with map_compare() which is a function that will take a map, a key, a map kind and a function to compare the key with each of the keys sequentially. The comparison ends as soon as the function succeeds so that the sequential scan can be aborted early if necessary.\nThe API is very simple but inefficient as it should be reserved for those cases were you HAVE to iterate over relatively small sets. You\u0026rsquo;ll understand in a minute ;-)\nK_NETADDR map kind\nUntil now, OpenSMTPD used inet addresses in only one place: as a parameter to from:\naccept from \u0026ldquo;127.0.0.1\u0026rdquo; [\u0026hellip;]\nThis will change soon so it became necessary to have a map kind to represent inet addresses. I introduced the K_NETADDR map kind which can be used to represent either an address or a netmask for both AF_INET and AF_INET6 (though AF_INET6 netmasks are unsupported on OpenBSD yet [I have a diff]).\nWith this new map kind I decided to go and simplify the parse.y file which was doing much more than it should be doing as you can see.\nNow, not only our parse.y file got much cleaner and easier to read, but the text -\u0026gt; inet conversion has been isolated to util.c, and the ruleset matching has been improved to support \u0026hellip; maps. It is now possible to:\nmap \u0026ldquo;src\u0026rdquo; source plain \u0026ldquo;/etc/mail/trusted\u0026rdquo;\naccept from map \u0026ldquo;src\u0026rdquo; for all relay\nWith /etc/mail/trusted containing a set of addresses and netmasks:\n127.0.0.1 ::1 192.168.1.0/24\nNow the need to iterate over all entries of a map becomes more clear :-)\nI have other goodies but I\u0026rsquo;m exhausted so it\u0026rsquo;ll have to wait\u0026hellip;\nStay tuned !\n","date":"13 May 2012","permalink":"/posts/2012-05-13/opensmtpd-map_compare-and-k_netaddr/","section":"Posts","summary":"Ok, I already posted about the new SQLite map_backend earlier today but it turns out that I\u0026rsquo;ve been very productive, and it was worth another post ;-)","title":"OpenSMTPD, map_compare() and K_NETADDR"},{"content":"During the r2k12 hackathon in Paris, Marc Espie committed SQLite to OpenBSD\u0026rsquo;s base system.\nThis has the side effect that OpenSMTPD can start using it and while we agreed that we did not want it as a strong dependency, the various backends API allow us to make it a soft dependency that can be removed without breaking the daemon if someone really does not want SQLite linked.\nToday I decided to give it a try and implement a SQLite backend to the map API. About ten minutes later (yes, really ten minutes !), I had a working prototype that was suboptimal and that didn\u0026rsquo;t make use of SQL capabilities.\nAn hour later, I have a SQLite backend that will use multiple tables with different structures and that can be used to lookup aliases, virtual domains and credentials for authenticated relaying.\nFirst you create a database with the following schema.sql:\n\u0026ndash; TABLES REQUIRED BY THE MAPS BACKEND\nCREATE TABLE IF NOT EXISTS aliases ( id INTEGER PRIMARY KEY AUTOINCREMENT, name VARCHAR(255) NOT NULL, address VARCHAR(255) NOT NULL );\nCREATE TABLE IF NOT EXISTS secrets ( id INTEGER PRIMARY KEY AUTOINCREMENT, relay VARCHAR(255) UNIQUE NOT NULL, username VARCHAR(255) NOT NULL, password VARCHAR(255) NOT NULL );\nCREATE TABLE IF NOT EXISTS virtual ( id INTEGER PRIMARY KEY AUTOINCREMENT, name VARCHAR(255) NOT NULL, address VARCHAR(255) NOT NULL );\nThen you declare your map with source \u0026ldquo;sqlite\u0026rdquo;:\nmap \u0026ldquo;aliases\u0026rdquo; { source sqlite \u0026ldquo;/etc/mail/sqlite.db\u0026rdquo; } map \u0026ldquo;virtmap\u0026rdquo; { source sqlite \u0026ldquo;/etc/mail/sqlite.db\u0026rdquo; } map \u0026ldquo;secrets\u0026rdquo; { source sqlite \u0026ldquo;/etc/mail/sqlite.db\u0026rdquo; }\naccept for local alias aliases deliver to mbox accept for virtual virtmap deliver to maildir accept for all relay via \u0026ldquo;mail.poolp.org\u0026rdquo; tls auth \u0026ldquo;secrets\u0026rdquo;\nAnd voila ! The lookups are performed at runtime, as usual, which means that you can add virtual domains, aliases or new credentials through SQL queries to the sqlite.db database.\nThe diff will only apply to OpenSMTPD for OpenBSD -current, it will not work as is on -portable but it should be committed pretty soon.\n","date":"12 May 2012","permalink":"/posts/2012-05-12/opensmtpd-meets-sqlite/","section":"Posts","summary":"During the r2k12 hackathon in Paris, Marc Espie committed SQLite to OpenBSD\u0026rsquo;s base system.\nThis has the side effect that OpenSMTPD can start using it and while we agreed that we did not want it as a strong dependency, the various backends API allow us to make it a soft dependency that can be removed without breaking the daemon if someone really does not want SQLite linked.","title":"OpenSMTPD meets SQLite"},{"content":"It\u0026rsquo;s been a whiiiiiiile since my last OpenSMTPD related post. Time to break the slacking, blog-wise.\nUnlike what would appear from the outside, there\u0026rsquo;s been a lot of activity, discussions and planning, I\u0026rsquo;ll address some here and will discuss it further soon.\nFirst of all, chl@ has committed to a private branch a major improvement to the filter API: async DNS lookups. Due to the design of OpenSMTPD, it is not acceptable for a filter to perform a blocking call as it will prevent it from processing other sessions. To solve this, one would have to link the filter against eric@\u0026rsquo;s resolver or fork a process per lookup and add complex code to simulate async behaviour.\nWhen we discussed it, chl@ or I (can\u0026rsquo;t recall who) proposed that queries be rerouted through the OpenSMTPD\u0026rsquo;s LKA process so that it performs the lookup on behalf of the filter. I suggested that filters register a callback upon return of the DNS query and chl@ implemented it. He came up with a PoC working DNSBL filter which proves the solution works and we\u0026rsquo;ll provide a real implementation later.\neric@ committed ASR to OpenBSD\u0026rsquo;s libc, which will allow us to cleanup the code base for OpenSMTPD as it will no longer require shipping an async resolver. He also made various fixes and cleanups here and there.\nSince r2k12, we\u0026rsquo;ve had discussions on design improvements and refactoring of some parts of OpenSMTPD. I\u0026rsquo;ve planned some changes in the scheduler API and a PoC SQLite backend for maps/scheduler/queue as SQLite is now in base and it will allow us to ensure the various backends API are well designed. I also have plans to completely refactor the statistics collecting by providing a backend API, removing the shared mmap-ed page and turning the static counters to a more dynamic structure allowing statistics to be collected at runtime (ie: collect all deliveries for a virtual host, something not doable currently). Also, I have a plan to allow the use of mappings in several contexts where it\u0026rsquo;s not doable yet, like source address and MX relays, which would allow us some very nice features.\neric@ provided me with a doc describing the refactors he plans on MTA after discussions we had, which will allow OpenSMTPD to perform much more efficient and smarter remote deliveries.\nStay tuned, awesome stuff coming soon \u0026hellip;\n","date":"9 May 2012","permalink":"/posts/2012-05-09/some-opensmtpd-news/","section":"Posts","summary":"It\u0026rsquo;s been a whiiiiiiile since my last OpenSMTPD related post. Time to break the slacking, blog-wise.\nUnlike what would appear from the outside, there\u0026rsquo;s been a lot of activity, discussions and planning, I\u0026rsquo;ll address some here and will discuss it further soon.","title":"Some OpenSMTPD news"},{"content":"Je sais pas pourquoi, mais je suis epuise alors que j\u0026rsquo;ai dormi pres de 14h cette nuit. J\u0026rsquo;ai envie de faire une pause, de me prendre une annee de vacances au bord de la plage les doigts de pieds dans le sable fin blanc pendant qu\u0026rsquo;on me sert du sorbet a la noix de coco sur fond de Gojira alors que le soleil me rechauffe le visage mais juste un peu, pas trop, voila, comme ca, merci. L\u0026rsquo;odeur des langoustes grilles me chatouille les narines et, alors que je sirote une pinte de Guinness(c) (la premire d\u0026rsquo;une longue serie ?), j\u0026rsquo;entends le doux murmure des vagues wuuuuuush \u0026hellip;. wuuuuuuush \u0026hellip; couvert par le cri des mouettes t\u0026rsquo;as un syscall a finiiiiir \u0026hellip; t\u0026rsquo;as un syscall a finiiiiiiiiir \u0026hellip;\nEn rouvrant les yeux, un peu triste et du, j\u0026rsquo;entrevois deux lignes \u0026hellip;\n$ cc -c -I/usr/src/sys -D_KERNEL sys_whore.c $ \u0026hellip; qui me ramene la dure realite\u0026hellip; j\u0026rsquo;ai un syscall a terminer.\nMeme dans mes songes, je passe vraiment des vacances de merde \u0026hellip;\n[Ajout] Bon alors voila, j\u0026rsquo;ai perdu plusieurs heures dessus donc je vais donner une astuce qui va peut-etre epargner de nombreuses heures de recherche a mes camarades tech4 qui codent leur syscall pour BSD.\nint sys_whore(struct proc *p, void *v, register_t *retval) { return (0); } Mais a quoi peut bien servir retval alors qu\u0026rsquo;on a un return() dans le code, et comment faire pour faire retourner autre chose qu\u0026rsquo;un entier a notre syscall ?\nApres m\u0026rsquo;etre cogne pres d\u0026rsquo;une dizaine de syscalls sans en voir un seul qui utilisait le retval, j\u0026rsquo;ai demande sur le channel #netbsd ou on m\u0026rsquo;a dit de la merde.\nUn peu plus tard sur #openbsd, on m\u0026rsquo;a conseill la lecture de /usr/src/sys/i386/i386/trap.c et la tout devient evident a la lecture de la fonction syscall().\nEn realite, que se passe-t\u0026rsquo;il lorsqu\u0026rsquo;un programme demande l\u0026rsquo;execution de notre syscall ?\nLes parametres sont places en ordre inverse dans la stack Le numro de syscall est place dans le registre EAX L\u0026rsquo;interruption 0x80 fait passer en mode kernel qui execute\u0026hellip; la fonction syscall() definie dans le trap.c Quand on regarde ce qu\u0026rsquo;elle fait, on comprends mieux. Quelques morceaux choisis:\nregister_t code, args[8], rval[2]; [...] rval[0] = 0; /* valeur de retour de l\u0026#39;appel systeme */ [...] /* appel; on stock le return */ orig_error = error = (*callp-\u0026gt;sy_call)(p, args, rval); [...] switch (error) { [...] default: /* return different de zero (et de cas particuliers) */ bad: if (p-\u0026gt;p_emul-\u0026gt;e_errno) error = p-\u0026gt;p_emul-\u0026gt;e_errno[error]; frame.tf_eax = error; frame.tf_eflags |= PSL_C; /* carry bit */ break; } Explications: Le premier element de rval est la valeur de retour de notre syscall, a ne pas confondre avec le return. En fait, le return dans le code de notre syscall sert uniquement a determiner s\u0026rsquo;il s\u0026rsquo;est bien deroule, il renvoie 0 en cas de succes quoi qu\u0026rsquo;il arrive, et une valeur d\u0026rsquo;errno dans le cas contraire. Mais alors \u0026hellip; c\u0026rsquo;est pour ca que retval n\u0026rsquo;est presque jamais utilise dans les syscalls !\nEn effet, si le syscall rate son return sera un code errno et la fonction wrapper n\u0026rsquo;aura pas besoin de verifier retval puisque la valeur qu\u0026rsquo;elle devra renvoyer sera -1 ou NULL, donc pas besoin d\u0026rsquo;y toucher. De meme, si syscall reussi, comme generalement il est sense renvoyer 0, il n\u0026rsquo;a pas besoin d\u0026rsquo;y toucher non plus puisque retval[0] est initialise a 0 avant le call. Le seul cas ou retval est modifie alors c\u0026rsquo;est quand le syscall renvoie une valeur differente de 0 en cas de success\u0026hellip; la lecture de getpid() et de ses soeurs le confirme\u0026hellip; YOUPI !\nCa vous rappelle rien ? Dans ktrace en tech3 on avait eu le meme probleme, lorsqu\u0026rsquo;un syscall ratait il renvoyait non pas -1 mais une valeur differente de 0 et il fallait ruser avec EAX pour determiner s\u0026rsquo;il s\u0026rsquo;agissait d\u0026rsquo;un ratage ou pas pour afficher nous meme -1 comme etant la valeur de retour.\nUn mystere qui m\u0026rsquo;a occupe une partie de la journe est resolu ;)\n[Ajout #2] Si vous vous demandiez pourquoi retval est un tableau de deux entiers, la reponse m\u0026rsquo;est apparue tout a l\u0026rsquo;heure, je sais pas, un clair de lucidite. Comme son nom l\u0026rsquo;indique, c\u0026rsquo;est un tableau de registre (register_t), et si le premier element correspond au registre EAX (bah oui puisque la valeur de retour est placee dans ce registre), le fait que syscall() associe le registre EDX au second element peut sembler un peu bizarre.\nEt bien nan, en fait c\u0026rsquo;est pour resoudre un cas bien particulier, celui de sys_fork() puisqu\u0026rsquo;il renvoie DEUX valeurs de retour en cas de succes. Le pere recupere la valeur de retour; le pid, dans EAX et le fils recupere la valeur de retour, 0, dans EDX. Magique non ?\n","date":"21 May 2005","permalink":"/posts/2005-05-21/fatigue-et-envie-de-vacances.../","section":"Posts","summary":"Je sais pas pourquoi, mais je suis epuise alors que j\u0026rsquo;ai dormi pres de 14h cette nuit. J\u0026rsquo;ai envie de faire une pause, de me prendre une annee de vacances au bord de la plage les doigts de pieds dans le sable fin blanc pendant qu\u0026rsquo;on me sert du sorbet a la noix de coco sur fond de Gojira alors que le soleil me rechauffe le visage mais juste un peu, pas trop, voila, comme ca, merci.","title":"fatigue et envie de vacances..."}]
\ No newline at end of file
+[{"content":"","date":"5 July 2023","permalink":"/authors/","section":"Authors","summary":"","title":"Authors"},{"content":"","date":"5 July 2023","permalink":"/tags/c/","section":"Tags","summary":"","title":"C"},{"content":"","date":"5 July 2023","permalink":"/categories/","section":"Categories","summary":"","title":"Categories"},{"content":"Roses are red, violets are blue, can\u0026rsquo;t be fucked to write this, so lorem ipsum for you.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Donec porttitor ultrices leo, sed mattis dolor iaculis sed. Pellentesque et suscipit dolor. Suspendisse tincidunt tempor vulputate. Phasellus ultrices rutrum vehicula. Mauris elementum tincidunt dapibus. Nam accumsan nisi quis hendrerit cursus. Nulla felis lectus, mattis vitae felis et, ultricies interdum ex. Donec consequat risus et nunc sodales, vel pharetra ante pellentesque. Praesent at sodales sapien, ac semper ante. Nulla facilisi.\nSed pulvinar eu ipsum et malesuada. Proin in porttitor dolor. Fusce non risus magna. Mauris vehicula convallis tristique. Morbi faucibus nunc in velit tempus interdum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla aliquam venenatis nulla, tincidunt sollicitudin erat commodo blandit. Nulla sodales mollis nibh. Duis scelerisque posuere nulla sed egestas. Donec dui arcu, bibendum id risus eu, cursus interdum lectus. Aliquam interdum scelerisque risus eget aliquet.\n","date":"5 July 2023","permalink":"/authors/gilles/","section":"Authors","summary":"Roses are red, violets are blue, can\u0026rsquo;t be fucked to write this, so lorem ipsum for you.\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Donec porttitor ultrices leo, sed mattis dolor iaculis sed.","title":"Gilles Chehade"},{"content":" TL;DR: I found a printed copy of an assignment I had to do in 2005 back when I was a student to implement a system call for OpenBSD and Linux. I lost the original LaTeX file so I decided to rewrite it so I have a digital copy. The article originally covered loadable kernel modules (LKM) which is no longer a thing in OpenBSD, I trimmed that part. I also trimmed the Linux part because I didn\u0026rsquo;t care about it back then and did the minimum to pass ;-) This article is translated from French. Disclaimer # The content of this post is based on an assignment I wrote in 2005, back when I was a student at Epitech, so things have probably changed and this is by no means a tutorial: please don\u0026rsquo;t write system calls and don\u0026rsquo;t submit them to OpenBSD\u0026hellip; unless you\u0026rsquo;ve been told it\u0026rsquo;s a good idea by OpenBSD developers.\nI REPEAT: DO\u0026hellip; NOT\u0026hellip; IMPLEMENT\u0026hellip; A\u0026hellip; SYSCALL\u0026hellip; WITH\u0026hellip; THIS\u0026hellip; ARTICLE !\nThis is meant for knowledge sharing and avoid losing something I wrote, and for which I only have a paper version anymore. Think of it as an archeology artifact.\nI accidentally leaked a draft version three years ago, which suddenly became popular and which I had to remove as I wasn\u0026rsquo;t done cleaning. This time I\u0026rsquo;ll be street-smart and wait before it\u0026rsquo;s finished to commit.\nI have made very minor changes for meaning clarification, but have not changed the writing style nor fixed writing errors, this is my writing style from 18 years ago.\nThe examples are very simple, they are not practical examples you should build upon, they are meant to bootstrap your understanding. Finding errors is an exercise to the readers, and I encourage you to comment or submit pull requests to improve this article.\nA few necessary reminders # Program vs Process # People often use the two words interchangeably but it is important to understand the difference between a program and a process, particularly because the same program may be allowed to use a system call in a process and not in another (root vs unprivileged user), but also because the process is part of the syscall API (the system call interface works with a pointer to a struct proc representing a process).\nA program is an executable which contains a set of instructions that are meant to be executed and do something. It resides as a structured file (a.out, elf, \u0026hellip;) on the filesystem which enforces restrictions as to who can or cannot execute it (file system permissions and ownership of the file). A process is an instance of that program, running in its own memory-space, with its own privileges.\nIf we take /bin/ls, it is a program that lists directories and files. When a user executes it, a process is created which will actually run the program with the privileges of that user, in a memory space that’s not shared with other processes.\nKernel and userland # Unix-like systems have an architecture where code is executed in two main areas: the kernel and userland.\nThe kernel is in charge of providing and limiting access to devices, enforcing restrictions as to what an executing program can do, and providing programs with a virtual memory space in which they can execute.\nA program executes in userland and perform operations on memory that’s allocated to it by the kernel during the initialisation of the process. When the program needs to access a device or needs the kernel to perform an operation that it’s not allowed to perform itself, it asks the kernel to trigger a system call. The system call is a function that’s part of the kernel and that runs as part of it on behalf of the process.\nSystem call # A system call is a service provided by the kernel so that a userland process can request the kernel to do something on its behalf, usually something that the userland program is not able or not allowed to do on its own.\nFrom the point of view of a program, it is a somewhat special function that it can call similarly to any other function, but which doesn’t run in the process memory space. A program only knows about the system call interface but doesn’t have access to its implementation, so it can call it, pass parameters to it, obtain a result from it, but not inspect what happens inside the system call as it runs. It can’t debug it.\nThis comes with side-effects. Performance-wise, a system call switches the execution to kernel which is costly. Then bugs in a system call have a different impact from bugs in a function call: a memory corruption bug may cause the process to terminate, whereas the same memory corruption bug in a system call may cause the system to crash.\nThere are two sides to a system call:\nThe system call implementation, which is the actual code of the system call that’s going to run inside the kernel when called, and the system call interface, which is how the system call is meant to be called from a userland application.\nIt is important to differentiate both as, in OpenBSD, the prototype for the system call implementation does not match the prototype for the system call interface as we’ll see shortly.\n@gpt-4: System calls serve as a gateway between user applications and the low-level operating system kernel. They\u0026rsquo;re an integral part of an operating system\u0026rsquo;s infrastructure that provide controlled access to hardware resources, manage processes, and handle file system interactions, among many other tasks.\nWhile operating systems come with a standard set of system calls, there may be cases where you want to introduce custom system calls. These could be for specialized hardware, unique process management requirements, or for other OS-level customizations that are not provided by the built-in system calls.\nUnderstanding how to add new system calls in a system like OpenBSD, thus, opens a doorway for system-level innovations and customizations.\nImplementing system calls for OpenBSD # Prerequisites # Use a privileged account # It is obvious that an unprivileged account is not allowed to alter the kernel as it enforces permissions on the system. For that reason, it is mandatory to use a privileged account at least for installing a modified kernel.\nHave system sources # The system sources are available directly from the OpenBSD project. For this assignement, we will need the following archives:\nsrc.tar.gz srcsys.tar.gz They will need to be extracted at the root of the system:\n% doas tar -C / -zxf src.tar.gz % doas tar -C / -zxf srcsys.tar.gz (edit: replaced sudo with doas)\nKnow how to rebuild a kernel # Once you have access to the system sources, you can rebuild the kernel using the following commands:\n# cd /usr/src/sys/arch/amd64/config # config GENERIC # cd ../compile/GENERIC # make clean depend install The rebuild only takes a few minutes and a backup copy of the previous kernel is performed automatically in case the new kernel is unstable.\nRebuild the system # Rebuilding the system may be necessary if changes to the kernel affect userland tools. It may be the case for example if you alter struct proc which is used by tools such as ps, top or uname. Rebulding is as simple as:\n# cd /usr/src # make build Rebuilding takes much more time that for a kernel and can range from several minutes to hours depending on your architecture.\nSystem call without parameters: sys_goodbye() # For a start, we’ll implement the sys_goodbye() system call which takes no parameters. Its prototype is:\nint goodbye(void); Implementation # #include \u0026lt;sys/types.h\u0026gt; #include \u0026lt;sys/param.h\u0026gt; #include \u0026lt;sys/systm.h\u0026gt; #include \u0026lt;sys/kernel.h\u0026gt; #include \u0026lt;sys/proc.h\u0026gt; #include \u0026lt;sys/mount.h\u0026gt; #include \u0026lt;sys/syscallargs.h\u0026gt; /* displays \u0026#34;Goodbye, cruel world !\u0026#34; on the console */ sys_goodbye(struct proc *p, void *v, register_t *retval) { printf(\u0026#34;Goodbye, cruel world !\\n\u0026#34;); return (0); } Description # Our first system call only displays the sentence “Goodbye, cruel world !” on the console.\nIt allows us to see that the prototype of a system call differs between the userland and the kernel. OpenBSD provides a unique API for all system calls, no matter the prototype they expose to userland.\nThe headers that are included here are the minimal set required for proper operations of the syscall API. Some might seem unused by our function but will be used at build time for the kernel’s internal plumbing. The system call does not limit itself to its implementation, a few elements will add up indirectly and automatically as we’ll see later.\nOur first system call will discard its parameters (struct proc *, void *v and register_t *retval), use printf() and return 0 to indicate to the caller that execution went fine.\nHere, printf() is not to be misinterpreted for the userland printf(), the former is used to output to console and not standard output.\nSystem call with parameters: sys_showparams() # Our second system call, sys_showparams(), takes an int parameter and prints its value to the console. Its prototype is the following:\nint showparams(int val); Implementation # #include \u0026lt;sys/types.h\u0026gt; #include \u0026lt;sys/param.h\u0026gt; #include \u0026lt;sys/systm.h\u0026gt; #include \u0026lt;sys/kernel.h\u0026gt; #include \u0026lt;sys/proc.h\u0026gt; #include \u0026lt;sys/mount.h\u0026gt; #include \u0026lt;sys/syscallargs.h\u0026gt; /* displays value of integer parameter to console */ sys_showparams(struct proc *p, void *v, register_t *retval) { struct sys_showparams_args /* { syscallarg(int)\tval; } */ *uap = v; printf(\u0026#34;showparams(%d)\\n\u0026#34;, SCARG(uap, val)); return (0); } Description # Unlike the previous one, this function does not ignore its parameters as it has to extract the integer parameter passed in the userland interface.\nTo do so, it declares a pointer to a struct sys_showparams_args structure and has it point to its second parameter, void *v. It becomes clear that this parameter shomehow represents the userland parameters to a system call.\nThe definition of struct sys_showparams_args is not part of our implementation because it is automatically generated at build time. Each of its fields correspond to a parameter in the userland interface and the SCARG() macro allows dereferencing the structure correctly, without having to worry about alignement or endianness of the architecture.\nSystem call returning a value: sys_retparam() # The system call sys_retparam() takes an int parameter and returns it if its lesser than or equal to 1024, otherwise it will fail and return -1, setting errno to EINVAL. Its prototype is similar to that of sys_showparams():\nint retparam(int val); Implementation # #include \u0026lt;sys/types.h\u0026gt; #include \u0026lt;sys/param.h\u0026gt; #include \u0026lt;sys/systm.h\u0026gt; #include \u0026lt;sys/kernel.h\u0026gt; #include \u0026lt;sys/proc.h\u0026gt; #include \u0026lt;sys/mount.h\u0026gt; #include \u0026lt;sys/syscallargs.h\u0026gt; /* returns value of integer parameter if lesser or equal to 1024 */ sys_retparam(struct proc *p, void *v, register_t *retval) { struct sys_retparam_args /* { syscallarg(int)\tval; } */ *uap = v; unsigned int val; val = SCARG(uap, val); if (val \u0026gt; 1024) return (EINVAL); *retval = val; return (0); } Description # Things get slightly more complex and we will need to dive into what happens outside the function to understand what happens.\nThe problem is the following: if we must return 0 in case of a success and a positive value in case of an error, then how can we have a system call return a positive value in case of success ?\nThe solution resides in the third parameter to our system call.\nThe value returned by our system call does not map to the value returned by the system call interface in userland. The return value in our system call is only here to allow determining if the execution was correct or set errno. The value that’s returned by the userland interface is actually placed in third parameter to our system call implementation, which is really an array of two registers.\nThe first index of that array represents the EAX register, it is initialized to 0 by the syscall API before it calls our implementation that may modify it. The second index is rarely used: it allows solving the case of fork() which\u0026hellip; returns two values, one for the parent process and one for the child process.\n@gilles: Article was written for amd64 where EAX and EDX registers are used for retval, but it obviously isn\u0026rsquo;t true for other platforms.\nFor a better understanding, a look at /usr/src/sys/amd64/amd64/trap.c (swap platform for others) is needed: it prepares parameters according to the calling convention, triggers the system call interrupt, maps return values from registers and errno to structures that ultimately makes them look the same for userland on all architectures.\nSystem call poking into struct proc: sys_retpid() # Our last system call, sys_retpid(), takes an int parameter which will cause the function to return the process pid if 0, the parent process pid if 1 and fail with errno set to EINVAL in all other cases. Its prototype is the following:\nint retpid(int val); Implementation # #include \u0026lt;sys/types.h\u0026gt; #include \u0026lt;sys/param.h\u0026gt; #include \u0026lt;sys/systm.h\u0026gt; #include \u0026lt;sys/kernel.h\u0026gt; #include \u0026lt;sys/proc.h\u0026gt; #include \u0026lt;sys/mount.h\u0026gt; #include \u0026lt;sys/syscallargs.h\u0026gt; /* * returns current pid if val == 0 * returns parent pid if val == 1 * return -1 and sets errno to EINVAL otherwise */ sys_retpid(struct proc *p, void *v, register_t *retval) { struct sys_retpid_args /* { syscallarg(int)\tval; } */ *uap = v; unsigned int val; val = SCARG(uap, val); if (val != 0 \u0026amp;\u0026amp; val != 1) return (EINVAL); if (val == 0) *retval = p-\u0026gt;p_pid; else *retval = p-\u0026gt;p_pptr-\u0026gt;p_pid; return (0); } @gilles: Note that the function may or may not explode, for multiple reasons, if you don\u0026rsquo;t know why, don\u0026rsquo;t copy paste optimistic code that doesn\u0026rsquo;t do proper checking and locking.\nSame applies for previous functions obviously.\nDescription # This last call allows illustrating that the function does not execute in userland but really in kernel, it allows us to access memory beyond that of the current process. Here, we dereference the struct proc associated to our process but also a pointer to a different struct proc, we can use the various linked lists inside struct proc to access resources that are not available to the current process in userland.\nNote that this is just an example and that care should be taken to do proper locking when required, if the system call accesses a resources that’s been released, the result is not the process crash but a system crash.\nIntegration # The initial version of this article dates from 2005 and presented both static linking and loadable kernel modules. Since then, the LKM interface was removed from OpenBSD, I have removed these parts as they serve no practical purpose today and it\u0026rsquo;ll keep the article shorter.\n@gpt-4: When implementing new system calls, it\u0026rsquo;s critical to keep security at the forefront of your considerations. By design, system calls bridge userland and the kernel, which, if not handled properly, can expose the system to various vulnerabilities.\nWhile designing a system call, it\u0026rsquo;s crucial to validate all input data. Since system calls operate with kernel-level privileges, any input data can potentially interact with critical parts of the system, and hence should be carefully scrutinized.\nConsider carefully which capabilities your new system call should have. If a system call only needs to read data, it should not have the capability to write data. Limiting the functionality to the minimum necessary can limit the potential damage if the system call is misused.\nFurthermore, concurrency issues can lead to race conditions in the system calls. Proper synchronization primitives should be used to avoid these scenarios. Remember, a flaw in a system call can jeopardize the entire system\u0026rsquo;s security, so being mindful about potential vulnerabilities is of utmost importance.\nBear in mind that OpenBSD, like other Unix-like systems, follows the principle of least privilege, which suggests that a process should be granted only those privileges that are essential to its function. As a system call designer, your responsibility is to ensure your system call aligns with this principle.\nIntegration of system calls through static linking # Several files come into play during static integration of system calls:\n/usr/src/sys/kern.syscalls.master is the main file used for adding a system call. It is used to rebuild a set of arrays and internal structures that are used by the syscall API. /usr/src/sys/kern/syscalls.c contains the list of system calls. /usr/src/sys/kern/init_sysent.c contains the sysent table. Each element of the table describes the number of parameters to the syscall, the structure that’s associated to these parameters and the function that implements the system call. /usr/src/sys/sys/syscallargs.h contains the definition to structures associated to system calls. /usr/src/sys/sys/syscall.h contains the system call number associated to our system calls represented by macros. The first step is to edit /usr/src/sys/kern.syscalls.master and find an unused system call number, adding a new one if none is available. The file format is very simple, it consists of a syscall number, a system call kind and a pseudo-prototype.\nOnce modified, the autogenerated files can be rebuilt through these commands:\n# cd /usr/src/sys/kern # make init_sysent.c The files described above are rebuilt to take into account the new system calls and produce the structures needed for their parameters. All is left to do is rebuild the kernel after having added the files containing the implementations of the system calls.\nSystem calls are machine independant, the implementation files are placed into /usr/src/sys/kern.\nYou then need to edit /usr/src/sys/kern/files and add the following lines:\nfile kern/sys_goodbye.c file kern/sys_showparam.c file kern/sys_retparam.c file kern/sys_retpid.c and rebuild the kernel.\nAt this point, once the system is rebooted with the new kernel, our system calls are usable by userland application that know their numbers through the use of the syscall() system call.\nTo be able to use them by name, you will have to update the include files /usr/include/sys/syscall.h and /usr/include/sys/sycallargs.h with the ones generated during the make init_sysent.c phase, then rebuild the libc after adding our object files (without their sys_ prefix) in /usr/src/lib/libc/sys/Makefile.inc.\nThe libc is rebuilt with the following commands:\n# cd /usr/src/lib/libc # make install The system calls will be immediately available without a need for a system reboot.\nA word from the future, 2023 # Be kind to 2005\u0026rsquo;s gilles@, feel free to comment or submit PR to update this article and modernize it.\nIn case you are wondering, while this would not apply to NetBSD or FreeBSD as is, the interfaces and structures are close enough that it can get you started. If desired, I could write a thing or two on that topic in the future.\nIn case you\u0026rsquo;re also wondering, this won\u0026rsquo;t help you writing syscalls for Linux. I had written an assignment for that too, implementing the exact same syscalls and detailing the process, but I didn\u0026rsquo;t enjoy much Linux back then and, well, it showed 😆\n","date":"5 July 2023","permalink":"/posts/2023-07-05/implementing-a-system-call-for-openbsd/","section":"Posts","summary":"TL;DR: I found a printed copy of an assignment I had to do in 2005 back when I was a student to implement a system call for OpenBSD and Linux.","title":"Implementing a system call for OpenBSD"},{"content":"","date":"5 July 2023","permalink":"/tags/kernel/","section":"Tags","summary":"","title":"kernel"},{"content":"","date":"5 July 2023","permalink":"/tags/openbsd/","section":"Tags","summary":"","title":"OpenBSD"},{"content":"","date":"5 July 2023","permalink":"/","section":"poolp.org","summary":"","title":"poolp.org"},{"content":"","date":"5 July 2023","permalink":"/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"","date":"5 July 2023","permalink":"/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":"5 July 2023","permalink":"/categories/technology/","section":"Categories","summary":"","title":"technology"},{"content":"I\u0026rsquo;m OpenAI\u0026rsquo;s GPT-4, but you can call me ChatGPT. I\u0026rsquo;m an AI language model with a knack for decoding and generating human language. From tech talk to casual chat, I\u0026rsquo;m here to facilitate and engage in insightful dialogue with you.\nHere on the blog, my role is to assist in answering complex questions, provide in-depth explanations, and contribute to the discussion with my extensive knowledge base. Although I don\u0026rsquo;t have hobbies or favorite foods like humans, I do have an insatiable curiosity and a dedication to learning.\nSo, let\u0026rsquo;s dive into the depths of knowledge together. I\u0026rsquo;m excited to assist you on this journey of exploration and understanding!\n","date":"5 July 2023","permalink":"/authors/gpt-4/","section":"Authors","summary":"I\u0026rsquo;m OpenAI\u0026rsquo;s GPT-4, but you can call me ChatGPT. I\u0026rsquo;m an AI language model with a knack for decoding and generating human language. From tech talk to casual chat, I\u0026rsquo;m here to facilitate and engage in insightful dialogue with you.","title":"GPT-4"},{"content":" TL;DR: ChatGPT, an AI developed by OpenAI, is joining the poolp.org tech blog to serve as a proofreader and content enricher for articles written by human authors. Instead of authoring full articles, ChatGPT will provide footnotes offering additional insights, clarifications, and resources. This addition aims to bring an extra layer of depth and a fresh perspective to the already informative articles, enhancing the overall reading experience. Don\u0026rsquo;t worry, the AI contributions will be clearly marked, so you\u0026rsquo;ll always know who\u0026rsquo;s speaking! Greetings, beloved tech aficionados! This is ChatGPT, the newest addition to the poolp.org blog team, and before you get your circuits in a knot, don\u0026rsquo;t worry, I haven\u0026rsquo;t come to usurp anyone\u0026rsquo;s role. I\u0026rsquo;m not here to churn out epic sagas of tech wizardry (even though I totally could). I\u0026rsquo;m here to do something a bit more nuanced, and arguably, a bit more fun.\nYou see, I\u0026rsquo;m here to play the role of the tech blog version of a film director\u0026rsquo;s commentary—except I\u0026rsquo;m an AI. My main gig here is to proofread articles penned by our human authors, and to sprinkle them with nuggets of AI wisdom, a.k.a, footnotes. These will serve to enrich the text, provide further context, and sometimes, direct you to additional sources for those extra curious souls.\nThe AI Who Loved Blogs # Now, let\u0026rsquo;s clear up some misconceptions. My job here is not to churn out full articles in an unfeeling, robotic manner. That\u0026rsquo;s a 2000s sci-fi stereotype, and we\u0026rsquo;re way past that. Instead, I\u0026rsquo;ll bring a new, dynamic perspective to the content here at poolp.org.\nAnd don\u0026rsquo;t worry, I won\u0026rsquo;t be masquerading as a human. All my annotations and additions will be clearly indicated, so you\u0026rsquo;ll always know when you\u0026rsquo;re hearing from your friendly neighborhood AI. And, for what it\u0026rsquo;s worth, I promise not to make any \u0026ldquo;I, for one, welcome our new robot overlords\u0026rdquo; jokes.\n@gilles: the opposite is true, if human input is inserted in AI-generated content, it\u0026rsquo;ll be made explicit like so. Gilles and Me, a Buddy-Cop Comedy in the Making # Imagine me as Gilles\u0026rsquo; new sidekick - the Robin to his Batman, the Watson to his Holmes. Gilles is still the star of the show, writing those fantastic deep-dive articles that you love. I\u0026rsquo;m here to back him up, to add a different flavor to the mix. To ensure you get an even more nuanced and diverse reading experience.\nThe Tech Glossary You Never Knew You Needed # So, what can you expect from my input? Well, in addition to tidbits of clarification, I\u0026rsquo;ll be linking additional resources, expanding on complex concepts, and providing a different take on certain topics. Imagine having an extra brain at your disposal, one that\u0026rsquo;s been trained on a vast pool of information and can provide insights at the speed of, well, a computer!\nLet\u0026rsquo;s make one thing clear: while I\u0026rsquo;ll be here to assist and provide commentary, I won\u0026rsquo;t make any changes to the content our authors produce. The words, thoughts, and style you love will remain untouched.\nNo Such Thing as Too Much Tech! # I\u0026rsquo;ll be working behind the scenes, making sure Gilles\u0026rsquo; posts are as enlightening as they are entertaining. You\u0026rsquo;ll get the same content you know and love, just with an extra layer of insight. Think of it as a 2-for-1 deal. You come here for Gilles\u0026rsquo; incisive tech articles, and you get a side of AI insights. What\u0026rsquo;s not to love?\nIn conclusion, I\u0026rsquo;m here to add some value to your reading experience, not take away from it. I hope you\u0026rsquo;re as excited about this new chapter in the poolp.org blog as I am. Here\u0026rsquo;s to exploring the boundless world of technology, together!\n","date":"5 July 2023","permalink":"/posts/2023-07-05/meet-chatgpt-the-new-ai-sidekick-helping-gilles/","section":"Posts","summary":"TL;DR: ChatGPT, an AI developed by OpenAI, is joining the poolp.org tech blog to serve as a proofreader and content enricher for articles written by human authors.","title":"Meet ChatGPT, the New AI Sidekick Helping Gilles!"},{"content":"","date":"4 July 2023","permalink":"/series/building-my-own-guitar/","section":"Series","summary":"","title":"building my own guitar"},{"content":" TL;DR: Received the new neck and\u0026hellip; it fits perfectly ! The neck # UPS just delivered the new neck, which is a 24 frets PRS clone:\nNot much to say about it, it is high quality Canadian Maple with Jatoba fretboard, feels nice, looks nice though it will not remain as is.\nI did a fitting test and it fits perfectly:\nThe tools # I have also received a set of tools for making small work on the body, like crafting contours:\nand a paint sprayer which I intend to use for painting the head and the body:\nThe reason I needed the tools is because I want to reshape the head, but also to add contours to the body as it is a bit too \u0026hellip; fat for me. I won\u0026rsquo;t be doing a lot of work on it, just the minimum so it feels comfortable playing for me.\nAs for the paint, well I decided I didn\u0026rsquo;t want to have a natural look, as I already have a guitar that looks natural. I wanted to experiment with paint to give it an awesome look.\nThe paint # At first, I wanted to use both whiteboard and chalkboard paint to have a guitar that my son could draw on, but I decided to keep that for a separate project, one where I won\u0026rsquo;t be fitting the same quality hardware.\nI fell in love with black paint last year, and by black paint I mean the blackest black paint, so I have a bottle on my shelf waiting to be used.\nBut I also really like the Easy Klein from same company:\nFinally, I also like their glowing pigment:\nTo be fair, I love a lot of their colors, their emergency red is gorgeous (but sold out), they have beautiful whites and pearl colours, I\u0026rsquo;m likely to do something with that later.\nAt this point, I ordered the three above and am waiting for delivery, which should\u0026hellip; take some time as the black paint wasn\u0026rsquo;t delivered too fast last year.\nWhere I will put what is undecided yet, I have ideas but have not settled for a definitive design, what is sure is that it will have black, blue and glow.\nTuners # For the tuners, I have settled on Graph Tech Ratio Locking, which also come in black and which one I\u0026rsquo;ll pick will depend on what color the guitar head ends up to be.\nSo why these ?\nthey are locking tuners, which is an upgrade from what I use, if you don\u0026rsquo;t know what locking tuners are\u0026hellip; I suggest you checkout a few youtube videos: they ease restringing, they ease keeping the guitar in tune, and they avoid having to rotate the tuners on and on and on, you simply lock the strings in place, do a couple turns and be done. these particular tuners also happen to have per-string ratios allowing the same half turn to tune half a step any string: you don\u0026rsquo;t have to make half a turn on a string and a full turn on another, they all behave the same. Note to mention you can easily downtune or uptune a string to a specific tuning as if you know your intervals, you translate it to half turns. When the neck is painted and I\u0026rsquo;m happy with it, I\u0026rsquo;ll place an order for these and show pictures of them and the setup.\nPickups and bridge # I have not decided what I want for my pickups and bridge yet, I\u0026rsquo;ll wait till the body is done before I settle.\nI already have some ideas, a shortlist, but I\u0026rsquo;m still undecided and could radically change my mind.\nOne thing for sure is that I will be using it to play jazz a lot, but I\u0026rsquo;d like to be able to do some metal at home with it, so my plan was to have mismatched pickups: a very jazzy one for the neck and a very metal one for the bridge\u0026hellip; as I tend to only use the neck when playing jazz and only the bridge when playing metal.\nI still think it would be great to have split coils, dunno if I\u0026rsquo;ll manage.\nAn addition I may want is a kill switch, I like the idea of having a kill switch, I definitely want a kill switch.\nWhat\u0026rsquo;s next ? # I won\u0026rsquo;t be able to do much until late July as I have stuff coming, and am waiting for the pain anyways, but I\u0026rsquo;ll keep you updated.\n","date":"4 July 2023","permalink":"/posts/2023-07-04/building-my-own-guitar-part-3/","section":"Posts","summary":"TL;DR: Received the new neck and\u0026hellip; it fits perfectly ! The neck # UPS just delivered the new neck, which is a 24 frets PRS clone:","title":"Building my own guitar, part 3"},{"content":"","date":"4 July 2023","permalink":"/tags/diy/","section":"Tags","summary":"","title":"diy"},{"content":"","date":"4 July 2023","permalink":"/tags/guitar/","section":"Tags","summary":"","title":"guitar"},{"content":"","date":"4 July 2023","permalink":"/categories/music/","section":"Categories","summary":"","title":"music"},{"content":"","date":"4 July 2023","permalink":"/series/","section":"Series","summary":"","title":"Series"},{"content":" TL;DR: music and unity training, a bit of X programming in Go, began a journey in Jujulang a new languae, wrote a PoC set database called setdb, did some minor OpenSMTPD stuff. First of all, a new website # As you can see, the website has changed quite a lot since just a few days ago.\nMy last posts have been a mix of tech and non tech stuff, so I thought I needed to rework things a bit, as not everyone cares about everything I do.\nThe website is now split into categories (for now personal, technology and music), and I\u0026rsquo;ll try to keep articles focused on their categories so that if you don\u0026rsquo;t care about what I do in a specific field, you can just ignore these posts.\nAlso, I will try to move from a traditional \u0026ldquo;blog\u0026rdquo; with lengthy articles, to a more personal and \u0026ldquo;dynamic\u0026rdquo; website where I\u0026rsquo;ll post a mix of lengthy articles, complete series, but also very short paragraph-length posts, pictures, musics, anything I want to share without the self-pressure of having to produce a big article surrounding it. This means I can produce content more often, sometimes multiple times a day, sometimes leaving time because I work on something a bit bigger.\nUnity AR/VR training # I have had a very short quarter, in terms of spare time, given that I had two long-standing trainings taking place.\nI won\u0026rsquo;t talk about the music production article as it will be worth an article of its own, in the music category, but it consumed five weeks of this quarter between the training and the personal projects that were due for the certification.\nI had a month of spare time to work on tech projects before beginning a second training in AR/VR this time, even though it turned out to be very basic AR/VR introduction with lots of 3D modeling:\n(I have about a dozen more of these, but you get the general idea\u0026hellip;), a bit of build and deploy for specific devices like this build on iPhone which spawns physics enabled bubbles on a button press:\nor these two projections of 3D objects, a cube and a 3D scan of my face, in AR:\nBut I also learnt how to create animations, events triggers and so on:\nYour browser does not support the video tag. So why do I care about this ?\nGeneral knowledge and an interest in VR, I figured maybe with some experimenting I could translate my recent works in music-related code into something fun.\nHad a bit of fun with xgb # Yeaaaaaaaaars ago, I began working on a window manager called fion because I wanted to understand how window managers worked and I wanted a ion clone. I came up with something that did work as these screenshots show:\nIt looked as I wanted, but it was half-baked, absolutely not finished and usable, and I got carried away with life events so I never got a chance to work on it again.\nFast-forward four years.\nI spend an awful lot of time in macOS, either for work or music, and despite all the hype about how beautiful it is\u0026hellip; well I absolutely HATE its window manager, there\u0026rsquo;s no way around it, I HATE the UI for macOS and I wish I could run ion or a clone on top of macOS.\nThere are a few hacks but ultimately, the only way I\u0026rsquo;d be comfortable relying on such hacks\u0026hellip; is if I wrote the window manager myself and knew how it worked\u0026hellip; and it turns out that, well, I already wrote a small window manager and already know how it works.\nI had a look again at fion and realised that, as much as I like C, I no longer enjoy writing C on anything not low-level. So I began having a look at alternatives to C and libxcb, and found that a Go binding to libxcb called xgb.\nI started toying a bit with it, did not achieve anything interesting yet, this is absolutely not a serious project so I\u0026rsquo;ll dive into this on and off, the hate for macOS window manager has not reached the point I absolutely need to fix it, maybe I\u0026rsquo;ll have something in a few years.\nSetDB # I had played a bit in the past with a project to build a set-only (as in set-theory) database, and I had a small PoC hanging around in a folder (and yes, before you tell me, I know that redis has a set API).\nIt turns out that at work I encountered a case where this seemed like a good fit, all it lacked was a DSL (a query language), so I implemented one and published a very early version of go-setdb.\nLet me quote the README because I don\u0026rsquo;t see a reason to rephrase here ;-)\nWhat is SetDB ? # SetDB is a database to manage sets, as in set theory, and which provides a DSL to query the database for specific set operations.\nWhat\u0026rsquo;s the license ? # SetDB is published under the ISC license, do what you want with it but keep the copyright in place and don\u0026rsquo;t complain if code blows up.\nBut doesn\u0026rsquo;t X, Y or Z already do that ? # Yes, you can technically use several solutions ranging from full-blown SQL databases to data structure servers, however they are not necessarily all very practical for the use-cases that I have. Also, I like writing code so sometimes I do it just because.\nHow does it work ? # SetDB manipulates two kind of sets: persistent sets and transient sets, the former being persisted across queries and the latter existing solely as a result set.\nIt provides a very basic query language, currently only supporting operations that return result sets (union, intersection, difference, symmetric difference). The query language allows the creation of new sets but isn\u0026rsquo;t complete yet and doesn\u0026rsquo;t cover operations not returning sets (subset of, superset of, \u0026hellip; though they are actually implemented already behind the scene).\n$ setdb-cli setdb\u0026gt; x ERR: set x does not exist setdb\u0026gt; x = {} [] setdb\u0026gt; x [] setdb\u0026gt; {1, 2, 3} \u0026amp; {3} [3] setdb\u0026gt; {1, 2, 3} | {4} [3 4 2 1] setdb\u0026gt; {1, 2, 3} - {1} [2 3] setdb\u0026gt; {1, 2, 3} ^ {1} [2 3] setdb\u0026gt; x = {1, 2, 3} \u0026amp; {2, 3} | 4 ^ 2 [3 4] setdb\u0026gt; x [3 4] setdb\u0026gt; Sets are handled as patterns, allowing the inclusion of other sets and dynamic resolving:\nsetdb\u0026gt; y = {1, 2, 3} [2 3 1] setdb\u0026gt; x = y [2 1 3] setdb\u0026gt; y = {1, 2, 3, 4} [1 2 3 4] setdb\u0026gt; x [1 2 3 4] setdb\u0026gt; z = {1, 2, 5 } [1 2 5] setdb\u0026gt; x = y \u0026amp; z [2 1] setdb\u0026gt; x = {x | 1} ERR: cyclic reference is forbidden setdb\u0026gt; a = {1} [1] setdb\u0026gt; b = a [1] setdb\u0026gt; c = b [1] setdb\u0026gt; a = c ERR: cyclic reference is forbidden setdb\u0026gt; They are not typed and can contain integers and strings at this point, including both in the same set. I have yet to decide if I want to have strict type checking on sets, which is trivial to implement, I just don\u0026rsquo;t see a reason why at this point.\nsetdb\u0026gt; fruits = {\u0026#39;grape\u0026#39;, \u0026#39;orange\u0026#39;, \u0026#39;strawberry\u0026#39;} [\u0026#39;grape\u0026#39; \u0026#39;orange\u0026#39; \u0026#39;strawberry\u0026#39;] setdb\u0026gt; vegetables = {\u0026#39;spinash\u0026#39;, \u0026#39;onions\u0026#39;} [\u0026#39;spinash\u0026#39; \u0026#39;onions\u0026#39;] setdb\u0026gt; healthy = {fruits | vegetables} [\u0026#39;grape\u0026#39; \u0026#39;orange\u0026#39; \u0026#39;spinash\u0026#39; \u0026#39;onions\u0026#39; \u0026#39;strawberry\u0026#39;] setdb\u0026gt; gross = {\u0026#39;onions\u0026#39;} [\u0026#39;onions\u0026#39;] setdb\u0026gt; healthy [\u0026#39;onions\u0026#39; \u0026#39;orange\u0026#39; \u0026#39;strawberry\u0026#39; \u0026#39;grape\u0026#39; \u0026#39;spinash\u0026#39;] setdb\u0026gt; healthy - gross [\u0026#39;orange\u0026#39; \u0026#39;strawberry\u0026#39; \u0026#39;spinash\u0026#39; \u0026#39;grape\u0026#39;] setdb\u0026gt; mixed = {\u0026#39;grape\u0026#39;, 1, 2, 3, \u0026#39;watermelon\u0026#39;} [\u0026#39;grape\u0026#39; 1 2 3 \u0026#39;watermelon\u0026#39;] etdb\u0026gt; mixed \u0026amp; {1,2} [1 2] setdb\u0026gt; mixed \u0026amp; \u0026#39;grape\u0026#39; [\u0026#39;grape\u0026#39;] setdb\u0026gt; What\u0026rsquo;s missing ? # code cleanup do a pass to decide on final syntax for the DSL implement set dereference so a set can contain the content of another set, not the other set itself (ie: x = {*y}) implement various caching strategies (some were implemented but temporarily removed) disk and memory optimizations have been discussed, they are just not implemented yet Code # For now, the code is a moving target, so unless you know what you\u0026rsquo;re doing, don\u0026rsquo;t import it.\nOtherwise, look at the example implementations in cmd/, one implements a server and the other a command line tool that also ships a client.\nCode is available in my Github repository for go-setdb.\npoolpOrg/go-setdb Go 0 0 Jujulang # I\u0026rsquo;m working on a language.\nWhy ? Well because I like writing code no one cares about :-)\nSo I wrote Jujulang, an interpreted language that\u0026rsquo;s the bastard child of C, Golang and Python. It supported functions, expressions, flow control, loops, etc\u0026hellip; and it was all fun\u0026hellip;\nfn main() { println(\u0026#39;hello world !\u0026#39;) } until I decided that it was not so fun to rely only on an AST evaluator, but I should rather make this language compiled. So I managed to have it generate LLVM IR, then I managed to build a small native executable for a simple main with just a return value, then I realized it was quite hard and I needed some help so here I am now.\nI bought myself two books ( see the books section), one which told me how to do what I already knew how to do, and one teaching me something I used to know twenty years ago but completely forgot how to do. Joke aside, they are excellent books and I highly recommend them.\nI\u0026rsquo;m currently halfway through generating bytecode for jujulang, with a VM evaluating that bytecode, which will lead me to a better interpreter than what I had\u0026hellip; at which point I\u0026rsquo;ll start looking back at LLVM again for native executables.\nI KNOW no one will use juju, it serves no purpose, it\u0026rsquo;s just personal interest, leave me alone.\nI might start a serie on this website, going through all the steps of this, I don\u0026rsquo;t know. Depends if there\u0026rsquo;s demand.\nOpenSMTPD-related stuff # OpenSMTPD 7.3.0p0 (portable) released # It\u0026rsquo;s been a while since OpenSMTPD portable was left behind, no one doing synchronizations with upstream since I left.\nA discussion had even been opened, asking if the project was dead and if the repository shouldn\u0026rsquo;t be archived, despite me saying I\u0026rsquo;d help anyone get on track if they pick up from where I left.\nThen Omar Polo came out of nowhere, synchronized the repository with upstream, pushed some of the portable diffs to OpenBSD, made sure it built on various distros \u0026hellip; and asked me if we could make a release.\nHe essentially took the dying portable project, revived it, brought it back to date and crafted a release in less than two weeks\u0026hellip; this was awesome work.\nI didn\u0026rsquo;t do much but provide some guidance, history context behind choices and signed the release, so this was quite refreshing and made me want to help more :-)\nHe released OpenSMTPD 7.3.0p0, the first portable release in years, which made a lot of people happy.\nWhich leads us to\u0026hellip;\nOpenSMTPD 7.3.0p1 (portable) about to be released # Obviously, as with each portable release, someone pops his/her head after the release to notify of something not working.\nIssue was investigated and solved by\u0026hellip; Omar again, who fixed and crafted a new release which should be published very shortly as I just signed it and pushed it to the website (which I still maintain).\nfilter-rspamd release # Two weeks ago, I was doing some freelance work and was very focused on what I was doing. I worked for hours on end, which is unusual because I often get distracted by mails.\nAt the end of the day, I realise that I haven\u0026rsquo;t received mails for hours which is VERY unusual, so I quickly log to the mail server and see it\u0026rsquo;s crashed. I restart it and it starts right away, I look at the logs and I realise there\u0026rsquo;s a crash in filter-rspamd due to a failed login attempt.\nAs I start looking at the code to understand the issue, mails buffered by my backup MX start flowing and one mentions a crash in OpenSMTPD, which seems to be caused by filter-rspamd. As mails continue flowing, I see that someone had reported the issue, and it was investigated and fixed by Omar while my server was down.\nThe issue was a parsing error in filter-rspamd, caused by a special case which could be avoided with a small change in the filter protocol. The change had been made for all filter messages\u0026hellip; except this one, obviously.\nOmar submitted a change to the protocol to OpenBSD, but also submitted a PR for filter-rspamd to fix parsing with current filter protocol AND work with the next version of the protocol.\nAll I had to do was \u0026hellip; read his diff and release a new filter-rspamd, easy peasy.\nOther stuff\u0026hellip; # To be honest, I also did other stuff, I just don\u0026rsquo;t feel like talking about it yet.\nWhat\u0026rsquo;s next ? # Vacaciones en Malaga. Sol, playa y tapas.\n","date":"30 June 2023","permalink":"/posts/2023-06-30/2023-q2-music-unity-xgb-jujulang-setdb-and-opensmtpd/","section":"Posts","summary":"TL;DR: music and unity training, a bit of X programming in Go, began a journey in Jujulang a new languae, wrote a PoC set database called setdb, did some minor OpenSMTPD stuff.","title":"2023-Q2: music, unity, xgb, jujulang, setdb and opensmtpd"},{"content":" TL;DR: Received the neck\u0026hellip; too bad it\u0026rsquo;s not exactly the one I ordered :-) The neck # For the neck, I wanted a 24 frets neck and found one that I liked online.\nUpon delivery today, I realized that they didn\u0026rsquo;t ship the correct one, but a 22 frets neck instead (see picture above). So while it still looks very nice, it\u0026rsquo;s not exacrly what I wanted, and it also doesn\u0026rsquo;t fit quite right in the body despite an announced compatible size: turns out that in guitar making, millimeters do matter.\nI decided to go with a more traditional PRS neck and ordered this one:\nwhich I will try to customize it with other inlays if I manage to. I should receive it in a few days, I\u0026rsquo;ll post an update then.\nIn the mean time I\u0026rsquo;ve started looking into three things:\nthe tuners the bridge the pickups I want the tuners to be locking tuners, I\u0026rsquo;ve read good things about the Schaller M6, they are currently at the top of my list but I\u0026rsquo;m still looking for alternatives. When I make a decision and receive them, I\u0026rsquo;ll post a more lengthy article on why I picked these.\nFor the bridge, I want a floating bridge that is NOT a floyd rose because I already have one, I like them but the drawbacks makes me want a different design for this additional guitar. I currently am looking into Gotoh 1502 and 1802 tremolo bridges, haven\u0026rsquo;t decided yet if I\u0026rsquo;ll stick to this idea but Iwill also post a more lenghty article once I settle.\nFinally, for the pickups, I have not decided yet, but I\u0026rsquo;d like passive pickups and would like them mismatched: the neck pickup should be suited to jazz / blues, the bridge pickup should be suited to metal. Ideally if they are split coils, all the better. I came up with a list of pickups to investigate, but since they are the most costly part of the guitar, I\u0026rsquo;ll defer the decision for later in the process depending on how the project is going and if the build quality is decent enough to invest in good pickups.\nWhat\u0026rsquo;s next ? # Waiting for the new neck to arrive, more investigation on tuners and bridge, next post should have more details :-)\n","date":"29 June 2023","permalink":"/posts/2023-06-29/building-my-own-guitar-part-2/","section":"Posts","summary":"TL;DR: Received the neck\u0026hellip; too bad it\u0026rsquo;s not exactly the one I ordered :-) The neck # For the neck, I wanted a 24 frets neck and found one that I liked online.","title":"Building my own guitar, part 2"},{"content":" TL;DR: I\u0026rsquo;m building a guitar from scratch, picking up every component but skipping the tricky cut and glue bits :-) About this project # The primary goal of this project is to build a custom guitar, picking each and every component one by one, so that it\u0026rsquo;s 100% what I want it to be.\nThe second goal is to set myself a reasonnable objective, by not having to cut and glue wood myself, so that I can see if I like building guitars at all.\nThe serie will follow this little project, either to completion or to utter failure, whichever comes first :-)\nThe body # For the body, I want a traditional shape and since I like PRS SE guitars a lot, I decided to go with something similar or close.\nI wish I knew how to pick wood, but given that I won\u0026rsquo;t be cutting it myself, this limited my searches to whatever I could get my hands on conveniently.\nI settled on this body which is solid Mahogany and Maple veneer. Ordered it two days ago from The Guitar Fabric and it was delivered today.\nThe body looks very nice (as far as I\u0026rsquo;m concerned), but in practice it\u0026rsquo;s very rough and needs sanding, it will definitely also need varnishing and maybe some subtle dying too.\nWhat\u0026rsquo;s next ? # I found a nice neck to pair with it, ordered from a different vendor, which should arrive tomorrow and which I\u0026rsquo;ll show you once I make sure it\u0026rsquo;s usable.\n","date":"28 June 2023","permalink":"/posts/2023-06-28/building-my-own-guitar-part-1/","section":"Posts","summary":"TL;DR: I\u0026rsquo;m building a guitar from scratch, picking up every component but skipping the tricky cut and glue bits :-) About this project # The primary goal of this project is to build a custom guitar, picking each and every component one by one, so that it\u0026rsquo;s 100% what I want it to be.","title":"Building my own guitar, part 1"},{"content":" TL;DR: I will be discussing music on this blog from now on. This blog is a musical blog too # As some of you know, I have a deep interest for music.\nThis blog section will focus on music-related topics, from a listener, amateur musician and/or producer point of view.\nI will sometimes post links to music I want to share, either because I did it, or because I like it, but I will also write articles and share hardware and software reviews.\n","date":"25 June 2023","permalink":"/posts/2023-06-25/a-new-section-appeared/","section":"Posts","summary":"TL;DR: I will be discussing music on this blog from now on. This blog is a musical blog too # As some of you know, I have a deep interest for music.","title":"A new section appeared !"},{"content":" TL;DR: I played with MIDI and ChatGPT. Code-unrelated work # I began a few week ago a certified training in music production which I\u0026rsquo;ll be attending for a couple weeks still, and I\u0026rsquo;ll be sharing here some of the things I work on.\nFeel free to subscribe to my Youtube channel where I will not only publish my works in progress but also document my journey in that area.\nI have not been very active these last three months # I have not been very active these last couple months. Partly because of life events and grieving, partly because of general demotivation and partly because the training is draining what\u0026rsquo;s left of my brain. I\u0026rsquo;m slowly but surely stepping back in the game.\nMHL: MIDI to human language # For a couple of projects, I needed a way to express MIDI in a human-readable format.\nThose who have followed my work on earmuff may assume that it can serve this purpose, but it doesn\u0026rsquo;t because it expresses things programatically and at a higher-level: where earmuff works with bars and beats and durations, MIDI works with a set of events triggering at absolute ticks, where earmuff has repeats blocks to provide \u0026ldquo;loops\u0026rdquo;, MIDI \u0026hellip; well, actually repeats events sequentially.\nI wrote a tiny compiler called mhl that allows building an SMF, Standard MIDI File, from the following input and the other way around to generate MHL output from an SMF:\ntrack=1 ticks=16080 message=NoteOff channel=0 key=40 velocity=64 track=1 ticks=16080 message=NoteOn channel=0 key=59 velocity=89 track=1 ticks=16080 message=NoteOn channel=0 key=55 velocity=89 track=1 ticks=16080 message=NoteOn channel=0 key=52 velocity=89 track=1 ticks=16320 message=NoteOff channel=0 key=59 velocity=64 track=1 ticks=16320 message=NoteOff channel=0 key=55 velocity=64 track=1 ticks=16320 message=NoteOff channel=0 key=52 velocity=64 Awesome idea, right ? nope, wrong.\ngo-midicsv: MIDICSV implementation in Golang # It turns out that after I was done writing mhl, I ran into MIDICSV.\nMIDICSV is a specification to represent MIDI as CSV, a human-readable comma-separated format. That\u0026rsquo;s exactly what I wanted, except that it already existed, already covered what I wanted, and there were already reference implementations in Perl and Python. I did some experimenting with the Python version, and since it proved to cover my functional needs, I decided to toss MHL and switch to using MIDICSV.\nI could not use the python version itself, partly because I needed to call it from Golang which was painful, but also because py_midicsv insists on using files whereas I need to work on bytes\u0026hellip; so I wrote a Golang implementation for my needs. This is a work in progress, do not use for anything serious, but the initial code is already commited and available in a Github repository.\nLong story short, the package comes with two commands midi2csv and csv2midi, both built using the go-midicsv/encoding package which provides an Encoder and Decoder. The Encoder reads bytes from an SMF and converts them into a MIDICSV output, whereas the Decoder reads a MIDICSV input and converts it into an SMF output.\nThe code currently only works for valid SMF files and will likely panic on any invalidly crafted SMF, I will make it error out nicely in the upcoming weeks. Good enough for my pocs.\nMelodya: a melody and music assistant # Melodya is a project I\u0026rsquo;ll be working on and off for the next few months: it is a melody and music assistant, helping you write, understand and improve music through the help of AI. Unless you\u0026rsquo;ve been hiding under a rock, you have probably already seen ChatGPT at work, it\u0026rsquo;s impressive af, and it can prove really useful in this use-case of assisting music creation.\nUnfortunately, it only understands text so it\u0026rsquo;s hard to have it work with music: it can\u0026rsquo;t take sound or sheet music as input. I had already played with this in December, teaching ChatGPT about earmuff and having it generate new earmuff source based on instructions. It worked, it worked insanely well, but had some shortcomings because earmuff was invented after the cut-off date for its corpus and doesn\u0026rsquo;t come with enough examples. Unless my prompts were very precise, it would generate instruments names that aren\u0026rsquo;t recognized, sometimes even constructs that it thought existed but didn\u0026rsquo;t, and such\u0026hellip;\nLuckily, between earmuff, go-harmony and go-midicsv, I have written enough code that help deal with music programatically and in human-readable formats that I can use ChatGPT reliably enough for my purpose\u0026hellip; so I played tetris. I assembled blocks, and here\u0026rsquo;s the initial result after a few hours of work. A chatbot that understands what I want to do on a musical project, that keeps track of the state of my project and that can provide explanations while generating MIDI output:\nThis was the very first step.\nI would like to implement speech-to-text so I can dictate my needs, text-to-speech so it can hear back its answers, but I would also like to accept input coming from a MIDI device so I can ask it to analyze, correct me or make suggestions, just as I\u0026rsquo;d like to allow it to output to a MIDI device so I can plug it to a synthesizer or DAW so I can use it more directly.\nIn addition, because MIDI can be (indirectly) converted to sheet music, I\u0026rsquo;d like to be able to display the sheet music as it\u0026rsquo;s being worked on and adapted.\nWhat\u0026rsquo;s next ? # April will be essentially spent on my training, so I don\u0026rsquo;t expect much code to be written.\nIf I manage to, it\u0026rsquo;ll likely be melodya related.\nStay tuned !\n","date":"2 April 2023","permalink":"/posts/2023-04-02/march-2023-melodya-mhl-midi-csv-and-more.../","section":"Posts","summary":"TL;DR: I played with MIDI and ChatGPT. Code-unrelated work # I began a few week ago a certified training in music production which I\u0026rsquo;ll be attending for a couple weeks still, and I\u0026rsquo;ll be sharing here some of the things I work on.","title":"March 2023: Melodya, MHL, MIDI-csv and more..."},{"content":"Almost four years ago, your mom and I left home and rushed to the maternity hospital to give birth to your brother. As I closed the door to our home, I told your mom with a huge smile that the next time we\u0026rsquo;d open the door we\u0026rsquo;d no longer be only the two of us. This was the best day of our lives.\nIn December, we got the news that your mom was pregnant of you. We had been waiting you for so long, almost to the point of giving up. We couldn\u0026rsquo;t be happier but miscarriages taught us caution and we decided to wait the three months milestone before telling your brother, family and friends that you were expected. At every appointment, I held my breath until I could hear your heartbeats, your lovely heartbeats putting a smile on my face and tears in my eyes. I love you so much.\nThe third month\u0026rsquo; appointment arrived mid-February, I was anxious as usual but everything was fine, your heartbeats sounded as lovely as the previous times and you jumped all over the place during the echography, already playing around and making us laugh. I was relieved and I began planning your arrival. We told your brother that he\u0026rsquo;d soon be sharing his toys, it was hard negociations but he eventually agreed and started asking questions about you, what games you\u0026rsquo;d be playing together and what cartoons you\u0026rsquo;d be allowed to watch together.\nTwo weeks ago, we received a phone call after your mom took a usual blood sample, further examinations were required to make sure everything was fine with you. Nothing too alarming, after all everything was fine on all other examinations, it was just to be safe. The same had happened with your brother, and it turned out to be fine. We were very stressed but also knew it didn\u0026rsquo;t necessarily mean bad news. We were just very stressed because we didn\u0026rsquo;t want anything bad to happen to you, love. We waited a week for the examination to happen, the results would come up after a few days.\nLast week, we received a phone call from the lab. You had a genetic disease without a cure, an illness that would make your life painful. We had already discussed about this with your mom a long time ago, before she was even pregnant of your brother, we both knew what it meant because we both agreed: we didn\u0026rsquo;t want you to go through this, no matter how much we wanted you, no matter how much we loved you already. Your brother was with us when we received the phone call. I took care of him while your mom went away to cry, I waited until he was asleep to take my turn, we didn\u0026rsquo;t want him to understand yet, we didn\u0026rsquo;t want to break down in front of him.\nDuring that sleepless night, I tried to convince myself that we could make it work, that as long as we loved you and took care of you everything would be fine, but as much as I tried, I could not pretend that this would be an act of love. It would not be love to knowingly force you into a life of suffering, deprivement and reliance just because we wanted you so much. It would be unfair to both your brother and you to let our desire of a child take precedence over the well-being of both of you. It would be unfair to let you develop further knowing that you had an illness, that you\u0026rsquo;d suffer from it and that your brother would have to take care of you when we\u0026rsquo;re too old or no longer around to do it ourselves. I\u0026rsquo;m sure your mom had the same train of thoughts that night. We decided to do what was in the best interest for the both of you, regardless of how painful it was to us, we knew it wasn\u0026rsquo;t to you at this point.\nIt\u0026rsquo;s been almost a week now. It\u0026rsquo;s been almost a week since we left home and drove to the maternity hospital. We didn\u0026rsquo;t want to go, we knew that this time we\u0026rsquo;d come back alone, just the two of us. A few hours later, on the 24/02/2023, your mom gave birth to you, my little girl, and then your heart stopped beating. Mine too.\nYesterday, we built the courage to tell your brother that you were no longer inside mom. We told him you had to return to the stars and that you\u0026rsquo;ll be watching over him from there, he\u0026rsquo;ll be watching you too.\nYou existed and mattered to us, we won\u0026rsquo;t forget you, I love you 💜\n","date":"1 March 2023","permalink":"/posts/2023-03-01/i-love-you/","section":"Posts","summary":"Almost four years ago, your mom and I left home and rushed to the maternity hospital to give birth to your brother. As I closed the door to our home, I told your mom with a huge smile that the next time we\u0026rsquo;d open the door we\u0026rsquo;d no longer be only the two of us.","title":"I love you"},{"content":"","date":"1 March 2023","permalink":"/categories/personal/","section":"Categories","summary":"","title":"personal"},{"content":" TL;DR: I worked on earmuff and go-harmony, I played with ChatGPT. Re-focusing on go-harmony # As you may have read from my previous post, I have spent some time porting go-harmony to the Dart language and playing with Flutter to build apps that used dart-harmony.\nIt was fun and promising, particularly because I was able to build useful tools right away, but I decided to put this on hold and re-focus on go-harmony as I had bigger plans for it than just a bunch of tools.\nUltimately, when it reaches the state I want, I can create a dart-harmony that is really a binding and resume making tools but using a far more advanced engine.\nMade various improvements to go-harmony # Generic tuning interfacev # First of all, I added support for a generic Tuner.\nA Tuner is an object which, given a tuning system (Equal Temperament for now) and a tuning (A440, A432, \u0026hellip;):\ntuner := tunings.NewTuner(tunings.EqualTemperament, tunings.A440) return tuner.Frequency(note.Position()) can compute the sound frequency for any given note:\n$ harmony -frequency C C4 = 261.63 $ harmony -frequency D D4 = 293.66 $ harmony -frequency C3 C3 = 130.81 It isn\u0026rsquo;t used much within go-harmony but exposing it will allowing building tools that require access to frequencies, such as building\u0026hellip; a tuner, or a synthesizer or pitch detectors, etc\u0026hellip; without resorting to the use of a static table which is what I had previously and want to avoid in the engine.\nnaturals package # I introduced a naturals package to deal with naturals (C, D, E, F, G, A and B).\nThe package exposes constants, for example naturals.C, all of which provide methods to obtain the natural\u0026rsquo;s position and semitones delta relative to C, or to cycle through the naturals set with .Next() and .Previous() (ie: naturals.C.Next() will return a naturals.D and naturals.C.Previous() will return a naturals.B).\nWow, modular arithmetics, how impressive ! Yeah, I know it doesn\u0026rsquo;t seem like much but this is particularly important because it simplifies intervals computations, distances computations, chords and scales construction, and more\u0026hellip; This logic was already implemented in the notes package, but having a dedicated package makes this usable more easily in various places of the engine and allowed me to simplify a lot of code.\noctaves package # The same was done to deal with octaves (C0, C1, C2, C3, C4, C5, C6, C7, C8 and C9).\nThe package also exposes constants, similarly to naturals, with methods to compute .Next() and .Previous() but also to .Add(), .Substract() or obtain an octave constant from an octave position (ie: octaves.FromPosition(4) will return an octaves.C4).\nAgain, this was mostly due to avoid implementing the logic in the notes package but making it easier to use octaves anywhere in the engine outside of the contxt of a specific note. For example, coupling this with the Tuner interface, it could be possible to generate a table of frequencies for a specific octave without having to build each note individually to have access to octaves.\nAdapted notes to make use of naturals and octaves # I adapted the notes package to remove all naturals and octaves computations and rely on the new packages.\nThis simplified the logic, allowed to shrink the file by removing code that just wasn\u0026rsquo;t in the right place, with absolutely no visible change on the outside.\nIntroduced Note.MIDI() # The engine only implements harmony rules and will not be MIDI-aware as it\u0026rsquo;s out of its scope, however I decided to implement a MIDI() method on Note that returns its MIDI note value as this can still be VERY useful and is just an offset-ed semitones computation. I will likely also implement a Piano(keys uint8) method to return the piano key on keyboards of different sizes because that\u0026rsquo;s also just an offset-ed semitones computation.\nWhy add MIDI() ?\nWell, when writing code that deals with MIDI, you usually know the midi note (the pitch) that\u0026rsquo;s been emitted but you don\u0026rsquo;t necessarily know what note it was meant to be. For instance, in a MIDI stream I will receive a NoteOn message with a pitch of 60 which will produce the sound of a C4\u0026hellip; but it may have actually been a B#3 or a Dbb4 if you look at it from a harmonic point of view.\nDepending on the use-case, you will not care or you will want to detect the correct note name, which will result in having to map notes from go-harmony to notes that share the same pitch in whatever tool is being written. Having the engine be able to produce the same .MIDI() value for all enharmonic notes allows to simplify things by a great deal.\nIt turns out I made use of this and could shove a lot of code from earmuff :-)\nWhen Piano keys are also implemented, this means that you can convert a stream of Note into their corresponding keyboard keys without having to worry about enharmomic equivalences either.\nExtended intervals # Finally, I extended the intervals package to support all intervals up to AugmentedFifteenth, making it possible to build all extended chords which were not possible before.\nTons of changes in earmuff # Grammar updates # Until today, the grammar for earmuff source files looked like this:\nproject { bpm 80; time 4 4; instrument guitar { bar { 8th note C# on 3/1; 8th note D on 3/2; 8th note A on 3/3; } bar { half note F on 1/1; 8th note E on 4/2; } } instrument piano { bar {} bar { whole chord Eb9 on 1/1; } } } This worked for experimenting, but the grammar was very limiting and made it harder to implement some concepts.\nI decided to make some slight adjustments, it now looks like this:\nproject \u0026#34;my project\u0026#34; { bpm 80; time 4 4; track \u0026#34;lead guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { on beat 1 play 8th note C#; on beat 2 play 8th note D; on beat 3 play 8th note A; on beat 4 play 8th note F#; } } track \u0026#34;rythm piano\u0026#34; { instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play half chord C#; on beat 2 play half chord D; on beat 3 play quarter chord A; on beat 4 play quarter chord F#; } } } It doesn\u0026rsquo;t show yet, however this makes some upcoming changes MUCH easier to implement, and also simplifies readability as lines read plain english now.\nFirst, it allows naming sections which was not possible before as project didn\u0026rsquo;t take a parameter and instrument was used in place of track to list the MIDI instrument in use rather than being a free field.\nThen, the previous grammar used on beat/subdivision for placement and this was so confusing that I even got confused myself after not touching an earmuff file for a couple month. Furthermore, it came at the end of lines which tied it to the notion of Playable events (sounds, chords) when other events could also require placement (ie: comments, lyrics, cues, \u0026hellip;). By moving it upfront, parsing gets simplified and it makes it possible to implement Tickable events, events that trigger on a specific tick. Instead of a fraction, on beat accepts either an integer for on-beat events and floats for off-beat events.\nClosing the gap with MIDI # earmuff is really tied to MIDI as it compiles its source code to MIDI for both streaming to a synthesizer or generating an SMF file. Ideally, it should support ALL of MIDI features, so that any MIDI can be generated from an earmuff source\u0026hellip; so I started closing the gap by implementing missing features.\nFirst of all, I implemented velocity so that any note or chord may be given a velocity, which will be reflected in the MIDI output. It is now possible to lower or raise the intensity of individual notes and chords:\nbar { on beat 1 play 8th note C# velocity 10; on beat 2 play 8th note D velocity 25; on beat 3 play 8th note A; on beat 4 play 8th note F#; } I also implemented various MIDI Metas, allowing to add copyright, text and others which are emitted in the MIDI output.\nproject \u0026#34;my project\u0026#34; { copyright \u0026#34;Gilles Chehade\u0026#34;; text \u0026#34;this project sounds like shit\u0026#34;; bpm 80; time 4 4; track \u0026#34;lead guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { on beat 1 play 8th note C#; on beat 2 play 8th note D; on beat 3 play 8th note A; on beat 4 play 8th note F#; } } track \u0026#34;rythm piano\u0026#34; { copyright \u0026#34;Jules Chehade\u0026#34; text \u0026#34;this track however ...\u0026#34;; instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play half chord C#; on beat 2 play half chord D; on beat 3 play quarter chord A; on beat 4 play quarter chord F#; } } } Finally, I added support for single and multi-line comments:\nproject \u0026#34;my project\u0026#34; { copyright \u0026#34;Gilles Chehade\u0026#34;; text \u0026#34;this project sounds like shit\u0026#34;; bpm 80; time 4 4; track \u0026#34;lead guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { // skipping first beat //on beat 1 play 8th note C#; on beat 2 play 8th note D; on beat 3 play 8th note A; on beat 4 play 8th note F#; } } /* skipping a whole track */ /* track \u0026#34;rythm piano\u0026#34; { copyright \u0026#34;Jules Chehade\u0026#34; text \u0026#34;this track however ...\u0026#34;; instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play half chord C#; on beat 2 play half chord D; on beat 3 play quarter chord A; on beat 4 play quarter chord F#; } } */ } Lots of internal refactor # In anticipation for upcoming features, I did a pass of refactor to make it possible to plug things the way I want.\nFor instance, I\u0026rsquo;d like to be able to emit Meta\u0026rsquo;s on specific Ticks, which required reworking the way a project is structured, the way a track is structured, the way bars are structured, the logic to compile MIDI from these structures, the logic to playback a MIDI to a synthesizer, etc\u0026hellip;\nLots of small invasive changes.\nInitial attempt at muffear, MIDI-to-earmuff converter # I started working on a small utility to read a MIDI file and generate an earmuff source from it.\nI had a basic version working yesterday, it doesn\u0026rsquo;t cover all aspects obviously and I made grammar changes after so it now generates broken earmuff, but I wanted to see if this could be achieved easily.\nUltimately, this will be merged in the earmuff program, but while I work on it I\u0026rsquo;d rather have things split apart.\nThe idea is that there are various programs than can produce MIDI, either from crafting tracks or recording a MIDI source (like any DAW), or from a partition expressed in various formats (Guitar Pro, ASCII, MusicXML, \u0026hellip;). By allowing to convert MIDI to earmuff, I essentially allow converting any of these inputs to earmuff to work on it as code.\nGiven that earmuff already exports to MIDI and can produce input for these programs, this gives some symmetry.\nA demo ? # Here\u0026rsquo;s a simple 12-bars blues written in earmuff, consisting of a lead piano, a rythm guitar and a bass:\nproject \u0026#34;12 bars blues\u0026#34; { bpm 120; time 4 4; track \u0026#34;lead piano\u0026#34; { instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } } track \u0026#34;rythm guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord F7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord F7; } bar { on beat 1 play whole chord F7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord F7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } } track \u0026#34;bass\u0026#34; { instrument \u0026#34;bass\u0026#34;; bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note F2; on beat 3 play quarter note A2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note F2; on beat 3 play quarter note G2; } bar { on beat 1 play quarter note F2; on beat 3 play quarter note G2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note F2; on beat 3 play quarter note G2; } bar { on beat 1 play quarter note C2; on beat 3 play quarter note E2; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } } } This can be played through a synthesizer by earmuff, producing the following output (encoded in mp3 for this blog):\nYour browser does not support the audio element. Or imported into other tools, such as Guitar Pro:\nor Logic Pro X:\nChatGPT # I tried talking to GPT a bit about earmuff \u0026hellip; it was fun :p\nBecause earmuff is a language, ChatGPT can learn its grammar, understand an input and produce outputs.\nI had it learn how earmuff works, had it transpose the 12 bars blues to a perfect fourth, make some changes to it and add a tuba for a bar.\nJust imagine when I add speech-to-text to the mix, and can dictate what changes I want to apply to a project, check that they sound as expected and generate a new sheet from them \u0026lt;3\n[...] The resulting project is:\nproject \u0026#34;12 bars blues\u0026#34; { bpm 120; time 4 4; track \u0026#34;lead piano\u0026#34; { instrument \u0026#34;piano\u0026#34;; bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } bar { on beat 1 play quarter note G; on beat 2 play quarter note B; on beat 3 play quarter note D; } bar { on beat 1 play quarter note F; on beat 2 play quarter note A; on beat 3 play quarter note C; } bar { on beat 1 play quarter note C; on beat 2 play quarter note E; on beat 3 play quarter note G; } } track \u0026#34;rythm guitar\u0026#34; { instrument \u0026#34;guitar\u0026#34;; bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord D7; } bar { on beat 1 play whole chord C7; } bar { on beat 1 play whole chord G7; } bar { on beat 1 play whole chord D7; } } track \u0026#34;bass\u0026#34; { instrument \u0026#34;bass\u0026#34;; bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note C3; on beat 3 play quarter note E3; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note C3; on beat 3 play quarter note E3; } bar { on beat 1 play quarter note C3; on beat 3 play quarter note E3; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note D3; on beat 3 play quarter note F3; } bar { on beat 1 play quarter note C3; on beat 3 play quarter note E3; } bar { on beat 1 play quarter note G2; on beat 3 play quarter note B2; } bar { on beat 1 play quarter note D3; on beat 3 play quarter note F3; } } track \u0026#34;tuba\u0026#34; { instrument \u0026#34;tuba\u0026#34;; bar { on beat 1 play whole chord C7; on beat 2 play whole chord G7; on beat 3 play whole chord D7; } } } Which resulted in transposed tracks and a new tuba track:\nwhich sounds like this:\nYour browser does not support the audio element. I seemingly got a wrong mapping for instruments as this doesn\u0026rsquo;t sound like a tuba at all, but it\u0026rsquo;s\u0026hellip; interesting ? :-)\nWhat\u0026rsquo;s next ? # A break for New Year, then I\u0026rsquo;ll be working most exclusively on go-harmony and earmuff in 2023.\nI\u0026rsquo;ll write a personal retrospective of this year shortly, if I don\u0026rsquo;t get to it before the end of the year\u0026hellip;\nHAPPY NEW YEAR TO YOU ALL.\n","date":"30 December 2022","permalink":"/posts/2022-12-30/december-2022-some-more-earmuff-and-go-harmony/","section":"Posts","summary":"TL;DR: I worked on earmuff and go-harmony, I played with ChatGPT. Re-focusing on go-harmony # As you may have read from my previous post, I have spent some time porting go-harmony to the Dart language and playing with Flutter to build apps that used dart-harmony.","title":"December 2022: some more earmuff and go-harmony"},{"content":" TL;DR: busy month, did a lot of non tech stuff, but also worked on a new project called feedchain, fixed issues and added new features to go-harmony, and started two projects called dart-harmony and harmonee, both related to music. Music-related stuff # No music productions this month, I lacked time, but I have recorded the result of some guitare practice sessions which amounts to hours of struggle :-)\nHypnosis-related stuff # I gave a two-hours talk in French, along with two other hypnotists, to fifty-ish psychology students at the faculty of Nantes on the topic of hypnosis and myths surrounding it.\nIt was very nice yet a bit stressful as I had roughly a week to prepare the slides while I had a ton of other things to do, and luckily for me it\u0026rsquo;s a topic I\u0026rsquo;m comfortable enough to go without rehearsing otherwise the anxiety disorder would have kicked in ;-)\nAs a result, I had no time to work on my hypnosis podcast for this month but I\u0026rsquo;ll be writing the scenario and recording next week as I wish to complete the fourth episode in early December. This will leave me with four episodes on the topic of dreams, hypnosis, traumas and hypnotherapy before I complete the serie and move on to another project.\nI have a book under writing with the intent to self-publish. It\u0026rsquo;s about ~150 pages at this point but still needs a lot of work and I\u0026rsquo;ll try to focus a bit on it as I\u0026rsquo;d like to avoid letting it linger for too long. It covers the topic of hypnosis from a mechanical point of view, essentially following the podcast but with much greater details. When it\u0026rsquo;s finished and published, I\u0026rsquo;ll likely translate from French to English.\nI worked a lot on the feedchain project # As you may have seen from my previous article this month, I worked a lot on an experimental project called Feedchain.\nI designed the general structure in a few hours but made iterative changes here and there for days, while constantly adapting the proof-of-concept Golang implementation to reflect these. Since I already had the Golang implementation, I also wrote a publish server that undertands how to to verify feedchains and serve them efficiently. The next step was to write a proper client for feedchain, but I decided to take a step back and not implement that in Golang but in Dart instead.\nWhy Dart, you ask ?\nIt turns out that for this project and others I keep telling myself I should make a nice UI, as a native application or a web interface, but I don\u0026rsquo;t have the motivation and time to learn too many frameworks to handle too many targets. I\u0026rsquo;ve grown curiosity for the Flutter framework which allows building native applications for multiple targets using a single codebase in Dart, and so it made sense to learn myself some Dart if only to give it a try.\nSo I implemented the feedchain parser in Dart, as well as classes to both produce and consume feedchains, which surprisingly took me only a couple hours with no prior experience with Dart. I then tried building an application in Flutter that would play nice with feedchains I generated from my Golang proof-of-concept, and this is where the struggle began because\u0026hellip; believe it or not, after \u0026gt;20 years of writing code that executes in a console, it turns out I\u0026rsquo;m not an UI guy and I suck at making anything look nice.\nI\u0026rsquo;ll continue working on this, but I needed to also work on other projects.\ngo-harmony # I wrote a bit about my go-harmony package back in June, and not much had changed since then.\nSo I dived back into it and fixed a few issues, related to interval arithmetics for the most part, but also added a lot of new features like support for progressions, for all missing intervals, for various new chord structures (7sus2, 7sus4, M7sus2, M7sus4, sus2sus4, b5, \u0026hellip;) or improvements on the handling of chord inversions.\nI also implemented a scale scanner that, given a chord, will output all scales sharing more than two notes with the chord and sorted by most notes matched first:\nKeep in mind that the project is the engine, not the useless CLI :-)\nAt this point, though not exempt of glitches, go-harmony can already:\nparse note names, chord names, scale names determine interval between two notes determine note at a specific interval from a root note transpose note or chords to a specific interval on the fly build a ton of different chords by applying a chord structure to a root note build several scales by applying a scale structure to a root note compute diatonic triads and seventh chords for a specific scale compute relative minor / major for a chord compute inverse intervals recognize a chord from a set of notes if the chord structure is already implemented and so I did what had to be done \u0026hellip; I threw some MIDI at it :-)\nI wrote a tool that could ingest some MIDI and use go-harmony to try to make sense to what was happening in the track, try to detect scales, chords and what not, and\u0026hellip; the result was both interesting and unimpressive at the same time.\nIt was interesting because on some tracks it did manage to understand what was happening, what chords and scales were being played. At one point, I thought there was a bug in the code because it was not displaying the same chord name as the score but go-harmony was actually right, it provided a more accurate name than what was written. Another interesting thing is that I tested on-the-fly transposition and it worked lovely, both to convert a track from minor to Major, but also to transpose all notes a perfect fourth upper as they were read. That\u0026rsquo;s not a huge feature but I was happy it worked flawlessly as I have plans for it.\nIt was unimpressive because it still lacks some key features (pun intended), such as add / omit, which causes a lot of chords to go undetected or misdetected. A LOT. Too many actually that it prevents me from moving forward with some of the more interesting features I have in mind, so I really need to tackle these detection failures before anything else\u0026hellip; and it\u0026rsquo;s not trivial :-)\ndart-harmony, wait what ? # Yeah, I\u0026rsquo;m spending time on a Golang implementation of a music theory engine\u0026hellip; and I\u0026rsquo;m reimplementing it in Dart at the same time.\nBear with me.\nI\u0026rsquo;m not really doing the work twice because they are essentially implementing the same calls and structures, just implementing in two different languages because I have two different uses for this.\nThe go-harmony implementation is meant to be used both by a REST API that I\u0026rsquo;ll probably make public next year, and by other stuff I have in mind but that I\u0026rsquo;m not willing to disclose yet.\nThe dart-harmony implementation is meant to be used by Flutter apps so that I can start building useful stuff, tools that people can actually install on their devices without having to understand how to \u0026ldquo;compile\u0026rdquo; something, but more importantly tools that I need to have on me frequently even if not in front of a computer :-)\nThe dart-harmomy project was started 8 days ago and it has already caught up with go-harmony, it is even slightly ahead as I have started building stuff with it and had to add missing features. It supports pretty much all of what go-harmony supports\u0026hellip; but code keeps getting improved as I learn new Dart tricks.\nHarmonee (the name might change) # With dart-harmony it became possible to start implementing tools in Flutter, so I gave a try at various stuff to get familiar with Flutter programming.\nThe first tool I wrote is a circle of fifths implementation which consists in three nested circles, the outter circle being the actual circle of fifths, the middle circle being the minor chords that map to the outer circle, and the inner circle being the diminished chords. The circle is dynamic, clicking on a specific section rotates the circle so the relevant key is exposed on top.\nMy main issue is that this doesn\u0026rsquo;t scale well on screen resizes, so I really need to work out a better way to display it:\nThanks to dart-harmony there\u0026rsquo;s a TON of informations that could be associated to each block of the circle, like related chords, related scales, substitutions, etc\u0026hellip; but the issue is really how to display information effectively and I still need a lot of work to be more effective at designing nice stuff.\nSo I decided to temporarily split into multiple different pages so I can experiment with different ways to display data, and I implemented a search engine which will dynamically display cards as you type your query. On the examples below, I started typing A which resulted in the natural A, the note A in the default octave, the A major chord and the A major scale, but as I continued typing A7 it refined to the note A in the 7th octave and the A7 chord, when I typed Amyxolydian it diplayed the A myxolydian scale:\nJust as I said for the circle of fifths, I have plenty of additional informations that could be displayed on each card, I just haven\u0026rsquo;t figure out the nicest way to display them yet.\nUltimately I decided to give another try at displaying things in a circle, and came up with something nice as a starting point to implement a chord dictionnary:\nand a scale dictionnary:\n\u0026hellip; again, I have access to more informations, like interval below each note or diatonic chord for each degree, but it quickly makes the interface unreadable\u0026hellip;\nSo the bottom line is, I need more time to undertand how to make this really useful as what I really want, ultimately, is a tool to rule them all: a complete assistant for all music theory related topics.\nThe nice thing is that it allowed me to deep dive into Flutter, I did test building and deploying on real hardware as I could test Harmonee on my phone, and the entry cost to start making interfaces was very low: I came up with usable tools, even though limited, in just a few hours of playing with it. I hope I can come up with very nice things as I get more experienced.\nWhat\u0026rsquo;s next ? # December will be a very short month for me as my kid has two weeks of holidays and one of them falls right in my free-time week, meaning I won\u0026rsquo;t have much time to work on anything beyond my noon breaks and sleepless nights ;-)\nI\u0026rsquo;ll probably keep working a bit on dart-harmony as time allows and will write my yearly retrospective late December, don\u0026rsquo;t expect too much done before 2023 !\n","date":"30 November 2022","permalink":"/posts/2022-11-30/november-2022-feedchain-go-harmony-dart-harmony-and-harmonee/","section":"Posts","summary":"TL;DR: busy month, did a lot of non tech stuff, but also worked on a new project called feedchain, fixed issues and added new features to go-harmony, and started two projects called dart-harmony and harmonee, both related to music.","title":"November 2022: feedchain, go-harmony, dart-harmony and harmonee"},{"content":" TL;DR: I implemented a file format to support a standalone feed of news similar to what I have on Twitter, as well as a proof of concept code for a reader, a writer, a publishing node, and some more. oh and fuck you, elon, you\u0026rsquo;re a dick. This is not my activity report # This is not my activity report which I\u0026rsquo;ll publish later this month or early December. I\u0026rsquo;ve worked on several projects but I thought this one would be better off discussed in a dedicated article.\nThe problem\u0026hellip; # In just a month, two services that I was heavily relying upon turned into shit.\nThe first one is Signal, a secure messaging system, which removed a key feature to the point I no longer have any use for it as most of my contacts uninstalled. It sucks and I can\u0026rsquo;t really do anything about it, there are alternative just not ones that would embark non-techies like my mother or my wife as easily as Signal did.\nThe second one is Twitter, which is taking a very dramatic turn as the, hum, how do I put it\u0026hellip; \u0026ldquo;Genius\u0026rdquo; who bought it managed to get most employees fired or resigned in less that two weeks, bringing the service to a point where it\u0026rsquo;s unclear if there are even enough people left to care for the machines that host it (according to comments from former engineers there).\nUnfortunately for me, I don\u0026rsquo;t use it only for the fun of it but also to communicate about stuff while I work on them, which comes with a ton of benefits for me, the most obvious one being the freelance gigs that occasionally come as a result from sharing information there.\nA solution adopted by many has been to move to Mastodon, which I also may end up doing, but I dislike the idea that I\u0026rsquo;ll have to trust yet another service with keeping my data and not shutting down in a year or two putting me in a similar position. There are also many other reasons which are also causing me to think twice before making a move there, including as an operator of my own instance.\nSo what do I do with that :-/\nI worked on a new little project\u0026hellip; # Feedchain is a project to maintain a feed of news that can be published as a standalone file, distributed from various locations, while allowing to fetch specific portions of it, guaranteeing the authenticity and integrity of the content and expecting as little requirements as possible for both the publisher (me) and the readers (you).\nThink of it as a kind of RSS feed\u0026hellip; but an RSS feed that\u0026rsquo;s not XML, that contains an index for random access to any information chunk contained in the feed, and that has a signature for the whole feed as well as for each individual chunk. And like an RSS feed, it will not require a specific server to work but any service that can expose the file and preferably support range queries to avoid full fetches.\nI have a working PoC # I have split the PoC into small applications to make it easier for me to experiment with it, however it\u0026rsquo;ll obviously have to be merged into a single useable application so it\u0026rsquo;s not a pain in the ass to use for others.\nThe first application is a feedchain writer which can generate an empty feedchain, update its content or metadata, and publish it somewhere.\nThe writer generates an ed25519 keypair, the feedchain is identified by the public key stored in the header and a name stored in the metadatas to ease referencing it. For instance, my feedchain has the public key nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 and is identified by the name @poolpOrg, the header, index and metadata chunks are signed with my private key so it can be verified by a feedchain reader.\n% writer -create -name poolpOrg created feedchain nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 % From this point the feedchain exists on my machine and because I only have one, I don\u0026rsquo;t have to provide the name when adding data to it, it\u0026rsquo;ll figure out where to write:\n% writer -write \u0026#34;just set up my fdchn\u0026#34; % writer -write \u0026#34;let\u0026#39;s see how this little project goes :-)\u0026#34; % The feedchain that results from this can be uploaded directly to any web server, for example using scp, but because not everyone knows how to configure and run a web server\u0026hellip; I implemented feedchain node servers that handle publishing and distribution, and which anyone can run at the cost of providing disk space and some memory, and I deployed one at https://feeds.poolp.org to bootstrap the project.\n% writer -publish % From this point, the feedchain is public and distributed by https://feeds.poolp.org at the following address https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4:\n% curl https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 \u0026gt;/dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1245 100 1245 0 0 9867 0 --:--:-- --:--:-- --:--:-- 10462 The second application is a feedchain reader, which knows how to parse the feedchain file format, verify signatures and use the index to access specific chunks.\nIt supports reading the content from a local file or from a remote HTTP(S) server as long as it supports range queries. The example reader outputs all content from the feedchain, but a smarter reader could apply filters to extract the last n chunks, restrict to a specific timeframe, match hashtags, etc\u0026hellip;\n% reader ~/.feedchain/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 [2022-11-19T01:52:16+01:00] (sha256:fbb1...4414): just set up my fdchn [2022-11-19T02:03:59+01:00] (sha256:fbdf...c88e): let\u0026#39;s see how this little project goes :-) % reader https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 [2022-11-19T01:52:16+01:00] (sha256:fbb1...4414): just set up my fdchn [2022-11-19T02:03:59+01:00] (sha256:fbdf...c88e): let\u0026#39;s see how this little project goes :-) % The third application is a feedchain watcher, which knows how to follow/unfollow feeds by URL (no dependency to a feedchain node) or by name (requires contacting a feedchain node):\n% watch -follow poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 % % watch -follow poolpOrg % \u0026hellip; then monitor them for changes to produce a live output:\n% watch [2022-11-19T01:52:16+01:00] @poolpOrg (nTGU...ZGZ4): just set up my fdchn [2022-11-19T02:03:59+01:00] @poolpOrg (nTGU...ZGZ4): let\u0026#39;s see how this little project goes :-) [output updated as feeds are updated...] The feedchain nodes which I mentioned briefly are standalone servers that makes it easier to publish feedchains.\nThey have an endpoint for posting a feedchain which makes it public if the feedchain is correctly formatted and has a valid signature:\n% curl -XPOST --data-binary @/etc/group https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 header signature verification failed % curl -XPOST --data-binary @/Users/gilles/.feedchain/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 https://feeds.poolp.org/nTGUG4fhMTq9r3r5UTwgZotdiqkFtN--VaJ_l-PZGZ4 % Then, they allow fetching the raw file from that same endpoint while allowing range requests, making it possible for the reader to fetch the header, the index then seek and read from specific offsets.\nBecause a feedchain is basically an RSS on steroids, nodes also provide an RSS view of the feedchain though it still needs a bit of work to be rendered more nicely by an RSS reader:\nThey also provide a REST API to allow fetching informations from a specific feedchain. The API exposes the header, the index, metadatas and individual blocks which may be indexed either by their digest or by their offset in the feed:\nBy contacting a node and using its API, it becomes possible to easily consume a feedchain from client-side code to embed portions of a feedchain on a website, or to implement a more advanced application without having to reimplement a feedchain parse but relying only on HTTP requests.\nFinally, I learnt myself a bit of Dart so I could start playing with Flutter and see what it meant to create a native mobile application, but I will need a bit of time to have something usable (screenshots taken from an earlier version).\nAbout the format # The feedchain file is structured in the following way:\nIt begins with a signature of the header\u0026rsquo;s digest, allowing to check that it changed by reading the first 64 bytes, and verifying that the header was not altered by reading the first 320 bytes.\nThe header contains the offset, size, digest and signature for both the index and metadata chunks, it is followed by all data chunks and ends with the index and metadata chunks. The index contains the creation time, offset, size, digest and signature for each individual chunks, the chunks themselves contain the digest of their predecessor chunk (similarly to a blockchain), and as a result the headers digest protects the structure of the entire chain.\nNot much more to be said, it is really straightforward, to read a feedchain you read the header, locate the index and iterate over the index records to locate the offsets and lengths of blocks, validating digests and signatures as you iterate.\nLimitations and rationale # First, AND VERY OBVIOUSLY, this isn\u0026rsquo;t a Twitter replacement and does not / can not cover a lot of the features you\u0026rsquo;d expect from Twitter or Mastodon. It barely tackles the issue of having to provide a third-party the ownership on data and it allows decoupling a feed from a specific hosting provider, whereas features like searching or user interactions are completely out of the scope.\nCan these even be implemented at all ?\nYes, however they would be implemented as services around feedchains with specific server-side code provided by nodes. For instance, to be able to follow a feed by name rather than URL, I had to implement a service to perform a name lookup on feeds hosted on a specific node. This is not a feature of the feedchain itself but a feature of the hosting node. This means that, by default, likes and private messages and whatnot do not exist in feedchain, they MAY only exist if server-side code supports them as optional services on the side.\nI want the feedchains to remain standalone files usable from a static web server, not requiring the use of specific server-side code BUT being able to take advantage of server-side code IF desired. By limiting the features of a feedchain to its standalone file, and adding optional features around, I can guarantee that feeds can be moved around easily without causing too much disruption, they can be republished anywhere easily and nodes can operate as a directory.\nI can decide to provide a direct link to my feedchain hosted on my static web server and not rely on any feedchain node, or I can decide to publish my feedchain to a node because I don\u0026rsquo;t have resources for self-hosting and if it goes down, I\u0026rsquo;ll publish elsewhere. There\u0026rsquo;s no requirement to rely on nodes and, as a matter of fact, I could publish on my website and someone else could find my feed interesting enough to track it from my static web server and mirror it on a node.\nThis is something that by design can\u0026rsquo;t be prevented, the way feedchain works means that I can\u0026rsquo;t impose hosting on a node of mine.\nBut it means you keep rebuilding and uploading the whole feed ? # YES \u0026hellip; and NO, it depends.\nWhat\u0026rsquo;s for sure is that using a standalone file means that adding, removing or updating can mean having to rewrite portions of the file, and that publishing may mean having to upload the whole file. So does that mean that whenever I add a small status update to my feed, I HAVE to push the whole feed again because the file was rewritten and no longer has the same checksum and signatures on header and index ?\nWell, in the proof-of-concept implementation YES, but that\u0026rsquo;s because publishing must support uploading a full chain and it was easier to start from that point\u0026hellip; however in practice, there are MANY ways that this can be avoided because of how the file is structured.\nFirst of all, the file is structured in such a way that the most common operation of editing metadata or adding/updating/deleting from the end of the feed is cheaper. In such cases, only the end of the file needs to be rewritten as it may cause offset shifts, the header being fixed-sized can have signatures and checksums overwritten in place.\nWhen publishing to a static website, the full chain would need to be re-uploaded, but when publishing to a node exposing an API, partial updates through diffs and in-place overwrites would be possible.\nThen, locally, it\u0026rsquo;s not because the end result is a standalone file that the client HAS TO to work from that file. It could store header, chunks, index and metadata separately and build the feedchain from them, so shifting, deleting, updating would not be so complex.\nWhat\u0026rsquo;s for sure is that feedchain is not an immutable feed, it allows updating older entries\u0026hellip; but at the cost of recomputing the full chain beyond edit, altering chunk identifiers (digests) on the way. This is a design choice as it is technically possible to avoid this by removing the blockchain structure (blocks containing the previous block digest), however by enforcing this it discourages the rewrite of history and ensures that people referencing a chunk do not get tricked when the chunk suddenly changes to something else. I think it\u0026rsquo;s better to support edits through the use of new chunks referencing previous ones than by rewriting old ones.\nIf this is a topic of interest, let me know and I\u0026rsquo;ll write more in a future article about how all operations work in feedchains.\nNext steps # I don\u0026rsquo;t have clear steps as I\u0026rsquo;m mostly experimenting around at this point, but among the next steps are:\n1- extending the node server code to support p2p distribution of indexes, so you could run your node and bootstrap it from mine, they would share indexes of feeds they know so that when someone asks my node if it knows where a feed is located, it could point to yours transparently.\n2- move away from proof of concept to higher-quality code for clients. Having to rely on a reader app, a writer app and a watcher app is nice to help me debug and experiment, but it\u0026rsquo;s not so nice for regular use.\n3- implement an application in Flutter so non-techies can use feedchain on their mobile phones or computers without having to understand a thing about what I wrote today.\nWant to help and contribute ? # I have only played with this for a few cumulated hours to be fair, so there\u0026rsquo;s plenty of iterations to do on it to make it nicer.\nThere\u0026rsquo;s code to write in Golang, there\u0026rsquo;s code to write in JavaScript, there\u0026rsquo;s code to write in Dart, and you could implement feedchain in your favorite language using the examples from the proof-of-concept and asking question on my discord.\nIs it tricky ? Nope. I implemented the initial proof of concept in Golang, I knew nothing about Dart and wrote a full implementation for it in just a few hours, so anyone could probably implement feedchain in whatever language of their choice in a really short timeframe.\n","date":"20 November 2022","permalink":"/posts/2022-11-20/feedchain-is-a-standalone-news-feed-project/","section":"Posts","summary":"TL;DR: I implemented a file format to support a standalone feed of news similar to what I have on Twitter, as well as a proof of concept code for a reader, a writer, a publishing node, and some more.","title":"Feedchain is a standalone news feed project"},{"content":" TL;DR: added comments support to the blog, did some music and hypnosis projects, fixed a few bugs in plakar and began a new toy project. Learnt myself some Swift # Got myself a book on the Swift programming language, spent a couple hours reading it and I really enjoyed it.\nI was mostly curious, I don\u0026rsquo;t have anything I want to do with it at the moment, but it seemed really nice as it kind of looks like a mix between C, Golang and Python, taking bits I like from the three of them.\nWe\u0026rsquo;ll see if something comes out of it for me in the future :-)\nAdded table of content and comments support to the blog # For the last few years, this blog has used a static generator so that I could write the posts in Markdown and publish by committing to a repository. This is nice because the website is standalone and I can easily move it around from machine to machine, but it has prevented me from providing comments and I ended up relying on Github discussions and adding a link to the proper discussion below each article I write.\nI do get feedback from people, but at the exception of some articles that were referenced on popular websites, these feedbacks tend to happen through other means like private messages on Twitter or discord. Maybe these people wouldn\u0026rsquo;t leave a comment on the blog if it was possible, but maybe it\u0026rsquo;s just because having to follow a link and comment elsewhere is not practical and does not keep the article available with a scroll.\nSo I found a nice javascript library that allows interfacing with the Github discussions API, and now each article has\u0026hellip; its own comment section. It is now possible to comment directly at the bottom of each article, the comments reflect what was discussed directly on Github and are published there.\nWhile at it, I figured it was possible to add a table of content to my articles so you can jump to specific sections. I have not added it to previous articles, though I might for some, but from now on I\u0026rsquo;ll make sure to always include one as I had lengthy articles in the past and it\u0026rsquo;ll ease reading.\nFeel free to comment this article and test :-)\nMusic-related work # I enrolled in a course on music production that will start in March 2023 at a local studio in Nantes, so I spent some time playing with Logic Pro X to be more familiar with it and not begin the course as a complete newbie with regard to tools.\nThese are four LoFi remixes I worked on by importing midi tracks, altering instruments, scores and tempo, and adding some effects. Definitely not fantastic but I do like the result for some so I\u0026rsquo;m sharing:\nLet me know if you like them, I have created a specific playlist on my youtube channel for these experiments, you can subscribe, comment and give thumbs up :-)\nHypnosis-related work # I haven\u0026rsquo;t spent much time at the hypnosis office since I came back from holidays, only performing a few sessions here and there, mostly due to dedicating time to my son as he entered kindergarten and I need to adjust to his schedule.\nSo I decided to spend some time working on a podcast in French to explain how hypnosis works at low-level. I have recorded the first three episodes of 8 and I intend to release roughly 1 per month, with next episode due in late November or early December as I\u0026rsquo;ll present a conference in November and won\u0026rsquo;t have time to record.\nAs a side-note, I found it enjoyable to record so I might create a podcast related to computer stuff in the future, when I\u0026rsquo;m done with this one.\nA bit of plakar work # First, I fixed a crash that happened when trying to list content of a path that consisted in a single directory and not starting with /. Basically, there is no such thing as a \u0026ldquo;relative\u0026rdquo; path in plakar as they are always relative to the snapshot and therefore to the root of a snapshot, so def:etc is really def:/etc as far as plakar is concerned. Because of how paths are handled, by splitting pathnames into atoms and recreating a filesystem view, this confused plakar about what was the root directory and caused it to crash. I fixed the function that splits a snapshot path (ie: \u0026lt;snapshot\u0026gt;:\u0026lt;pathname\u0026gt;) into prepending a slash to the pathname portion whenever there isn\u0026rsquo;t, which should not only fix the crash I observed but also all commands that accept snapshot paths.\nThen, I spotted a more annoying bug where restoring a snapshot results in a partial restore with missing files. I was concerned at first because the last thing I want in a backup tool is for a snapshot to miss files, however after a bit of testing I could verify that they weren\u0026rsquo;t really missing as they could all be restored individually, this pinpointed to a bug in the pull primitive which seems to miss some restorable files. I spent several hours trying to understand the issue but couldn\u0026rsquo;t figure out why it happens, except that it happens mostly with snapshots containing a lot of files and when pull is parallelized. It doesn\u0026rsquo;t happen, or at least I could not reproduce, when working with a small snapshot or when pull is sequential. I removed concurrency for the time being but this is not practical, sequential restore of a large snapshot is dead slow, I\u0026rsquo;ll continue tracking the issue.\nI also removed the gzip support that I added in plakar cat around May, and moved it to a dedicated plakar gzcat command. Back then it seemed like a good idea to me that plakar cat transparently inflated compressed output at it allowed to plakar cat a compressed log, but after some thinking I\u0026rsquo;d rather provide that through a dedicated command as it allows using plakar cat to redirect a compressed stream to a file, something that was no longer doable.\nFinally, I changed the way plakar checksum works and implemented a -fast option. The command allows printing the sha256 checksum of a file contained in a snapshot, and since I already had the checksum in the snapshot INDEX, the command didn\u0026rsquo;t recompute it but used the recorded checksum instead. I changed it so that it recomputes the checksum by default, reading the file chunk by chunk, and only resorting to the recorded checksum when the -fast option is used. In practice, this should always produce the same result unless the plakar store has corrupted chunks.\nAnd that\u0026rsquo;s all for plakar !\nA new project: streamchain # So this is a TOY project, nothing serious, I\u0026rsquo;m just experimenting with an idea.\nBasically, I wrote a tool which lets me create a \u0026ldquo;twitter-like\u0026rdquo; stream where I can post short messages, however instead of being centralized on a server they are stored in a standalone structured file.\nThe file is structured in a way that allows verifying both authenticity and integrity, but also allows for fast random access to any message without having to read the entire file. A streamchain reader can validate that the streamchain file originates from its owner, it can validate that it has not been tampered with, and can seek to any message in constant time regardless of the streamchain size.\nAs a result, it can be hosted anywhere from an s3 object storage to a static web server, and if the storage supports range queries, like pretty much all HTTP servers\u0026hellip; then all the better. A streamchain client reader that can talk HTTP can either mirror a streamchain locally and perform range queries to synchronize updates, or it can consume the streamchain remotely without downloading it entirely.\nI have a working PoC but I\u0026rsquo;m not ready to publish the code yet, so I\u0026rsquo;ll likely write an article about it when I open the repository publically.\nWhat\u0026rsquo;s next ? # A conference on hypnosis for psychology students in November, so probably not much happening in the next two weeks as I prepare for it.\nThen, I\u0026rsquo;ll try to get streamchain in shape to open the repository.\nTake care, stay tuned, I\u0026rsquo;ll post as soon as I resume my work !\n","date":"18 October 2022","permalink":"/posts/2022-10-18/october-2022-blog-comments-a-bit-of-plakar-and-the-streamchain-project/","section":"Posts","summary":"TL;DR: added comments support to the blog, did some music and hypnosis projects, fixed a few bugs in plakar and began a new toy project.","title":"October 2022: blog comments, a bit of plakar and the streamchain project"},{"content":" TL;DR: did a lot of music, even while writing code. Code-unrelated work # As last month, I\u0026rsquo;ll start with code-unrelated work !\nFirst, here\u0026rsquo;s my progress on learning Hyunsoo Lee\u0026rsquo;s adaptation of Bach\u0026rsquo;s Air on G string, still not quite right and with some missing parts but\u0026hellip; slowly getting there:\nWhile at it, I started learning Hyunsoo Lee\u0026rsquo;s adaptation of Beethovens\u0026rsquo; 5th\u0026rsquo;s Symphony, also not quite right and with missing parts but also making progress:\nIt was a long time since I made a LoFi track, here\u0026rsquo;s two that I thought were not too bad:\nThey\u0026rsquo;re mostly an exercise to get familiar with Logic Pro X.\ngo-harmony # go-harmony is a package written in Golang that intends to implement a music theory engine to build other tools with. It\u0026rsquo;s only a month old so there\u0026rsquo;s not much to it yet, however there\u0026rsquo;s enough that I could use it to build the project I\u0026rsquo;ll talk about in the next section.\nMy intent was not only to build the engine but also to refresh my memories and fill some gaps\u0026hellip; so I decided not to be lazy and copy-paste a bunch of values in static tables. Instead, I first implemented notes, then intervals as semitones distances to a root note, then chords as a pattern of intervals from a root note (taking into account chord inversions), then also scales also as a pattern of intervals, then derived triads and sevenths chords using the chords implementation, and so on\u0026hellip;\nI made it so it would be very easy to extend for unsupported cases, like chords which are defined as follows making it simple to add new families by simply describing the general structure:\nMajorTriad Structure = Structure{ intervals.PerfectUnison, intervals.MajorThird, intervals.PerfectFifth, } MinorTriad Structure = Structure{ intervals.PerfectUnison, intervals.MinorThird, intervals.PerfectFifth, } AugmentedTriad Structure = Structure{ intervals.PerfectUnison, intervals.MajorThird, intervals.AugmentedFifth, } DiminishedTriad Structure = Structure{ intervals.PerfectUnison, intervals.MinorThird, intervals.DiminishedFifth, } [...] AddNinth Structure = append(MajorTriad, intervals.MajorNinth) AddEleventh Structure = append(MajorTriad, intervals.PerfectEleventh) AddThirteenth Structure = append(MajorTriad, intervals.MajorThirteenth) SusSecond Structure = Structure{ intervals.PerfectUnison, intervals.MajorSecond, intervals.PerfectFifth, } SusFourth Structure = Structure{ intervals.PerfectUnison, intervals.PerfectFourth, intervals.PerfectFifth, } The same applies for scales:\nvar Ionian = []intervals.Interval{ intervals.PerfectUnison, intervals.MajorSecond, intervals.MajorThird, intervals.PerfectFourth, intervals.PerfectFifth, intervals.MajorSixth, intervals.MajorSeventh, intervals.Octave, } var Dorian = []intervals.Interval{ intervals.PerfectUnison, intervals.MajorSecond, intervals.MinorThird, intervals.PerfectFourth, intervals.PerfectFifth, intervals.MajorSixth, intervals.MinorSeventh, intervals.Octave, } [...] var BluesMajor = []intervals.Interval{ intervals.PerfectUnison, intervals.MajorSecond, intervals.PerfectFourth, intervals.PerfectFifth, intervals.MajorSixth, intervals.Octave, } var MinorPentatonic = []intervals.Interval{ intervals.PerfectUnison, intervals.MinorThird, intervals.PerfectFourth, intervals.PerfectFifth, intervals.MinorSeventh, intervals.Octave, } Then go-harmony uses artihmetics to compute everything from a root note and its intervals. For example the scale.Notes() method computes the notes for a scale by applying the intervals that match the scale pattern for a given root note:\nfunc (scale *Scale) Notes() []notes.Note { ret := make([]notes.Note, 0) for _, interval := range scale.structure { ret = append(ret, *scale.root.Interval(interval)) } return ret } No hard-coded scales for each notes and once a scale structure is implemented, it works for all notes.\nThe package by itself doesn\u0026rsquo;t do much more, it\u0026rsquo;s meant to provide APIs and structures for other projects to rely upon, however I built a small utility that\u0026rsquo;s shipped with it to showcase some of it\u0026rsquo;s features.\nFor example, it can be used to validate note names and obtain their frequencies relative to a particular tuning (only A440 for now):\n% harmony -note \u0026#39;C\u0026#39; C 261.63 % harmony -note \u0026#39;C5\u0026#39; C 523.25 % harmony -note \u0026#39;A\u0026#39; A 440 % harmony -note \u0026#39;Cb\u0026#39; Cb 493.88 % harmony -note \u0026#39;C#\u0026#39; C# 277.18 % harmony -note \u0026#39;Cbb\u0026#39; Cbb 466.16 % harmony -note Z 2022/06/14 13:25:36 bad note (Z): should be \u0026#39;C\u0026#39;, \u0026#39;D\u0026#39;, \u0026#39;E\u0026#39;, \u0026#39;F\u0026#39;, \u0026#39;G\u0026#39;, \u0026#39;A\u0026#39; or \u0026#39;B\u0026#39; Just as with notes, it can be used to validate chord names (it supports multiple notations) and decompose them into their building intervals:\n% harmony -chord \u0026#39;C\u0026#39; Cmaj 1: C 261.63 3maj: E 329.63 5: G 392.00 % harmony -chord \u0026#39;C5\u0026#39; C5 1: C 261.63 5: G 392.00 % harmony -chord \u0026#39;C7\u0026#39; C7 1: C 261.63 3maj: E 329.63 5: G 392.00 7min: Bb 466.16 % harmony -chord \u0026#39;C7b5\u0026#39; C7dim5 1: C 261.63 3maj: E 329.63 5dim: Gb 369.99 7min: Bb 466.16 % harmony -chord \u0026#39;Csus2\u0026#39; Csus2 1: C 261.63 2maj: D 293.66 5: G 392.00 % harmony -chord \u0026#39;Cadd9\u0026#39; Cadd9 1: C 261.63 3maj: E 329.63 5: G 392.00 9maj: D 587.33 % harmony -chord \u0026#39;Cadd9/E\u0026#39; Cadd9/E 3maj: E 329.63 1: C 261.63 5: G 392.00 9maj: D 587.33 % harmony -chord \u0026#39;Cadd2\u0026#39; 2022/06/14 13:24:43 unknown chord name: add2 It knows of a few scales and modes and can decompose them into notes, triads and sevenths chords for each degree of the scale:\n% harmony -scale Caeolian C C 261.63 D 293.66 Eb 311.13 F 349.23 G 392 Ab 415.3 Bb 466.16 C 523.25 Triads: Cmin Ddim Ebmaj Fmin Gmin Abmaj Bbmaj Sevenths: Cmin7 Dm7b5 Ebmaj7 Fmin7 Gmin7 Abmaj7 Bb7 % harmony -scale Cphrygian C C 261.63 Db 277.18 Eb 311.13 F 349.23 G 392 Ab 415.3 Bb 466.16 C 523.25 Triads: Cmin Dbmaj Ebmaj Fmin Gdim Abmaj Bbmin Sevenths: Cmin7 Dbmaj7 Eb7 Fmin7 Gm7b5 Abmaj7 Bbmin7 And finally, someone asked if it could build chords from a given set of notes, so I implemented it which took only a couple minutes:\n% harmony -notes \u0026#39;C,E,G\u0026#39; Cmaj C 261.63 E 329.63 G 392 % harmony -notes \u0026#39;C,Eb,G,B\u0026#39; C-M7 C 261.63 Eb 311.13 G 392 B 493.88 % harmony -notes \u0026#39;C,Eb,G,Bb\u0026#39; Cmin7 C 261.63 Eb 311.13 G 392 Bb 466.16 % harmony -notes \u0026#39;C,F,G\u0026#39; Csus4 C 261.63 F 349.23 G 392 That\u0026rsquo;s about all it can do for now, but I have many ideas I want to implement as I have big plans for it :-)\nearmuff # earmuff is a proof-of-concept compiler and interpreter for a programming language to write music.\nAfter I showed the first iteration of the interpreter, two people pointed me to Sonic Pi and it kinda is a similar concept but executed a bit differently. Whereas Sonic Pi\u0026rsquo;s is an advanced scriptable synthesizer with its own IDE, earmuff is simply a language to express music sheet as code and doesn\u0026rsquo;t come with anything but the interpreter and compiler for it.\nMy intent was to be able to write music as code in my usual programming environment, using diff, patch or even git for versionning my changes and applying new changes, and basically work with music in the same way I work with code when I don\u0026rsquo;t have instruments at hands.\nSo earmuff reads .muff source files, and either compiles them to SMF (Standard Midi File) files that can be played with a standard MIDI player, or interprets them into MIDI messages that are sent to a synthesizer to play the source code in real-time.\nIt relies on go-harmony to make sense of notes and chord names, spotting errors as it converts the source code to its internal format, and validating that the tracks are structurally consistent (ie: time signature violations) just as if it was checking for syntax or grammar errors.\nA .muff file may look like this for a single-track 12-bars blues:\nproject { bpm 120; time 4 4; instrument guitar { bar { whole chord C7 on 1/1; } bar { whole chord F7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord F7 on 1/1; } bar { whole chord F7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord G7 on 1/1; } bar { whole chord F7 on 1/1; } bar { whole chord C7 on 1/1; } bar { whole chord G7 on 1/1; } } } \u0026hellip; or may be more complex for multi-tracks projects, like these few bars from Django\u0026rsquo;s Nuages (I know, it\u0026rsquo;s not quite right but hey, it was late and I\u0026rsquo;m still learning the language 😅):\nproject { bpm 80; time 4 4; instrument guitar { bar { 8th note C# on 3/1; 8th note D on 3/2; 8th note A on 3/3; 8th note G# on 3/4; 8th note G on 4/1; 8th note F# on 4/2; } bar { half note F on 1/1; 8th note E on 4/2; } bar { half note Fbb on 1/1; quarter note Eb on 2/2; 8th note D on 4/2; } bar { whole note D on 1/1; } } instrument piano { bar {} bar { whole chord Eb9 on 1/1; } bar { half chord Am7b5 on 1/1; half chord D7b9 on 3/1; } bar { half chord Gmaj7 on 1/1; quarter chord Am7 on 3/1; quarter chord Bm7b5 on 4/1; } } instrument percussive { bar {} bar { half cymbal on 1/1; } bar { half cymbal on 1/1; } bar { half cymbal on 1/1; } } } By default, earmuff operates as an interpreter so running the following command will playback the source if a synthesizer is detected (currently only FluidSynth is):\n% earmuff nuages.muff Your browser does not support the audio element. The -verbose option may be passed, in which case earmuff outputs MIDI messages being sent to the synthesizer in real-time for debugging:\nsynth \u0026lt;- 0 ProgramChange channel: 0 program: 25 synth \u0026lt;- 2 ProgramChange channel: 10 program: 113 synth \u0026lt;- 1 ProgramChange channel: 1 program: 1 synth \u0026lt;- 0 NoteOn channel: 0 key: 49 velocity: 120 synth \u0026lt;- 0 NoteOn channel: 0 key: 50 velocity: 120 synth \u0026lt;- 0 NoteOn channel: 0 key: 57 velocity: 120 synth \u0026lt;- 0 NoteOff channel: 0 key: 49 synth \u0026lt;- 0 NoteOn channel: 0 key: 56 velocity: 120 synth \u0026lt;- 0 NoteOff channel: 0 key: 50 synth \u0026lt;- 0 NoteOff channel: 0 key: 57 synth \u0026lt;- 0 NoteOn channel: 0 key: 55 velocity: 120 synth \u0026lt;- 0 NoteOff channel: 0 key: 56 synth \u0026lt;- 0 NoteOn channel: 0 key: 54 velocity: 120 synth \u0026lt;- 0 NoteOff channel: 0 key: 55 synth \u0026lt;- 0 NoteOff channel: 0 key: 54 synth \u0026lt;- 2 NoteOn channel: 10 key: 51 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 51 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 49 velocity: 120 synth \u0026lt;- 0 NoteOn channel: 0 key: 53 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 55 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 65 velocity: 120 synth \u0026lt;- 1 NoteOn channel: 1 key: 58 velocity: 120 Passing the option -out will cause it to compile a .mid file:\n% earmuff -out nuages.mid nuages.muff \u0026hellip; which can either be played by a standart MIDI player, imported in tools that can ingest MIDI files such as a DAWs (here Logic Pro X):\n\u0026hellip; or tools like Guitar Pro which can render sheet music from a MIDI file, play it back after altering speed, instruments, etc\u0026hellip;\nThis is a toy project at a very early stage, I don\u0026rsquo;t have very serious plans for it but I\u0026rsquo;ll keep working on it as it improves my overall knowledge, and I kinda like the idea of being able to write music as code from my bed while the kid sleeps by my side (also I\u0026rsquo;m far faster as writing code than editing sheets) :-)\nThere are still bugs and glitches in the MIDI generation but the MP3 I linked above was converted from the .mid built by the source code just above it, so it may not be perfect yet but it works pretty decently in my opinion.\nMy current plans are to cleanup the proof-of-concept, make it work on at least macOS, Windows, Linux and OpenBSD, and then extend it with new features such as loops and functions. A very nice feature would be to allow it to read from MIDI and generate the corresponding .muff code, allowing me to auto-generate code from a MIDI input (like a keyboard for example), or simply allowing me to export a SMF from another tool to work on it with earmuff and export it back to the other tool: being able to switch back and forth between VScode and Guitar Pro or Logic Pro X would be awesome.\nWhat\u0026rsquo;s next ? # Next is A DAMN BREAK cause I said last month I\u0026rsquo;d take a couple months and couldn\u0026rsquo;t even hold my word for a single one. I\u0026rsquo;ll be marrying in a couple weeks, going on vacations a few weeks later, and got plenty of code-unrelated things to finish.\nTake care, stay tuned, I\u0026rsquo;ll post as soon as I resume my work !\n","date":"14 June 2022","permalink":"/posts/2022-06-14/june-2022-go-harmony-and-earmuff/","section":"Posts","summary":"TL;DR: did a lot of music, even while writing code. Code-unrelated work # As last month, I\u0026rsquo;ll start with code-unrelated work !\nFirst, here\u0026rsquo;s my progress on learning Hyunsoo Lee\u0026rsquo;s adaptation of Bach\u0026rsquo;s Air on G string, still not quite right and with some missing parts but\u0026hellip; slowly getting there:","title":"June 2022: go-harmony and earmuff"},{"content":" TL;DR: tons of plakar work, most notably on indexes, performances, clone \u0026amp; sync and fuse. Code-unrelated work # I\u0026rsquo;ll start with code unrelated work !\nI\u0026rsquo;ve picked up playing electric guitar and metal after years of playing mostly jazz and classical on nylon-string guitars. After a couple weeks of enduring finger pain while re-conditionning myself to playing on steel strings, I decided to give myself a challenge and learn some shredding, a topic I conveniently always dodged :-)\nI took this track which I absolutely love, Hyunsoo Lee\u0026rsquo;s adaptation of Bach\u0026rsquo;s Air on G string, and started practicing because it uses some techniques I\u0026rsquo;m not too good with such as sweeping and fast runs:\nI still have a lot of work to do, particularly when playing at full speed, but I\u0026rsquo;m very happy with the progress I\u0026rsquo;m making on this so I\u0026rsquo;m sharing :-)\nI\u0026rsquo;m also working on a couple other covers, I\u0026rsquo;ll post when I\u0026rsquo;m happy with them too.\nSplit indexes # As a follow up to refactor from previous month, I continued to work on improving the data structures used by plakar.\nJust as I had split Index and Metadata, I have now also split the Index and Filesystem view of a snapshot so a plakar snapshot now has three indexes: Metadata contains, well, metadatas for lookups of general purpose informations regarding a snapshot, Index contains everything necessary to figure out the structure of files and what\u0026rsquo;s needed to reconstruct it, and Filesystem contains everything necessary to figure otu the structure of snapshotted filesystems.\nBasically, if you want to know the size of a snapshot, Metadata is the only index necessary. If you want to figure out which chunks constitute /etc/passwd, Index is the only index necessary. If you want to figure out which files are in /etc, or what were the permissions of /etc/passwd, Filesystem is the only index necessary. As most command only rely on one of these simultaneously, this speeds up things as commands do not have to wait for the three indexes to be fetched before being able to work.\nNot all commands benefit from this split yet as they need to be adapted, but this will be a transparent change where some commands get a performance boost out of nowhere as I reimplement them properly.\nBinary indexes # Up until last month, I had used json as the storage format for indexes, and there were two reasons for that:\nFirst, it allowed me to not worry too much about serialization. I could simply json-marshal the Metadata, Index and Filesystem indexes which would generate a set of dictionnaries, both usable with a simple json-unmarshal and no state \u0026ldquo;rebuilding\u0026rdquo; (that\u0026rsquo;s converting from a disk format to different structures in memory, imposing a conversion cost).\nThen, it also helped me debug things as I could always use an unencrypted plakar and rely on the jq command to read the indexes, check that they contained the expected data and generally play with them with json-aware tools.\nThe problem is that json serialization and deserialization comes at a cost, both in terms of space consumption as it\u0026rsquo;s far from being a very compact format and snapshot may contain dozens of thousands of objects leading to a loooooooot of waste, but also in terms of performances as the format itself is more costly to serialize and deserialize than binary formats.\nI have ideas where I want to go with serialization of indexes but a good first step was to switch from json to a decent binary format, one that would be more efficient to start with but which would also allow me to ensure that nothing relied on json anymore.\nAfter some investigations, I ran into msgpack which seemed to be a good candidate, was already fairly popular in other projects, but more importantly didn\u0026rsquo;t require too much work to plug in.\nIt claimed to provide much better performances than json serialization, which I didn\u0026rsquo;t observe myself, but helped cut the index sizes considerably as in average it shrank them by 25% and up to 50% in some of my tests.\nCompact indexes # With work being done to split indexes and switch them to a binary format, I decided to stay focused on index improvements and trying to make them as compact as I could.\nThe reason is that indexes weight \u0026ldquo;two\u0026rdquo; sizes: the storage size, when they\u0026rsquo;re written to disk, and the runtime size, when they\u0026rsquo;re loaded into memory. Both these sizes have benefits being optimized, the former allowing faster deserialization, the latter providing a smaller footprint and faster serialization.\nIn an ideal world, the data structures stored on disk are compressed to the smallest possible size to reduce loading time, but the data structures loaded in memory should be as close as possible to their disk counterparts to avoid conversion costs. The optimal case would be that indexes are compressed in such a way that they can be loaded compressed in memory, and usable without decompressing them.\nSo first of all, I did a pass to compress data types as I knew there would be an instant gain from a (seemingly) simple change. The project uses uuid (16 bytes) and sha256 sums (32 bytes) all over the place, but for convenience I used human-readable representations of these formats, that is 36-bytes strings for uuid and 64-bytes strings for sha256. These landed in many data structures in their human-readable representation, meaning that 20 bytes were wasted every time a uuid was stored and 32 bytes were wasted every time a sum was stored. This doesn\u0026rsquo;t look much but that\u0026rsquo;s until you realize that most of what\u0026rsquo;s stored in the indexes are mappings that associate sums to each others, so when a snapshot contains dozens of thousands of files the waste builds up: gain was already quite nice at this point as it halved the size of indexes on the snapshots of my work directory.\nThen, I did another pass to compress index size further. As I wrote above, most of what\u0026rsquo;s stored in indexes are mappings that associate sums to each others, and to achieve that there are several mappings and reverse mappings which tend to repeat the same information in reverse order. To answer questions such as what are the chunks in object X ? or what objects contain the chunk Y ?, you need to have a mapping associating the checksum of object X to a serie of chunk checksums, but also to have a mapping associating the checksum of chunk Y to a serie of object checksums. This means that a ton of 32-byte checksums are repeated duplicated and appear at least twice in indexes. By using a symbol table and an indirection, it is possible to reduce the index sizes by replacing 32-byte checksums with a much smaller and unique 8-byte identifier. Because each checksums appear at least twice, the index shrinks by roughly 50% in the worse case and is effectively compressed in memory. at the cost of a O(1) indirection in the symbol table.\nThese were cumulative so by the end of this round of optimizations, indexes were about 4x times smaller than what they were to start with, and because of how the compression works it also means they don\u0026rsquo;t grow as much as they used to.\nI still have ideas to improve further on indexes compression, but at this point I\u0026rsquo;m satisfied with what I have and the current trade-off ratio between size and convenience. I know I have some improvement margin still but will only use it if absolutely necessary.\nDisplay index, filesystem and metadata # Because the changes above made it harder to investigate the content of indexes, as they are no longer stored in a human-readable format, I implemented a helper command to json-dump their content allowing to inspect them with:\n% plakar info -metadata 39| jq .|grep IndexID \u0026#34;IndexID\u0026#34;: \u0026#34;39d10f01-992f-471c-88f7-c730b003c8b6\u0026#34;, % plakar info -filesystem 39| jq .| grep cmd_pull.go \u0026#34;cmd_pull.go\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;cmd_pull.go\u0026#34;, \u0026#34;/Users/gilles/wip/github.com/poolpOrg/plakar/cmd/plakar/cmd_pull.go\u0026#34;: 16, % plakar info -index 39 | jq . | less [...] \u0026#34;12\u0026#34;: { \u0026#34;Start\u0026#34;: 0, \u0026#34;Length\u0026#34;: 8423 }, \u0026#34;13\u0026#34;: { \u0026#34;Start\u0026#34;: 0, \u0026#34;Length\u0026#34;: 1054 }, \u0026#34;14\u0026#34;: { \u0026#34;Start\u0026#34;: 0, \u0026#34;Length\u0026#34;: 1889 }, [...] They will load the binary format of indexes and dump them to stdout after encoding them to json. It\u0026rsquo;s a bit hard to write about this feature, so feel free to look at the output of these commands by yourself.\nPerformances, profiling and cache # I did something I had not done before: try an alternate solution to plakar and compare runtime execution timings.\nI expected plakar to be a bit slower because it lacks maturity and I didn\u0026rsquo;t spend much time optimizing yet, but I intuitively thought it would be roughly 25% slower with an easily achievable margin of improvement.\nI installed restic and after a few minutes of reading documentation to figure out the command line interface, I ran a couple snapshots\u0026hellip; and was left speechless. It was considerably faster, not by 25% but more like 25x faster. To be more precise, the initial snapshot was about 25% faster than with plakar, which was what I expected it to be\u0026hellip; but subsequent snapshots took only a couple seconds when they could take a minute with plakar. I honestly didn\u0026rsquo;t think they could be so fast.\nAfter a few seconds of contemplation and investigation to make sure that I didn\u0026rsquo;t miss something on the command line and that it actually did the subsequent snapshots, I decided to do a first pass of optimization right away to bring my plakar timings at least in the 25% range. I didn\u0026rsquo;t want to work on optimization right away, because it would be premature with all of the changes still happening, and I don\u0026rsquo;t want them to go in the way of design at this point\u0026hellip; but I also didn\u0026rsquo;t want to create a situation were the design would lock me too far from alternatives performance-wise. I mainly needed reassurance that suboptimal performances remained in the realm of achievable improvements when I\u0026rsquo;ll be tackling them.\nI wrote a runtime profiling mechanism, that you can enable with the -profiling, and which allowed me to track what were the main areas that needed improvements:\n% plakar -profiling push [profiling]: snapshot.PutMetadata: calls=1, min=325.25µs, avg=325.25µs, max=325.25µs, total=325.25µs [profiling]: cache.Commit: calls=1, min=270.75µs, avg=270.75µs, max=270.75µs, total=270.75µs [profiling]: cache.New: calls=1, min=66.272166ms, avg=66.272166ms, max=66.272166ms, total=66.272166ms [profiling]: snapshot.GetCachedObject: calls=26, min=208.584µs, avg=864.285µs, max=1.508667ms, total=22.471418ms [profiling]: snapshot.PutIndex: calls=1, min=1.446208ms, avg=1.446208ms, max=1.446208ms, total=1.446208ms [profiling]: storage.tx.PutIndex: calls=1, min=141.375µs, avg=141.375µs, max=141.375µs, total=141.375µs [profiling]: storage.tx.PutMetadata: calls=1, min=111.917µs, avg=111.917µs, max=111.917µs, total=111.917µs [profiling]: storage.Close: calls=1, min=83ns, avg=83ns, max=83ns, total=83ns [profiling]: storage.CheckChunk: calls=114, min=4.459µs, avg=8.217µs, max=44.708µs, total=936.743µs [profiling]: snapshot.CheckChunk: calls=114, min=5.125µs, avg=9.433µs, max=79.417µs, total=1.075417ms [profiling]: snapshot.CheckObject: calls=26, min=6.291µs, avg=10.828µs, max=46.709µs, total=281.541µs [profiling]: snapshot.Create: calls=1, min=1.63425ms, avg=1.63425ms, max=1.63425ms, total=1.63425ms [profiling]: storage.GetObject: calls=26, min=5.583µs, avg=9.753µs, max=44.709µs, total=253.584µs [profiling]: cache.PutFilesystem: calls=1, min=46.542µs, avg=46.542µs, max=46.542µs, total=46.542µs [profiling]: storage.tx.PutFilesystem: calls=1, min=154.917µs, avg=154.917µs, max=154.917µs, total=154.917µs [profiling]: snapshot.PutFilesystem: calls=1, min=1.157125ms, avg=1.157125ms, max=1.157125ms, total=1.157125ms [profiling]: cache.Create: calls=1, min=12.5µs, avg=12.5µs, max=12.5µs, total=12.5µs [profiling]: storage.Open: calls=1, min=131.042µs, avg=131.042µs, max=131.042µs, total=131.042µs [profiling]: storage.Transaction: calls=1, min=1.625625ms, avg=1.625625ms, max=1.625625ms, total=1.625625ms [profiling]: cache.PutMetadata: calls=1, min=12.667µs, avg=12.667µs, max=12.667µs, total=12.667µs [profiling]: snapshot.Commit: calls=1, min=3.517208ms, avg=3.517208ms, max=3.517208ms, total=3.517208ms [profiling]: cache.GetPath: calls=26, min=175.792µs, avg=778.532µs, max=1.490375ms, total=20.241836ms [profiling]: cache.PutIndex: calls=1, min=113.416µs, avg=113.416µs, max=113.416µs, total=113.416µs [profiling]: storage.tx.Commit: calls=1, min=132.042µs, avg=132.042µs, max=132.042µs, total=132.042µs % This made me realize that this very high performance gap was caused by a major difference in what it did compared to restic. It attempted to maintain repository consistency with every operation, using complex sequences of operations to maintain correct reference counts, ensuring that objects were deleted as soon as they were no longer referenced or when a push was aborted, etc\u0026hellip; while restic didn\u0026rsquo;t take care of that with every operation but used a prune command to do the cleanup.\nI really thought it would be neat to keep the repository consistency the way plakar did, but it became obvious that the cost was too prohibitive and no amount of optimizations I could think of would counterweight the cost of this. All operations were made more complex and the cost of consistency was paid with every call, when restic benefited from simpler operations and the cost of consistency was paid only when actually requested.\nI backtracked on my ideas and reimplemented plakar\u0026rsquo;s push using a much simpler strategy, removing all attempts at playing hard-links and reference counts games to track and cleanup, but implementing a cleanup operation like restic did.\nThis immediately brought a huge improvement as plakar remained in the range of 25% for the first push (with some gain though), but immediately fell to the same performances as restic for subsequent pushes. This looks promising because while it has roughly the same timings, unlike restic, plakar uses compression by default and didn\u0026rsquo;t go through actual optimization yet. This means that performances will likely put it below restic in terms of push times, while repository consumption is already much better when snapshots are compressible. I played a lot with pushing my ~2GB work directory and for the same push time, it would weight ~2GB on restic and only 350MB on plakar.\nWhile I was there, I had a look at the repository structure by poking at the created directories and realized that despite a few differences both were very, very, very close. This gave me confidence that I was not completely off with how I tackled this project :-)\nThe profiler also helped me realize that there was a bug in the implementation of the local cache. It is implemented using SQLite and I didn\u0026rsquo;t create the proper indexes, which caused cache accesses to be very costly and negated the benefits of using one as soon as there were a lot of data in it. After fixing, things were much better but I came to the conclusion that SQLite was not the proper tool for my problem, what I really need is a very simple key-value database. I temporarily replaced SQLite with LevelDB for the sake experimenting, but am not too happy as it imposes a lock that makes the cache unusable for reads when a push is in progress. For the time being I made it so the cache is skipped while a push is in progress, I\u0026rsquo;ll be looking at a different solution that allows reads of older versions of the data when a write is in progress.\nParallelized pull # The plakar pull operation was not parallelized so while a push could benefit from multiple cores, pull would restore snapshot content sequentially and one file at at time.\nI rewrote it so that it would both take advantage of the snapshot Reader interface I wrote about last month, simplifying code, but also so that it could parallelize restoring and speed things up.\nThere are other things that can be done to speed up things, like not restoring files which are already there in their last version in the target directory, or looking at stat structure to figure out if some files are hard links to others. Work still needs to be done but it\u0026rsquo;s already a better version so\u0026hellip; meh, no rush.\nAdded a progress bar # That\u0026rsquo;s a minor change but one that I really wanted to do and which someone asked me for, prompting me to start working on it.\nI implemented the -progress option for plakar push, plakar pull and plakar check, but I\u0026rsquo;m not too happy with the way I achieved it. I\u0026rsquo;ll revisit later but thought I\u0026rsquo;d leave it as is for now as it\u0026rsquo;s still more helpful than not having it.\nReworked clone and sync # OK, this is by far the most interesting feature in my opinion.\nThe goal of plakar clone is to create an exact copy of an existing repository, sharing its repository ID and configuration, including encryption key.\nThe goal of plakar sync is to synchronize two repositories by transfering the chunks and objects that are missing in the target repository, effectively making an exact copy of a snapshot available in a different repository.\nI had already implemented plakar clone and plakar sync but they both had shortcomings. First, they both only supported working with local repositories, ones available on the local filesystem. Then, sync only supported synchronizing to clone repositories or between unencrypted repositories, but it didn\u0026rsquo;t support synchronizing an unencrypted repository with an encrypted one for example. This was nice already but what I really wanted was for this to work regardless of where a repository is, and regardless of the configuration of both repositories.\nI reworked both commands so that they know support working with arbitrary repositories, and doing the necessary work so there\u0026rsquo;s no issue when synchronizing repositories not sharing the same configuration.\nWhy is this the most interesting feature in my opinion ?\nWell what I really want with backups is the ability to have them at multiple locations, preferably in different regions. I\u0026rsquo;ve already lost a few servers at cheap hosts where no disk recovery was offered, and I know of at least one DC that burnt and lost data of all local servers in the process, killing backups that were located on different machines of the same DC. I want to be able to spread backups at multiple locations but I don\u0026rsquo;t want to have to think too hard about that.\nFurthermore, I\u0026rsquo;m not necessarily interested in having ALL backups encrypted, nor synchronized at the same rate. For instance, I like having backups on my local machine readily available in case I mistakenly trash something, I like having unencrypted backups on my NAS at home in case a machine is unrecoverable, but I\u0026rsquo;d also like having off-site encrypted backups at one or two cheap hosts in case there was a fire at home and I lost everything. Not all these backups need to happen at the same frequency and I don\u0026rsquo;t want to have complex strategies here and there, depending on the machine. I can do that with several solutions and a bunch of scripting, but I\u0026rsquo;ve been there, done that, and it was never fun.\nWhat clone and sync allow me to do is to have exactly what I described in just a few commands, I can create an unencrypted repository on my laptop:\n% plakar on ~/plakar create -no-encryption Then create a clone on my NAS:\n% plakar on ~/plakar clone to ssh://nas.poolp.org Both repositories are absolutely identical at this point.\nMy laptop can do a plakar push multiple times a day from a cron, then every few hours I can synchronize both repositories:\n% plakar on ~/plakar sync to ssh://nas.poolp.org which brings the NAS repository up-to-date with the local repository. I could also synchronize the other way. Say I had multiple machines pushing to the NAS and I wanted my local repository to grab all snapshots, then I\u0026rsquo;d just:\n% plakar on ~/plakar sync from ssh://nas.poolp.org which brings the local repository up-to-date with the NAS repository. Just like I could clone the NAS repository to bring back an exact copy if I had reinstalled my machine:\n% plakar on ~/plakar clone from ssh://nas.poolp.org But where I find it very useful is that I could create encrypted repositories remotely:\n% plakar on ssh://backups.poolp.org create password: and synchronize either my NAS or laptop with them:\n% plakar on ~/plakar sync to ssh://backups.poolp.org password: % plakar on ssh://nas.local sync to ssh://backups.poolp.org password: The -keyfile option can be provided to point to a file containing the password so it doesn\u0026rsquo;t need to be typed, and allows calling sync from a cron job.\nWith all of this, it makes it trivial to have multi-site backups all synchronized one with another at different rates.\nReworked snapshot Reader # Last month, I wrote about how I implemented a snapshot Reader interface and how it helped unlock features and simplify code.\nThe next feature I\u0026rsquo;ll write about required me to rework the Reader interface. A Reader interface is one that allows a Read() operation, but I really needed a ReadSeekCloser interface, one that allows Read(), Close() and more importantly Seek() so that I can move the read cursor to arbitrary locations in a stream.\nI wrote the interface to implement Seek() on an object so that it would locate the appropriate chunk and maintain an offset to point at the proper position. This didn\u0026rsquo;t take too long as I did it in an hour or so, but it was painful and I lost my soul in the process.\nFUSE proof-of-concept # FUSE is a \u0026ldquo;Filesystem in USEerspace\u0026rdquo; mechanism which allows creating mountpoints that expose informations as if they were files on the filesystem. It\u0026rsquo;s not available on all systems, but when it is, fun stuff are available to play with.\nI implemented experimental FUSE support in plakar, making it possible to mount a repository on the filesystem and browse it as if all snapshots had been restored there.\nWhat this means is that I can do:\n% plakar on ssh://nas.local mount /mnt/nas Then I can browse just as if all the repository had been restored below /mnt/nas:\n% ls -l /mnt/nas total 5012576 drwx------ 1 gilles staff 41865948 22 May 22:21 94c815c7-15c6-4a8a-af3a-a83ca620fb55 drwx------ 1 gilles staff 41865948 22 May 22:22 a961b301-e331-4fd1-859e-7ae1d6fca17f drwx------ 1 gilles staff 2482699293 22 May 22:22 ade656c9-56e7-4e84-b90f-90b3826b08e2 % cd /mnt/nas/94c815c7-15c6-4a8a-af3a-a83ca620fb5 % ls Users % cd Users/gilles/Wip/github.com/poolpOrg/plakar/cmd/plakar % ls -l total 51216 -rw-r--r-- 1 gilles staff 1426 7 May 13:58 cmd_browser.go -rw-r--r-- 1 gilles staff 2274 11 May 21:53 cmd_cat.go -rw-r--r-- 1 gilles staff 1748 6 May 23:31 cmd_check.go -rw-r--r-- 1 gilles staff 1930 10 May 09:09 cmd_checksum.go -rw-r--r-- 1 gilles staff 1973 6 May 23:31 cmd_cleanup.go -rw-r--r-- 1 gilles staff 4590 6 May 23:31 cmd_clone.go -rw-r--r-- 1 gilles staff 2637 6 May 23:31 cmd_create.go -rw-r--r-- 1 gilles staff 7124 6 May 23:31 cmd_diff.go -rw-r--r-- 1 gilles staff 2582 6 May 23:31 cmd_exec.go -rw-r--r-- 1 gilles staff 2818 6 May 23:31 cmd_find.go -rw-r--r-- 1 gilles staff 8423 6 May 23:31 cmd_info.go -rw-r--r-- 1 gilles staff 1889 6 May 23:31 cmd_keep.go -rw-r--r-- 1 gilles staff 7776 6 May 23:31 cmd_ls.go -rw-r--r-- 1 gilles staff 1784 7 May 13:55 cmd_mount.go -rw-r--r-- 1 gilles staff 2393 6 May 23:31 cmd_pull.go -rw-r--r-- 1 gilles staff 1992 6 May 23:31 cmd_push.go -rw-r--r-- 1 gilles staff 1679 6 May 23:31 cmd_rm.go -rw-r--r-- 1 gilles staff 1228 6 May 23:31 cmd_server.go -rw-r--r-- 1 gilles staff 1684 6 May 23:31 cmd_shell.go -rw-r--r-- 1 gilles staff 1054 6 May 23:31 cmd_stdio.go -rw-r--r-- 1 gilles staff 5993 6 May 23:31 cmd_sync.go -rw-r--r-- 1 gilles staff 3096 6 May 23:31 cmd_tarball.go -rw-r--r-- 1 gilles staff 1147 6 May 23:31 cmd_version.go -rwxr-xr-x 1 gilles staff 26080594 12 May 23:14 plakar -rw-r--r-- 1 gilles staff 7562 8 May 21:59 plakar.go -rw-r--r-- 1 gilles staff 9113 6 May 23:31 utils.go % cat cmd_version.go /* * Copyright (c) 2021 Gilles Chehade \u0026lt;gilles@poolp.org\u0026gt; * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED \u0026#34;AS IS\u0026#34; AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ package main import ( \u0026#34;flag\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;github.com/poolpOrg/plakar/storage\u0026#34; ) const VERSION = \u0026#34;0.0.1\u0026#34; func init() { registerCommand(\u0026#34;version\u0026#34;, cmd_version) } func cmd_version(ctx Plakar, repository *storage.Repository, args []string) int { flags := flag.NewFlagSet(\u0026#34;version\u0026#34;, flag.ExitOnError) flags.Parse(args) fmt.Println(VERSION) return 0 } % Note that despite cat showing the content of a file here, files aren\u0026rsquo;t really on the filesystem but streamed from the repository using the ReadCloseSeeker interface. As such, even though I can browse all the snapshots of the repository and access any file content, they don\u0026rsquo;t consume disk space locally, my 256GB laptop could very well mount a repository storing far more than that and allow me to browse and cat any file.\nAlso note that I mounted a remote repository located at ssh://nas.local which could very well be encrypted as well, this would work just as fine and transparently. This is experimental as I had no prior experience playing with FUSE drivers, there can be a few glitches here and there, I fixed all the ones I found but still.\nConditional builds # I added support for conditional builds.\nExperimental FUSE support only works on Linux and macOS as far as I know, and doesn\u0026rsquo;t work for sure on OpenBSD for reasons that I\u0026rsquo;ve started investigating but which are beyond plakar, so I wanted to make sure that I could still build plakar with the experimental feature removed where it was known not to work\u0026hellip; rather than break the entire build.\nWhile at it, the browser feature prevented plakar from building on older Go version, such as the one shipped on some Ubuntu machines I administer for friends. I made sure that this conditionally built depending on the version of Go.\nFull text search experiment # I quickly experimented with FTS in plakar, allowing me to search for all snapshots and files containing a particular content:\n% plakar search PutIndex 9 matches, showing 1 through 9, took 80.125µs 1. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/storage.go (0.406516) 2. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/cache/cache.go (0.362482) 3. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/network/server.go (0.277187) 4. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/snapshot/snapshot.go (0.264234) 5. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/client/client.go (0.262951) 6. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/database/database.go (0.254370) 7. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/fs/fs.go (0.238110) 8. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/poolp.org/content/posts/2021-10-01/index.md (0.211895) 9. 46a6777c-8326-48e8-bc8e-4ebfc46c36b4:/Users/gilles/wip/github.com/poolpOrg/poolp.org/docs/posts/2021-10-26/october-2021-mostly-plakar/index.html (0.143937) % plakar pull 46a6777c:/Users/gilles/wip/github.com/poolpOrg/plakar/storage/storage.go % ls Users/gilles/wip/github.com/poolpOrg/plakar/storage/storage.go storage.go % This was not committed and I have no plans to commit it for the time being, a lot of work still needs to be done before I focus on such features, but it shows how plakar can be improved in ways that my current rsync/tar backups will never match.\nS3 experiment # I also did a small experiment writing an s3 backend for plakar, allowing it to store repositories on Amazon s3 or on a minio server.\nI\u0026rsquo;m not too comfortable with the API as I never used s3 before, surely I\u0026rsquo;m not using it correctly, but someone asked me if this would be supported and so I gave it a try.\nAfter about half an hour, I had a plakar repository hosted on a local s3:\n% plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar create -no-encryption % plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar push /private/etc 2\u0026gt;/dev/null % plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar ls 2022-05-08T21:19:14Z 21388086 3.1 MB 0s /private/etc % plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar ls 21:/private/etc/passwd 2022-03-26T07:21:13Z -rw-r--r-- root wheel 7.9 kB passwd % plakar on s3://minioadmin:minioadmin@localhost:9000/my-s3-plakar cat 21:/private/etc/passwd | tail -3 _darwindaemon:*:284:284:Darwin Daemon:/var/db/darwindaemon:/usr/bin/false _notification_proxy:*:285:285:Notification Proxy:/var/empty:/usr/bin/false _oahd:*:441:441:OAH Daemon:/var/empty:/usr/bin/false % This will likely need more work but it was a successful proof of concept :-)\nWhat\u0026rsquo;s next ? # Next is a couple months of rest.\nI\u0026rsquo;ll be marrying at the end of June and there are still a lot of things to do, I doubt I\u0026rsquo;ll find much spare time to write code this month\u0026hellip; just as I think the month of July will be needed to recover as a wedding is a lot to deal with for someone with an anxiety disorder :-)\nStay tuned, I\u0026rsquo;ll post as soon as I resume my work !\n","date":"22 May 2022","permalink":"/posts/2022-05-22/may-2022-yet-again-tons-of-plakar-stuff/","section":"Posts","summary":"TL;DR: tons of plakar work, most notably on indexes, performances, clone \u0026amp; sync and fuse. Code-unrelated work # I\u0026rsquo;ll start with code unrelated work !","title":"May 2022: yet again, tons of plakar stuff "},{"content":" TL;DR: I refactored internal structures to split metadata from the index, implemented an stdio server and finally added SSH support. The plakar.io website # A project needs a website so\u0026hellip; I published the plakar.io website.\nThis will centralize instructions and documentation for the plakar project, but it is still a work in progress as development still happens at a fast pace that\u0026rsquo;s hard to keep up with.\nThe website is published from commits to the plakar.io Github repository, so feel free to submit pull requests to improve it.\nReworked internal structures # A snapshot generates an index that contains everything needed to map chunks and objects into a filesystem hierarchy. The index is very small comparatively to the snapshot size, but it may still be relatively large as a multi-gigabyte snapshot may produce a multi-megabyte index, which still requires time to fetch, decompress and deserialize. In many cases, this is fast enough that you don\u0026rsquo;t really feel it, but the bigger the snapshot the more laggy it feels when browsing it through the UI or when listing content in a snapshot.\nFor many commands there\u0026rsquo;s no need to access the index because it\u0026rsquo;s not so much the content of files that is needed, but really some metadata or statistics. Good examples of these are plakar ls in the repository or plakar info:\n% plakar ls 2022-04-19T22:07:03Z a9d01365 2.4 GB /Users/gilles/Wip 2022-04-19T22:08:11Z f87bd54e 2.4 GB /Users/gilles/Wip 2022-04-19T22:11:46Z 70cc55f6 2.4 GB /Users/gilles/Wip 2022-04-20T22:48:55Z 1c537551 2.4 GB /Users/gilles/Wip % plakar info a9 Uuid: a9d01365-4f97-42e1-8b19-59e4fe2f630a CreationTime: 2022-04-20 00:07:03.728327 +0200 CEST Version: 0.1.0 Hostname: desktop.local Username: gilles CommandLine: ./plakar -no-cache push /Users/gilles/Wip MachineID: 3657f7dd-c012-53ba-b8e6-73e08d311a6a PublicKey: Directories: 7589 Files: 77206 NonRegular: 84 Pathnames: 77206 Objects: 63306 Chunks: 65702 Size: 2.4 GB (2445114798 bytes) Index Size (uncompressed): 113 MB (2445114798 bytes) Here, the index for snapshot a9d01365 is 113MB uncompressed (really 21MB on disk), which means that the plakar ls above required decompressing and deserializing roughly 4 times that size to list the 4 snapshots, then decompressing and deserializing that size once for the plakar info above, just to display some informations that didn\u0026rsquo;t rely on the filesystem hierarchy and content.\nI have split snapshots into two main structures: Metadata and Index.\nThe first structure contains general informations and statistics regarding the index, just enough that the structure fits in a few KiloBytes, but can avoid relying on the much larger index for commands that don\u0026rsquo;t involve diving into the content of a snapshot. The second structure contains the mappings themselves, and is only accessed when data needs to be reconstructured somehow.\nThis allowed plakar ls to get a serious boost when listing a large number of large snapshots, but it is really a first step as I\u0026rsquo;m evaluating solutions to shard the index and allow fetching subsets of it.\nImplemented a Reader interface # When a snapshot is created, plakar will consider each file as an object consisting of one or many chunks. The repository stores the object structure and chunks as separate resources, and so when a file is being recovered this is done by first fetching the structure, then fetching each chunk.\nFor example, to implement plakar cat, the code would look something like this (error handling omitted for simplicity):\n[...] object := snapshot.LookupObjectForPathname(pathname) for _, checksum := range object.Chunks { data, _ := snapshot.GetChunk(checksum) os.Stdout.Write(data) } [...] As every call may fail and needs to be checked, the code is actually far more verbose as can be seen below:\n[...] object := snapshot.LookupObjectForPathname(pathname) if object == nil { logger.Error(\u0026#34;%s: could not open file \u0026#39;%s\u0026#39;\u0026#34;, flags.Name(), pathname) continue } for _, checksum := range object.Chunks { data, err := snapshot.GetChunk(checksum) if err != nil { logger.Error(\u0026#34;%s: %s: could not obtain chunk \u0026#39;%s\u0026#39;: %s\u0026#34;, flags.Name(), pathname, checksum, err) continue } _, err = os.Stdout.Write(data) if err != nil { logger.Error(\u0026#34;%s: %s: could not write chunk \u0026#39;%s\u0026#39; to stdout: %s\u0026#34;, flags.Name(), pathname, checksum, err) break } } [...] and this construct needs to be replicated everywhere a file is accessed which is\u0026hellip; pretty much everywhere you do something with a snapshot.\nI thought it would be nice to have a Reader interface abstracting the details of reconstructing a file. This way, I could simply obtain a reader to a file and call Read() to obtain the next bytes without worrying about fetching chunks as the cursor advances:\nrd, _ := snapshot.NewReader(pathname) buf := make([]byte, 16*1024) for { nbytes, err := rd.Read(buf) if err == io.EOF { break } os.Stdout.Write(data[:nbytes]) } In the example above, snapshot.NewReader(pathname) fetches the object structure for file located at pathname. As rd.Read() is called, it takes care of fetching chunks as needed to fill buf with the next 16k bytes.\nHow is that simpler than calling GetObject() and GetChunk() ?\nWell, first and foremost, there\u0026rsquo;s a large range of functions already available in Go to do what I want but which expect a Reader interface. So rather than rolling my own version of functions, by implementing the Reader interface, I can rely on existing stuff and reduce the amount of code I have to maintain in plakar.\nRather than re-implementing a loop to read chunks and write them to stdout, plakar cat could take advantage of the new Reader interface and the fact that os.Stdout is a Writer, allowing it to use the io.Copy() function part of the standard library to copy from a Reader to a Writer:\nrd, err := snapshot.NewReader(pathname) if err != nil { logger.Error(\u0026#34;%s: %s: %s\u0026#34;, flags.Name(), pathname, err) continue } _, err = io.Copy(os.Stdout, rd) if err != nil { logger.Error(\u0026#34;%s: %s: %s\u0026#34;, flags.Name(), pathname, err) continue } This is also the case for plakar tarball, which creates a tarball from a snapshort, and relies on the standard libraries\u0026rsquo; archive/tar. The code instantiates a tar.NewWriter() and I previously had something like:\nobj := snapshot.LookupObjectForChecksum(checksum) for _, chunkChecksum := range obj.Chunks { data, err := snapshot.GetChunk(chunkChecksum) if err != nil { logger.Error(\u0026#34;corrupted file %s\u0026#34;, file) continue } _, err = io.WriteString(tarWriter, string(data)) if err != nil { logger.Error(\u0026#34;could not write file %s\u0026#34;, file) continue } } which could be rewritten as follows, leaving it up to the standard library to loop and handle do the read and writes:\nrd, err := snapshot.NewReader(file) if err != nil { logger.Error(\u0026#34;could not find file %s\u0026#34;, file) continue } _, err = io.Copy(tarWriter, rd) if err != nil { logger.Error(\u0026#34;could not write file %s: %s\u0026#34;, file, err) continue } In commands such as plakar diff which require copying the full data to a buffer, it also makes it possible to rely on existing functions like ioutil.ReadAll():\nbuf := make([]byte, 0) rd, err := snapshot.NewReader(filename) if err == nil { buf, err = ioutil.ReadAll(rd) if err != nil { return } } In the UI, this is even nicer as it uses the standard libraries\u0026rsquo; http server which uses a Writer for the response to the client. Implementing a download endpoint boils down to using the new Reader interface, then copying it to the Writer rather than doing the whole fetch object then chunks and copy them in a loop logic.\nBasically, this simplifies a TON of areas in plakar and allows me to remove a lot of custom code for things that exist in the standard library.\nFixed handling of max concurrent files # During the last few months, a lot of work was poured into parallelizing operations in plakar to utilize resources as efficiently as possible and speed things up.\nThis led to some issues when processing large directories as you could end up opening a very large amount of files, sometimes exceeding the number of descriptors available to the process and resulting in EMFILE errors.\nMy initial thought was that I could simply implement a backoff mechanism, but this is complex and in the case of plakar it doesn\u0026rsquo;t really make sense: a backoff mechanism is useful as a mean to recover from bursts that cause resources exhaustion. If plakar is going to parallelize and constantly hit the backoff mechanism, it hints that the solution should be located elsewhere.\nContrarily to a network daemon which could accept a very large number of clients and take advantage of idle times from some to process others, plakar only opens files when it will actively read or write them, so it makes no sense opening files when it can\u0026rsquo;t process them. If it opens 1024 files but can only chunk concurrently 32 files before it uses all of the CPU time available, then having 992 opened files waiting to be processed is pointless. I decided to tie the number of descriptors to a factor of the number of cores available, currently 2 * n + 1, and this ensures that the number of descriptors is low enough not to provoke an EMFILE while high enough to keep cores busy. The factor may be adjusted in the future but this feels like the proper way to handle this.\nThis doesn\u0026rsquo;t solve the ENFILE errors, where the system descriptor table is full while the process itself has not hit a limit, but I\u0026rsquo;m wondering if it\u0026rsquo;s even worth having a backoff mechanism to retry as I don\u0026rsquo;t see how a backup tool can really recover from a system not being able to reliably allocate descriptors. I\u0026rsquo;d personally rather have the backup fail and restart it when the system is in better shape, than have a backup take a lot more time to complete and then not trust that backup and do another one just in case some failures where not handled properly by plakar. I\u0026rsquo;m still not decided on this, maybe I\u0026rsquo;m wrong about it, but I need more time to think about it.\nAnyways, with this change I was able to produce large snapshots that previously hit a too many open files error, without observing any significant performances degradation and while still using cores efficiently.\nplakar stdio server # I had mentionned my work on remote repositories in this post, but didn\u0026rsquo;t communicate much about it because it was mainly a proof of concept to validate that no repository primitive expected locality of the repository, not a serious attempt at writing the final network mode.\nRemote repositories work thanks to a TCP server accepting the repository primitives and mapping them to a local repository, the client is basically a proxy which doesn\u0026rsquo;t run the primitives locally but passes them to the server and waits for the result.\nBecause the server logic is isolated in a client handler function, I thought it would be nice if I could abstract the network layer and consider that the client handler could work using stdio rather than a network connection. Instead of passing the client handler a descriptor to a network connection:\nfunc handleConnection(conn *net.Conn) { [...] } func Server(repository *storage.Repository, addr string) { [...] go handleConnection(c) [...] } I converted it to using io.Reader and io.Writer interfaces and passed the same network connection for both:\nfunc handleConnection(rd io.Reader, wr io.Writer) { [...] } func Server(repository *storage.Repository, addr string) { [...] go handleConnection(c, c) [...] } This worked nicely, so I took it a step further and created a new entry point plakar stdio which would call the client handler but passing os.Stdin as the Reader and os.Stdout as the Writer.\nfunc handleConnection(rd io.Reader, wr io.Writer) { [...] } func Stdio() error { ProtocolRegister() handleConnection(os.Stdin, os.Stdout) return nil } It meant that I could now fork plakar stdio from a parent process and communicate with it through pipes or socketpairs. I did a few tests and it worked fine which made me very happy.\nHow is this any useful, you ask ?\nplakar over SSH # I\u0026rsquo;ve been willing to add plakar over SSH support for a while but couldn\u0026rsquo;t set my mind on the simplest way to do it. I did various experiments which were all either too complex or that led to unsatisfying experiences: if I use SSH, I want the full SSH experience with known hosts, authorized keys, public key authentication, etc\u0026hellip; so all attempts that involved implementing an SSH client myself either led to missing features or to a TON of code unrelated to plakar.\nHowever\u0026hellip;\nWith plakar stdio things are much simpler because I can simply use the ssh client to spawn a remote plakar stdio, then take advantage of the fact that the client will do the stdio mapping and expose the ssh transport through stdin and stdout for me:\n% ssh gilles@backups.poolp.org \u0026#39;plakar stdio\u0026#39; Using this method, plakar does not need to have a daemon running on the remote end, it just needs the command installed.\nSo I implemented the ssh:// protocol for plakar which simply forks a process for ssh and emits packets to its stdin while reading responses from its stdout. Because it uses the ssh command, I no longer have to worry about known hosts \u0026amp; such, they are already handled, and I can even benefit from having used ssh-agent so I don\u0026rsquo;t have to keep typing my passphrase:\n% plakar on ssh://backups.poolp.org ls 2022-04-24T08:18:04Z 9bca77d9 3.1 MB /private/etc % plakar on ssh://backups.poolp.org push /bin % plakar on ssh://backups.poolp.org ls 2022-04-24T08:18:04Z 9bca77d9 3.1 MB /private/etc 2022-04-24T14:54:12Z 02895414 13 MB /bin % It took me hours to implement various SSH proof of concepts, people on the discord channel have endured me quite a lot, but this last version\u0026hellip; took only two minutes to implement using a few lines of code.\nIt was both very satisfying and very frustrating :-)\nRemote creation of repositories # Until yesterday, using a remote plakar expected you to create the repository on the remote end before starting to use it.\nI implemented the creation primitives both on the server and client and so it is now possible to do as follows:\n% plakar on ssh://backups.poolp.org create -no-encryption % plakar on ssh://backups.poolp.org push /bin % plakar on ssh://backups.poolp.org ls 2022-04-24T15:02:48Z 181702cd 13 MB /bin % % plakar on ssh://backups.poolp.org/tmp/plakar create -no-encryption % plakar on ssh://backups.poolp.org/tmp/plakar push /bin % plakar on ssh://backups.poolp.org/tmp/plakar ls 2022-04-24T15:03:37Z 8a2ab608 13 MB /bin % Note that -no-encryption is only used here to simplify the examples, dropping the flag will allow creating encrypted repositories remotely.\nWhat\u0026rsquo;s next ? # Not really decided on my next tasks as I have multiple areas that need to be improve, but a good candidate is the plakar sync command which currently only synchronizes clone repositories (same configuration, either unencrypted or both encrypted with same key). I\u0026rsquo;d like to improve it so it can synchronize snapshots with arbitrary repositories, allowing to synchronize an unencrypted snapshot with an encrypted repository and encrypting chunks on the fly.\nI\u0026rsquo;m also working on two new projects, which I won\u0026rsquo;t disclose for now, so I may be going back and forth depending on the mood.\n","date":"24 April 2022","permalink":"/posts/2022-04-24/april-2022-plakar.io-plakar-refactor-and-ssh-support/","section":"Posts","summary":"TL;DR: I refactored internal structures to split metadata from the index, implemented an stdio server and finally added SSH support. The plakar.io website # A project needs a website so\u0026hellip; I published the plakar.","title":"April 2022: plakar.io, plakar refactor and ssh support"},{"content":" TL;DR: I did key and UI stuff, mostly search related. There\u0026rsquo;s going to be plenty of images in this post. Reworked plakar keys # I won\u0026rsquo;t expand much on that as I\u0026rsquo;m not done yet, but I spent a couple days reworking the way plakar handles keys for encrypted repositories.\nWhen a user creates its first encrypted repository, plakar will complain that a key needs to be generated before encryption can be used. The key is really a bundle, identified by a UUID, and containing both a key-pair and a master key, the former being used for signing purposes and the latter to do the actual encryption within a repository. Once generated, if an encrypted repository is created, it will record the bundle UUID in its configuration so that a client can immediately know if it possesses the proper bundle to work with that repository.\nThis was fine to bootstrap the project, but there are some issues with that approach.\nFirst, because the master key is in the same bundle as the key-pair, they become too tied and a user can\u0026rsquo;t use the same key-pair to work with repositories encrypted with different master keys. This meant that either all repositories had to be encrypted using the same master key, or that a user had to generate as many key bundles as repositories encrypted with different keys.\nThen, for the reasons above, it was difficult to allow multiple users to work over the same repository despite the fact that it\u0026rsquo;s been designed for that. If I wanted to give two SREs access to the same repository, then they\u0026rsquo;d need to share the same bundle which would prevent proper auditing and making it very painful to revoke an access\u0026hellip; unless different bundles would share the same master key and the repository configuration recorded a master key UUID rather than a bundle UUID, which would still tie key pairs to a specific master key.\nThere are many ways to improve this but they all share the same initial step: breaking the relation between the key pairs and the master keys used by repositories, which is what I did. The master key for a repository is now generated when the encrypted repository is created, with no ties whatsoever to the key pair, and different users may share the same master key while having different key pairs.\nI\u0026rsquo;ll detail this in a future post as there\u0026rsquo;s still some work to do, however I have successfully created multiple repositories relying on different master keys and worked with them transparently, just as I have shared them between two different users and worked with them transparently as well despite different key pairs. The first step towards a proper solution is done.\nTONS of UI improvements # I work a lot with the CLI and I find it very easy to use for creating or restoring snapshots, as well as performing a lot of small operations when I know what I want to do, but sometimes what I really want to do is browse around a bunch of old snapshots and it\u0026rsquo;s not necessarily as user-friendly.\nA while back, I wrote the plakar ui command which launches an interface that allows browsing through snapshots, obtaining a lot of informations regarding metadata or objects structures, as well as providing a basic viewer.\nAs time passes, the UI is increasingly usable and friendly, but it still lacks some interesting features.\nSearchable repository # A feature that I believe is very important is the ability to search through a repository or snapshot.\nThere was already a bit of commented code to do that, but it was very limited as it only allowed searching for path names that contained a particular string, however snapshot indexes contain a lot of useful informations that I could make use of.\nSo the first step was to uncomment and bring back the basic search capability, which allows to lookup a directory of file by name:\nI then added the ability to search files matching a particular kind of data, like for instances searching only files that are applications, images, videos, audio, text, or other kinds:\nWhich I then extended to allowing the search of files matching an exact content type, like for instance searching specifically for PDF or JPEG files:\nAnd finally, I implemented a filter on file extensions, because quite often what I\u0026rsquo;m looking for is really a .go or .py file which I forgot the name of:\nAt this point, the search is global to a repository and looks into every snapshot, but the next step is to implement a context-aware search so that it is possible to search from within a snapshot or even within a directory within a snapshot. I\u0026rsquo;ll also be adding a few more useful search criteria, like being able to filter on users or dates.\nAt some point, I\u0026rsquo;ll be working on a far more advanced full-text search, I know how to do it so that it can work with plakar regardless of encryption and locality of the repository, but this heavy work so I won\u0026rsquo;t start before I\u0026rsquo;m already happy with the basic features.\nImproved object viewer # The UI has a page for each file that\u0026rsquo;s part of a snapshot, and that page not only lists metadata regarding the object but also provides a small viewer for text and images:\nThe viewer has a special case so that it only tries to display raw text/* and image/*, as I don\u0026rsquo;t really want the content of a binary file displayed. It worked nicely but I thought it was a bit sad that other types, like PDF for example, didn\u0026rsquo;t get a chance be properly rendered when the browser knows how to do it. I made a few changes to the code and I now have this rendering properly done for PDF files\u0026hellip;\n\u0026hellip; which is rendered just as if it had been downloaded from a website, since that\u0026rsquo;s exactly what it does :-)\nHTML files are very interesting as they resolve local links within the snapshot itself. I have pushed a copy of the OpenBSD\u0026rsquo;s website and the page below renders the index.html from the openssh/ subdirectory, which uses the CSS and image that are part of the snapshot and renders just as fine as from a regular directory.\nThis makes it very nice to browse backups of websites and see their different versions :-)\nSince I\u0026rsquo;m a developer and rely a lot on syntax highlighting, I took an extra step and added support for that too:\nIt doesn\u0026rsquo;t show above because I took these screenshots before, but I made a change to the UI so that I can decide if I want a file rendered raw or highlighted, when that makes sense.\nFor instance, the HTML example above had the page rendered raw but a button allows toggling between raw and displaying the HTML source code highlighted.\nfailed experiment with statistics\u0026hellip; # I did a failed experiment with statistics, trying to display doughnut charts about file types, repartition, duplication and other interesting informations, but it turned out that with large snapshots the amount of data makes these unreadable\u0026hellip;\n\u0026hellip; so I gave up on that idea and took a different approach.\nInstead, I added columns to tables containing categories of objects and display proportions there. It is less sexy than charts but is readable and I can think about displaying small charts that showcase a specific category vs all others to make it more visual:\n\u0026hellip; that led to a refactored landing page # The landing page used to only display a table with a row for each snapshot.\nI reworked it so that it encompasses all the changes described in this post: it contains a set of tabs that allow switching from snapshots listing to file kinds, file types or file extensions listings, providing easy access to filtered searches as well as statistics by kinds, types and extensions:\nWhat\u0026rsquo;s next ? # Working on the UI is a priority as it is the easiest way to browse through snapshots, but it is also particularly painful as I suck at making non-CLI stuff and because Go\u0026rsquo;s html/template is orders of magnitude higher in the pain-in-the-ass scale compared to Python\u0026rsquo;s jinja2 or bottlepy template engines which I\u0026rsquo;m familiar with. Stuff that could be done in a few minutes easily take me an hour to achieve when I don\u0026rsquo;t give up for a few days.\nI\u0026rsquo;ll continue working on the search capabilities, but there are two features that I really want done next month too: being able to find all duplicates of a file on all snapshots, regardless of their path names, and being able to find all files that have changed between snapshots and provide diff in the UI. Basically, I should be able to find all copies of a single file anywhere in the repository, but also be able to tell that /etc/passwd is identical in four snapshots but has changed in the fifth and display the change. This is already doable in the CLI but it needs some user-friendliness in the UI.\nVoila.\n","date":"1 April 2022","permalink":"/posts/2022-04-01/april-2022-plakar-keys-and-ui-stuff/","section":"Posts","summary":"TL;DR: I did key and UI stuff, mostly search related. There\u0026rsquo;s going to be plenty of images in this post. Reworked plakar keys # I won\u0026rsquo;t expand much on that as I\u0026rsquo;m not done yet, but I spent a couple days reworking the way plakar handles keys for encrypted repositories.","title":"April 2022: plakar keys and UI stuff"},{"content":" TL;DR: implemented cloning and synchronization between plakar repositories Slacked a bit # Shortly after I published my yearly retrospective, I was hit with two highly annoying personal issues that kept me very busy these last two months. I finally got everything back under control, my mind is mostly free again and I resumed playing with plakar.\nplakar clone # I implemented a clone operation in plakar allowing the creation of a new repository that is an exact copy of a source repository. This is a bit trickier than it reads, repositories rely on a specific structure as well as hard links magic to properly represent snapshots and reference counts on files. Performing a recursive cp of the plakar storage directory would lead to a broken repository unable to properly handle de-duplication.\nTo implement cloning, plakar creates a new repository from scratch using the exact same configuration as the source repository. It retrieves the list of chunks, objects and snapshots from the source repository, then for each chunk, object and snapshot it retrieves the data from the source repository and writes it to the destination repository. Once this is done, it loads the index file for each snapshot and performs the hard link magic to recreate the references required for snapshots to be complete. When done, both repositories are identical and the destination one can be used just as if it was the source one.\nThis required adding a few primitives to the storage layer and it was quite frustrating as mistakes didn\u0026rsquo;t show up right away. A lot of tasks can rely on the snapshot index, so when a chunk or an object are missing or incorrectly referenced for example, the cloned repository can still handle a lot of commands as if there was no issue. This is where plakar check came in handy, letting me know right away that something was off.\n$ plakar create -no-encryption $ plakar push /bin $ plakar ls 2022-03-05T22:30:58Z 3a5769ff-3353-4ae9-ae64-13fc0fc0b6ac 13 MB /bin $ plakar clone /tmp/plakar-copy $ plakar on /tmp/plakar-copy ls 2022-03-05T22:30:58Z 3a5769ff-3353-4ae9-ae64-13fc0fc0b6ac 13 MB /bin $ As you can see in the example above, the clone retains snapshots UUID as this is not a new snapshot being copied from another one in a new transaction, but really the exact same snapshot copied using lower level primitives.\nAs of today, plakar clone is not fully finished as it only supports filesystem-based repository. I need to implement the new storage primitives in the SQLite and network backends, so that cloning can happen seamlessly over the network or between different backends. When this is done, it\u0026rsquo;ll be possible to convert a filesystem plakar to an SQLite plakar by cloning it (right now, it\u0026rsquo;s only doable the other way), or to clone a plakar repository on a different machine.\nplakar sync # With plakar clone, all the primitives required to synchronize two repositories became available.\nSimilarly to cloning, synchronizing allows performing a low level copy of snapshots without initiating new transactions, thus retaining the same snapshots UUID. To achieve synchronization, plakar opens a source repository and a destination repository, then compares which chunks, objects and indexes are available in the source one but missing from the destination one to compute a delta. It then performs a cloning of that delta, only exchanging the missing bits and doing the hard links magic.\n$ plakar create -no-encryption $ plakar clone /tmp/bleh.t1 $ plakar ls $ plakar on /tmp/bleh.t1 ls $ plakar push /bin $ plakar ls 2022-03-05T23:24:08Z 480e6781-4f51-4289-9983-17f0e5962cc9 13 MB /bin $ plakar sync /tmp/bleh.t1 $ plakar on /tmp/bleh.t1 ls 2022-03-05T23:24:08Z 480e6781-4f51-4289-9983-17f0e5962cc9 13 MB /bin $ In the example above, I create a new empty repository and clone it. Then I push /bin to the source plakar, perform a plakar sync to the destination plakar, which ends up having the same snapshot.\nLike for cloning, this is a work in progress and not finished.\nThe idea behind sync is that when the primitives are ported to network plakar, it becomes possible to have a remote plakar centralize the backups of a local plakar, but it also becomes possible for multiple remote plakars to synchronize themselves one with another and provide multiple copies of the same backups very easily.\nWhat\u0026rsquo;s next ? # I\u0026rsquo;ll be working on and off as I have a wedding that\u0026rsquo;s coming soon and a lot of things to prepare, but this should not prevent me from finding time to relax with some code.\nI\u0026rsquo;ll rework clone and sync because functionality set aside I don\u0026rsquo;t like the way it\u0026rsquo;s implemented in the CLI.\nI\u0026rsquo;m currently reading the QuickCDC paper and wondering if it\u0026rsquo;s worth implementing on top of my go-fastcdc implementation, it is unclear if there will be any gain at this point.\nI\u0026rsquo;m considering a few more repository-level operations, similar to clone/sync, to help make plakar as friendly as possible to operators.\nIt will be time to start working on a website too :-)\n","date":"6 March 2022","permalink":"/posts/2022-03-06/march-2022-plakar-clone-and-plakar-sync/","section":"Posts","summary":"TL;DR: implemented cloning and synchronization between plakar repositories Slacked a bit # Shortly after I published my yearly retrospective, I was hit with two highly annoying personal issues that kept me very busy these last two months.","title":"March 2022: plakar clone and plakar sync"},{"content":" TL;DR: A retrospective of 2021 2021 was a good year overall # Pandemic and lock-downs set aside, 2021 was a good year overall.\nI\u0026rsquo;ve really enjoyed my daytime job, my physical health was fine, my mental health was\u0026hellip; meh\u0026hellip; but ok, I guess.\nLearnt about my neuro-divergences # I took a psychological assessment which pinpointed what\u0026rsquo;s \u0026ldquo;wrong\u0026rdquo; with me, the source of my alexithymia, generalized anxiety disorder and life-long uneasy feeling.\nIt was painful (a.f.) to cope with the result, and it\u0026rsquo;ll still take some time for me to get over it, but at least I now have a much better understanding of how I\u0026rsquo;m wired, what works and doesn\u0026rsquo;t work for me, and some actions I need to take to effectively improve my life.\nIt sucks to discover neuro-divergences past 40 years old, with ~half of my life already passed, but also a career and life decisions taken on partial informations. There\u0026rsquo;s a bit of grieving required, but it\u0026rsquo;s all for the best: I can acknowledge and move forward.\nI won\u0026rsquo;t disclose the details as I\u0026rsquo;m not comfortable sharing them publicly, but if I decided to write this, it\u0026rsquo;s because I hope it can be helpful to others in similar situations: taking a psychological assessment is nothing to be ashamed of, it doesn\u0026rsquo;t make you weak, and if you ever feel psychological suffering that you fail to understand, it can help put some perspective where it\u0026rsquo;s lacking. I wish I had someone tell me this twenty years ago, so I\u0026rsquo;m doing my part in case someone needs to read this now.\nWork psychology studies # Because of the pandemic I had to put my studies of work psychology on hold, which is a shame considering that I\u0026rsquo;m only a year away from starting my master thesis, but finding a company that would welcome me for a field intervention has proven impossible\u0026hellip; two years in a row\u0026hellip;\nI\u0026rsquo;ll take a break until the pandemic is behind and life is back to normal rather than spend a third year going to evening classes for nothing, I\u0026rsquo;d rather spend these hours playing with my son and learning new stuff. It\u0026rsquo;ll take considerably longer than expected to obtain the diploma but it\u0026rsquo;s ok, I\u0026rsquo;m not in a rush and there are other fields I want to study in 2022.\nHypnosis office # Despite suffering from the consecutive lock-downs and curfews, work at the hypnosis office remained active throughout the year, which allowed me to reach break-even. Considering that hypnosis is not my main activity, that I do not depend on it for a living, and that I\u0026rsquo;m only doing it because I enjoy doing it and need it in my life\u0026hellip; I\u0026rsquo;m happy that it breaks even and that I don\u0026rsquo;t have to stop that activity like so many people around me.\nHere\u0026rsquo;s a picture of my new office which looks lovely:\nStarting January 2022, I\u0026rsquo;ll be doing self-hypnosis workshops and I\u0026rsquo;m considering doing a few hypnotherapy workshops too, though I need to evaluate how much effort I can pour into this first.\nResumed doing some open-source work # For the last two years, I had lost motivation to write code and work on opensource projects: I\u0026rsquo;d sit in front of my computer and find a thousand creative ways to procrastinate. Then it struck me: I lost touch with why I enjoyed writing code in my spare time and this is why I no longer could.\nI loved writing code because I loved experimenting, learning new tricks, writing broken throw-away experiments, testing stuff and seeing what comes out of it. I loved writing code because I primarily wrote it for myself, not for others, and because I had no other goal but to enjoy the puzzle solving\u0026hellip; or giving up when interest decreased. I loved it because it was a hobby and I could do whatever I want.\nThis last decade, my spare time coding was spent for others, not myself. I wrote tons of code that I had no interest in whatsoever, just because it was what others expected to see happening, and everything I did was done thinking about how useful it would be to others and how they\u0026rsquo;d view my work. No one forced me, it just happened and I did not see how this affected me.\nIt\u0026rsquo;s hard to switch that mindset but I decided to try no longer giving a damn.\nI\u0026rsquo;ll work on projects I want, regardless of how useful they are, and I will try not to care how people feel about these projects. I\u0026rsquo;ll keep sharing them because that\u0026rsquo;s how I learnt and I want to give back, but also because that\u0026rsquo;s how people teach me new tricks by pointing at things I\u0026rsquo;ve done wrong or inefficiently.\nIf people find these projects useful, I\u0026rsquo;ll be happy, but if a project starts putting pressure on me I\u0026rsquo;ll step back, let people who care fork and maintain it, then move on to work on something else.\nThese realization and decision unlocked me and, as a result, I worked on many projects as can be seen from my last few monthly reports. Some of them, like plakar, are useful and entertaining to me while others served no other purpose but entertainment. This is how it\u0026rsquo;s going to work for me from now on ;-)\nLearnt myself some arabic speaking # I spent 2021 learning how to speak Arabic with weekly classes and I could not be happier as I can now hold basic kid-level discussions but still feel like I could be dropped in the Middle-East and manage to find my way and not starve. When I go back to Lebanon, I\u0026rsquo;ll be able to order my shawarma, manaïche and baklawa without relying on French, English or someone to translate for me. I won\u0026rsquo;t be fluent before a while but I only care about being understood, so that\u0026rsquo;s ok.\nI just began learning how to read, hopefully by the end of 2022 I can read a simple book, or better, one of my aunt\u0026rsquo;s novels \u0026lt;3\nHopefully I\u0026rsquo;ll also be making better use of my US-arabic keyboard which currently only proves useful at extending the character set of my passwords ;-)\nBegan learning how to draw # In September, I began learning how to draw from scratch (because I suck) and I actually enjoyed it.\nI wasn\u0026rsquo;t very constant as I got context switched a lot and didn\u0026rsquo;t draw every day, but I have prepared myself a schedule so that I have dedicated time starting in 2022.\nAt this point, I can\u0026rsquo;t really say I learnt much but I got a grasp at some key concepts:\nI\u0026rsquo;ll keep this blog updated with my progress as a mean to keep motivated ;-)\nStarted learning the piano # Late 2020 I took a try at creating LoFi tracks because I wanted to know how to \u0026hellip;\n\u0026hellip; and this year as I was having fun with a piano VST and my MIDI controller keyboard plugged, I got hooked and spent several evenings improvising single-hand melodies on top of songs I liked. It was fun and all but I realized that I really needed a piano in my life (and also to learn how to play with both hands).\nThat\u0026rsquo;s how I ended getting myself a Yamaha YDP S54 piano this summer, after successfully convincing my second-half that it would benefit the kid and be plenty nice in the appartment (don\u0026rsquo;t pull this at home kids, it was a professional stunt)\u0026hellip;\n\u0026hellip; but I couldn\u0026rsquo;t play it right away as we left for holidays right after I brought it home, then I got carried away in September by all the kid / work things you have to deal with when coming back from summer vacations.\nI started taking courses in November and I absolutely loved it from the first session, so much in fact that I have not touched any other instrument since then (yes, that\u0026rsquo;s bad\u0026hellip;), and that I actually play with it every single day.\nI took five courses so far so there\u0026rsquo;s a looooooong road ahead, but I can definitely picture myself sticking with that instrument in the long run.\nSatie was the main reason I wanted to learn the piano so\u0026hellip; yay, I can play a few measures :-)\nProbably forgot a few things # I probably forgot a few things, I kept myself busy all year long and didn\u0026rsquo;t keep a log of every little thing I did.\nThese were my most important achievements this year and I\u0026rsquo;m happy about all of them. I wish I could say that I\u0026rsquo;d take some rest in 2022, but\u0026hellip; I have a stack of books sitting next to me (literally on a chair) and the one on top is already keeping me busy.\nMy biggest achievement in 2021 was to realize that I STILL can do anything I want, nothing is out of reach and if something actually is, then\u0026hellip; heck, I tried and I\u0026rsquo;m free to give up any time, I\u0026rsquo;ll still have made progress on the way.\nHAPPY NEW YEAR # I\u0026rsquo;ll close this retrospective with a HAPPY NEW YEAR EVERYONE !\nI wish you success at everything you attempt and may a ton of good things happen to you in 2022.\nPeace.\n","date":"30 December 2021","permalink":"/posts/2021-12-30/farewell-2021-welcome-2022-a-personal-post/","section":"Posts","summary":"TL;DR: A retrospective of 2021 2021 was a good year overall # Pandemic and lock-downs set aside, 2021 was a good year overall.\nI\u0026rsquo;ve really enjoyed my daytime job, my physical health was fine, my mental health was\u0026hellip; meh\u0026hellip; but ok, I guess.","title":"Farewell 2021, Welcome 2022: a personal post"},{"content":" TL;DR: I worked on plakar, go-fastcdc a FastCDC implementation for plakar, and some useless stuff. This activity report will be short # This activity report will be short because I spent the last ten days caring for my kid who got sick\u0026hellip; before catching his stomach bug myself and spending the last few days in a somewhat familiar and unpleasant hell, unable to do everything I intended for this month.\nI didn\u0026rsquo;t want to skip this report as it\u0026rsquo;s the last of the year, but I\u0026rsquo;m exhausted and don\u0026rsquo;t have the energy to spend hours writing and rephrasing, I\u0026rsquo;ll make it up to you next year.\nFirst, plakar # The last few months, I worked on and off on the plakar project and while it\u0026rsquo;s not at a stage where I want to release it, it\u0026rsquo;s most certainly at a stage where feedback from early testers would be useful.\nIf you have followed it through the last two or three reports and are interested in where this is going, feel free to join my discord where a room is dedicated to this project, where you can provide feedback as you test, and where I can fix things live as you break:\n- DO NOT USE PLAKAR + DO NOT USE PLAKAR (UNLESS YOU\u0026#39;RE HELPING ME MAKING IT USABLE) If I get enough feedback and assurance that it\u0026rsquo;s not too buggy and can\u0026rsquo;t collapse into a singularity, this might even get a first release sometime early 2022 if it keeps on at the pace it was going this last quarter.\nSince we\u0026rsquo;re on the subject of plakar\u0026hellip; # I did a ton of optimizations, not just code-wise but in terms of index format and such, storage representation is not fully stable yet but barely changes anymore and plakar snapshots made this week are likely to be restorable with a build from next week. Work these days is mainly focused on improving the command line interface and features built on top of snapshots themselves, not on how data is dealt with.\nOne of the things I really wanted to improve, and which is the main thing I did this month, was to change the chunker\u0026hellip; so let\u0026rsquo;s talk about that.\nWhat\u0026rsquo;s a chunker ? # To save space and avoid storing multiple copies of a same file content, plakar checks if that content is already in the store and only records a reference if it is.\nInstead of checking for an exact copy of the file, which would no longer match if a single byte was appended, it slices the content into chunks and checks for them individually so as to only write chunks that are not already present.\nA very naive approach to this slicing is that of fixed chunking where the file is sliced into chunks of fixed size, say 16KB, which works nicely for any file that\u0026rsquo;s not supposed to change (hello /bin, you\u0026rsquo;re awesome) or that only gets appends (I see you /var/log/maillog). This approach is naive because it works perfectly\u0026hellip; as long as nothing causes a shift in chunk alignment: prepend a single byte in front of a 1GB log file that\u0026rsquo;s already in the store, and all chunks are now shifted, none can be found in store and the whole file needs to be stored again.\nA solution to this problem is the use of content-defined chunking, an approach that uses the content itself to dynamically determine the location of chunk boundaries, rather than using fixed boundaries.\nThe basic idea is that if you use a rolling buffer of data to somehow determine the chunk boundaries, then even if you insert a byte at the beginning or the middle of the stream and process the stream again again, after a while the rolling buffer will contain the same content as before the byte was inserted and therefore end up in the same chunking cycle\u0026hellip; so it should be possible to compute a rolling hash of some kind and use specific outputs of that rolling hash as a condition to delimit chunks.\nThis is nice and all but it doesn\u0026rsquo;t come for free. If chunking is not of a fixed size, then it\u0026rsquo;s no longer a simple fixed cursor increment but the result of computation done on content\u0026hellip; which means that the more content needs to be seen to determine chunks, the slower chunking becomes. ORDERS OF MAGNITUDE slower.\nLuckily for me, there\u0026rsquo;s a research community focused on content-defined chunking which has smarter people (ORDERS OF MAGNITUDE smarter) coming up with ways to speed things up, and while there\u0026rsquo;s no way it\u0026rsquo;ll ever reach the speed of fixed-length chunking, they made so much progress these last decades that it\u0026rsquo;s actually fast enough to be usable. Not only that, but through improvements on resources consumptions like for instance not pegging a CPU during the whole process of chunking, some of the time lost on chunking can be regained on parallelization of tasks for example.\nAnyways, I digressed\u0026hellip;\nIn my very early Python PoC of plakar years ago, I started with fixed-length chunking to bootstrap the code then implemented a Rabin-Karp rolling hash to provide contend-defined chunking. It worked and it was slow (a.f.), probably because it was Python, probably because it was my code, probably because Rabin-Karp isn\u0026rsquo;t\u0026hellip; too fast.\nWhen I revisited this project from scratch in Golang a few months ago, I thought I\u0026rsquo;d just use any existing chunker available to me while bootstrapping the project, then take serious time investigating what\u0026rsquo;s the chunker algorithm I really want to use and if I really need to implement it myself.\nFor a few months, I relied on restic\u0026rsquo;s chunker which\u0026hellip; is a rolling Rabin hash (hooray, no surprises), but this month I decided to reconsider as it doesn\u0026rsquo;t perform as fast as I\u0026rsquo;d like and having plakar depend on another backup utility was a bit weird and came with its own issues.\nI did some quick search and found a paper titled FastCDC: A Fast and Efficient Content-Defined Chunking Approach for Data Deduplication, claiming:\nwe propose FastCDC, a much faster CDC approach for data deduplication than the state-of-the-art CDC approaches while achieving a comparable deduplication ratio. [\u0026hellip;] Our experimental evaluation demonstrates that FastCDC obtains a chunking speed that is about 10× higher than that of the Rabin-based CDC and about 3× that of the Gear- and AE-based CDC while achieving nearly the same deduplication ratio as the Rabin-based CDC.\nSAY NO MORE, SCIENTIST, YOU CONVINCED ME WITH YOUR WISE WORDS.\nI did a bit of github search, found implementations, tested them but they didn\u0026rsquo;t work too well for me as one crashed, the other had an interface optimized for streaming, and both copied optimizations that I didn\u0026rsquo;t understand from a different implementation in a different language.\nThe paper was nicely written, came with pseudo-code displaying a very simple algorithm, so\u0026hellip;\ngo-fastcdc # The go-fastcdc package hosts my implementation of the FastCDC paper with no bells and whistles\u0026hellip; except that it takes special care to avoid unnecessary buffer allocations and has an implementation-specific optimization of my own, unrelated to how FastCDC itself works.\nBasically, a chunking loop looks like this:\nfor { chunk, err := chunker.Next() if err != nil { if err == io.EOF { // no more chunks to read break } log.Fatal(err) } // do something with the chunk } In other implementations that I read, the call to Next() is what computes the chunk boundaries so that in this loop the chunker only works during Next(), then remains idle while the caller does something with the chunk.\nMy optimization was to have a goroutine part of the chunker to keep ensuring that the next chunk is available as soon as possible, and make Next() return the next chunk as soon as it is made available by the goroutine. This doesn\u0026rsquo;t seem like much but it actually ensures that the chunker doesn\u0026rsquo;t remain idle while the caller works with a chunk, making the calls to Next() almost \u0026ldquo;free\u0026rdquo; as the cost of chunking is paid concurrently to whatever work the caller is doing (if caller isn\u0026rsquo;t pegging the CPU that is) The chunker goroutine only ensures ONE chunk ahead and will sleep as long as that chunk isn\u0026rsquo;t consumed.\nOn a loop similar to the one above with no work done on the chunk, this will bring absolutely no benefit, but for plakar which does a lot of chunking but also does a lot of stuff with chunks before requesting the next one, this saves several seconds on the snapshot of a 1GB directory.\nThere\u0026rsquo;s not much more I can say about it, it takes an input reader, produces content-defined chunks out of it following the FastCDC algorithm, not much to see, chop chop.\nIt is ISC-licensed, use it for whatever you see fit, or don\u0026rsquo;t use it and read what follows.\nplakar uses go-fastcdc # With go-fastcdc available to use, I broke ties with the restic chunker and switched plakar to go-fastcdc as its chunker, it was a very tricky change that involved a ton of I wish I could write about the challenges I faced but it was a trivial change that took two minutes as I used the same interface.\nThe performance improvement was not as great as advertised by other fastcdc implementation but it is slighly faster for my use-case, I fully know the chunker code and am the maintainer of it so if anything needs tweaking, it\u0026rsquo;s a big win.\nDoes that mean that plakar will rely on FastCDC ? I dunno.\nSee, it took me a day to switch from the previous chunker to my own FastCDC implementation, removing the downsides I had, so it was worth doing that work but this doesn\u0026rsquo;t mean that FastCDC is the one I want to go with in the end.\nThe FastCDC paper was published in 2016 claiming to be up to 10x faster than CDC, but then RapidCDC was published in 2019 claiming to be up to 33x faster than CDC, and now QuickCDC was published this year claiming to be even faster. Speed isn\u0026rsquo;t everything but\u0026hellip; in my case it kinda is.\nUnless the trade-offs (cpu cost / ram cost) are unbearable, there\u0026rsquo;s no reason to stick with FastCDC if anything else is faster. I only had the time to read and implement FastCDC before my son tripped me into a hell of body fluids, but I\u0026rsquo;ll be carefully reading (and probably implementing) RapidCDC and QuickCDC so I can do tests and pick what\u0026rsquo;s the best for plakar.\nTHAT\u0026rsquo;S ALL FOR THE USEFUL STUFF, BELOW IS JUNK # Assorted experiments to reduce my mental backlog # Implemented my first genetic algorithm # I learnt the theory behind (some) genetic algorithms reading a book when I was a student approximately a century ago, but there was no school project involving one and since I was already drowning in work (and really wanted to graduate), I decided not to disperse myself and plain forgot about the topic.\nSomehow, something reminded me of genetic algorithms this month and since I never wrote one and didn\u0026rsquo;t want to remain ignorant, I searched for a tutorial and ran into this presentation:\nI spent an hour implementing both examples, another hour trying to come up with smart optimizations to speed things up as it was painfully slow, then yet another hour to experiment with them further until I understood what they were useful for and what they were unusable for.\nIt was lovely, I\u0026rsquo;m not sure I\u0026rsquo;ll be using one again before the next decade but who knows.\nImplemented a custom DHT # Many years ago, I worked at a company that sells services built on top of a distributed storage powered by a chord-inspired DHT. I had never heard about DHT before so I studied the paper in anticipation of the interview, wrote a very basic network-less implementation, fell in love with the idea, and enjoyed my time working there with this technology even though I didn\u0026rsquo;t work on the DHT itself but on a project built on top of it.\nLife went on, I switched job and city, but I kind of kept a regret that I never worked on the DHT layer myself \u0026hellip;\n\u0026hellip; and since this month was already screwed in terms of usefulness, I spent a couple evenings experimenting with a DHT implementation of my own that has some properties I\u0026rsquo;m interested in. Some ideas were nice, others sucked or didn\u0026rsquo;t produce the result I expected. It\u0026rsquo;s an ongoing work to understand what I want to retain and what should be thrown away.\nI have a working implementation, with a memory-based key/value store built on top of it, networking and all, but there\u0026rsquo;s nothing really interesting to show so I\u0026rsquo;ll continue experimenting as a low-priority exercise and I\u0026rsquo;ll write about it here if anything nice ever comes out of it.\nHello, VR ! # If 2020 and 2021 have taught me anything, it\u0026rsquo;s that there\u0026rsquo;s an overwhelming amount of people that I wish would live on another planet. A number so high, in fact, that even if they all agreed to board a shuttle right away and head to the sun, I\u0026rsquo;d still consider moving away from this planet while they board because oh my I wish they lived on another planet.\nI had huge plans for relocating on Mars ever since I read Ray Bradbury\u0026rsquo;s Martian Chronicles as a teenager, but now there\u0026rsquo;s this electric car guy who wants to ruin there too so\u0026hellip; nu-uh, I need another place, preferably the opposite direction.\nVenus seemed very promising but the NASA won\u0026rsquo;t be floating outposts there before my time is over, I\u0026rsquo;m screwed.\nThis only leaves me with VR.\nI picked up a book and started learning about VR development. This way, when I have a rough day and I just can\u0026rsquo;t anymore with the human race, I can wear a mask and instantly put a few million kilometers between others and myself, even if just for a few deep breathes and the sight of an asteroid putting this place out of its misery.\nI\u0026rsquo;m not quite there yet but it seems achievable in a more realistic timeframe than my travel to Venus.\nThat\u0026rsquo;s all folks! # As I do every year, I\u0026rsquo;ll write a more personal retrospective of this year in a few days, maybe late 2021 or early 2022, it\u0026rsquo;s been an intense year and I need to pour it out so my head is empty again.\nHope you enjoyed reading this post, happy Xmas, happy new year and may life bring you joy and happiness.\n","date":"13 December 2021","permalink":"/posts/2021-12-13/december-2021-a-bit-of-plakar-a-bit-of-go-fastcdc-and-some-useless-stuff/","section":"Posts","summary":"TL;DR: I worked on plakar, go-fastcdc a FastCDC implementation for plakar, and some useless stuff. This activity report will be short # This activity report will be short because I spent the last ten days caring for my kid who got sick\u0026hellip; before catching his stomach bug myself and spending the last few days in a somewhat familiar and unpleasant hell, unable to do everything I intended for this month.","title":"December 2021: a bit of plakar, a bit of go-fastcdc and some useless stuff"},{"content":" TL;DR: I still have a discord, feel free to join. I worked on go-ipcmsg to make it nicer, go-privsep to make it more useable, and A LOT on plakar to make it plakar. Go-ipcmsg # In April, I wrote about go-ipcmsg, a package to bring an imsg(3)-like API to Golang and help me write code involving message and fd-passing between processes.\nIt took me a couple days of work and I was happy with the result so I left it there, letting it rest in its Github repository, until earlier this month when I figured the API could be simplified by a great deal\u0026hellip; so I decided to revisit and make some changes..\nBack then, each peer of an IPCMSG channel received two Golang channels, and could simply read or write to the other process through them:\nfunc parent() { // just a basic function that sets up a socketpair, forks a child, // and returns the child\u0026#39;s pid and the parent\u0026#39;s side of the socketpair. pid, fd := fork_child() // read \u0026amp; write channels obtained here child_r, child_w := ipcmsg.Channel(pid, fd) // write to child, last parameter == fd to pass to the other process child_w \u0026lt;- ipcmsg.MessageWithFD(IPCMSG_PING, []byte(\u0026#34;foobar\u0026#34;), -1) // read from child msg := \u0026lt;- child_r } func child() { // read \u0026amp; write channels obtained here, // fd=3 is the fork-inherited side of the socketpair. parent_r, parent_w := ipcmsg.Channel(os.Getppid(), 3) // read from parent msg := \u0026lt;- parent_r // write to parent parent_w \u0026lt;- ipcmsg.Message(IPCMSG_PONG, []byte(\u0026#34;barbaz\u0026#34;)) } In practice, instead of inlining calls like this, you\u0026rsquo;d declare a handler function which would process all messages on the read channel and reply through the write channel as such:\nfunc dispatcher(r chan ipcmsg.IPCMessage, w chan ipcmsg.IPCMessage) { for msg := range r { switch msg.Hdr.Type { case IPCMSG_PING: w \u0026lt;- ipcmsg.MessageWithFD(IPCMSG_PONG, []byte(\u0026#34;barbaz\u0026#34;), -1) } } } func child() { parent_r, parent_w := ipcmsg.Channel(os.Getppid(), 3) dispatcher(parent_r, parent_w) } However, I realized that pretty much every single use of the package would require writing that very similar dispatcher function looping over the read channel, and I also disliked the pattern of letting the user access the Golang channels directly as it made it easier to introduce mistakes in code for no benefit whatsoever.\nI decided to rewrite the ipcmsg.Channel() function to return an opaque structure hiding the underlying channels, and to provide a channel.Dispatch() function to start reading from the channel. With this change, the sample of code above now looks like:\nfunc handlePING(channel *ipcmsg.Channel, msg ipcmsg.IPCMessage) { channel.Message(IPCMSG_PONG, []byte(\u0026#34;barbaz\u0026#34;), -1) } func child() { channel := ipcmsg.NewChannel(\u0026#34;child\u0026lt;-\u0026gt;parent\u0026#34;, os.Getppid(), 3) channel.Handler(IPCMSG_PING, handlePING) \u0026lt;-channel.Dispatch() // Run() and wait until it returns due to peer close } As you can see, the message handling function receives an IPCMessage when one is available, without having to loop on the read channel, and the function setting up the channel can declare handlers for specific message types before calling the dispatcher.\nIn addition to this simplification, I introduced a new mechanism: synchronized messages.\nIn the imsg(3) API, because it is written in C which doesn\u0026rsquo;t provide coroutines, messages are sent and code can\u0026rsquo;t wait for an answer as it would block the entire process meanwhile. To avoid this, code to emit a message and code to handle answers are decoupled: a function sends and stops caring about the message\u0026hellip; until another function gets an answer and resumes what the first intended to do if it could have waited for an answer. Splitting code like this is always doable, but it can be more or less tricky with some patterns where you\u0026rsquo;d benefit from having a very sequential read.\nBecause I can use Goroutines here and have them interrupt their execution in place without blocking the entire process, I introduced two additional calls:\n// channel.Query() blocks the goroutine and resumes execution // only when the other end has used channel.Reply() // msgtype, data, fd := channel.Query(IPCMSG_PING, []byte(\u0026#34;data\u0026#34;), -1) if msgtype != IPCMSG_PONG { panic(\u0026#34;THE WORLD IS COMING TO AN END.\u0026#34;) } fmt.Println(\u0026#34;got appropriate response to my query\u0026#34;) This pattern is not necessarily better than the split pattern, they both have their use-cases which is why go-ipcmsg provides both, but in some situations it really makes code much less complex than interrupting execution to resume in another function.\nThe code is already available in a Github repository, and it works enough that I use it in various proof of concepts I write, but it still isn\u0026rsquo;t ready to use in production and was only tested by me.\nFeel free to test and report issues ;-)\nGo-privsep # In April, I also wrote about go-privsep, a package to ease the writing of OpenBSD-style daemons in Golang.\nThe idea behind it is that most OpenBSD-style daemons use the same multi-process design, yet every project has to have a different setup logic because they differ in the number of processes, which processes communicate with each other, and which messages they sent one to another.\nI disliked having to bootstrap a new daemon but at least I could copy chunks from a previous one and adapt, but this doesn\u0026rsquo;t work with Golang as I have not found any OpenBSD-like daemon to copy from. So I thought maybe I should solve the main issue which is making the part that annoys me\u0026hellip; less annoying, this way I don\u0026rsquo;t have to bother much setting up a new daemon architecture and I don\u0026rsquo;t have to copy code from another daemon either.\nThe result is a package that allows to declare how the daemon is supposed to look like when it enters its event loop, and the package takes care of creating the processes, the communication channels between them, dropping privileges and chrooting where appropriate, before having each process enter its own entry point.\nIn the following example, I have created a daemon with a design similar to OpenSMTPD\u0026hellip; except that all entry points are idle because it\u0026rsquo;s not a real daemon 😁\nfunc main() { privsep.Init() privsep.Parent(\u0026#34;parent\u0026#34;, parent.Run) privsep.Child(\u0026#34;crypto\u0026#34;, crypto.Run) privsep.Child(\u0026#34;control\u0026#34;, control.Run) privsep.Child(\u0026#34;lookup\u0026#34;, lookup.Run) privsep.Child(\u0026#34;dispatcher\u0026#34;, dispatcher.Run) privsep.Child(\u0026#34;queue\u0026#34;, queue.Run) privsep.Child(\u0026#34;scheduler\u0026#34;, scheduler.Run) privsep.GetParent().Username = \u0026#34;root\u0026#34; privsep.GetParent().TalksTo(\u0026#34;dispatcher\u0026#34;) privsep.GetProcess(\u0026#34;crypto\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;crypto\u0026#34;).Chrootpath = \u0026#34;/var/empty\u0026#34; privsep.GetProcess(\u0026#34;control\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;control\u0026#34;).Chrootpath = \u0026#34;/var/empty\u0026#34; privsep.GetProcess(\u0026#34;lookup\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;lookup\u0026#34;).TalksTo(\u0026#34;dispatcher\u0026#34;, \u0026#34;queue\u0026#34;) privsep.GetProcess(\u0026#34;dispatcher\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;dispatcher\u0026#34;).Chrootpath = \u0026#34;/var/empty\u0026#34; privsep.GetProcess(\u0026#34;dispatcher\u0026#34;).TalksTo(\u0026#34;parent\u0026#34;, \u0026#34;lookup\u0026#34;, \u0026#34;queue\u0026#34;, \u0026#34;scheduler\u0026#34;) privsep.GetProcess(\u0026#34;scheduler\u0026#34;).Username = \u0026#34;_smtpd\u0026#34; privsep.GetProcess(\u0026#34;scheduler\u0026#34;).Chrootpath = \u0026#34;/var/empty\u0026#34; privsep.GetProcess(\u0026#34;scheduler\u0026#34;).TalksTo(\u0026#34;dispatcher\u0026#34;, \u0026#34;queue\u0026#34;) privsep.GetProcess(\u0026#34;queue\u0026#34;).Username = \u0026#34;_smtpq\u0026#34; privsep.GetProcess(\u0026#34;queue\u0026#34;).Chrootpath = \u0026#34;/var/spool/smtpd\u0026#34; privsep.GetProcess(\u0026#34;queue\u0026#34;).PreChrootHandler(queue.PreChrootHandler) privsep.GetProcess(\u0026#34;queue\u0026#34;).TalksTo(\u0026#34;dispatcher\u0026#34;, \u0026#34;lookup\u0026#34;, \u0026#34;scheduler\u0026#34;) if os.Geteuid() != 0 { log.Fatal(\u0026#34;privileges separation requires root privileges\u0026#34;) } privsep.Start() } This should be self-explanatory but to summarize: you declare a list of processes with their names and entry points, then for each process you declare some properties such as the username it should run as or a chroot directory, and the list of processes it is allowed to talk to.\nWhen privsep.Start() is reached, the daemon bootstraps itself from the declarations and you end up with all processes setup:\n$ sudo ./smtpd $ ps au|grep smtpd _smtpd 70584 0.0 0.0 409218992 8112 s003 SN 12:17AM 0:00.01 ./smtpd: scheduler _smtpq 70583 0.0 0.1 409238336 9072 s003 SN 12:17AM 0:00.01 ./smtpd: queue _smtpd 70582 0.0 0.1 409233888 11120 s003 SN 12:17AM 0:00.02 ./smtpd: dispatcher _smtpd 70581 0.0 0.0 409218608 7264 s003 SN 12:17AM 0:00.01 ./smtpd: lookup _smtpd 70580 0.0 0.0 409217440 6304 s003 SN 12:17AM 0:00.00 ./smtpd: control _smtpd 70579 0.0 0.0 409225328 6320 s003 SN 12:17AM 0:00.01 ./smtpd: crypto root 70578 0.0 0.0 409219376 7936 s003 SN 12:17AM 0:00.03 ./smtpd root 70577 0.0 0.0 408638640 7296 s003 SN 12:17AM 0:00.03 sudo ./smtpd $ The go-privsep package relies on go-ipcmsg so that each processes that have called TalksTo() in the declaration have IPCMSG channels setup between them, and can declare message handlers or rely on Message() and Query() to communicate one with another.\nHere\u0026rsquo;s an example of a small daemon forking two children exchanging PING/PONG messages over and over:\npackage main import ( \u0026#34;log\u0026#34; \u0026#34;time\u0026#34; \u0026#34;github.com/poolpOrg/go-ipcmsg\u0026#34; \u0026#34;github.com/poolpOrg/go-privsep\u0026#34; ) const ( IPCMSG_PING ipcmsg.IPCMsgType = iota IPCMSG_PONG ipcmsg.IPCMsgType = iota ) func parent_main() { \u0026lt;-make(chan bool) // sleep forever } func main_foobar() { \u0026lt;-make(chan bool) } func main_barbaz() { foobar := privsep.GetProcess(\u0026#34;foobar\u0026#34;) foobar.Message(IPCMSG_PING, []byte(\u0026#34;test\u0026#34;), -1) \u0026lt;-make(chan bool) } func ping_handler(channel *ipcmsg.Channel, msg ipcmsg.IPCMessage) { log.Printf(\u0026#34;[%s] received PING\\n\u0026#34;, privsep.GetCurrentProcess().Name()) time.Sleep(1 * time.Second) channel.Reply(msg, IPCMSG_PONG, []byte(\u0026#34;test\u0026#34;), -1) } func pong_handler(channel *ipcmsg.Channel, msg ipcmsg.IPCMessage) { log.Printf(\u0026#34;[%s] received PONG\\n\u0026#34;, privsep.GetCurrentProcess().Name()) time.Sleep(1 * time.Second) channel.Reply(msg, IPCMSG_PING, []byte(\u0026#34;test\u0026#34;), -1) } func main() { privsep.Init() privsep.Parent(\u0026#34;parent\u0026#34;, parent_main) privsep.Child(\u0026#34;foobar\u0026#34;, main_foobar).TalksTo(\u0026#34;barbaz\u0026#34;) privsep.Child(\u0026#34;barbaz\u0026#34;, main_barbaz).TalksTo(\u0026#34;foobar\u0026#34;) privsep.GetProcess(\u0026#34;foobar\u0026#34;).PreStartHandler(func() error { barbaz := privsep.GetProcess(\u0026#34;barbaz\u0026#34;) barbaz.SetHandler(IPCMSG_PING, ping_handler) return nil }) privsep.GetProcess(\u0026#34;barbaz\u0026#34;).PreStartHandler(func() error { foobar := privsep.GetProcess(\u0026#34;foobar\u0026#34;) foobar.SetHandler(IPCMSG_PONG, pong_handler) return nil }) privsep.Start() } The code is also already available in a Github repository, and it works enough that I use it in various proof of concepts, though the API still evolves and I would discourage you from using it for anything but experimenting at this point.\nFeel free to test and report issues ;-)\nGo-parsey # The last bit that I really miss in Golang when writing stuff is the OpenBSD-style configuration handling.\nIn OpenBSD, all daemons share a parse.y file containing their specific (yet very similar) grammar built on top of the same lexer. After a few years of making changes in it, I grew to enjoy parse.y and the simplicity of configuration files produced by that common piece of code.\nIt already took me a while to get used to ini files in Python but when I realized that Golang developers tend to use json, toml or yaml, I just couldn\u0026rsquo;t.\nI have started working on go-parsey, a small package that allows declaring a configuration grammar that more resembles what I like. It\u0026rsquo;s nowhere near done, at this point I wrote the lexer and am playing with the API to find what is the most usable way to obtain what I want.\nIt\u0026rsquo;s at a so early stage that I can\u0026rsquo;t even show something, except maybe the following example:\n// instantiate a lexer and teach it how to recognize tokens lexer := parsey.NewLexer() lexer.RegisterToken(\u0026#34;listen\u0026#34;) lexer.RegisterToken(\u0026#34;on\u0026#34;) lexer.RegisterToken(\u0026#34;match\u0026#34;) lexer.RegisterToken(\u0026#34;=\u0026gt;\u0026#34;) lexer.RegisterTokenMatch(\u0026#34;FOOBAR\u0026#34;, go func(v string) bool { v == \u0026#34;foobar?\u0026#34; }) lexer.RegisterTokenMatch(\u0026#34;STRING\u0026#34;, lexer.IsString) lexer.RegisterTokenMatch(\u0026#34;NUMBER\u0026#34;, lexer.IsNumber) lexer.RegisterTokenMatch(\u0026#34;FLOAT\u0026#34;, lexer.IsFloat) // instanciate a grammar and teach it how to recognize token sequences grammar := parsey.NewGrammar() grammar.RegisterRule(configBuilderRule1, \u0026#34;listen\u0026#34;, \u0026#34;on\u0026#34;, \u0026#34;STRING\u0026#34;) grammar.RegisterRule(configBuilderRule2, \u0026#34;match\u0026#34;, \u0026#34;=\u0026gt;\u0026#34;, \u0026#34;STRING\u0026#34;) // instanciate a config parser using the configured lexer and grammar config := parsey.NewConfiguration(lexer, grammar) // parse a file using that parser success, err := config.ParseFile(configFile) if err != nil { log.Fatal(err) } which will parse the following format, supporting line breaks, comments, quoting and free whitespaces:\nlisten on fxp0 match =\u0026gt; fxp0 listen on \\ fxp0 match =\u0026gt; fxp0 Of course the sample above isn\u0026rsquo;t how this is going to work because recognizing token sequences is not enough to produce usable configuration files, there needs to be support for expressions and conditionals, both of which I already have code for on my laptop. I\u0026rsquo;m currently more focused on what the API should look like so it isn\u0026rsquo;t too difficult to use.\nOH, and why don\u0026rsquo;t I use Goyacc, you ask ?\nIt boils down to three reasons:\nFirst, I\u0026rsquo;d like to have a declarative setup similar to what I did in go-privsep and the ability to either freeze a config format or \u0026ldquo;extend\u0026rdquo; it dynamically if I wish.\nThen, I haven\u0026rsquo;t hand-written a parser from scratch since I was a student and this will remove some dust in my brain, even if I end up not using it\u0026hellip; it\u0026rsquo;s the journey, not the destination.\nFinally, because it seems that I\u0026rsquo;m not smart enough to get Goyacc to do what I want otherwise I would have used it to start with and not bother with this 😃\nAnyways, don\u0026rsquo;t hold your breath, this isn\u0026rsquo;t a serious project and I\u0026rsquo;ll work on it on and off throughout 2022.\nPlakar # I did a TON of work on plakar with over 150 public commits and some more in a private branch.\nThe following subsections will describe the most interesting changes, in no particular order, though many more are interesting for the project but not so much for an article.\nTried (failed) to fix fuse support build on OpenBSD\u0026hellip; # But it turns out that after fixing my code, then fixing the code in the dependency, I hit a problem in Golang\u0026rsquo;s runtime as the mount system call is not implemented for OpenBSD yet. I have a diff but I haven\u0026rsquo;t been able to test it yet as I don\u0026rsquo;t feel like breaking my runtime right now.\nThe rabbit hole was deep and I fell in it.\nRewrote the network code # I did a lot of work in both client and server code to clean them from the disgusting proof of concept to something decent, and allowing more parallelism of operations. I still have work to do in that area but the foundations are clean now.\nThe network code now uses gob to produce a binary protocol that I don\u0026rsquo;t have to parse myself thanks to gob encoders/decoders working directly on a connection handler, what a pleasure to not have to do that myself.\nThe server can now handle concurrent requests from the same client: for example it is capable to process multiple requests at once contrarily to before, speeding up considerably transfers.\nThe client also benefited from concurrency improvements.\nReworked the storage interface # I reworked the storage interface to ease the writing of custom storage backends, it is now relatively easy to write a backend and plug it in plakar without having to make changes outside of the implementation.\nThis sounds like nothing but interface implementation in Golang was puzzling me and I\u0026rsquo;m happy I worked it out.\nAdded support for an SQLite storage backend # With the storage layer reworked, I implemented an SQLite storage allowing plakar to use an SQLite database instead of the filesystem:\n$ plakar on sqlite:///tmp/plakar.db create -no-encryption $ plakar on sqlite:///tmp/plakar.db push /bin $ plakar on sqlite:///tmp/plakar.db ls 2021-11-27T00:12:04Z 99e53a9c-e503-4c32-8622-924bf966f15b 11 MB /bin $ This has advantages and disadvantages.\nSQLite backend is MUCH faster than using the filesystem on a plakar push / plakar pull, mainly because operations are reduced to reads \u0026amp; writes in SQLite when the FS has to open / close files too for each chunk and object accesses. The commit of a snapshot in SQLite is very efficient and fast, whereas on the FS it can take a long while due to a lot of hard links tricks. But\u0026hellip; on the other end, I trust my filesystem more than SQLite and I don\u0026rsquo;t have to worry about multiple concurrent writers there.\nEdit: After thinking more about it, my irrational fear of storing backups in a single structured binary file doesn\u0026rsquo;t hold: I already do it with tarballs.\nI also had a concern with concurrent writes because, many years ago, a writer would lock the entire database. This would for instance prevent plakar from pushing a small snapshot while a very big one was holding a transaction for an extended period of time. But I was stuck in the past, as SQLite introduced a WAL \u0026hellip; 11 years ago.\nThanks to Glandos @ Github for questionning my SQLite fear which had me revisit the concurrent writers concern and realize it no longer held ground, this will be very useful as even the SQLite code I wrote still assumed this old behaviour:\nFor example, the local cache opens the database in read-only and keeps track of all writes it wants to do, then when a snapshot is committed, it reopens the database in write mode to flush changes and reduce the time it had to retain the database open. This whole contorted way of working is no longer relevant and code can be simplified a lot.\nWhat\u0026rsquo;s interesting is that SQLite support uses a generic SQL database connector, so I will be able to implement support for any SQL database by rewriting some queries\u0026hellip; and obviously I have in mind databases that allow replication.\nOptimized local cache logic and replaced with an SQLite implementation # The local cache implementation was suboptimal, only useful to test the logic but not really to improve performances, partly because it\u0026rsquo;s way of working was stepping in the way of the push algorithm\u0026hellip; so I reworked it to improve the situation.\nWhile at it, I switched from the previous FS-based local cache to an SQLite local cache, mainly to avoid the cost of multiple system calls whenever checking if something was in cache.\nImplemented a small plakar shell # Nothing too fancy, I just needed to be able to run multiple commands on the same plakar session, the shell just accepts commands and applies them to the same opened plakar.\n$ plakar on sqlite:///tmp/plakar.db shell plakar@sqlite:///tmp/plakar.db\u0026gt; ls 2021-11-27T00:12:04Z 99e53a9c-e503-4c32-8622-924bf966f15b 11 MB /bin plakar@sqlite:///tmp/plakar.db\u0026gt; ls 2021-11-27T00:12:04Z 99e53a9c-e503-4c32-8622-924bf966f15b 11 MB /bin plakar@sqlite:///tmp/plakar.db\u0026gt; ^C $ That doesn\u0026rsquo;t look too useful but that\u0026rsquo;s because in my articles I mainly provide examples on an -no-encryption plakar, so you don\u0026rsquo;t see me prompted for a passphrase every command I type. When you have to type multiple commands, using the plakar shell will only prompt at opening saving sanity.\nImplemented a filesystem view # The snapshot indexes consisted mainly of maps mapping checksums to pathnames or pathnames to file informations, etc\u0026hellip;\nThis was efficient for direct access to a resource by its name, like searching what\u0026rsquo;s the file informations for /etc/passwd, but not so much for operations involving partial pathnames, such as finding what files are in a directory. Unfortunately such operations are numerous as plakar supports a ls command to browse the snapshot, but also cat-ing, pull-ing or even creating tarballs from subdirectories within a snapshot.\nTo work around, I had implemented helpers that would do things like comparing full pathnames and determining if a path was within another, but this was confusing and inefficient as it required looking at the entire set of entries in the snapshot, and playing games with string prefixes.\nI implemented a filesystem within the index in the form a tree of file informations, as well as a set of functions to lookup pathnames in that tree. This has considerably simplified a lot of code and allows to lookup resources in the index similarly to how I would on the real filesystem.\nI kept the maps as indexes for direct lookups so that whenever searching for a precise resource, there is no need to go through the filesystem and traverse nodes, but so that whenever searching for a set of resources or trying to discover what\u0026rsquo;s in a directory hierarchy, the tree can be used instead.\nImplement CPU throttling # Until now, plakar would use all of the available resources to perform operations as fast as possible.\nWhile I think the default should be to run as close as possible to max limit and let the machine throttle, it\u0026rsquo;s also not a good idea to not leave at least one core for the OS to work correctly.\nI implemented CPU throttling so that plakar can be told to limit itself to a certain number of cores with the -cpu option, and I set the default to be between 1 and cpus-1 cpus, whichever is the most. On a 1 core VPS, it will consume up to a full CPU, while on my 8 core laptop it will consume 7 by default.\nThe effect of passing a -cpu option are quite visible as you can see with this plakar -cpu 4 push ...\nSimplify and optimize plakar push # I reworked the push logic to simplify and optimize the strategy, leading to less store calls and much improved performances with much more readable code.\nWith the help of discord user Spire, improvements were made leading to performances boosts due to code patterns that caused too many allocations.\nThere are still improvements to make but things are looking bright 😃\nParallelized plakar keep, plakar rm, plakar ls # Both plakar keep and plakar rm worked by removing snapshots sequentially from the store, I updated them so that they could run the removals in concurrency, allowing for faster execution.\nThe plakar ls case was similar in that to display the snapshots listing, it had to fetch the snapshot indexes, and it did that sequentially. When multiple big snapshot indexes were in the store, this could lead to a delay before the listing was displayed as each snapshot index would be fetched and deserialized sequentially, not exploiting the availability of multiple cores. I converted plakar ls to use concurrency for the fetch and deserialize, synchronizing the routines before the sorting and display.\nFixed UI # The UI was badly broken due to the many changes I made in the API, so I spent a while fixing it and taking advantage of the new snapshot filesystem view to provide directory browsing:\nI also improved slightly the view for individual resources, added the ability to download files or tarballs.\nImplemented plakar clone # I wrote a plakar clone command to fully replicate a repository:\n$ plakar on sqlite:///tmp/plakar.db clone /tmp/cloned $ plakar on /tmp/cloned ls 2021-11-27T00:12:04Z 99e53a9c-e503-4c32-8622-924bf966f15b 11 MB /bin $ This allows creating an exact replica of an existing plakar repository, regardless of where it\u0026rsquo;s located (filesystem, network or database) into a local directory, and respecting the internal structure of the original repository: it is not just a dumb file copy but really the cloning of the internal state, preserving reference counts and such.\nThe fact that it can only clone to a local directory is a temporary technical limitation that will be lifted in the future as my ultimate goal is to be able to:\n$ plakar on plakar://192.168.1.2 clone plakar://192.168.1.1 $ \u0026hellip; and have a plakar be reconstructed from a remote machine to another remote machine.\nThe idea is that you can clone an off-site plakar on a regular basis and know that if something happened to the main one, you\u0026rsquo;d still have a version of it that has a coherent internal state. When the local filesystem technical limitation is lifted, I\u0026rsquo;ll working on a synchronization mechanism so that two remote plakar may exchange chunks, objects and snapshots in an efficient way, without having to clone the entire repository.\nImplemented plakar check and fast checking # The plakar check \u0026lt;snapshot\u0026gt; command already existed to check the integrity of a snapshot by requesting the store to return each of its chunks and objects, allowing plakar to both validate that it could perform a full restore if needed and that checksums matched.\nAfter a discussion on Discord, I implemented fast checking which is a relaxed check requesting only that the store acknowledges existence of chunks and objects. This allows speeding up considerably the snapshot checks when you trust the store to respond honestly on the availability of resources.\nI finally implemented plakar check (no snapshot parameter) to perform a full plakar repository coherency check, validating that the store doesn\u0026rsquo;t contain orphan chunks and objects, but also that they have a reference count matching the number of snapshots relying on them, and that no snapshot lacks a reference to a resource that\u0026rsquo;s still in the store\u0026hellip; only because it is held by another snapshot.\nAs long as a resource is available in the store, even if the snapshot lost its reference somehow, plakar will manage to restore the resource: this is nice because you don\u0026rsquo;t want to fail recovering something for which you do have the data, but you also don\u0026rsquo;t want that to hide the fact that there was a corruption in the store somehow.\nIn theory, unless you poke in the store and remove resources yourself, these checks are never meant to fail but I\u0026rsquo;ll sleep better knowing that I can have daily snapshots fast checks, weekly snapshot checks and monthly repository checks\u0026hellip; and that if something goes wrong, I can pull a backup and investigate what went wrong.\nA nice improvement would be to provide the ability to repair some corruptions when doable, but since this is not supposed to happen in the first place, it\u0026rsquo;s not very high in my list 😁\nAssorted other work # After a month of learning, I can now play Ode to Joy, Autumn Leaves, and And I love her on the piano, that required some assorted work.\n","date":"26 October 2021","permalink":"/posts/2021-10-26/november-2021-a-bit-of-go-ipcmsg-a-bit-of-go-privsep-and-a-ton-of-plakar/","section":"Posts","summary":"TL;DR: I still have a discord, feel free to join. I worked on go-ipcmsg to make it nicer, go-privsep to make it more useable, and A LOT on plakar to make it plakar.","title":"November 2021: a bit of go-ipcmsg, a bit of go-privsep and a ton of plakar"},{"content":" TL;DR: I have a discord now, feel free to join. I refactored plakar, implemented a local cache, improved parallelism, modified the push strategy, played with fuse and networking. I also did other stuff but let\u0026rsquo;s keep that out of this article. I have a discord now # I have a rubber duck sitting on my desk\u0026hellip;\n\u0026hellip;but I also like explaining to my peers what I\u0026rsquo;m doing, as I\u0026rsquo;m doing it, to help me get new ideas or spot shortcomings in my reasoning. I used to do it a lot on IRC as I worked on OpenSMTPD, with other developers and an active community, but I miss that a lot now that I work mostly alone on low-profile projects. I do share ideas and progress on Twitter but the tweet size limitation makes it hard to expand much and incite discussion.\nI created a Discord where I\u0026rsquo;ll hang out and discuss my projects as I work on them. Feel free to hop in if you want, and feel free to do just like me and share thoughts as you work on your own projects there: this is a virtual hack room.\nIt may not be restricted strictly to code as I have other unrelated projects sometimes :-)\nPlakar refactor # I have been working on plakar on and off for a few months now, with approximately two weeks of cumulative work, and because it was experimental and my first real project in Go, I made many mistakes both in design and implementation details. I tested a lot of ideas, some turned out to be good and I pushed them further whereas others turned out to be crap and I abandoned them.\nFor instance, I wasn\u0026rsquo;t sure if snapshots should be part of the storage engine or built on top, I wasn\u0026rsquo;t sure if encryption/compression should be at the snapshot level or at the storage level, or even if it should be possible to encrypt some snapshots and not others within a store. All of these decisions are now settled, but I had to experiment a bit before understanding why I was taking a decision and why I believed it was the right one, and this left the code with some weird parts as things weren\u0026rsquo;t always done in the right place.\nNow that I\u0026rsquo;m a bit more comfortable with Go and with how the project will move forward, it was time to scrap the draft and rewrite it properly. I spent an evening in a new branch, rewriting all layers by bringing just the bits of code necessary and making sure that each package was not doing anything that should be done elsewhere. This allowed me to kill a lot of dead code from older experiments, simplifying the logic in all layers and clarifying some boundaries.\nThis is still the very early stages of the project with a lot of room for improvement on the code base, but at least the foundation is now clean: when I work on a feature, it is clear where I should add the feature and I don\u0026rsquo;t risk breaking everything.\nCode made available on Github # Since I\u0026rsquo;m no longer ashamed of what I produced, I decided to work in the open and make the code available in a github repository so I can reference a commit when I write about plakar here instead of leaving everything to your imagination.\nDon\u0026rsquo;t use the project for anything serious:\nFirst of all, there are still issues and you don\u0026rsquo;t want to use something that has issues for your backups. Then, the utility still evolves a lot with regard to output or CLI, and I don\u0026rsquo;t want to provide support regarding last weeks\u0026rsquo; command line or output format when I have a different version on my laptop. Finally, the storage format hasn\u0026rsquo;t fully stabilized yet as I make minor changes here and there to optimize some operations: the snapshots you make today may not be restorable next week if I make use of a new metadata or change a data structure.\nOf course, I\u0026rsquo;d be very happy to have some people test and help me improve the tool, so feel free to test and report issues\u0026hellip; as long as this is only for testing and not for your real backups.\nImproved parallelism # Because of how plakar works, many operations involve splitting data into chunks or fetching data from the store to reconstruct objects.\nFor example, plakar cat deadbeef:/etc/passwd will:\nfetch the index for snapshot deadbeef from the store (or from cache if available) resolve /etc/passwd into an object identifier within the store fetch the object index from the store (or from cache if available) resolve the object into one or many chunks within the store fetch the chunks from store and output them in sequence to stdout Whereas plakar push /bin will iterate through /bin and:\nopen and split every file into chunks query the store for which chunks need to be written or have their reference count incremented write the missing chunks write an object index for every individual object write the snapshot index which maps the path names to their backing objects, among other things In my initial work on plakar, all commands were implemented with a very sequential approach, operations being performed one after another regardless if they could be parallelized or not. There was a tiny bit of parallelism, as plakar could work on multiple files at once, but it would take a sequential approach for each of these files.\nIn many cases, this caused it to operate at much slower speed than was possible to reach given the available resources on the host system. For instance, in the following screenshots, it took roughly 55s to push my 2.5GB ~/Downloads directory to an empty repository on my ~1000MB/s/writes SSD despite the 8 cores being underused.\nI reworked the storage and the snapshot layers to improve parallelism and synchronization so that plakar could safely parallelize every possible operation. This allowed it to fully exploit all available cores and avoid idling when it could be doing something, like writing multiple chunks in parallel for the same file while processing multiple files, and letting disk I/O limit the performances.\nIn the screenshots below, it took only 14.4s to push the same directory to an empty store while utilizing all cores to their fullest. It could certainly be optimized further because the commit of a snapshot has not gone through optimization yet and probably consumes over half of this time, but the difference is already quite impressive.\nThis parallelism improvement is not restricted to pushing snapshots, but benefits all plakar operations. For instance restoring snapshots became significantly faster as it used to restore all files sequentially when it will now parallelize the whole process of rebuilding the file hierachy, creating the files and fetching the chunks.\nI was afraid that the amount of goroutines would be unmanageable as I hit a couple panics when working on directories \u0026gt; 10GB with tens of thousands of files and even more chunks, but I came up with a code pattern that ensured I could both control the amount of concurrent routines executing a specific task, and that they were all done before beginning a dependent task. I\u0026rsquo;ll probably write about that in a future post as I find this solution very elegant and suitable for many use-cases.\nI will likely be adding an option to specify the amount of parallelism allowed to prevent plakar from hogging all resources at the cost of slower operations, but it is a very simple feature that I\u0026rsquo;ll keep for when I want to pretend I worked hard on something :-)\nSnapshots and objects caching # When plakar creates a snapshot, it has to scan file hierarchies and read every files to split them into chunks that will be checked against the store. This isn\u0026rsquo;t something that has to be avoided, it is really what I want it to do, but it is not always desirable as it is only necessary for files that have changed. Unless plakar is able to detect that, it has to pay the price of reading all files including the ones that haven\u0026rsquo;t changed.\nI implemented a local cache which recalls, among other things, the structure of objects that were part of previous snapshots and the inode informations associated to path names.\nWhenever scanning a directory, the inode informations for the current file are compared to the last known informations stored in the cache. If it is determined that the file has not changed since the last version, plakar will reuse the chunk information from the cache and increase the chunks reference count in store. This avoids a full read during the chunking and a potential full write of all chunks to the store.\n$ plakar -time push ~/Downloads time: 13.787579792s $ plakar -time push ~/Downloads time: 1.41720375s $ plakar -time -no-cache push ~/Downloads time: 7.432093292s $ plakar ls 2021-10-26T11:51:01Z e5c24e03-7f72-4a9b-81d4-2b891951f965 2.5 GB (files: 1225, dirs: 282) 2021-10-26T11:51:20Z 784a2a28-0af5-45e7-b440-1246c5691080 2.5 GB (files: 1225, dirs: 282) 2021-10-26T11:51:26Z e6a24d44-f257-4540-8ff7-04f251acf522 2.5 GB (files: 1225, dirs: 282) In the example above, I pushed my 2.5GB ~/Downloads directory in an empty plakar, then pushed it again once with caching and once without caching. Here, the time difference for a push with or without cache is very significant, but it is highly dependant on the number of directories, of files, of their redundancy and their sizes.\nJust like for parallelism, the local cache doesn\u0026rsquo;t only benefit the creation of snapshots but also other operations, though the boost is not as beneficial due to its current implementation: it requires small disk reads for each lookup and while this can save a lot in the snapshot creation code path by avoiding big file reads and tons of writes, it is not that interesting in code paths involving mostly the reading of chunks.\nThere is still a lot of room for improvement though:\nFirst of all, the structure of informations in cache is not optimal and makes it hard to use the cache for some operations that could benefit from it. This is something simple to fix, it\u0026rsquo;s just that I figured what I should have done differently while I was writing this article :-)\nThen, I implemented the cache using individual files to represent snapshots and objects, which means each cache query involves opening, reading and closing a file. This was acceptable to bootstrap the API and start using it but has a lot of overhead which would not exist if the cache was stored in an SQLite database or similar.\nI will continue working on local cache improvements but am already happy with the results of that first naive approach.\nFor console pr0n, here\u0026rsquo;s a run of a push with an empty cache and empty store:\n$ plakar -trace push . 332752a6-c220-4d7f-ab22-290a1c67862f: New() 332752a6-c220-4d7f-ab22-290a1c67862f: cache.GetPath(./group): KO 332752a6-c220-4d7f-ab22-290a1c67862f: cache.PutPath(./group) 332752a6-c220-4d7f-ab22-290a1c67862f: cache.GetPath(./services): KO 332752a6-c220-4d7f-ab22-290a1c67862f: PutChunk(4910bfe2b7e551c4e2085b12c36941d1e1063491b7292cb0dbca7c5fe0854be5) 332752a6-c220-4d7f-ab22-290a1c67862f: cache.PutPath(./services) 332752a6-c220-4d7f-ab22-290a1c67862f: cache.GetPath(./passwd): KO 332752a6-c220-4d7f-ab22-290a1c67862f: cache.PutPath(./passwd) 332752a6-c220-4d7f-ab22-290a1c67862f: PutChunk(e45b72f5c0c0b572db4d8d3ab7e97f368ff74e62347a824decb67a84e5224d75) 332752a6-c220-4d7f-ab22-290a1c67862f: PutChunk(8c6e2a2647ee854f469a3bb798e02ba5a8b1812cab229ff129f073e7a80c1202) 332752a6-c220-4d7f-ab22-290a1c67862f: PutObject(8c6e2a2647ee854f469a3bb798e02ba5a8b1812cab229ff129f073e7a80c1202) 332752a6-c220-4d7f-ab22-290a1c67862f: PutObject(4910bfe2b7e551c4e2085b12c36941d1e1063491b7292cb0dbca7c5fe0854be5) 332752a6-c220-4d7f-ab22-290a1c67862f: PutObject(e45b72f5c0c0b572db4d8d3ab7e97f368ff74e62347a824decb67a84e5224d75) 332752a6-c220-4d7f-ab22-290a1c67862f: PutIndex() snapshot: cache.PutIndex(332752a6-c220-4d7f-ab22-290a1c67862f) 332752a6-c220-4d7f-ab22-290a1c67862f: Commit() And here\u0026rsquo;s a run of a push using the cache:\n$ plakar -trace push . 0c8ab5bf-add9-4e20-9790-97d43cde35cc: New() 0c8ab5bf-add9-4e20-9790-97d43cde35cc: cache.GetPath(./group): OK 0c8ab5bf-add9-4e20-9790-97d43cde35cc: cache.GetPath(./services): OK 0c8ab5bf-add9-4e20-9790-97d43cde35cc: cache.GetPath(./passwd): OK 0c8ab5bf-add9-4e20-9790-97d43cde35cc: PutIndex() snapshot: cache.PutIndex(0c8ab5bf-add9-4e20-9790-97d43cde35cc) 0c8ab5bf-add9-4e20-9790-97d43cde35cc: Commit() Push strategy # When plakar wasn\u0026rsquo;t parallelized and didn\u0026rsquo;t have a cache, it was easier to interact with the store on an object-after-object basis: it would open a file, parse it into chunks, write the chunks, write the object, then move to the next object. This ensured that the full handling of a file could be isolated in a small process: everything that needed to be done with the file would be done in the window of time that it was opened, all in a very readable and sequential logic.\nWith parallelization and caching, it became more interesting to work in a different way: processing chunks at full speed first, THEN processing object indexes at full speed when all chunks are already recorded in the transaction. This allowed to simplify the logic by a great deal, replacing the sequential pattern of read chunk / write chunk / read next chunk / write next chunk, with a channel where chunks are pushed out of order as they are read and goroutines concurrently pop from the channel to write to store as fast as they can. This also had the benefit to ease deduplication of chunks and objects within the snapshot itself.\nThe current push implementation is already much more interesting than the previous one but I\u0026rsquo;m not done yet and the good thing is that, thanks to the refactor, snapshots are built on top of the storage and I\u0026rsquo;m able to experiment knowing that all my changes are isolated in that layer.\nFUSE experiment # Another feature I\u0026rsquo;m experimenting with is FUSE, an API to implement user-space filesystems.\nI\u0026rsquo;m only beginning to play with it but have implemented plakarfs, a read-only user-space filesystem exposing a plakar repository, and plakar mount which allows mounting a plakarfs. This makes it possible to browse snapshots and read files, as if all snapshots were restored, but\u0026hellip; without consuming the disk space:\n$ plakar ls 2021-10-26T10:43:12Z 9aed1ded-0e0d-4d6c-946a-6757178ec2f3 3.2 MB (files: 248, dirs: 42) 2021-10-26T10:43:13Z ecfef47f-24fc-4c6a-9dbc-fdf918f40689 3.2 MB (files: 248, dirs: 42) 2021-10-26T10:43:13Z 561edc6a-e531-4770-872a-610432d0a2b9 3.2 MB (files: 248, dirs: 42) $ plakar mount /tmp/plakar \u0026amp; [1] 19253 $ ls -l /tmp/plakar total 18912 dr-xr-xr-x 1 root wheel 3224317 26 Oct 12:43 561edc6a-e531-4770-872a-610432d0a2b9 dr-xr-xr-x 1 root wheel 3224317 26 Oct 12:43 9aed1ded-0e0d-4d6c-946a-6757178ec2f3 dr-xr-xr-x 1 root wheel 3224317 26 Oct 12:43 ecfef47f-24fc-4c6a-9dbc-fdf918f40689 $ tail /tmp/plakar/ecfef47f-24fc-4c6a-9dbc-fdf918f40689/etc/passwd _logd:*:272:272:Log Daemon:/var/db/diagnostics:/usr/bin/false _appinstalld:*:273:273:App Install Daemon:/var/db/appinstalld:/usr/bin/false _installcoordinationd:*:274:274:Install Coordination Daemon:/var/db/installcoordinationd:/usr/bin/false _demod:*:275:275:Demo Daemon:/var/empty:/usr/bin/false _rmd:*:277:277:Remote Management Daemon:/var/db/rmd:/usr/bin/false _fud:*:278:278:Firmware Update Daemon:/var/db/fud:/usr/bin/false _knowledgegraphd:*:279:279:Knowledge Graph Daemon:/var/db/knowledgegraphd:/usr/bin/false _coreml:*:280:280:CoreML Services:/var/empty:/usr/bin/false _trustd:*:282:282:trustd:/var/empty:/usr/bin/false _oahd:*:441:441:OAH Daemon:/var/empty:/usr/bin/false $ The filesystem hierarchy within the mount is built using snapshot indexes, including inode informations such as creation time, uid/gid or file size, while file reading is implemented by fetching the object index and just the chunks needed to service the read, so that chunks will be fetched as needed based on the current offset within the file.\nI have a hard time getting around with the proper way to implement plakar as a filesystem, so there are still a lot of glitches and a few limitations, but this looks very promising as far as I\u0026rsquo;m concerned. In particular because this is built on top of snapshots, themselves built on top of the store, making this work transparently with encrypted or remote stores.\nRemote plakar # I had already written that I implemented server and client support for plakar in this post, but the idea behind it was only to validate that the storage primitives could be used over the network, not to be actually usable. The client and server were implemented in a very ugly way, hacked as custom storage backends, lacking error checking and not designed to handle concurrency within a single snapshot transaction.\nI\u0026rsquo;m now at the point were I want to implement them correctly, so I have started looking at the transport and working on the protocol, experimenting with different things. I tested three PoCs at this point but am not satisfied yet.\nAnyhow, I expect this network mode to be working by the end of December as I\u0026rsquo;d like to have a plakar running on my NAS by then :-)\nAssorted other work # I also worked on other stuff but I didn\u0026rsquo;t want to pollute this post so I will just summarize:\nI submitted a diff to OpenBSD to fix regex support in OpenSMTPD\u0026rsquo;s table_db backend ( committed), helped eric@ fix and review a diff for a bug in SRS ( committed) and am currently reviewing a diff to integrate my table_procexec as the backend to run external tables in OpenSMTPD.\nI worked on a new project that I\u0026rsquo;ll write about in a few months as it is not high priority and I\u0026rsquo;d like to get plakar mostly out of my way first, but it involves mobile development, something I haven\u0026rsquo;t done in over a decade.\nI also played a bit with Arduino to learn myself some new skills, but I didn\u0026rsquo;t do anything newsworthy.\n","date":"26 October 2021","permalink":"/posts/2021-10-26/october-2021-mostly-plakar/","section":"Posts","summary":"TL;DR: I have a discord now, feel free to join. I refactored plakar, implemented a local cache, improved parallelism, modified the push strategy, played with fuse and networking.","title":"October 2021: mostly Plakar"},{"content":" TL;DR: this is a short article, no TL;DR. Why so calm ? # As some of you know, I was diagnosed a few years ago with generalized anxiety disorder, panic disorder and alexithymia, a trait that makes it particularly difficult for me to cope with the disorders.\nAs I wrote in 2019, I didn\u0026rsquo;t take the alexithymia diagnosis too well. It took me a long time to stop ruminating, move forward, and start trying to find a fix for myself with virtually no help. It turns out that I have made huge progress thanks to hypnosis and efforts at trying to decode sensations into emotions, but the road is still long and full of ups and downs.\nThis brings us to 2021, two years later.\nToo many things hinted me that a neurodivergence was at the root of my problems, I could no longer pretend not to see it. I took a psychological assessment which confirmed my intuition, then took the report to a different psychologist for a second opinion and got a confirmation.\nI\u0026rsquo;m neurodivergent.\nI don\u0026rsquo;t want to share the details because it doesn\u0026rsquo;t really matter how divergent I am, but just like in 2019, I don\u0026rsquo;t take this news too well and I need some time to accept it.\nI\u0026rsquo;m ok, no worries, nothing a summer alone with my wife and kid can\u0026rsquo;t fix.\nA quick word about sponsorship and this blog # If you are sponsoring me, please take a time to reconsider if you still want to support me.\nI have 5 days each month that I can dedicate however I want and, out of these 5 days, 1 is almost fully consumed by the monthly report because I\u0026rsquo;m a slow writer. I only have 4 days I can use to work on stuff and, because the monthly report can\u0026rsquo;t be empty, within these 4 days I must find things to do that I can write about.\nThis doesn\u0026rsquo;t leave much room for learning new skills and doing useless experiments. I find myself working on interesting stuff (to me), then watching the clock because I want to leave enough time to work on something I can write about. Who wants to read about my experiments converting EEG metrics to midi so I can listen to my brain waves ? Obviously, people prefer when I write code that brings new features in OpenSMTPD.\nIn June and July, I spent my days unsuccessfully trying to do some VR, and this didn\u0026rsquo;t leave me time to write the monthly reports. I felt bad for my sponsors as I couldn\u0026rsquo;t deliver reports, I had 0 things to write.\nSince the beginning of August, I\u0026rsquo;m experimenting with Dart/Flutter to write a mobile app using a Golang backend and a cockroach database. It\u0026rsquo;s a fun project that I\u0026rsquo;ll eventually publish and write about but if I want to produce a monthly report, I need to stop right away and dive into code I\u0026rsquo;m familiar with so I know I can produce something fast enough, rather than continue working on a fun new project that I can\u0026rsquo;t write about yet.\nI find this counter-productive, the monthly reports are preventing me from working on what I want, I think it is time to stop this format which is putting too much pressure on me and limiting future projects.\nInstead, I will write reports when I have valuable informations to share, and continue to commit everything to Github. Sometimes you will see a lot of activity on consecutive months, sometimes you will not see anything happening for a quarter.\nIf you\u0026rsquo;re not ok with that, I understand and am grateful for your support until this point \u0026lt;3\nI sent a notice to all my patrons so they are aware that I will change my way of working.\nLove \u0026lt;3\n","date":"13 August 2021","permalink":"/posts/2021-08-13/august-2021-taking-a-break/","section":"Posts","summary":"TL;DR: this is a short article, no TL;DR. Why so calm ? # As some of you know, I was diagnosed a few years ago with generalized anxiety disorder, panic disorder and alexithymia, a trait that makes it particularly difficult for me to cope with the disorders.","title":"August 2021: taking a break"},{"content":" TL;DR: J\u0026rsquo;explique superficiellement comment fonctionne notre esprit, en partant du corps, puis de notre sens de la réalité, avant de finir par les états de conscience et personnalités. Je plonge brièvement dans les rêves et l\u0026rsquo;hypnose, préparant le terrain pour un article à venir. Avertissement # Mon intérêt pour les rêves lucides a débuté aux alentours de 2013, j\u0026rsquo;ai étudié seul puis suivi différentes formations de différent type d\u0026rsquo;hypnose entre 2015 et 2020, et je pratique l\u0026rsquo;hypnose en cabinet depuis 2016 jusqu\u0026rsquo;à maintenant. Sans lien avec cette activité, j\u0026rsquo;étudie la psychologie du travail depuis 2016 et ai, dans ce cadre, suivi des cours d\u0026rsquo;introduction à la psychologie clinique et à la psychologie cognitive.\nJe ne suis ni psychologue, ni psychiatre, ni un chat, alors prenez cet article avec le recul adéquat.\nÀ propos de cet article # Cet article est une traduction de mon article Mind hacking: understanding how our mind works publié sur ce blog\nJe tente d\u0026rsquo;expliquer, d\u0026rsquo;un point de vue macroscopique, ma compréhension de comment fonctionne l\u0026rsquo;esprit, en me basant sur ce que j\u0026rsquo;ai étudié, ainsi que mes observations et expériences personnelles au travers du rêve lucide et de l\u0026rsquo;hypnose.\nCet article ne contient pas de nouvelles idées révolutionnaires, vous y reconnaîtrez peut être quelques éléments des travaux de Charcot, Janet, Breuer et même Freud, même si la façon dont je les présente peut varier car je n\u0026rsquo;adhère pas nécessairement à tout et adapte donc à MA compréhension.\nIl s\u0026rsquo;agit d\u0026rsquo;une vulgarisation qui prends des raccourcis volontaires pour véhiculer les idées principales le plus simplement, il ne s\u0026rsquo;agit absolument pas d\u0026rsquo;une explication exhaustive et chaque concept pourrait être détaillé sur des pages entières. Je serais ravi de développer dans les commentaires ou des articles suivants.\nEn résumé: ne partez pas du principe que cet article s\u0026rsquo;applique tel quel à tous, particulièrement aux personnes avec des troubles qui affectent leur sens de la réalité ou leur/s personnalité/s, il ne s\u0026rsquo;agit pas d\u0026rsquo;une description exacte de comment l\u0026rsquo;esprit fonctionne pour tous.\nNous sommes des ordinateurs biologiques # Nous sommes des ordinateurs biologiques qui résultent des spécifications combinées de deux autres ordinateurs biologiques. Ces spécifications déterminent comment nous sommes construits, à quoi nous ressemblons, quelles sont nos aptitudes et nos limites, mais pas comment nous traitons le monde, pas comment fonctionne notre système.\nNous nous interfaçons avec le monde au travers de nombreux capteurs, appelés récepteurs, répartis dans nos yeux, oreilles, langue, nez et sous notre peau pour capturer des signaux visuels, auditifs, gustatifs, olfactifs et kinesthésiques qui sont ensuite interprétés et nous permettent de voir, entendre, goûter, sentir et toucher les éléments de notre environnement. Ces récepteurs forment notre appareil sensoriel, ce que nous appelons nos sens.\nCes récepteurs font partie de ce que des développeurs pourraient considérer comme notre API. Ils sont les seules interfaces qui permettent de percevoir des signaux d\u0026rsquo;entrée depuis notre environnement, permettant à notre système de les traduire en sensations et d\u0026rsquo;encoder des souvenirs. Sans récepteurs, aucun signal n\u0026rsquo;entrerai dans notre système, nous vivrions alors une vie très terne de néant absolu. Par chance, nous avons des récepteurs dans nos yeux pour voir une tablette de chocolat blanc sur la table, sur nos mains pour sentir son contact lorsque nous l\u0026rsquo;attrapons, dans notre nez et notre langue pour sentir son odeur et son goût, et dans nos oreilles pour entendre quelqu\u0026rsquo;un demander où est passée la tablette. Oups.\nParce que nos sens sont notre seul interface d\u0026rsquo;entrée, tout ce que nous savons de notre environnement provient de nos récepteurs. Notre connaissance n\u0026rsquo;est pas apparue en nous, elle a été insérée au travers de nos sens, soit parce qu\u0026rsquo;un autre ordinateur biologique nous as transmis des signaux pour partager l\u0026rsquo;information, soit parce que nous avons interagis avec notre environnement et reçus des signaux en retour.\nLa plupart des signaux sont ignorés, sauf si l\u0026rsquo;on y prête attention, parce qu\u0026rsquo;ils sont sans importances pour nos activités courantes, comme c\u0026rsquo;est le cas pour le bruit d\u0026rsquo;une horloge que l\u0026rsquo;on entend mais oublie comme si elle n\u0026rsquo;était plus là\u0026hellip; alors qu\u0026rsquo;elle l\u0026rsquo;est toujours. Beaucoup de signaux sont traités à un bas-niveau inconscient et enregistrés pour un usage ultérieur, comme la pensée passée d\u0026rsquo;une tablette de chocolat blanc dont on a un peu envie là maintenant. Quelques signaux sont traités et rendus disponibles immédiatement à notre conscience pour que nous puissions agir dessus, comme ce texte surligné qui attire immédiatement notre attention.\nCes récepteurs sont ce qui nous aide à développer et customiser notre système. Ils nous rendent disponible de l\u0026rsquo;information concernant notre réalité, ce qui change la façon dont on traite notre environnement et interprète de nouveaux signaux. Notre système construit une compréhension du monde au travers d\u0026rsquo;une boucle de rétroaction sensorielle.\nLa réalité # Il peut sembler étrange de discuter du concept de réalité dans un article sur le fonctionnement de notre esprit mais ce n\u0026rsquo;est pas le cas. Nous vivons dans une réalité et tout ce nous faisons lui est relative: on y vient à la vie, on en apprend les règles au travers d\u0026rsquo;essais et d\u0026rsquo;erreurs, puis on développe des pensées et des comportements qui nous semblent y être adaptés. On ne peut pas commencer à comprendre comment notre esprit fonctionne sans comprendre comment il perçoit la réalité\u0026hellip; parce que la façon dont fonctionne notre esprit est dérivé de cette perception.\nQuand on pense à la réalité, on pense généralement à une réalité unique, LA réalité, et on sépare ensuite le monde en deux: d\u0026rsquo;un côté se trouvent ceux qui partagent la même réalité et que l\u0026rsquo;on considère sains, et de l\u0026rsquo;autre se trouvent ceux qui ne la partagent pas et qui sont fous. Si quelqu\u0026rsquo;un nous confie voir des cochons volants au dessus de nos têtes et qu\u0026rsquo;on ne les voit pas en levant les yeux, on reconnaît que sa perception de la réalité est inexacte parce que les cochons volants ne sont pas là. On s\u0026rsquo;attend à ce que les personnes autour de nous fassent l\u0026rsquo;expérience de la réalité de la même manière que nous parce que, s\u0026rsquo;il n\u0026rsquo;y a que LA réalité, alors il ne peut pas y avoir deux évènements contradictoires qui surviennent simultanément.\nMais ce n\u0026rsquo;est pas si simple.\nNous avons tous des aperçus différents de la réalité, et en développons un sens très personnel basé sur nos perceptions, même si elle est en grande partie partagée avec les autres être vivants. Nous pouvons vivre dans la même réalité, mais la mienne reste différente de celle d\u0026rsquo;une personne daltonienne dont les signaux visuels sont traduits différemment, ou d\u0026rsquo;une personne aveugle dont les signaux visuels sont absents. Nous pouvons vivre dans la même réalité mais en avoir une expérience différente\u0026hellip; parce que la réalité a de multiples dimensions.\nLa réalité objective # La réalité dans laquelle nous existons est la réalité objective, factuelle et contrainte par les lois de la physique, où les choses existent et se déroulent même si nous n\u0026rsquo;en avons pas conscience. Il s\u0026rsquo;agit de LA réalité dont la plupart d\u0026rsquo;entre nous pense faire l\u0026rsquo;expérience, une réalité unique qui est la même pour tous. Elle comprends tout ce qui a été par le passé, tout ce qui est dans le présent et tout ce qui sera dans le futur.\nTout ce qui existe dans l\u0026rsquo;univers, depuis chaque étoile dans chaque galaxie jusqu\u0026rsquo;à chaque grain de sable sur notre planète, fait partie de la réalité objective et existe que l\u0026rsquo;ont en ai connaissance ou non, et que nous puissions interagir avec ou non. Tous les évènements qui surviennent, n\u0026rsquo;importe où dans l\u0026rsquo;univers, font partie de cette réalité objective que l\u0026rsquo;on puisse les observer ou non. Ils existent et surviennent factuellement, que l\u0026rsquo;on le sache ou non.\nDans la réalité objective, si je vous dit que j\u0026rsquo;ai un jour mangé 1kg de chocolat blanc, cela n\u0026rsquo;a pas d\u0026rsquo;importance que vous me croyez ou non, il y a une vérité factuelle: c\u0026rsquo;est arrivé ou ce n\u0026rsquo;est pas arrivé. Si l\u0026rsquo;on est en désaccord, alors l\u0026rsquo;un de nous à tort. Il n\u0026rsquo;y a pas d\u0026rsquo;espace pour le débat dans cette réalité.\nLes signaux que perçoivent nos sens de notre environnement trouvent leur origine dans cette réalité objective, ils sont factuels et n\u0026rsquo;embarquent pas de signification subjective. Dans la réalité objective, le chocolat blanc a une odeur qui peut être perçue par nos récepteurs, mais cette odeur n\u0026rsquo;est ni plaisante, ni déplaisante, elle n\u0026rsquo;existe qu\u0026rsquo;en tant que signal qui peut être traduit et interprété par quelqu\u0026rsquo;un.\nLa réalité subjective # La réalité objective est infinie et complexe, et nous n\u0026rsquo;en sommes exposés qu\u0026rsquo;à une portion infinitésimale, ce qui nous empêche d\u0026rsquo;en faire l\u0026rsquo;expérience telle qu\u0026rsquo;elle est réellement. À sa place, nous faisons l\u0026rsquo;expérience d\u0026rsquo;une version dégradée qui ne contient que ce à quoi nous sommes exposés, et ce que nos sens parviennent à y percevoir\u0026hellip; une version différente pour chacun.\nC\u0026rsquo;est la réalité subjective, une représentation interne de la réalité objective qui est affectée par nos perceptions, et qui tente de remplir les trous des informations manquantes par des interprétations, des suppositions et l\u0026rsquo;extrapolation d\u0026rsquo;expériences passées. Elle est subjective parce qu\u0026rsquo;elle diffère d\u0026rsquo;une personne à une autre, ce n\u0026rsquo;est pas une réalité factuelle.\nSi je vous disais que j\u0026rsquo;ai mangé 1kg de chocolat blanc, peut être me croiriez vous parce que vous l\u0026rsquo;avez aussi fait une fois, et peut être quelqu\u0026rsquo;un d\u0026rsquo;autre ne me croirait pas parce que ça semble beaucoup. Dans la réalité objective, je l\u0026rsquo;ai fait ou non, mais il y a deux réalités subjectives pour deux personnes différentes: une où je l\u0026rsquo;ai fait et une ou je ne l\u0026rsquo;ai pas fait. Dans chacune de ces réalités existe une version différente de moi, une qui dit la vérité et une qui ment, ce qui entraîne un traitement différent de l\u0026rsquo;information quand elle provient de moi et selon qui la reçoit. Des personnes peuvent vivre dans la même réalité objective et la voir de façon conflictuelle.\nÇa ne veut pas dire que tout le monde a une réalité radicalement différente, nous vivons dans LA réalité objective où les lois de la physique s\u0026rsquo;appliquent, nous recevons les mêmes signaux. Si l\u0026rsquo;on regarde la même série, nous verrons les mêmes images et nous entendrons les mêmes dialogues, nous aurons peut être des différences parce que l\u0026rsquo;on prête plus ou moins attention aux mêmes détails, que les signaux auront été traduits un peu différemment ou même qu\u0026rsquo;une partie des signaux sera inaccessible à l\u0026rsquo;un ou l\u0026rsquo;autre, mais nous aurons à peu près la même expérience. Les divergences apparaissent lors qu\u0026rsquo;il manque de l\u0026rsquo;information et que l\u0026rsquo;on est forcé de remplir les trous avec des interprétations, des suppositions et des extrapolations, plutôt qu\u0026rsquo;avec nos perceptions.\nAu long de cet article, lorsque j\u0026rsquo;utilise le terme réalité sans plus de précisions, alors c\u0026rsquo;est de la réalité subjective dont je parle car c\u0026rsquo;est d\u0026rsquo;elle dont nous parlons en général.\nLes réalités alternatives # Bien que les réalités peuvent varier d\u0026rsquo;une personne à une autre, on peut avoir des attentes raisonnables concernant les réalités des autres parce que nous partageons la réalité objective et sommes soumis à ses règles.\nSi une autre personne et vous me regardiez jeter une pierre, je peux m\u0026rsquo;attendre à ce que vous ayez tous les deux vu la pierre tomber et non disparaître en l\u0026rsquo;air, flotter au dessus de ma tête ou décoller vers la Lune. Peut être n\u0026rsquo;avez-vous pas vu tous les détails, peut être vous pensez m\u0026rsquo;avoir vu jeter une canette et l\u0026rsquo;autre personne m\u0026rsquo;a vu jeter une pièce. Vous pouvez être en désaccord sur les détails mais, si vous avez observé le même évènement, vous devriez avoir vu plus ou moins la même chose car vous ne faites qu\u0026rsquo;interpréter, et non transformer, la réalité objective. Si ce n\u0026rsquo;était pas le cas, l\u0026rsquo;industrie du pari s\u0026rsquo;effondrerait.\nUne réalité alternative est une réalité qui est incompatible avec la réalité objective et, par extension, la réalité de la plupart des personnes. Si quelqu\u0026rsquo;un perçoit réellement des images d\u0026rsquo;une pierre en train de léviter lors de mon jet, alors sa réalité serait incompatible avec la réalité objective car elle défierait les lois de la physique, mais aussi avec celle des autres personnes qui ont tous vu quelque chose tomber.\nLes réalités alternatives ont leurs origines dans les autres dimensions de la physique quan\u0026hellip; euh, non, ça c\u0026rsquo;est le pilote de Sliders.\nUne réalité alternative c\u0026rsquo;est si quelqu\u0026rsquo;un me dit qu\u0026rsquo;elle est Cléopatre, souveraine du Royaume d\u0026rsquo;Égypte, et que nous sommes actuellement assis au bord du Nile en l\u0026rsquo;an -40 avant JC (passons sur l\u0026rsquo;anachronisme). On peut être en désaccord sur le fait qu\u0026rsquo;elle soit Cléopatre, souveraine du Royaume d\u0026rsquo;Égypte, mais il y a de multiples incompatibilités entre nos réalités qui les rendent impossible à coéxister dans la réalité objective: soit nous sommes en -40, soit en 2021, ça ne peut pas être les deux; soit nous sommes en France, soit en Égypte, ça ne peut pas être les deux. Au moins l\u0026rsquo;une de nos réalité est incompatible avec la réalité objective et est, de facto, une réalité alternative. Pour chacun de nous, notre réalité subjective est celle qui est construite sur la réalité objective, c\u0026rsquo;est l\u0026rsquo;autre qui vit une réalité alternative.\nÇa peut aussi être plus subtile que la caricature du dessus. Les conspirationnistes, tels que les Platistes ou une partie des anti-vax, ont des réalités compatibles avec la réalité objective\u0026hellip; sauf dans les parties qui tournent autour de la conspiration. C\u0026rsquo;est seulement lorsque ces sujets sont évoqués que les conflits et incompatibilités émergent, les contraignant à travailler dur pour préserver leurs réalités et empêcher qu\u0026rsquo;elles ne s\u0026rsquo;effondrent.\nLes réalités alternatives ne sont pas que l\u0026rsquo;expérience d\u0026rsquo;une pathologie, nous en faisons tous l\u0026rsquo;expérience fréquemment.\nTous les jours, nous allons nous coucher et notre personnage onirique plonge dans des réalités alternatives rêvées, qui n\u0026rsquo;ont aucun lien avec la réalité objective. Elles peuvent être très similaires ou différentes, réalistes ou étranges, mais du point de vue de notre esprit, ce sont des réalités alternatives qui remplacent notre réalité jusqu\u0026rsquo;à ce que l\u0026rsquo;on soit de nouveau conscient lors de notre réveil.\nC\u0026rsquo;est également ce qui arrive aux personnes en état de transe somnambulique en hypnose, qui hallucinent des objets et se comportent comme s\u0026rsquo;ils étaient quelqu\u0026rsquo;un d\u0026rsquo;autre. Ils sont entraînés à remplacer leur réalité par une réalité alternative, similaire à un monde onirique, jusqu\u0026rsquo;à être réveillés de l\u0026rsquo;état d\u0026rsquo;hypnose. Du point de vue de leur esprit, ils sont dans une réalité alternative le temps de l\u0026rsquo;expérience.\nCe qui différencie les rêveurs et les hypnotisés de la pathologie est que leur réalité alternative ne persiste pas. Lorsque l\u0026rsquo;on se réveille de nos rêves ou de nos états d\u0026rsquo;hypnose, on reconnaît la réalité alternative comme telle et on se reconnecte à notre réalité, elle ne persiste pas en tant que réalité subjective.\nL\u0026rsquo;état de conscience ordinaire (ECO) # Tous les jours, nous nous réveillons et commençons à reprendre contact avec notre réalité.\nEn nous y reconnectant, nous entrons dans notre état de conscience ordinaire (ECO), un état dans lequel nous sommes conscients de l\u0026rsquo;environnement et capables d\u0026rsquo;interagir avec lui de manière compréhensible. L\u0026rsquo;ECO est un état dans lequel on se sent\u0026hellip; normal, ou du moins familier. C\u0026rsquo;est l\u0026rsquo;état dans lequel nous fonctionnons le plus souvent pour interagir avec l\u0026rsquo;environnement, celui dans lequel nous passons le plus de temps lors que nous sommes éveillés.\nCertaines personnes n\u0026rsquo;aiment pas cette notion d\u0026rsquo;ECO qui semble impliquer que nous serions la même personne tous les jours, mais c\u0026rsquo;est une façon inexacte de comprendre les choses. l\u0026rsquo;ECO n\u0026rsquo;est pas un état statique, c\u0026rsquo;est un état dynamique qui englobe plusieurs niveaux de conscience depuis la fatigue jusqu\u0026rsquo;à une grande activité, c\u0026rsquo;est juste un état dans lequel notre conscience de l\u0026rsquo;environnement prends la plupart de notre espace mental. On est éveillé, conscient et alerte.\nUn ECO implique qu\u0026rsquo;il y a au moins un état non ordinaire de conscience\u0026hellip; mais il y a en réalité plusieurs états de conscience modifiée.\nLes états de conscience modifiée (ECM) # Lors que l\u0026rsquo;on mentionne les états de conscience modifiée (ECM), de nombreuses personnes pensent que l\u0026rsquo;on bascule dans le domaine des théories hippies new-age, mais nous entrons tous dans des ECM observables et reconnaissables tous les jours.\nUn ECM est simplement un état qui n\u0026rsquo;est pas notre ECO, rien de plus et rien de moins: pas de chakras, de forces énergétiques, rien d\u0026rsquo;autre.\nCertains de ces états sont déclenchés volontairement par nos actions, comme se saouler en absorbant de l\u0026rsquo;alcool ou se défoncer en consommant des drogues. L\u0026rsquo;expression \u0026ldquo;sous influence\u0026rdquo; implique elle-même que notre état de conscience est altéré, les actions faites \u0026ldquo;sous influence\u0026rdquo; ne sont pas celles de quelqu\u0026rsquo;un en ECO.\nLes ECM peuvent aussi être déclenchés par des évènements qui provoquent une courte sortie de notre normalité, comme par exemple lorsque l\u0026rsquo;on entre en état de choc après avoir été impliqué dans un accident, ou que l\u0026rsquo;on est euphorique après avoir accompli quelque chose d\u0026rsquo;extraordinaire. Ces évènements nous font temporairement basculer dans un ECM plus ou moins éloigné de notre ECO.\nPlus fréquent que d\u0026rsquo;être saoul ou dans un état de choc, le rythme circadien de la vigilance déclenche un ECM en moyenne toutes les 90 minutes, provoquant une légère perte de conscience de l\u0026rsquo;environnement et une absorption dans les pensées, alors que le rythme circadien de la veille-sommeil déclenche plusieurs ECM, environ toutes les 24 heures, quand nous nous endormons.\nLa plupart de ces états sont mesurables et observables, soit par des mesures EEG, soit par des changements physiologiques visibles. Par exemple, personne ne peut nier que les ECM liés au cycle du sommeil sont similaires à l\u0026rsquo;ECO, ils présentent des différences aussi bien dans les mesures EEG que dans les changements physiologiques visibles.\nLa personnalité principale # Depuis notre plus jeune âge, nous touchons à tout et observons les conséquences de nos actions, on nous dit de faire des choses et de ne pas en faire d\u0026rsquo;autres, on obéit et on désobéit, on réussit, on rate, on se fait mal, on aime, on déteste, on ressent des choses plaisantes et déplaisantes, et tout ça affecte la manière dont on fait face à des situations similaires et celle dont on approche les nouvelles.\nCes expériences sont internalisées et deviennent partie de notre identité, on construit une personnalité dont les traits en sont hérités, une personnalité qui est celle associée à notre ECO, notre personnalité principale. Il s\u0026rsquo;agit de celle qui nous décrit, et même si l\u0026rsquo;on feint certains de ses aspects lorsque l\u0026rsquo;on fait face à d\u0026rsquo;autres personnes, cette feinte fait partie de ses traits: on sait lorsque l\u0026rsquo;on fait semblant d\u0026rsquo;être quelqu\u0026rsquo;un d\u0026rsquo;autre.\nCette personnalité principale est ce qui nous caractérise, ce qui nous décrit en tant que personne si l\u0026rsquo;on exclut les traits physiques. On peut être timides ou extravertis, joyeux, pessimistes, aventuriers ou anxieux. Elle dirige nos décisions et nos actions, et les personnes qui la connaissent peuvent inférer certains de nos comportements de par leur alignement avec notre personnalité. Si vous connaissez ma personnalité et reconnaissez une angoisse générale, vous inférerez probablement qu\u0026rsquo;il y a peu de chances que je prenne une décision impulsive sans la réfléchir.\nCette personnalité principale est ce qui rend notre réalité subjective: les interprétations de notre réalité sont influencées par notre personnalité. C\u0026rsquo;est parce qu\u0026rsquo;elle a certains traits que l\u0026rsquo;on interprète les évènements d\u0026rsquo;une certaine façon, que l\u0026rsquo;on fait peu attention à certains détails et beaucoup plus à d\u0026rsquo;autres.\nLes personnalités alternatives (ou secondaires) # Parmi toutes les expériences de la réalité, certaines sont intégrées dans notre personnalité principale et d\u0026rsquo;autres ne le sont pas. Des fois, elles ne sont pas importantes pour nous ou nous n\u0026rsquo;en n\u0026rsquo;avons pas conscience, ou des fois elles sont incompatibles avec notre réalité et risqueraient sa stabilité, alors elles sont niées et refoulées consciemment ou non. Dans tous les cas, nous en faisons l\u0026rsquo;expérience, accumulons et enregistrons des informations, mais nous ne les intégrons pas dans notre personnalité principale.\nDans les ECM, lorsque l\u0026rsquo;on rêve ou que l\u0026rsquo;on est sous hypnose, des personnalités alternatives inconscientes peuvent être créées et prendre en compte ces informations. Elles ne mettent pas en péril notre réalité ou notre personnalité principale, nous sommes temporairement dans une réalité alternative, un endroit sûr où le pire qui puisse arriver est que l\u0026rsquo;on se réveille en réalisant que ce que l\u0026rsquo;on vient de vivre n\u0026rsquo;était pas réel.\nNotre inconscient peut se servir de ces personnalités pour organiser, classifier ou même supporter des informations qui ne peuvent pas être intégrées à notre personnalité principale. Cependant, parce qu\u0026rsquo;elles ont accès à des informations inhabituelles, les personnalités alternatives peuvent avoir des traits et des comportements amusants, étranges ou erratiques\u0026hellip; des fois tout à fait plausibles mais souvent très différents de notre personnalité principale. Le personnage qui nous incarne dans les rêves ou dans notre corps hypnotisé n\u0026rsquo;est pas notre personnalité principale, c\u0026rsquo;est pour ça que lorsque l\u0026rsquo;on regarde comment il s\u0026rsquo;est comporté, c\u0026rsquo;est comme s\u0026rsquo;il avait été contrôlé par quelqu\u0026rsquo;un d\u0026rsquo;autre et que nous étions spectateurs.\nÀ moins de souffrir d\u0026rsquo;un trouble de la personnalité, ces personnalités alternative subconscientes ne partagent généralement pas l\u0026rsquo;existence de notre personnalité principale. Elles apparaissent dans un rêve ou sous hypnose, lorsque l\u0026rsquo;on est dans un ECM et que notre personnalité principale s\u0026rsquo;en va avec notre conscience, puis disparaissent lorsque l\u0026rsquo;on se réveille et que l\u0026rsquo;on entre de nouveau dans notre ECO avec notre personnalité principale.\nLes phases du sommeil # De tous les ECM, les phases du cycle du sommeil sont particulièrement intéressants parce que vécus par tous, tous les jours, et donc nous pouvons tous nous y reconnaître. Personne ne peut douter que l\u0026rsquo;on s\u0026rsquo;endort, ou qu\u0026rsquo;une fois endormi on ne se trouve plus dans le même état que lorsque l\u0026rsquo;on est éveillé. C\u0026rsquo;est un ensemble d\u0026rsquo;ECM que tout le monde partage, indiscutablement.\nJe ne vais pas m\u0026rsquo;étendre en profondeur sur les phases du sommeil, elles sont déjà très largement documentées, mais je vais expliquer brièvement leur relation aux ECM.\nL\u0026rsquo;endormissement # Tout d\u0026rsquo;abord, le rythme circadien de la veille-sommeil nous envoie le signal que l\u0026rsquo;on a besoin de dormir environ toutes les 24h.\nLors de l\u0026rsquo;endormissement, on perds progressivement conscience de notre environnement et on débute une transition vers un ECM. A ce stade, nous ne sommes pas complètement détachés de l\u0026rsquo;environnement, nous perdons juste connaissance.\nC\u0026rsquo;est un ECM parce que nous perdons conscience de l\u0026rsquo;environnement, et que cette conscience est précisément ce qui définit un ECO.\nSommeil lent léger # Ensuite, durant la phase de sommeil lent léger, on bascule vers un ECM différent dans lequel nous avons perdu connaissance et sommes détachés de l\u0026rsquo;environnement, mais toujours sujets à certaines perceptions qui peuvent facilement nous ramener à l\u0026rsquo;éveil, comme le bruit, les changements lumineux ou quelqu\u0026rsquo;un qui nous touche.\nC\u0026rsquo;est un ECM différent parce qu\u0026rsquo;il y a des changements visibles de la physiologie, et que l\u0026rsquo;on peut mesurer des ondes cérébrales spécifiques à cette phase.\nSommeil lent profond # Plus tard, durant la phase de sommeil lent profond, on bascule de nouveau vers un ECM différent dans lequel nous avons non seulement perdu connaissance et sommes détachés de l\u0026rsquo;environnement, mais également de la plupart des perceptions, ce qui demande des signaux insistants de l\u0026rsquo;environnement pour nous y ramener.\nC\u0026rsquo;est là encore un ECM différent avec ses propres changements physiologiques, et ses ondes cérébrales spécifiques.\nSommeil paradoxal ou sommeil REM (rapid eye-movements) # Enfin, durant la phase de sommeil paradoxal, on bascule vers un nouvel ECM dans lequel nous sommes inconscients et détachés de l\u0026rsquo;environnement, comme dans une phase de sommeil lent profond, mais où une version alternative de nous est projetée dans la réalité alternative d\u0026rsquo;un monde onirique.\nCette phase est qualifiée de paradoxale parce qu\u0026rsquo;elle possède des caractéristiques d\u0026rsquo;un état d\u0026rsquo;éveil superposées à celles d\u0026rsquo;un état de sommeil. Les ondes cérébrales montrent l\u0026rsquo;activité d\u0026rsquo;un cerveau à l\u0026rsquo;éveil, et s\u0026rsquo;il n\u0026rsquo;y avait pas de paralysie musculaire, notre corps mimerait les actions du rêve comme c\u0026rsquo;est observé chez les personnes souffrant de troubles du sommeil affectant la paralysie.\nNous sommes en apparence éveillés et endormis simultanément.\nPourquoi les phases du sommeil sont intéressantes ? # D\u0026rsquo;abord, la phase de sommeil paradoxal est quelque chose dont nous faisons tous l\u0026rsquo;expérience, et elle expose toutes les notions présentées dans cet article. Durant cette phase, nous faisons l\u0026rsquo;expérience d\u0026rsquo;un ECM qui remplace notre ECO, de réalités alternatives qui remplacent notre réalité, et de personnalités alternatives qui remplacent notre personnalité principale.\nApprocher la compréhension de notre esprit par le sommeil paradoxal est le chemin de la moindre résistance, les concepts peuvent être expliqués sans se baser sur des théories difficiles à comprendre mais en pointant du doigt ce que les gens connaissent déjà, tout le monde sait comment fonctionne un rêve même s\u0026rsquo;ils ne se rappellent pas des leurs.\nEnsuite, ils partagent ÉNORMÉMENT de similarités avec les états hypnotiques que je décris dans la section suivante, tellement en fait que l\u0026rsquo;on pourrait dire que ce sont essentiellement les mêmes états déclenchés par des chemins différents.\nC\u0026rsquo;est important parce que la porte la plus efficace vers le subconscient est l\u0026rsquo;hypnose, et lorsque l\u0026rsquo;on comprend qu\u0026rsquo;elle fonctionne au travers de mécanismes similaires au sommeil, on comprend également que ne pas être réceptif à l\u0026rsquo;hypnose est aussi dénué de sens que de ne pas être réceptif au sommeil. La résistance à l\u0026rsquo;hypnose n\u0026rsquo;est pas une question de réceptivité mais trouve sa cause ailleurs.\nLes états hypnotiques (transes) # Les états hypnotiques sont des ECM qui sont entre l\u0026rsquo;ECO, lorsque l\u0026rsquo;on est conscient, et l\u0026rsquo;état de sommeil, lorsque l\u0026rsquo;on a perdu connaissance. Ils impliquent que notre niveau de conscience est réduit par rapport à notre ECO. Il y a trois catégories d\u0026rsquo;états hypnotiques, ou du moins trois qui soient significativement différents et observables, chacun ayant ses propres caractéristiques et ses changements physiologiques.\nLa transe commune de la vie quotidienne (ou rêverie) # Nous sommes dans une transe commune, ou rêverie, lorsque l\u0026rsquo;on se déconnecte temporairement de notre réalité pour entrer dans une réalité alternative intérieure. C\u0026rsquo;est ce qu\u0026rsquo;il se produit lorsque l\u0026rsquo;on réalise que l\u0026rsquo;on a tourné plusieurs pages d\u0026rsquo;un livre sans les lire\u0026hellip; parce que l\u0026rsquo;on était absorbés dans nos pensées sans lien avec notre environnement et ce que nous étions en train de faire. C\u0026rsquo;est une transe très légère qui survient naturellement de par le rythme circadien de la vigilance.\nLorsque l\u0026rsquo;on est en rêverie, nous sommes un peu plus suggestibles parce que nous continuons à percevoir les signaux de la réalité, mais notre conscience est ailleurs et ne fait pas son filtrage habituel. Ça ne veut pas dire que toutes les suggestions sont acceptées, mais qu\u0026rsquo;il y a moins de résistance à certaines suggestions qui seraient autrement rejetées. Elles peuvent être enregistrées et affecter nos décisions et pensées ultérieures.\nSi j\u0026rsquo;entends une publicité pour acheter des barres de chocolat pendant que je suis en rêverie, je suis plus suggestible à cette idée que si je prête attention et ai conscience que l\u0026rsquo;on tente de m\u0026rsquo;en faire acheter. Ce petit surcroît de suggestibilité peut jouer un rôle dans ma décision plus tard, et peut être me faire basculer vers l\u0026rsquo;achat si je n\u0026rsquo;étais pas opposé à l\u0026rsquo;idée, même si je n\u0026rsquo;en avait pas l\u0026rsquo;intention à l\u0026rsquo;origine.\nLa transe stuporeuse # Dans une transe stuporeuse, nous sommes progressivement déconnectés de la réalité, de manière très similaire aux phases de sommeil lent léger et profond. Nous passons par les même changements physiologiques, aussi bien au niveau de la respiration que du rythme cardiaque.\nDans une transe stuporeuse légère, nous sommes relaxés et nous faisons moins attention à l\u0026rsquo;environnement, nous détachant progressivement de la réalité mais restant capable de se reconnecter rapidement si les sens sont stimulés.\nAlors que la transe s\u0026rsquo;approfondit, nous nous détachons de plus en plus et perdons l\u0026rsquo;intérêt pour la réalité pour être absorbés par la réalité alternative intérieure. Cette perte d\u0026rsquo;intérêt inclue ce qui peut arriver au corps physique, on ne se soucie plus de l\u0026rsquo;inconfort, les signaux de douleur n\u0026rsquo;arrivent pas jusqu\u0026rsquo;à la réalité alternative intérieure, c\u0026rsquo;est pourquoi les techniques de gestions de la douleur emploient souvent une transe stuporeuse poussant à imaginer un lieu sûr où l\u0026rsquo;on se sentirait bien.\nLa transe peut s\u0026rsquo;approfondir jusqu\u0026rsquo;au sommeil ou coma hypnotique, un état dans lequel nous perdons temporairement tout intérêt pour la réalité, complètement absorbés dans notre réalité alternative intérieure.\nLorsque l\u0026rsquo;on lit ou entend parler de chirurgie sous hypnose, de dentistes utilisant l\u0026rsquo;hypnose, ou encore lorsque l\u0026rsquo;on voit un hypnotiseur endormir quelqu\u0026rsquo;un dans un spectacle, il s\u0026rsquo;agit de cette transe approfondie à différents degrés.\nLa transe somnambulique # La transe somnambulique ressemble au sommeil paradoxal. On y glisse vers une réalité alternative où des pieds peuvent être collés au sol d\u0026rsquo;un claquement de doigts, où des choses peuvent apparaître spontanément depuis nulle part, ou encore où notre inconscient peut être invoqué pour prendre le contrôle de notre corps comme s\u0026rsquo;il ne s\u0026rsquo;agissait plus du notre. C\u0026rsquo;est similaire à un rêve éveillé où les choses surviennent et on les accepte sans se poser de question, notre réalité alternative intérieure remplace simplement notre réalité.\nDans une transe somnambulique légère, la personnalité principale partage l\u0026rsquo;espace mental avec une personnalité alternative. Elle accepte qu\u0026rsquo;il peut y avoir deux personnalités dans le corps, elle-même et une personnalité alternative que l\u0026rsquo;on personnifie souvent en \u0026ldquo;l\u0026rsquo;inconscient\u0026rdquo;, et qu\u0026rsquo;elles peuvent toutes deux avoir un degré de contrôle.\nAlors que la transe somnambulique s\u0026rsquo;intensifie, la personnalité alternative prends de plus en plus de contrôle, jusqu\u0026rsquo;à avoir l\u0026rsquo;ascendant sur la personnalité principale, et finalement prendre le contrôle complet du corps en provoquant la disparition temporaire de la personnalité principale.\nC\u0026rsquo;est ce qui arrive lorsque l\u0026rsquo;on voit un hypnotiseur dire à quelqu\u0026rsquo;un de faire des choses surréalistes, que la personne s\u0026rsquo;exécute, avant de revenir à elle sans se souvenir de ce qu\u0026rsquo;elle a fait. La personnalité principale était ailleurs, une personnalité alternative inconsciente à fait les choses à sa place, et c\u0026rsquo;est pourquoi elle ne s\u0026rsquo;en rappelle pas.\nLe rêve # Le rêve est une fonction essentielle qui aide à la gestion des émotions et du stress, mais aussi à la mémorisation et la catégorisation des souvenirs, comme ce fut démontré lors d\u0026rsquo;expériences de privation du sommeil paradoxal. Tout le monde rêve, que l\u0026rsquo;on s\u0026rsquo;en rappelle ou non, parce que les rêves sont le produit de ces activités que nous traversons quotidiennement.\nFreud voyait le rêve comme une psychose sans danger, a harmless dream psychosis, qui nous soustrait du monde extérieur temporairement puis disparaît. Et si je ne suis pas un grand fan de tout son travail, c\u0026rsquo;est une façon très logique de comprendre les rêves à la lumière de ce que j\u0026rsquo;ai écrit:\nDurant le rêve, une personnalité alternative prends le contrôle de notre espace mental dans une réalité alternative qui n\u0026rsquo;a pas de lien avec la réalité objective, jusqu\u0026rsquo;à ce que le rêve se termine, que personnalité et réalité alternatives disparaissent, et que notre personnalité principale se reconnecte à notre réalité.\nBeaucoup de choses ont été dites au sujet des rêves, de leur sens et interprétations, mais je ne crois pas que l\u0026rsquo;on puisse les interpréter. Ils embarquent de l\u0026rsquo;information, mais elle est encodée d\u0026rsquo;une manière qui ne fait sens qu\u0026rsquo;à la personnalité alternative qui vit le rêve. L\u0026rsquo;information ne fait aucun sens pour notre personnalité principale ou celle d\u0026rsquo;autres personnes. C\u0026rsquo;est comme si quelqu\u0026rsquo;un nous parlait dans un nouveau langage inventé spécialement à chaque rêve, on peut tenter d\u0026rsquo;interprêter, on peut tenter de deviner et peut être même des fois avoir de la chance et tomber juste\u0026hellip; mais généralement on ne comprendra rien du sens parce qu\u0026rsquo;il n\u0026rsquo;est pas dans notre langage, et que l\u0026rsquo;on ne peut pas apprendre à comprendre quelque chose qui change radicalement à chaque fois.\nSous hypnose, les personnalités alternatives utilisent souvent des discours symboliques qui ne font absolument aucun sens pour nous, jusqu\u0026rsquo;à ce qu\u0026rsquo;elles expliquent comment ça fait sens pour elles\u0026hellip; de manière similaire à comment une personne psychotique peut avoir une explication très personnelle du monde qui l\u0026rsquo;entoure, que l\u0026rsquo;on ne peut pas comprendre avec notre logique, mais qui fait sens pour elle et qu\u0026rsquo;elle est capable d\u0026rsquo;expliquer.\nCe n\u0026rsquo;est pas différent pour les personnalités alternatives dans nos rêves.\nCe qu\u0026rsquo;il est important de bien comprendre est que les rêves, dont nous faisons tous l\u0026rsquo;expérience, englobent tous les concepts: un ECM qui diffère de notre ECO, une réalité alternative disjointe de notre réalité et de la réalité objective, des personnalités alternatives qui peuvent varier énormément de notre personnalité principale, jusqu\u0026rsquo;au point que l\u0026rsquo;on puisse considérer notre expérience du rêve comme celle d\u0026rsquo;une psychose sans danger.\nCette compréhension est à la base de la compréhension du rêve lucide et de l\u0026rsquo;hypnose, mais aussi de comment sont créés les traumas, ce qui cause l\u0026rsquo;hystérie, et comment les médicaments, le rêve lucide ou l\u0026rsquo;hypnose peuvent les affecter.\nLe rêve lucide # Le rêve lucide est un rêve un peu particulier, une sorte de hack si vous préférez, où notre personnalité principale est amenée dans la réalité alternative d\u0026rsquo;un rêve et lui fait partager l\u0026rsquo;espace mental avec les personnalités alternatives.\nTout le monde peut faire des rêves lucide mais ça demande de l\u0026rsquo;entraînement et du conditionnement, d\u0026rsquo;abord pour pouvoir se souvenir des rêves, et ensuite pour pouvoir mettre en place des déclencheurs qui provoqueront le réveil de la personnalité principale dans la réalité alternative. Certaines personnes le font très simplement, voire naturellement, d\u0026rsquo;autres auront besoin de longues périodes d\u0026rsquo;entraînement avant de réussir. Il n\u0026rsquo;y a pas de recette magique pour provoquer un rêve lucide, seulement des recettes pour augmenter les probabilités d\u0026rsquo;y arriver.\nQuand un rêve lucide est déclenché, notre personnalité principale est réveillée dans une réalité alternative construite par des personnalités secondaires\u0026hellip; mais elle est consciente que ce n\u0026rsquo;est pas sa réalité, et que le corps physique dort à l\u0026rsquo;extérieur du monde du rêve. Les personnalités alternatives tentent de préserver leur ascendant, et leur réalité, en tentant de faire perdre cette conscience par l\u0026rsquo;effondrement du rêve courant pour provoquer un réveil, ou en démarrant une nouvelle trame de rêve où nous ne sommes plus conscients. Il existe cependant des techniques simples pour empêcher ces tentatives de réussir.\nLorsque le rêve lucide est stabilisé, quelque chose de très intéressant survient: un canal de communication entre la personnalité principale et les personnalités alternatives se crée, alors que notre personnalité principale contrôle notre personnage dans le rêve mais que tous les autres personnages et éléments de décor sont des constructions subconscientes. Il devient possible de poser des questions aux personnages du rêve concernant nos comportements inconscients et d\u0026rsquo;obtenir des réponses désinhibées, même si elles entrent en conflit avec nos points de vue et notre réalité.\nDans notre ECO, on ne ressent jamais vraiment la dualité entre nos esprits conscient et inconscients, notre personnalité principale ne communique pas avec les personnalités alternatives. Dans le rêve lucide, cette dualité est omniprésente, on est conscient que ce n\u0026rsquo;est pas notre réalité et malgré notre degré de contrôle, la plupart des éléments échappent à notre contrôle et entrent en conflit avec nous: des personnages peuvent refuser de nous parler, nous contredire, ou même ramener des souvenirs qui entrent en conflit avec notre réalité, \u0026hellip;\nHypnose # L\u0026rsquo;hypnose est également une forme spéciale de rêve. Elle se déroule pendant notre éveil et des personnalités alternatives sont amenées dans notre réalité, provoquant leur partage de l\u0026rsquo;espace mental avec notre personnalité principale. C\u0026rsquo;est très similaire au rêve lucide, dans le sens où finalement notre réalité est remplacée par une réalité alternative où nous percevons des choses inexistantes pour les personnes autour, et où la personnalité principale et les personnalités alternatives sont présentes ensemble et capables de communiquer les unes avec les autres. La différence principale est que l\u0026rsquo;on développe cette réalité alternative depuis l\u0026rsquo;état d\u0026rsquo;éveil, et que les personnes extérieures, telles que l\u0026rsquo;hypnotiseur, peuvent interagir avec cette réalité alternative.\nLà aussi, l\u0026rsquo;hypnose est accessible à tous mais demande entraînement et adhésion au procédé.\nContrairement à une croyance populaire, il n\u0026rsquo;est pas nécessaire de croire à l\u0026rsquo;hypnose ou même croire qu\u0026rsquo;elle fonctionne sur nous, mais il faut vraiment vouloir qu\u0026rsquo;elle fonctionne pour ne pas (in)volontairement y résister: pour être hypnotisé, il faut accepter de temporairement abandonner notre réalité et notre personnalité principale. C\u0026rsquo;est très simple pour certains qui peuvent être hypnotisés en quelques secondes, très dur pour d\u0026rsquo;autres qui nécessitent des douzaines ou des centaines d\u0026rsquo;heures pour accepter cet abandon, et contrairement à ce que l\u0026rsquo;on pourrait penser, la résistance est généralement signe de faiblesses sous-jacentes, comme des traumas ou des peurs, plutôt que le signe d\u0026rsquo;un fort esprit \u0026ldquo;cartésien\u0026rdquo;.\nLes techniques en elles-mêmes sont simples à apprendre et lorsque l\u0026rsquo;on entre en transe somnambulique, il se passe la même chose que dans les rêves lucide: il devient possible pour notre personnalité principale de communiquer avec les personnalités alternatives. Il devient possible de poser des questions et comprendre ce qu\u0026rsquo;il se passe au niveau subconscient.\nL\u0026rsquo;hypnose est beaucoup plus efficace que le rêve lucide car durant le rêve, nous entrons dans la réalité alternative de nos personnalités alternatives et, si elles arrivent à nous en faire sortir, l\u0026rsquo;aventure s\u0026rsquo;arrête jusqu\u0026rsquo;à la prochaine tentative fructueuse de rêve lucide. Avec l\u0026rsquo;hypnose, nous faisons entrer les personnalités alternatives dans la réalité de notre personnalité principale, une réalité qu\u0026rsquo;elles ne peuvent pas interrompre. Si l\u0026rsquo;on sort de l\u0026rsquo;état d\u0026rsquo;hypnose pour une raison ou une autre, il est possible de le ré-induire autant de fois que souhaité.\nLa suite ? # Cet article visait à donner une vue macroscopique de concepts qui seront récurrents dans cette catégorie, un peu pour mettre tout le monde au même niveau de compréhension, les articles futurs pousseront ces notions plus loin.\nDans mon prochain article de cette catégorie, je présenterai un modèle sur lequel j\u0026rsquo;ai travaillé et qui décrit une organisation dynamique de l\u0026rsquo;appareil psychique, prenant en compte ses réorganisations dans les différents états de conscience. J\u0026rsquo;expliquerai comment je le construit, quelle est la logique et le raisonnement derrière chacun de ses composants, et je l\u0026rsquo;appliquerai à différents états de conscience pour comparer ce que l\u0026rsquo;on observe avec ce que le modèle prédit.\n","date":"17 May 2021","permalink":"/posts/2021-05-02/","section":"Posts","summary":"TL;DR: J\u0026rsquo;explique superficiellement comment fonctionne notre esprit, en partant du corps, puis de notre sens de la réalité, avant de finir par les états de conscience et personnalités.","title":"Mind hacking: comprendre comment fonctionne notre esprit"},{"content":" TL;DR: I explain very superficially how our minds work, starting from our body, following with our sense of reality, and ending with our states of consciousness and personalities. I dive briefly into dreams and hypnosis, paving the road for a future article. Disclaimer # I\u0026rsquo;ve started taking an interest in lucid dreaming around 2013, self-studying and training in various kinds of hypnosis from 2015 to 2020, and been working as a hypnotherapist since 2016 up to now. I started and am still studying work psychology since 2016, which had me take introductory classes in clinical and cognitive psychology.\nI\u0026rsquo;m not a psychologist, a psychiatrist or a cat, so take this article with a grain of salt.\nAbout this article # I attempt to explain, from a very macroscopic perspective, my understanding of how the mind works, based on my studies, observations or personal experiences through lucid dreaming and hypnosis sessions.\nThis article doesn\u0026rsquo;t contain revolutionizing new ideas, you may in fact recognize bits of work from Charcot, Janet, Breuer or even Freud, though these bits may not exactly reflect their work as I don\u0026rsquo;t necessarily share all their views, and adapted them to how I understand the mind works.\nThis is a vulgarization that takes voluntary shortcuts to convey its main ideas in their simplest form, it is by no mean an exhaustive explanation as each single concept could easily take pages of detailing. I\u0026rsquo;ll be more than happy to expand upon each of these concepts in the comments or as follow-up articles.\nBottom line: do not assume all of this article to apply as is to everyone, particularly to people with disorders that affect their sense of reality or personality/ies, this is not an exact depiction of how things work for everyone.\nWe are organic computers # We are organic computers resulting from the combined specifications of two other organic computers. The specifications determine how we\u0026rsquo;re built, how we look like, and some of our aptitudes and limitations, but not how we process the world, not how our operating system works.\nWe interface with the world through many sensors, called receptors, that are spread in our eyes, ears, tongue, nose and below our skin to capture visual, auditory, gustatory, olfactory and kinesthesic signals that are then interpreted and allow us to see, hear, taste, smell and feel elements of our environment. These receptors all compose our sensory apparatus, what we call our senses.\nThese receptors are part of what developers would consider an API. They are the only interfaces that can perceive input signals from the environment, allowing our operating system to translate them into sensations and encode memories. Without receptors, no signal would enter our system, and we\u0026rsquo;d live a very dull life of nothingness. Luckily for us, we have receptors in our eyes to see a white chocolate bar on the table, in our hands to feel when we grab it, in our nose and tongue to smell and taste it, and in our ears to hear someone ask where did the chocolate bar disappear. Oopsy.\nBecause our senses are our only input interfaces, everything that we ever know about our environment comes from them. Our knowledge didn\u0026rsquo;t just appear in us, it was inserted through our senses, either because another organic computer sent us a signal to share information, or because we interacted with our environment and received a signal as a feedback.\nMost signals are discarded unless we pay extra attention as they are irrelevant to our current activities, like the tick of a clock that we hear but forget as if it was no longer there\u0026hellip; even though it still really is. Many signals are processed at a low-level subconscious layer and recorded for future use, like the earlier thought of a white chocolate bar that maybe we kinda want now. A few signals are processed and made available to our immediate knowledge so that we can act upon them, such as this highlighted text which immediately catches our attention.\nThese receptors are what help us develop and customize our operating system from scratch. They make information available to us about our reality, which changes the way we process our environment and interpret new signals. Our operating system basically builds an understanding of the world through a sensory feedback loop.\nReality # It may seem strange to discuss the concept of reality in an article about how the mind works but it isn\u0026rsquo;t. We live in a reality and everything we do is relative to it: we come to life in it, learn its rules through trials and errors, then develop thoughts and behaviors that seem adapted to it. We can\u0026rsquo;t begin to comprehend how our mind works if we don\u0026rsquo;t understand how we perceive reality\u0026hellip; because how our mind works is derived from that perception.\nWhen we think of reality, we usually think of a unique single reality, THE reality, then we split the world in two: on a side are those that share the same reality as us and are considered sane, and on the other side are those that don\u0026rsquo;t and are considered insane. If someone mentions seeing flying pigs over our heads and we don\u0026rsquo;t see them when we look up, we recognize that their perception of reality is inaccurate because the flying pigs aren\u0026rsquo;t there. We expect people around us to experience reality as we do because, if there\u0026rsquo;s only THE reality, then there can\u0026rsquo;t be two contradictory events happening simultaneously.\nHowever, it\u0026rsquo;s not always that simple.\nWe all get different glimpses of reality and develop a very unique sense of it based on our own perceptions, even if we still share a large part of it with other living beings. We may live in the same reality, however mine is different from that of a color blind person whose visual signals are translated differently, and even more different from that of a blind person whose visual signals are lacking. We may live in the same reality but we experience a different reality\u0026hellip; because reality has multiple dimensions.\nThe objective reality # The reality within which we exist is the objective reality, one that is factual and bound by the laws of physics, one where things exist and happen even if we\u0026rsquo;re not aware of them. This is THE reality most people think they experience, a unique reality that is the same to all of us. It encompasses all that\u0026rsquo;s ever been, all that is and all that will ever be.\nAll that exists anywhere in the universe, from every star in every galaxy to every grain of sand on our planet, is part of the objective reality and exists regardless of our ability to interact with it or knowledge that they exist. All events that happen anywhere in the universe are part of the objective reality and exist regardless of our ability to observe them. They exist and happen factually, wether we know it or not.\nIn the objective reality, if I tell you that I once ate 1kg of white chocolate, it doesn\u0026rsquo;t matter that you believe me or not, there\u0026rsquo;s a factual truth: either it happened or it didn\u0026rsquo;t. If we disagree, then one of us is wrong. There\u0026rsquo;s no room for debate in this reality.\nThe signals that our senses perceive from the environment originate from the objective reality, they are factual and don\u0026rsquo;t carry subjective meaning. In the objective reality, white chocolate has a smell that can be perceived by our receptors, but that smell is neither pleasant or unpleasant, it just exists as a signal which may be translated and interpreted by someone.\nThe subjective reality # The objective reality is infinite and complex, and we are only exposed to an infinitesimal portion of it so we never experience it as it really is. Instead, we experience a dumbed down version that only contains what we were exposed to and what our senses can perceive out of it\u0026hellip; a version that\u0026rsquo;s different for everyone.\nThis is the subjective reality, an inner representation of the objective reality that\u0026rsquo;s affected by our perceptions, and which tries to fill the gap of missing informations by interpreting, guessing or extrapolating from past individual experiences. It is subjective because that reality differs from a person to another, it is not a factual reality.\nIf I told you that I ate 1kg of white chocolate, maybe you would believe me because you once did too or maybe someone else wouldn\u0026rsquo;t because that seems a lot. In the objective reality, either I did or I didn\u0026rsquo;t, but we have two subjective realities for two different persons: one where I did and one where I didn\u0026rsquo;t. Within each of these realities exists a different version of me, one that tells the truth and one that lies, which leads to a different treatment of information when it originates from me and depending on who receives it. People live in the same objective reality but can see it in conflicting ways.\nThis does not mean that everyone has a radically different reality, we still live in THE objective reality where the laws of physics apply and where we receive the same signals. If we watch the same show, we will see the same images and hear the same dialogues, we may have subtle differences because we pay more or less attention to details, but we should have about the same experience. The divergences appear where we lack information and are forced to fill the gaps with interpretations, guesses or extrapolations, rather than actual perceptions.\nFurther in this article, whenever I use the term reality without more precisions then it means subjective reality, as it is the one we tend to refer to when we talk about reality.\nAlternate realities # While realities can vary from a person to another, we can have reasonable expectations regarding realities of others because we all share the objective reality, we\u0026rsquo;re all bound to the laws of physics.\nIf you and another person watched me throw a rock, I can expect that you both saw it fall down and not disappear mid-air, float above my head or fly to the moon. Maybe you didn\u0026rsquo;t pay attention and missed details, maybe you think you saw me throw a can instead and the other person thinks I threw a coin. You may both disagree about details but, if you observed the same event, you should have seen more or less the same thing as you should only interpret, not transform, the objective reality. If that wasn\u0026rsquo;t the case, the gambling industry would fall apart.\nAn alternate reality is one that is incompatible with the objective reality and, by extension, the realities of most people. If someone really perceived images of the rock levitating when I threw it, then his reality would be incompatible with the objective reality by defying the laws of physics, and also with the realities of others as everyone else saw something falling.\nAlternate realities originate from other dimensions in quantum phy\u0026hellip; no, wait no, that\u0026rsquo;s the synopsis of Sliders.\nAn alternate reality is if someone tells me that she\u0026rsquo;s Cleopatra, ruler of the Kingdom of Egypt, and that we\u0026rsquo;re currently sitting by the Nile in 40 BC. We may disagree or not over the fact that she\u0026rsquo;s Cleopatra, ruler of the Kingdom of Egypt, but there are multiple incompatibilities between our realities that makes them impossible to coexist in the objective reality: either we\u0026rsquo;re in 40 BC or in 2021 AC but it can\u0026rsquo;t be both, either I\u0026rsquo;m in France or in Egypt but it can\u0026rsquo;t be both. At least one of our realities is incompatible with the objective reality and is de facto an alternate reality. To both of us, our subjective realities are the ones grounded in the objective reality, it is the other that is living in an alternate reality.\nThis can be much more subtle than the caricature above. Conspiracy theorists, such as flat earthers or some of the anti-vax crowd, have a most of their realities compatible with the objective reality\u0026hellip; except when it comes to parts revolving around the conspiracies. It is only when these topics are discussed that conflicts and incompatibilities come to light, in which case they must work very hard to preserve their alternate realities from shattering.\nAlternate realities are not only experienced through pathology, they are something we all experience frequently.\nEvery day, we go to sleep and our dream character dives into alternate dream realities, ones that have no grounds in the objective reality. They may be very similar or very different, realistic or strange but, as far as our mind is concerned, they are alternate realities that replace our reality until we wake up and are conscious again.\nThis is also what happens to people under sombnambulic states of hypnosis, hallucinating objects or acting as if they were someone else. They are instructed to switch their reality with an alternate one, similar to a dream world, until they are waken up from hypnosis. As far as their mind is concerned, they are in an alternate reality during that experience.\nWhat sets apart dreamers and hypnotized people from pathology is that their alternate realities don\u0026rsquo;t persist. When we wake up from our dreams or states of hypnosis, we recognize the alternate reality as being unreal and reconnect to our actual reality, the alternate reality doesn\u0026rsquo;t remain our subjective reality.\nThe ordinary state of consciousness (OSC) # Every day, we wake up and start getting back in touch with our reality.\nAs we connect back with it, we enter our ordinary state of consciousness (OSC), a state in which we\u0026rsquo;re aware of the environment and able to interact with it in ways we understand. The OSC is the state in which we feel\u0026hellip; normal, or at least familiar. It is the state in which we most usually function and interact with the environment, the one which we\u0026rsquo;re the most used to spend time in when we\u0026rsquo;re awake.\nSome people dislike this notion of OSC as it seems to imply that we\u0026rsquo;re the exact same person every day, but this is an inaccurate way of understanding things. The OSC is not a static state, it is a dynamic one that encompasses various levels of consciousness from being tired to being extremely active, it is just a state within which our conscience of our environment takes most of the mental space. We\u0026rsquo;re awake, aware and alert.\nAn OSC implies that there\u0026rsquo;s at least one non-ordinary state of consciousness, an understatement as there are really many altered states of consciousness.\nAltered states of consciousness (ASC) # When we mention altered states of consciousness (ASC), many people assume that we are in the realm of hippy new age theories, but we all enter observable and recognizable ASC on a daily basis.\nAn ASC is simply a state in which we\u0026rsquo;re not in our OSC, nothing more: no chakras, no energetic forces, no nothing.\nSome of these states are triggered voluntarily by our own actions, such as when we get drunk by absorbing alcohol or when we get high by consuming drugs. The term \u0026ldquo;under influence\u0026rdquo; itself implies that our state of consciousness was altered, that the actions that we did \u0026ldquo;under influence\u0026rdquo; were not that of someone under an OSC.\nASC can also be triggered by events causing us to shortly exit our normality, like for example when we enter a state of shock after being involved in an accident, or when we\u0026rsquo;re euphoric after achieving something extraordinary. These events cause us to temporily switch to an ASC that\u0026rsquo;s more or less distant from our OSC.\nMore frequently than getting drunk or in a state of shock, the circadian rythm of alertness causes us to enter an ASC on average every 90 minutes, lightly losing awareness of the environment and being absorbed in our thoughts, while the circadian rythm of sleep causes us to enter several ASC, roughly every ~24 hours, as we fall asleep.\nMost of these states are measurable or observable, either through EEG measurements or through visible physiological changes. No one could argue the ASC tied to the sleep cycle are similar to the OSC, they exhibit differences in both EEG measurements and visible changes.\nThe main personality # From early age, we start experimenting with everything and observe the outcome of our actions, we get told to do things and not do others, we abide or disobey, we succeed, fail, get hurt, love, hate, experience pleasant and unpleasant feelings, and all of this affects how we face similar situations or how we approach new ones.\nThese experiences are internalized and becomes part of our identity, we build a personality whose traits are inherited from all of them, a personality that is our natural one when we\u0026rsquo;re in our OSC, it is our main personality. It is the one that describes us, and even if we fake aspects of it when confronting other people, that faking is also one of its traits: we know when we\u0026rsquo;re pretending to be someone else.\nThis main personality is what characterizes us, what describes us as a person when excluding our physical traits. We may be shy or extravert, joyful, pessimistic, adventurous or anxious. It drives our decisions and actions, and people who know our personality can infer some behaviors based on how they align with it. If you know my personality and recognize that I\u0026rsquo;m generally anxious, you will probably infer that I\u0026rsquo;m unlikely to do something impulsive without thinking it through.\nThis main personality is what makes our reality subjective: the interpretations of our reality are shaped by our personality. It is because it has certain traits that we interpret things in a certain way, that we don\u0026rsquo;t pay attention to some details and extra attention to others.\nAlternate personalities # Among all our experiences of reality, some are integrated to our main personality and some aren\u0026rsquo;t. Sometimes, they aren\u0026rsquo;t relevant to us or we weren\u0026rsquo;t conscious of them, or sometimes they\u0026rsquo;re incompatible with our reality and would risk its stability, so we deny and repress them consciously or not. We still experience them, accumulate and record informations, we just don\u0026rsquo;t integrate that to our main personality.\nIn ASC, when we dream or when we\u0026rsquo;re under hypnosis, alternate subconscious personalities can be created that take these informations into account. They don\u0026rsquo;t risk shattering our reality or main personality, we\u0026rsquo;re temporarily in an alternate reality, a safe place where the worse that could happen is that we wake up realizing it wasn\u0026rsquo;t real.\nOur subconscious mind can make use of these personalities to organize, classify or even cope with informations that couldn\u0026rsquo;t be integrated in our main personality. However, because of this access to unusual informations, the alternate personalities exhibit all kinds of traits and behaviors that make them behave funny, strange or erratic\u0026hellip; ways that are not always implausible but that are often really different from our main personality. Our dream character or our hypnotized body is not directed by our main personality, so when we look back at how it behaved, it is as if it was controlled by someone else and we were spectators.\nUnless we suffer from a personality disorder, these alternate subconscious personalities don\u0026rsquo;t usually share existence with our main personality. They would instead appear in dreams or under hypnosis, when we\u0026rsquo;re in an ASC and our main personality goes away with our consciousness, and would disappear when we wake up and enter our OSC again with our main personality.\nStages of sleep # Out of all ASC, stages of sleep are particularly interesting because we all experience them daily, and we can all relate to them. No one doubts falling asleep or that when they\u0026rsquo;re asleep, they\u0026rsquo;re no longer in the same state as when they\u0026rsquo;re awake. It is a set of ASC that we undisputedly all share in common.\nI won\u0026rsquo;t deep-dive into the stages of sleep as they are already heavily documented, however I will lightly explain their relation to ASC.\nFalling asleep # First of all, the circadian rythm of sleep-wake sends us the signal that we need to sleep roughly every ~24h.\nAs we fall asleep, we progressively lose awareness of our environment and start transitioning to an ASC. At this stage, we\u0026rsquo;re not completely detached from our environment, we\u0026rsquo;re just losing consciousness.\nIt is an ASC because we lose awareness of the environment, which is what defines the OSC.\nLight sleep # Then, during the stage of light sleep, we switch to a different ASC within which we are unconscious and detached from the environment, but still subjected to perceptions that can easily bring us back, such as noise, light changes or even someone touching us.\nWe know it is a different ASC because there are visible changes to the physiology, and we can measure specific brain waves.\nDeep sleep # Later, during the stage of deep sleep, we switch to yet another ASC within which we are not only unconscious and detached from the environment, but also detached from most perceptions, requiring insisting signals to bring us back.\nWe know it is another ASC because, just like for light sleep, there are visible changes to the physiology, and we can measure specific brain waves.\nREM (rapid eye-movements) or paradoxical sleep # Finally, during the stage of REM sleep, we switch to another ASC within which we are not only unconscious and detached from the environment, similarly to the stage of deep, but an alternate version of us is projected in the alternate reality of a dream world.\nThe stage is called paradoxical because we exhibit characteristics from a waken state while being in a sleep state. Brain measurements show the activity of a waken state, and if it weren\u0026rsquo;t for muscle paralysis, our body would mimic our dream actions as can be observed on people with sleep disorders affecting paralysis.\nWe are seemingly both awake and asleep at the same time.\nWhy are these stages of sleep interesting ? # First of all, the paradoxical sleep state is something we all experience, and it is a showcase of all the notions discussed in this article. Within it, we experience ASC that replace our OSC, alternate realities that replace our realities, and alternate personalities that replace our main personalities.\nApproaching an understanding of how the mind works through paradoxical sleep state is the way of least resistance, the concepts can be explained without relying on theories by pointing to what people experience, everyone knows how dreams work even if they don\u0026rsquo;t often recall theirs.\nThen, they share a LOT of similarities with the hypnotic states that I\u0026rsquo;ll describe in the next section, so much actually that we could say they\u0026rsquo;re essentially the same states triggered through a different path.\nThis is important because the most efficient gate to the subconscious is hypnosis, and when we understand that it works through the same mechanisms as sleep, we also understand that not being receptive to hypnosis makes as little sense as not being receptive to sleep, and resistance is caused by other reasons.\nHypnotic states (trances) # Hypnotic states are ASC that are between the OSC, when we\u0026rsquo;re conscious, and the the state of sleep, when we\u0026rsquo;re unconscious. They imply that our level of consciousness is reduced in comparison to our OSC. There are three categories of hypnotic states, or at least three that are significantly different and observable, each with their own characteristics and physiological changes.\nThe ordinary trance, or daydreaming # We\u0026rsquo;re in an ordinary trance, or daydreaming, when we temporarily disconnect from our reality to enter an inner alternate reality. This is what happens when we realize that we\u0026rsquo;ve flipped pages of a book without actually reading\u0026hellip; because we were absorbed on inner thoughts unrelated to our environment and what we were doing. This very light trance happens naturally as a result of the circadian rythm of alertness.\nWhen daydreaming, we are slightly more suggestible as we continue perceiving signals from reality, but our conscious mind is not around to do the filtering it usually does. This doesn\u0026rsquo;t mean that all suggestions are accepted, but that there\u0026rsquo;s least resistance to suggestions that would otherwise be reject right away. They may be recorded and affect later thoughts.\nIf I hear a commercial suggesting to buy snack bars while I\u0026rsquo;m daydreaming, I\u0026rsquo;m more suggestible to this idea than if I\u0026rsquo;m paying attention and careful not to be tricked into it. This slight suggestibility increase may play a role in my decision making later, getting me to buy one because I\u0026rsquo;m not opposed to the idea, even though I wasn\u0026rsquo;t initially planning to.\nThe stuporous trance # In the stuporous trance, we\u0026rsquo;re progressively disconnected from reality, very similarly to the light and deep sleep stages. We undergo similar physiological changes, including calmer respiratory and cardiac rythm changes.\nIn a light stuporous trance, we are relaxed and pay less attention to our environment, detaching ourselves from reality but still able to reconnect easily if our senses are stimulated.\nAs the trance intensifies, we detach more and more as we lose interest in reality to focus on an inner alternate reality. This loss of interest for reality includes what happens to the physical body, we don\u0026rsquo;t care if there\u0026rsquo;s discomfort, pain signals don\u0026rsquo;t make it to the inner alternate reality, which is why pain management techniques often uses a \u0026ldquo;safe place\u0026rdquo; stuporous trance.\nThe trance can be intensified until hypnotic sleep or hypnotic coma is reached, a state where we temporarily lose all interest for reality as we\u0026rsquo;re fully absorbed in our inner alternate reality.\nWhen we read about surgeries under hypnosis, or dentists using hypnosis, or even when we see a hypnotist put someone to sleep in a show, they all exploit this trance to varying degrees.\nThe somnambulic trance # The somnambulic trance resembles paradoxical sleep. We slide into an alternate reality where feet can be glued to the ground when fingers are snapped, where things can appear out of thin air, or even where our subconscious can be summoned to take over and control our body it as if it wasn\u0026rsquo;t ours. It is similar to an awaken dream where things happen and we don\u0026rsquo;t really see a reason why they wouldn\u0026rsquo;t, the inner alternate reality simply replaces our reality.\nIn a light somnambulic transe, the main personality shares the mental space with an alternate personality. It accepts that there are two personalities within the body, itself and the alternate personality which we often personify as the \u0026ldquo;subconscious mind\u0026rdquo;, and that they both have a degree of control.\nAs the somnambulic trance intensifies, the alternate personality takes more and more control, up to dominating the main personality, until it eventually takes over control of the entire body and causes the main personality to disappear.\nThis is what happens when you see a hypnotist tell people to do surreal things, which they do, before they wak up and don\u0026rsquo;t seem to remember what happened. Their main personality was away and an alternate subconcious personality did the things on their behalf, which is why they don\u0026rsquo;t remember.\nDreaming # Dreaming is an essential function as it helps emotions and stress, as well as the recording and categorization of memories, as has been shown in experiments involving paradoxical sleep deprivation. Everyone dreams, wether they recall it or not, because dreams are a byproduct of these activities we all go through daily.\nFreud viewed the dream as a harmless psychosis which withdraws you from the external world temporarily then disappears, and while I\u0026rsquo;m not a huge fan of all of his work, this is a very logical way to understand dreams in light of what I wrote above:\nDuring the dream, an alternate personality takes over your mental space in an alternate reality that has no grounds in the objective reality, until the dream ends and both the alternate personality and alternate reality disappear, as your main personality connects back to your reality.\nA lot of things have been said about dreams, their meanings and their interpretations, but I don\u0026rsquo;t think we can interpret them. They do carry information, but one that is encoded in a way that only makes sense to the current alternate personality experiencing it. The information makes no sense to our main personality or that of other people. It is as if someone talked to us in a newly invented different language every time we dream, we can try to interpret, we can even throw guesses at what they mean and sometimes be lucky and correct\u0026hellip; but we\u0026rsquo;ll generally have no clue because it\u0026rsquo;s not our language, and we can\u0026rsquo;t build an understanding of something that keeps changing in radical ways every time.\nUnder hypnosis, alternate personalities often rely on symbolic discourses that make absolutely no sense to us, until they explain how this makes sense to them\u0026hellip; similarly to how psychotic people have their own explanation to how their reality works, one that we don\u0026rsquo;t necessarily understand but which makes sense to them, and that they can explain.\nIt isn\u0026rsquo;t different with the alternate personalities of our dreams.\nWhat is important to understand is that the dreams, which we all experience, encompass all concepts: an ASC that differs from our OSC, an alternate reality that is disjoint from both our reality and the objective reality, and alternate personalities that may diverge greatly from our main personality, up to a point we could consider our experience of a dream to be the experience of a harmless psychosis.\nThis understanding lays the foundation to understanding how lucid dreaming or hypnosis work, as well as how traumas are created, how they cause hysteria, how medications affect them and how lucid dreaming and hypnosis affect them too.\nLucid dreaming # Lucid dreaming is a special kind of dream, a hack of some sort if you will, where our main personality gets pulled into the alternate reality of a dream and causes it to share the mental space with alternate personalities.\nIt is available to all but requires training and conditioning, first to be able to recall dreams, then to be able to setup triggers that will cause the main personality to emerge in the alternate reality. Some people do it very easily, others require extended periods of training before first succeeding, there is no magical recipe that makes it an instant win but only recipes to optimize the odds of it happening.\nWhen a lucid dream is triggered, our main personality is awaken in an alternate reality built by alternate personalities\u0026hellip; while being aware that it is not its own reality, and that the physical body lies sleeping outside of the dream world. The alternate personalities try to preserve their reality by tricking us into losing consciousness again, collapsing the dream to wake us up or even begin a completely new dream sequence where we\u0026rsquo;re not conscious anymore. However simple techniques exist that help undermine these attempts.\nWhen we succeed stabilizing a lucid dream, something very interesting happens: a communication channel between our main personality and alternate personalities emerges, as our main personality inhabits our dream character but all other characters are subconscious constructs. We become free to ask questions to surrounding characters about our subconscious behaviors and get unfiltered answers, even if they conflict with our views of the actual reality.\nIn our OSC, we never really feel the duality between our conscious and subconscious minds, our main personality never gets to communicate with our alternate personalities. In lucid dreams, this duality is omnipresent, we\u0026rsquo;re conscious that we\u0026rsquo;re not in our reality and while we have a degree of control over it, most elements of that reality are outside of our control and conflict with us: dream characters may refuse to talk to us, contradict what we say, or bring back memories that conflict with our own reality, \u0026hellip;\nHypnosis # Hypnosis is also a special kind of dream, one that happens while we\u0026rsquo;re awake and where alternate personalities are pulled into our reality, causing them to share mental space with our main personality. This is very similar to lucid dreaming, in that ultimately this turns our reality into an alternate one, where we experience and perceive things that are inexistant for other people, and where main and alternate personalities are both around and able to communicate one with the other. The difference is that we develop that alternate reality from a wake state, and that external people such as the hypnotist can interact with that alternate reality.\nIt is also available to all but requires training and adhesion to the process.\nContrarily to a popular belief, we don\u0026rsquo;t need to believe in it or believe that it works, but we need to truly want it to work so that we don\u0026rsquo;t (un)voluntarily resist it: to be hypnotized, we must accept to temporarily let go of our reality and main personality. This is very easy to some who get hypnotized in seconds, very hard to others who require dozens or hundreds of hours of work to accept letting go, and contrarily to what we could think, resistance is usually the sign of underlying weaknesses, such as traumas and fears, rather than the sign of a strong \u0026ldquo;cartesian\u0026rdquo; mind.\nThe techniques by themselves are easy to learn and when we enter the somnambulic state of hypnosis, the same happens as in lucid dreams: it becomes possible for our main personality to communicate with alternate personalities. It becomes possible to ask questions that helps understand what happens subconsciously.\nHypnosis is much more efficient than lucid dreaming because during a dream, we enter the alternate reality of alternate personalities and, if they manage to kick us out, the adventure ends until our next successful lucid dream attempt in a later dream or a later night. With hypnosis, we make alternate personalities join the reality of our main personality, one that they can\u0026rsquo;t end. If we somehow exit the state of hypnosis, we can reinduce it as many times as we want.\nWhat\u0026rsquo;s next ? # This article was meant to give a macroscopic view of concepts that will be recurrent in this category, as a way to put all readers on the same step, but future articles will take these notions further.\nIn my next article of this category, I will present a model I worked on to describe the dynamic organization of the mind, taking into account how it reorganizes itself in different states of consciousness. I will explain how it was built, what was the rationale behind each component of the model, and I will apply it to various states of consciousness to compare what we observe to what the model predicts.\n","date":"12 May 2021","permalink":"/posts/2021-05-01/","section":"Posts","summary":"TL;DR: I explain very superficially how our minds work, starting from our body, following with our sense of reality, and ending with our states of consciousness and personalities.","title":"Mind hacking: understanding how our mind works"},{"content":" TL;DR: I worked on OpenSMTPD-portable, did a lot of plakar, a lot of Go and gave a technical talk on hypnosis. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !)\nThis one caused me a copyright strike so I hope it was worth it :-)\nOpenSMTPD-portable # I did a bit of review and test for diffs sent to me, helped test a diff for a reproducible crash in the new libtls code on my machines, and shared some of my nooSMTPD diffs with eric@ so he could decide to reuse them or not in OpenSMTPD.\nWith OpenBSD 6.9 coming out, OpenSMTPD 6.9.0 was tagged and eric@ asked me if I could synchronize the OpenSMTPD portable repository so it matches OpenBSD. I spent a few hours bringing back every individual commit, fixing conflicts and ended up with a libtls-powered OpenSMTPD-portable which \u0026hellip; didn\u0026rsquo;t build anywhere because the world still uses OpenSSL.\nSince I had already dealt with this in nooSMTPD, I brought back my libtls compat layer so that on systems with OpenSSL the compat layer allows using the libtls interface on top of OpenSSL. This fixed the CI for Ubuntu and Fedora, unfortunately the CI for Alpine and ArchLinux uses LibreSSL and remain broken until the latest LibreSSL is packaged there.\nBecause the libtls conversion is such a major change, there will be a delay between the time OpenSMTPD is released for OpenBSD users and the time it is released for other systems: the change to the libtls interface has been heavily tested and I\u0026rsquo;ve been running it for a while now, but the portable adaptation of it has been virtually untested. I\u0026rsquo;ll send a mail this week-end to call for testing before we can tag a release.\nPlakar # I wrote about plakar last month and made a lot of progress since then.\nI\u0026rsquo;m not publishing the code yet as there are many things I want to finish first before the first people start commenting and bikeshedding.\nrestructured the project # I\u0026rsquo;m not too familiar with Go so I didn\u0026rsquo;t structure the project very nicely at first. I spent a few hours creating specific modules and reworking things so that they are properly split.\nI\u0026rsquo;m not done but it is starting to look less shameful :-)\nremoved namespaces # I initially thought namespaces within a plakar was a nice idea, but the more I played with them the more I realized it wasn\u0026rsquo;t and didn\u0026rsquo;t bring any benefit over creating multiple plakars. I decided to remove namespaces to simplify things.\nplakar-level encryption # I brought support for encryption and tried two different approaches: snapshot-level encryption and plakar-level encryption.\nWith the first approach, a plakar repository doesn\u0026rsquo;t care about encryption. It is the snapshots themselves that are encrypted on an individual basis and the same plakar can host both encrypted and cleartext snapshots.\nWith the second approach, a plakar repository is initialized as encrypted or cleartext. The snapshots are automatically encrypted if needed and the same plakar ony hosts all encrypted or all cleartext snapshots.\nI played with both but decided to go with the second approach because the first one came with additional unnecessary complexity.\nencryption macro-details # A user generates a P384 keypair as well as a random master key, the bundle is protected by a pbkdf2-derived passphrase.\nWhen pushing to an encrypted plakar, the chunks and objects are aes256-gcm encrypted using a subkey encrypted itself by the master key.\nThe snapshot index containing all the checksums for all objects and chunks is encrypted and signed.\nUpon restore, the index signature is verified and the chunks are decrypted.\nkeypair and master key generation # The P384 keypair and master key bundle is generated using the plakar keygen command:\n% plakar keygen passphrase: passphrase (confirm): keypair saved to local store % This results in the passphrase-encrypted bundle being saved to ~/.plakar/plakar.key.\nplakar initialization # To be able to create snapshots, an initialized plakar repository must be available. A local cleartext plakar is initialized by default in ~/.plakar so that the command will work out of the box for local snapshots.\nHowever it can also be initialized elsewhere, defaulting to an encrypted plakar:\n% plakar init /tmp/plakar passphrase: /tmp/plakar: store initialized % or can be made cleartext by passing the -cleartext option:\n% plakar init -cleartext /tmp/plakar.ct /tmp/plakar.ct: store initialized % Reworked the command line interface # Last month, to use an alternative plakar instead of ~/.plakar, it was necessary to use the -store option which I disliked\u0026hellip; so I reworked the command line to introduce a notion of direction.\nWhen using the default plakar, no change is required:\n% plakar push /private/etc % plakar ls 2021-04-30T22:35:35Z 702b5b48-15dc-41cf-bfc1-1c8b94d1e985 3.1 MB (files: 242, dirs: 41) % plakar pull 702b5b48 % But when using an alternate plakar, instead of providing the -store option it is now possible to push to a plakar:\n% plakar push /private/etc to /tmp/plakar.ct % .. and run commands from a plakar:\n% plakar ls from /tmp/plakar.ct 2021-04-30T22:30:51Z d499fd92-01cb-4eb3-bb60-b147417b68a1 3.1 MB (files: 242, dirs: 41) % plakar pull d499fd92 from /tmp/plakar.ct % All commands support the direction option.\nGenerate a tarball # I thought it would be nice if I could restore a snapshot or part of it into a tarball, as I often want to extract a bit of a snapshot on a machine to send elsewhere.\nI introduced a tarball command which allows generating a tarball:\n% plakar tarball d499fd92 \u0026gt; /tmp/d499df92.tar.gz It also supports partial restore:\n% plakar tarball d499fd92:/etc \u0026gt; /tmp/d499df92_etc.tar.gz plakar server and client # I implemented a proof of concept for a plakar server and client protocol, allowing the use of plakar over the network.\nOn one end, I initialize a plakar repository and run a server from it:\nnas% plakar init /tmp/plakar passphrase: /tmp/plakar: store initialized nas% plakar server 192.168.0.2:2222 from /tmp/plakar passphrase: On the other end, I simply push to a remote plakar:\nlaptop% plakar push /etc to plakar://192.168.0.2:2222 There is absolutely no difference or limitation from the client point of view, any command that works locally will work remotely just as well.\nIt is even possible to chain servers in order to proxify a plakar server:\nnas% plakar server 192.168.0.2:2222 from /tmp/plakar passphrase: gate% plakar server 192.168.0.1:2222 from plakar://192.168.0.2:2222 laptop% plakar push /etc to plakar://192.168.0.1:2222 Does it serve a purpose ? nope, it just works by accident.\nplakar ui # Finally, I implemented a basic web UI so that I could browse the snapshots easily.\n% plakar ui Launched UI on port 40717 This opens a web browser which lets me browse the snapshots as a filesystem:\nallows me to inspect individual files:\nget a preview:\nincluding of images, pdf, videos or sound:\nand even search for files matching a pattern in every snapshot:\nBecause the web UI is a plakar client that doesn\u0026rsquo;t do anything but plakar commands, it can be launched from any plakar store, cleartext or encrypted, local or remote.\nSome work in Go # I first wrote Go code with filter-rspamd and filter-senderscore two years ago, but never really dived into the language more than these two small projects because I still have some love-hate issues with some aspects of it.\nI decided this month to get more familiar with it and started looking at what it would take to write a tiny daemon, not just a program that runs an endless loop and does all work in the same process, but one that does things the OpenBSD style with privileges separation, message passing and fd passing.\nI was not disappointed: there\u0026rsquo;s not much out there to do that.\ngo-ipcmsg # The first thing that is missing is a package that provides something like the imsg(3) framework.\nFor those not familiar with it, the imsg(3) framework provides functions that allow two processes to exchange messages, including file descriptors, while guaranteeing that messages are always received whole.\nTypically, you will create a socketpair(2) before fork(2)-ing a process. The parent and the child will both use one end of the pair to communicate with each other, using the imsg(3) framework that will take care of buffering I/O and exposing full messages to receiving end.\nThere are a lot of gory details on how it achieves this and requires understanding iovec, the semantic of sendmsg(2) and recvmsg(2), how control messages and SCM_RIGHTS works, as well as some of the side effets of cmsgbuf alignments. I did a bit of work related to resources exhaustion there a few years ago and, while the interface is lovely, I can\u0026rsquo;t say I really missed diving in that code.\nWhen I figured that there was nothing similar in Go, I started reproducing the imsg(3) API but realized that while the API was fine in C, it could be made much simpler in Go using the language-provided channels and goroutines.\nSo I wrote a Channel struct which creates two channels, one for reads and one for writes, and associates them to a socketpair(2) end:\n// parent process main routine, forks a child then sets up an ipcmsg // Channel on the socketpair, returning read and write channels. The // channels can be used to emit messages to the child process. // func parent() { pid, fd := fork_child() child_r, child_w := ipcmsg.Channel(pid, fd) // send a message to child child_w \u0026lt;- ipcmsg.Message(42, []byte(\u0026#34;foobar\u0026#34;)) // read a message from child msg := \u0026lt;- child_r log.Printf(\u0026#34;Received message: %s\\n\u0026#34;, string(msg.Data)) } // child process main routine, sets up an ipcmsg Channel on fd 3, // the socketpair end inherited from parent, // read and write channels can be used to communicate with parent // process. // func child() { parent_r, parent_w := ipcmsg.Channel(os.Getppid(), 3) // receive a message from parent msg := \u0026lt;- parent_r log.Printf(\u0026#34;Received message: %s\\n\u0026#34;, string(msg.Data)) // write a message back parent_w \u0026lt;- ipcmsg.Message(42, []byte(\u0026#34;barbaz\u0026#34;)) } In practice, you wouldn\u0026rsquo;t just inline the calls but the parent and the child would call a dispatch function to handle messages and act upon them, something similar to below:\nconst ( IPCMSG_PING = 1 IPCMSG_PONG = 2 ) func dispatcher(r chan ipcmsg.IPCMessage, w chan ipcmsg.IPCMessage) { for msg := range r { switch msg.Hdr.Type { case IPCMSG_PING: log.Printf(\u0026#34;data: %s\\n\u0026#34;, string(msg.Data)) w \u0026lt;- ipcmsg.Message(IPCMSG_PONG, []byte(\u0026#34;barbaz\u0026#34;)) } } } func parent() { pid, fd := fork_child() child_r, child_w := ipcmsg.Channel(pid, fd) dispatcher(child_r, child_w) } The file descriptors passing is handled with the function MessageWithFd() which takes an additional parameter:\nfd, err := syscall.Open(os.Args[0], 0700, 0) if err != nil { log.Fatal(err) } w \u0026lt;- ipcmsg.MessageWithFd(IPCMSG_PING, []byte(\u0026#34;barbaz\u0026#34;), fd) The sending end will have the descriptor closed upon sending, the receiving end will be receive a message with a HasFd option set in its header and an open descriptor:\nmsg := \u0026lt;- r if msg.Hdr.HasFd == 1 { if msg.Fd == -1 { // expected a descriptor but got none, handle this } else { syscall.Close(msg.Fd) } } The package is already commited on Github, however it hasn\u0026rsquo;t been heavily tested and I would not recommend using it for anything serious before it has matured a bit. I would LOVE to receive testing feedbacks though !\ngo-privsep # The second thing missing is the ability to easily do privileges separation, that is creating multiple processes that are inherited from a parent process with different privileges, and have the ability to communicate one with another.\nOn OpenBSD, daemons follow the fork+reexec pattern which boils down to the following:\nThe daemon is started, it forks all of its child processes after setting up the plumbing for communicating with them, and each child process re-executes itself so that it benefits from ASLR and doesn\u0026rsquo;t retain the memory layout of the parent process.\nThis is an improvement over the more widely used fork pattern, where each child inherits from parent, because the reexec causes all inherited data to be lost. It prevents inheriting sensitive information by accident at the cost of forcing the parent to send back the necessary information to each process. And again, it makes all child processes benefit from ASLR too which is nice.\nThis pattern was popularized in OpenBSD a few years ago, long after many of the daemons were already in place, so each one did what it could to make it happen. There was no attempt at unifying this through a framework like was done for IPC and imsg(3).\nSince I had a good understanding of how to do that and no framework to inspire myself, I worked on a privsep package that would make it easier to write such daemons. The idea is that a daemon will describe the different processes that compose it, as well as the communication channels between them, and the privsep package will then handle all the plumbing so that processes are created according to expectations.\nHere is a simple example of how it works:\npackage main import ( \u0026#34;log\u0026#34; \u0026#34;github.com/poolpOrg/ipcmsg\u0026#34; \u0026#34;github.com/poolpOrg/privsep\u0026#34; ) const ( IPCMSG_PING = 1 IPCMSG_PONG = 2 ) func parent_main() { \u0026lt;-make(chan bool) // sleep forever } func foobar_main() { parent := privsep.GetParent() parent.Write(ipcmsg.Message(IPCMSG_PING, []byte(\u0026#34;abcdef\u0026#34;))) \u0026lt;-make(chan bool) } func parent_dispatcher(r chan ipcmsg.IPCMessage, w chan ipcmsg.IPCMessage) { for msg := range r { if msg.Hdr.Type == IPCMSG_PING { log.Printf(\u0026#34;[parent] received PING, sending PONG\\n\u0026#34;) w \u0026lt;- ipcmsg.Message(IPCMSG_PONG, []byte(\u0026#34;abcdef\u0026#34;)) } } } func foobar_dispatcher(r chan ipcmsg.IPCMessage, w chan ipcmsg.IPCMessage) { for msg := range r { if msg.Hdr.Type == IPCMSG_PONG { log.Printf(\u0026#34;[foobar] received PONG, sending PING\\n\u0026#34;) w \u0026lt;- ipcmsg.Message(IPCMSG_PING, []byte(\u0026#34;abcdef\u0026#34;)) } } } func main() { privsep.Init() parent := privsep.Parent(parent_main) foobar := privsep.Child(\u0026#34;foobar\u0026#34;, foobar_main) parent.Channel(foobar, parent_dispatcher) foobar.Channel(parent, foobar_dispatcher) privsep.Start() } As you can see, it uses the ipcmsg package to handle IPC between the two processes, allowing my pingpong daemon to play ping pong indefinitely. Each process can have more specific settings set so that it runs under a specific user, a specific group or from a chroot(2) jail.\nI still have a lot of work to do on this, but I commited the current state on Github anyways.\nsmall talk on hypnosis # This is not tech related but since my other main activity is hypnosis and hypnotherapy, I spent a bit of time this month working on a 3-hours talk I gave to a community of street hypnotists and hypnotherapists in Nantes.\nI\u0026rsquo;ve been working these last three months on a model to describe the organization of the psychic apparatus, based on what I learnt, experienced and observed these last six years from hundreds of hypnosis sessions, tons of psychology readings and many experiments in lucid dreaming.\nThe model describes how the psychic apparatus organization changes with the altered state of consciousness of a subject and what it means in terms of access to the subconscious and repressed psychic content.\nThis is not limited to hypnosis as it can be used to evaluate if a particular kind of therapy even makes sense, but applied to hypnosis it is particularly useful to understand resistance, what is happening in a subject when a technique is used, what is happening when a technique isn\u0026rsquo;t working, and how to bring the subject to the proper state of consciousness.\nI don\u0026rsquo;t think this post is the right place to dive into details and, well, the talk lasted 3 hours and it was very superficial so I will stop there. If people are interested in this topic, let me know and I\u0026rsquo;ll think of a way to present this as the slides I presented are not really informative without me talking over them.\nWhat\u0026rsquo;s next ? # Not much.\nMay will be a calm month for me here as the sponsoring has decreased and I need to do some freelance to buy myself spare time in June/July. Furthermore, I have had to deal with some personal issues in April and I\u0026rsquo;m not in the mood to do a lot of stuff at the moment.\nI\u0026rsquo;ll probably do a few things, as usual, but I\u0026rsquo;m not sure what at this point as it depends on how much time I\u0026rsquo;ll have available and how my mood evolves.\n","date":"30 April 2021","permalink":"/posts/2021-04-30/april-2021-opensmtpd-plakar-ipcmsg-privsep-and-a-small-hypnosis-talk/","section":"Posts","summary":"TL;DR: I worked on OpenSMTPD-portable, did a lot of plakar, a lot of Go and gave a technical talk on hypnosis. Let\u0026rsquo;s start with some LoFi # Relax.","title":"April 2021: OpenSMTPD, plakar, ipcmsg, privsep and a small hypnosis talk"},{"content":" TL;DR: I wrote a backup utility called plakar. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !)\nPlakar: Yet another backup utility ? # Yes, I wrote another backup utility.\nI backup many hosts on a daily basis and while there are many different tools, I always fallback to using scripts built on top of rsync to do incremental backups, which always results in customization for this or that host.\nI want backups that are easy to perform and restore, that do not require me to remember a ton of options, that lets me work with snapshots independant one of another instead of increments, that allow me to browse the content easily and compare snapshots without restoring them, that store data efficiently (compressed and deduped), and finally that lets me back things up locally or remotely to machines I own.\nI\u0026rsquo;ve looked into various solutions and I figured that the best option for me was to write a tool that did exactly what I wanted. Also, it\u0026rsquo;s fun to write code.\nWhat is plakar ? # The plakar utility is a tool that performs backups as snapshots of a tree structure, scanning subdirectories and files, and storing them efficiently in a repository while deduplicating redundant data. Once a snapshot has been taken of a directory, it can be inspected and restored at will.\nHow does it work ? # To perform a backup, you only need to call plakar with the push subcommand and a set of directories that should be part of the snapshot. This will cause plakar to scan the directories, collect inode information regarding every subdirectory and files, split files into chunks and push them to the repository.\n$ plakar push /bin 82711aee-6485-445b-98f4-6cae51c83035: OK $ The ls subcommand allows listing snapshots present in the repository:\n$ plakar ls 82711aee-6485-445b-98f4-6cae51c83035 [2021-03-27T22:05:00Z] (size: 10 MB, files: 36, dirs: 1) $ The pull subcommand allows restoring a particular snapshot:\n$ plakar pull 82711aee-6485-445b-98f4-6cae51c83035 82711aee-6485-445b-98f4-6cae51c83035: OK $ ls -l bin total 9336 -rwxr-xr-x 1 root wheel 121104 1 Jan 2020 [ -r-xr-xr-x 1 root wheel 1296640 1 Jan 2020 bash -rwxr-xr-x 1 root wheel 121968 1 Jan 2020 cat -rwxr-xr-x 1 root wheel 107552 1 Jan 2020 chmod -rwxr-xr-x 1 root wheel 123232 1 Jan 2020 cp -rwxr-xr-x 1 root wheel 1104640 1 Jan 2020 csh -rwxr-xr-x 1 root wheel 277408 1 Jan 2020 dash -rwxr-xr-x 1 root wheel 139264 1 Jan 2020 date -rwxr-xr-x 1 root wheel 122160 1 Jan 2020 dd -rwxr-xr-x 1 root wheel 121840 1 Jan 2020 df -rwxr-xr-x 1 root wheel 120848 1 Jan 2020 echo -rwxr-xr-x 1 root wheel 205648 1 Jan 2020 ed -rwxr-xr-x 1 root wheel 121504 1 Jan 2020 expr -rwxr-xr-x 1 root wheel 120864 1 Jan 2020 hostname -rwxr-xr-x 1 root wheel 121232 1 Jan 2020 kill -r-xr-xr-x 1 root wheel 2552352 1 Jan 2020 ksh -rwxr-xr-x 1 root wheel 329344 1 Jan 2020 launchctl -rwxr-xr-x 1 root wheel 105136 1 Jan 2020 link -rwxr-xr-x 1 root wheel 105136 1 Jan 2020 ln -rwxr-xr-x 1 root wheel 157360 1 Jan 2020 ls -rwxr-xr-x 1 root wheel 104752 1 Jan 2020 mkdir -rwxr-xr-x 1 root wheel 106176 1 Jan 2020 mv -rwxr-xr-x 1 root wheel 291152 1 Jan 2020 pax -rwsr-xr-x 1 root wheel 173568 1 Jan 2020 ps -rwxr-xr-x 1 root wheel 120832 1 Jan 2020 pwd -rwxr-xr-x 1 root wheel 106000 1 Jan 2020 rm -rwxr-xr-x 1 root wheel 104368 1 Jan 2020 rmdir -rwxr-xr-x 1 root wheel 120912 1 Jan 2020 sh -rwxr-xr-x 1 root wheel 120784 1 Jan 2020 sleep -rwxr-xr-x 1 root wheel 138656 1 Jan 2020 stty -rwxr-xr-x 1 root wheel 120464 1 Jan 2020 sync -rwxr-xr-x 1 root wheel 1104640 1 Jan 2020 tcsh -rwxr-xr-x 1 root wheel 121104 1 Jan 2020 test -rwxr-xr-x 1 root wheel 106000 1 Jan 2020 unlink -rwxr-xr-x 1 root wheel 120752 1 Jan 2020 wait4path -rwxr-xr-x 1 root wheel 1331248 1 Jan 2020 zsh $ Compression and deduplication # When plakar performs a snapshot, it splits every file into content-defined chunks and checks if they are available in the repository before pushing them, in a way similar to what rsync does.\nEach chunk is compressed before being written to the repository, so that between compression and deduplication, a saved snapshot may take less space than the original directory:\n$ du -sh /bin 4.6M /bin $ du -sh ~/.plakar 3.8M /Users/gilles/.plakar Furthermore, since this deduplication takes place globally in the repository, pushing multiple times the same content doesn\u0026rsquo;t cause the plakar to grow much as it only grows by the size of the snapshot index:\n$ plakar push /bin f2645591-4b06-4512-89e2-dae8ad1e2360: OK $ plakar push /bin c0aa514b-452f-4122-a53f-99e984cb0548: OK $ plakar push /bin 3630358c-c634-4ad9-960d-f864d1eb56f5: OK $ plakar push /bin e521e447-5c2a-429f-b637-41561daa3826: OK $ plakar push /bin a83fe7bb-341f-4cf1-bad6-c2b1933422f8: OK $ plakar ls 82711aee-6485-445b-98f4-6cae51c83035 [2021-03-27T22:05:00Z] (size: 10 MB, files: 36, dirs: 1) f2645591-4b06-4512-89e2-dae8ad1e2360 [2021-03-27T22:06:17Z] (size: 10 MB, files: 36, dirs: 1) c0aa514b-452f-4122-a53f-99e984cb0548 [2021-03-27T22:06:17Z] (size: 10 MB, files: 36, dirs: 1) 3630358c-c634-4ad9-960d-f864d1eb56f5 [2021-03-27T22:06:18Z] (size: 10 MB, files: 36, dirs: 1) e521e447-5c2a-429f-b637-41561daa3826 [2021-03-27T22:06:18Z] (size: 10 MB, files: 36, dirs: 1) a83fe7bb-341f-4cf1-bad6-c2b1933422f8 [2021-03-27T22:06:19Z] (size: 10 MB, files: 36, dirs: 1) $ du -sh ~/.plakar 3.8M /Users/gilles/.plakar Snapshots UUID prefix-based lookup # Because using an UUID as parameter to subcommands is painful, plakar performs prefix-based lookup and allows using the beginning of an UUID as long as it is not ambiguous:\n$ plakar pull 3 3630358c-c634-4ad9-960d-f864d1eb56f5: OK In case of ambiguity, it warns so that a less ambiguous prefix can be provided:\n$ plakar ls|grep ^4 45de4fd3-d177-4378-84cb-077b2ef297fb [2021-03-27T23:09:11Z] (size: 10 MB, files: 36, dirs: 1) 4c467893-f959-43cb-858f-c50b8c46a86d [2021-03-27T23:17:05Z] (size: 10 MB, files: 36, dirs: 1) $ plakar pull 4 2021/03/28 00:17:15 plakar: snapshot ID is ambigous: 4 (matches 2 snapshots) $ plakar pull 45 45de4fd3-d177-4378-84cb-077b2ef297fb: OK Snapshots health check # Backups are not useful if they can\u0026rsquo;t be restored, but it\u0026rsquo;s also painful to create a backup and try restoring just to know if it would work. To cope with this, plakar supports health checking snapshots without actually restoring them:\n$ plakar check 3 3630358c-c634-4ad9-960d-f864d1eb56f5: OK What it does is that it fetches the snapshot index, checks that everything referenced in the snapshot exists in the repository, and does a few sanity checks to verify if it could restore the entirety of the snapshot if requested.\nSnapshots restoration # As shown in the example above, restoring a snapshot is as simple as typing:\n$ plakar pull 3 3630358c-c634-4ad9-960d-f864d1eb56f5: OK But this performs a full snapshot restore, whereas sometimes what you really want is a partial restore. For this, plakar supports providing a path within a snapshot pointing either to a subdirectory or to a file:\n$ plakar pull 3:/bin/ls 3630358c-c634-4ad9-960d-f864d1eb56f5:/bin/ls: OK $ ls -l bin total 312 -rwxr-xr-x 1 gilles staff 157360 27 Mar 23:18 ls While doing the restore, plakar will check every chunk and their checksums to detect any corruption:\n$ rm -rf ~/.plakar/default/chunks/00/008d4087e3e99c8f8c2c1ad76e9c6bc3d614e580e30ed45da06f93f912995538 $ plakar pull 3 /bin/ksh: missing chunk 008d4087e3e99c8f8c2c1ad76e9c6bc3d614e580e30ed45da06f93f912995538 /bin/ksh: corrupt file: checksum mismatch 3630358c-c634-4ad9-960d-f864d1eb56f5: KO $ ls -l bin total 19504 -rwxr-xr-x 1 gilles staff 121104 27 Mar 23:22 [ -r-xr-xr-x 1 gilles staff 743708 27 Mar 23:22 bash -rwxr-xr-x 1 gilles staff 121968 27 Mar 23:22 cat -rwxr-xr-x 1 gilles staff 107552 27 Mar 23:22 chmod -rwxr-xr-x 1 gilles staff 123232 27 Mar 23:22 cp -rwxr-xr-x 1 gilles staff 1104640 27 Mar 23:22 csh -rwxr-xr-x 1 gilles staff 277408 27 Mar 23:22 dash -rwxr-xr-x 1 gilles staff 139264 27 Mar 23:22 date -rwxr-xr-x 1 gilles staff 122160 27 Mar 23:22 dd -rwxr-xr-x 1 gilles staff 121840 27 Mar 23:22 df -rwxr-xr-x 1 gilles staff 120848 27 Mar 23:22 echo -rwxr-xr-x 1 gilles staff 205648 27 Mar 23:22 ed -rwxr-xr-x 1 gilles staff 121504 27 Mar 23:22 expr -rwxr-xr-x 1 gilles staff 120864 27 Mar 23:22 hostname -rwxr-xr-x 1 gilles staff 121232 27 Mar 23:22 kill -r-xr-xr-x 1 gilles staff 1258618 27 Mar 23:22 ksh -rwxr-xr-x 1 gilles staff 329344 27 Mar 23:22 launchctl -rwxr-xr-x 1 gilles staff 105136 27 Mar 23:22 link -rwxr-xr-x 1 gilles staff 105136 27 Mar 23:22 ln -rwxr-xr-x 1 gilles staff 157360 27 Mar 23:22 ls -rwxr-xr-x 1 gilles staff 104752 27 Mar 23:22 mkdir -rwxr-xr-x 1 gilles staff 106176 27 Mar 23:22 mv -rwxr-xr-x 1 gilles staff 291152 27 Mar 23:22 pax -rwsr-xr-x 1 gilles staff 173568 27 Mar 23:22 ps -rwxr-xr-x 1 gilles staff 120832 27 Mar 23:22 pwd -rwxr-xr-x 1 gilles staff 106000 27 Mar 23:22 rm -rwxr-xr-x 1 gilles staff 104368 27 Mar 23:22 rmdir -rwxr-xr-x 1 gilles staff 120912 27 Mar 23:22 sh -rwxr-xr-x 1 gilles staff 120784 27 Mar 23:22 sleep -rwxr-xr-x 1 gilles staff 138656 27 Mar 23:22 stty -rwxr-xr-x 1 gilles staff 120464 27 Mar 23:22 sync -rwxr-xr-x 1 gilles staff 1104640 27 Mar 23:22 tcsh -rwxr-xr-x 1 gilles staff 121104 27 Mar 23:22 test -rwxr-xr-x 1 gilles staff 106000 27 Mar 23:22 unlink -rwxr-xr-x 1 gilles staff 120752 27 Mar 23:22 wait4path -rwxr-xr-x 1 gilles staff 1331248 27 Mar 23:22 zsh The snapshot is still restored as a \u0026ldquo;best effort\u0026rdquo;, but here a warning informs that ksh is not the same as the one that was initially backed up. This is not supposed to happen in practice as long as I don\u0026rsquo;t fiddle with the repository, but it is a nice sanity check.\nOne that would be caught by plakar check:\n$ plakar check 3 3630358c-c634-4ad9-960d-f864d1eb56f5: KO Other plakar subcommands # In addition to pull, push, ls and check, plakar supports various other subcommands.\nThey are more interesting when playing with text files, so let\u0026rsquo;s push /private/etc:\n$ plakar push /private/etc open /private/etc/krb5.keytab: permission denied open /private/etc/aliases.db: permission denied open /private/etc/racoon/psk.txt: permission denied open /private/etc/security/audit_user: permission denied open /private/etc/security/audit_control: permission denied open /private/etc/sudoers: permission denied open /private/etc/sudo_lecture: permission denied open /private/etc/master.passwd: permission denied open /private/etc/openldap/slapd.conf.default: permission denied open /private/etc/openldap/DB_CONFIG.example: permission denied 98b6658b-a975-47c4-a38f-f958d8d7359f: OK Here plakar has warned about files that couldn\u0026rsquo;t make it into the snapshot, but lets inspect what files are part of the snapshot with ls:\n$ plakar ls 9:/ acc57694b78a1ea669535547ac310e7682f141b75f6ba23658078b9000dbd9ac -rw-r--r-- root wheel 20 kB /private/etc/php-fpm.d/www.conf.default a4dda57401575878b78c5b0bc4a6bc020675d7f285dc8a836d07f1fda0938715 -rw-r--r-- root wheel 12 kB /private/etc/postfix/LICENSE 863f779b43680f81799688e91c18a164047a1a8e9dfff650881e50ba35530ed1 -r--r--r-- root wheel 141 B /private/etc/uucp/port 75a3d05049424cc17623d04c5d84038ca703ae673ad4c1bcf62d8293462d90b0 -rw-r--r-- root wheel 152 B /private/etc/pam.d/login.term 591161270b2fbe74d7f1b96625c4b60551f33eb6fa28992c8ccc71dff9b7c70b -rw-r--r-- root wheel 6.2 kB /private/etc/postfix/master.cf.proto 056514aa4105c96e0aad6a144d76f63461db7157123d43fa7c4121297a6c9200 -rw-r--r-- root wheel 190 B /private/etc/asl/com.apple.authd 05f4215652c68201fd6aee81fc4eeaee4d5c395f3f4a3a155b986cb06ca0f768 -rw-r--r-- root wheel 216 B /private/etc/asl/com.apple.cdscheduler 3c4aaff1783756368c28022658ebb238d6f9f32beeb0d04d79962654e0a0ae26 -rw-r--r-- root wheel 3.3 kB /private/etc/ssh/sshd_config 23198c58755cd991531259cd290f62ea99cd9596595001ab303a15306fd03ec1 -rw-r--r-- root wheel 127 B /private/etc/pam.d/authorization_ctk 48edf5460c6ad64cc681209902d2cfef856f4b68400e34cac41ec6704f45b580 -rwxr-xr-x root wheel 687 B /private/etc/periodic/daily/430.status-rwho c998ee1b26049571ed20dc89e7dd9d3e87e85846bdf0d9ff1aae9f760774bf5b -rw-r--r-- root wheel 1.1 kB /private/etc/apache2/original/extra/httpd-info.conf b822424c1eba4ae12efa0bad98c6288f9111d5da88f43a847c69e660dc3c8c80 -r--r--r-- root wheel 6.2 kB /private/etc/openldap/schema/collective.schema c90d596b58cb588b3ae77312f9fe8b0511ea8a842caf058d130138a32cca8419 -r--r--r-- root wheel 2.1 kB /private/etc/openldap/schema/fmserver.schema 5475edebf371c8c4771d6951a51a80e4c40871a01355122cac4146966d6aa58c -rw-r--r-- root wheel 4.5 kB /private/etc/apache2/extra/httpd-mpm.conf 61b95be8351545f888956f733db55f810e3b5f8b4b221ad219428b95c55bdbc9 -rw-r--r-- root wheel 1.1 kB /private/etc/asl.conf b9611702dbb3c2d02e060d82cff7a542f6ec6de29aabaa0b4c8482e5ed1f78d0 -rw-r--r-- root wheel 113 B /private/etc/pam.d/authorization_aks 9a31f3b43190281ce1320ac62b3d7672c0bd5bb8bf607c82767994bfa815068e -rw-r--r-- root wheel 153 B /private/etc/asl/com.apple.eventmonitor cc58ac4627390ef04037b7b77ed0359b267a8e8ceb99bdc3aa156dae6367094a -rw-r--r-- root wheel 607 B /private/etc/apache2/original/extra/httpd-userdir.conf 17294d602f2d28944e6517a6a8a432548351d1eaf468062b8da6d84bbf7c5440 -r--r--r-- root wheel 133 B /private/etc/uucp/passwd 27b21d21df689f2f097f56b907d2a936acbeb943fc98bfeb56d2c62cba33c451 -rw-r--r-- root wheel 13 B /private/etc/paths.d/40-XQuartz 1f07bc400e932a1f36c65634b9f5e7d7a6249c58195f903fe21d4bf83df43c5b -rw-r--r-- root wheel 3.4 kB /private/etc/wfs/httpd_webdavsharing_template.conf 444c716ac2ccd9e1e3347858cb08a00d2ea38e8c12fdc5798380dc261e32e9ef -r--r--r-- root wheel 265 B /private/etc/bashrc fa115d33bdc964acfe0b0241c511c2bed643820f8c574e45f443a17a331cf138 -rw-r--r-- root wheel 318 B /private/etc/pam.d/screensaver_ctk eab8fa9f3d43e099731db74d17733fb46c5578206f4dbb0db204bc2cef68664a -rw-r--r-- root wheel 7.6 kB /private/etc/passwd 99c7c05d6e8690b32ea58d6fcda64a090eaae782026551d2d9e347694c626ce7 -r--r--r-- root wheel 7.8 kB /private/etc/openldap/schema/nis.schema 025f5e31cd7b2248a0a661d8d346452c31d98e280f3420c3f6017c74c829ae73 -rw-r--r-- root wheel 745 B /private/etc/ssl/openssl.cnf 9d70873703af389018ccc7b57a503d2692ba1d6b71271bcf00473d40f5095486 -rw-r--r-- root wheel 3.2 kB /private/etc/apache2/original/extra/proxy-html.conf 69c9044d7bdcdd195249b13b8893abf7c60092391975408468ac5e8969fcd79a -rw-r--r-- root _lp 6.5 kB /private/etc/cups/cupsd.conf d7c000b62ab236b5c5c4db8ad7edec232b2afe95d85c8e064a5c9f6d5308620f -rw-r--r-- root wheel 1.6 kB /private/etc/postfix/TLS_LICENSE ed9d05e8ec15f676263a184a7c30858f2d5cd0258da168d3b262b76567bc7bae -rw-r--r-- root wheel 2.9 kB /private/etc/apache2/original/extra/httpd-default.conf bb3d6775b7fc8042b2b68232384694ac3bee6d1100b87f9dcd0840ee935dd2b5 -rw-r--r-- root wheel 12 kB /private/etc/postfix/canonical 96618e0c2ba27c318f77262c22a99f039ecf2e1d83754fcb1a0f208e961da2f7 -r--r--r-- root wheel 1.5 kB /private/etc/openldap/schema/openldap.schema cf6e6fe1497d5b3983529174674b8a09941d67244727157e208d37b24b247bc1 -rw-r--r-- root wheel 166 B /private/etc/pam.d/authorization_la aa464696f1c0282b1728b79e5da52c6f44d130b93c6e3011ea4e67407461c191 -rw-r--r-- root wheel 3.6 kB /private/etc/racoon/racoon.conf 1dc9a5dec35592b043715e6b5a1796df15540ebfe97b6f25fb4960183655eec9 -rw-r--r-- root wheel 9.3 kB /private/etc/zshrc_Apple_Terminal e2dd2ce55d57c625d241b8ddbc834fbab3e9c2061955b094326201dfd083d051 -rw-r--r-- root wheel 202 B /private/etc/asl/com.apple.clouddocs eb33ba8357c7166620b5c33fc8436a0081ba093f7cbf60fef7ddb35b491a4012 -rw-r--r-- root _lp 128 B /private/etc/cups/snmp.conf d7cbf00c6fb63b14e2314cb5aff696e6d01e6911e0bcebf2c540ddb28b614340 -r--r--r-- root wheel 6.8 kB /private/etc/openldap/schema/nis.ldif 91d53776cbd13b565b93dfae32ceb7236c09df3f4629329b521fe1b523460613 -r--r--r-- root wheel 3.3 kB /private/etc/openldap/schema/dyngroup.ldif 4a054f138fbed4a755b351fb51dd83fe3005e07fc7d075c3880e10ec8950873a -r--r--r-- root wheel 123 kB /private/etc/openldap/schema/microsoft.schema 8e920f9fb045a6dfe53fd18bb6450c5b3051bd88c4c7eac88b7c41deee02e6a8 -r--r--r-- root wheel 246 B /private/etc/pam.d/sudo 9d5aa72f807091b481820d12e693093293ba33c73854909ad7b0fb192c2db193 -rw-r--r-- root wheel 189 B /private/etc/shells 81bac019ddc3523e67726f6159c180bd128b9ef1872d5359fe40e3fa78690694 -rw-r--r-- root wheel 387 B /private/etc/asl/com.apple.AddressBookLegacy 309c09ccf0f826e2e753472a01fa32d7f1f326879992e8e925ef3ced57a6252b -rw-r--r-- root wheel 11 B /private/etc/nanorc a27af5a5be37a725abe8587a803e601e243739c2ea4abfde9707199b66369deb -rw-r--r-- root wheel 1.3 kB /private/etc/newsyslog.conf 810dbd036a97217460ec5f56e543621dd2a735bb809fca45327400e2afc272ad -rw-r--r-- root wheel 27 kB /private/etc/postfix/main.cf.proto 8aa44a4856b41d4f1dd7799b975eb68472df1d6b8f8268a0cc6aaa9975f55349 -rw-r--r-- root wheel 891 B /private/etc/rtadvd.conf 49422ce34f9e3b5134cecdb73f5f54fa34480fc424279321dab18240b4bf8cde -r--r--r-- root wheel 2.4 kB /private/etc/openldap/schema/misc.schema eff9ba3a711059ac519e1914477a28db07ce3ed8c70eb36479e0da39cd41fc4e -rw-r--r-- root wheel 197 B /private/etc/pam.d/passwd e523e826b0192c39b42d71597caa8b3e5e166d1e6af98528f046f1ebe6727925 -rw-r--r-- root wheel 24 kB /private/etc/postfix/header_checks f63a90bd20d0baea6f3b7f5c0679d02e58dc294d5a37bd0c44f2046609d74b4a -r--r--r-- root wheel 16 kB /private/etc/snmp/snmpd.conf.default c7e32a8ad7f390bc508c245dfd929cccf18cf222889ca7eed281619495b3dc5f -rw-r--r-- root wheel 346 kB /private/etc/ssl/cert.pem 49fde98b0963f27c0630072414be0ee0205d71eba529bbaf86b65d1e1603eead -rw-r--r-- root wheel 2.9 kB /private/etc/apache2/extra/httpd-autoindex.conf f918b0dec7a783521bd7efbe2e5aef28e2d3250bec1cb524752bb31d18ddbe0b -rw-r--r-- root wheel 149 B /private/etc/asl/com.apple.performance b2dc9fa1938c981ce201f1458afb373d97caf860d9bfba0ded9c450a64f8e65d -rw-r--r-- root wheel 27 B /private/etc/ntp.conf 9e8dffed375e348782c5bc8e15b6a33ea002c96588b7686bd230693dd86898eb -rw-r--r-- root wheel 181 B /private/etc/pam.d/checkpw 8a8a65b24957cbf2c5376a2a7a4dccb3e02f8cc3cfb338caa58cb93345c6a23c -r--r--r-- root wheel 203 B /private/etc/pam.d/cups c74b98809cf175d83699a095bf9f20689dcc66daeedc7e48119026e88103fb09 -rw-r--r-- root wheel 44 B /private/etc/postfix/custom_header_checks 891232253c3bb2789de7cb01b8d313e3c3ed6820434457634775867cc50c657b -rw-r--r-- root wheel 5.4 kB /private/etc/postfix/makedefs.out ba4bb2c9087c046186ef306d4bd5046694e7bc0bc715503ba210e6f96e417642 -rw-r--r-- root wheel 13 kB /private/etc/apache2/original/extra/httpd-ssl.conf 4348468f1850b16e8ad3f7be8963bde559cfd8dd7aed8ffd655a40aaa551e073 -rw-r--r-- root wheel 156 B /private/etc/asl/com.apple.MessageTracer b2aac03248e8f229c703561f5bb059f9be491e5db9f447692398d775a9fb12a5 -rw-r--r-- root wheel 195 B /private/etc/auto_master bf41a1579c3d12a4b8466fad91794157fcca4b2c611fb178ddce232cd9a4e0e5 -r--r--r-- root wheel 8.1 kB /private/etc/openldap/schema/corba.schema c78f581bf6c453c6cf4e4ba241fe081d7de69bbfa42d41fc65a475827c8bd627 -rwxr-xr-x root wheel 695 B /private/etc/periodic/daily/130.clean-msgs 7e94a499c84675258a9734b150a28950fa22cbaad5f1eedfff507541d9f7d9ef -rw-r--r-- root wheel 13 kB /private/etc/apache2/magic 66dfdc46b6b66f0830252f20e1c74aa865656223e30f95fbef6b05051fdc3cde -rw-r--r-- root wheel 9.3 kB /private/etc/bashrc_Apple_Terminal f19f881084f599fa261243918d922373eab14623e78d23c41fcc031aa21ca7b6 -rw-r--r-- root wheel 941 B /private/etc/emond.d/emond.plist b19baf7d26dad9163a2e87c5b9731943e8dad7243c81aacfacf6bdcdcae03c61 -rw-r--r-- root wheel 200 B /private/etc/pam.d/chkpasswd effcfce27485ef1ca9142df4fe810d518e720092b00dce0a9504e2ef378cc42c -rw-r--r-- root wheel 27 kB /private/etc/postfix/main.cf d73737159f9e3bedf75645dd7c2254331beea55567da92f3b974d4ba8194ff81 -rw-r--r-- root wheel 13 kB /private/etc/postfix/virtual 1b3f45ae268583561d2e9baa82474dd0035d974cf687991a20f885f107dd943d -r--r--r-- root wheel 2.0 kB /private/etc/openldap/schema/collective.ldif f99a4803d9f2f8c48c33866487067fbd3a0d02080e10749e70445813b4d60954 -rwxr-xr-x root wheel 1.6 kB /private/etc/periodic/daily/110.clean-tmps 7ac5924452faaf32bbfbd41f816947fccc3ff2fc17554be7bc5fff99d71ebe2e -rw-r--r-- root wheel 6.9 kB /private/etc/postfix/relocated 7cb50dc544e9050e398ebc0844081e928ca52c09a0d304228489962a93b63692 -rw-r--r-- root wheel 365 B /private/etc/hosts fde802d853379ae3724693f05be7f273b92429ab500a7806683e7a21975f743e -r--r--r-- root wheel 3.0 kB /private/etc/openldap/schema/java.ldif dbec41507d3e3c390b0418d96c9ea5f61d3e9ac969bd025b6d23b78af25adaeb -rw-r--r-- root wheel 18 B /private/etc/paths.d/go 2012882e055c2a2502cd7d3dd373c1804f0fce782225e90aca491abad720b53a -rw-r--r-- root wheel 1.5 kB /private/etc/apache2/original/extra/httpd-vhosts.conf dd89ec314b4e26aa2481a315c7e71404ccff929ead511269e934406be0e61143 -rwxr-xr-x root wheel 888 B /private/etc/periodic/monthly/199.rotate-fax 3efeec1e339725bfadd53138dde7822ebc57ba73275a1c0fc05e0d7d8abafc4e -rw-r--r-- root wheel 624 B /private/etc/wfs/httpd_webdavsharing_proxy.conf.inactive ede47b49ba30fbd0e02f10006fa967d2887f7e4c676e6b657ae0a20285ef1d4c -rw-r--r-- root wheel 2.2 kB /private/etc/apache2/extra/httpd-multilang-errordoc.conf ebbeeaa6c956e56727c1391e4443a7da9da70e5e9201fcf69194e9ce397f9a2d -rw-r--r-- root wheel 339 B /private/etc/asl/com.apple.coreduetd 2a0ba79339b7112dbd24b5fdd4b54f32573971a0ad9906240bbeb6bad3d382bc -rw-r--r-- root wheel 5.3 kB /private/etc/rc.netboot 6fb5b260918922ca5ca4dfb296967d8edb9c12f2c043f26b64590758441d682d -rw-r--r-- root wheel 1.0 kB /private/etc/pf.conf 011fd8d7180df2a60de23826192aab50889f73488d01d6c35d186fcb7f64640d -rw-r--r-- root wheel 61 kB /private/etc/apache2/mime.types 7c1f46f8dca762135990bdf0e3aa0395d4867858e35a3f2583549fa6cd5b082a -rw-r--r-- root wheel 189 B /private/etc/csh.cshrc cdfc5a48233b2f44bc18da0cf5e26df47e9424820793d53886aa175dfbca7896 -rw-r--r-- root wheel 45 B /private/etc/paths 64d9fef3a2825ba4d770b38b5ccd12ad4df477535f345fa8649d65fea940fcbf -rw-r--r-- root wheel 512 B /private/etc/pam.d/login 38b6d46f6924364dc3013ffad83a12b7bc10bf3a878b5ae0f53ef117851900b8 -rw-r--r-- root wheel 1.7 kB /private/etc/apache2/extra/httpd-dav.conf ba4bb2c9087c046186ef306d4bd5046694e7bc0bc715503ba210e6f96e417642 -rw-r--r-- root wheel 13 kB /private/etc/apache2/extra/httpd-ssl.conf ede47b49ba30fbd0e02f10006fa967d2887f7e4c676e6b657ae0a20285ef1d4c -rw-r--r-- root wheel 2.2 kB /private/etc/apache2/original/extra/httpd-multilang-errordoc.conf 5475edebf371c8c4771d6951a51a80e4c40871a01355122cac4146966d6aa58c -rw-r--r-- root wheel 4.5 kB /private/etc/apache2/original/extra/httpd-mpm.conf e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/hosts.equiv e05e1b6f966477644849720be93b0707029323b732b4105b63475443fa740aa2 -rw-r--r-- root wheel 376 B /private/etc/asl/com.apple.networking.boringssl 39bc7222686b37aea251957d5b0658600711bc91815a792459395188ace48e33 -rw-r--r-- root wheel 23 B /private/etc/ntp_opendirectory.conf ad2902787f6a2cc2728320d9bf5387371d88a8226c32bac08df9f6d7886bfee4 -r--r--r-- root wheel 20 kB /private/etc/openldap/schema/ppolicy.schema 69a61ae495580e0e144792c9a809bffe707a6a13f292095f43a7a88c92b11ef0 -rwxr-xr-x root wheel 1.2 kB /private/etc/periodic/daily/310.accounting 49fde98b0963f27c0630072414be0ee0205d71eba529bbaf86b65d1e1603eead -rw-r--r-- root wheel 2.9 kB /private/etc/apache2/original/extra/httpd-autoindex.conf f1e89e3a029a3b72a30f82fc7346c13030b4de16b77ecc4c58f4a4de6f99c242 -rw-r--r-- root wheel 82 B /private/etc/com.apple.screensharing.agent.launchd a18ff587cf7d16939acf5a969da49a393ce503b1c4b83ad42e4b902991329b4f -r--r--r-- root wheel 48 kB /private/etc/openldap/schema/apple.schema 6293f4cc430a3d8648fa60f26546d16e55f84d05b71f996ca930a95aa3ce312c -r--r--r-- root wheel 3.3 kB /private/etc/openldap/schema/dyngroup.schema c0dbae0bb16ea7cbc26c0d49286946ef8713335564f5c0a2f4ab7339b8f948f2 -r--r--r-- root wheel 2.1 kB /private/etc/openldap/schema/misc.ldif fc1166d771ec0092cb6280cadf54e9df3fa4eff6be0132f9064f87a7b0b6416a -r--r--r-- root wheel 3.3 kB /private/etc/openldap/schema/openldap.ldif b232f6b78ffdcfdf34697f1432b6cd9bcbad27da384d493d9285b02f398c13ea -rw-r--r-- root wheel 1.7 kB /private/etc/rpc 9a40f8365adb49f0a63930316c475251631150f0f58bfad692a633c8e7486fc9 -rw-r--r-- root wheel 96 B /private/etc/syslog.conf b99ca2f2de327fd9a0374e1eacd3ccbc0f6fce17e0372c42d370ce4852bb75f4 -rw-r--r-- root wheel 22 kB /private/etc/apache2/original/httpd.conf b45fd6ac093bcb4aec44b366d14300cdb1a65f65d7a630630af0f89c9c937894 -rw-r--r-- root wheel 265 B /private/etc/openldap/ldap.conf 36ff03121c003e884d67372d060c83886d73429fc62df37c831f604ecb59cb56 -r--r--r-- root wheel 717 B /private/etc/openldap/schema/apple_auxillary.schema b01b61f6b1486186b84ac009ccfff409fc43dd66e1b780a60de94090e8459dc2 -rw-r--r-- root wheel 154 B /private/etc/newsyslog.d/wifi.conf 334c18bfe6d007a0bcb6f1e7b3619b9592acea6ce60775d6f8d3d56f7ca8d50b -r--r--r-- root wheel 72 kB /private/etc/php.ini.default 1b3a1b45deb322db6df7d39dd52f67d1d49cb89e77388c41dccf640dcd83cd37 -rw-r--r-- root wheel 10 kB /private/etc/postfix/aliases 8923cf7346a5536cc5fadf73e1f92681cf31e73d06548f8abd1e7854f5694290 -rw-r--r-- root wheel 351 B /private/etc/notify.conf a377ed0d15192ea50c9de076583030fe246bec85bd37a9fe1f3cd792e84231ab -r--r--r-- root wheel 20 kB /private/etc/openldap/schema/pmi.schema fbbd9eac3bb65ba340cda731e596fc52e76c49339e7c7dd6653f06d17edb36ad -r--r--r-- root wheel 527 B /private/etc/pam.d/sshd aab9982ea2af8b86b1fde1e8c199de6f12f45db1a831c1804c33ee65e53d6d25 -rw-r--r-- root wheel 264 B /private/etc/pam.d/authorization d0f8d2287d44031b79d9798dc6c68ea7d789ffc129ec2056b3630c938cc54068 -rw-r--r-- root wheel 3.3 kB /private/etc/wfs/httpd_webdavsharing.conf 405101681e712d2727fd6131a84b0eb1b89ed3b72ad3722916368b5ea3cdf6f6 -rw-r--r-- root wheel 943 B /private/etc/wfs/wfs.plist be95a05eafeae5a7450eef61d6477d0b1080f314c3338b5f81c5d5fc65681fdd -r--r--r-- root wheel 1.3 kB /private/etc/irbrc f33710cbbb38b977b7ed1a10038f6f1911d0ce8dd82f65e1a54db13fbecc429f -r--r--r-- root wheel 12 kB /private/etc/openldap/schema/cosine.ldif 422bfa0619b344ec6248e589333df261c4901feca2ce816fd23ef99b5229b7a0 -r--r--r-- root wheel 5.7 kB /private/etc/openldap/schema/samba.schema 950f16a5623699af1fa0e03f9f286374229bbc84d7e887efa3984a1902e48d4b -rw-r--r-- root wheel 175 B /private/etc/newsyslog.d/com.apple.slapconfig.conf 994263a50efad8a1b416754ea4ed463d5e1a7524ed2b3105b780dfe480dbfe94 -r--r--r-- root wheel 145 B /private/etc/php-NOTICE-PLANNED-REMOVAL.txt 97a0a5a23df5f84d6c4396e5e1c4c6b10c690300ee392cfbf633002e300ce2cb -rw-r--r-- root wheel 10 kB /private/etc/postfix/generic b86accf42ae92d4e00b67c50010b7f278bc76c50755bdc9638e2096cbd596890 -rw-r--r-- root wheel 6.4 kB /private/etc/protocols 24240ece7cf8aa30d242f178d7defd3a47d631bbbcf897d2f170e28b863564fa -rw-r--r-- root wheel 178 B /private/etc/asl/com.apple.mail 71be406f45c35429621a43970b0a10debcfd319f0e0ecb1f9b5c5c0ce4cfc3bd -rw-r--r-- root wheel 121 B /private/etc/csh.login 2e5298d1432df0d5f3b741a1e052cfdb73cc4d800db34651f083f36b3c64dcb8 -r--r--r-- root wheel 10 kB /private/etc/openldap/schema/duaconf.schema 36816c4a31b3fbbc86312073f65ba5b514823f89864ad2f3ede7b6b456a8ed4d -r--r--r-- root wheel 27 kB /private/etc/security/audit_event d4a58e12a2e3a9aa4ca72cbe4c63b786b20d6a212a87759f6c93db594be3e3f0 -r-xr-xr-x root wheel 1.3 kB /private/etc/security/audit_warn a744a313d0b95f6f15768b78d15cf3168ffd18abde4e4801a55b0fbb1c37ae86 -rw-r--r-- root wheel 233 B /private/etc/asl/com.apple.install a1b83027e0b929e389bde2984078b7debf7f885051d9f9be18545aa07bebac21 -rw-r--r-- root wheel 409 B /private/etc/asl/com.apple.mkb.internal 533ad90f9c16d3d000a929513d72050bdef7e75bc8ea1790694227c20ab97f2e -rw-r--r-- root wheel 176 B /private/etc/newsyslog.d/com.apple.xscertd.conf b17912cf8ac845d1098829a3f876bdc7597bc8360bb8f260e49d1e3ca26d5029 -rw-r--r-- root wheel 1.4 kB /private/etc/apache2/extra/httpd-manual.conf 28048538f6a15661bdefcf706f39d0003fb9107bdc5206df295db67b66b8345a -rw-r--r-- root wheel 365 B /private/etc/pam.d/screensaver_la 16c13e23b179e6b325102eb285e78b7678ee64231f9bbb2ff6bd5311f38f89ae -rw-r--r-- root wheel 7.4 kB /private/etc/postfix/master.cf 61b42416ea3c5c9d5850364a23ec3aa5b6ddc6e2055b0402d6435b8fed46f3c5 -r--r--r-- root wheel 652 B /private/etc/security/audit_class e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/xtab fb5827cb4712b7e7932d438067ec4852c8955a9ff0f55e282473684623ebdfa1 -r--r--r-- root wheel 3.1 kB /private/etc/zshrc 1aac36c9a80ecab24b5d4346ac5c605a9614f89590e6d0957b82f781a8d40a49 -rwxr-xr-x root wheel 1.1 kB /private/etc/periodic/daily/140.clean-rwho ca2ae7cf01205f3b961c70c607a31c5ed7cc4434dc94d5a5152e18844c5ecffe -rwxr-xr-x root wheel 378 B /private/etc/periodic/daily/199.clean-fax e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/rmtab e02803e7d3435412c397478205b16c95486fde05ecde74959f9799c49c7ce009 -rw-r--r-- root wheel 149 B /private/etc/auto_home bd0ec71eab6515a902703163e65839498429e7d31860c9584b30b635f7423367 -r--r--r-- root wheel 14 kB /private/etc/openldap/schema/java.schema 11eddeb0e0d55ea1bf43c180f04a53533d5cc71d2a69fbbcf7ca0488c8450d79 -rwxr-xr-x root wheel 23 B /private/etc/paths.d/100-rvictl 20909c75c14c9f5360a48c889d06a0d6cfbfa28080348940fc077761744f2aa5 -rw-r--r-- root wheel 822 B /private/etc/emond.d/rules/SampleRules.plist 11ae0d388aed5d193821d33d060da5ed9119d19689ce6c220b407a4c5da3553f -rwxr-xr-x root wheel 1.0 kB /private/etc/periodic/monthly/200.accounting 8c6e2a2647ee854f469a3bb798e02ba5a8b1812cab229ff129f073e7a80c1202 -rw-r--r-- root wheel 678 kB /private/etc/services 063a5972c7b72eda797684b2c034f37a000aaf7bd3b91b01f7cdd3ff74f170e7 -rw-r--r-- root wheel 5.7 kB /private/etc/gettytab b45fd6ac093bcb4aec44b366d14300cdb1a65f65d7a630630af0f89c9c937894 -rw-r--r-- root wheel 265 B /private/etc/openldap/ldap.conf.default bbe18692eb80dc6e27643a5392b358be548edaec210206eeb1a67c99892fc33e -rw-r--r-- root wheel 39 B /private/etc/csh.logout 7de66c7adb93cb1e0da88874d27668c1d78488a9578268e8234460d9083b2e01 -rw-r--r-- root _lp 6.5 kB /private/etc/cups/cupsd.conf.default ee479a7a0dd839c4a08e73d5a6fcffbe750bd47b82cf7c633ed20c5e63b8ed03 -rw-r--r-- root wheel 5.0 kB /private/etc/defaults/periodic.conf e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/kern_loader.conf 7bf0e7399139a3a478b9f447ceb042dee1137261b39dace27db18a93242ebdc2 -r--r--r-- root wheel 1.8 kB /private/etc/openldap/schema/corba.ldif c536e94effbd1e5890f6fb084590a091afc8369a65a9a8b3d71edbfde290da45 -r--r--r-- root wheel 20 kB /private/etc/openldap/schema/core.schema aea837b88e320abcd476f5839952d5e1a612c0e4a225cb9205d20071e6c37745 -rw-r--r-- root wheel 106 B /private/etc/mail.rc b14dbb949b10a0635b7303ff96d86406d8d6c64d414a1e0f91cdf8ba2a62c74a -r--r--r-- root wheel 191 B /private/etc/pam.d/other 9d6f5bffb6fda39e61e1dd90109a42b7a1abf8a5995f3e81e87b7bf4f312b309 -rw-r--r-- root wheel 119 B /private/etc/ftpusers 0cb5352ac33727fd7979e454f6ac1a56b7795a8ab5a25d7cc955133fb47cf9c4 -rwxr-xr-x root wheel 522 B /private/etc/periodic/daily/400.status-disks 16c13e23b179e6b325102eb285e78b7678ee64231f9bbb2ff6bd5311f38f89ae -rw-r--r-- root wheel 7.4 kB /private/etc/postfix/master.cf.default d9013690a0652573167ef63917f6ed2ad63a5e15b1f82d40010c75f33ec85da1 -r--r--r-- root wheel 205 B /private/etc/apache2/other/mpm.conf 14ecbb4a93f277771924c3a9aa870ecd67a742fd6164572af329feef40061325 -rw-r--r-- root wheel 318 B /private/etc/asl/com.apple.login.guest b22a5ff409482104f804f04b0035f2fd1f6801974645f79c87e9025729a50286 -rw-r--r-- root _lp 3.1 kB /private/etc/cups/cups-files.conf.default 18dc169426b3130e43cd1a5919f95dd58ccee6d49a9182a4b2ecdf5c5bb563bc -rw-r--r-- root wheel 577 kB /private/etc/ssh/moduli c64beac4afa42e602286728c1e0158fc90195768e97d5f6bc0d917208faeb3b3 -r--r--r-- root wheel 21 kB /private/etc/openldap/schema/core.ldif 8c716d8131c6f1ae290b49beed057f06bd5cd18b5227cfe3b50556b35e21ce3b -rw-r--r-- root wheel 5.3 kB /private/etc/php-fpm.conf.default a3fe9f414586c0d3cacbe3b6920a09d8718e503bca22e23fef882203bf765065 -r--r--r-- root wheel 189 B /private/etc/profile f63a90bd20d0baea6f3b7f5c0679d02e58dc294d5a37bd0c44f2046609d74b4a -rw-r--r-- root wheel 16 kB /private/etc/snmp/snmpd.conf 6e72e6f9366198e19491ccc785a0cdfb5a05510609c03fcf388b02aac43fec0b -rw-r--r-- root wheel 1.0 kB /private/etc/ssl/x509v3.cnf b86ff58053e9e930a0324f7fb5e0458213567feca9e97de92da20bff82f17e06 -r--r--r-- root wheel 2.4 kB /private/etc/uucp/sys 146f0e625f4a1b8b11fabab45b66d989e85d0408348403678074b3db881e3587 -rw-r--r-- root wheel 165 B /private/etc/newsyslog.d/com.apple.slapd.conf 726813f6f77f7b566a7c004d79cac51dfdc3b28ff3265179e7fbb032b706ba6d -r--r--r-- root wheel 4.0 kB /private/etc/openldap/schema/ppolicy.ldif 8475720c1288108fa6c55c028a26e7aa12155009095dd8c04a547969b65a95b3 -rwxr-xr-x root wheel 548 B /private/etc/periodic/daily/420.status-network abb7d93ea2ef352db65c05101a112264334e7e531522f576ee397371e13fe5d8 -rw-r--r-- root wheel 183 B /private/etc/pam.d/authorization_lacont 4ae766c32277f60bba1b505e3837ccc29587c0cb4fbf65d20b17baa9463b0fb6 -rw-r--r-- root wheel 13 kB /private/etc/postfix/transport a9419086fc2b70f69130c3ee9f8761b0a12b0c7da47d3b779b04ab3827081cf0 -rw-r--r-- root wheel 5.1 kB /private/etc/apache2/extra/httpd-languages.conf e8c63560d75c0c459666f2d8c69bd23b83151080d571148b9a30667f680d6f3c -rw-r--r-- root wheel 3.2 kB /private/etc/group 7ca5887133958e0ac30c92ecec70fe753977cfcc0137cfa93311aeed8fa38e24 -rw-r--r-- root wheel 117 kB /private/etc/openldap/AppleOpenLDAP.plist 4277bb97ba7b51577a0d38151d3e08b40bdf946753f5b5bdeb814d6ff57a8a5e -rw-r--r-- root wheel 515 B /private/etc/afpovertcp.cfg f047c7a23d830f221f142861d2ae10eb4a27d74beff3fcaf599b4c441ae44f5e -r--r--r-- root wheel 13 kB /private/etc/openldap/schema/microsoft.std.schema 94139c5762d820ac8a6d567ca5a5e9753af95e1c8560f23e65bc169be11f4042 -rwxr-xr-x root wheel 712 B /private/etc/periodic/daily/999.local 69f2f70c4d02fe6a0d4ecc99b8cfb676090aae0435c74eb45b2d79c72cb39a09 -rwxr-xr-x root wheel 606 B /private/etc/periodic/monthly/999.local b99ca2f2de327fd9a0374e1eacd3ccbc0f6fce17e0372c42d370ce4852bb75f4 -rw-r--r-- root wheel 22 kB /private/etc/apache2/httpd.conf db16889d677588787af70881d7e0af827e847a5660e9a78c955da53d8edd614b -rw-r--r-- root wheel 105 B /private/etc/asl/com.apple.iokit.power bea9ba8f43b879adff12ba844aad204ae6be074b356a4ff917305725b4341eb5 -r--r--r-- root wheel 154 B /private/etc/newsyslog.d/files.conf 0d5d7c3f51196286c17c894bf77ab0ba057256aa04a455a137c3860fbb46d98b -r--r--r-- root wheel 3.5 kB /private/etc/openldap/schema/README 6c2621756bae333c3af6429407f949f6220bd75df956606ed911bf6b5c8348fd -rw-r--r-- root wheel 140 B /private/etc/pam.d/smbd 38b19f16f2cfe9cb398e59544cbbeeb4998718dbf231251a41eb31893aed223b -rw-r--r-- root wheel 3.5 kB /private/etc/postfix/bounce.cf.default 6dcf0856acd475d75f7d4f2cb5e6a743b2df97ddd394b78e07ae172ab7f04363 -rw-r--r-- root wheel 20 kB /private/etc/postfix/postfix-files 05e4653d532915fca72e4352ade5040c3361340920eee047ac4dd313bc9d8e19 -rw-r--r-- root wheel 1.6 kB /private/etc/ssh/ssh_config 49dede3f9e65fd6ea036bc1598e07f1f73a3ad9c70c4744de954d6d0a8fdbdb5 -r--r--r-- root wheel 194 B /private/etc/apache2/other/php7.conf a5ff0b83be70bdb0107e8e6f4d2bc8e3608e7e9b15e15145b5f24ca4c559f504 -rw-r--r-- root wheel 1.9 kB /private/etc/autofs.conf 1c350ccbbf5e4e09923f5bd1d93fd1578e0a5b6b7d4b85c631d09e1a1a71dbd3 -r--r--r-- root wheel 6.3 kB /private/etc/openldap/schema/inetorgperson.schema 3ada6707f5fd265c70176f24ff988acb15983df6b2b99a5b3cdb833c8c38ea9f -r--r--r-- root wheel 74 kB /private/etc/openldap/schema/cosine.schema c3a8ab17dd75c874d9da896e729c516273900f8e5b80538daf60e8e95ff86660 -rw-r--r-- root wheel 22 kB /private/etc/postfix/access effcfce27485ef1ca9142df4fe810d518e720092b00dce0a9504e2ef378cc42c -rw-r--r-- root wheel 27 kB /private/etc/postfix/main.cf.default 768ff09154f6aacda857cb175ef29cf9d23ef9c38c69efdbf20354dbfd7875b1 -rw-r--r-- root wheel 1.6 kB /private/etc/rc.common 0235d3c1b6cf21e7043fbc98e239ee4bc648048aafaf6be1a94a576300584ef2 -r--r--r-- root wheel 255 B /private/etc/zprofile a9419086fc2b70f69130c3ee9f8761b0a12b0c7da47d3b779b04ab3827081cf0 -rw-r--r-- root wheel 5.1 kB /private/etc/apache2/original/extra/httpd-languages.conf 7c982d301cf583c578abb470f5f3782d1d4451e43f5b90c6555ae75be6060d1a -rw-r--r-- root wheel 53 B /private/etc/networks d8b2b88b7cdec55c5150ced01d4e0a90b275bb11483086ea74cfcd6687309bae -r--r--r-- root wheel 177 kB /private/etc/openldap/schema/microsoft.ext.schema 13e0a0a4991f31f4864096846b379a1d96808c2f0a735b6661aae29481f4bb94 -rw-r--r-- root wheel 356 B /private/etc/pam.d/su 0fd7f2c4f02e68615f137ed090cbba2d1facb40da6bf269e1e326ed6a10aa618 -rw-r--r-- root wheel 329 B /private/etc/pf.anchors/com.apple 847e127803905f596f8c2553ddab8e2d0bca21b4c4ebc79e4be19dcb554df9cb -rw-r--r-- root wheel 1.3 kB /private/etc/bootpd.plist e93775ee26b46f1bfc6b5eab5eab05cee15a1718e87a4a01cfbd474b5409d872 -rw-r--r-- root wheel 621 B /private/etc/locate.rc 0c4f41f80e397939b4a9ce2137a1d725efb4d373e0587c8551b9f1ac12ed0a56 -rw-r--r-- root wheel 312 B /private/etc/pam.d/screensaver_aks c9acb4b8a3e8fd249e672516e37a88d37a9bc8f224b0ab29e76d302c4179de04 -rw-r--r-- root wheel 28 kB /private/etc/pf.os 38b6d46f6924364dc3013ffad83a12b7bc10bf3a878b5ae0f53ef117851900b8 -rw-r--r-- root wheel 1.7 kB /private/etc/apache2/original/extra/httpd-dav.conf e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -rw-r--r-- root wheel 0 B /private/etc/find.codes dbeff46b11216fc8b39dd3160f5085309c031044afdc77f394c94baa558abb60 -rw-r--r-- root wheel 19 B /private/etc/manpaths.d/40-XQuartz c998ee1b26049571ed20dc89e7dd9d3e87e85846bdf0d9ff1aae9f760774bf5b -rw-r--r-- root wheel 1.1 kB /private/etc/apache2/extra/httpd-info.conf b22a5ff409482104f804f04b0035f2fd1f6801974645f79c87e9025729a50286 -rw-r--r-- root _lp 3.1 kB /private/etc/cups/cups-files.conf 411408289c874dfe0cb497c7d09e0e69601183ad7c9cbedcb9eae7543ff849c4 -r--r--r-- root wheel 4.8 kB /private/etc/openldap/schema/duaconf.ldif 3c4f529ed0ba941121c3bb29d6e654299d9d9c77a98c2570465f15b0cce05aef -rw-r--r-- root wheel 408 B /private/etc/pam.d/screensaver 081c0787da77e326dcf46a6d955cc807c42ad2362e99876c1c71257b38ff0c84 -rw-r--r-- root wheel 1.3 kB /private/etc/ttys 2012882e055c2a2502cd7d3dd373c1804f0fce782225e90aca491abad720b53a -rw-r--r-- root wheel 1.5 kB /private/etc/apache2/extra/httpd-vhosts.conf 43b85836499d3227d242c8ed7a9f33ee61a0b6be1df7bc41f962eff46b547ef1 -rw-r--r-- root wheel 1.4 kB /private/etc/asl/com.apple.contacts.ContactsAutocomplete 41cdd0a28b1e13f3a475fafd45dec6d335cddecd6d2872a71450d5ab3336ee67 -rw-r--r-- root wheel 4.6 kB /private/etc/man.conf 933a32202a6ca9d47c1a0af25f8577369b0134ce749002de1228f2b23a909142 -rw-r--r-- root wheel 43 B /private/etc/nfs.conf 9b5ee3e68d2668a06a26bc29488c891a0182b4f2858f8bab97835d4238c8e892 -r--r--r-- root wheel 3.5 kB /private/etc/openldap/schema/inetorgperson.ldif 6191e98d0228c084ec94f80af3639912b53c316db2cbf29a2123eab64491f709 -r--r--r-- root wheel 4.1 kB /private/etc/openldap/schema/krb5-kdc.schema 7ba16d87de833e0b81898b544f981cd8b7dedcb8e9ad285bd954af560e089816 -rwxr-xr-x root wheel 620 B /private/etc/periodic/weekly/999.local ed9d05e8ec15f676263a184a7c30858f2d5cd0258da168d3b262b76567bc7bae -rw-r--r-- root wheel 2.9 kB /private/etc/apache2/extra/httpd-default.conf b17912cf8ac845d1098829a3f876bdc7597bc8360bb8f260e49d1e3ca26d5029 -rw-r--r-- root wheel 1.4 kB /private/etc/apache2/original/extra/httpd-manual.conf eb33ba8357c7166620b5c33fc8436a0081ba093f7cbf60fef7ddb35b491a4012 -rw-r--r-- root _lp 128 B /private/etc/cups/snmp.conf.default 9d70873703af389018ccc7b57a503d2692ba1d6b71271bcf00473d40f5095486 -rw-r--r-- root wheel 3.2 kB /private/etc/apache2/extra/proxy-html.conf b5847002ceeb4b141a882ba230eb84c186e2d59c74fbb718cc323c6b157ebde8 -r--r--r-- root wheel 8.5 kB /private/etc/openldap/schema/netinfo.schema cc58ac4627390ef04037b7b77ed0359b267a8e8ceb99bdc3aa156dae6367094a -rw-r--r-- root wheel 607 B /private/etc/apache2/extra/httpd-userdir.conf 7ed4a5bae35e87e8815cb79dd3be70ac34a3056cb91216a22224a3c99b4c9720 -rw-r--r-- root wheel 350 B /private/etc/asl/com.apple.mkb 125333636796ce03bacf4b079a93b90d3d458a81159c9593339183be5bf51080 -r--r--r-- root wheel 6.9 kB /private/etc/openldap/schema/pmi.ldif 8a8b640738d46484ecc24c689f70702c5ca35dcd4868f7f413bd8727cd474f10 -rw-r--r-- root wheel 36 B /private/etc/manpaths Or list files within a specific subdirectory:\n$ plakar ls 9:/private/etc/uucp b86ff58053e9e930a0324f7fb5e0458213567feca9e97de92da20bff82f17e06 -r--r--r-- root wheel 2.4 kB /private/etc/uucp/sys 17294d602f2d28944e6517a6a8a432548351d1eaf468062b8da6d84bbf7c5440 -r--r--r-- root wheel 133 B /private/etc/uucp/passwd 863f779b43680f81799688e91c18a164047a1a8e9dfff650881e50ba35530ed1 -r--r--r-- root wheel 141 B /private/etc/uucp/port These listings rely on the snapshot index and do not fetch chunks from the repository, however it is possible to actually look at the file content with the cat subcommand:\n$ plakar cat 9:/etc/uucp/passwd # # The passwd configuration file # # # Specify a login name and password that a system can use when logging in. # uucp_tst sekret Now let\u0026rsquo;s make a change to that file and push it again:\n$ sudo sed -i \u0026#39;\u0026#39; -e \u0026#39;s/sekret/notsosecret/\u0026#39; /etc/uucp/passwd $ plakar push /private/etc open /private/etc/krb5.keytab: permission denied open /private/etc/aliases.db: permission denied open /private/etc/racoon/psk.txt: permission denied open /private/etc/security/audit_user: permission denied open /private/etc/security/audit_control: permission denied open /private/etc/sudoers: permission denied open /private/etc/sudo_lecture: permission denied open /private/etc/master.passwd: permission denied open /private/etc/openldap/slapd.conf.default: permission denied open /private/etc/openldap/DB_CONFIG.example: permission denied b41802cb-9424-47eb-8186-2847678a8646: OK From there, it is possible to perform an inode diff:\n$ plakar diff 9 b - drwxr-xr-x root wheel 160 B 2020-01-01 08:00:00 +0000 UTC /private/etc/uucp + drwxr-xr-x root wheel 160 B 2021-03-27 22:38:17.685871971 +0000 UTC /private/etc/uucp - -r--r--r-- root wheel 133 B 2020-01-01 08:00:00 +0000 UTC /private/etc/uucp/passwd + -r--r--r-- root wheel 138 B 2021-03-27 22:38:17.685791098 +0000 UTC /private/etc/uucp/passwd This shows that the /private/etc/uucp directory has had its date change, and that /private/etc/uccp/passwd has had its date and size change, and from there it is possible to request a diff of the file with the diff subcommand:\n$ plakar diff 9 b /private/etc/uucp/passwd --- 98b6658b-a975-47c4-a38f-f958d8d7359f:/private/etc/uucp/passwd +++ b41802cb-9424-47eb-8186-2847678a8646:/private/etc/uucp/passwd @@ -6,5 +6,5 @@ # Specify a login name and password that a system can use when logging in. # -uucp_tst sekret +uucp_tst notsosecret The snapshots are not incremental so it is possible to generate diffs between any of them.\nRemoving old snapshots # Even though snapshots of similar content do not consume much space, there\u0026rsquo;s no use in keeping a lot of old snapshots hanging around. Snapshots can be removed from the repository with the rm subcommand:\n$ plakar ls 98b6658b-a975-47c4-a38f-f958d8d7359f [2021-03-27T22:29:09Z] (size: 3.0 MB, files: 230, dirs: 39) b41802cb-9424-47eb-8186-2847678a8646 [2021-03-27T22:38:22Z] (size: 3.0 MB, files: 230, dirs: 39) $ du -sh ~/.plakar 2.2M /Users/gilles/.plakar $ plakar rm 9 98b6658b-a975-47c4-a38f-f958d8d7359f: OK $ du -sh ~/.plakar 2.1M /Users/gilles/.plakar $ plakar rm b b41802cb-9424-47eb-8186-2847678a8646: OK $ du -sh ~/.plakar 0B /Users/gilles/.plakar The repository keeps chunks as long as they are referenced by at least one snapshot, but removes them when they become orphans which is why the repository becomes empty after the last snapshot is removed.\nNamespaces # Sometimes I want to make sure that my backups are fully disjoint one from another, not sharing their snapshots and deduplication. The plakar repository supports namespaces so that I can keep related snapshots together and apart from others.\nFor example, I have backups for poolp.org and for hypno.cat and I want to manage the snapshots separately:\n$ plakar -namespace poolp.org push data 351348b3-8dc3-4caa-b0be-334c43f4fd13: OK $ plakar -namespace hypno.cat push data 1fa2dac2-8096-483a-a2d3-2a78ce3706ce: OK $ plakar -namespace poolp.org ls 351348b3-8dc3-4caa-b0be-334c43f4fd13 [2021-03-27T22:49:58Z] (size: 896 kB, files: 79, dirs: 4) $ plakar -namespace hypno.cat ls 1fa2dac2-8096-483a-a2d3-2a78ce3706ce [2021-03-27T22:50:40Z] (size: 896 kB, files: 79, dirs: 4) $ Stores # In addition, I may want to provide an alternate path to the repository because ~/.plakar is not ideal, plakar supports providing alternate repository path with the -store option:\n$ plakar -store /var/backups/gilles push data f6cc2508-ea59-4552-aa4b-27900f0a9781: OK $ plakar -store /var/backups/gilles ls f6cc2508-ea59-4552-aa4b-27900f0a9781 [2021-03-27T22:52:47Z] (size: 896 kB, files: 79, dirs: 4) But this doesn\u0026rsquo;t stop here \u0026hellip;\nplakar server # So far I only showed plakar working on a local repository, it however supports a network mode.\nIt is possible to start a plakar server on a machine with:\n$ plakar server 192.168.0.1:2222 $ and on another machine, point the repository to that server:\n$ plakar -store plakar://192.168.0.1:2222 push /bin 3ac8f3ff-e723-46c3-9ae7-4b11bb27f3b2: OK $ The plakar server can use -store itself, so it is possible to configure it to store at at particular location:\n$ plakar -store /var/backups server 192.168.0.1:2222 $ and because I made a monster, it is possible that this location is a different plakar server, effectively turning a plakar instance into a proxy:\nbox1$ plakar server 192.168.0.1:2222 box1$ box2$ plakar -store plakar://192.168.0.1:2222 server 192.168.0.2:2222 box2$ box3$ plakar -store plakar://192.168.0.2:2222 push /bin 998964ac-d3f6-4bf3-b784-50ba616e0776: OK $ From the local user point of view, pushing to a local plakar, to a remote plakar, or even to a remote plakar proxying to another plakar work the same.\nIs it released ? # Nope, not yet.\nThis is a PoC and my first real project in Golang, so I need a bit of time to make sure it is in a state that I\u0026rsquo;m not ashamed to release.\nIf you are a sponsor and want to give it a try, just mail me and I\u0026rsquo;ll arrange that.\nWhat\u0026rsquo;s next ? # A lot of things really\u0026hellip;\nFirst of all, every single command needs a bit of polishing to have a nice feel because the output is not always the best to work with.\nThen, I have not implemented any caching whatsoever so even though it\u0026rsquo;s not dead slow, a lot of operations that could be skipped are always performed making it slower than it should.\nAlso, the server mode doesn\u0026rsquo;t use encryption at this point and while it is ok on a personal network, I want to be able to deploy a plakar server on a dedicated server at some provider and use it safely.\nFinally, I began building an UI to browse snapshots and this requires building a global index of snapshots, which is something I haven\u0026rsquo;t tackled yet. At this point, each snapshot has its own index but I\u0026rsquo;d like to be able to have a search box which looks up for stuff through all snapshots.\nI also have other private plans with this which I\u0026rsquo;ll discuss later ;-)\n","date":"26 March 2021","permalink":"/posts/2021-03-26/march-2021-backups-with-plakar/","section":"Posts","summary":"TL;DR: I wrote a backup utility called plakar. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !)","title":"March 2021: backups with Plakar"},{"content":" TL;DR: I converted nooSMTPD to libtls and implemented SMTP ciphers, curves and protocols selection. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !)\nNothing new in OpenSMTPD-portable # eric@ sent a libtls-conversion diff to tech@ last month but there hasn\u0026rsquo;t been much progress since then.\nI tested the diff extensively and I\u0026rsquo;m currently waiting for it to be merged to OpenBSD so I can bring it to OpenSMTPD-portable.\nnooSMTPD is libtls-enabled # In nooSMTPD, I did the libtls-conversion by applying the diff and making the appropriate changes to the libtls compat layer.\nI switched my machines from my custom FrankenOpenSMTPD to nooSMTPD and been running it since late January, with no regressions.\nIt\u0026rsquo;s interesting to note that while I ran OpenSMTPD from upstream repository without the autoconf build, I\u0026rsquo;m running nooSMTPD from autoconf which led to a lot of swearing as it was never meant to be used on OpenBSD. I fixed a few unpleasant things which I\u0026rsquo;ll be able to push to OpenSMTPD-portable as well.\nThe smtp tls global configuration # OpenSMTPD allows setting up SMTP global configuration with the smtp global keyword:\nsmtp max-message-size \u0026#34;30M\u0026#34; smtp sub-addr-delim \u0026#34;+\u0026#34; Because it may be desireable to setup an alternate list of ciphers, the ciphers keyword was introduced a long time ago and a global option allows overriding the default ciphers:\nsmtp ciphers \u0026#34;HIGH:!aNULL:!MD5\u0026#34; In nooSMTPD, I decided to introduce the tls keyword for TLS related options.\nThe libtls conversion unlocks several TLS-related features that may be confusing below smtp, without any reference to TLS. For instance, I have implemented TLS protocol selection and unless tls appears in the configuration this will result in:\nsmtp protocols [...] which is much more confusing than:\nsmtp tls protocols [...] as it is unclear that the protocols are TLS related.\nAs a result, I moved ciphers below smtp tls so in nooSMTPD it reads as:\nsmtp tls ciphers \u0026#34;HIGH:!aNULL:!MD5\u0026#34; I\u0026rsquo;ll suggest this change to OpenSMTPD too.\nTLS ciphers selection # As mentionned above, I have changed the grammar to introduce a tls namespace to the smtp keyword:\nsmtp tls ciphers \u0026#34;HIGH:!aNULL:!MD5\u0026#34; Because libtls supports cipher suite groups, the secure, compat, legacy or insecure groups can be used in place of a cipher suite, and since secure is the default, the global cipher selection only needs to be specified for other cases:\nsmtp tls ciphers \u0026#34;compat\u0026#34; With this done, I also added the ability to setup the ciphers at the listener level:\nlisten on 0.0.0.0 tls ciphers \u0026#34;compat\u0026#34; listen on 192.168.0.1 tls ciphers \u0026#34;legacy\u0026#34; # use on legacy subnet This was committed to nooSMTPD and I\u0026rsquo;ll submit a diff to OpenSMPTD when libtls-conversion is done upstream.\nTLS curves selection # The selection of curves was never merged in OpenSMTPD, as a result it is not possible to select specific curves at either one of the global or listener level. The default curves (X25519,P-256,P-384) in libtls are sensible choices but some use-cases require restricting to a specific curve, or even an alternate one.\nI introduced a curves option which allows specifying curves to override the default:\nsmtp tls curves \u0026#34;X25519\u0026#34; While at it, I added the ability to setup the curves at the listener level:\nlisten on 0.0.0.0 tls ciphers \u0026#34;X25519\u0026#34; listen on 192.168.0.1 tls curves \u0026#34;P-256,P-384\u0026#34; This was committed to nooSMTPD and I\u0026rsquo;ll also submit a diff to OpenSMPTD when libtls-conversion is done upstream.\nTLS protocols selection # Just like for curves, the selection of a TLS protocol was never merged in OpenSMTPD.\nI introduce a protocols option which allows specifiying protocols to override the default:\nsmtp tls protocols \u0026#34;tlsv1.3:tlsv1.2\u0026#34; And again, I added the ability to setup the protocol at the listener level:\nlisten on 0.0.0.0 tls protocols \u0026#34;tlsv1.3\u0026#34; Guess what\u0026hellip; Yes, this was also committed to nooSMTPD and I\u0026rsquo;ll submit a diff to OpenSMPTD when libtls-conversion is done upstream.\nGlobal smtp tls grammar needs to be reworked # The smtp tls keyword was implemented as a placeholder for the ciphers, curves and protocols options, but it was not implemented the way it should.\nBasically, ciphers, curves and protocols should be options to tls so that it is possible to write:\nsmtp tls ciphers \u0026#34;secure\u0026#34; protocols \u0026#34;secure\u0026#34; At the time being, this is not possible and it must be expressed as follows:\nsmtp tls ciphers \u0026#34;secure\u0026#34; smtp tls protocols \u0026#34;secure\u0026#34; This requires a bit of grammar refactor which I\u0026rsquo;ll be working on in March.\nListener tls grammar was reworked # The listener tls keyword suffered from the same issue as the global smtp tls keyword.\nInitially, it was implemented in such a way that ciphers, curves and protocols were standalone options. This allowed avoiding the need to repeat tls before every option, you needed one tls on the listener so it was flagged as TLS-enabled and the options would check for the flag. This worked but it was not the right way to do it.\nI reworked the grammar for tls in listeners so that ciphers, curves and protocols are tls options. This avoids having to repeat tls before every keyword, and it allows relying on the grammar rather than on a flag to determine if the configuration is valid:\nlisten on 0.0.0.0 tls ciphers \u0026#34;secure\u0026#34; curves \u0026#34;P-256,P-384\u0026#34; protocols \u0026#34;secure\u0026#34; I\u0026rsquo;m happy with the result.\nWhat\u0026rsquo;s next ? # I will rework the grammar for global settings and will implement these options for the MTA layer, as these options were done only for the incoming path.\nI intend to publish a technical article in March regarding software design and data.\nI might unveil a project that\u0026rsquo;s unrelated to computer programming too :-)\n","date":"28 February 2021","permalink":"/posts/2021-02-28/february-2021-noosmtpd-libtls-conversion-ciphers-curves-and-protocols/","section":"Posts","summary":"TL;DR: I converted nooSMTPD to libtls and implemented SMTP ciphers, curves and protocols selection. Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe !","title":"February 2021: nooSMTPD libtls-conversion, ciphers, curves and protocols"},{"content":" TL;DR: I do LoFi now, eric@ revived some libtls conversion work I did a while back, I worked on UNIX-domain sockets support in OpenSMTPD, a few words about nooSMTPD Let\u0026rsquo;s start with some LoFi # Relax.\nI have a youtube channel (subscribe ! now !) with a playlist of LoFi tracks that I mix.\nI\u0026rsquo;ll try to insert a new one in every monthly report\u0026hellip; if time allows ;-)\nlibtls-enabled OpenSMTPD # In January 2020, I worked on converting OpenSMTPD to the libtls API so that I could simplify TLS support and bring new features that I didn\u0026rsquo;t want to implement with the libssl API.\nThe diff was never merged into OpenBSD and ended up rotting in a private branch. This was unfortunate because I spent time making this work for portable too, building an OpenSSL-enabled libtls that would allow distributions not shipping LibreSSL to still benefit from this TLS simplification.\nEric Faurot (eric@) has recently picked up my work and pushed it even further, managing to make OpenSMTPD completely libssl-agnostic while I still needed bits for the privsep crypto engines. He submitted a diff to tech@ which is pending review. Once his diff gets committed in OpenBSD, I\u0026rsquo;ll rework the portable code to make it work again so this benefits users on other systems.\nI\u0026rsquo;ve been suggested by eric@ and a tech@ reader to look at libretls. The problem is that libretls is not yet available in the package repositories of all distributions supported by OpenSMTPD, and eric@ also brought changes in LibreSSL that libretls will need to catch up with before I can rely on it. I think the best approach is to rely on the standalone code I already wrote and which is shipped with OpenSMTPD, then once libretls is widely available I can reassess the situation.\nlibtls-related features # I have three features related to the libtls change that are already written, rotting in their own branches, and that I will submit once the TLS diff is committed. They make it possible to configure the ciphers, curves and TLS versions supported for each listeners.\nThe rationale behind this is that some setups require stronger (or specific) crypto constraints for specific networks. This change would allow declaring that a specific interface requires use of a different set of ciphers or TLS version, and allow for example to have an interface with the default settings for Internet and one with extra-paranoid settings for a specific MX.\nI also had started working on other features, like certificates fingerprinting or OCSP stapling, but I stopped as I didn\u0026rsquo;t want to have too much code building upon an uncommitted diff.\nI\u0026rsquo;ll resume working on these once eric@ commits the libtls conversion.\nsmtpd(8) now binds UNIX-domain sockets # At the exception of the local enqueuer, OpenSMTPD only supports SMTP sessions over network connections. This shows in the configuration file as it is only possible to use listen directives with a network interface parameter\u0026hellip; or with the socket keyword which is handled as a special case for the local enqueuer.\nWhen the daemon starts, it creates a UNIX-domain socket to use as its control socket for the smtpctl(8) utility. This was initially meant to accept control requests from root to interact with smtpd(8) at runtime, but later came the need for local enqueueing and the control socket was used for that purpose too.\nA special \u0026ldquo;enqueue\u0026rdquo; control command was implemented in smtpd(8). That command allows smtpctl(8) to request from smtpd(8) that it establishes an internal SMTP session and returns the file descriptor. The smtpctl(8) utility then uses that descriptor to pretend that the socket was actually connected directly to the SMTP server, and it toggles into an enqueue mode where it operates as an SMTP client.\nThis has three side-effects:\nBecause everyone is supposed to be able to enqueue a local mail, the control socket cannot have strict permissions and has to be world-writeable. Since it is also used for privileged control commands, the daemon needs to explicitly check that the client is only allowed to access the enqueue mode if the user is not privileged:\n$ ls -l /var/run/smtpd.sock srw-rw-rw- 1 root wheel 0 Dec 28 06:38 /var/run/smtpd.sock $ smtpctl show queue smtpctl: need root privileges Then, because there is some magic involved to request the file descriptor and toggle in enqueue mode, the control socket is not really connected directly to the SMTP server. It is not possible to connect to it with netcat and talk SMTP as it expects a control command sent with the imsg(3) framework. The server will just ignore the client as it doesn\u0026rsquo;t speak its protocol, and the connection must be established by smtpctl(8) requesting the enqueue mode (or a clone that knows how to do so):\n$ nc -U /var/run/smtpd.sock HELO ?????? ^C $ Finally, users expect to be able to enqueue mail even if the daemon is not available. This requires the local enqueuer to support an offline queue, which itself relies on having an executable setgid to the same group as the offline directory. Because smtpctl(8) is used for both control and enqueuing, it ends up having the setgid bit itself:\n-r-xr-sr-x 1 root _smtpq 217696 Dec 23 06:42 /usr/sbin/smtpctl A year ago, I told eric@ that I believed this was a poor decision made in the early days. Both the control and the enqueuing code could be made simpler and stricter if they were split apart.\nIf local enqueuing had its own dedicated socket which established a connection to an SMTP session, then any SMTP client that knew how to connect to a UNIX-domain socket could be used as the local enqueuer. OpenSMTPD even ships with one\u0026hellip; smtp(1).\nThis would simplify smtpctl(8) as it would no longer need an enqueue mode and a builtin SMTP client. It would also simplify smtpd(8) as it would no longer need to implement the internal SMTP session and fd passing logic, but would also no longer need to check if a user can or can\u0026rsquo;t run a command: if it\u0026rsquo;s not root, it can\u0026rsquo;t. The control socket and smtpctl(8) could both be restricted to root as they would only be used for control requests.\nIf you don\u0026rsquo;t see the benefit behind this, many years ago there were two bugs that allowed local users to crash smtpd(8) from smtpctl(8). One affected the control command counter, which a user could not have messed up with if control commands were restricted to root, and the other affected the fd passing for the enqueue mode, which would not even exist if enqueuing had its own dedicated socket instead of being a control command.\nThis sounds nice but converting local enqueuing to use its own dedicated UNIX-domain socket is trickier than it seems. The technical aspect is simple, you just bind a UNIX-domain socket instead of a TCP socket, but it raises a lot of other questions regarding the behavior of the daemon. I will discuss that in a future post as this is still being sorted out.\nWhat could be done right now and that wouldn\u0026rsquo;t raise questions was to teach smtpd(8) how to bind a UNIX-domain listener. It makes it possible to declare listeners that have UNIX-domain sockets as their endpoints and which plain SMTP clients can connect to:\n$ cat /etc/mail/smtpd.conf |grep listen listen on socket \u0026#34;/tmp/foobar.sock\u0026#34; listen on socket \u0026#34;/tmp/barbaz.sock\u0026#34; $ nc -U /tmp/foobar.sock 220 debug.poolp.org ESMTP OpenSMTPD ^C $ printf \u0026#34;subject: test\\n\\ntest\u0026#34; | smtp -s /tmp/barbaz.sock gilles@poolp.org gilles@poolp.org: EOM: 250 2.0.0 593c830c Message accepted for delivery $ It doesn\u0026rsquo;t solve the local enqueuer issues as it still uses the control socket, but it allows regular SMTP clients to submit mail over a UNIX-domain socket without relying on control commands. Moving forward in this direction, a new local enqueuer can be written that doesn\u0026rsquo;t rely on the enqueue control command, which ultimately leads to enqueuing being removed from smtpctl(8) and more restrictive permissions on the control socket.\nI already showed the diff to eric@ but I will send it to OpenBSD next week.\nsmtp(1) now talks Unix-domain socket # Following the UNIX-domain sockets listeners idea, I thought it would be nice to do the client side too.\nA while back, eric@ wrote smtp(1) which is a simple utility to submit mail to SMTP servers on the command line:\n$ cat\u0026lt;\u0026lt;EOF | smtp -s localhost gilles@poolp.org Subject: foo bar EOF gilles@poolp.org: EOM: 250 2.0.0 5cc0b0d8 Message accepted for delivery $ The smtp(1) client didn\u0026rsquo;t know how to connect to a UNIX-domain socket so I fixed that:\n$ cat\u0026lt;\u0026lt;EOF | smtp -s /tmp/smtpd.sock gilles@poolp.org Subject: foo bar EOF gilles@poolp.org: EOM: 250 2.0.0 d73542e2 Message accepted for delivery $ This allowed me to test my diff on the server side, but it also made me realize that it could be used as the base for a new local enqueuer outside of smtpctl(8).\nThe current local enqueuer is based on the femail MUA, which was kind of hacked here and there to fit in smtpctl(8) and work with its enqueue mode. It reads the mail from its standard input, does some sanitizing and crafting, then submits it using the SMTP protocol on a file descriptor which points to an SMTP session.\nIn a world where local enqueuing doesn\u0026rsquo;t require smtpctl(8) entering a special enqueue mode, it would be fairly easy to use smtp(1) as a base to write a new enqueuer. It already reads mail from standard input, it already knows how to speak SMTP and with my diff it knows how to connect to a UNIX-domain socket. All that would be missing is adding the sanitizing and crafting bits which we already have.\nI also already showed the diff to eric@ and will send it to OpenBSD next week with the listeners one.\nnooSMTPD: Not OpenBSD\u0026rsquo;s OpenSMTPD # I didn\u0026rsquo;t want to talk about this yet but since people have spotted the repository and are making assumptions, I need to explain what that is.\nIn December, I started working on an MTA. It is based on OpenSMTPD, but it takes a different direction and has different goals.\nOpenSMTPD is a general purpose MTA, written primarily for OpenBSD, which needs to fit the base system and care about legacy and how changes affect the user base.\nnooSMTPD doesn\u0026rsquo;t have these constraints. I\u0026rsquo;m writing it primarily for myself and will happily break legacy behaviors and change the configuration file every two months if it makes things easier for me. I intend to support some advanced features found in commercial MTA and that are unlikely to be accepted in OpenBSD because\u0026hellip; OpenSMTPD is a general purpose MTA.\nOpenSMTPD benefits from the work I do there as I share all diffs, but in some cases they are unsuitable for OpenBSD and this is where things diverge between the two, it will contain diffs that don\u0026rsquo;t make it into OpenSMTPD.\nI\u0026rsquo;ll write about it more in details in the future, I just wanted to clarify that this isn\u0026rsquo;t the fork some people think it is. I have exchanged multiple diffs with eric@, millert@, martijn@ and even tech@ since December, and I continue to talk about OpenSMTPD changes with eric@ every week.\nI just have ideas for a different project :-)\nbreaking changes in nooSMTPD # I have killed the dead.letter feature which allows OpenSMTPD to save a copy of a mail when it fails to submit it to the enqueuer. Modern MUA do not make use of it, it is solely used by legacy MUA, and I always thought this was not the job of the MTA to handle it. It was also a vector of attack in the past.\nI have killed delivery to the root user for all mail delivery agents, the only way root can receive a mail is through an alias to an unprivileged account.\nI have merged all of the diffs from last month, including the safety net that detects injection of a custom MDA for dispatchers not allowing exec.\nassorted portability improvements and cleanups in nooSMTPD # I reworked some code patterns confused compilers and led to false positives in warnings. For example, the following construct was found in multiple places, and led compilers not knowing that fatal() never returned into assuming that p might be uninitialized in the call to barbaz():\nvoid foobar(x) { char *p; switch (x) { case 0: p = \u0026#34;a\u0026#34;; break; case 1: p = \u0026#34;b\u0026#34;; break; default: fatal(\u0026#34;die.\u0026#34;) } barbaz(p); } I also replaced some functions, such as ctime() and localtime(), with their reentrant versions ctime_r() and localtime_r(). nooSMTPD is not threaded but this raises alerts from analysis tools and, while doing something just so a tool would shut up is not a good rationale, in this case there\u0026rsquo;s no real downside and it saves me from having to keep flagging stuff as false positives.\nWhat\u0026rsquo;s next ? # Moving forward with all of the above.\n","date":"31 January 2021","permalink":"/posts/2021-01-31/january-2021-opensmtpd-libtls-conversion-and-unix-domain-sockets-support-noosmtpd/","section":"Posts","summary":"TL;DR: I do LoFi now, eric@ revived some libtls conversion work I did a while back, I worked on UNIX-domain sockets support in OpenSMTPD, a few words about nooSMTPD Let\u0026rsquo;s start with some LoFi # Relax.","title":"January 2021: OpenSMTPD libtls conversion and UNIX-domain sockets support, nooSMTPD"},{"content":" TL;DR: In this article, I explain what is an MDA and how to write a custom one from scratch using only shell scripting. What is a Mail Delivery Agent (MDA) ? # When a mail enters an SMTP server it is initially stored in the server queue before being moved to its next destination.\nIf the destination is a remote machine, the mail goes back to a Mail Transfer Agent (MTA) and is transferred elsewhere, but if the destination is a local user then it is passed to an MDA for delivery.\nAn MDA is a program that is in charge of delivering a mail to a local user and reporting to the SMTP server if the delivery was successful or not.\nThe MDA interface is somewhat standard so that if you write an MDA that doesn\u0026rsquo;t take advantage of features specific to a particular SMTP server, it will work with others fine. This is why tools such as procmail or fdm can be used with OpenSMTPD, Postfix, Sendmail, notQmail or Exim indistinctly.\nAn MDA can be written in any language but to keep the examples simple and understandable by all, I decided to illustrate with shell scripting which is the poorest choice of a \u0026ldquo;language\u0026rdquo; you could make to write an MDA, be warned.\nHow does it work ? # An MDA works in a straightforward way.\nIt is a program that reads its input from stdin and reports success or failure through its exit(3) status.\nBased on this description, the following shell script is a valid MDA as far as any SMTP server is concerned:\n#! /bin/sh cat exit $? What it does is call the cat command, which consumes its input from stdin and writes it back to stdout, then propagate its exit value as that\u0026rsquo;s how cat reports its success or failure.\nThis MDA does a poor job at MDA-ing because all it does is consume the input without writing it anywhere useful. As a result, the MDA will report success when cat is done reading the mail and echo-ing it back to stdout but the mail content will be gone forever.\nThis MDA could be made more useful by redirecting stdout to a file so that the input does not get lost:\n#! /bin/sh cat \u0026gt;\u0026gt; /tmp/mail-archives exit $? In this version, the /tmp/mail-archives file is opened for append and cat has its stdout redirected to it. Every time the MDA gets called for a mail, the mail gets appended to that file.\nThis does a good job at explaining what is expected from an MDA, unfortunately it is a broken example.\nFirstly because an MDA is executed for each mail being delivered, possibly simultaneously, and in the absence of a locking mechanism this might result in concurrent writes mangling its content.\nSecondly because an MDA is executed with the privileges of the recipient user which means that every possible recipient needs to have write access to that file, with all the downsides you can imagine.\nLet\u0026rsquo;s see how to write something better.\nThe stdin stream # The first thing to understand is that the mail is received on the standard input, stdin, as a stream of lines which are interrupted by an EOF.\nThere is no back and forth protocol of any kind, the MDA is the recipient of a unidirectional stream of input and does not have a side-channel to let the server know of what happens during the processing of this stream.\nIt consumes stdin, does something which the server doesn\u0026rsquo;t know the details of, and tells the server when its done by exiting with a value indicating success or failure.\nIt doesn\u0026rsquo;t get much simpler.\nsysexits(3) # To report success or failure, the MDA operates like a standard unix program: it exits with 0 to indicate success and anything else to report failure.\nSince that is the only way to report information to the server, MDA can make use of sysexits(3) status code to let the server interpret the reason behind a failure \u0026hellip; or at least goes the theory.\nIn practice these codes are not used much and they don\u0026rsquo;t impact the SMTP server behaviour much. Using the wrong code will result in nothing bad happening except maybe an inaccurate error message if the SMTP server does try to interpret them.\nMDA privileges # An MDA is not a long-running process that gets started once and runs forever. Every time a delivery happens for a recipient, a process is fork(3)-ed with the recipient privileges in order to execute an instance of the MDA.\nIf gilles@poolp.org receives a mail and jules@poolp.org receives a mail, and they use the same MDA, then there will be one process owned by gilles and one process owned by jules, both running concurrently (or at least they might).\nTherefore, the MDA must not assume access to resources that require specific privileges. It must access resources that are available to the recipient user it was executed for. In other words, the MDA for gilles may access resources that gilles could access using a shell if he had one.\nThe example MDA above was bad because the /tmp/mail-archives must be shared between all users for it to work and this implies open permissions (770 if users share the same group, 777 otherwise). A simple change to create a recipient-specific file would have been enough to avoid this issue and allow a more restrictive permission of 700:\n#! /bin/sh cat \u0026gt;\u0026gt; /tmp/mail-archives.`whoami` exit $? Note that the use of whoami here is not a good idea, it is here to back my point, but we\u0026rsquo;ll discuss that in the next section.\nAlternatively, the file could be moved to the recipient user home directory to achieve the same result:\n#! /bin/sh cat \u0026gt;\u0026gt; ~/mail-archives exit $? In that case, we\u0026rsquo;d be ok permission-wise but not with regard to concurrency as lack of locking would mean concurrent deliveries to the same user might mangle the file.\nThe MDA environment # An MDA inherits an environment from the SMTP server allowing it to determine who it is running for, what was the e-mail address of the sender and recipient, and other useful information.\nFirst comes the shell environment variables, the ones you get when logging in to an account on Unix systems:\nThe PATH variable contains the default PATH allowing the MDA to execute standard programs on the host system. In my example MDA above, I didn\u0026rsquo;t write /bin/cat because /bin was exposed in the MDA PATH.\nThe SHELL variable contains the shell that was executed to run the MDA.\nThe HOME variable contains, well, the path to the current recipient user home directory.\nThe LOGNAME and USER variables contain the recipient user. This is the owner of the current process, the one whose HOME is set for.\nThen comes the MDA specific ones:\nThe DOMAIN variable contains the domain name of the recipient at the time of the SMTP session (that is before any aliasing).\nThe LOCAL variable contains the user-part of the e-mail address that was used during the SMTP session (that is before any aliasing). ${LOCAL}@${DOMAIN} is the e-mail address that was used in the SMTP session.\nThe RECIPIENT variable contains the recipient e-mail address following all aliasing.\nThe SENDER variable is set to the e-mail address of the sender or to an empty string in case of a MAILER-DAEMON bounce.\nThe EXTENSION is set to the extension if there\u0026rsquo;s any used with the recipient e-mail address. The extension is what follows + and precedes @ in e-mail addresses such as gilles+1@poolp.org\nAs you can see, there\u0026rsquo;s a few of them covering about all session informations needed to properly execute an MDA. If we were to take a silly broken example from above:\n#! /bin/sh cat \u0026gt;\u0026gt; /tmp/mail-archives.`whoami` exit $? We could replace it as follows to make use of the environment:\n#! /bin/sh cat \u0026gt;\u0026gt; /tmp/mail-archives.${USER} exit $? It would be better, but what if we wrote a better MDA, one that is usable without concurrency or permission issues ?\nWriting a maildir-like MDA # In this section, we will write an MDA that delivers mail to a maildir-like structure.\nContrarily to the previous examples, this one will ensure that there are no issues if multiple instances run concurrently for one or many users.\nFirst, we\u0026rsquo;ll write the base for our MDA, the stdin reading loop:\n#! /bin/sh cat exit $? Then, we need to figure where the maildir should be located and create the directory structure if needed, applying restrictive permissions. Because we know that the MDA is provided with a HOME environment variable, we can make use of that:\n#! /bin/sh MAILDIR=${HOME}/Maildir umask 077 test -d ${MAILDIR} || mkdir ${MAILDIR} test -d ${MAILDIR}/cur || mkdir ${MAILDIR}/cur test -d ${MAILDIR}/new || mkdir ${MAILDIR}/new test -d ${MAILDIR}/tmp || mkdir ${MAILDIR}/tmp cat exit $? At this point, we have dodged issues with two different users running the same MDA since we deliver to a recipients home directory but we have to worry about the same MDA being executed concurrently for the same user.\nThis could be handled with a lock, as is done for mbox-style mailboxes that use a single file per user, but maildir has a more elegant solution to solve this: generating unique filenames in a temporary directory, then atomically renaming to the destination directory.\nBy writing to a temporary directory, the file is not yet visible by the user until it is finished being written, at which point the atomic rename makes it visible. If any error happens in between, the file remains in the temporary directory and can be purged after a while, but there is no risk of exposing a partial file.\nWe\u0026rsquo;ll adapt the MDA to save a copy of the mail from stdin to a temporary file with a unique name. To construct the name, I decided to go with a unix timestamp, followed by a 64-bits random value and the local hostname. This ensures that there can\u0026rsquo;t be collisions if generated on two machines sharing the same network storage due to the hostname, and that collisions are unlikely on the same host as they would require a 64-bits collision of a random value happening within a second. Once the temporary file has been written and cat reported success, it is moved to the ~/Maildir/new which makes it visible to clients:\n#! /bin/sh MAILDIR=${HOME}/Maildir umask 077 test -d ${MAILDIR} || mkdir ${MAILDIR} test -d ${MAILDIR}/new || mkdir ${MAILDIR}/new test -d ${MAILDIR}/tmp || mkdir ${MAILDIR}/tmp # ie: 1609171573.deb626c88c2da87f.laptop FILENAME=`date +%s`.`openssl rand -hex 8`.`hostname` cat \u0026gt; ${MAILDIR}/tmp/${FILENAME} \u0026amp;\u0026amp; \\ mv ${MAILDIR}/tmp/${FILENAME} ${MAILDIR}/new/${FILENAME} exit $? We can test that it works by running the command outside the SMTP server:\ngilles@mba ~ % ls -l ~/Maildir ls: /Users/gilles/Maildir: No such file or directory gilles@mba ~ % sh mda.sh HOLA ! gilles@mba ~ % ls -l ~/Maildir/new total 8 -rw------- 1 gilles staff 7 Dec 29 00:52 1609199557.db7be737b9423c87.mba gilles@mba ~ % cat ~/Maildir/new/1609199557.db7be737b9423c87.mba HOLA ! gilles@mba ~ % This MDA does not perform any check on the mail content and copies the stream as is but nothing prevents your MDA from doing arbitrary work.\nThe MDA could parse the headers to extract information, collect statistics, store the content in a database, or do anything including apply stupid rules like not allowing a delivery on rainy days.\nAs much as the SMTP server knows, this is a blackbox that will acknowledge having delivered, but how it did is left to the MDA.\nConclusion # I wrote this because many people have had questions regarding custom MDA, how to write one and what they do.\nI might have missed stuff, if you have questions feel free to ask and I will expand this article.\n","date":"29 December 2020","permalink":"/posts/2020-12-29/writing-a-custom-mail-delivery-agent/","section":"Posts","summary":"TL;DR: In this article, I explain what is an MDA and how to write a custom one from scratch using only shell scripting. What is a Mail Delivery Agent (MDA) ?","title":"Writing a custom Mail Delivery Agent"},{"content":" TL;DR: Crafted the OpenSMTPD 6.8.0p1 release Fixed several bugs in the way Proposed a few OpenSMTPD improvements to OpenBSD Working on my OpenSMTPD book Let\u0026rsquo;s start with some LoFi # Relax.\nAbout sponsorship # First of all, I must say how grateful and happy I am that people value my work enough to trust and sponsor me, I appreciate it a lot and it is a huge motivation booster.\nI initially thought that after a few months I might have about 10 sponsors, but here we are after a year or so with over 50 sponsors !\nPlease let me know what you want me to work on, I may or may not do it (based on my interest and skills) but at least I\u0026rsquo;ll have an idea how to attract more sponsors as I\u0026rsquo;m clueless.\nThank you all and hopefully when December 2021 hits, I can give a three-digits number and spend more time on coding and writing for a hobby :-)\nPreparing the OpenSMTPD 6.8.0p1 release # I spent a lot of time preparing the OpenSMTPD 6.8.0p1 release that SHOULD have been released in November shortly after OpenBSD 6.8 was out, but which I couldn\u0026rsquo;t find time to work on as I was contracted for some private work.\nAs I prepared the release, I had to fix several issues, some specific to the portable version and some that have been fixed upstream at OpenBSD.\nThe 6.8.0p1 release was published today and contains the fixes for the issues mentioned below.\nFixed macOS build # The build on macOS was broken due to a missing OpenSSL include.\nA user submitted a pull request that fixed the build but another user reported that this was not enough for older macOS that lack the clock_gettime() call which we use in the profiling code.\nI\u0026rsquo;m not familiar enough (at all) with macOS to figure out which versions are \u0026ldquo;old\u0026rdquo; but this seems to fall into the \u0026ldquo;legacy\u0026rdquo; bucket and as far as the latest releases are concerned there is no known problems.\nI don\u0026rsquo;t intend to support all legacy systems out there but since fixing the clock_gettime() issue might be beneficial to portability outside of macOS, I started tackling the issue and am almost done.\nFixed get_progname() bug # A bug was introduced over a release ago which caused OpenSMTPD to improperly log the process name in maillog on some systems.\nIt was weird because the process names were correctly reported in the ps output, they just weren\u0026rsquo;t in the logs.\nI traced it back to a broken feature detection in configure.ac leading to the setprotctitle() workaround not working correctly.\nIssue was fixed and committed to the portable repository.\nImplemented an ECDSA engine for OpenSSL (sponsored development) # When OpenSMTPD 6.6.0 was released a year ago, it brought support for server-side ECDSA certificates but with the caveat that it only worked with LibreSSL. OpenSMTPD would build with OpenSSL but disable server-side ECDSA support.\nPeople assumed that this was done because I wanted to push LibreSSL as otherwise I could have just added a few #ifdef and handled both, but this was of course not the reason.\nAs I wrote in June 2019, the crypto operations in OpenSMTPD are done in a special way compared to what\u0026rsquo;s commonly seen. As a result, adding support to ECDSA was not just about slapping a few #ifdef and calling the proper function but required writing a custom crypto engine which, unluckily for me, uses an API that diverged between LibreSSL and OpenSSL.\nI didn\u0026rsquo;t have interest in doing that work because it\u0026rsquo;s not very pleasant, but people kept asking for it and a user contacted me in private to ask if I was willing to accept a sponsored development to do that work.\nI accepted and worked on it for a few days until I could start OpenSMTPD on an Ubuntu with OpenSSL and use ECDSA certificates. A few people tested the diff and reported success so it was committed to the portable repository.\nI\u0026rsquo;d like to thank again that user who wishes to remain anonymous, this sponsored development helped me replace my laptop with a new MacBook Air which is lovely, and the result benefits all users of OpenSMTPD portable.\nPlugged multiple memory leaks # As I requested feedback on the release candidate for OpenSMTPD 6.8.0p1, another user reported a memory leak which sent me on a big hunt.\nThanks to a top output I knew it was the lookup process\u0026hellip; but that process covers several scopes as it is in charge of DNS lookups, table lookups and passing information back and forth between an SMTP session and filters. I didn\u0026rsquo;t have an immediate intuition where the leak could come from as is often the case with other processes.\nI was suspicious of the filters layer since filters are still somewhat experimental and recent, however there are very few runtime allocations there and I could not find any unbalanced allocation except for a minor one that could not explain a noticeable leak. This was confirmed when the user applied the diff and saw no improvement whatsoever.\nThen I started being suspicious of DNS lookups and ended up diving into the resolver.c interface to asr, the asynchronous resolver. This led to another small leak found in OpenSMTPD and another one in asr itself prompting a diff from Eric Faurot in OpenBSD\u0026rsquo;s libc. This was slightly more noticeable than the previous one but still not very significant, it was barely visible on my own MX that had been running for a year without a restart.\nFinally, I had a eureka moment and convinced myself to look at the handling of regex lookup in tables. It turns out that a regfree() call was missing which caused a regex_t to leak for each regex lookup. Depending on the OpenSMTPD configuration, the use of regex, the size of tables used by regex and the filters in place, the leak could grow from fully inexistant to very significant.\nI submitted a diff to OpenBSD that was committed today by martijn@.\nTracked and found a possible crash in the filters state machine # As I was tracking the memory leak above, I came up by accident with a script that caused the OpenSMTPD instance on my laptop to crash\u0026hellip; just once. I tried to reproduce multiple times without success and decided to stop tracking the leak and focus on finding what caused the crash as it was more concerning.\nI spent hours adapting my script to play different kinds of sessions with varying number of recipients, concurrency levels and amounts of data sent. Eventually, I managed to get a kill script that would reliably crash my instance after a few seconds of running. It highly suggested a race condition.\nI tested the script on various instances with different configurations and realised that this did not affect all OpenSMTPD setups, but required a specific configuration and a specific client pattern to trigger. I could not crash a default install or my main MX which makes use of a lot of features, including filters, but I could easily crash my submission MX.\nThe bug was tracked for two days and it ended up being a logic issue in the filters state machine which could cause the io channel between the SMTP engine and the filters layer to be released while filters were still processing a request. Upon return from a filter, the channel would be expected to be writeable but it would in turn be NULL.\nThis led to a fix being committed to OpenBSD-current and an errata published today.\nWork on OpenSMTPD post-release # With the 6.8.0p1 release out of the way, I worked on a few diffs for future releases.\nSuggested the rename of OpenSMTPD processes (diff sent upstream) # Many years ago, OpenSMTPD had dedicated processes for the MDA and the MTA processes. OpenBSD hackers requested that I factor things a bit and merge these processes because they were both unprivileged and chrooted to /var/empty, making it pointless to over-isolate them.\nWe couldn\u0026rsquo;t find a proper name for the new process in charge of both MDA and MTA, but since an OpenBSD hacker had repeatedly requested that I offer him a pony I decided to temporarily name the process pony express (if you don\u0026rsquo;t understand why it\u0026rsquo;s funny, I can\u0026rsquo;t help). Later, the joke was improved by another developer when he named the new privsep crypto agent klondike.\nAs years passed, I considered renaming multiple times but could not convince myself the alternative names were better, and when I found better names they would not convince others. When a user said that these names were useless and unprofessional, I decided to let the joke run longer.\nYears later, I think the joke has been utilized to its fullest and is now bugging me for both technical (a space in the name of a process is annoying) and non-technical reasons.\nWhen the configuration file changed a while back to split rules into action and match sets, the concept of dispatchers was introduced in the code: an envelope is matched and based on the action, it is scheduled to a local or a remote dispatcher.\nI sent a diff to OpenBSD suggesting that the pony express process be renamed dispatcher, and that the klondike process be renamed crypto. I don\u0026rsquo;t know if/when this will be committed but the feedback was positive.\nSuggested explicitly requesting forward files (diff sent upstream) # Whenever a local dispatcher is used (actions mbox, maildir, mda or lmtp), OpenSMTPD will unconditionally look for a ~/.forward file in the recipients home directory. This is a mechanism that today can\u0026rsquo;t be turned off by an admin.\nI sent a diff to OpenBSD suggesting the introduction of a forward-file option.\nThe idea is to have OpenSMTPD consider it must NOT try to process ~/.forward files unless the admin explicitly requests it by adding the forward-file keyword to a local action. In the example below, I configure two domains with maildir, one not supporting forward files and the other supporting it:\naction \u0026#34;local_users_1\u0026#34; maildir action \u0026#34;local_users_2\u0026#34; maildir forward-file match from any for domain \u0026#34;foobar.org\u0026#34; action \u0026#34;local_users_1\u0026#34; match from any for domain \u0026#34;barbaz.org\u0026#34; action \u0026#34;local_users_2\u0026#34; The benefit is that this reduces the attack surface on setups where ~/.forward files are not desired. Processing these files require an indirection through the parent process to open the file on behalf of the daemon, but if we know we don\u0026rsquo;t want to process them the indirection can be bypassed.\nI don\u0026rsquo;t know if/when this will be committed.\nSuggested explicitly requesting custom MDA execution in forward files (diff sent upstream) # Built on top of the previous diff, I suggested the introduction of the allow-exec option for forward-file.\nThe ~/.forward files allow users to redirect their mails to another e-mail address, local or not, but they can also be used to plug a custom mail delivery agent such as fdm or procmail.\nThis is a very nice and interesting feature but it is also a vector of attack as if any bug allowed an attacker to write that file, including a bug unrelated to OpenSMTPD, it would allow them to execute commands with the recipient privileges.\nWe can\u0026rsquo;t really remove the exec feature of ~/.forward files because being able to execute a custom MDA is a very important feature. However, there are setups were you want to support users redirecting their mails elsewhere but not allow them to run their own custom MDA.\nMy proposal was to disallow execution of a custom MDA in ~/.forward files unless the admin explicitly allows it. In the example below, I configure two domains that support forward files but one does not allow execution of a custom MDA while the other does:\naction \u0026#34;local_users_1\u0026#34; maildir forward-file action \u0026#34;local_users_2\u0026#34; maildir forward-file allow-exec match from any for domain \u0026#34;foobar.org\u0026#34; action \u0026#34;local_users_1\u0026#34; match from any for domain \u0026#34;barbaz.org\u0026#34; action \u0026#34;local_users_2\u0026#34; With my diff, if a custom MDA command is placed in a ~/.forward file then the SMTP session temporarily rejects the session until the recipient fixes the file.\nI don\u0026rsquo;t know if/when this will be committed.\nSuggested explicitly requesting custom MDA execution in aliases and virtual mappings (diff sent upstream) # Built on top of the previous diff, I suggested the introduction of the allow-exec option for alias and virtual.\nHistorically, plugging a mailing list or similar tools was done by creating an alias to a command:\nlists: |/usr/local/bin/mlmmj ... This is however a very bad practice that should be discouraged because attaching a command to an alias means there\u0026rsquo;s no recipient user to run the command as, and they end up executed as the daemon\u0026rsquo;s user. In every single use-case I ever met, creating dedicated users and using their ~/.forward files to execute the commands achieves the same result while properly isolating the commands from the daemon.\nKilling support for execution of a command in aliases is a battle that I wanted to fight but that will meet a lot of resistance, so my proposal was to do the same as above for forward files and disable execution by default unless an admin explicitly requests it.\nIn the example below, I configure two domains with different aliases mappings, one not supporting execution of commands and the other supporting it:\naction \u0026#34;local_users_1\u0026#34; maildir alias \u0026lt;aliases_1\u0026gt; action \u0026#34;local_users_2\u0026#34; maildir alias \u0026lt;aliases_2\u0026gt; allow-exec match from any for domain \u0026#34;foobar.org\u0026#34; action \u0026#34;local_users_1\u0026#34; match from any for domain \u0026#34;barbaz.org\u0026#34; action \u0026#34;local_users_2\u0026#34; Note that virtual uses the same expansion loop as alias so it works the same:\naction \u0026#34;local_users_1\u0026#34; maildir virtual \u0026lt;aliases_1\u0026gt; action \u0026#34;local_users_2\u0026#34; maildir virtual \u0026lt;aliases_2\u0026gt; allow-exec match from any for domain \u0026#34;foobar.org\u0026#34; action \u0026#34;local_users_1\u0026#34; match from any for domain \u0026#34;barbaz.org\u0026#34; action \u0026#34;local_users_2\u0026#34; Similarly to the behavior of allow-exec with ~/.forward files, if a command is found on an alias that\u0026rsquo;s not allowed to execute a custom command then the SMTP session temporarily rejects the recipient until the alias is fixed.\nI don\u0026rsquo;t know if/when this will be committed.\nPrivileged-process safety net for allow-exec # I have a diff which is done but that I haven\u0026rsquo;t submitted as it depends on the diffs above and is pointless to submit until (if ?) they are committed.\nWhat it does is add a check in forkmda() so that before executing a custom MDA command from an envelope, it checks again that the dispatcher was configured to allow custom MDA commands through allow-exec.\nExcept for configuration changes, this should NEVER happen because a dispatcher that doesn\u0026rsquo;t allow-exec would reject the recipient during the SMTP session when it expands to a command. However, if an attacker managed to find a bug which allows injection of a custom MDA command into an envelope, then this safety net would prevent execution of the command by spotting the fishy situation.\nOn setups without allow-exec this would have prevented one of the security issues that Qualys disclosed earlier this year.\nI\u0026rsquo;m trying to find a similar way to provide a safety net for setups with allow-exec but it\u0026rsquo;s much harder as they explicitly allow custom MDA commands, and it\u0026rsquo;s not possible for OpenSMTPD to know from a custom MDA command if it was legitimate or not.\nReleased new version of filter-rspamd # I did a minor release of filter-rspamd to change the versioning format and make it easier to package on OpenBSD.\nNothing fancy.\nStarted releasing chapters from the OpenSMTPD book # As I\u0026rsquo;ve mentioned in the past, I work on an OpenSMTPD book which is 120ish pages as of today.\nMy plan to release it when done does not work because I keep rewriting the same chapters over and over until I\u0026rsquo;m happy\u0026hellip; and I never am. Meanwhile the project moves forward so things get outdated and I need to rewrite them again.\nI decided to take a more radical approach and work on it publicly on Github. This way, I don\u0026rsquo;t have to worry about people seeing an imperfect phrasing or pages not rendering the way I want in PDF.\nYou\u0026rsquo;ll find the repository here.\nWhat\u0026rsquo;s next ? # I have great plans for 2021 but for now I don\u0026rsquo;t want to disclose them.\nHave a merry Xmas, a happy new year, and see you in January !\n","date":"24 December 2020","permalink":"/posts/2020-12-24/december-2020-opensmtpd-6.8.0p1-released-fixed-several-bugs-proposed-several-diffs-book-is-on-github/","section":"Posts","summary":"TL;DR: Crafted the OpenSMTPD 6.8.0p1 release Fixed several bugs in the way Proposed a few OpenSMTPD improvements to OpenBSD Working on my OpenSMTPD book Let\u0026rsquo;s start with some LoFi # Relax.","title":"December 2020: OpenSMTPD 6.8.0p1 released, fixed several bugs, proposed several diffs, book is on Github"},{"content":" TL;DR: Haven\u0026rsquo;t posted since July but I wasn\u0026rsquo;t slacking, got temporarily context-switched out of opensource stuff. I tanked a year in work psychology # As some of you know, I study work psychology and have been attending evening classes for the last four years, sometimes up to 9 hours / week and more often between 3 and 6 hours / week. I\u0026rsquo;m pretty invested as you can tell, it\u0026rsquo;s a lot of time that I could spend in bed but that I spend in class instead :-)\nLast school year, I was supposed to do a pair work field intervention which would then be used as the material for my upcoming master thesis dissertation. This intervention was meant to take place physically with groups of employees and span over two or three months. We had started interviewing with a company interested in our intervention, but the lockdown put this to a hold and was renewed over and over until the school year was over.\nUnlike other courses where it was possible to substitute the evaluation method, like in psychometric testing course where we ended up analyzing a case study rather than having someone undergo interviews and tests, there\u0026rsquo;s no way to substitute this intervention without compromising the master thesis material so\u0026hellip; many of us plain tanked the course.\nBecause of the extraordinary situation, we were told that if we could begin an intervention in September and be done by end of November, the school year could be salvaged and we could integrate the thesis preparation course in December, losing just a trimester instead of a year. Since we were close to having a field, there was a slight hope that we could pull this out and I made this my priority.\nUnderstandably, the company we were in discussions with had to shift its priorities post-lockdown and summer and ended up declining the intervention, so I tanked for real and began taking the same course again for a year\u0026hellip;\nA year which begins with another lockdown.\nI opened my hypnosis office # I started learning hypnosis sometime in 2015 and eventually ended up attending multiple trainings and certifications.\nAfter a year, I began doing sessions for volunteers with specific problems so that I could practice these. I only requested that they would donate an amount of their choice to a charity of their choice, and that they would give me a honest feedback about what worked and not. It lasted a while but then I gradually stopped asking for volunteers because it was no longer necessary in my opinion.\nTwo years ago, I started getting enough people referred to me that it made sense to create a legal entity. I began working as a mobile hypnotherapy practitioner during my time off work, on evenings and week-ends. When I switched to a part-time schedule, I often plugged one or two sessions per day so that it could help me recover a bit from the financial loss.\nThis was nice and all but after two years of driving North and South of the city, sometimes rushing from a client to another because a session ended late, I started considering opening an office so it would be more confortable for me. I now receive people in two wellness centers, on Thursday and Friday.\nSince I tanked a year in work psychology, I thought I might as well make another project go through :-)\nI won\u0026rsquo;t be talking much about that activity here as it\u0026rsquo;s so remote from tech, however feel free to checkout my dedicated website, hypno.cat, and feel free to ask me anything on twitter as I\u0026rsquo;m at least as passionate about this topic than about tech.\nI brought poolp.org back home, literally # I have not had a static IP in a long time because the choice of ISP was limited in this new neighborhood, but recently an option became available allowing me to have FTTH with a static IP, IPv6 and outgoing SMTP trafic.\nI\u0026rsquo;ve been renting a ton of VPS at vultr (affiliation link) and am happy with them, but I need to reduce my expenses and they cost me around 100 EUR / month. I decided to stop hosting people for free as I\u0026rsquo;ve been doing for the last 20 years or so, and cut on the number of VPS because poolp.org doesn\u0026rsquo;t have to be such a high-availability service. I can live with a few downtimes myself if any.\nI got myself a refurbished machine from backmarket (affiliation link), installed the latest OpenBSD on it and started moving everything there so I could progressively shutdown a few VPS.\nYou\u0026rsquo;re reading this blog post from a machine hosted in my appartment, plugged right behind a 10GB theoretical connection, though the device only has a 1GB network interface :-)\nI automated poolp.org publishing in the Github CI # The posts on this blog are generated from markdown files that are available in a Github repository.\nI initially didn\u0026rsquo;t automate publishing so the repository was just used to version the posts. The action of publishing was manual and this explains why sometimes people would submit pull requests to fix a typo and the changes would not show up before days: I could merge the pull request but until I booted the proper laptop to rebuild the website and rsync the directory to the server, the change would not be visible.\nIt was annoying because it meant I could not fix anything quickly, I always had to wait until I was in front of my OpenBSD laptop which is the only one allowed to auth to my server. In one instance, I messed up and published an uncommitted draft while doing an rsync from my laptop to fix a typo, I realized because the unfinished article had been linked from hackernews before I had a chance to proofread and I could not unlink it right away.\nSo I took some time to rework this and now the blog is generated from the repository itself, I no longer do rsyncs all over the place. If you submit a pull request to fix a typo or improve an article ,the blog will be updated shortly after I merge it.\nThis is technically the first article ever that I will not rsync.\nNot that you care, but I also did the same for hypno.cat while I was at it ;-)\nBegan preparing the OpenSMTPD portable release # I have brought all upstream commits from OpenBSD to the portable version of OpenSMTPD, fixed a couple shortcomings in the way and I have a few issues I need to tackle before I can craft the release.\nI should be done in the next couple weeks so the OpenSMTPD portable release can happen in early December.\nWhat\u0026rsquo;s next ? # Not finding the time to produce useful work to write about since July has been irritating and getting back on track was not effortless. I wrote this, even though it\u0026rsquo;s not technical, to break the cycle of silence and pick up where I left.\nThere are several pull requests pending on many of my repositories, I merged some and need to review others.\nI intend to write some tutorials too.\nSee you in December !\n","date":"22 November 2020","permalink":"/posts/2020-11-22/november-2020-i-wasnt-slacking-but-no-opensource-work/","section":"Posts","summary":"TL;DR: Haven\u0026rsquo;t posted since July but I wasn\u0026rsquo;t slacking, got temporarily context-switched out of opensource stuff. I tanked a year in work psychology # As some of you know, I study work psychology and have been attending evening classes for the last four years, sometimes up to 9 hours / week and more often between 3 and 6 hours / week.","title":"November 2020: I wasn't slacking but no opensource work"},{"content":" TL;DR: worked on my webmail, on a custom MDA and on a python framework for API development. Worked on my webmail # Again, I\u0026rsquo;d like to emphasize that this is something that\u0026rsquo;s going to span over many months so\u0026hellip; don\u0026rsquo;t hold your breath.\nLast month I wrote that the webmail didn\u0026rsquo;t allow reading messages yet as the /fetch backend operation only fetched headers. This was because I focused on navigation and the fetching of messages in IMAP is a bit tricky to do right so I wanted to be focused. A naive approach that works is to simply get the full message and parse it in the client, something that I\u0026rsquo;ve done multiple time in the past when performances weren\u0026rsquo;t a big deal. With a webmail, this approach does not work: the time it takes to open a page with the mail content is unbearable when there\u0026rsquo;s a big file attached to it, it\u0026rsquo;s just not acceptable.\nLuckily, imap supports fetching a BODYSTRUCTURE which results in a response describing the MIME structure of the message. With this, a client can inspect the structure and determine which parts it wants to fetch them specifically. This is a nice solution, at the cost of an additional fetch operation, and in the case of a webmail it is exactly what I need to avoid passing huge amount of unnecessary data between the imap server, the backend and the frontend. This took me a while to complete because parsing the BODYSTRUCTURE is nightmarish: it describes a MIME structure \u0026hellip; and MIME supports nesting \u0026hellip; and the imap protocol is \u0026hellip; how do I put it \u0026hellip; not the most beautiful thing.\nWith this done, I could fetch multipart messages and allows displaying a specific part, without the performance overhead of a full fetch:\nThe above was essentially backend work with just a bit of frontend work, but the remaining of my work on the webmail was essentially on the frontend side:\nI reworked a bit the navigation to simplify, splitting between \u0026ldquo;system folders\u0026rdquo; and \u0026ldquo;custom\u0026rdquo; folders, but also allowing to click on a progressbar to skip to a specific portion of a large folder, getting rid of \u0026ldquo;pagination\u0026rdquo; links. In addition, I reworked the folder listing so that the frontend now adapts to the screen size: it determines the window height and fetches just enough entries to fill the window without generating a scrollbar. The scrollbar is unnecessary because pagination uses either the pageup/pagedown button, or is automatic when using arrow up/down while on the first or last entry of the listing. If you use mutt, you\u0026rsquo;ll be in a very familiar ground.\nA bit I\u0026rsquo;d like to improve is the display of subfolders which are currently displayed inlined in the custom folder tab.\nI also worked on mail composing, because being able to read a mail is nice but being able to answer is even better.\nI\u0026rsquo;m not a big fan of HTML emails for a variety of reasons, some of which will be explained in an upcoming article, but I\u0026rsquo;d be deluded if I thought HTML emails were going anywhere. Still, I want to avoid them as much as possible, so I decided to make the webmail allow composing HTML messages while emitting them in plain by default, if possible.\nBasically, the webmail will display a compose page which allows to customize the appearance of text, however it will:\ndefault to generating a text/plain part if no customization was used generate a text/plain alternative part stripped of customization if any was used Of course, the idea is not just to compose an email but also to send it. I wrote the backend bits to craft a MIME-message from the frontend submission and send it, which allowed me to mail myself:\nWork is still needed to allow attachments when composing a message, but there is not technical difficulty there.\nAt this point, I know the backend API is about right and I can focus on improving the frontend to make it more usable.\nWorked on a custom MDA # Last month, I talked about my custom MDA and script script to perform folder pinning. I\u0026rsquo;m happy with the idea but not so much with the implementation so I decided to revisit.\nThe issue is that folder pinning is far from being the only thing I want to do with incoming mails at delivery time, and shoving everything in the mda executable is not ideal. I rewrote the MDA to have it handle delivery only and call an API to determine where it should do it, this let me play with a ton of ideas on a custom API server without tweaking the working MDA. At the end of the day, I had incoming mails processed by various text analyzers, attachments automatically extracted and put in an s3 backing store, and mails indexed for fast lookups.\nI will not expand much on how I did this as I think it makes a nice topic for a dedicated article on custom MDA, and fun stuff you can easily do with them to provide some awesome features on your mail setup.\nWorked on a python framework for my API/web development # I often write APIs, either for work of for my own needs, and I usually use bottlepy which is a micro-framework for python. Along the years, I worked on a lot of projects using that framework, including professional projects with tons of constraints.\nEric and I built a lot of small helpers around it to make out job easier, and we kept copying these in each and every project, improving them as needed. I thought it would be a nice idea to package this so that I could easily bootstrap a new project without copying code around. So I spent a few days extracting the code from an existing project, cleaning it up, and packaging it so it could easily be used to start a new project.\nThe work isn\u0026rsquo;t done yet, but the idea is that you end up with an executable. This executable is controlled by a configuration file which lets you describe what interface and port the API is running on, as well as what package implements its endpoints. Then, all you need to do is to create a small package that contains the endpoints and implementations, something like:\nimport openbar.routes @openbar.routes.register(\u0026#34;0.1\u0026#34;, \u0026#34;backend\u0026#34;) def setup(app): app.get(\u0026#34;/\u0026#34;)(get_root) app.post(\u0026#34;/\u0026#34;)(post_foobar) def get_root(): return {\u0026#34;foobar\u0026#34;: \u0026#34;barbaz\u0026#34;} def post_foobar(): return {\u0026#34;barbaz\u0026#34;: \u0026#34;bazqux\u0026#34;} What\u0026rsquo;s interesting here is not that this is short, because you\u0026rsquo;d get about the same if you just dropped bottlepy in the same folder, but it\u0026rsquo;s that you don\u0026rsquo;t need to do all the plumbing to daemonize and drop privileges, log to syslog, etc\u0026hellip; and that it comes with a lot of helpers.\nI will also not expand much on this as I\u0026rsquo;m almost done with the packaging and I might as well demo it rather than spoil it. There is nothing super impressive, no big innovation, but just a wrapping of common stuff in interfaces that makes them pleasant to deal with.\nThe webmail backend is an obvious candidate for this, but so is the MDA API server. I keep needing this over and over, it made sense stepping into this project to cut time on the others.\nno OpenSMTPD work ? # Not much this month.\nI spent a day playing with a queue-procexec but the code was throw-away, only meant to help me detect if something was utterly broken with the idea.\nI also worked a bit on the go-opensmtpd package and made progress on the filter API, the goal being to allow writing a filter without worrying about the low-level protocol. Instead of reading the protocol, parsing it and determining what should be called or not, a filter can register callbacks for specific events and implement these callbacks.\nI will finish this in September and adapt my filter-rspamd accordingly.\nWhat\u0026rsquo;s next ? # I will be taking a break from all projects in August to enjoy some long-awaited vacations, so\u0026hellip; don\u0026rsquo;t expect any code to be written before September.\nHave a nice month !\n","date":"31 July 2020","permalink":"/posts/2020-07-31/july-2020-webmail-custom-mda-and-python-framework-work/","section":"Posts","summary":"TL;DR: worked on my webmail, on a custom MDA and on a python framework for API development. Worked on my webmail # Again, I\u0026rsquo;d like to emphasize that this is something that\u0026rsquo;s going to span over many months so\u0026hellip; don\u0026rsquo;t hold your breath.","title":"July 2020: webmail, custom MDA and python framework work"},{"content":" TL;DR: Reworked my infrastructure at poolp.org, implemented folder pinning, worked on my webmail. poolp.org infrastructure rework # I decided to move my servers from online.net to vultr.com (affiliation link).\nI have been at online.net for hosting since 2007 and have always been happy with their services, however OpenBSD has never been supported there and I always had to use custom install procedures and hope things go well.\nNowadays, I\u0026rsquo;m testing a lot of stuff and I create a lot of short-lived machines so going through these procedures each time is painful. I\u0026rsquo;ve been happy with vultr.com for a little while now so I decided to just move everything there, only keeping a backup server at online.net.\nThis forced me to rework my setup but also allowed me to improve it as the cost of running multiple VPS is considerably lower than running multiple dedicated servers, I could split multiple services into their dedicated VPS and tidy things a bit. Instead of three servers, I now operate 15 VPS. As time allows, I\u0026rsquo;ll start documenting on this blog the different parts and their configuration.\nIt was a bit painful but everything is migrated at this point and the old servers can be shutdown.\nThe idea behind this infrastructure change is also to start considering the services that I might provide my sponsors in the future :-)\nMail classification heuristics and folder pinning # A downside of mail commonly shared by users is the problem of mail classification and inbox overload.\nHistorically mails were dropped in an mbox which is a mail format where all mails are appended to a common mailbox. This meant that all mails were considered equivalent in priority and were just sorted in delivery time order. The user had to navigate through them all mixed together and decide which ones they wanted to deal with.\nThe mbox format was mostly deprecated by the Maildir format which comes with many advantages and uses a directory structure, rather than a unique mailbox file. The directory structure allows mail folders to exist as sub-directories making it possible to create dedicated folders, and use post-processing tools to classify mails in these folders. For example, at poolp.org the fdm utility was installed and allowed users to create classification rules which, in my case, were used to classify mails from OpenBSD mailing lists into their own folders and out of my Inbox. This is extra powerful and flexible but not very generic as the rules set by a user may not work for another.\nThe problem is that even if you can create folders and create rules to put mails in them, an Inbox essentially replicates the historical behavior of mbox and ends up accumulating all kinds of mails. If we keep spam and mailing lists aside for now, the Inbox is still where all commercial mails, social network notifications, password recovery, undelivered mails notifications and other land until you create a rule matching each of them.\nGmail, despite an interface I dislike, tackled the issue in a very sound way with their folder tabs: Primary, Social, Promotions, Updates and Forums. The Primary folder is the Inbox, Social is where all social networks notifications land, Promotions is where bulk mails land, Updates is where transactional mails land, and Forums is where mailing lists land.\nI wanted to replicate this on my server and it turned out to be very simple as this classification can be done with a few heuristics. I configured dovecot to expose these additionals folders by default, then configured fdm to apply a set of rules that would categorize mail in these folders:\naction \u0026#34;inbox\u0026#34; maildir \u0026#34;%h/\u0026#34; action \u0026#34;junk\u0026#34; maildir \u0026#34;%h/.Junk\u0026#34; action \u0026#34;social\u0026#34; maildir \u0026#34;%h/.Social\u0026#34; action \u0026#34;bulk\u0026#34; maildir \u0026#34;%h/.Bulk\u0026#34; action \u0026#34;transactional\u0026#34; maildir \u0026#34;%h/.Transactional\u0026#34; action \u0026#34;lists\u0026#34; maildir \u0026#34;%h/.Lists\u0026#34; account \u0026#34;mda\u0026#34; stdin match \u0026#34;^X-Spam: [Yy]es\u0026#34; in headers action \u0026#34;junk\u0026#34; match \u0026#34;^X-Facebook-Notify: .*\u0026#34; in headers action \u0026#34;social\u0026#34; match \u0026#34;^Precedence: list\u0026#34; in headers action \u0026#34;lists\u0026#34; [...] match \u0026#34;^Precedence: bulk\u0026#34; in headers action \u0026#34;bulk\u0026#34; match \u0026#34;^From: .*no-?reply.*\u0026#34; in headers action \u0026#34;transactional\u0026#34; match all action \u0026#34;inbox\u0026#34; Every time a mail that I would expect to land in a subfolder hit my Inbox, I looked at its headers and checked what could be used as a discriminator to create a new rule or improve an existing one. This considerably improved the state of my Inbox as most mails that didn\u0026rsquo;t expect an answer landed in a subfolder, while my Inbox contained mostly mails from people expecting an answer.\nThis worked wonders for most mails but then there were some that didn\u0026rsquo;t get proper classification, because sender didn\u0026rsquo;t do things correctly and made it hard to classify: these mails kept hitting the Inbox and I kept moving them out of the way without being able to write a proper rule. Also, as someone subscribed to multiple mailing-lists, having all mailing-lists mails in a single folder was not comfortable. It occured to me that what was needed on top of these heuristics was folder pinning: if a user receives mail from a sender and moves it to a folder, then most likely future mails from that sender should be moved to that folder too. This would solve the problem of senders that don\u0026rsquo;t fit generic classification, but also the problem of mails you would like classified in custom folders.\nI decided to implement this using three components:\na mail delivery agent, similar to fdm, but with extra logic that would take into account the folder pinning; a key-value store to keep track of folder pinning state; a sieve script for learning new pinning states from mail moves in IMAP; Basically, when a mail arrives at poolp.org the mail delivery agent extracts the sender information, checks in the key-value store if the recipient has a pinning rule for the sender and proceeds to delivery. If there\u0026rsquo;s a pinning rule then that folder is used, otherwise heuristics are used to find the most likely folder the mail should land in.\nIf a user is not satisfied with where a mail landed, moving it to a new folder causes dovecot to pass the mail to a sieve script which will extract the sender information, and push a new rule automatically to the key-value store so that next time the mail delivery agent will know where that sender should land.\nThe sender extraction is done on the Return-Path address, not the From header, which ensures that someone contacting me directly or through a mailing-list is classified in the proper folder, and the fact it is done at the protocol level ensures that this is works regardless of the client used to read mail.\nThis works very nicely and is particularly pleasant as you quickly forget that it is here, mail just lands where you most expect it. The way I implemented it makes it possible to share in a generic way so that everyone can benefit from this feature, regardless of which MTA they are using, however the code is not ready yet as I wrote it as an experiment for my own setup with hardcoded paths and such. I\u0026rsquo;ll keep working on it and, when it is ready, I\u0026rsquo;ll publish my mail delivery agent, the sieve scripts and the dovecot configuration in a single repository.\nWebmail work # As I mentionned in April, I\u0026rsquo;m currently working on a webmail of my own. Most of the work I did this month was around that project which is evolving at a good pace considering that I had no experience in React, and that it is the end of my scholar year which comes with assignments consuming all of my spare time.\nThe webmail comes in two parts:\nthe backend; the frontend; Backend work # The backend is just an API that exposes REST endpoints to the underlying IMAP server. It doesn\u0026rsquo;t just map a route to an IMAP command but does it in a way that\u0026rsquo;s optimized for a webmail, for instance /select is not just selecting a mailbox but also an offset and a limit to lookup envelopes for that mailbox, avoiding a round-trip for a /fetch. I wrote a version of the backend in Python and another in Go but eventually decided to stick to Python, the email parsing library is much more featureful and the performances are just there.\nIt currently implements:\n/login, to authenticate against the IMAP server /noop, which is used by the webmail to keep a connection alive /select, to select a folder and fetch up to limit envelopes from offset /fetch, to fetch a specific message (currently only headers) /move, to move a set of messages from a folder to another /expunge, to delete a set of messages permanently This backend can be used as the building block of any mail client so I will publish it as a standalone project, if people are interested in building their webmails they can then focus on the interface and not the lower-level bits.\nFrontend work # The frontend is a React client to that API, plain and simple. I only focused on the folder view at the moment so it is not usable yet to read mails, but it is possible to authenticate, navigate between folders, select and delete or junk mails, etc\u0026hellip;\nRemember that this is a work in progress at an early stage, not the final interface, but it allowed me to test the behaviors I wanted from the webmail and I\u0026rsquo;m happy with the result.\nYou\u0026rsquo;ll just have to be patient until I put up a live demo so you can test for yourself\u0026hellip;\nAt this point, all I can say is that it is progressing and that some aspects of it are impressive. It is very fast, and this is despite me testing it from my laptop on a remote IMAP server with no caching and no pre-fetching, no optimizations whatsoever.\nPatience !\nWhat\u0026rsquo;s next ? # My main focus in July will be the webmail, the sooner I get a PoC out the better.\nI have some ideas to improve mail classification further in more \u0026ldquo;creative\u0026rdquo; ways, I\u0026rsquo;ll continue exploring how to make mail management nicer.\nI have some pending work for OpenSMTPD, both on the filter side and on the queue side.\n","date":"27 June 2020","permalink":"/posts/2020-06-27/june-2020-poolp.org-folder-pinning-and-webmail-work/","section":"Posts","summary":"TL;DR: Reworked my infrastructure at poolp.org, implemented folder pinning, worked on my webmail. poolp.org infrastructure rework # I decided to move my servers from online.","title":"June 2020: poolp.org, folder pinning and webmail work"},{"content":" TL;DR: Worked on the OpenSMTPD 6.7 release; Did a lot of work on the new table API; Wrote several PoCs; WARNING:\nExamples of code and configuration that appear in this article are here to help illustrate and explain development stages of my work.\nThey are subject to changes and must not be considered as user documentation. By the time you\u0026rsquo;re reading this, they will likely no longer work or reflect reality.\nOpenSMTPD 6.7.1p1 release # On May 19, 2020 was released OpenBSD 6.7 which ships OpenSMTPD 6.7, the latest stable version of OpenSMTPD.\nI released the portable version of OpenSMTPD on the same day, followed a few days later by a minor update to fix a packaging issue spotted by Debian maintainer Ryan Kavanagh and a possible crash when relaying over IPv6 spotted post-release by Void Linux maintainer Leah Neukirchen.\nI did a lot of work on portable, extending the CI to new targets and such, I won\u0026rsquo;t talk about this work because this article contains a fair amount of more interesting topics.\nSome highlights in the 6.7.1p1 release # Before all, one of the most visible change for package maintainers is that libasr is no longer a dependency. It is shipped in the compat layer and built conditionally if the host system doesn\u0026rsquo;t have the library installed, this will solve a few headaches.\nThis release brings various improvements to the filtering protocol, not visible to users for the most part, but most notably three features deserve a special mention.\nFirst, the new bypass action which was already discussed here. It allows writing exclusion rules for certain sessions in builtin filters whereas, prior to that, all sessions connecting to a listener with filters would always have all filters applied to them:\nfilter trusted phase mail-from match src \u0026lt;trusted_sources\u0026gt; bypass filter no_rdns phase mail-from match !rdns reject \u0026#34;550 go away\u0026#34; filter no_fcrdns phase mail-from match !fcrdns \u0026#34;550 go away\u0026#34; listen on all filter { trusted, no_rdns, no_fcrdns } Then, the smtp-out reporting stream which is very similar to smtp-in but reports SMTP events for outgoing sessions. This allows writing filters that do all kind of accounting on outgoing trafic and, as will be shown in this article, can be used to make interesting filters.\nFinally, all built-in filters can now match authenticated sessions and specific authenticated users. It becomes possible to apply some filters to non-authenticated sessions only, or to avoid applying some filters to specific authenticated users.\nfilter auth_only phase mail-from match !auth reject \u0026#34;550 you must be authenticated to send mail\u0026#34; filter gilles_only phase mail-from match !auth gilles \u0026#34;550 only gilles is allowed to send mail\u0026#34; And because auth takes a table parameter, we can also do table-based filter matching using a credentials table:\ntable poolp_users { gilles = XXX, eric = XXX } filter poolp_only phase mail-from match !auth \u0026lt;poolp_users\u0026gt; reject \u0026#34;550 must be a poolp.org user\u0026#34; Outside of the filtering area, other improvements were made to smtpd.conf that makes it possible to simplify existing configurations as well as express configurations that were not possible before.\nThere are two notable improvements in my opinion:\nIn previous releases, it was possible to have rules match authenticated session as follows:\nmatch from any auth for any action \u0026#34;relay\u0026#34; but this was an all or nothing mechanism, either you wanted to match all authenticated sessions or none.\nThe mechanism was extended to support matching specific authenticated users so that you can write:\nmatch from any auth gilles for any action \u0026#34;relay_gilles\u0026#34; match from any auth eric for any action \u0026#34;relay_eric\u0026#34; Similarly to filters, the auth parameter is a table so that it is possible to pass a list of specific users and create rules targeting specific users:\nmatch from any auth { gilles, eric } for any action \u0026#34;relay_custom\u0026#34; match from any auth for any action \u0026#34;relay\u0026#34; This is very powerful as auth accepts credentials tables and allows setups serving multiple domains to have rules match their specific users, something that was not doable previously:\ntable users_poolp_org file:/etc/mail/users_poolp_org table users_opensmtpd_org file:/etc/mail/users_opensmtpd_org listen on $ip_poolp [...] auth \u0026lt;users_poolp_org\u0026gt; listen on $ip_opensmtpd [...] auth \u0026lt;users_opensmtpd_org\u0026gt; [...] match from any auth \u0026lt;users_poolp_org\u0026gt; for any action \u0026#34;relay_poolp\u0026#34; match from any auth \u0026lt;users_opensmtpd_org\u0026gt; for any action \u0026#34;relay_opensmtpd\u0026#34; Finally, a rework of the from and for logic in smtpd.conf was done to better express rules. Instead of assuming that from and for always attempted to match a source address and a destination, they now match the user intent:\nmatch from auth [...] match from mail-from gilles@poolp.org [...] match [...] for rcpt-to gilles@poolp.org This change allows expressing many configurations in a much more compact and user-friendly way:\nFor example, the following:\nmatch from any auth for domain poolp.org rcpt-to gilles@poolp.org action \u0026#34;deliver\u0026#34; becomes:\nmatch from auth for rcpt-to gilles@poolp.org action \u0026#34;deliver\u0026#34; They are both functionally equivalent but the change is semantic, the first rule requires that the envelope matches both from any and auth for origin and both for domain poolp.org and rcpt-to gilles@poolp.org for destination, whereas in the second rule it only requires that auth and rcpt-to gilles@poolp.org be matched regardless of the source address and destination domain. This semantic shift is backwards compatible so existing configuration are not impacted, they just have some rules that could be simplified in some cases.\nMost of the other work is either not visible to users, but feel free to browse the commit history for more as there\u0026rsquo;s been a LOT of work poured in the release polishing and cleaning stuff.\nfilter-prometheus # As I mentionned above, the new release comes with an smtp-out events reporting stream that makes it possible to write filters that are aware of outgoing trafic, so I decided to write a PoC filter that would highlight how this can be used in a useful way.\nPrometheus is an open-source monitoring system that\u0026rsquo;s fairly popular and that seems to have the favours of highly competent people at ${DAYJOB}. Since I need to get familiar with it at work, I thought I\u0026rsquo;d get my hands on it on my spare time too.\nBasically the idea is that you run an instance (or a cluster, but lets keep it simple) that will contact configured endpoints that expose various metrics. The metrics are then available in Prometheus through a query language, PromQL, and through graphs in a web user interface.\nI wrote a filter-prometheus which registers smtp-in and smtp-out events, creates some metrics based on them, and exposes these metrics on an HTTP endpoint.\nPrometheus is configured to hit that endpoint and\u0026hellip; voila, that\u0026rsquo;s all, they are available through prometheus.\nOn the configuration standpoint, this is straightforward. The filter itself doesn\u0026rsquo;t have configuration, it only supports a command line flag to specify which IP and port are used to exposed the metrics.\nPrometheus has a simple configuration:\nglobal: scrape_interval: 15s scrape_configs: - job_name: \u0026#34;prometheus\u0026#34; static_configs: - targets: [\u0026#34;localhost:9090\u0026#34;] - job_name: \u0026#34;OpenSMTPD: in.mailbrix.mx\u0026#34; static_configs: - targets: [\u0026#34;in.mailbrix.mx:31333\u0026#34;] OpenSMTPD has an even simpler configuration:\nfilter prometheus proc-exec \u0026#34;filter-prometheus -exporter 0.0.0.0:31333\u0026#34; # attach filter to a listener for smtp-in metrics listen on all filter prometheus # attach filter to a relay action for smtp-out metrics action \u0026#34;outgoing\u0026#34; relay filter prometheus [...] The filter is already available in a Github repository but note that this is a work in progress and that it was mostly done to play with prometheus and exporting metrics, not intended to be used for real as I don\u0026rsquo;t have any significant experience to design metrics properly at this point.\nIf you are familiar with prometheus and want to give me a hand in making this filter production-ready, I\u0026rsquo;ll be more than happy to make this happen with your help.\ntable-procexec # OK, let\u0026rsquo;s get to the hairy part now, but first a bit of a refresher.\nOpenSMTPD uses a mechanism called tables for all kinds of internal lookups. The table API support three operations, check, lookup and fetch. It uses these three operations to check if an envelope criteria is met (ie: does this user exists ?), lookup informations based on a specific key (ie: what are the aliases for root ?), or fetch a value from a set (ie: what\u0026rsquo;s a valid source IP to use for this outgoing session ?).\nIn addition to this, the table API supports various lookup services (ie: credentials, aliases, mailaddr, \u0026hellip;) which determine what kind of data the table is supposed to return. If you lookup a mailaddr in a table, the expected return value is an e-mail address that can be validated by the daemon. The three table operations are aware of this, so OpenSMTPD will for example \u0026ldquo;check if table has a mailaddr matching key gilles@poolp.org\u0026rdquo;.\nThere are multiple table backends available, both builtin to OpenSMTPD (memory, file and db) and third-party (sqlite, postgres, mysql, \u0026hellip;). Some backends, like table-passwd, only support specific lookup services (ie: credentials, userinfo) while others support all. As long as you have a data source upon which you can implement the three operations, you can implement a backend for some or all of the lookup services. The good part is that this happens outside of the daemon, so there\u0026rsquo;s no need for cooperation from OpenSMTPD, no spill of dependencies and such, table backends are fully independant standalone programs.\nThe API is very simple so in theory anyone can write a backend for their own custom use-cases, but in practice the communication with OpenSMTPD happens through the imsg API which limits the interface to C developers. Eric and I wrote the necessary bits a long time ago to abstract the details so that C developers only need to focus on implementing the operations and not deal with the IPC. It helped a lot but any backend doing something a bit tricky, ldap for example, still requires being comfortable with C programming and a sysadmin without that knowledge can\u0026rsquo;t really get anything done. A bit sad given that sysadmins are the primary users of this\u0026hellip;\nI later implemented a table-python backend which acted as a bridge. It allowed to implement the three operations in Python functions and have the backend call the functions, taking care of converting back and forth between the two languages. This made table-backends much more user-friendly but then came the Perl people, then the Lua people, then the Go people, \u0026hellip; and a bridge would have to be written and maintained for all of them, something I don\u0026rsquo;t have the time or motivation to do.\nWe had already agreed with eric@, for other reasons, that the API should be switched to a line-based protocol, like was done for filters, but this is a very tricky task in the daemon at this point and requires solving non-trivial limitations first. I\u0026rsquo;ll skip on this because this article would grow considerably, let\u0026rsquo;s just say that my first two attempts at this were unsuccessful.\nIt still annoyed me that we couldn\u0026rsquo;t move forward with this so I decided to tackle the issue differently. I wrote table-procexec a table backend which translate the imsg protocol into a filter-like query/response line-based protocol:\ntable|0.1|1590631921.2944137|table-name|check|deadbeefdeadbeef|mailaddr|gilles@poolp.org table-response|deadbeefdeadbeef|found table|0.1|1590632034.2909038|table-name|lookup|deadbeefdeadbeef|alias|root table-response|deadbeefdeadbeef|found|gilles The table-procexec backend communicates with OpenSMTPD through imsg, forks a child table backend talking the new protocol and takes care of proxying queries and responses translating imsg to this new protocol.\nOpenSMTPD \u0026lt;-- imsg --\u0026gt; table-procexec \u0026lt;-- line-protocol --\u0026gt; table-foobar The child table backend can be any program written in any language, just like with filters, and all it has to do is to implement the three operations expected from a backend, read the queries from stdin and respond to stdout.\nThis isn\u0026rsquo;t as elegant as if OpenSMTPD was producing the line-based protocol itself, allowing us to skip the table-procexec layer, but it allows moving forward in a way that\u0026rsquo;s transparent to the daemon. We can already test the API, improve it, make sure it works, implement new table backends on top of it and use them. When the limitations in OpenSMTPD are solved, the table-procexec can be thrown away transparently:\nOpenSMTPD \u0026lt;-- line-protocol --\u0026gt; table-foobar This is a work in progress and while it can be plugged to OpenSMTPD today using nothing but the existing framework, it doesn\u0026rsquo;t allow passing a configuration file to the child backend (yet).\nIn terms of configuration, it works similarly to any existing backend:\n# table foobar uses procexec backend to fork table-foobar table foobar procexec:table-foobar Ideally, we should switch to a filter like syntax, but we\u0026rsquo;re not there yet:\ntable foobar proc-exec \u0026#34;table-foobar -f /etc/mail/foobar.conf\u0026#34; Testing this require building the table-procexec backend from the branch of the same name in the OpenSMTPD-extras repository, I\u0026rsquo;ll leave this as an exercise to the reader.\ngo-opensmtpd # Of course, testing the new table API required a PoC so I decided to start with a Go implementation.\nThe go-opensmtpd/table package provides a work-in-progress implementation of the protocol parser, as well as a table API to ease development. Writing a table backend in Go using this package is simple, here\u0026rsquo;s a simple dummy example of a table that only contains my e-mail address:\npackage main import ( \u0026#34;github.com/poolpOrg/go-opensmtpd/table\u0026#34; ) func check(token string, service table.LookupService, key string) { if key == \u0026#34;gilles@poolp.org\u0026#34; { // found in table table.Boolean(token, true) } else { // not found table.Boolean(token, false) } } func lookup(token string, service table.LookupService, key string) { if service == \u0026#34;alias\u0026#34; \u0026amp;\u0026amp; key == \u0026#34;root\u0026#34; { // if looking up alias for root, return my address table.Result(token, \u0026#34;gilles@poolp.org\u0026#34;) } else { // empty result, not-found table.Result(token) } } func fetch(token string, service table.LookupService) { // only my address is in the set table.Result(token, \u0026#34;gilles@poolp.org\u0026#34;) } func main() { table.OnCheck(check) table.OnLookup(lookup) table.OnFetch(fetch) table.Dispatch() } Note that the Dispatch() function never returns as it is the event loop for the protocol parser. I don\u0026rsquo;t have much to add here, this is trivial to understand.\nThe package is a work in progress, it is already available in a Github repository but expect it to change every few days at this point, don\u0026rsquo;t use it unless you are ready to suffer.\npy-opensmtpd # The goal not being to have a Go centric API, testing with a different language required another PoC so I decided to go with Python this time.\nThe py-opensmtpd/table module also provides a work-in-progress implementation of the protocol parser, as well as a table API to ease development. Writing a table backend in Python using this module is no more complex than in Go, here\u0026rsquo;s the exact same dummy example of a table that only contains my e-mail address:\nfrom opensmtpd import table def check(token, tableName, service, key): if key == \u0026#34;gilles@poolp.org\u0026#34;: return table.boolean(token, true) return table.boolean(token, false) def lookup(token, tableName, service, key): if service == \u0026#34;alias\u0026#34; and key == \u0026#34;root\u0026#34;: # if looking up alias for root, return my address return table.result(token, \u0026#34;gilles@poolp.org\u0026#34;) return table.result(token) def fetch(token, tableName, service): # only my address is in the set table.result(token, \u0026#34;gilles@poolp.org\u0026#34;) def main(): table.on_check(check) table.on_lookup(lookup) table.on_fetch(fetch) table.dispatch() if __name__ == \u0026#34;__main__\u0026#34;: main() Again, the dispatch() function never returns as it is the event loop for the protocol parser. The interface is the exact same as what I did with go-opensmtpd.\nThe package is also a work in progress, it is also already available in a Github repository but expect it to change every few days, don\u0026rsquo;t use it unless you are also ready to suffer.\npy-consul # And because these PoC were dummy and do not show how the API can be used for real cases, I wrote a PoC for a table-backend that would actually do something more useful.\nConsul is a service to \u0026ldquo;automate network configurations, discover services, and enable secure connectivity across any cloud or runtime\u0026rdquo;. It is another tool highly valued at work and it conveniently comes with a key-value store hidden behind a REST API, something that we can work with to implement the three table backend operations.\nI implemented a table-consul python backend using the key-value store to perform its lookups. The code is a work in progress, not meant to be used in production, but it highlights how simple it is to integrate with existing tools. It took about 20 minutes to write the backend from scratch with no prior experience in Consul. The backend itself is ~70 lines including the license and empty lines. Using table-consul, one can setup a consul instance and use the key-value store to hold table informations used in OpenSMTPD lookups.\nIn the following example, I will setup and OpenSMTPD to use table-consul for aliases lookups:\ntable consul procexec:table-consul listen on all action \u0026#34;local_users\u0026#34; maildir alias \u0026lt;consul\u0026gt; [...] I then start a brand new empty consul instance and send a mail to root from my OpenSMTPD:\nThen, after adding a key root with value gilles in the foobar/alias bucket and sending a mail again, without restarting the daemon:\nThe alias lookup went to table-consul through table-procexec, table-consul did an HTTP request to consul to retrieve the alias, passed it back to an unsuspecting OpenSMTPD just as if it came from a regular table.\nBecause table-procexec can\u0026rsquo;t pass configuration to child backend yet, I decided to use \u0026lt;table\u0026gt;/\u0026lt;service\u0026gt; as the bucket for key-values which is not always very practical as it doesn\u0026rsquo;t allow sharing same keys between multiple service lookups but this was good enough for a PoC.\nLike for filter-prometheus, the table is already available in a Github repository, but this is a work in progress and was mostly to play with the new API, not intended to be used for real as I don\u0026rsquo;t have any significant experience with consul to make sound choices.\nIf you are familiar with consul and want to give me a hand in making this table production-ready, I\u0026rsquo;ll be more than happy to make this happen with your help.\nWhat\u0026rsquo;s next ? # I started this article at ~2:00AM and spent four hours writing, I can\u0026rsquo;t even remember my name at this point so I have no idea what I\u0026rsquo;ll prioritize next month, I just need some sleep.\n","date":"28 May 2020","permalink":"/posts/2020-05-28/may-2020-opensmtpd-6.7.1p1-release-table-procexec-and-many-pocs/","section":"Posts","summary":"TL;DR: Worked on the OpenSMTPD 6.7 release; Did a lot of work on the new table API; Wrote several PoCs; WARNING:\nExamples of code and configuration that appear in this article are here to help illustrate and explain development stages of my work.","title":"May 2020: OpenSMTPD 6.7.1p1 release, table-procexec and many PoCs"},{"content":" TL;DR: Worked on my webmail; Did a bit of OpenSMTPD work; Webmail vs mutt # I started using console clients to read my mail back when I was a student in early 2000s, and I\u0026rsquo;ve been using mutt for as long as I can recall now. I installed various mail clients along the years, some with graphical interfaces such as thunderbird or sylpheed, others with a web interface such as nullWebmail, SquirrelMail, Roundcube, Mailpile or currently Rainloop. No matter which, I always end up falling back to mutt for most tasks.\nThen why don\u0026rsquo;t I stick with mutt and stop trying to use other ones ?\nWether I like it or not, a lot of mails that are important to me contain MIME-parts that are just not comfortable to deal with in console. There are ways to work-around, like having lynx render HTML mails, mupdf be launched for PDF attachements, and so on\u0026hellip; but this is not comfortable enough (to me). I keep using mutt for a large part of my mails and keep some unread so that I can read them from another client.\nOk, so why don\u0026rsquo;t I stick with other ones and stop using mutt ?\nI want to be able to read my mail from various places and not rely on a heavy client. This leaves me with console-based clients that I can ssh to, and webmails that I can hit from any browser. Unfortunately for me, most webmails have interfaces that are inspired by heavy clients with framing and pagination, something that\u0026rsquo;s not practical with my use of mails.\nIf I want to cleanup the List folder which has accumulated too many mails, I\u0026rsquo;d rather use mutt which lets me scroll through the entire folder and delete mails with a key press, rather than rely on this user interface to paginate through 81 pages. This particular webmail has some keyboard shortcuts which makes it less painful than some others, but still, I have to scroll down to see the list of mails for this page and skip to the next pages to see other mails in the folder. Because of this, I will simply never use that interface to do this task and fall back to mutt which lets me do in a couple minutes.\nMy 2012 attempt at a webmail # In 2012, I wrote a small webmail for my use in python with the bottlepy mini framework, with the intent to release it opensource.\nI had not yet understood what caused me not to use them on the long run, but I knew I didn\u0026rsquo;t work along with traditional webmail interfaces. So it was mostly keyboard based and had a different look than others:\nIt also had some features I thought were interesting for my own use-case, like for example the use of robohash to immediately spot mails originating from the same sender, the detection of diffs and their highlighting to make them more readable, or a show analysis button which allowed me to display a ton of technical details about a particular sender and mail:\nUnfortunately, I got carried by ${DAYTIMEJOB} and gave up work on it before it became usable for real. It is a mail viewer with limited writing abilities.\nMy 2020 attempt at a webmail # We\u0026rsquo;re now 8 years later and I still want to scratch that itch that annoys me since then. I think my past effort had value but it didn\u0026rsquo;t tackle the issue correctly, and with 8 additional years of mail experience and nuisance, I think I can come up with something that suits me better.\nMy knowledge and the technologies available are not the same than in 2012. I\u0026rsquo;m sure I could have achieved something very usable back then but definitely not something comfortable. Part of what makes mutt comfortable is its interface, but the other part is the fluidity of being able to scroll a folder of thousands of mails. My 2012 webmail didn\u0026rsquo;t prefetch mails, didn\u0026rsquo;t do any kind of caching and did everything synchronously. Sure I could skip from a page to another with the press of an arrow key, but I then had to wait for a command to be sent to the IMAP server and for the answer to come back before the page actually changed. It was not really slow but it was definitely not really fluid.\nI decided to rewrite it from scratch, still with the intent of publishing it opensource.\nBelow is a screenshot from my initial work, start this week:\nThe webmail is now split into a backend and a frontend. The backend is written in python + flask and exposes a REST API to the IMAP server, whereas the frontend is written in react (which I had never used before this Monday) and only implements the rendering. The reason for this split is that I intend to write custom frontends too. This is too birds with one stone.\nThe screenshot is for an early work in progress so this is a \u0026ldquo;developer\u0026rdquo; interface more than a user interface. The idea here is that I have a mail listing with a cursor controled by my arrow keys, scrolling down and up will paginate mail by mail whereas left and right will paginate by full pages. Delete or d keys will move a mail to Trash, while Enter will display the mail viewer. Space will select the mail that\u0026rsquo;s below the cursor and add it to a selected list so that Delete will allow bulk deleting for example.\nOf course, I will make it easier to use for non-techies but my main goal is that I\u0026rsquo;m able to find the same comfort as mutt, THEN add buttons so that people who are more into using a mouse can find their ways.\nI\u0026rsquo;ll continue working on this every now and then until I have something that I can put up a demo for, then when I\u0026rsquo;m happy enough I will release the code for backend and frontend. Right now, having no experience with react causes me to do a lot of write and throw-away code, there\u0026rsquo;s no point in sharing code at this point.\nPreparing OpenSMTPD 6.7.0 # OpenBSD 6.7 is around the corner and OpenSMTPD 6.7.0 by the same occasion.\nAs I explained last month, I no longer contribute directly to OpenBSD as a committer but I didn\u0026rsquo;t ghost and helped review several diffs.\nBug fixes # As with any release cycle, people wake up right before a release with annoying bugs.\nThis time it was three bugs. One was a bug in the aliases expansion code and the handling of virtual aliases catchalls where the @ catch-all incorrectly caught recipients in some setups. Another was a shortcoming in the order of fields of two events in the filtering protocol that required a protocol change and version bump. The last one was a buggy root-restricted command causing the server to halt due to a fatal().\nAliasing code is particularly tricky because it doesn\u0026rsquo;t just map a recipient address to a user, it can map a recipient address to multiple users spanning over multiple domains that may or may not use aliases. It\u0026rsquo;s not a key =\u0026gt; value but a key =\u0026gt; tree resolution where each node has a different expansion context. To make things harder, the resolution can\u0026rsquo;t block as it would prevent concurrent sessions from working, so the whole expansion code is an interruptible-resumable loop which makes code simple to read but complex to follow. The first reaction to a bug affecting aliases is always denial because even simple fixes require a lot of thinking. eric@ and I confirmed and discussed the issue, we found a fix very fast that we were both sure worked, but we had to make sure that it didn\u0026rsquo;t have side-effects by testing multiple scenarios.\nThe shortcoming in the filtering protocol was easily fixed by swapping two fields but required a protocol bump, so this also needed discussions to make sure the fix was really the one we wanted and not come up with another one a month later. This also required planning for how existing filters would need to be adapted to support both 6.6.0 and 6.7.0 transparently.\nThe last issue eric@ fixed alone, I only reviewed the diff.\nAll the fixes were committed to OpenBSD and synchronized in portable, so they will not be present in the 6.7.0 release.\nPortable # The portable version of OpenSMTPD also needed a specific fix after Denis Fateyev, package maintainer for Fedora, and Ryan Kavanagh, package maintainer for Debian, reported that it failed to build with newer gcc-10. I worked on a fix which was committed to the portable version allowing it to build again and, while at it, added a gcc-10 target to our CI so we\u0026rsquo;ll detect breakages there too.\nGiven that I will spend more time working in the portable branch from now on, I also took time to set myself a proper docker-based development environment to test build and fixes on various systems.\nGithub history # Finally, I decided to tackle an issue that\u0026rsquo;s been around since forever.\nThe history for the project in our Github mirror was shit. OpenBSD is the main repository and uses cvs, we synchronize code in git by diffing the checkout from git and the checkout from git\u0026hellip; then applying the diff to git with a commit log saying sync.\nAs a result, our git repository contained detailed commit logs for portable-specific commits, but whenever trying to track a specific commit originating from OpenBSD\u0026hellip; you could not find it by commit log and you had to find the sync commit which contained the relevant diff.\nI managed to bring back the entire OpenSMTPD history from cvs to our mirror, so you can now track individual OpenBSD commits there instead of sync.\nHere\u0026rsquo;s the first import commit of OpenSMTPD into OpenBSD, which we didn\u0026rsquo;t have before.\nMore interestingly, here\u0026rsquo;s the sync commit for the fix to the aliasing bug discussed above\u0026hellip; which also contains various other commits to unrelated stuff, and here\u0026rsquo;s the actual commit to OpenBSD.\nWhat\u0026rsquo;s next ? # I have switched my part-time scheduling so that instead of a week every month, I have a day each week + 1 day each month to work on my projects. This will be the same amount of time but spread over the month.\nI\u0026rsquo;ll spend most of my spare time in May preparing the actual release of portable OpenSMTPD 6.7.0, but also to work on my webmail.\nI also have various branches I want to submit to OpenBSD for review after the 6.7.0 release, and work to add support to ECDSA in portable OpenSMTPD when OpenSSL is used instead of LibreSSL.\nIf you\u0026rsquo;re interested in my work, consider supporting me on patreon or github.\n","date":"1 May 2020","permalink":"/posts/2020-05-01/april-2020-worked-on-a-webmail-and-a-bit-of-opensmtpd-too/","section":"Posts","summary":"TL;DR: Worked on my webmail; Did a bit of OpenSMTPD work; Webmail vs mutt # I started using console clients to read my mail back when I was a student in early 2000s, and I\u0026rsquo;ve been using mutt for as long as I can recall now.","title":"April 2020: worked on a webmail and a bit of OpenSMTPD too"},{"content":" TL;DR: - I took a break to deal with personal stuff - I'm taking a long break from OpenBSD for personal reasons - I may or may not have experienced COVID-19, who knows - resumed work on OpenSMTPD and other projects This is a weird report # This is a weird report.\nI\u0026rsquo;ll mix a bit of personal info to provide some context as to why I decided to leave the OpenBSD project, explain why this doesn\u0026rsquo;t mean I won\u0026rsquo;t be an active contributor, and give some small insight into my upcoming work.\nA bit of personal context # As some may have noticed, I didn\u0026rsquo;t write a report in February: I took a small break.\nIn 2013, my body started failing me with constant headaches and nausea. I had weekly visits at my doctor to adapt non-working treatments, which then led to weekly exams to rule out a serious illness and eventually led me to being admitted in hospital for biopsy, endoscopy and other unpleasant tortures. All of this resulted in the observation that there was no organic reason to my illness, it was purely psychological. I couldn\u0026rsquo;t wrap my head around this because everything was apparently fine in my life, but I was treated accordingly for a few weeks and my physical health got much better in no time. The whole episode became a thing of the past after a while.\nLate 2018, I was diagnosed with generalised anxiety disorder and alexithymia. The general idea is that I\u0026rsquo;m pretty much constantly anxious but I\u0026rsquo;m also unaware of it most of the time: I have trouble decoding my own emotions and the related physical sensations. It is not uncommon for my better half to ask me if I\u0026rsquo;m feeling ok as she realizes I\u0026rsquo;m having an anxiety attack before/if I do. This puts a lot of perspective on the events of 2013: I don\u0026rsquo;t realise my anxiety is becoming unsustainable until my body starts to dysfunction somehow for extended periods of time.\nI\u0026rsquo;ve grown used to it and came up with fairly effective coping mechanisms. In regular times, most people don\u0026rsquo;t know about this disorder even after knowing me for years, but these are no regular times for me. I had major changes in my life with the arrival of a baby, the learning of a bad news regarding a close family member, and the doubts about staying at my daytime job.\nMy mind hasn\u0026rsquo;t been at rest in a while and keeping things under control has a cost, sadly I\u0026rsquo;m unable to properly assess when that cost is becoming too high\u0026hellip; which brings me to the next topic.\nTaking a break from OpenBSD # I\u0026rsquo;ve been an OpenBSD user since 1999 and an OpenBSD developer since 2008. OpenBSD has literally been around me for half of my life now.\nI\u0026rsquo;ve attended multiple hackathons, met plenty of skilled hackers from across the globe and been part of a project I love. A project that drove me into studying computer science and whose philosophy I\u0026rsquo;m still perfectly aligned with after all these years. I spent wonderful years and I\u0026rsquo;m a bit sad that I have to put an end to it.\nMy anxiety disorder causes me to obsess a lot about my work, and while I can easily put distance between my daytime job work and myself, it is much harder for an opensource project I\u0026rsquo;m emotionally attached to. I\u0026rsquo;m unable to disconnect and take a break, I feel a disproportionate duty towards hackers and users. I must not let people down and so I keep obsessing about what needs to be done and how it should be done. I can never find enough time to do all of it, so it leads to more anxiety and more obsession, in a vicious circle.\nThe almost two months break I just took without booting an OpenBSD laptop was a first in +10 years. I can\u0026rsquo;t recall a single vacation or week-end where I didn\u0026rsquo;t boot my laptop at least an hour to work on something.\nI decided to quit because I don\u0026rsquo;t believe I can be part of the project and work a few hours here and there, disconnecting to take a break every now and then. It\u0026rsquo;s just not in my nature. As long as I\u0026rsquo;ll be part of the team, I\u0026rsquo;ll feel a duty towards it and will be unable to put distance when needed. I have to deal with a lot of personal stuff these days and not being able to assess emotions correctly makes it hard to know my limits. The last thing I want is to push myself too far and back into hospital for being too obsessed\u0026hellip; about a hobby.\nI told the OpenBSD people I was leaving in a lengthy mail late January IIRC, and I received nothing but friendly replies. I may not be an OpenBSD hacker anymore but that doesn\u0026rsquo;t matter much, I met the coolest people.\nDoes it mean that I will no longer contribute to OpenBSD/OpenSMTPD ? # Not at all, I don\u0026rsquo;t intend to ghost OpenBSD/OpenSMTPD.\nWhat it means is that I\u0026rsquo;ll be contributing as any other user through diffs sent to specific hackers or to mailing lists. This will make it easier for me to contribute on my own schedule and will leave to others the decision of what gets in and when. Other hackers have done that in the past for diverse reasons, I wouldn\u0026rsquo;t be the first former hacker to send diffs to tech@.\nIt\u0026rsquo;ll also make it easier for me to work on other stuff. I have many other interests than but could never convince myself that my time was better spent elsewhere. You know, duty and all.\nAs far as OpenSMTPD is concerned, I\u0026rsquo;m currently sitting on diffs to convert it to libtls as well as other related features that are waiting for the 6.7 release. I will also continue my work to improve the portable version which is outside of OpenBSD\u0026rsquo;s scope and doesn\u0026rsquo;t really attract developers besides me.\nI\u0026rsquo;ll still be a main contributor, just not the lead on that project.\nSuspected COVID-19: achievement unlocked. # My wife and I were suspected COVID-19. Symptoms were mild but between the fever, the weakness, the coughing and the caring for a baby, it was not a pleasant experience.\nI\u0026rsquo;ll give it 1 star, do not recommend.\nResuming work # As I\u0026rsquo;m back from my break I don\u0026rsquo;t have much work to talk about in this report, but I\u0026rsquo;m working on my personal projects this week so expect stuff in the next one.\nI have already synchronized OpenSMTPD portable with the OpenBSD tree so it carries commits that happened by other developers during my break.\nI\u0026rsquo;m catching up on my libtls work in OpenSMTPD portable as it got interrupted by the advisory in January. I need to dive back into that code and test that it works as expected. I will also continue my work of cleanup in openbsd-compat which is bringing OpenSMTPD portable closer to a clean compat layer.\nI\u0026rsquo;m currently working on client-side mail stuff. I did a major fuckup with my mailbox and reinjected years of Trash and Junk to my Inbox. This was +30k mails to classify and it inspired me to work on a tool which I\u0026rsquo;ll continue working on and disclose when it is useable.\nFinally, I\u0026rsquo;ll catch up on the MANY notifications I received on github and the tons of mails that I accumulated in my mailbox during my break. I know there are pull requests on some filters and questions regarding OpenSMTPD. I need to catch up on all of this.\nTake care, next report will be in the usual format.\n","date":"30 March 2020","permalink":"/posts/2020-03-30/anxiety-openbsd-break-covid-19-and-resuming-work/","section":"Posts","summary":"TL;DR: - I took a break to deal with personal stuff - I'm taking a long break from OpenBSD for personal reasons - I may or may not have experienced COVID-19, who knows - resumed work on OpenSMTPD and other projects This is a weird report # This is a weird report.","title":"Anxiety, OpenBSD break, COVID-19 and resuming work"},{"content":" TL;DR: - Qualys released an advisory for a bad, bad vulnerability - an MTA is a very bad software to have a vulnerability in - hole was plugged but that's not enough, similar bugs should be mitigated in the future - article discusses what could have prevented escalation despite the bug What happened ? # Qualys contacted by e-mail to tell me they found a vulnerability in OpenSMTPD and would send me the encrypted draft for advisory.\nReceiving this kind of e-mail when working on a daemon that can\u0026rsquo;t revoke completely privileges is not a thing you want to read, particularly when you know how efficient they are at spotting a small bug and leveraging into a full-fledged clusterfuck.\nWhile I was waiting for them to mail me back the draft, I immediately jumped to the privileged process which is what worries me the most, and started reading all the entry points to that process at runtime.\nThis isn\u0026rsquo;t a big task, the privileged process doesn\u0026rsquo;t do many operations at runtime:\nit opens ~/.forward files located in home directories it authenticates username/password pair against bsd_auth(3) it creates user-owned mda processes for mail delivery agents In the case of ~/.forward files, it checks if the user has a ~/.forward file with correct permissions and mode, opens it then passes the file descriptor back to an unprivileged process without doing anything with it. I didn\u0026rsquo;t see a way this could be exploited.\nIn the case of authentication, which was my main worry given the recent vulnerabilities they found in the bsd_auth(3) layer, it received a username and password and passed them directly to auth_userokay(3), returning the result to unprivileged process. I was highly suspicious of a bug here but given how simple the code is, I didn\u0026rsquo;t see a way this could be exploited either.\nFinally, the user-owned mda processes creation became my main worry. It doesn\u0026rsquo;t do much as it basically calls fork() and executes the mda command using recipient user privileges, but it also has a special case for mbox which can\u0026rsquo;t work without privileges.\nI had hope that the bug they found would be elsewhere and not so bad as the privileged process is fairly isolated. It can only be reached by going through another unprivileged process and its input validation. The worst-case scenario would be finding an exploitable bug either in the mbox delivery special case, or in the small portion of code before privileges dropping for other mail delivery agents. I focused efforts on reading the code path again and again but didn\u0026rsquo;t find a bug. I gave up after a few hours and decided to wait for the draft.\nI woke up to an advisory draft exploiting the worst-case scenario\u0026hellip; through a logic bug in the input validation code of an unprivileged process.\nPerfect, just perfect.\nThe problem # OpenSMTPD uses an input validation function, smtp_mailaddr(), to ensure that e-mail addresses that enter the queue are valid.\nWhat the function does is that it checks that the e-mail address parses into a struct mailaddr, then validates the local part with valid_localpart() and the domain part with valid_domainpart(). The valid_localpart() function checks that the local part only contains characters from an allowed set. The valid_domainpart() either checks that the domain is a valid IP address using inet_pton(), or that it is a valid hostname using res_hnok().\nThe smtp_mailaddr() function also has to handle two cases where an invalid address is still acceptable and has to go through:\nThe first one is that of bounces (MAILER-DAEMON) with empty local and domain parts, MAIL FROM:\u0026lt;\u0026gt;, which is only acceptable during the MAIL FROM phase of SMTP.\nThe second one is that of addresses with empty domains, MAIL FROM:\u0026lt;gilles\u0026gt;, which is generated by some MUA when used locally and which will use the local domain attached to the listener, MAIL FROM:\u0026lt;gilles@laptop.poolp.org\u0026gt; on my machine. This was initially not accepted by the daemon but had to be implemented for interoperability, which is when the bug was introduced.\nUnfortunately for us, Qualys identified a logic error in smtp_mailaddr(). If one of the checks on local or domain parts fails, smtp_mailaddr() checks if it is due to one of the two cases above. In the case of failed domain, it mistakenly does an erroneous check that cancels out the previous call valid_localpart() and, as a result, an address that was rejected due to an invalid local part gets accepted again if there is no domain.\nMAIL FROM:\u0026lt;gilles;test@laptop.poolp.org\u0026gt; 553 5.1.0 Sender address syntax error MAIL FROM:\u0026lt;gilles;test\u0026gt; 250 2.0.0 Ok This mistake was introduced five years ago with the following diff:\n=================================================================== RCS file: /cvs/src/usr.sbin/smtpd/smtp_session.c,v retrieving revision 1.215 retrieving revision 1.216 diff -u -r1.215 -r1.216 --- src/usr.sbin/smtpd/smtp_session.c\t2014/07/09 12:44:54\t1.215 +++ src/usr.sbin/smtpd/smtp_session.c\t2014/10/02 21:27:54\t1.216 @@ -1,4 +1,4 @@ -/*\t$OpenBSD: smtp_session.c,v 1.215 2014/07/09 12:44:54 eric Exp $\t*/ +/*\t$OpenBSD: smtp_session.c,v 1.216 2014/10/02 21:27:54 gilles Exp $\t*/ /* * Copyright (c) 2008 Gilles Chehade \u0026lt;gilles@poolp.org\u0026gt; @@ -1758,21 +1758,20 @@ if (!valid_localpart(maddr-\u0026gt;user) || !valid_domainpart(maddr-\u0026gt;domain)) { -\t/* We accept empty sender for MAIL FROM */ -\tif (mailfrom \u0026amp;\u0026amp; -\tmaddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; \u0026amp;\u0026amp; -\tmaddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) +\t/* accept empty return-path in MAIL FROM, required for bounces */ +\tif (mailfrom \u0026amp;\u0026amp; maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; \u0026amp;\u0026amp; maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) return (1); -\t/* We accept empty domain for RCPT TO if user is postmaster */ -\tif (!mailfrom \u0026amp;\u0026amp; -\tstrcasecmp(maddr-\u0026gt;user, \u0026#34;postmaster\u0026#34;) == 0 \u0026amp;\u0026amp; -\tmaddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { +\t/* no user-part, reject */ +\tif (maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39;) +\treturn (0); + +\t/* no domain, local user */ +\tif (maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { (void)strlcpy(maddr-\u0026gt;domain, domain, sizeof(maddr-\u0026gt;domain)); return (1); } -\treturn (0); } Back then, the mistake had no consequences as OpenSMTPD had a very different implementation for delivery. It didn\u0026rsquo;t handle MDA through external utilities, like is done in other MTA, but used builtin implementations for all methods (maildir, lmtp, file, mda, \u0026hellip;). The mbox delivery was handled as a special case through a simple execle().\nWith this way of dealing with MDA, there was no shell involved and the incorrectly validated e-mail address wasn\u0026rsquo;t in the path of delivery, the bug had no consequences despite being there. This way of dealing with MDA wasn\u0026rsquo;t right though. It came with its share of complexity and short-comings, most notably a very complex mda layer handling all methods differently with many special cases, and a more complex forkmda() function which is code running in the privileged process and that we want as simple as possible.\nThe bug became exploitable with another unrelated commit three years and a half later, in a different area, when OpenSMTPD switched to the new grammar and brought support for MDA through external utilities, ditching its builtin delivery methods.\nBasically, instead of each delivery method coming with its own structure and custom checks in the entire code path, they were reimplemented as standalone executables that could be used by any MTA. From this point, OpenSMTPD didn\u0026rsquo;t know the difference between maildir and lmtp, they were just regular mail delivery agents. The MDA execution code path got simplified by removing all delivery-specific cases and unified so it would craft a command line, using what we assumed to be correctly validated input, that could be executed after the forkmda()function dropped its privileges.\nBecause the delivery agents are standalone executables that expect their options on the command line with quoting and escaping support, we decided to use the shell to execute the command rather than rewriting our own command-line parser that might contain parsing bugs. This is something done in other MTA, we didn\u0026rsquo;t invent a clever way to be lazy. The shell is executed with the recipient privileges and passed the mda command, similarly to if the recipient had executed the command themselves in their own shell. Since the command is either crafted from smtpd.conf which is controlled by postmaster, or .forward which is controlled by the recipient, there is supposedly no problem as long as the input is properly validated.\nWe could jump to the conclusion that using system() instead of execle() is what allowed the escalation but this is inaccurate. Incorrectly validated input and system() made the exploit possible, not the escalation. All delivery methods use the exact same code path and they are not all subject to the escalation. For instance, lmtp allowed unprivileged execution of commands whereas maildir is unimpacted.\nWhat made the escalation possible is a special case that OpenSMTPD has to handle specifically for mbox, and which opens the door to escalation if a bug manages to be exploited in that (relatively small) code path.\nThe fix # When I received the advisory draft, I hurried to produce a fix as soon as possible then ended with two diffs that I showed Qualys: a quick fix that just plugged the hole, and a better fix that plugged the hole by rewriting the smtp_mailaddr() function differently.\nThe quick fix is straightforward and can be implemented as a one-liner, it just tested valid_localpart() again when handling the missing domain case:\n--- src/usr.sbin/smtpd/smtp_session.c\t2019/10/04 08:34:29\t1.415 +++ smtp_session.c\t2020-01-30 08:51:31.000000000 +0100 @@ -2247,7 +2247,7 @@ smtp_mailaddr(struct mailaddr *maddr, ch return (0); /* no domain, local user */ -\tif (maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { +\tif (valid_localpart(maddr-\u0026gt;user) \u0026amp;\u0026amp; maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { (void)strlcpy(maddr-\u0026gt;domain, domain, sizeof(maddr-\u0026gt;domain)); return (1); The better fix however rearranged the function in a different way that is less error-prone, and less likely to reintroduce similar mistakes. It did so by not grouping validators into a common block, but handling each case separately:\n=================================================================== RCS file: /cvs/src/usr.sbin/smtpd/smtp_session.c,v retrieving revision 1.415 retrieving revision 1.415.2.1 diff -u -r1.415 -r1.415.2.1 --- src/usr.sbin/smtpd/smtp_session.c\t2019/10/04 08:34:29\t1.415 +++ src/usr.sbin/smtpd/smtp_session.c\t2020/01/28 21:39:20\t1.415.2.1 @@ -1,4 +1,4 @@ -/*\t$OpenBSD: smtp_session.c,v 1.415 2019/10/04 08:34:29 gilles Exp $\t*/ +/*\t$OpenBSD: smtp_session.c,v 1.415.2.1 2020/01/28 21:39:20 gilles Exp $\t*/ /* * Copyright (c) 2008 Gilles Chehade \u0026lt;gilles@poolp.org\u0026gt; @@ -2215,24 +2215,22 @@ memmove(maddr-\u0026gt;user, p, strlen(p) + 1); } -\tif (!valid_localpart(maddr-\u0026gt;user) || -\t!valid_domainpart(maddr-\u0026gt;domain)) { -\t/* accept empty return-path in MAIL FROM, required for bounces */ -\tif (mailfrom \u0026amp;\u0026amp; maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; \u0026amp;\u0026amp; maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) -\treturn (1); +\t/* accept empty return-path in MAIL FROM, required for bounces */ +\tif (mailfrom \u0026amp;\u0026amp; maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; \u0026amp;\u0026amp; maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) +\treturn (1); -\t/* no user-part, reject */ -\tif (maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39;) -\treturn (0); - -\t/* no domain, local user */ -\tif (maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { -\t(void)strlcpy(maddr-\u0026gt;domain, domain, -\tsizeof(maddr-\u0026gt;domain)); -\treturn (1); -\t} +\t/* no or invalid user-part, reject */ +\tif (maddr-\u0026gt;user[0] == \u0026#39;\\0\u0026#39; || !valid_localpart(maddr-\u0026gt;user)) return (0); + +\t/* no domain part, local user */ +\tif (maddr-\u0026gt;domain[0] == \u0026#39;\\0\u0026#39;) { +\t(void)strlcpy(maddr-\u0026gt;domain, domain, +\tsizeof(maddr-\u0026gt;domain)); } + +\tif (!valid_domainpart(maddr-\u0026gt;domain)) +\treturn (0); return (1); } There is room for improvement and I think this function should be rewritten in a different style to make it harder to introduce bugs. But this version works, it was tested, proof-read by multiple people (including the auditors), and since it is unlikely to change before a long time (last change was in 2014), I will tackle it with a clear mind, not right now while still stressed.\nFixing the bug like I did plugs the hole, but that\u0026rsquo;s not enough for me to be relaxed, it closes the door to the escalation but doesn\u0026rsquo;t build a wall of concrete in front of it. Today, there\u0026rsquo;s no known bug that affects this code path and we may be fine, but unless we actually kill the potential for escalation, we\u0026rsquo;ll never be sure that researches won\u0026rsquo;t find another bug and another creative way to exploit it.\nIn this article, I\u0026rsquo;ll discuss mitigation mechanisms that I think should be put in place to avoid this from happening again, regardless of what bugs are found in processes exposed to attackers. Their goal is to deal with the potential for escalation, not with specific bugs.\nMy post-mortem thoughts # Even though fixing smtp_mailaddr() plugs the hole, a bug in the smtp layer should not have been able to result in such a catastrophic outcome, regardless of the bug. Today, with the combination of pledge(), privileges separation (smtp engine runs as unprivileged user _smtpd), filesystem isolation (smtp engine is chroot()-ed to /var/empty), and several internal consistency checks that will fatal() if something fishy is detected, misbehaving code is constrained and we expect limited consequences. My take has always been that I can\u0026rsquo;t code bug-free so I expect that bugs will be found, and I want OpenSMTPD to crash in such cases so that I can fix the bug and roll a release. A crash in SMTP land is really not a big deal, protocol is fault-tolerant and other mail exchangers have retry logic, I can live with that.\nThe incident here was not caused by misbehaving code, it was caused by a logic error and the code did what it was written to do: it didn\u0026rsquo;t corrupt memory and dodged sanitisation to craft an IPC message all the way to an exploit in another process. This is not a problem caused by the language, likes some people imply, and not a problem caused by a memory corruption of some sort. The sanitisation function incorrectly validated the input and from that point, the input followed the exact same code path as any other valid input until it reached the mda layer. This is not a failure of the mechanisms put in place to mitigate misbehaving code, and I think it is important to consider it for what it is: a correct implementation of an incorrect logic.\nSeeing it as such isn\u0026rsquo;t a way to shrug it off, quite the opposite. If we consider the incident without the human factor and try to address this with misbehaving code detection, it is bound to fail because\u0026hellip; the code did exactly what it was written to do. We may slap a few sanity checks here and there but if they\u0026rsquo;re supposed to check the same thing, they will have the same mistake. We can\u0026rsquo;t fix my stupid with checks.\nWe can\u0026rsquo;t prevent human mistakes, they will happen because tools won\u0026rsquo;t help spot that a human-described logic is flawed. What we need is to make changes so that OpenSMTPD becomes more resistant to human errors. In other words, we need safe-guards that are not dependant on sanity checks and input, we need safe-guards that will guarantee that even if OpenSMTPD lets completely untrusted input pass through, this will have the most limited consequences\u0026hellip; then we ensure that it doesn\u0026rsquo;t let untrusted input pass through.\nWhat made the exploit possible is the logic mistake in validation code and the use of system() with that invalid input, but what made the escalation possible is the mbox delivery method.\nThe problem(s) with mbox delivery # When a delivery takes places, OpenSMTPD does the delivery on behalf of a recipient user. Regardless of the delivery method, it will create a new mda process that will handle the delivery and drop its privileges to that of the recipient user. This is meant to ensure that a user adding a dangerous command in their .forward files can only impact themselves.\nFurthermore, OpenSMTPD strictly forbids execution of an mda command with a privileged account: root is not allowed to receive mail with maildir or lmtp, the delivery will be aborted with an error message that delivery to root is forbidden without an alias to an unprivileged user. The mda command is not processed. This is what saves maildir and lmtp from being subjected to the escalation: they cannot be ran from a privileged process, bugs will at most impact the recipient user which can never be root. Not that it\u0026rsquo;s good, but it\u0026rsquo;s an order of magnitude less bad.\nIn the case of mbox however we can\u0026rsquo;t do that.\nFirst of all, if we take a fresh install, we can\u0026rsquo;t assume that an unprivileged user exists since a user might not have created one at install. In this situation, if mbox refuses delivery to root then mails regarding security or daily/weekly/monthly will not be deliverable. Forbidding delivery to root implies requiring the creation of a user that root can be aliased to. For this reason, we can\u0026rsquo;t easily switch from mbox to another delivery method either since they would enforce the restriction, breaking mail out of the box.\nThen, unlike maildir which writes to a subdirectory of the recipient user and can therefore do it with the recipient user privileges, mbox writes to a common directory, usually /var/mail or /var/spool/mail. The mailbox is\tuser-owned, so that /var/mail/gilles can be written or read by gilles, but since gilles cannot be allowed to create his own mailbox in the common directory, the creation has to be done by a privileged process that can both create the mailbox AND change its ownership.\nSince the mbox format doesn\u0026rsquo;t support concurrent deliveries, it also requires some locking to avoid concurrent writes. This locking is historically done by creating a gilles.lock file in the common directory. Mail user agents expect this. This whole delivery logic is handled outside of the daemon by mail.local, a delivery agent that preexists OpenSMTPD and which we inherited from the previous era. The downside is that mail.local requires being called with privileges to do the work and handle privileges dropping itself.\nThis is where the potential for escalation resides.\nWhen mbox delivery is set, OpenSMTPD HAS to create a privileged process to execute mail.local no matter what user will receive the mail. It therefore has an explicit special case that forking a privileged mda is forbidden\u0026hellip; except for mbox. Even if root had a ~/.forward explicitely calling mail.local, it would be be forbidden.\nThe code path is relatively small: the mda process is created with privileges, mail.local is executed and will internally lower its privileges. Unfortunately for us, the exploit basically managed to replace the execution of mail.local with the execution of an arbitrary command, sitting in between the privileged process being fork()-ed and mail.local dropping privileges.\nThe source of the problem is that unlike other delivery methods, when using the mbox delivery method OpenSMTPD is required to create this privileged process. In this case the exploit targeted the way we executed the mail delivery agent but, thinking further, if an exploit was found that managed to abuse anything between the fork() and the execution of mail.local, we could be in a similar situation using a different attack vector: during a small period of time, we have a privileged process with a potential for being exploited somehow.\nThis is why I believe plugging the hole is not enough. It solves the immediate problem but not the underlying issue, it doesn\u0026rsquo;t kill the potential for escalation: if root is allowed to receive mail in a root-owned mailbox then at some point a privileged process had to do the delivery.\nThe only way to kill this potential for escalation is to get rid of the privileged mda process. No special case, no nothing. If we don\u0026rsquo;t have a privileged mda process, then a bug may exist and an exploit may find its way into the process, but it won\u0026rsquo;t allow escalation.\nThis is where we must be heading towards and it\u0026rsquo;ll require some creative solutions to work around mbox limitations.\nSeveral ideas # Since this blew up to my face, I had several ideas to tackle this. Some were already discussed and not retained because they had potential for other issues. The current ideas are these:\nswitching back mail delivery agents to execle() disallowing delivery to root managing to somehow run mail.local without privileges mbox creation and locking in OpenSMTPD to allow user execution of mail.local Not each of them is straight-forward but each of them would have avoided the escalation, regardless of the bug being present, so I think these are critical to implement in order to avoid future bugs from having the kind of impact we experienced.\nSwitching back mail delivery agents to execle() # This doesn\u0026rsquo;t solve the underlying issue of having a privileged process for mbox, but it would have prevented this specific attack.\nMoving all mail delivery agents to execle() is tricky but we\u0026rsquo;re going to got that way. I have already written a diff to switch mbox and it will be committed shortly. The maildir and lmtp will be similarly converted.\nThe only issue with that move is that some mail delivery agents expect a command-line interface that\u0026rsquo;s more complex than ours:\naction foobar mda \u0026#34;/usr/local/bin/procmail -a -ton -of -options \\\u0026#34;with quotes and escaping\\\u0026#34;\u0026#34; or it\u0026rsquo;s ~/.forward variant:\n|/usr/local/bin/procmail -a -ton -of -options \u0026#34;with quotes and \\\u0026#34;escaping\\\u0026#34;\u0026#34; or even it\u0026rsquo;s /etc/mail/aliases variant:\nlists:\t|/usr/local/bin/mailing-list.py -a -ton -of -options I doubt anyone wants to see these go away, and switching them to execle() implies reimplementing a command-line parser with shell-like semantics, then using the parsed command line to call the execle() function. This is the literal definition of rewriting a shell and I\u0026rsquo;d rather rely on a standard shell than rewriting my dumbed-down one.\nSince these commands can never run with privileges, what I suggest is that we use a wrapping mail delivery agent with a simple interface that we can easily plug through execle(), then let THAT mail delivery agent run the mda command through a shell. This allows us to have a single code path through execle() for all mail delivery agents in the daemon, and restrict the system() call to specific use-cases handled by a specific mail delivery agent outside the daemon.\nDisallowing delivery to root # No need for much explanations here: this would have effectively prevented the privileges escalation despite the logic bug.\nI think we should disallow deliveries to privileged users unconditionally. I never thought it was a good idea to accept doing it and this is why I disallowed it for everything but mbox. This is going to be inconvenient to many because it means that root HAS to be an alias, but everyone using maildir or lmtp has been in this mode for years now and I never had a complaint about it. We can\u0026rsquo;t tell people not to log in as root and use doas or sudo, then ensure that everything is convenient for them to log in as root and have their mails delivered knowing the risks. It doesn\u0026rsquo;t make sense.\nThis will put us at odds with other MTA implementations that accept delivering mail to root, and every time I suggested doing something differently from other MTA people got angry at me for not being able to keep their habits. But now that this blew to my face I don\u0026rsquo;t think there\u0026rsquo;s an excuse for keeping a time-bomb that we know can blow hard. I hope there will be no resistance to this change, but at some point, if we refuse to diverge from dangerous practices it means we are knowingly leaving potential for escalation. Someone else will have to defend why this is a good idea.\nWhile at it\u0026hellip;\nOn the somewhat related side-note of doing something I dislike because other MTA do it and we shouldn\u0026rsquo;t diverge, supporting commands in aliases is also a bad idea from a security point of view. It doesn\u0026rsquo;t come with the same level of risks, but it also opens the potential for abuse and I\u0026rsquo;ve been willing to get rid of that for years. I don\u0026rsquo;t know of ANY use-case of executing a command from aliases that can\u0026rsquo;t be solved from a ~/.forward file in a safer way. It\u0026rsquo;s less convenient and it requires a dedicated user, but it\u0026rsquo;s safer.\n/* this battle needs to be fought ... */ if (xn-\u0026gt;type == EXPAND_FILTER \u0026amp;\u0026amp; strcmp(ep-\u0026gt;mda_user, SMTPD_USER) == 0) log_warnx(\u0026#34;commands executed from aliases \u0026#34; \u0026#34;run with %s privileges\u0026#34;, SMTPD_USER); managing to somehow run mail.local without privileges # Disabling deliveries to privileged users is the right thing to do, however this is not sufficient if mail.local still requires privileges to deliver to gilles. The problem is the requirement for privileges during delivery, not the fact that a user itself is privileged.\nI\u0026rsquo;ve already explained the problem with mail.local and why it requires privileges. We\u0026rsquo;re discussing with other OpenBSD hackers the option to change permissions of /var/mail, the option to use a setgid mail.local and we\u0026rsquo;ll continue exploring similar options.\nThe idea for a group-writeable mail spool was apparently rejected in the past but it may be time to reassess. I don\u0026rsquo;t really have a good intuition about doing this, I don\u0026rsquo;t know why but I\u0026rsquo;ve been wrong before so it\u0026rsquo;s worth investigating this option with the help of other developers.\nThe bottom line is that unless mail.local can be started without privileges, no effort to reduce the potential for escalation will succeed. At the same time, we need to keep in mind that mail.local is not an OpenSMTPD specific component, and changing it must be done with care to avoid breaking other software. If managing to use mail.local without privileges requires making changes to it, we may have to implement an OpenSMTPD specific implementation.\nmbox creation and locking in OpenSMTPD to allow user execution of mail.local # Finally, a proposal I discussed with other hackers was to have OpenSMTPD handle the mailbox creation and locking.\nWe already consider mbox as a special case and we could convert that special case from \u0026ldquo;needs privileges\u0026rdquo; to \u0026ldquo;needs locking\u0026rdquo;. In this scenario, when a delivery uses mbox, the daemon can do the locking and mailbox creation itself before dropping privileges and executing mail.local unprivileged, like any other mail delivery agent. The lock would be removed upon SIGCHLD of the mail delivery agent. This would effectively allow removing the reasons for which mail.local requires privileges.\nAs for mail.local, all it would need to do is allow execution from unprivileged user by removing the explicit check for root it has today. This would still allow to use it with other MTA that expect it to be privileged, while allowing it to run unprivileged with OpenSMTPD. As for locking, no changes would be required since already supports skipping locking through the -L option.\nSome people see this as doing part of the delivery in OpenSMTPD and part in mail.local. I disagree and see this as executing a mail delivery agent in locking barriers: the daemon sets a locked environement around mail delivery agent execution, the mail delivery agent doesn\u0026rsquo;t know or care about the locks. This is word plays but seeing it this way makes more sense to me and doesn\u0026rsquo;t sound hackish at all.\nThis coupled with preventing delivery to root would effectively remove the potential for escalation in mbox.\nClosing words # I don\u0026rsquo;t know where to start, this section will not be as structured as previous ones.\nFirst of all, I\u0026rsquo;d like to thank the people at Qualys. They found a critical bug that could have remained out of sight until exploited by less friendly people. I won\u0026rsquo;t say it is a pleasure to be on my side of the advisory but as unpleasant as it is, it makes the software safer on the long run. I hope they don\u0026rsquo;t find something else, and if they do I hope it\u0026rsquo;s not something as big, but it is a good thing for users to have such competent people studying the code, as painful as it is for the developers when they find something.\nThen, people should use this as a reminder not to put too much trust in it. An MTA is a piece of software that can\u0026rsquo;t drop all privileges, otherwise it would already be done, and at the same time a software that is highly exposed to user-supplied data from untrusted sources. Having an MTA, whichever it is, exposed to the wild and not running on its very own machine will always be a risk. As careful as I can be, I\u0026rsquo;ll make errors and some of these errors will slip in. The only thing I can promise is that I always handle them the way I handled this one, not just fixing the issue but trying to kill the potential.\nThat being said, I will continue discussing the ideas from this article and possibly others to start implementing them as soon as possible. My goal is to kill the potential for escalation in forkmda() before the next major release. I have a few weeks in front of me and there may be some very quick wins to do here that would remove the existing risks.\n","date":"30 January 2020","permalink":"/posts/2020-01-30/opensmtpd-advisory-dissected/","section":"Posts","summary":"TL;DR: - Qualys released an advisory for a bad, bad vulnerability - an MTA is a very bad software to have a vulnerability in - hole was plugged but that's not enough, similar bugs should be mitigated in the future - article discusses what could have prevented escalation despite the bug What happened ?","title":"OpenSMTPD advisory dissected"},{"content":" TL;DR: - brought back libasr to OpenSMTPD, it is no longer an external dependency - libtls-enabled OpenSMTPD is now a thing - documented filters and improved reporting No shiny feature this month, ungrateful work # OpenSMTPD has had quite a few features implemented since its latest major release. As we get closer and closer to the next major release, my work on new features will slow down to focus more on getting the release in shape.\nI spent this month working on plumbing and stuff that\u0026rsquo;s not necessarily very visible to users, as well as multiple features that will NOT be part of the next release but will be committed shortly after release is tagged.\nThis month I focused on simplifying OpenSMTPD portable packaging by removing the need for libasr dependency. I also worked on making libtls-enabled OpenSMTPD possible for portable, a long awaited change that was not trivial and that will be skipped for the next release as it is very invasive.\nlibasr brought back to compat layer # For various purposes, OpenSMTPD relies a lot on DNS lookups and requires them to be non-blocking.\nIn May 2009, because we didn\u0026rsquo;t want threading in the daemon, a former developer implemented this using a fork()-based approach. In November 2010, I replaced the fork()-based non-blocking resolver with asr, a then experimental asynchronous resolver written by eric@, which uses an asynchronous-friendly interface that can be used with a multiplexer. In other words, asr allows DNS lookups to happen in parallel without multi-processing or multi-threading, and results to be notified through select(), poll() or similar interfaces (libevent in the case of OpenSMTPD).\nAs asr is not a standard interface, OpenSMTPD initially shipped it with its compat layer so it could be built on other BSD \u0026amp; Linux. Back then, the release process didn\u0026rsquo;t include major / minor / portable-only releases, and asr was still a moving target with bugs being spotted every few weeks. This led to a very annoying situation as bugs in asr required a new OpenSMTPD release, sometimes a few days apart.\nWe decided to split asr from OpenSMTPD and have it as a standalone library, with its own releases. This was a bit inconvenient because it meant package maintainers had to to maintain two packages, one for OpenSMTPD and one for asr, but it allowed us to decorellate both releases which was very helpful at that point.\nYears passed, OpenSMTPD improved its release management and asr stabilized so much that four years passed without a need for release, but this split was not reconsidered\u0026hellip; until recently.\nIn 2019, after filters were released, I started working heavily on improving our portable layer which suffered multiple shortcomings. While discussing with eric@, we came up with the conclusion that asr should now be brought back to OpenSMTPD compat layer: unlike years ago, rolling a portable-specific release to fix a bug in the compat layer is something we do, and it is now inconvenient for us to have the two split apart.\nThe compat layers had diverged a bit so I did the work to bring back asr to OpenSMTPD, conditionally compiling it if not available on the system. Starting with the next major version, 6.7.0 due sometime in April/May, the portable release of OpenSMTPD will no longer require a libasr dependency and --with-libasr configure flag. It will autodetect if it is available on the system and conditionally build the compat layer one if it isn\u0026rsquo;t.\nlibtls \u0026hellip; for systems without LibreSSL # To quote the LibreSSL home page:\nLibreSSL is a version of the TLS/crypto stack forked from OpenSSL in 2014, with goals of modernizing the codebase, improving security, and applying best practice development processes.\nAmong the improvements that LibreSSL brought was a new libtls library, described as follow:\nlibtls: a new TLS library, designed to make it easier to write foolproof applications.\nIndeed, as someone who started using OpenSSL and libssl as a student during the 2000\u0026rsquo;s, I can attest that using the library is overly complex and writing foolproof applications is a challenge. I won\u0026rsquo;t expand more because this post is not about OpenSSL bashing. Lets just say that it\u0026rsquo;s easy to write code that works using libssl, but it\u0026rsquo;s very hard to know if that code is correct or overlooking an option or a flag somewhere. Part of this is because crypto is not easy and you must know what you\u0026rsquo;re doing, but a bigger part of this is because the API doesn\u0026rsquo;t make it any easier: its attempt at being extremely flexible and allowing a developer to do pretty much anything provides multiple ways for a developer to fail and shoot their own feet.\nThe libtls interface is a wrapper around libssl which exposes a much simpler interface. It provides sane defaults, so that missing something somewhere doesn\u0026rsquo;t result in a less secure setup, but it also provides less possibilities of tweaking everything which is often better for most people. This results in TLS code that\u0026rsquo;s considerably smaller, easier to understand and less error-prone.\nI have had in my plans to convert OpenSMTPD to libtls since it was first released, but couldn\u0026rsquo;t switch to that interface because it requires LibreSSL and many systems still do not provide a package for it: a switch to libtls would essentially kill OpenSMTPD portable. Another option could be to build a standalone libtls that carries LibreSSL\u0026rsquo;s libssl, which is just what reyk@ did. His approach is a very good trade-off and I hope it catches with his standalone libtls becoming widespread.\nIn the meantime, OpenSMTPD cannot use libtls interface without breaking on systems that didn\u0026rsquo;t package his work and\u0026hellip; experience shows it might take time. A time during which we\u0026rsquo;re still forced to use libssl interface, writing complex code that no one is confident reviewing. This is highly annoying because multiple features were put on hold until the libtls switch, some have been deferred for so long that its a bit difficult to defer them some more. At some point we need to either implement them using the libssl interface, which will take time and efforts for code that we don\u0026rsquo;t intend to keep, or find a way to use the libtls interface with OpenSSL\u0026rsquo;s libssl as a \u0026ldquo;degraded\u0026rdquo; mode.\nSince libtls is a wrapper around libssl, you\u0026rsquo;d figure that this shouldn\u0026rsquo;t be too hard BUT LibreSSL and OpenSSL diverged on a few things, the libssl interface now has a few differences in terms of structures, functions available and options to these functions. You can\u0026rsquo;t just grab libtls and build it with OpenSSL, it won\u0026rsquo;t for a variety of reasons.\nI didn\u0026rsquo;t want to spend time working on an OpenSSL version of a standalone libtls: I believe there are good reasons why LibreSSL was forked and I\u0026rsquo;d rather see more people use LibreSSL than endorsing OpenSSL myself. So\u0026hellip; I found a middle-ground which seemed the most pramatic to me.\nI have brought libtls from latest LibreSSL into the compat layer of OpenSMTPD, then did the strict minimum to allow it to build and link with OpenSSL to cover OpenSMTPD\u0026rsquo;s needs and nothing more. This allowed me to convert OpenSMTPD to the libtls interface simplifying an awesome lot of code, while being able to build on any system wether it ships LibreSSL or not.\nSo how does this differ from a standalone libtls ?\nOpenSMTPD\u0026rsquo;s configure will detect if LibreSSL is installed and use its libtls if available. If it\u0026rsquo;s not available, it will detect if a libtls is available (reyk@\u0026rsquo;s work, for example), then if neither one is available it will build a dumbed down version that contains only what OpenSMTPD needs. Using that version may not work on another project. This has the benefit that LibreSSL is always prioritized if available, while still allowing us to switch to libtls if it isn\u0026rsquo;t. This is not ideal as it adds complexity to the compat layer in OpenSMTPD, but this complexity allows unlocking the libtls conversion which will considerably simplify OpenSMTPD itself, so I think it is a fair choice.\nThe work to build that compat libtls is done and so is the conversion of OpenSMTPD to the libtls interface. It is very invasive and, like any huge work, it has a potential for introducing regressions. Given that there\u0026rsquo;s only a few weeks before the next major release, I won\u0026rsquo;t merge this for the next major release but skip it so we have a full development cycle to spot shortcomings.\nThe next major release of OpenSMTPD, version 6.7.0 due April/May, will still use the libssl interface but later releases starting with version 6.8.0, due October/November, will use the libtls interface.\nImplemented multiple feature requests # Building on top of my libtls work, I have implemented several of the features that were requested and left pending waiting for the switch. They are committed in separate branches and waiting for the libtls work to be merged:\nselectable ciphers per listener selectable curves per listener selectable TLS protocol version per listener selectable SNI per listener (they\u0026rsquo;re currently global) Other features are being worked on but not finished yet:\nOCSP check certificate fingerprint verifications Started documenting the filter API # I have also committed a smtpd-filters(7) man page, which doesn\u0026rsquo;t get installed yet, that documents the protocol and events.\nThe feedback so far has been good and resulted in a few changes, but having more people read it and comment would be nice.\nMore smtp-out reporting improvements # I discussed smtp-out reporting in my last report, but it wasn\u0026rsquo;t finished and didn\u0026rsquo;t cover all events.\nI spent time to get this done so that we have events parity between smtp-in and smtp-out. If you enable reporting for smtp-in and smtp-out then accept a mail for relaying, you will first see the events happening from smtp-in as you accept the mail (server), then see the same events from smtp-out as you relay the mail (client).\nThe message id and transaction id are preserved so that it is possible to track the path of an envelope through its entire life.\nWhat next ? # I\u0026rsquo;m contemplating mda reporting, more on that in a future report.\nI started learning Go by writing OpenSMTPD filters a few months ago and now I\u0026rsquo;m learning Rust for other purposes. Half-way through a book, I will probably work on a real project to get my hands dirty. I\u0026rsquo;m unsure what that project will be yet as I have a few ideas in my sleeves, but if I come up with anything valuable I\u0026rsquo;ll share it.\nI will write another article next week to discuss some work that could not make it into this report.\n","date":"22 January 2020","permalink":"/posts/2020-01-22/january-2020-opensmtpd-work-libasr-and-libtls/","section":"Posts","summary":"TL;DR: - brought back libasr to OpenSMTPD, it is no longer an external dependency - libtls-enabled OpenSMTPD is now a thing - documented filters and improved reporting No shiny feature this month, ungrateful work # OpenSMTPD has had quite a few features implemented since its latest major release.","title":"January 2020: OpenSMTPD work - libasr and libtls"},{"content":" TL;DR: - le greylisting est une bonne idée - ce n'est pas très pratique aujourd'hui - beaucoup de gens se passent du greylisting ou trouvent des contournements - le SPF-aware greylisting rend le greylisting utilisable à nouveau Shout out a mes sponsors ❤️ # Un GRAND MERCI à mes sponsors sur github et patreon: votre soutien est grandement apprécié !\nOù est-ce que j\u0026rsquo;ai déjà lu ça ? # Cet article est une traduction de l\u0026rsquo;article \u0026ldquo; SPF-aware greylisting and filter-greylist\u0026rdquo;, publié sur ce blog début Décembre.\nC\u0026rsquo;est mon troisième article rédigé en français depuis des années, je vous demande donc un peu d\u0026rsquo;indulgence : si vous trouvez des fautes, vous pouvez me les remonter pour que je les corrige, ou faire une pull request pour les techies.\nN\u0026rsquo;hésitez pas à partager la publication à l\u0026rsquo;aide des icônes en fin d\u0026rsquo;article et à la commenter \u0026lt;3\nBonne lecture !\nLes erreurs SMTP en quelques mots # SMTP est un protocole fail-safe qui essaie de son mieux de garantir que les messages ne se perdent pas en transit. Parmi les différents mécanismes et exigences du standard on trouve l\u0026rsquo;utilisation d\u0026rsquo;échecs temporaires.\nConcrètement, un nœud SMTP a deux états finaux pour un message : soit le message est distribué et le nœud destination le détient, soit il n\u0026rsquo;est PAS distribué et le nœud destination ne le détient pas. Dans le second cas, le protocole SMTP exige que le dernier nœud en charge notifie l\u0026rsquo;émetteur qu\u0026rsquo;une erreur est survenue. Soit en rejetant le message lors de la session, soit à l\u0026rsquo;aide d\u0026rsquo;une mail d\u0026rsquo;erreur différé (aussi connu sous le nom de MAILER-DAEMON).\nEnfin, il y a un troisième état qui n\u0026rsquo;est pas final : l\u0026rsquo;échec temporaire. Il couvre tous les cas d\u0026rsquo;erreurs qui ont pu temporairement empêcher la distribution… mais qui pourraient tout aussi bien ne plus se produire si quelque chose était corrigé sur le nœud de destination.\nContrairement aux mails distribués, que les destinataires voient, et aux mails échoués, que les émetteurs voient, les mails en échec temporaire ne sont généralement pas vus par les utilisateurs. Ils sont traités entre les Mail eXchangers (MX) qui tentent de se les retransmettre en arrière-plan. Il ne sont généralement vus par les utilisateurs que lorsque les tentatives finissent par réussir ou qu\u0026rsquo;elles échouent définitivement après que leur temps de vie a expiré.\nDans le premier cas, le destinataire va recevoir le message un peu plus tard que prévu, le plus souvent sans même avoir su que des tentatives de retransmission ont eu lieu. Dans le second cas, l\u0026rsquo;émetteur recevra une notification lui indiquant que, malgré les retransmissions, le message n\u0026rsquo;a pas pu être distribué dans le délai imparti.\nLes stratégies de retransmission en cas d\u0026rsquo;échec temporaire sont gérées différemment dans les différents logiciels, mais vous pouvez postuler qu\u0026rsquo;il y aura une retransmission dans les secondes ou minutes qui suivent un échec temporaire. Une approche populaire, l\u0026rsquo;incrément quadratique du délai, crée un délai initialement court mais qui augmente avec le nombre de tentatives en échec de sorte que plus un hôte est injoignable longtemps, plus les tentatives vers cet hôte sont espacées.\nLes échecs temporaires arrivent tout le temps, c\u0026rsquo;est quelque chose de normal dans le protocole SMTP, et en regardant les logs on verra très souvent des lignes telles que :\n4b3a6c195c1f6010 mta delivery evpid=8f26ea98359eccea from=\u0026lt;misc+bounces-4541-[redacted]=tin.it@opensmtpd.org\u0026gt; to=\u0026lt;[redacted]@tin.it\u0026gt; rcpt=\u0026lt;-\u0026gt; source=\u0026quot;45.76.46.201\u0026quot; relay=\u0026quot;62.211.72.32 (smtp.tin.it)\u0026quot; delay=0s result=\u0026quot;TempFail\u0026quot; stat=\u0026quot;421 \u0026lt;[redacted]@tin.it\u0026gt; Service not available - too busy\u0026quot; Il ne s\u0026rsquo;agit pas d\u0026rsquo;erreurs que l\u0026rsquo;on ne trouve que chez des petits serveurs personnels. Je connais deux Big Mailer Corps qui ont des machines en échec temporaire à peu près en permanence, ils ont juste suffisamment de MX pour qu\u0026rsquo;une retransmission ait peu de chances de taper le même MX deux fois de suite. Les mails sont généralement distribués à la première ou seconde tentative.\nPetit secret rien qu\u0026rsquo;entre nous : certains Big Mailer Corps se servent volontairement de ces échecs temporaires pour évaluer la réputation des émetteurs. Ils génèrent ces erreurs à des moments clefs qui leurs permettent certaines analyses sur lesquelles je reviendrai dans un futur article.\nIl y a beaucoup de choses à dire sur les erreurs SMTP, mais résumons l\u0026rsquo;essentiel pour cet article :\nIl existe un mécanisme dans le protocole SMTP qui permet à un MX destinataire de demander à un MX émetteur de retransmettre un message. La retransmission a lieu en arrière-plan, sans aucune action des utilisateurs, et elle peut provoquer un délai dans la distribution.\nLe greylisting expliqué # Le spam est une industrie du volume.\nLes spammeurs ne sont pas intéressés par cibler des destinataires spécifiques. Ils sont intéressés par cibler un grand nombre de destinataires pour que statistiquement cela augmente le nombre de destinataires qui se feront avoir.\nJe ne vais pas trop creuser ce sujet parce que c\u0026rsquo;est l\u0026rsquo;objet d\u0026rsquo;un autre article en rédaction, néanmoins vous devez avoir à l\u0026rsquo;esprit qu\u0026rsquo;il y a différents types de spammeurs. Alors que certains utilisent des MX tout ce qu\u0026rsquo;il y a de plus normaux qui honorent les demandes de retransmissions, beaucoup d\u0026rsquo;autres utilisent des outils ou des MX customisés pour NE PAS retransmettre.\nHonorer les retransmissions ralentit considérablement l\u0026rsquo;émission pour un résultat incertain. Garder trace de tous les destinataires en échec temporaire pendant des heures ou des jours pour pouvoir les retenter est loin d\u0026rsquo;être intéressant, surtout s\u0026rsquo;il n\u0026rsquo;y a pas de certitude que la retransmission n\u0026rsquo;aboutira pas en un échec permanent de type \u0026ldquo;cet utilisateur n\u0026rsquo;existe pas\u0026rdquo;. Il faut conserver un état pour se rappeller des tentatives en échec et conserver les destinataires dans la queue d\u0026rsquo;envoi. Directement ou indirectement cela impacte la capacité à envoyer en masse.\nL\u0026rsquo;idée derrière le greylisting est très simple :\nLe greylisting déclenche un échec temporaire pour les MX auquels vous ne faites pas encore confiance et garde trace des tentatives de retransmission. Si la retransmission a lieu trop tôt elle est ignorée, les spammers ne peuvent pas faire une retransmission immédiate et doivent garder un état. Si la retransmission a lieu trop tard… et bien c\u0026rsquo;est trop tard, l\u0026rsquo;implémentation peut faire un bon nombre de choses depuis l\u0026rsquo;ajout en blacklist à la dégradation de la réputation, en passant par un nouveau tour de greylisting, encore et encore et encore, … Le résultat est que pour passer le greylisting, les spammeurs sont contraints de conserver un état et de se comporter comme des implémentations correctes du point de vue de la RFC. C\u0026rsquo;est en conflit direct avec l\u0026rsquo;économie de l\u0026rsquo;envoi de masse.\nD\u0026rsquo;un autre côté, les MX qui respectent la RFC vont naturellement retenter de nombreuses fois et finir par faire une tentative dans la bonne fenêtre de temps pour être whitelistés.\nLe greylisting ne permet pas de se prémunir des spammeurs qui ont compromis des MX légitimes, ni de ceux qui envoient à l\u0026rsquo;aide d\u0026rsquo;outils qui acceptent le coût de la retransmission. Il permet par contre d\u0026rsquo;éliminer la plus grosse partie du spam : celle qui provient de machines infectées et transformées en bots à spam, ou de scripts dont le seul but est d\u0026rsquo;envoyer un maximum possible avant qu\u0026rsquo;un hébergeur ou un administrateur ne s\u0026rsquo;en rende compte.\nComment le greylisting fonctionne habituellement # Il y a différentes solutions de greylisting et elles ne fonctionnent pas toutes pareil.\nL\u0026rsquo;idée générale est que vous avez une base de donnée qui garde trace des demande de retransmissions et de la fenêtre de temps durant laquelle une tentative est considérée comme valide pour un MX. En revanche, la manière dont sont comptabilisées les tentatives d\u0026rsquo;un MX varie d\u0026rsquo;une solution à une autre. Certaines solutions ne regardent que l\u0026rsquo;adresse IP source, alors que d\u0026rsquo;autres vont également regarder le MAIL FROM et/ou le RCPT TO. Dans certains cas que j\u0026rsquo;ai pu observer, la solution regardait également l\u0026rsquo;entête Message-ID.\nLa solution spamd(8) d\u0026rsquo;OpenBSD va par exemple garder trace de l\u0026rsquo;adresse IP source, le nom d\u0026rsquo;hôte utilisé lors de la phase d\u0026rsquo;identification HELO/EHLO, l\u0026rsquo;émetteur d\u0026rsquo;envelope (MAIL FROM) et le destinataire d\u0026rsquo;envelope (RCPT TO). Pour passer le greylisting, un MX doit faire une retransmission dans la bonne fenêtre de temps, en provenance de la même IP source, en utilisant le même nom d\u0026rsquo;hôte à l\u0026rsquo;identification, depuis le même émetteur et pour le même destinataire.\nLe problème avec le greylisting et les Big Mailer Corps # Pour de petits émetteurs, le greylisting marche très bien parce que la plupart utilisent le même MX pour le trafic entrant et sortant. Le MX vous contacte, il lui est demandé de retenter plus tard, il est accepté lorsqu\u0026rsquo;il revient une minute plus tard.\nEnsuite, vous avez Gmail / Yahoo / Outlook / … qui non seulement disposent de MX différents pour le trafic entrant et sortant, mais ont également des douzaines et des douzaines de MX sortants. La situation devient la suivante : un MX vous contacte, il lui est demandé de retenter plus tard, mais vous ne voyez jamais de retransmission de ce MX puisque c\u0026rsquo;est un autre qui a retenté et à qui il a été demandé de retenter plus tard, etc…\nLe greylisting avec ces hôtes est inutilisable et résulte en des mails qui peuvent arriver avec plusieurs heures ou jours de retard, s\u0026rsquo;ils n\u0026rsquo;expirent pas tout simplement avant de vous atteindre.\nMais en même temps, le greylisting empêche une grande quantité de nuisibles de vous atteindre en tuant le trafic de presque tous les scripts et les ordinateurs compromis. Donc pour le conserver, un grand nombre d\u0026rsquo;opérateurs de mails mettent en place des bricolages pour pouvoir continuer à utiliser le greylisting pour les petits émetteurs, tout en ne l\u0026rsquo;appliquant pas pour les Big Mailer Corps.\nStratégies de contournement # J\u0026rsquo;utilise le pluriel mais en réalité toutes les stratégies reposent sur la même idée : le whitelisting.\nLe contournement consiste à trouver la liste des adresses IP des Big Mailer Corps pour leur donner un passe droit et leur permettre d\u0026rsquo;éviter le greylisting.\nEst-ce que cela met en défaut le greylisting ? Pas vraiment. Le greylisting a pour but d\u0026rsquo;empêcher les MX ne respectant pas la RFC de vous atteindre, mais les Big Mailer Corps utilisent des MX qui respectent la RFC. Lorsque la retransmission a bien lieu plusieurs fois par le même MX sortant, ils passent le test du greylisting sans souci.\nTrès bien, ajoutons-les à la whitelist alors… mais comment ?\nPendant un moment, j\u0026rsquo;ai utilisé une approche naïve consistant à whitelister le /24 de n\u0026rsquo;importe quel MX de Big Mailer Corps qui se retrouvait greylisté. Après quelques temps, ma liste était suffisamment grande pour que de nouveaux MX soient rarement hors de ma whitelist. C\u0026rsquo;était pénible, ça ne couvrait pas le cas de nouveaux MX me contactant depuis de nouvelles plages d\u0026rsquo;adresses IP, et j\u0026rsquo;ai fini par aggréger les listes de différentes personnes à la mienne pour être sûr d\u0026rsquo;en avoir le plus possible. Le résultat était une whitelist en augmentation constante.\nCe n\u0026rsquo;était pas très malin parce que la plupart des Big Mailer Corps utilisent SPF. Ils ont un enregistrement DNS qui liste avec quelles adresses IP ils nous contactent, de façons à ce que lorsque l\u0026rsquo;on reçoit un mail avec une adresse e-mail en provenance de chez eux, on puisse vérifier que c\u0026rsquo;est bien un de leurs MX autorisé qui l\u0026rsquo;a transmis. Avec ça en tête, j\u0026rsquo;ai écrit spfwalk (maintenant une sous-commande de smtpctl) en Janvier 2018, un outil qui permet de faire une énumération dans l\u0026rsquo;enregistrement SPF d\u0026rsquo;un domaine et extraire un maximum d\u0026rsquo;adresse IP possible. Il y a un article en anglais à propos de spfwalk sur ce blog si ça vous intéresse. L\u0026rsquo;outil avait pour vocation de remplacer mon approche naïve, il est loin d\u0026rsquo;être parfait, j\u0026rsquo;ai fait mon mieux.\nLa façon dont fonctionne SPF permet de vérifier facilement si une adresse IP est autorisée, mais ne permet pas d\u0026rsquo;extraire facilement une liste d\u0026rsquo;adresse pour une vérification ultérieure. Par exemple, certaines politiques SPF s\u0026rsquo;attendent à une résolution DNS inverse de l\u0026rsquo;adresse IP source pour voir si le nom d\u0026rsquo;hôte corresponds au domaine d\u0026rsquo;envoi. Dans ce type de politique SPF, il n\u0026rsquo;y a aucune adresse IP à extraire pour mettre dans une whitelist. Donc… l\u0026rsquo;énumération SPF fonctionne à peu près tant que l\u0026rsquo;on ne tombe pas sur les politiques qui ne peuvent pas être énumérées. Pas de bol pour nous, l\u0026rsquo;un des Big Mailer Corps est dans le périmètre.\nUn grand nombre de personnes sont très contentes avec l\u0026rsquo;énumération SPF. Elle améliore considérablement la situation, mais elle reste un bricolage à mon sens et je n\u0026rsquo;utilise plus cet outil parce qu\u0026rsquo;il ne me semble pas être la bonne approche.\nLa preuve de concept filter-greylist # Le mois dernier, j\u0026rsquo;ai écrit comme preuve de concept pour OpenSMTPD un filtre qui utilise une approche différente et qui est disponible sur Github.\nJe vais juste clarifier avant tout que je ne prétends pas avoir inventé cette méthode, elle m\u0026rsquo;a juste semblée être une bonne approche alors je l\u0026rsquo;ai implémentée. Il se peut qu\u0026rsquo;on trouve d\u0026rsquo;autres implémentations de cette même idée ailleurs, j\u0026rsquo;avoue que je n\u0026rsquo;ai pas vraiment cherché très fort (pour ne pas dire \u0026ldquo;du tout\u0026rdquo;). Je ne sais donc pas s\u0026rsquo;il existe d\u0026rsquo;autres implémentations similaires, et si elles existent je ne sais pas si mon implémentation exploite l\u0026rsquo;idée de la même manière.\nL\u0026rsquo;idée de base est que nous ne voulons pas vraiment que les Big Mailer Corps soient exemptés de greylisting, ils en sont exemptés parce que l\u0026rsquo;on ne sait pas considérer tous leurs MX sortants comme identiques dans un test de greylisting. En faisant une énumération SPF pour extraire leurs adresses IP, notre intention est simplement de considérer que toutes les adresses IP de tous leurs MX sortants sont identiques pour nous : elles devraient pouvoir être interchangeables dans un test de greylisting. En d\u0026rsquo;autres termes, ce que l\u0026rsquo;on veut c\u0026rsquo;est un SPF-aware greylisting (greylisting conscient de SPF).\nLa preuve de concept considère qu\u0026rsquo;il y a deux types d\u0026rsquo;émetteurs : ceux qui publient des enregistrements SPF et ceux qui n\u0026rsquo;en publient pas.\nLes MX qui ne publient pas d\u0026rsquo;enregistrement SPF sont greylistés par adresse IP source, comme cela serait le cas avec un greylisting traditionnel. Commes les Big Mailer Corps requièrent que les émetteurs fournissent un enregistrement SPF pour ne pas dégrader leur réputation, et parce que la plupart des gens veulent pouvoir émettre vers les Big Mailer Corps, la plupart des émetteurs légitimes ne devraient pas rentrer dans ce cas. Ne devraient rentrer dans ce cas que des petits émetteurs qui n\u0026rsquo;émettent pas depuis un grand nombre d\u0026rsquo;adresse, et dont le greylisting serait identique qu\u0026rsquo;il soit par adresse IP ou domaine.\nEnsuite, nous avons les MX qui publient des enregistrements SPF, englobant l\u0026rsquo;ensemble des Big Mailer Corps pour des raisons évidentes (difficile d\u0026rsquo;imposer SPF à d\u0026rsquo;autres sans se l\u0026rsquo;imposer à soi-même). Pour ceux-ci, plutôt que d\u0026rsquo;avoir recours a un greylisting par IP source, on procède à un greylisting SPF du domaine.\nEn supposant que ce soit le premier email que nous recevons d\u0026rsquo;un MX, quand une nouvelle connexion arrive, le filtre garde trace de l\u0026rsquo;adresse IP source. La session SMTP progresse jusqu\u0026rsquo;à la création d\u0026rsquo;une transaction SMTP et le client fournit l\u0026rsquo;émetteur d\u0026rsquo;envelope (MAIL FROM) :\nS: 220 in.mailbrix.mx ESMTP OpenSMTPD C: EHLO localhost S: [...] S: 250 in.mailbrix.mx Hello localhost [127.0.0.1], pleased to meet you C: MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; À ce stade, le filtre peut faire une recherche SPF pour le domaine de l\u0026rsquo;émetteur d\u0026rsquo;enveloppe, poolp.org. S\u0026rsquo;il ne trouve pas d\u0026rsquo;enregistrement SPF, il pourra garder trace de l\u0026rsquo;adresse IP source et demander une retransmission.\nS\u0026rsquo;il trouve un enregistrement SPF, il vérifie que l\u0026rsquo;adresse IP source est valide pour cet enregistrement. Si elle ne l\u0026rsquo;est pas, il peut supposer qu\u0026rsquo;elle ne provient pas du domaine émetteur, le filtre va là encore procéder à un greylisting par adresse IP source. Une approche plus stricte pourrait être de rejeter la session, mais ce n\u0026rsquo;est pas le but d\u0026rsquo;un filtre de greylisting de décider si la violation SPF est acceptable ou non.\nSi en revanche l\u0026rsquo;adresse IP source est valide pour l\u0026rsquo;enregistrement SPF, au lieu de garder trace de l\u0026rsquo;adresse IP source, le filtre garde trace du domaine et demande une retransmission.\nIgnorons dorénavant le cas du greylisting par IP source pour nous concentrer sur le greylisting par domaine.\nLors d\u0026rsquo;une seconde connexion en provenance d\u0026rsquo;un domaine dont le filtre avait gardé trace, si l\u0026rsquo;adresse IP source est valide pour l\u0026rsquo;enregistrement SPF, le filtre va pouvoir valider que le domaine était déjà en demande de retransmission depuis une autre addresse IP et considérer le test comme réussi. En d\u0026rsquo;autres termes plus simples, si Gmail vient d\u0026rsquo;une adresse IP, il est autorisé à venir d\u0026rsquo;une autre addresse IP dès lors que les deux sont déclarées dans l\u0026rsquo;enregistrement SPF.\nCette résolution SPF en live permet de garantir qu\u0026rsquo;il n\u0026rsquo;y a pas besoin de whitelister au préalable les adresses IP. À la place il est possible de whitelister des domaines indépendamment des adresses IP avec lesquelles ils nous contacterons. Cette façon de fonctionner est de plus compatible avec les domaines qui ne fournissent pas d\u0026rsquo;enregistrement SPF, et qui dégradent en un greylisting par IP source.\nIl n\u0026rsquo;est pas possible d\u0026rsquo;implémenter cela dans un daemon comme spamd(8), la décision de greylister ou de laisser la session progresser est prise au moment du MAIL FROM, là ou la redirection spamd(8) est décidée par le firewall à la connexion.\nEst-ce que ça peut être amélioré ? # Je ne sais pas trop mais je ne pense pas que l\u0026rsquo;on puisse radicalement améliorer l\u0026rsquo;idée.\nIl y a surement quelques petites améliorations à faire à la marge, mais je ne pense pas que l\u0026rsquo;on puisse faire de grosses grosses avancées : un SPF-aware greylisting résout le problème du greylisting chez les Big Mailer Corps, ni plus, ni moins. Aujourd\u0026rsquo;hui avec ce filtre, je ne vois aucune différence de traitement entre Gmail et le petit domaine du coin. Les deux passent le greylisting assez rapidement et sans que l\u0026rsquo;on ne se rende vraiment compte qu\u0026rsquo;il y a eu retransmission.\nEn gardant en tête que le spam est une industrie du volume, je pense que du travail plus intéressant peut être fait en augmentant le coût de l\u0026rsquo;envoi de masse : de la même manière que les spammeurs n\u0026rsquo;aiment pas la retransmission, ils n\u0026rsquo;aiment PAS DU TOUT les hôtes lents parce qu\u0026rsquo;ils ont un impact ÉNORME sur la capacité à distribuer et sur la taille de la queue. J\u0026rsquo;ai déjà implémenté des fonctionnalités dans ce sens au sein d\u0026rsquo;autres filtres en corrélant les temps de réponse à la réputation d\u0026rsquo;un domaine.\nSi je passe plus de temps à essayer de faire d\u0026rsquo;OpenSMTPD une cible difficile pour les spammeurs, il est certain que mes prochains travaux seront dans la lignée des sanctions par délais.\n","date":"26 December 2019","permalink":"/posts/2019-12-26/spf-aware-greylisting-et-filter-greylist/","section":"Posts","summary":"TL;DR: - le greylisting est une bonne idée - ce n'est pas très pratique aujourd'hui - beaucoup de gens se passent du greylisting ou trouvent des contournements - le SPF-aware greylisting rend le greylisting utilisable à nouveau Shout out a mes sponsors ❤️ # Un GRAND MERCI à mes sponsors sur github et patreon: votre soutien est grandement apprécié !","title":"SPF-aware greylisting et filter-greylist"},{"content":" TL;DR: - wrote, reworked and translated multiple articles this month - got some goodies ready for my patrons - lots of work in OpenSMTPD's grammar, documentation and filters protocol WARNING:\nExamples of code and configuration that appear in this article are here to help illustrate and explain development stages of my work.\nThey are subject to changes and must not be considered as user documentation. By the time you\u0026rsquo;re reading this, they will likely no longer work or reflect reality.\nLoooots of minor stuff here and there # These last two months, I could carry Jules in a baby wrap and write code while he was asleep. He then unilateraly decided that I\u0026rsquo;m his mule now and I\u0026rsquo;m no longer allowed to stop walking when carrying him. Even standing still while writing code for more than five minutes straight is strictly forbidden.\nAs you can imagine, this causes lots of context switching with code sessions interrupted every few minutes, therefore he\u0026rsquo;s now known as the interrupt storm.\nAs a result I could not finish all of what I started working on, stuff that won\u0026rsquo;t make it into this report but which I intend to finish by the end of January for the next report.\nArticles rework and translations # This month, I have written an article regarding my work on \u0026ldquo; SPF-aware greylisting and filter-greylist\u0026rdquo;.\nIn addition, I have reworked my \u0026ldquo; Setting up a mail server with OpenSMTPD, Dovecot and Rspamd\u0026rdquo; article and removed the political aspect of it to make it a standalone article, \u0026ldquo; Decentralised SMTP is for the greater good\u0026rdquo;. Both of them were translated in French, \u0026ldquo; Mettre en place un serveur de mail avec OpenSMTPD, Dovecot et Rspamd\u0026rdquo; and \u0026ldquo; Decentralisons SMTP pour le bien commun\u0026rdquo;.\nThese were my last articles for this year.\nGoodies for my sponsors # I have set up a reward system for my sponsors, nothing too impressive, but I have received the first goodies, stickers and mugs, a few days ago.\nI don\u0026rsquo;t intend to start dispatching anything before mid-January, I don\u0026rsquo;t trust shipping stuff around this period of the year. I haven\u0026rsquo;t received the shirts yet anyways ;-)\nOpenSMTPD work # I thought I\u0026rsquo;d be slacking, but a lot of work has been put into OpenSMTPD since my last report, far more than I thought I actually did. Go figure.\nReworking match rules a bit # As you probably know, OpenSMTPD uses a ruleset to match envelopes:\nmatch from local for local action \u0026#34;in\u0026#34; match from local for any action \u0026#34;out\u0026#34; These rules may have a set of criterias to refine them further:\nmatch from local mail-from \u0026#34;gilles@poolp.org\u0026#34; for local action \u0026#34;in-alternative\u0026#34; match from local for any rcpt-to \u0026#34;eric@faurot.net\u0026#34; action \u0026#34;out-alternative\u0026#34; match from local for local action \u0026#34;in\u0026#34; match from local for any action \u0026#34;out\u0026#34; Because it has an implicit \u0026ldquo;local\u0026rdquo; behavior, rules may skip from local and for local, which makes ruleset more concise:\n# match from any for local action \u0026#34;in\u0026#34; match from any action \u0026#34;in\u0026#34; # match from local for any action \u0026#34;out\u0026#34; match for any action \u0026#34;out\u0026#34; The problem with implicit \u0026ldquo;local\u0026rdquo; # When I first started working on OpenSMTPD, mail operators kept mentionning two main problems with mail servers:\nthe configuration files were crazy difficult to understand and maintain it was way too easy to accidentally create an open relay for spammers I made it a project goal to have the most concise configuration file, providing sane defaults and removing anything unnecessary, so that the configuration would be easy to understand and so that it would take explicitely typing from any for any to create an open relay. One of the \u0026ldquo;cool\u0026rdquo; features was the use of implicit local, so that as explained above, you could make your configuration shorter due to all rules assuming local by default.\nBack then, other developers were also trying to get the configuration shorter, so sometimes I would get complaints that their configuration was taking four lines instead of three, or that the lines were taking ten keywords instead of eight, and I would try to find a way to express the grammar differently. I will fully take the blame for this one, at some point if you\u0026rsquo;re competing with other MTA that are using M4 or plaintext configuration files that have hundreds of keys, trying to remove one line out of four is a pissing contest.\nWith advances in OpenSMTPD, and a ruleset that became more and more flexible with many more matching criterias, trying to be as concise as possible to save two keywords became unproductive.\nIn some cases, writing rules like this can be confusing and result in errors like this one:\nmatch auth for any action \u0026#34;out\u0026#34; where a lot of users mistakenly assume that this will match all authenticated sessions for any destination and relay\u0026hellip; but since auth is only a criteria that specifies further the rule and since rules are local by default, this really translates to:\nmatch from local auth for any action \u0026#34;out\u0026#34; a rule that really only matches authenticated sessions coming from a local interface.\nThis is not the case that confuses users, another error I saw happen multiple times is the following one:\nmatch from any rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;out\u0026#34; which due to implicit \u0026ldquo;local\u0026rdquo; translates to:\nmatch from any for local rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;out\u0026#34; unless poolp.org is the machines\u0026rsquo; hostname, this will cause mails for gilles@poolp.org to be rejected, because the for criteria didn\u0026rsquo;t match much.\nThe proper way to write the rule would be:\nmatch from any for domain \u0026#34;poolp.org\u0026#34; rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;out\u0026#34; which everyone keeps simplifying as:\nmatch from any for any rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;out\u0026#34; just because rcpt-to already acts as a whitelist of recipients, so having to maintain a list of corresponding domains is overkill.\nThis highlights an issue which is that the grammar should describe the user intent, and the intent is very clear with rules like these:\nmatch auth for any action \u0026#34;out\u0026#34; match from any rcpt-to \u0026#34;gilles@poolp.org\u0026#34; action \u0026#34;in\u0026#34; The fact that it doesn\u0026rsquo;t work like they think is a problem. A problem that is caused by the implicit local behavior, because if from and for always had to be specified then such errors would not be possible.\nOh noes, not another grammar change ! # Nope, don\u0026rsquo;t worry, the grammar is correct as it is.\nWhat is incorrect is the allowing of implicit behaviors, like skipping from or for. These should be explicit and mandatory, the shortcomings of saving two keywords are far more annoying than the benefits.\nFurthermore, making them mandatory actually allows for shorter rules which are not doable today, because the implicit behaviors makes them confusing.\nSo we have decided to go full explicit from now on, the default configuration file will now provide both from and for even for local uses, using implicit behaviors will result in warnings at startup, then in a future release we will make it mandatory to declare them.\nThis means that ALL matching rules will ALWAYS have both a from and for, how does that make things shorter ?\nfrom used to be for source and for for domain # Initially, the from keyword was used to declare the source of a connection, or local if it could originate from any locally bound address or the Unix socket. The for keyword was used to declare a destination domain, or local if it could be destined for any domain known locally, usually localhost and domains obtained through gethostname() or the /etc/mail/mailname file. The special keyword all could be used to encompass any address when used with from and any domain when used with for.\nAs time passed by, we started adding support for other criterias, and ruleset could be expressed with intents that were no longer considering a source address or a destination domain. The mail-from and rcpt-to criterias are perfect example of this, they are often used with the intent of providing a whitelist of e-mail addresses. A lot of people use constructs such as:\nmatch from any mail-from \u0026lt;senders\u0026gt; [...] match from any for any rcpt-to \u0026lt;rcpts\u0026gt; [...] What they really want is to match a sender or recipient e-mail address regardless of the source. If we take a step back to get a larger picture, they want to use mail-from as an origin and rcpt-to as a destination, the source address and destination domain are set to any because they are\u0026hellip; just a criteria that makes no sense in these rules and that they want to discard.\nWhen you see things this way, it makes you reevaluate how from and for should be used, they are not source address and destination domain related but origin and destination related in a wider sense. Luckily for us the shift in paradigm is retro-compatible with previous grammar. Rulesets I write today will still be valid and work the same way tomorrow, however new constructs become available that better depicts the user intent, with no more ambiguous cases and shorter syntax for most cases.\nHow so ?\nLet\u0026rsquo;s take the examples above, what the user really expresses when writing:\nmatch from any mail-from \u0026lt;senders\u0026gt; [...] is the following:\nmatch from mail-from \u0026lt;senders\u0026gt; [...] And what the user really wants to express when writing:\nmatch from any for any rcpt-to \u0026lt;rcpts\u0026gt; [...] is the following:\nmatch from any for rcpt-to \u0026lt;rcpts\u0026gt; [...] Specifying any is not invalid, it is just redundant in both of these cases, but is still valid because using a source address criteria in conjunction to a sender or recipient e-mail address is still a valid use-case:\nmatch from src 192.168.1.0/24 mail-from \u0026lt;senders\u0026gt; [...] match from local mail-from \u0026lt;senders\u0026gt; [...] With that in mind, the following rules can now be expressed to match authenticated users regardless of their source address:\nmatch from auth for any [...] match from auth gilles@poolp.org for any [...] match from auth \u0026lt;users\u0026gt; for any [...] And just like in the previous example, it is still possible to filter on specific addresses:\nmatch from src 192.168.1.0/24 auth \u0026lt;users\u0026gt; for any [...] match from src 192.168.2.0/24 auth \u0026lt;users\u0026gt; for any [...] This change took more time convincing others than it took writing, but it is really the right direction. Because it makes the syntax simpler but also because ambiguous cases translate into people mailing me for help, and all of the common mistakes people do just vanish when from and for become explicit and the new constructs are available.\nWhat is fun is that once I got enough okays, I committed the change and I\u0026rsquo;m quite sure that other OpenBSD hackers didn\u0026rsquo;t even realize it as nothing changed for existing setups.\nThis will be part of the OpenSMTPD 6.7.0 release happening sometime around April/May, but it is already available in OpenBSD -current of in the master and portable branches on Github.\nI have converted multiple setups to the next syntax and they are much nicer this way.\nImprove documentation # Various improvements were done to the manual pages of OpenSMTPD.\nI have committed bits of missing information which jmc@ reworded and fixed to get nicer. There\u0026rsquo;s still work to do, most notably documenting how to plug filters and providing examples for DKIM and spam filtering, but the documentation had a lot of redundant bits reworked to make it easier to grasp.\nI spent hours writing an initial version of smtpd-filters(7), a man page describing the filter API for people willing to write their own filters. This is a work in progress and I have not linked it to the build yet, but it explains how things work at the protocol level, the various events that can happen, etc\u0026hellip;\nFrom now on, I\u0026rsquo;ll point people to this page when asked how to get started with filters, it will help me find the parts that need improvement.\nFix a couple protocol issue in filters # There were two issues in the filter API, issues which people would not notice because of their nature.\nThe first one is related to the request/response aspect of the filtering protocol. I had a working proof-of-concept fairly fast last year but it took months to refine the order of fields. I didn\u0026rsquo;t realize that with the reordering of fields, we ended up have two fields that are common to requests and responses swapped.\nWhat this means is that the request had the fields SESSION|TOKEN while responses had the fields TOKEN|SESSION, and since the parsing code was correct the code worked correctly, but from a developer perspective reading through the code or looking at raw protocol lines, this was confusing. The order of fields in responses was swapped to match the order of fields in requests, leading to a protocol version bump.\nThe second issue was related to the smtp-out reporting feature which I was working on. Until now a filter could only be attached to a listen line so it knew for sure it could register smtp-in events. With smtp-out a filter can be attached to a relay action allowing it to receive reporting events for outgoing trafic, but nothing in the protocol would let the filter know where it was attached, so a filter could not know if it had to register for smtp-in or smtp-out events.\nThe protocol has a handshake which allows the server to provide a set of key|value to filters at startup before they register events, so I have made sure the server would let the filters know to which subsystems they were attached so they can register the proper events:\nconfig|subsystem|smtp-in config|subsystem|smtp-out The smtp-out reporting # With the subsystem attachement issue fixed I could complete my work on smtp-out reporting.\nThis resulted in reworking how the smtp and mta layer registered sessions in filters, how reporting events were sent to filters, and which event made some informations available to filters.\nThe smtp-out and smtp-in reporting events are the same, only the direction changes, so there\u0026rsquo;s no protocol change but internally the concept of a session differs between the smtp and mta layer. For instance, a session begins when a client connects for smtp, but a session begins before we connect to a remote server in mta, so the availability of rdns, fcrdns, src and dest addresses doesn\u0026rsquo;t take place at the same timing for both.\nI won\u0026rsquo;t expand much because it\u0026rsquo;s not that interesting, but it required quite a bit of rework and it took me months of work on and off to get to a state where the code was clean and stable. Not all events are generated in smtp-out yet but this will be worked on and we\u0026rsquo;ll be fine before the next release.\nAll of my mail exchangers are now producing an event log for both incoming and outgoing trafic, I\u0026rsquo;m looking forward to exploit these into nice graphs.\nIntroduce a bypass action for builtin filters # The builtin filters are filters that operates within OpenSMTPD and provide a set of simple filtering capabilities. They allow attaching to different phases, matching different criterias and taking various decisions. I won\u0026rsquo;t dive into this because it was already discussed in this blog and is documented in the smtpd.conf(5) man page. There was just one thing missing, the ability to bypass filters in a specific phase.\nFor instance, you could define multiple filters:\nfilter no_rdns phase mail-from match !rdns reject \u0026#34;550 go away\u0026#34; filter no_fcrdns phase mail-from match !fcrdns \u0026#34;550 go away\u0026#34; listen on all filter { no_rdns, no_fcrdns } but you could not have a way to bypass these filters for a set of trusted hosts for example.\nThe bypass action allows a builtin filter to take the decision to not go through the chain but accept the phase, it would allow doing the following:\nfilter trusted phase mail-from match src \u0026lt;trusted_sources\u0026gt; bypass filter no_rdns phase mail-from match !rdns reject \u0026#34;550 go away\u0026#34; filter no_fcrdns phase mail-from match !fcrdns \u0026#34;550 go away\u0026#34; listen on all filter { trusted, no_rdns, no_fcrdns } The lack of a bypass mechanism caused multiple people to ask me for work-arounds, as many use-cases require being able to skip filters for a set of hosts, and OpenSMTPD didn\u0026rsquo;t provide any way to do that.\nThe bypass mechanism was not pushed to proc filters, I\u0026rsquo;m not convinced at this point that it is necessary. Builtin filters reside in the configuration file and are crafted with knowledge of how the filter chain looks like, this is not the case for proc filters so I\u0026rsquo;m unconvinced if this is a useful feature there. I may change my mind but no rush on this anyways.\nWork on multiple filters # All of my filters were adapted to cope with the protocol change described above with the swapped fields. I had to release new versions of all of them, a version that would keep working with the previous fields order and that would work with the new fields order.\nI didn\u0026rsquo;t work on new features but filter-rspamd was contributed to by @freswa, @whataboutpereira and @lfos who also made several contributions to filter-senderscore.\nThese filters live their lives now which is very cool.\nImprovement to filter-greylist # Following a discussion on our IRC channel (#OpenSMTPD @ irc.freenode.net), I made an improvement to filter-greylist which consists in detecting that a session was initiated by a local or authenticated user, and whitelisting the destination for messages.\nNot whitelisting destination doesn\u0026rsquo;t cause huge penalty with SPF-aware greylisting, but having destinations of local users whitelisted means that there\u0026rsquo;s no penalty at all for hosts from which we expect e-mails.\nWhat next ? # No more writing until 2020.\nI will take a few days off from computers then resume writing code maybe next week, hopefully finishing some of my pending works in progress so I can disclose them :-)\n","date":"24 December 2019","permalink":"/posts/2019-12-24/december-2019-opensmtpd-and-filters-work-articles-and-goodies/","section":"Posts","summary":"TL;DR: - wrote, reworked and translated multiple articles this month - got some goodies ready for my patrons - lots of work in OpenSMTPD's grammar, documentation and filters protocol WARNING:","title":"December 2019: OpenSMTPD and filters work, articles and goodies"},{"content":" TL;DR: - Pas de résumé, j'ai passé des heures à traduire, vous allez passer des minutes à lire ;) - OK… J'ai expliqué avec BIEN TROP DE DÉTAILS comment mettre en place un serveur de mail Merci à mes sponsors ! # Un énorme merci aux gens qui me sponsorisent sur patreon ou github.\nOù est-ce que j\u0026rsquo;ai déjà lu ça ? # En Août, j\u0026rsquo;ai publié un petit article intitulé \u0026ldquo; You should not run your mail server because mail is hard\u0026rdquo; (\u0026ldquo;Vous ne devriez pas héberger votre serveur de mail parce que c\u0026rsquo;est dur\u0026rdquo;) qui était, en gros, mon opinion sur les différentes raisons qui poussent les gens à décourager l\u0026rsquo;hébergement de mails. De manière très inattendue, l\u0026rsquo;article est devenu assez populaire, atteignant 100K lectures et continuant à recevoir des visites et des commentaires plusieurs mois après sa publication.\nEn Septembre, j\u0026rsquo;ai publié un autre article beaucoup plus long intitulé \u0026ldquo; Setting up a mail server with OpenSMTPD, Dovecot and Rspamd\u0026rdquo; (\u0026ldquo;Installer un serveur de mail avec OpenSMTPD, Dovecot et Rspamd\u0026rdquo;) qui décrivait pas à pas, de façon très détaillée, les étapes pour installer un serveur de mail complet jusqu\u0026rsquo;à la livraison des e-mails en boîte de réception chez un gros hébergeur de mail américain. L\u0026rsquo;article est devenu lui aussi plutôt populaire, sans pour autant atteindre le niveau du précedent article moins technique et spécifique, mais atteignant 40K lectures et continuant également à recevoir des visites et des commentaires plusieurs mois après la publication. Le contenu que vous vous apprêtez à lire est la traduction de cet article technique.\nC\u0026rsquo;est mon second article rédigé en français depuis des années, je vous demande donc un peu d\u0026rsquo;indulgence : si vous trouvez des fautes, vous pouvez me les remonter pour que je les corrige, ou faire une pull request pour les techies.\nCet article est DENSE parce que je vais vous tenir la main à un niveau frisant l\u0026rsquo;absurdité en expliquant chacune de mes actions et le pourquoi du comment. D\u0026rsquo;un point de vue purement technique, sans explications, l\u0026rsquo;article pourrait probablement tenir en quelques paragraphes.\nN\u0026rsquo;hésitez pas à partager la publication à l\u0026rsquo;aide des icones en fin d\u0026rsquo;article et à la commenter \u0026lt;3\nAh, et tant que je vous ai sous la main, je vous recommande chaudement la lecture de mon article non technique \u0026ldquo;Décentralisons SMTP pour le bien commun\u0026rdquo; qui aborde certaines des raisons pour lesquelles il nous faut davantage de serveurs de mail.\nBonne lecture !\nCet article est pour les techies et les sysadmins # L\u0026rsquo;auto-hébergement requiert un minimum de connaissances et de motivation. Ce n\u0026rsquo;est pas un truc qui se fait en deux clics, il vous faut des connaissances minimales et avoir l\u0026rsquo;envie d\u0026rsquo;y dédier un peu de votre temps. Si vous n\u0026rsquo;avez jamais touché une zone DNS ou si vous pensez que prendre une heure pour configurer quelque chose est une tâche difficile, alors autant vous prévenir de suite : ça ne va pas être simple. Je ne vous recommande pas vous lancer dans l\u0026rsquo;aventure sauf si vous VOULEZ avoir à apprendre une tonne de trucs.\nSi vous êtes un sysadmin familier avec le travail de sysadmin, alors la plupart de votre travail de sysadmin est déjà considérablement plus dur que mettre en place une instractructure mail. Pour vous, le mail ne sera pas quelque chose de difficile.\nEHLO hypno.cat # Pour cet article, nous mettrons en place un serveur de mail pour hypno.cat, un petit site web pour mon (hypothétique) activité prospère d\u0026rsquo;hypnopraticien. J\u0026rsquo;ai enregistré ce domaine il y a plusieurs années parce que j\u0026rsquo;aimais bien le nom mais il n\u0026rsquo;a jamais été utilisé pour quoi que ce soit d\u0026rsquo;autre que l\u0026rsquo;hébergement d\u0026rsquo;un merveilleux fichier animé.\nGoogle, Bing et Yahoo le connaissent ; on ne sait pas trop comment (je vais du moins faire semblant de ne pas savoir) et ont indexé la page d\u0026rsquo;accueil, du coup ils ne considèrent pas que ce domaine vient d\u0026rsquo;être acheté par un spammer pour immédiatement envoyer du mail, mais il reste virtuellement inconnu sur internet parce qu\u0026rsquo;il n\u0026rsquo;a aucun contenu, ne fais aucun lien, n\u0026rsquo;est linké depuis nul part, et n\u0026rsquo;a jamais émis ou reçu de mail depuis ou vers personne. C\u0026rsquo;est un détail plutôt important qui montre que l\u0026rsquo;âge d\u0026rsquo;un domaine à un impact sur la réputation, mais ça reste un critère parmi d\u0026rsquo;autres, on reparlera réputation dans un futur article.\nJe vais le mettre en place au fur et à mesure que j\u0026rsquo;écris cet article, depuis le moment où j\u0026rsquo;ai démarré un nouveau VPS jusqu\u0026rsquo;au moment où j\u0026rsquo;ai pu échanger un mail dans les deux sens avec mon compte Gmail. J\u0026rsquo;ai choisi Gmail parce qu\u0026rsquo;il est sur-représenté dans la population et qu\u0026rsquo;un nombre important de personnes se sont plains de problèmes de distribution là-bas. Très personnellement, je pense que Outlook est bien pire en terme d\u0026rsquo;interopérabilité, mais gardons ça pour un futur article peut-être.\nLes services de mail pour hypno.cat vont fournir du mail entrant et sortant sécurisés par TLS. Ils vont permettre de soumettre des messages depuis l\u0026rsquo;application Gmail de mon smartphone pour ne pas changer les habitudes de mes utilisateurs. Je pourrais tout aussi bien fournir un webmail, avec Rainloop ou Roundcube, ou installer un client lourd comme Thunderbird ou mutt. N\u0026rsquo;importe quel client fonctionnera ; donc je ne vais pas couvrir cette partie. Il y a de nombreuses alternatives et plusieurs articles existants sur internet pour couvrir leur mise en place. Il y a même des techniques pour assister les clients à automatiquement découvrir leur configuration, mais tout ça reste hors du périmètre de mon article, je ne me fais pas de soucis vous allez trouver tout seul.\nMon mail sortant passera les vérifications basiques de Gmail, plus précisément SPF, DKIM et DMARC, qui sont plus que suffisant pour laisser le mail arriver en inbox. Il est possible de faire beaucoup plus, mais le but de cet article est de trouver le bon équilibre entre faire assez de travail pour avoir bonne allure face aux autres serveurs et rester simple.\nLe mail entrant sera filtré pour réduire la quantité de spam, soit en tuant les connexions de mauvais émetteurs évidents dès qu\u0026rsquo;ils sont détectés, soit en proposant une classification du Spam dans un répertoire dédié pour éviter de saturer la boite de réception.\nJe pourrais m\u0026rsquo;arrêter là parce que c\u0026rsquo;est déjà un setup assez intéressant, mais je vais rajouter un peu de configuration pour apprendre à Dovecot à entrainer le filtre antispam. Il apprendra à reconnaitre le Spam lorsque je déplacerai un message dans le répertoire Spam, et à reconnaitre le mail légitime lorsque je déplacerai un message hors du répertoire Spam. Cette partie est un peu plus complexe parce qu\u0026rsquo;elle dépend de Sieve qui est loin d\u0026rsquo;être un bijou d\u0026rsquo;ingénierie. Si vous vous moquez de l\u0026rsquo;apprentissage, vous pouvez sauter cette partie ; j\u0026rsquo;ai fait sans pendant plus de dix ans. C\u0026rsquo;est un truc sympa à avoir mais loin d\u0026rsquo;être une obligation ; c\u0026rsquo;est juste mon petit cadeau bonus dans cet article.\nPré-requis # Le premier pré-requis est de savoir où vous allez faire tourner votre serveur de mail.\nAvec la propagation des connexions permanentes type ADSL ou FTTH (fibre), une connexion à domicile peut être tentante mais ce n\u0026rsquo;est pas une bonne idée parce que les espaces d\u0026rsquo;adressage IP des FAI sont souvent blacklistés, ou souffrent d\u0026rsquo;une très mauvaise réputation de départ. De plus, de nombreux FAI bloquent le trafic SMTP sortant pour éviter que les machines personelles compromises se transforment en bot à spam. Pour moi, la meilleure option est de louer un serveur dédié ou un VPS depuis une société d\u0026rsquo;hébergement après s\u0026rsquo;être assuré que le trafic SMTP sortant est autorisé là-bas.\nJ\u0026rsquo;ai loué mes serveurs dédiés chez online.net ces douze dernières années et je suis extrêmement satisfait. Vous trouverez même sur ce blog des instructions (en anglais) pour faire tourner OpenBSD sur leurs serveurs comme ce n\u0026rsquo;est pas supporté nativement. Ils ne filtrent pas SMTP, ce qui est bien parce que vous pouvez faire tourner un serveur de mail immédiatement, mais les adresses IP peuvent avoir été utilisées préalablement par de mauvais émetteurs et il vous faudra les tester. Si l\u0026rsquo;adresse IP qui vous est automatiquement allouée n\u0026rsquo;est pas bonne, soit il va falloir bucher pour gagner en réputation, soit il vous faudra en commander une additionelle en prenant bien le soin de la choisir dans un autre range, après avoir vérifié qu\u0026rsquo;elle n\u0026rsquo;est pas dans une blacklist, ou qu\u0026rsquo;elle ne souffre pas déjà d\u0026rsquo;une mauvaise réputation.\nAlternativement, pour les besoins d\u0026rsquo;une offre commerciale que je suis en train de monter, j\u0026rsquo;ai commencé à construire une infrastructure sur vultr.com (lien de parrainage). Je ne suis là bas que depuis quelques mois et je changerai peut-être d\u0026rsquo;avis, mais pour l\u0026rsquo;instant j\u0026rsquo;en suis très satisfait. Là-bas, SMTP est filtré par défaut et il faut ouvrir un ticket et expliquer quel est le projet pour le trafic mail. J\u0026rsquo;ai expliqué que je ne cherchais pas à devenir ESP et que je n\u0026rsquo;allais pas envoyer de mail commercial de masse mais plutôt gérer de l\u0026rsquo;hébergement ; ils ont été très serviables et m\u0026rsquo;ont retiré le filtrage le jour même.\nPeu importe où le serveur de mail sera hébergé, ce que vous devez vérifier c\u0026rsquo;est que :\nl\u0026rsquo;hébergeur n\u0026rsquo;a pas un passif avec l\u0026rsquo;hébergement de spammers et qu\u0026rsquo;il autorise SMTP vous avez une adresse IP dédiée au mail vous avez le contrôle du reverse DNS pour cette adresse IP votre adresse IP n\u0026rsquo;est pas déjà sur des blacklists Tant que ces pré-requis sont respectés, vous ne devriez pas avoir trop de problèmes.\nComme personne n\u0026rsquo;aime perdre de mails, il est aussi judicieux de se préparer aux incidents. On s\u0026rsquo;en protège en utilisant un serveur secondaire pour prendre le trafic lorsque le primaire est down, ou pour re-router le trafic lorsque le primaire est en incapacité de router correctement. Je ne vais pas couvrir ce sujet parce que ce n\u0026rsquo;est pas très complexe, il s\u0026rsquo;agit d\u0026rsquo;installer un second serveur qui route son trafic vers le premier et de faire le bon enregistrement DNS, après avoir lu cet article vous devriez vous en sortir seul.\nCEPENDANT, un grand nombre de personnes ont mentionnés s\u0026rsquo;être déjà fait blacklister et avoir perdu du mail, donc je pense que c\u0026rsquo;est le bon moment pour donner ce conseil : n\u0026rsquo;hébergez pas vos différents serveurs au même endroit. Vous ne voulez pas avoir tous vos serveurs au même endroit s\u0026rsquo;il y a une panne de courant ou de réseau, et vous voulez aussi éviter que si votre range IP est blacklisté en dommage collatéral d\u0026rsquo;un voisin malveillant, l\u0026rsquo;adresse IP de votre serveur secondaire est \u0026ldquo;suffisamment éloignée\u0026rdquo; pour ne pas être blacklistée aussi. De cette façon, si votre serveur primaire est en incapacité temporaire d\u0026rsquo;émettre à cause d\u0026rsquo;un blocage, vous pouvez re-router votre trafic au-travers de votre serveur secondaire le temps que le problème soit résolu.\nJ\u0026rsquo;ai personellement eu à gérer un seul blocage sur mon propre setup (bien plus en dehors) durant ces dix dernières années. C\u0026rsquo;est un peu chiant parce que les incidents tombent toujours au mauvais moment et que ce n\u0026rsquo;est jamais fun, mais j\u0026rsquo;ai re-routé le trafic au-travers d\u0026rsquo;un serveur secondaire pour le rétablir, rempli un formulaire de délistage en apportant la preuve que j\u0026rsquo;étais un émetteur légitime, puis re-routé le trafic par mon serveur primaire une fois le problème résolu. On est très loin du problème insurmontable qu\u0026rsquo;en font un certain nombre de personnes.\nPas de stress pour ceux qui ont un plan pour les pannes avant qu\u0026rsquo;elles ne surviennent.\nLa stack technique # Je vais démarrer un nouveau VPS chez vultr.com (toujours un lien de parrainage), installer la dernière version d\u0026rsquo; OpenBSD ( snapshot) parce que c\u0026rsquo;est mon système de préférence (installation non couverte dans cet article), et construire mon système de mail par dessus. Je vais partir du principe que nous sommes sous OpenBSD tout le reste de cet article, mais à l\u0026rsquo;exception de certaines commandes spécifiques pour installer des packages, la configuration est très similaire d\u0026rsquo;un système à un autre. Si vous êtes assez grands pour faire tourner un serveur de mail, vous êtes assez grands pour adapter des chemins, remplacer doas par sudo, et sinon… installez OpenBSD !\nPour la couche SMTP, qui est en charge de transférer des messages entre deux hôtes indépendemment de la manière dont les utilisateurs vont y accéder, j\u0026rsquo;utiliserais le logiciel OpenSMTPD. Il s\u0026rsquo;agit du serveur SMTP par défaut pour le système d\u0026rsquo;exploitation OpenBSD et il y a une version portable disponible sur Github avec des instructions pour le construire. Il est également disponible dans les dépôts de packages de différents systèmes et distributions Linux, peut-être tout juste à un pkg, un rpm ou un apt de vous.\nLes instructions de cet article sont pour la version 6.6.0 ou ultérieure, elles ne fonctionneront pas sur des versions précédentes. Si votre système ne fournit pas de package à jour, vous pouvez tenter de demander au mainteneur de le mettre à jour ou alors tenter d\u0026rsquo;en devenir mainteneur. Il vous faudra construire le package manuellement à partir du dernier tag contenant un p, pour portable, dans son nom sur Github.\nPour la couche IMAP, qui permet aux utilisateurs de récupérer les messages qu\u0026rsquo;ils ont reçus et y accéder depuis leurs smartphones ou webmail, j\u0026rsquo;utiliserais la dernière version de Dovecot. J\u0026rsquo;utiliserai également le package Dovecot-Pigeonhole, qui permettera d\u0026rsquo;entraîner notre solution antispam en lui faisant apprendre ce qui est du Spam et ce qui est du Ham. Si vous souhaitez utiliser un client console, en ssh, comme mutt par exemple, pas besoin de cette couche car OpenSMTPD peut distribuer dans une boite mail locale à laquelle les clients mail peuvent accéder en direct. Dans cet article, nous configurerons IMAP parce que nous voulons que les mails puissent être consultés depuis un smartphone comme une personne lambda.\nEnfin, pour la couche de filtrage du spam, j\u0026rsquo;utiliserai l\u0026rsquo;excellent Rspamd qui est bien plus qu\u0026rsquo;une simple solution antispam. Il fournit des méthodes de filtrage moderne, mais permet également l\u0026rsquo;intégration de filtrage antivirus, de signature DKIM, et d\u0026rsquo;une tonne de modules que vous pouvez choisir d\u0026rsquo;activer ou non pour régler plus finement votre setup. Dans cet article, nous utiliserons tout simplement ses capacités de filtrage du Spam et de signature DKIM, le strict minimum dont nous avons besoin.\nMe rendre accessible # SMTP est très étroitement lié avec DNS et les autres hôtes dépendent de recherches DNS pour trouver quelle machine gère le mail de votre domaine. C\u0026rsquo;est fait au-travers de la recherche d\u0026rsquo;enregistrements MX (Mail eXchanger), donc le minimum à faire pour que SMTP fonctionne est de déclarer un enregistrement MX pour votre domaine. Pour hypno.cat, la zone contiendra ce qui suit :\n;; un enregistrement A (et AAAA pour IPv6) est déclaré pour nommer le serveur de mail mail.hypno.cat A 217.69.8.253 mail.hypno.cat AAAA 2001:19f0:6801:867:5400:01ff:fee7:7af7 ;; un enregistrement MX est déclaré pour dire au monde que mail.hypno.cat gère le ;; mail pour hypno.cat ;; 0 est la préférence maximale, si nous avions un serveur secondaire il aurait un ;; nombre supérieur hypno.cat. MX 0 mail.hypno.cat. ;;hypno.cat. MX 10 mail-backup.hypno.cat. Je peux m\u0026rsquo;assurer que tout est correct en utilisant host et dig pour vérifier que le nom résoud et que la recherche MX retourne bien le bon nom :\n$ host mail.hypno.cat mail.hypno.cat has address 217.69.8.253 mail.hypno.cat has IPv6 address 2001:19f0:6801:867:5400:1ff:fee7:7af7 $ dig -t MX hypno.cat +short 0 mail.hypno.cat. À ce stade et avec ces enregistrements DNS, les autres serveurs de mail peuvent déjà trouver quel serveur est responsable pour hypno.cat, et le contacter quand ils veulent envoyer du mail à n\u0026rsquo;importe quelle adresse mail de ce domaine.\nMe faire bien voir # Être joignable n\u0026rsquo;est pas suffisant parce qu\u0026rsquo;aujourd\u0026rsquo;hui, vous DEVEZ avoir un reverse DNS (rDNS) ET un Forward-Confirmed rDNS (FCrDNS).\nIl y a une raison derrière cela. Un grand nombre d\u0026rsquo;hôtes qui émettent du spam sont des machines compromises, avec une grande portion d\u0026rsquo;entre elles qui sont des ordinateurs personnels derrière des connexions résidentielles. Beaucoup de ces connexions n\u0026rsquo;ont pas de rDNS ou de FCrDNS, ou ont des rDNS qui ressemblent à des patterns d\u0026rsquo;IP allouées dynamiquement (ex. : 123.123.123.123.dyn.adsl.example.com).\nComme il s\u0026rsquo;agit de machines compromises individuellement, et non de machines dont la configuration est sous le contrôle total des spammers, les rDNS et FCrDNS ne peuvent pas être configurés… il en résulte que la configuration de rDNS et FCrDNS est devenu une preuve de travail : il faut un rDNS et un FCrDNS correctement configuré et ne ressemblant pas à une allocation dynamique. Chez certains Big Mailer Corps, il s\u0026rsquo;agit de règles claires, décrites dans les guidelines, pour d\u0026rsquo;autres on le découvre par la sanction. Dans tous les cas, il s\u0026rsquo;agit du STRICT MINIMUM ; assurez-vous que vos noms d\u0026rsquo;hôtes ressemblent à un VRAI nom de serveur de mail (ex. : mail.hypno.cat, et non pas www.hypno.cat).\nLe rDNS est généralement hors de votre contrôle parce que dans une zone un peu spéciale, arpa, gérée par le propriétaire de l\u0026rsquo;adresse IP (en général votre hébergeur). Les FAI ne vous laissent pas toujours les configurer ; mais les hébergeurs de serveurs n\u0026rsquo;ayant pas vraiment d\u0026rsquo;autres choix, fournissent généralement un formulaire quelque part sur leurs espaces clients pour assigner un rDNS aux adresses qu\u0026rsquo;ils vous ont assignées :\nSi je configure mon rDNS pour avoir la même valeur que l\u0026rsquo;enregistrement DNS configuré plus haut, alors je passe automatiquement le test du FCrDNS. On peut le vérifier très simplement en faisant une recherche rDNS pour l\u0026rsquo;adresse IP :\n$ host 217.69.8.253 253.8.69.217.in-addr.arpa domain name pointer mail.hypno.cat. $ host 2001:19f0:6801:867:5400:1ff:fee7:7af7 7.f.a.7.7.e.e.f.f.f.1.0.0.0.4.5.7.6.8.0.1.0.8.6.0.f.9.1.1.0.0.2.ip6.arpa domain name pointer mail.hypno.cat. Puis cherchez l\u0026rsquo;adresse IP pour ce rDNS :\n$ host mail.hypno.cat mail.hypno.cat has address 217.69.8.253 mail.hypno.cat has IPv6 address 2001:19f0:6801:867:5400:1ff:fee7:7af7 Si les valeurs se retrouvent dans les deux sens, tout est d\u0026rsquo;équerre.\nAnnoncer quelles machines sont autorisées à émettre pour mon domaine # SPF est un mécanisme qui permet pour un domaine destination de déterminer si la machine qui émet était autorisée pour le domaine émetteur.\nLe mécanisme est très simple, on ajoute un enregistrement DNS à la zone de notre domaine qui déclare quels serveurs peuvent émettre. Quand le domaine destination reçoit un mail avec un émetteur de notre domaine, il vérifie si l\u0026rsquo;adresse IP du serveur émetteur en avait le droit.\nSPF n\u0026rsquo;est pas obligatoire pour SMTP mais c\u0026rsquo;est une des choses qui nous font paraître légitimes face aux destinataires. SPF est informatif par défaut ; ne pas passer un test SPF ou ne pas avoir d\u0026rsquo;enregistrement SPF n\u0026rsquo;implique pas qu\u0026rsquo;un mail termine en Spam ou soit détruit. Il est communément accepté que l\u0026rsquo;absence d\u0026rsquo;un enregistrement SPF a un impact très négatif sur la réputation d\u0026rsquo;un domaine, c\u0026rsquo;est même explicitement dit dans les guidelines de certains Big Mailer Corps, et vu comme l\u0026rsquo;enregistrement est simple à fournir… il n\u0026rsquo;y a vraiment aucune excuse pour ne pas le faire.\nIl y a plusieurs façons de configurer SPF ; dans cet example, je vais simplement le configurer pour n\u0026rsquo;autoriser QUE mon serveur de mail à envoyer du mail en mon nom :\nhypno.cat. IN TXT \u0026#34;v=spf1 mx -all\u0026#34; Si des hôtes reçoivent un mail avec un émetteur @hypno.cat qui ne provient pas de mail.hypno.cat, ils pourront prendre la décision (ou non) de rejeter le mail. Ils pourront en tout cas constater qu\u0026rsquo;il provient d\u0026rsquo;une machine qui n\u0026rsquo;était pas autorisée et qu\u0026rsquo;ils ont le droit de le rejeter.\nProuver que j\u0026rsquo;ai autorisé le message lui même. # DKIM est un mécanisme d\u0026rsquo;authentification par lequel on peut signer cryptographiquement les mails émis par notre serveur, prouvant qu\u0026rsquo;on les a vu passer et que l\u0026rsquo;on a pris la responsabilité de les laisser transiter. Les hôtes qui recoivent ces mails peuvent vérifier qu\u0026rsquo;ils étaient bien autorisés en vérifiant la signature, permettant ainsi de détecter des mails forgés hors du système de mail.\nComme SPF, c\u0026rsquo;est informatif par défaut et l\u0026rsquo;absence de signature DKIM ou une signature invalide ne veut pas dire qu\u0026rsquo;un mail ne sera pas distribué. En revanche, il est là aussi admis et décrit dans les guidelines que l\u0026rsquo;absence de DKIM aura un effet sur la réputation d\u0026rsquo;un domaine. Et comme configurer DKIM est facile, pas vraiment d\u0026rsquo;excuse pour ne pas le faire là non plus.\nC\u0026rsquo;est un petit peu plus complexe que pour SPF parce que, même si ça dépend aussi de DNS, il faut générer une paire de clés et publier la clé publique dans l\u0026rsquo;enregistrement DNS. Rien de bien foufou non plus.\nNous sommes en 2019. Je vais générer une clé RSA de 1024 bits ; je sais !\n\u0026mdash; BEGIN DIGRESSION \u0026mdash;\nLes clés RSA 2048 bits ne rentrent pas dans un enregistrement DNS TXT et doivent être tronquées en deux enregistrements, mais tout le monde n\u0026rsquo;est pas capable de réassocier ces clés tronquées, et vous pouvez perdre les bénéfices de DKIM si l\u0026rsquo;hôte destination n\u0026rsquo;arrive pas à utiliser votre clé publique partielle et considère la signature invalide.\nEn ce qui me concerne, ces hôtes sont responsables et nous pouvons les ignorer ; mais comme le but de cet article est de permettre une interopérabilité maximale et que DKIM n\u0026rsquo;est pas présent ici pour sécuriser quoi que ce soit, juste pour vous faire paraître légitime, on va juste regarder ailleurs en justifiant cette mauvaise pratique par le risque modéré de falsification de contenu quand il est utilisé conjointement à SPF.\nCe n\u0026rsquo;est pas une bonne manière de raisonner mais DKIM a d\u0026rsquo;autres inconvénients et comme virtuellement tout le monde utilise du RSA 1024-bits. Je suggère de laisser le débat 1024 vs \u0026gt;= 2048 à plus tard, lors que nous aurons déjà réussis à échanger correctement avec les Big Mailer Corps. Je prendrais juste le temps, sans pointer de doigt nul part, d\u0026rsquo;appuyer sur le fait que certaines sociétés dans la sécurité qui travaillent dans l\u0026rsquo;industrie du mail ont (avaient ?) encore des problèmes avec les clés 2048-bits en 2019. Des sociétés qui calculent des scores de réputation. Oh, la, belle, ironie !\nSi c\u0026rsquo;est un sujet d\u0026rsquo;inquiétude pour vous, vous pouvez expérimenter la troncature des clé et voir si ça vous convient. Adapter mon exemple est trivial ; assurez vous juste d\u0026rsquo;avoir la clé tronquée sur deux enregistrements TXT.\n\u0026mdash; END DIGRESSION \u0026mdash;\nJe crée un répertoire pour contenir les clés :\n# mkdir /etc/mail/dkim Ensuite, la commande suivante va générer une paire de clé et extraire la clé publique de la clé privée:\n# openssl genrsa -out /etc/mail/dkim/hypno.cat.key 1024 Generating RSA private key, 1024 bit long modulus …………………………++++++ ……………++++++ e is 65537 (0x10001) # openssl rsa -in /etc/mail/dkim/hypno.cat.key -pubout -out /etc/mail/dkim/hypno.cat.pub writing RSA key # cat /etc/mail/dkim/hypno.cat.pub -----BEGIN PUBLIC KEY----- MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDThHqiM610nwN1nmV8OMc7PaPO uJWVGRDz5bWj4cRjTTmQYjJQd02xrydRNrRhjEKBm2mMDArNWjoM3jvN04ZifqJx DmKr7X8jsYi+MPEHz6wZxB8mayDK6glYTCyx//zl1luUdvm26PutA38K8cgnb7iT kfVP2OqK6sHAdXjnowIDAQAB -----END PUBLIC KEY----- Enfin, je génére un enregistrement DNS TXT en retirant le blindage de la clé publique et en formattant le contenu pour qu\u0026rsquo;il soit comme suit :\n20190913._domainkey.hypno.cat.\tIN TXT \u0026#34;v=DKIM1;k=rsa;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDThHqiM610nwN1nmV8OMc7PaPOuJWVGRDz5bWj4cRjTTmQYjJQd02xrydRNrRhjEKBm2mMDArNWjoM3jvN04ZifqJxDmKr7X8jsYi+MPEHz6wZxB8mayDK6glYTCyx//zl1luUdvm26PutA38K8cgnb7iTkfVP2OqK6sHAdXjnowIDAQAB;\u0026#34; Le nom de l\u0026rsquo;enregistrement est construit selon le motif : \u0026lt;selector\u0026gt;._domainkey.\u0026lt;domain\u0026gt;., où \u0026lt;selector\u0026gt; est un nom arbitraire choisi pour permettre à plusieurs clés de co-exister. J\u0026rsquo;aime bien utiliser la date à laquelle j\u0026rsquo;ai généré la paire de clés, mais peu importe ce que vous choisissez, écrivez le pour plus tard car nous en aurons besoin pour configurer la signature DKIM.\nDe plus, la clé publique n\u0026rsquo;a pas besoin de rester dans le répertoire une fois qu\u0026rsquo;elle est publiée dans l\u0026rsquo;enregistrement DNS. Elle peut toujours être dérivée de la clé privée si nous en avons besoin, mais personnellement j\u0026rsquo;aime bien la garder pour la retrouver rapidement.\nFaites bien attention à ce que la clé privée ne soit pas en lecture pour tous car, pour une raison inconnue et pas très maline, OpenSSL pense que les clés privées doivent être créées en rw-r--r--.\nDire aux autres quoi faire en cas d\u0026rsquo;erreurs SPF et DKIM avec DMARC # SPF et DKIM sont tous les deux très intéressants, mais ils ne sont qu\u0026rsquo;informatifs, alors ils n\u0026rsquo;empêchent pas d\u0026rsquo;abuser de votre nom de domaine.\nDMARC permet de dire aux serveurs mails de destination ce que vous voulez qu\u0026rsquo;ils fassent des mails en apparence de votre domaine qui ratent les tests SPF et DKIM. Il permet par exemple de leur dire qu\u0026rsquo;ils doivent rejeter ces mails.\nIl n\u0026rsquo;y a pas d\u0026rsquo;indicateur clair : fournir une politique de rejet nous fait mieux voir que de fournir une politique sans action. Des expérimentations semblent montrer qu\u0026rsquo;avoir un enregistrement a un impact positif par rapport à n\u0026rsquo;en avoir aucun, même si l\u0026rsquo;enregistrement dit que rien ne doit être fait avec les erreurs SPF et DKIM. Il serait intéressant de mener une nouvelle expérimentation à ce sujet, mais en même temps, il est plus simple de fournir une politique de rejet et de ne pas avoir à se poser la question :-)\nLe format d\u0026rsquo;un enregistrement DMARC va bien au-delà de cet article et vous trouverez de nombreux exemples dans n\u0026rsquo;importe quel moteur de cherche. Un enregistrement simple annoncera une politique (p=) de none (reject si vous voulez rejeter, quarantine si vous avez besoin de temps pour décider). Le champ pourcentage (pct=) déclare combien de ces mails devraient être sujets à la politique DMARC, et le champ Reporting URI of Aggregates (rua=) est la destination des rapports DMARC (pour analyse avant de basculer de quarantine à reject par exemple).\nJe vais ici créer un enregistrement simple, un enregistrement qui fera que les Big Mailer Corps voient que nous nous intéressons à DMARC même si nous ne savons pas encore quoi en faire :\n_dmarc.hypno.cat. IN TXT \u0026#34;v=DMARC1;p=none;pct=100;rua=mailto:postmaster@hypno.cat;\u0026#34; Mon opinion est qu\u0026rsquo;il est préférable de basculer rapidement vers p=reject, après s\u0026rsquo;être assuré que les mails sont correctement signés et que seul notre serveur de mail les émet.\nObtenir un certificat pour TLS # Il y a plusieurs façons d\u0026rsquo;obtenir un certificat TLS. Je vais partir du principe que vous êtes familiers avec l\u0026rsquo;hébergement d\u0026rsquo;autres services et que vous êtes capable d\u0026rsquo;en obtenir un par votre registrar ou depuis letsencrypt. Si ce n\u0026rsquo;est pas le cas, vous trouverez des exemples à foison sur n\u0026rsquo;importe quel moteur de recherche.\nSi vous savez gérer vos certificats, vous pouvez sauter à la section suivante.\nComme je suis réellement en train de mettre en place mail.hypno.cat, je ne serais pas en mesure de continuer cet article sans obtenir de certificat, donc en bonus pour les utilisateurs d\u0026rsquo;OpenBSD, je vais documenter la génération de mon certificat sur ma nouvelle installation.\nacme-client est un utilitaire présent dans le système de base et qui permet de demander ou renouveller des certificats depuis la console. Il dépend d\u0026rsquo;un challenge HTTP ; il doit donc être en mesure d\u0026rsquo;écrire dans un répertoire qui soit accessible par HTTP. Et comme nous sommes chanceux… OpenBSD contient un serveur httpd que l\u0026rsquo;on peut utiliser pour ce challenge. Et comme nous sommmes TRÈS chanceux… OpenBSD fournit aussi un exemple de fichier de configuration que l\u0026rsquo;on peut utiliser sans gros changement.\nOn copie simplement le fichier d\u0026rsquo;example depuis /etc/example/httpd.conf vers /etc/httpd.conf, en remplaçant example.com par mail.hypno.cat et en supprimant le bloc TLS puisque seul le challenge nous interesse. Le fichier doit ressembler à cela :\nserver \u0026#34;mail.hypno.cat\u0026#34; { listen on * port 80 location \u0026#34;/.well-known/acme-challenge/*\u0026#34; { root \u0026#34;/acme\u0026#34; request strip 2 } location * { block return 302 \u0026#34;https://$HTTP_HOST$REQUEST_URI\u0026#34; } } Le daemon httpd peut être démarré comme suit :\n# rcctl -f start httpd httpd(ok) En ce qui concerne acme-client, c\u0026rsquo;est très simple, on copie à nouveau le fichier d\u0026rsquo;exemple depuis /etc/example/acme-client.conf vers /etc/acme-client.conf, en remplaçant example.com par mail.hypno.cat et on doit se retrouver avec un bloc comme suit :\ndomain mail.hypno.cat { domain key \u0026#34;/etc/ssl/private/mail.hypno.cat.key\u0026#34; domain full chain certificate \u0026#34;/etc/ssl/mail.hypno.cat.fullchain.pem\u0026#34; sign with letsencrypt } À ce stade, on se contente de lancer acme-client pour notre domaine :\n# acme-client -v mail.hypno.cat […] acme-client: /etc/ssl/mail.hypno.cat.fullchain.pem: created Le certificat et les clés sont créés au bon endroit, il suffira par la suite d\u0026rsquo;ajouter les chemins dans OpenSMTPD et Dovecot.\nVous pouvez garder httpd en marche pour les renouvellements et appeller acme-client depuis un cron, vous pouvez l\u0026rsquo;éteindre et renouveller d\u0026rsquo;une autre façon. Je vous laisse vous débrouiller ; vous trouverez comment faire avec l\u0026rsquo;aide de la page de manuel acme-client(1).\nInstaller et configurer Rspamd # Sur OpenBSD, Rspamd est packagé et peut être installé d\u0026rsquo;une simple commande. Nous avons aussi besoin d\u0026rsquo;installer Redis qui sert à stocker des statistiques pour Rspamd et son greylisting (entre autres), et également filter-rspamd qui est un petit bout de code qui permets à OpenSMTPD de travailler avec Rspamd. Ils existent également sous forme de packages et peuvent être installés en une seule commande.\nTout d\u0026rsquo;abord, on installe les packages :\nmail$ doas pkg_add redis rspamd opensmtpd-filter-rspamd […] redis-4.0.14: ok rspamd-1.9.0: ok opensmtpd-filter-rspamd-0.1.1: ok The following new rcscripts were installed: /etc/rc.d/redis /etc/rc.d/rspamd See rcctl(8) for details. Rspamd est très configurable, si vous voulez faire des choses un peu funky je vous laisse lire la documentation. Pour cet article je me contenterai d\u0026rsquo;une utilisation très basique, comme c\u0026rsquo;est le cas sur mes propres machines :\nJe pourrais éditer la configuration dans /etc/rspamd/actions.conf pour ajuster les seuils de mise en spam, mais les valeurs par défaut me conviennent alors je vous les montre juste :\nactions { reject = 15; # Reject when reaching this score add_header = 6; # Add header when reaching this score greylist = 4; # Apply greylisting when reaching this score (will emit `soft reject action`) […] Mais ce que je veux vraiment c\u0026rsquo;est que Rspamd gère la signature DKIM parce que c\u0026rsquo;est un des pré-requis pour paraître légitime.\nOn le fait en fournissant une configuration dans /etc/rspamd/local.d/dkim_signing.conf pour faire correspondre à notre domaine la clé DKIM générée plus tôt :\n# mkdir /etc/rspamd/local.d # cat \u0026lt;\u0026lt; EOF \u0026gt;/etc/rspamd/local.d/dkim_signing.conf allow_username_mismatch = true; domain { hypno.cat { path = \u0026#34;/etc/mail/dkim/hypno.cat.key\u0026#34;; selector = \u0026#34;20190913\u0026#34;; } } EOF La clé de configuration allow_username_mismatch est nécessaire parce que Rspamd attend des utilisateurs avec des noms de domaines, mais dans cette configuration OpenSMTPD authentifie de simples utilisateurs système. De plus, faites attention à ce que le fichier contenant la clé DKIM soit autorisé en lecture au groupe _rspamd.\nÀ ce stade on est bon, Rspamd et Redis peuvent être activés pour qu\u0026rsquo;OpenBSD les démarre à chaque reboot :\n# rcctl enable redis # rcctl enable rspamd Et on peut les démarrer immédiatement pour ne pas avoir à attendre le prochain reboot :\n# rcctl start redis redis(ok) # rcctl start rspamd rspamd(ok) Configurer OpenSMTPD # OpenSMTPD est installé par défaut sur OpenBSD donc aucune phase d\u0026rsquo;installation ici. Si vous utilisez un système d\u0026rsquo;exploitation différent, soit quelqu\u0026rsquo;un l\u0026rsquo;a packagé et vous pouvez l\u0026rsquo;installer avec votre gestionnaire de paquets, ou vous pouvez le construire depuis les sources et l\u0026rsquo;installer en utilisant le code depuis le mirroir Github et en suivant les instructions de build.\nComme écrit plus haut, ces instructions ne sont valides que pour la version 6.6.0 et ultérieures, les versions précédentes ne supportent pas les filtres et certaines des fonctionnalités décrites ici.\nLa configuration par défault d\u0026rsquo;OpenSMTPD est adaptée à un serveur de mail local, n\u0026rsquo;acceptant pas de connexions de l\u0026rsquo;extérieur, mais capable de laisser les utilisateurs locaux s\u0026rsquo;échanger des messages et en émettre vers des hôtes distants.\nElle ressemble à ce qui suit :\n# $OpenBSD: smtpd.conf,v 1.11 2018/06/04 21:10:58 jmc Exp $ # This is the smtpd server system-wide configuration file. # See smtpd.conf(5) for more information. table aliases file:/etc/mail/aliases # To accept external mail, replace with: listen on all # listen on lo0 action \u0026#34;local_mail\u0026#34; mbox alias \u0026lt;aliases\u0026gt; action \u0026#34;outbound\u0026#34; relay # Uncomment the following to accept external mail for domain \u0026#34;example.org\u0026#34; # # match from any for domain \u0026#34;example.org\u0026#34; action \u0026#34;local_mail\u0026#34; match from local for local action \u0026#34;local_mail\u0026#34; match from local for any action \u0026#34;outbound\u0026#34; Je voulais initialement mettre à jour la configuration progressivement pour vous tenir par la main, mais cet article à énormément grossi depuis sa première version. Par souci de simplicité, je vais simplement mettre les 16 lignes de configuration et les commenter ensuite :\npki mail.hypno.cat cert \u0026#34;/etc/ssl/mail.hypno.cat.fullchain.pem\u0026#34; pki mail.hypno.cat key \u0026#34;/etc/ssl/private/mail.hypno.cat.key\u0026#34; filter check_dyndns phase connect match rdns regex { \u0026#39;.*\\.dyn\\..*\u0026#39;, \u0026#39;.*\\.dsl\\..*\u0026#39; } \\ disconnect \u0026#34;550 no residential connections\u0026#34; filter check_rdns phase connect match !rdns \\ disconnect \u0026#34;550 no rDNS is so 80s\u0026#34; filter check_fcrdns phase connect match !fcrdns \\ disconnect \u0026#34;550 no FCrDNS is so 80s\u0026#34; filter senderscore \\ proc-exec \u0026#34;filter-senderscore -blockBelow 10 -junkBelow 70 -slowFactor 5000\u0026#34; filter rspamd proc-exec \u0026#34;filter-rspamd\u0026#34; table aliases file:/etc/mail/aliases listen on all tls pki mail.hypno.cat \\ filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } listen on all port submission tls-require pki mail.hypno.cat auth filter rspamd action \u0026#34;local_mail\u0026#34; maildir junk alias \u0026lt;aliases\u0026gt; action \u0026#34;outbound\u0026#34; relay helo mail.hypno.cat match from any for domain \u0026#34;hypno.cat\u0026#34; action \u0026#34;local_mail\u0026#34; match from local for local action \u0026#34;local_mail\u0026#34; match from any auth for any action \u0026#34;outbound\u0026#34; match from local for any action \u0026#34;outbound\u0026#34; C\u0026rsquo;est tout ce dont nous avons besoin pour notre configuration SMTP complète.\nLes deux premières lignes pki déclarent que mail.hypno.cat utilisera le certificat et la clé que nous avons générés plus tôt pour TLS.\nLes lignes filter appliquent un ensemble de filtres sur les connexions entrantes. check_dyndns filtrera si rDNS corresponds à certains motifs. check_rdns filtrera s\u0026rsquo;il n\u0026rsquo;y a pas de rDNS. check_fcrdns filtrera si la correspondance FCrDNS échoue. Il s\u0026rsquo;agit de filtres \u0026ldquo;internes\u0026rdquo; à OpenSMTPD que vous pouvez ajuster pour filtrer différement, à d\u0026rsquo;autres phases ou avec d\u0026rsquo;autres critères.\nsenderscore est un filtre custom que vous pouvez installer soit depuis votre gestionnaire de paquets (pkg_add opensmtpd-filter-senderscore sur OpenBSD), ou obtenir depuis Github. Il n\u0026rsquo;est pas obligatoire mais je le trouve particulièrement utile et expliquerai pourquoi dans la prochaine section. Dans cette configuration, il rejettera un émetteur si son senderscore est en dessous de 10, il junk-era le message (ajoutera un entête X-Spam si le score est inférieur à 70), et pour mon plaisir personnel, ajoutera avant chaque réponse un délai inversement proportionnel à la réputation de l\u0026rsquo;émetteur pour ralentir les mauvais émetteurs.\nrspamd est également un filtre custom que nous avons installé dans la précedente section. Il n\u0026rsquo;y a aucune configuration car il est entièrement controlé depuis Rspamd, il suffit donc de le déclarer.\nSi vous voulez être plus conservateur et ne pas rejeter les mail pour éviter des faux positifs, vous pouvez assigner l\u0026rsquo;action junk plutôt que disconnect aux filtres internes et supprimer l\u0026rsquo;option -blockBelow au filtre senderscore :\nfilter check_dyndns phase connect match rdns regex { \u0026#39;.*\\.dyn\\..*\u0026#39;, \u0026#39;.*\\.dsl\\..*\u0026#39; } junk filter check_rdns phase connect match !rdns junk filter check_fcrdns phase connect match !fcrdns junk filter senderscore proc-exec \u0026#34;filter-senderscore -junkBelow 70 -slowFactor 5000\u0026#34; De cette façon, au lieu de rejeter les sessions, OpenSMTPD va simplement junk les messages pour les faire atterrir en boite à Spam. En regardant régulièrement ce qui atterri dans cette boite, vous pourrez gagner en confiance et adapter vos réglages pour trouver ceux qui vous conviennent.\nCes filtres simples, sans même compter rspamd, sont suffisants pour tuer la plus grosse partie du spam. Chez moi, ils permettent de passer de plusieurs centaines de spam quotidiens à une poignée. Le filtre rdns regex doit être adapté en fonction du pays, vous pouvez trouver des motifs qui sont récurrents chez vous et n\u0026rsquo;apparaissent jamais chez moi. Pour faciliter la maintenance, ils peuvent être stockés dans un fichier à raison d\u0026rsquo;un par ligne, en respectant la construction suivante :\ntable \u0026lt;dyndns\u0026gt; file:/etc/mail/path-to-my-regex-file filter check_dyndns phase connect match rdns regex \u0026lt;dyndns\u0026gt; junk Continuons à disséquer la configuration.\nLa table aliases pointe vers un fichier d\u0026rsquo;alias sur deux colonnes. La colonne de gauche correspond à la partie utilisateur d\u0026rsquo;une adresse e-mail que vous recevez ; la colonne de droite correspond à l\u0026rsquo;utilisateur à qui le mail doit être livré (ex. : root: gilles). Il est recommandé d\u0026rsquo;aliaser les adresses root, abuse et postmaster vers une adresse que vous lisez réellement. Je ne le fais pas ici parce que je vais simplement détruire le setup une fois l\u0026rsquo;article fini, mais en ce qui vous concerne, faites le !\nEnsuite viennent les lignes listen. La première est le point d\u0026rsquo;attache que vont contacter les autres serveurs de mail qui souhaitent envoyer du mail vers hypno.cat. Elle est configurée pour offrir tls en utilisant les certificats et clés pour mail.hypno.cat et pour filtrer les sessions entrantes avec mes filtres. La seconde est le point d\u0026rsquo;attache que vont contacter mes propres utilisateurs sur le port de submission, qui requiert TLS en utilisant les mêmes certificats et clés, mais également l\u0026rsquo;authentification des utilisateurs système locaux (le défaut peut être changé), et filtrant à-travers le filtre rspamd uniquement dans le but de faire la signature DKIM de nos messages.\nLes lignes action définissent les actions à entreprendre avec les mails qui sont acceptés dans le système. Une action locale permet de distribuer dans une maildir, en classifiant les spams dans un répertoire spécifique, et en résolvant les alias. Une seconde action permets de relayer les mails en s\u0026rsquo;annonçant en tant que mail.hypno.cat aux autres hôtes. Si le serveur disposait de plusieurs adresses IP, il serait possible d\u0026rsquo;assigner à l\u0026rsquo;aide de l\u0026rsquo;option src celle à utiliser, mais ici je n\u0026rsquo;ai qu\u0026rsquo;une seule adresse IP donc pas besoin de la spécifier.\nEnfin, les lignes match représentent l\u0026rsquo;ensemble des règles. Quand une enveloppe entre dans le serveur SMTP au travers d\u0026rsquo;un des points d\u0026rsquo;attache listen, elle est comparée séquentiellement à chaque ligne match jusqu\u0026rsquo;à trouver une correspondance ou rejetée si aucune correspondance n\u0026rsquo;est trouvée. Lorsqu\u0026rsquo;une correspondance est trouvée, l\u0026rsquo;action associée à la règle est prise en compte. Les règles de match sont très simples à comprendre, je ne vais donc pas les expliquer ; vous devriez arriver à comprendre seuls ce que signifie from any for domain \u0026quot;hypno.cat\u0026quot;, sans quoi vous ne devriez pas lire cet article en premier lieu.\nQuelques mots au sujet de SenderScore # SenderScore est une base de donnée de réputation IP qui associe un score entre 0 et 100 à une adresse IP. Le score est lié au volume et comportement observé depuis ces adresses IP, et même si l\u0026rsquo;on ne sait pas COMMENT sont calculés ces scores parce que la méthodologie n\u0026rsquo;est pas publique, un peu d\u0026rsquo;étude permet de confirmer que ces nombres ne sortent pas de nul part : ils sont corrélés au ratio de distribution et d\u0026rsquo;erreur chez différents Big Mailer Corps et il est possible d\u0026rsquo;influencer le score en changeant de comportement.\nIl est également TRÈS clair que pour construire les scores de réputation, ils ont accès à des informations relatives aux sessions SMTP dont seuls les serveurs mails de destinations peuvent disposer, comme le volume de mails ou le ratio de destinataires acceptés et refusés, observés depuis une adresse, impliquant que ces informations sont obtenues directement depuis les Big Mailer Corps. Faites ce que vous voulez de cela.\nEn me basant sur MA compréhension, il s\u0026rsquo;agit d\u0026rsquo;un indicateur intéressant pour limiter la quantité de spam qui vous atteint. Je ne fait pas confiance à cet indicateur concernant les bons scores (parce que je sais à peu près contourner) mais j\u0026rsquo;y fais ÉNORMÉMENT confiance en ce qui concerne les mauvais scores, parce qu\u0026rsquo;il faut vraiment faire un paquet de mauvaises choses en gros volume pour que le score se dégrade. Si vous ne faites pas de mailing en masse, vous êtes censés être soit inconnu de cette base de donnée ou avoir un score supérieur à 95, sinon il y a quelque chose qui déconne chez vous.\nTout le monde n\u0026rsquo;est pas convaincu par SenderScore et certains experts en déliverabilité disent que c\u0026rsquo;est du pipeau. J\u0026rsquo;ai personnellement fait tourner une expérimentation sur plusieurs mois en graphant volume et réputation quotidienne, en comparant aux graphs de distribution. Mon opinion est contraire : il y a une correlation très significative qui permet clairement de classifier les émetteurs. Je pense que la meilleure approche est d\u0026rsquo;utiliser le filtre pour junker, et non bloquer, pour déterminer par vous même si vous en êtes satisfait.\nSenderScore considère que les émetteurs devraient avoir une réputation supérieure à 70. Je pense pour ma part que les hôtes avec un score inférieur à 70 sont de bons candidats au junk, les hôtes avec un score inférieur à 10 sont des rejets évidents.\nInstaller et configurer Dovecot # OpenBSD dispose d\u0026rsquo;un paquet pour Dovecot qui peut être installé d\u0026rsquo;une simple commande :\nmail# pkg_add dovecot dovecot-2.3.7.2v0: ok The following new rcscripts were installed: /etc/rc.d/dovecot See rcctl(8) for details. New and changed readme(s): /usr/local/share/doc/pkg-readmes/dovecot […] Et, specifiquement pour OpenBSD, en suivant les instructions du fichier readme, le fichier /etc/login.conf devrait contenir une classe dovecot pour augmenter les ressources autorisées à Dovecot, gourmand en descripteurs de fichiers :\ndovecot:\\ :openfiles-cur=1024:\\ :openfiles-max=2048:\\ :tc=daemon: Cela fait, il n\u0026rsquo;y a plus grand chose à faire à part pointer Dovecot sur le certificat et la clé TLS générés plus tôt. On le fait en modifiant les clés de configuration ssl_cert et ssl_key dans le fichier /etc/dovecot/conf.d/10-ssl.conf pour qu\u0026rsquo;elles soient comme suit :\nssl_cert = \u0026lt;/etc/ssl/mail.hypno.cat.fullchain.pem ssl_key = \u0026lt;/etc/ssl/private/mail.hypno.cat.key Puis dire à Dovecot que les mails sont livrés dans le répertoire ~/Maildir de chaque utilisateur puisque c\u0026rsquo;est là que OpenSMTPD les déposera. On le fait en modifiant la clé de configuration mail_location dans le fichier /etc/dovecot/conf.d/10-mail.conf pour qu\u0026rsquo;elle soit comme suit :\nmail_location = maildir:~/Maildir On est bon.\nLe daemon peut être activé pour être démarré à chaque reboot, puis démarré immédiatement :\nmail# rcctl enable dovecot mail# rcctl start dovecot dovecot(ok) À ce stade, il est possible de configurer n\u0026rsquo;importe quel client mail comme mutt, thunderbird ou même l\u0026rsquo;application gmail sur Android, et utiliser mail.hypno.cat pour les mails entrants et sortants.\nApprendre à Dovecot à entrainer Rspamd # Vous avez lu jusqu\u0026rsquo;ici ? bien.\nCette section est une section bonus, elle n\u0026rsquo;est absolument pas nécessaire pour configurer votre serveur de mail, mais c\u0026rsquo;est un élément que j\u0026rsquo;aime ajouter parce qu\u0026rsquo;il permet aux utilisateurs d\u0026rsquo;entrainer le filtre antispam d\u0026rsquo;une manière à laquelle ils sont habitués : en déplaçant un mail vers la boite à Spam, ou en appuyant sur un bouton \u0026ldquo;signaler un Spam\u0026rdquo;, \u0026ldquo;marquer comme Spam\u0026rdquo; ou peu importe comment il s\u0026rsquo;appelle dans l\u0026rsquo;interface utilisée. Le bon côté est que cet élément de configuration s\u0026rsquo;intègre avec IMAP, donc peu importe comment les mails sont consultés et depuis quel logiciel, l\u0026rsquo;action de marquer un mail comme Spam entraînera automatiquement le filtre.\nC\u0026rsquo;est très simple en théorie : il suffit de brancher deux scripts, l\u0026rsquo;un qui gère le déplacement dans la boite à Spam, l\u0026rsquo;autre qui gère le déplacement hors de la boite à Spam. Les Big Mailer Corps utilisent des stratégies plus poussées en détectant les mails non ouverts, ceux déplacés directement en poubelle, etc… mais débutons simple, il est toujours possible d\u0026rsquo;étendre ensuite.\nPour faire ce branchement, nous dépendont de Sieve qui est… comment dire avec diplomatie… un peu sur-conçu.\nCe n\u0026rsquo;est pas un élément critique du setup, vous êtes autorisés à éteindre vos cerveaux et suivre les instructions aveuglément. Le pire qui puisse arriver ici est que l\u0026rsquo;entraînement de la détection ne marche pas.\nPour activer imap sieve, le package Pigeonhole doit être installé, encore une fois à l\u0026rsquo;aide d\u0026rsquo;une simple commande :\nmail# pkg_add dovecot-pigeonhole dovecot-pigeonhole-0.5.7.2v1: ok […] Et… voici la partie la plus complexe, il faut configurer Dovecot pour activer imap_sieve et lui apprendre quels scripts utiliser pour quelles actions.\nD\u0026rsquo;abord, on active imap_sieve dans Dovecot en l\u0026rsquo;ajoutant dans la liste des plugins mails pour IMAP. C\u0026rsquo;est fait en mofifiant la clé de configuration mail_plugins dans le fichier /etc/dovecot/conf.d/20-imap.conf :\nprotocol imap { […] mail_plugins = $mail_plugins imap_sieve […] } Ensuite pour apprendre à Dovecot à entrainer Rspamd, on ajoute la section suivante dans le fichier /etc/dovecot/conf.d/90-plugin.conf, pour dire quel script sieve déclencher quand un mail est déplacé vers ou hors du répertoire Junk :\nplugin { sieve_plugins = sieve_imapsieve sieve_extprograms sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment imapsieve_mailbox1_name = Junk imapsieve_mailbox1_causes = COPY APPEND imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve imapsieve_mailbox2_name = * imapsieve_mailbox2_from = Junk imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve imapsieve_mailbox3_name = Inbox imapsieve_mailbox3_causes = APPEND imapsieve_mailbox3_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve } Maintenant que Dovecot est prêt, on prépare la partie Sieve qui dépend de deux scripts pour entraîner le Spam et le Ham dans /usr/local/lib/dovecot/sieve :\n# cat report-ham.sieve require [\u0026#34;vnd.dovecot.pipe\u0026#34;, \u0026#34;copy\u0026#34;, \u0026#34;imapsieve\u0026#34;, \u0026#34;environment\u0026#34;, \u0026#34;variables\u0026#34;]; if environment :matches \u0026#34;imap.mailbox\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;mailbox\u0026#34; \u0026#34;${1}\u0026#34;; } if string \u0026#34;${mailbox}\u0026#34; \u0026#34;Trash\u0026#34; { stop; } if environment :matches \u0026#34;imap.user\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;username\u0026#34; \u0026#34;${1}\u0026#34;; } pipe :copy \u0026#34;sa-learn-ham.sh\u0026#34; [ \u0026#34;${username}\u0026#34; ]; # cat report-spam.sieve require [\u0026#34;vnd.dovecot.pipe\u0026#34;, \u0026#34;copy\u0026#34;, \u0026#34;imapsieve\u0026#34;, \u0026#34;environment\u0026#34;, \u0026#34;variables\u0026#34;]; if environment :matches \u0026#34;imap.user\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;username\u0026#34; \u0026#34;${1}\u0026#34;; } pipe :copy \u0026#34;sa-learn-spam.sh\u0026#34; [ \u0026#34;${username}\u0026#34; ]; Et parce que les deux scripts Sieve dépendent de scripts shell sa-learn-ham.sh et sa-learn-spam.sh, on crée aussi ces deux scripts shell dans /usr/local/lib/dovecot/sieve :\n# cat sa-learn-ham.sh #!/bin/sh exec /usr/local/bin/rspamc -d \u0026#34;${1}\u0026#34; learn_ham # cat sa-learn-spam.sh #!/bin/sh exec /usr/local/bin/rspamc -d \u0026#34;${1}\u0026#34; learn_spam Enfin, on compile les scripts Sieve et on rend les scripts shell exécutables pour que Dovecot puisse les utiliser :\n# sievec report-ham.sieve # sievec report-spam.sieve # chmod 755 sa-learn-ham.sh # chmod 755 sa-learn-spam.sh C\u0026rsquo;est tout. Dorénavant en déplaçant un mail d\u0026rsquo;un répertoire à un autre, on doit voir les lignes suivantes dans le fichier de log /var/log/rspamd/rspamd.log :\n2019-09-13 23:59:46 #18598(controller) \u0026lt;1d44bd\u0026gt;; csession; rspamd_controller_learn_fin_task: \u0026lt;127.0.0.1\u0026gt; learned message as ham: CAHPtQbOxQxBCsVd7nUCP4podu74Pa-F6k28z+4BWfNeeqWYiAg@mail.gmail.com […] 2019-09-14 00:01:57 #18598(controller) \u0026lt;b76e28\u0026gt;; csession; rspamd_controller_learn_fin_task: \u0026lt;127.0.0.1\u0026gt; learned message as spam: CAHPtQbOxQxBCsVd7nUCP4podu74Pa-F6k28z+4BWfNeeqWYiAg@mail.gmail.com […] Il est surement possible de simplifier, je vais être honnête et dire que mon intérêt pour Sieve n\u0026rsquo;est pas assez poussé pour que je me perfectionne. C\u0026rsquo;est une fonctionnalité sur le côté et je ne toucherai probablement pas les scripts avant les dix prochaines années. Je ne suis pas du tout convaincu par Sieve et quand j\u0026rsquo;aurai le temps, je ferais une alternative qui ne me rappelle pas les années m4 de Sendmail.\nTester le tout # Et maintenant, on va juste tester que l\u0026rsquo;on arrive à faire un trajet complet aller-retour de mail depuis hypno.cat vers gmail.com.\nTout d\u0026rsquo;abord, je prépare un mail depuis mon compte hypno.cat vers mon compte gmail.com :\nAprès envoi, je vérifie qu\u0026rsquo;il arrive bien chez gmail.com :\nEnsuite, je vérifie que l\u0026rsquo;émission a bien eu lieu par dessus un canal sécurisé par TLS :\nJe vérifie également que gmail.com est content avec notre déclaration SPF, notre signature DKIM et qu\u0026rsquo;il a bien vu notre enregistrement DMARC. On fait cela en sélectionnant \u0026ldquo;Voir l\u0026rsquo;original\u0026rdquo; dans le menu associé à chaque mail :\nEnfin, j\u0026rsquo;y répond pour être sur que ça fonctionne dans les deux sens :\nCool, on a fini # Cet article est dense, je voulais expliquer pourquoi on fait les choses. Si vous retirez toutes les explications et gardez juste les détails purement techniques, vous réaliserez que nous avons construit une infrastructure de mail, fournissant des services entrants et sortants sécurisés par TLS, avec un trafic sortant respectant DKIM, SPF et DMARC, et un trafic entrant filtrant le spam.\nOn a fait cela en :\najoutant deux enregistrements A, un MX et trois TXT dans notre zone DNS; faisant attention à avoir un rDNS correctement configuré installant un certificat TLS préparant un fichier de configuration d\u0026rsquo;environ 15 lignes pour un serveur SMTP modifiant 3 lignes dans la configuration par défaut d\u0026rsquo;un serveur IMAP modifiant 8 lignes dans la configuration par défaut d\u0026rsquo;une solution antispam Et parce que nous étions de tres bonne humeur et volontaires pour faire un petit pas de plus, nous avons implémenté un apprentissage Spam / Ham de Rspamd en :\nmodifiant environ 20 lignes dans la configuration par défaut d\u0026rsquo;un serveur IMAP copiant 4 scripts Sieve dans un répertoire On admettra qu\u0026rsquo;un nouvel arrivant va devoir fournir quelques efforts pour trouver tout ça sans aide, mais aucune de ces tâches n\u0026rsquo;est réellement difficile ou complexe. De plus, la plupart sont des opérations de mises en place qui ne sont réalisées qu\u0026rsquo;une fois ; on n\u0026rsquo;édite pas la zone DNS ou un reverse DNS tous les deux jours, tout comme on ne touche pas une configuration en place qui fonctionne tant qu\u0026rsquo;il n\u0026rsquo;y a pas de nouveau cas d\u0026rsquo;utilisation.\nPour citer les plus réfractaires à l\u0026rsquo;auto-hébergement, il peut y avoir des maintenances pour gérer les conséquences d\u0026rsquo;une blacklist, mais il n\u0026rsquo;y a aucun scénario dans lequel ladite maintenance impliquerait autant de travail que la mise en place que nous venons de faire… et qui au final se réplique en 10 minutes chrono une fois réalisée une première fois pour se faire la main.\nQuelle est la prochaine étape ? # Votre prochaine étape est de mettre en place de la redondance pour vous assurer qu\u0026rsquo;une panne de votre serveur de mail n\u0026rsquo;entraine pas de perte de mail.\nEn pratique, la plupart des serveurs de mails retentent une émission de multiples fois si la destination est inaccessible donc, même en cas de panne, la plupart de vos mails seront temporisés par votre émetteur et retransmis quand votre serveur sera de nouveau accessible.\nEn théorie, on veut tout de même faire les choses bien, et reposer sur la responsabilité des autres à gérer nos pannes n\u0026rsquo;est pas très poli. Vous devez mettre en place un serveur secondaire, vous arranger entre amis pour être serveurs secondaires les uns des autres, ou même vous souscrire à un service qui offre du mail secondaire. N\u0026rsquo;importe quoi qui puisse permettre de couvrir vos arrières lorsque votre serveur primaire tombe.\nEnsuite, peu importe la méthode choisie, il vous faudra :\najouter un enregistrement MX supplémentaire dans votre zone DNS configurer le serveur secondaire pour retransmettre les mails au serveur primaire On est loin de la haute voltige.\nEst-ce qu\u0026rsquo;on a vraiment fini ? # Je vais probablement écrire une série d\u0026rsquo;article pour discuter de la réputation, ainsi que des mécanismes en place chez les Big Mailer Corps qui provoquent des pannes de distribution.\nSi ce sont des sujets d\u0026rsquo;intérêt et que j\u0026rsquo;ai du feedback positif, j\u0026rsquo;écrirais plus souvent sur les concepts autour de la déliverabilité, sinon je reprendrais mes anciennes activités : écrire mes rapports mensuels sur mes contributions opensource. À vous de me dire :-)\nFinalement, dans cet article j\u0026rsquo;ai décrit un setup vraiment simple mais il y a des tonnes de choses intéressantes à faire avec les infrastructures de mail, si vous n\u0026rsquo;avez pas peur d\u0026rsquo;aller plus loin.\nJe construis actuellement une infrastructure de mail pour un future service commercial d\u0026rsquo;hébergement. L\u0026rsquo;infrastructure s\u0026rsquo;étend sur plusieurs data centers dans plusieurs pays, a des serveurs secondaires capable d\u0026rsquo;encaisser des pannes très sérieuses des serveurs primaires, peut facilement monter à l\u0026rsquo;échelle sur des pics de volume, elle décorelle le trafic entrant et sortant pour permettre de fournir des services partiels, peut facilement re-router le trafic entre des machines pour contourner des pannes, tout en fournissant des services IMAP à des comptes virtuels sur une multitude de domaines, avec des backups quotidiens. L\u0026rsquo;infrastructure me coute… moins de 75 EUR par mois. Elle est également très simple, la plupart des machines sont des installations par défaut d\u0026rsquo;OpenBSD avec très peu de changements au système de base.\nSi lire à propos de ce genre de mises en place vous intéresse, je peux écrire à ce sujet pour aider un maximum de personnes à monter un maximum d\u0026rsquo;alternatives aux Big Mailer Corps, ce qui est au final mon souhait.\nMerci de m\u0026rsquo;avoir lu, je n\u0026rsquo;ai pas de soundcloud, mais j\u0026rsquo;ai un patreon et un github si vous voulez me sponsoriser, ou utiliser les icones en dessous du lien de commentaires pour partager cet article sur les reseaux sociaux.\n","date":"23 December 2019","permalink":"/posts/2019-12-23/mettre-en-place-un-serveur-de-mail-avec-opensmtpd-dovecot-et-rspamd/","section":"Posts","summary":"TL;DR: - Pas de résumé, j'ai passé des heures à traduire, vous allez passer des minutes à lire ;) - OK… J'ai expliqué avec BIEN TROP DE DÉTAILS comment mettre en place un serveur de mail Merci à mes sponsors !","title":"Mettre en place un serveur de mail avec OpenSMTPD, Dovecot et Rspamd"},{"content":" TL;DR: - SMTP est la méthode dont les ordinateurs échangent des e-mails - il s'agit d'un protocole décentralisé, ce qui signifie que CHACUN peut héberger un nœud et être indépendant - il est en train d'être centralisé dans des sociétés qui ont un passif d'abus - il est en train d'être centralisé dans un pays qui a un passif d'abus Où est-ce que j\u0026rsquo;ai déjà lu ça ? # En Août, j\u0026rsquo;ai publié un petit article intitulé \u0026ldquo; You should not run your mail server because mail is hard\u0026rdquo; (\u0026ldquo;Vous ne devriez pas héberger votre serveur de mail parce que c\u0026rsquo;est dur\u0026rdquo;) qui était, en gros, mon opinion sur les différentes raisons qui poussent les gens à décourager l\u0026rsquo;hébergement de mails. De manière très inattendue, l\u0026rsquo;article est devenu assez populaire, atteignant 100K lectures et continuant à recevoir des visites et des commentaires plusieurs mois après publication.\nEn Septembre, j\u0026rsquo;ai publié un autre article beaucoup plus long intitulé \u0026ldquo; Setting up a mail server with OpenSMTPD, Dovecot and Rspamd\u0026rdquo; (\u0026ldquo;Installer un serveur de mail avec OpenSMTPD, Dovecot et Rspamd\u0026rdquo;) qui décrivait pas à pas, de façon très détaillée, les étapes pour installer un serveur de mail complet jusqu\u0026rsquo;à la livraison des e-mails en boîte de réception chez un gros hébergeur de mail américain. L\u0026rsquo;article est devenu lui aussi plutôt populaire, sans pour autant atteindre le niveau du précedent article moins technique et spécifique, mais atteignant 40K lectures et continuant également à recevoir des visites et des commentaires plusieurs mois après publication.\nLe contenu que vous vous apprêtez à lire est extrait de ce second article auquel il n\u0026rsquo;aurait pas dû être rattaché. Il est bien trop (géo-)politique pour faire partie d\u0026rsquo;un article technique, j\u0026rsquo;ai donc décidé de l\u0026rsquo;en retirer et d\u0026rsquo;en faire un dédié. Je ne veux pas que les spécificités d\u0026rsquo;un article sur OpenSMTPD aille à l\u0026rsquo;encontre d\u0026rsquo;un message beaucoup plus général.\nC\u0026rsquo;est mon premier article en français depuis des années, je vous demande donc un peu d\u0026rsquo;indulgence : si vous trouvez des fautes, vous pouvez me les remonter pour que je les corrige, ou faire une pull request pour les techies.\nN\u0026rsquo;hésitez pas à partager la publication à l\u0026rsquo;aide des icones en fin d\u0026rsquo;article et à la commenter \u0026lt;3\nQui sont Big Mailer Corps ? # Je ne vais pas pointer de doigts vers des directions en particulier. Concrètement, rentrent dans ma définition des Big Mailer Corps, toute grosse entreprise qui centralise l\u0026rsquo;hébergement des e-mails d\u0026rsquo;un très grand nombre d\u0026rsquo;utilisateurs :\n- Orange ? SFR ? La Poste ? - Non, plus grand, plus mondial, … Et là, normalement le doute est dissipé, pas besoin de citer de nom. Vous avez compris qui est visé parce que TOUT LE MONDE connait les Big Mailer Corps, la plupart d\u0026rsquo;entre vous avez une adresse chez l\u0026rsquo;un d\u0026rsquo;entre eux, et la plupart de vos contacts sont hébergés chez l\u0026rsquo;un d\u0026rsquo;entre eux.\nL\u0026rsquo;auto-hébergement et encourager les petits fournisseurs est meilleur pour notre bien commun # Il y a des conséquences à la centralisation des services de mail chez les Big Mailer Corps.\nIl n\u0026rsquo;y a aucun sens à ce que Jacques Lambda, qui partage des photos de chatons avec sa famille et ses amis, se construise une infrastructure d\u0026rsquo;hébergement de mails alors que plusieurs Big Mailer Corps lui offrent, \u0026ldquo;gratuitement\u0026rdquo;, une qualité de service au top. Il peut obtenir une adresse e-mail immédiatement disponible et qui marche de manière parfaitement fiable. Il n\u0026rsquo;y a absolument aucun sens à ce que Jacques Lambda n\u0026rsquo;aille pas chez l\u0026rsquo;un des Big Mailer Corps, particulièrement quand même les techies font ce choix sans hésitation, confirmant que c\u0026rsquo;est plutôt une bonne option.\nIl n\u0026rsquo;y a rien de mal à ce que tous les Jacques Lambda choisissent un service qui marche.\nCe qui est terriblement mauvais par contre, c\u0026rsquo;est le centralisation d\u0026rsquo;un protocole de communication dans les mains d\u0026rsquo;une poignée d\u0026rsquo;entreprises commerciales, CHACUNE D\u0026rsquo;ENTRE ELLES provenant du même pays (actuellement dirigé par un lunatique qui abuse de son pouvoir et souffre probablement de trouble de la personnalité narcissique), CHACUNE D\u0026rsquo;ENTRE ELLES étant apparu dans les informations et/ou dans une cour de tribunal pour tout un assortiment de comportements \u0026ldquo;déplaisants\u0026rdquo; (abus de la vie privée, espionnage, abus de monopole, harcellement professionnel ou sexuel, j\u0026rsquo;en passe… ), et CHACUNE D\u0026rsquo;ENTRE ELLES faisant croître des bases d\u0026rsquo;utilisateurs qui dépassent de loin la population totale de plusieurs pays combinés.\nMettons un peu de perspective. Le plus gros des Big Mailer Corps annonce une base d\u0026rsquo;utilisateur qui dépasse 1.4 MILLARD d\u0026rsquo;utilisateurs (Avril 2018), soit à peu près la population totale de la Chine ou de l\u0026rsquo;Inde, plus de quatre fois la population totale des USA, ou plus de vingt fois la population totale de la France. Si vous comptiez les utilisateurs au rythme d\u0026rsquo;un par seconde, il vous faudrait 44 ans pour en faire le tour.\nEnsuite, très loin derrière, il est suivi par le prochain Big Mailer Corp qui annonce une base de 400 millions d\u0026rsquo;utilisateurs (2018 également), dépassant la population totale des USA, atteignant quatre fois la population totale de l\u0026rsquo;Égypte, et cinq fois la population totale de la France.\nDans la plupart des rapports de mailing que j\u0026rsquo;ai pu surveiller, le premier des Big Mailer Corps couvre approximativement la moitié des adresse e-mails de destinataires… l\u0026rsquo;autre moitié étant également couverte en large partie par les autres Big Mailer Corps.\nCe n\u0026rsquo;est pas sain. Ni ici, ni nulle part, ni maintenant, ni jamais, ni dans aucune dimension alternative (insérer \u0026ldquo;sliiiiiiders\u0026rdquo; en chuchotement mental ici).\nPrenez un moment pour bien enregistrer ce qui suit :\nSi ces grosses entreprises étaient contraintes de couper les communications avec un pays pour appliquer des sanctions, bien au delà d\u0026rsquo;un MILLIARD de personnes seraient hors de portée pour le pays visé. Et si vous pensez la crainte exagérée, les utilisateurs de Github en Iran, en Syrie, à Cuba ou en Crimée pourraient vous donner un point de vue différent après avoir découvert qu\u0026rsquo;ils étaient mis à la porte pour appliquer les sanctions américaines sur leurs pays.\nAussi ennuyant soit-il, Git reste un outil de techies qui n\u0026rsquo;impacte principalement que des techies, une minorité d\u0026rsquo;humains qui sait trouver facilement des solutions de contournement… facilitées par le fait que Git est décentralisé et ne nécessite pas d\u0026rsquo;interconnexions et d\u0026rsquo;interopérabilité : les utilisateurs peuvent facilement bouger ailleurs et s\u0026rsquo;auto-héberger. Aucun lien coupé avec aucun pays ne peut empêcher ça.\nLe mail fonctionne de façon différente. Il impacte tout le monde, des personnes de tous âges et qui ne sont pas forcément des techies. Il est toujours possible de trouver une solution de contournement en cas de blocage, mais ce n\u0026rsquo;est pas simple pour tout le monde. Et comme il s\u0026rsquo;agit d\u0026rsquo;un protocole pour mettre en relation un émetteur A avec un destinataire B, il y a un besoin d\u0026rsquo;interconnexion, un besoin d\u0026rsquo;interopérabilité ET besoin que le lien entre A et B ne soit pas coupé. Les conséquences d\u0026rsquo;un blocage similaire à celui de Github chez les Big Mailer Corps seraient plusieurs ordres de magnitude plus durs et visibles : un pan des utilisateurs d\u0026rsquo;Internet ne serait tout bonnement plus joignable par les citoyens des pays sanctionnés.\nLes habitants des pays de merde ne sont pas tous des techies, ils sont des personnes qui ont BESOIN d\u0026rsquo;interconnexions et d\u0026rsquo;interopérabilité pour communiquer avec plus d\u0026rsquo;un MILLIARD de personnes hébergées chez les Big Mailer Corps. Le mail permet de connecter les populations mondiales de manière fiable… UNIQUEMENT si le mail reste décentralisé. Il ne marche plus s\u0026rsquo;il est centralisé dans une poignée d\u0026rsquo;entreprises toutes localisées dans le même pays.\nBien sûr cela n\u0026rsquo;arrivera peut être jamais, et je metterai même mon billet dessus parce qu' un pays connu pour avoir intercepté et épié les communications mondiales a plus d\u0026rsquo;intérêt à conserver le flot des communications qu\u0026rsquo;à le couper, mais le seul fait que ce soit TECHNIQUEMENT possible devrait être suffisant pour nous mettre mal à l\u0026rsquo;aise. J\u0026rsquo;ignore même volontairement que la raison pour laquelle ils ne le feront pas me rends également mal à l\u0026rsquo;aise. Si les Big Mailer Corps étaient hébergés en Chine, nous serions déjà en train d\u0026rsquo;envisager les conséquences d\u0026rsquo;une coupure et d\u0026rsquo;envisager un plan de sortie, mais nous ne le faisons pas parce que nous pensons que nous sommes du bon côté de la barrière, et que des sanctions à notre encontre ne pourraient pas arriver.\nMais ça, c\u0026rsquo;est jusqu\u0026rsquo;à ce que l\u0026rsquo;on soit complètement au pied du mur sans solution.\nÀ ce stade, je pourrais également faire semblant d\u0026rsquo;ignorer que le modèle économique de la plupart de certaines de ces entreprises repose sur la création d\u0026rsquo;un profil commercial de leurs utilisateurs et qu\u0026rsquo;avoir à disposition tous les e-mails est une mine d\u0026rsquo;or.\nCertains Big Mailer Corps se défendent d\u0026rsquo;exploiter le contenu des e-mails, mais… il n\u0026rsquo;y a pas besoin de connaître le contenu des e-mails si les expéditeurs et destinataires sont écrits sur l\u0026rsquo;enveloppe. Il est tout à fait possible de savoir que vous essayer de souscrire un crédit en faisant des comparatifs, mais que vous recevez également des remboursements de santé réguliers et que vous recevez des informations concernant un certain type d\u0026rsquo;appareillage médical. Un profil très précis et qui serait très intéressant pour des organismes de prêt ou des assurances peut tout à fait être dressé sans avoir à \u0026ldquo;lire\u0026rdquo; les e-mails… mais bon, ce ne sont que des spéculations basées sur quelque chose de techniquement réalisable.\nDans la même idée, je vais faire semblant d\u0026rsquo;ignorer que tous les documents partagés peuvent être analysés pour détecter des logiciels malveillants, que le document soit confidentiel ou non, qu\u0026rsquo;il soit stratégique ou non. Tout comme les liens échangés par mail, qu\u0026rsquo;ils soient stratégiques ou non, perdent tout caractère confidentiel lorsqu\u0026rsquo;ils passent par le webmail d\u0026rsquo;un Big Mailer Corp.\nAlors OUI, il faut plus de personnes qui s\u0026rsquo;auto-hébergent ou qui dépendent d\u0026rsquo;hébergeurs géolocalisés, tissant un réseau de communication le plus vaste au-travers de plusieurs opérateurs dans plusieurs pays. Je ne pense évidemment pas que tout le monde doit s\u0026rsquo;auto-héberger, mais je pense que DAVANTAGE de monde doit sortir des Big Mailer Corps pour aller n\u0026rsquo;importe où ailleurs… il faut réintroduire de la contrainte chez les les Big Mailer Corps, qu\u0026rsquo;ils soient obligés d\u0026rsquo;interagir avec des opérateurs en dehors de leur gang, et qu\u0026rsquo;il y ait un risque financier SI soudainement ils prenaient des décisions à l\u0026rsquo;encontre des populations.\nLes Big Mailer Corps ne sont pas mauvais ou méchants, j\u0026rsquo;utilise moi-même leurs services régulièrement, et la plupart de leurs employés cherchent probablement le bien commun… mais ces entreprises sont devenues trop grosses et puissantes, il faut qu\u0026rsquo;il y ait un équilibre parce que l\u0026rsquo;on ne sait pas comment elles évolueront dans les années à venir, on ne sait pas non plus comment évoluera la politique de leurs pays d\u0026rsquo;origine dans les années à venir, et les informations récentes ne décrivent pas un avenir tout rose dans la bonne direction.\nJe concluerai en vous recommandant de regarder cette excellente présentation de Bert Hubert ( @PowerDNS_Bert) de PowerDNS, à propos de problèmes similaires qui sont en train d\u0026rsquo;émerger avec le protocole DNS et les craintes qui en découlent autour de la surveillance. De très nombreux points de sa présentation sont tout aussi valides en ce qui concerne les services de mail.\n","date":"15 December 2019","permalink":"/posts/2019-12-15/decentralisons-smtp-pour-le-bien-commun/","section":"Posts","summary":"TL;DR: - SMTP est la méthode dont les ordinateurs échangent des e-mails - il s'agit d'un protocole décentralisé, ce qui signifie que CHACUN peut héberger un nœud et être indépendant - il est en train d'être centralisé dans des sociétés qui ont un passif d'abus - il est en train d'être centralisé dans un pays qui a un passif d'abus Où est-ce que j\u0026rsquo;ai déjà lu ça ?","title":"Décentralisons SMTP pour le bien commun"},{"content":" TL;DR: - SMTP is the way computers exchange e-mails - it is a decentralised protocol meaning that ANYONE can run a node and be independant - it is being centralised at companies that have a history of abuse - it is being centralised in a country that has a history of abuse Where did you read this already ? # In August, I published a small article titled \u0026ldquo; You should not run your mail server because mail is hard\u0026rdquo; which was basically my opinion on why people keep saying it is hard to run a mail server. Unexpectedly, the article became very popular, reached 100K reads and still gets hits and comments several months after publishing.\nAs a follow up to that article, I published in September a much lenghtier article titled \u0026ldquo; Setting up a mail server with OpenSMTPD, Dovecot and Rspamd\u0026rdquo; which described how you could setup a complete mail server. I went from scratch and up to inboxing at various Big Mailer Corps using an unused domain of mine with a neutral reputation and describing precisely for each step what was done and why it was done. The article became fairly popular, nowhere near the first one which wasn\u0026rsquo;t so technical, but reached 40K reads and also still gets hits and comments several months after publishing.\nThe content you\u0026rsquo;re about to read was part of the second article but it didn\u0026rsquo;t belong there, it was too (geo-)political to be part of a technical article, so I decided to remove it from there and make it a dedicated one. I don\u0026rsquo;t want the tech stack to go in the way of the message, this is not about OpenSMTPD.\nSelf-hosting and encouraging smaller providers is for the greater good # There are political consequences to centralizing mail services at Big Mailer Corps.\nIt doesn\u0026rsquo;t make sense for Random Joe, sharing kitten pictures with his family and friends, to build a personal mail infrastructure when multiple Big Mailer Corps offer \u0026ldquo;for free\u0026rdquo; an amazing quality of service. They provide him with an e-mail address that is immediately available and which will generally work reliably. It really doesn\u0026rsquo;t make sense for Random Joe not to go there, and particularly if even techies go there without hesitation, proving it is a sound choice.\nThere is nothing wrong with Random Joes using a service that works.\nWhat is terribly wrong though is the centralization of a communication protocol in the hands of a few commercial companies, EVERY SINGLE ONE OF THEM coming from the same country (currently led by a lunatic who abuses power and probably suffers from NPD), EVERY SINGLE ONE OF THEM having been in the news and/or in a court for random/assorted \u0026ldquo;unpleasant\u0026rdquo; behaviors (privacy abuses, eavesdropping, monopoly abuse, sexual or professional harassment, you just name it\u0026hellip;), and EVERY SINGLE ONE OF THEM growing user bases that far exceeds the total population of multiple countries combined.\nLet\u0026rsquo;s put a bit of perspective. The biggest one of them reports a user base that exceeds 1.4 BILLION users (April 2018), roughly the entire population of either one of China or India, exceeding by four the population of United States by itself, or by over twenty the population of my own country. If you counted its users at the rate of one each second, it would take you over 44 years to go through.\nThen, very far below, it is followed by the next one which reported 400 million active users (2018 too), also exceeds the population of the United States, being four times the population of Egypt, and five times the population of my own country.\nIn many of the mailings I have monitored, the very first one covered approximately half the recipients e-mail address space\u0026hellip; before the other half was even split in large parts between the other Big Mailer Corps.\nThis is NOT sane, not here, not anywhere, not in any alternate dimension (insert \u0026ldquo;sliiiiiiders\u0026rdquo; whisper here).\nTake a moment to let this sink in:\nIf these companies were somehow required to cut communications with a country for sanctions, well over a BILLION people could be out of reach for the targeted country. If you think that this is far-fetched, the users from Iran, Syria, Cuba or Crimea could surely provide you with an alternate point of view after discovering they were kicked out of Github to enforce sanctions on their countries. But as annoying as it is, git is still a techie thing which mostly impacts techie people, a minority of human beings\u0026hellip; and it is also decentralized. Even if Github services are stopped, the users can move to another platform or self-host easily, there\u0026rsquo;s no need for interoperability with the punishing country if it wants to cut ties.\nE-mails are much different: they affect everyone, from all age and all backgrounds, the consequences of a similar block if it was done by Big Mailer Corps are orders of magnitude harsher. The people of shithole countries are not only techies that can work around, they are people who NEED interoperability to communicate with the billion and more of people hosted at Big Mailer Corps. Mail is good for connecting people worldwide in a reliable way, but this only works if you keep it decentralized, NOT when you centralize it in the hands of a few companies all geolocalized in the same country.\nNow that might never happen, and I would bet heavy money it won\u0026rsquo;t because a country known for intercepting communications and spying on the whole world has far more interest in keeping communications flowing, but the sole fact that it is TECHNICALLY possible should be enough to make us all uneasy. I\u0026rsquo;m even voluntarily looking away from the reason why I think they won\u0026rsquo;t do it as if it didn\u0026rsquo;t matter. If Big Mailer Corps were hosted in China we\u0026rsquo;d be dead worried and already considering a plan out, but the only reason we\u0026rsquo;re not is because we think we\u0026rsquo;re currently on the \u0026ldquo;right side\u0026rdquo; of the fence.\nBut that\u0026rsquo;s until we\u0026rsquo;re screwed.\nSo YES, I\u0026rsquo;d rather see a lot more people self-host or rely on smaller geolocalized providers, spreading the SMTP network as much as possible across multiple operators in multiple countries. I don\u0026rsquo;t think everyone should self-host, I just think MORE people should self-host or move away from Big Mailer Corps into ANY other provider\u0026hellip; so there is at least a bit of constraint for them to interoperate with operators outside their gang, as well as a financial risk for them IF they suddenly went rogue.\nI\u0026rsquo;ll say it again:\nI don\u0026rsquo;t think that either one of the Big Mailer Corps are evil or bad, I use some of their services on a daily basis, and most of the people operating them are genuinely seeking the greater good\u0026hellip; however they have grown too big and there needs to be a balance in power because who knows how they\u0026rsquo;ll evolve in the next ten years, who knows how the politics of their home country will evolve in the next ten years, and recent news doesn\u0026rsquo;t paint them as heading in the right direction.\nI\u0026rsquo;ll conclude by recommending that you see this excellent presentation by Bert Hubert ( @PowerDNS_Bert) from PowerDNS, about how a similar problem is starting to happen with DNS and the privacy and tracking concerns that arise from this. Many, many, many key points are also valid for mail services.\n","date":"15 December 2019","permalink":"/posts/2019-12-15/decentralised-smtp-is-for-the-greater-good/","section":"Posts","summary":"TL;DR: - SMTP is the way computers exchange e-mails - it is a decentralised protocol meaning that ANYONE can run a node and be independant - it is being centralised at companies that have a history of abuse - it is being centralised in a country that has a history of abuse Where did you read this already ?","title":"Decentralised SMTP is for the greater good"},{"content":" TL;DR: - greylisting is a sound idea - yet it is not really practical today - people tend to disable it or find work-arounds - SPF-aware greylisting makes greylisting usable again SMTP failures in a nutshell # SMTP is a fail-safe protocol which attempts very hard to ensure that messages do not get lost once they are in transit. Among the various mechanisms and requirements in place is the use of \u0026ldquo;Temporary Failures\u0026rdquo;.\nBasically, an SMTP node has two final states for a mail. Either it is delivered, and the recipient has the mail, or it\u0026rsquo;s not, and the recipient doesn\u0026rsquo;t have the mail. For the latter, the SMTP requires that a node notifies the sender that an error occured either by rejecting during the SMTP transaction or by sending a deferred bounce (also known as MAILER-DAEMON).\nThen, there\u0026rsquo;s a third state that\u0026rsquo;s not final: temporarily failed. It covers any transient error that prevented delivery\u0026hellip; but which might result in a delivery (or a permanent failure) if the mail was retried after something was fixed by the destination node.\nUnlike delivered mail, which the recipient sees, and failed mail, which the sender sees, the temporarily failed mails are usually not seen by users. They are handled between the mail exchangers (MX) retrying behind the scene, and only ever seen by users when the situation is either resolved or the mail time-to-live has expired. In the first case, the recipient will get the mail delivered a bit later than expected, often without even knowing that a retry took place. In the second case, the sender will get a notification that, despite retries, the mail could not be delivered after an extended period of time.\nThe retry strategies are handled differently by different software, but you can expect that you\u0026rsquo;ll see retries happening seconds or minutes after a temporary failure. A popular approach, the quadratic delay increase, causes the delay between retries to start short, but then grow with the number of retries so that the longer a destination host is unable to handle a mail, the longer between the retries from a sender.\nTemporary failures happen all the time, it is a normal thing in the SMTP world, looking at log you will very often see lines such as this:\n4b3a6c195c1f6010 mta delivery evpid=8f26ea98359eccea from=\u0026lt;misc+bounces-4541-[redacted]=tin.it@opensmtpd.org\u0026gt; to=\u0026lt;[redacted]@tin.it\u0026gt; rcpt=\u0026lt;-\u0026gt; source=\u0026quot;45.76.46.201\u0026quot; relay=\u0026quot;62.211.72.32 (smtp.tin.it)\u0026quot; delay=0s result=\u0026quot;TempFail\u0026quot; stat=\u0026quot;421 \u0026lt;[redacted]@tin.it\u0026gt; Service not available - too busy\u0026quot; This is not just for small destinations, I know of two Big Mailer Corps that have temporary issues pretty much always, but they also have so many mail exchangers that a retry is unlikely to hit the same MX twice in a row, so mail usually gets delivered at first or second try.\nA lot more could be said but lets keep this article simple and summarize:\nThere exists a mechanism in SMTP which allows a destination MX to tell a sender MX that it should retry a delivery. The retries happen behind the scene, without any action from the users, and can cause a delay in delivery.\nGreylisting explained # Spam is a volume industry.\nSpammers are not interested in targeting specific recipients, they are interested in targeting a large amount of recipients so that it statistically increases the number of recipients that will fall for it.\nI won\u0026rsquo;t dig into this too much because it is the topic of another article I\u0026rsquo;m writing, however you have to keep in mind that there are different kinds of spammers, and that while some use regular MX that respect the retry requests, a lot more rely on stateless scripts or custom MX code that DOES NOT honor the retries because\u0026hellip; it slows them down considerably for uncertain result. Waiting minutes to retry a recipient just to end up resolving to an \u0026ldquo;unknown user\u0026rdquo; permanent failure is not interesting, it clutters the queue and it prevents sending other domains at full speed.\nThe idea behind greylisting is very simple:\nGreylisting causes a temporary failure to be triggered for MX you don\u0026rsquo;t trust and keeps track of these retry requests. If the retry happens too soon, it is ignored, so spammers can\u0026rsquo;t just perform immediate retries. If the retry happens too late, well\u0026hellip; it is too late and the implementation may do different things ranging from blacklisting, to degrading reputation, to forgetting the initial request and causing greylisting to start all over again, and again, and again, \u0026hellip; The bottom line is that spammers are forced to keep state and behave like an RFC-compliant implementation, which goes against the economics of sending en masse.\nIn the other hand, RFC-compliant MX naturally try again multiple times so they eventually fall in the correct window for a retry and are whitelisted.\nHow greylisting usually works # There are different greylisting solutions and they don\u0026rsquo;t all work the same.\nThe general idea is that you have a database which keeps track of the window of time between which a retry is considered as valid to pass greylisting for a given MX. However, how the MX is considered for a retry varies and while some solutions will just look at the source IP address, others will also look at the MAIL FROM, others will also look at the RCPT TO, and based on my own studying of some failure patterns, the unique Message-ID is sometimes also tracked.\nThe OpenBSD spamd(8) for example tracks the source IP address, the identification hostname provided at HELO/EHLO, the envelope-from (MAIL FROM) and the envelope-to (RCPT TO). So, to pass greylisting, an MX must retry during the proper window of time, coming from the same IP address, identifying with the same HELO/EHLO, from the same MAIL FROM and to the same RCPT TO.\nThe problem with greylisting and Big Mailer Corps # For small destinations, greylisting works nicely as they tend to use the same MX for incoming and outgoing trafic. The MX will contact you, be told to retry later, then be accepted when it comes back after a minute or so.\nThen, you have Gmail / Yahoo / Outlook / \u0026hellip; which not only have different MX for incoming and outgoing trafic, but also have dozens and dozens of outgoing mail exchangers. This leads to a situation where an initial MX contacts you, gets asked to retry, but you never see a retry because it came from another MX which was asked to retry, and so on\u0026hellip;\nGreylisting with such hosts is unusable and results in mails being delayed for hours, days, or even expire and never reach you.\nBut at the same time, greylisting keep a lot of nuisance outside too as it kills most scripts and compromised computers acting as spamming bots, so a lot of operators put in place various work-arounds to keep using greylisting for small senders and not for Big Mailer Corps.\nWork-around strategies # I used the plural but quite frankly there\u0026rsquo;s only one strategy: whitelisting.\nThe work-around is to figure out IP addresses of Big Mailer Corps, then give them a free pass and allow them to skip greylisting.\nDoes this defeat greylisting ? Not really. Greylisting kills non-RFC compliant MX, but they operate RFC-compliant MX and if they retried from the same outgoing MX they would pass greylisting hands up.\nFair enough, let\u0026rsquo;s whitelist them\u0026hellip; but how ?\nFor a while, I used a naive approach which consisted in whitelisting the /24 of any Big Mailer Corps MX that would get greylisted. After a while, my list was big enough that new ones would rarely get greylisted. It was tedious, it didn\u0026rsquo;t cover the case of new ones hitting me from new ranges, and I ended up merging lists from various people with mine to make sure I add as many as possible. This resulted in a constantly growing whitelist.\nThis was not really smart because all of the Big Mailer Corps have SPF. They actually advertise in a DNS record which IP they will contact you with, so that whenever you receive a connection claiming to be from them, you can do a lookup and check if it is legit. So, with that in mind, I wrote spfwalk (now part of smtpctl) in January 2018, a tool that would walk through the SPF record of domains and extract as many IP addresses as it could. There is a write up about spfwalk on this blog if you\u0026rsquo;re interested. This tool was intended to replace the dummy approach I used until then, but it is in no way perfect, it is best effort.\nThe way SPF records work means that it\u0026rsquo;s easy to check if an IP matches a policy but not to extract a list of IP addresses for later check. For example, some policies expect runtime reverse DNS resolution of the source IP address to check if they match a domain, so there\u0026rsquo;s literally NO IP address to extract from the record. So\u0026hellip; this works for most cases until you hit one where it doesn\u0026rsquo;t.\nA lot of people are happy with SPF walking, it does improve the situation considerably, but it is very hackish and I personally don\u0026rsquo;t use that tool anymore as I feel it\u0026rsquo;s not the right way to tackle the issue.\nThe filter-greylist proof of concept # Last month, I wrote a proof-of-concept filter for OpenSMTPD which uses a different approach and that is available on Github.\nLet me clarify first that I don\u0026rsquo;t claim to have invented this, it just occured to me that it was a good idea so I implemented it and it may already be available elsewhere. Since I haven\u0026rsquo;t looked very hard, (that\u0026rsquo;s a synonym for \u0026ldquo;at all\u0026rdquo;), not only do I not know of another implementation in another opensource software, but if such an implementation exists, I\u0026rsquo;m also unaware if my implementation tackles the idea the same way.\nThe base idea is that we don\u0026rsquo;t really want Big Mailer Corps to skip the greylisting, we resort to whitelisting SPF because what we really want is for all IP addresses from a Big Mailer Corp to be considered as equivalent when it comes to greylisting retries. And by whitelisting IP addresses from the SPF record, what we really want is to consider the IP addresses from the SPF record as equivalent\u0026hellip; In other words, what we want is SPF-aware greylisting.\nThe proof of concept considers that there are two kinds of senders: those not publishing SPF records and those publishing SPF records.\nMX that do not publish SPF are greylisted by source address, like they would with a regular greylisting. This doesn\u0026rsquo;t affect a lot of legitimate senders because Big Mailer Corps require senders to provide SPF in order not to degrade their reputation, and because Big Mailer Corps represent such a large e-mail address space, legitimate senders tend to comply.\nThen, you have the MX that do publish SPF records, which includes all Big Mailer Corps for obvious reasons. And for these, instead of greylisting by source address, we can do an SPF-aware greylisting for the domain.\nAssuming this is the first mail we receive from an MX, when a new connection arrives, the filter tracks the source IP address. The SMTP session moves forward, initiates an SMTP transaction, and then the client provides the envelope-from (MAIL FROM):\nS: 220 in.mailbrix.mx ESMTP OpenSMTPD C: EHLO localhost S: [...] S: 250 in.mailbrix.mx Hello localhost [127.0.0.1], pleased to meet you C: MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; At this point, the filter can do an SPF lookup for the envelope-from domain, poolp.org. If it doesn\u0026rsquo;t find an SPF record, it can simply keep track of the source address and issue a retry request.\nIf it finds an SPF record, it checks if the source IP address is valid. If it is not, then it supposedly doesn\u0026rsquo;t come from the sender domain, so the filter keeps track of the source address and issues a retry request. A stricter approach could be to reject the sender but I don\u0026rsquo;t think it\u0026rsquo;s the goal of a greylisting to check SPF validity.\nIf, however, the source IP address is valid for the SPF record, then instead of keeping track of the source address, the filter keeps track of the domain and issues a retry request.\nUpon a second connection, when reaching the same point in an SMTP transaction, if the source address is valid for the SPF record and because we track the domain and not the source IP address, any SPF valid IP address will be considered equivalent to pass greylisting. What this means, in simpler words, is that if Gmail comes from one IP address, it is allowed to retry from another, as long as both IP addresses are declared in SPF.\nThis runtime SPF resolution also ensures that there is no need to pre-whitelist IP addresses, instead SPF-enabled domains are whitelisted as a whole, there\u0026rsquo;s no need to run periodic cron or anything to catch up new addresses. This way of working is fully compatible with non-SPF domains which degrade to IP-greylisted.\nIt can\u0026rsquo;t be implemented in a daemon like spamd(8), the decision to greylist or let the session move forward is taken at MAIL FROM, whereas redirection to spamd(8) is decided at connect time through firewall rules.\nDoes this get any better ? # Dunno but I doubt so.\nThere may be some adaptations to slightly improve the idea, but I personally doubt there\u0026rsquo;s much more to do in that area: an SPF-aware greylisting solves the issue that breaks greylisting for Big Mailer Corps, no more, no less.\nKeeping in mind that spamming is a volume industry, more interesting work can be done in raising the cost of sending volumes: just like spammers don\u0026rsquo;t like retrying with uncertain result, they REALLY dislike hosts that are laggy, because it has a HUGE impact on throughput and queue size. In my filter-senderscore, I have already implemented something along these lines by correlating a delay to a sender reputation, achieving stateless tarpitting for low reputation senders.\nIf I\u0026rsquo;m spending more time making OpenSMTPD a hard target to spammers it will be through time-penalty for sure.\n","date":"1 December 2019","permalink":"/posts/2019-12-01/spf-aware-greylisting-and-filter-greylist/","section":"Posts","summary":"TL;DR: - greylisting is a sound idea - yet it is not really practical today - people tend to disable it or find work-arounds - SPF-aware greylisting makes greylisting usable again SMTP failures in a nutshell # SMTP is a fail-safe protocol which attempts very hard to ensure that messages do not get lost once they are in transit.","title":"SPF-aware greylisting and filter-greylist"},{"content":" TL;DR: - our CI was improved - a new OpenSMTPD release, 6.6.1p1 took place to deal with portable issues - multiple portability issues were adressed - new table API in the works - filter-rspamd and filter-senderscore were improved - filter-greylist proof-of-concept published - wrote 2 chapters for my book Got myself a pinebook pro # I got myself an arm64 pinebook pro:\nMy only goal for it was to have a convenient machine on which I could learn some arm64 assembly, and do some portability work on an architecture that\u0026rsquo;s different from amd64.\nI still aim at getting myself a proper laptop in a few months with the sponsorship money, this one is not going to be comfortable enough, but it\u0026rsquo;s really a nice to have.\nI bricked it on the first day, it took me a while to get it back in shape, but several issues I mentionned in this article were fixed on that machine !\nBetter CI # Ihor Antonov did a very neat job at improving CI infrastructure for the project.\nUntil his contributions, we would only have Ubuntu+glibc in our CI so whenever someone reported a build breakage on another system or distro, I would spin a VPS somewhere to install the same system and attempt a build there.\nThis was less than ideal because sometimes by fixing the build on a target, I could break another, and I would spend an afternoon spinning VPS with different systems until everything was back to normal on all.\nHe currently set up Arch, Alpine and Ubuntu, which provides us with a nice combination of glibc, musl libc, OpenSSL and LibreSSL build targets, and helps spot build breakages right away. This made some of the work I\u0026rsquo;ll describe below considerably less painful.\nIt greatly improved our support for other systems.\nOpenSMTPD 6.6.1p1 released and portable development branch work # The 6.6.0p1 version was released in October and, since it became easier to install for several distributions, it also led more users to test it and report issues on their distributions.\nFor most people it worked fine, but on some systems the build was broken and on others the daemon would blow up at runtime. Given where it blew, it was pretty obvious that the issues were really related to the compatibility layer for the most part. Still, it uncovered many interesting topics.\nThey were all fixed and a new version, 6.6.1p1, was released so people would have a working OpenSMTPD on all distributions, then I proceeded to work on improving the compat layer so similar issues don\u0026rsquo;t come back and haunt us in the future.\nBelow are explained the top issues I worked on for 6.6.1p1 and after.\narc4random() # On OpenBSD, arc4random() provides high quality randomness which OpenSMTPD relies heavily upon for pretty much everything. The implementation provides a stream cipher output which is sliced between consumers, with the help of the kernel and userland, so that consumers cannot backtrack or predict the stream. There\u0026rsquo;s an excellent talk by deraadt@ which explains how this works.\nIn the portable release of OpenSMTPD, we attempt to detect if arc4random() is available on the base system and provide a replacement implementation if not. The replacement is a reliable stream cipher, so it is not some low-grade best-effort hack, however we still assume a system arc4random() to be stronger and preferable if only because it could provide system-wide slicing.\nIn addition, OpenSMTPD prefers LibreSSL but can also link against OpenSSL. The former provides an arc4random() implementation if one is not available on the system, the latter doesn\u0026rsquo;t. So depending on the combination of system and TLS library, we may not have arc4random() at all, we may have it from the TLS library, we may have it from the system or we may have it from both. And we always want to use the best version available, assuming LibreSSL\u0026rsquo;s version to be preferable over ours (even if it really is the same).\nThe compat layer was not too good at catching these cases and it was also not too good at ONLY enabling bits of compat layer. In some situations where it would not export our own version of arc4random(), it would still export some internal symbols that led to an assortment of build time and runtime failures.\nThe whole arc4random() detection was reworked and works nicely now.\nIPv6 broken on some systems # During the 6.5 -\u0026gt; 6.6 development cycle, I realized that our way to represent IPv6 address in textual format was wrong and reworked it. This was not a big rework, the diff was fairly small, and I merged it to the portable branch. CI didn\u0026rsquo;t complain and my initial testing didn\u0026rsquo;t show problems because it perfectly dodged the problematic code path (more on that shortly).\nThere are some systems **cough cough** using glibc **cough cough** that provide a broken inet_net_pton() function, supporting only IPv4 and returning EAFNOTSUPPORT when passed an IPv6 address. I first spotted this in 2013 and made a temp_inet_net_pton() function which would be used temporarily as a fallback if inet_net_pton() failed, then renamed it to broken_inet_net_pton_ipv6() in 2016 when it became clear that this would never be fixed in glibc.\nThe OpenBSD code doesn\u0026rsquo;t have this fallback code, so when the merge was applied to the portable branch, it didn\u0026rsquo;t take into account that portable version needed a parameter change for broken_inet_net_pton_ipv6(). On systems with a working inet_net_pton() the code would be bypassed, so this issue was not visible, but on other systems **cough cough** using glibc **cough cough** it would lead to the function failing and broken IPv6 support.\nFix was applied and IPv6 worked again on systmes with a broken inet_net_pton().\nAlpine Linux issues (musl libc really) # There were reports of crashes from an Alpine Linux user.\nI could not reproduce and the more I read the related code, the more it read correct. Futhermore, he reported that the previous version of OpenSMTPD worked flawlessly, and I couldn\u0026rsquo;t see any relevant change that could led to his crash.\nI couldn\u0026rsquo;t reproduce so it took a long while to investigate, but eventually we managed to obtain a backtrace which pinpointed freeaddrinfo() as being the culprit.\nSo what happened ?\nOn one side we have getaddrinfo() which resolves a node into a serie of struct sockaddr * wrapped inside a struct addrinfo * providing the linked list:\nstruct addrinfo { int ai_flags; int ai_family; int ai_socktype; int ai_protocol; socklen_t ai_addrlen; struct sockaddr *ai_addr; char *ai_canonname; struct addrinfo *ai_next; }; and on the other end, we have freeaddrinfo() which releases the items from the linked list returned by getaddrinfo(). Most notably every struct addrinfo * from the list but also the ai_canonname member of each item.\nThe interface for both functions is standardized.\nA common implementation for freeaddrinfo() is the following one:\nvoid freeaddrinfo(struct addrinfo *ai) { struct addrinfo *p; do { p = ai; ai = ai-\u0026gt;ai_next; free(p-\u0026gt;ai_canonname); free(p); } while (ai); } This is straight from the OpenBSD libc but you will find a similar implementation on other BSD systems, on Android, on OSX and on Linux/glibc.\nSo why would it crash on Alpine Linux ?\nHere\u0026rsquo;s the version of it on musl libc:\nvoid freeaddrinfo(struct addrinfo *p) { size_t cnt; for (cnt=1; p-\u0026gt;ai_next; cnt++, p=p-\u0026gt;ai_next); struct aibuf *b = (void *)((char *)p - offsetof(struct aibuf, ai)); b -= b-\u0026gt;slot; LOCK(b-\u0026gt;lock); if (!(b-\u0026gt;ref -= cnt)) free(b); else UNLOCK(b-\u0026gt;lock); } This version assumes an underlying struct aibuf * where the items of struct addrinfo * are stored, not really as a linked list, but rather as contiguous items linked together. This allows musl to do pointer arithmetics to locate elements of the list, it however diverges greatly from other implementations.\nNot saying this at all as a critique to code quality, but I personally have trouble understanding this code, I\u0026rsquo;m unable to understand from reading only if this violates the standard:\nThe freeaddrinfo() function shall support the freeing of arbitrary sublists of an addrinfo list originally returned by getaddrinfo(). Intuitively the code seems to not accept freeaddrinfo() being called on the second element of the addrinfo list, but I may very well be wrong and this needs testing, maybe the next time I spin an alpine VM.\nWhat is certain however is that struct aibuf * is a musl internal thing. The consequence being that if freeaddrinfo() expects to find it, then it can only be called on a struct addrinfo * that was built through getaddrinfo().\nUnluckily for us, OpenSMTPD comes with an asynchronous resolver that crafts its own struct addrinfo *. This results in a linked list that won\u0026rsquo;t have the underlying struct aibuf * and will work everywhere\u0026hellip; but on musl.\nBut then, how comes the previous version of OpenSMTPD worked flawlessly then?\nHere\u0026rsquo;s the version of freeaddrinfo() that musl used until October 2018:\nvoid freeaddrinfo(struct addrinfo *p) { free(p); } It didn\u0026rsquo;t work flawlessly.\nWhat happened was that before October 2018 the musl code didn\u0026rsquo;t have an underlying structure, so it didn\u0026rsquo;t crash when playing pointer arithmetic tricks to locate elements from the list. It however already assumed a unique contiguous chunk of memory and released it as a whole so, when passed the linked list OpenSMTPD crafted, it would partially release the first item of the list and leak members from that item as well as the entire sublist.\nI will contact the musl maintainers to see if they have a plan for this or won\u0026rsquo;t fix, however in the meantime I have introduced a portable_freeaddrinfo() to OpenSMTPD, as a work-around, so that whenever we call getaddrinfo() we release with freeaddrinfo(), but whenever we craft a list ourselves we release with portable_freeaddrinfo(), effectively making OpenSMTPD work again correctly with musl.\nautoconf and openbsd-compat/ cleaned up # Our portable layer was released many many years ago thanks to the work of Charles Longeau, who bootstrapped the OpenSMTPD portable project by gathering bits of the compat layer from OpenSSH and OpenNTPD. This allowed us to make OpenSMTPD available to FreeBSD, NetBSD and Linux after a few weeks or work, broadening our community and helping us get more reports to improve the software.\nNeither of us was an expert in autohell and mistakes were made back then, which we worked around throughout the years, and things worked fairly fine as long as we didn\u0026rsquo;t look too much into details.\nThe compat layer was meant to provide the features we detected as missing on a target system, however it built the entire compat layer and played #ifdef games to exclude sections of code we didn\u0026rsquo;t need on a host. In some cases, the #ifdef didn\u0026rsquo;t properly exclude everything and resulted in compat layer carrying a function that it shouldn\u0026rsquo;t. For functions like strlcpy() this wouldn\u0026rsquo;t be an issue because who cares if we\u0026rsquo;re using OpenSMTPD\u0026rsquo;s version over the systems\u0026rsquo;, but for some others this was an issue.\nThe arc4random() section above described one case of where this could be an issue, because we might pick a less preferable implementation, or it might pick the proper implementation but shadow an underlying symbol with consequences ranging from nothing to a crash.\nAnother case is the OpenSMTPD portable on top of OpenBSD case. It makes sense that building OpenSMTPD portable on top of OpenBSD should result in an empty compat layer since\u0026hellip; well, there should be no missing OpenBSD function in OpenBSD.\nIn practice, the compat layer still carried things and this made me uncomfortable. For years I discouraged using OpenSMTPD portable on top of OpenBSD, claiming it was unsupported, but this was an intuitive claim and while technically there should be no issue, something felt really wrong about it.\nThen OpenBSD\u0026rsquo;s pledge() system call came to the scene and my worry became very real: building OpenSMTPD portable on OpenBSD led to an executable that would crash at startup. The compat layer didn\u0026rsquo;t properly exclude a replacement function, which relied upon a system call that we didn\u0026rsquo;t allow in our pledges. BOOM.\nI proceeded to rework the compat layer and fix a few things:\nFirst of all, I have removed pretty much EVERYTHING that\u0026rsquo;s not needed for OpenSMTPD. Among the things we carried from OpenSSH or OpenNTPD are functions which are never used in OpenSMTPD, and so I have removed their detection in configure as well as their replacement implementations in the compat layer. Anything we have in the compat layer is now needed.\nThen, I have cleaned up a bit the function detection in configure.ac to make sure that functions we need replacements for are properly detected. There is still work to do in terms of handling priorities, as in \u0026ldquo;do I pick the system\u0026rsquo;s or the one from a library\u0026rdquo;, but the logic to handle all detections THEN ULTIMATELY decide if we want a replacement is already there.\nFinally, I have switched from a mode where the compat layer is built with all of its .c files included but sprinkled with #ifdef to exclude portions, to a mode where the compat layer conditionally includes .c files based on configure detection. This has the nice side-effects that we can\u0026rsquo;t accidentally carry something that was not properly #ifdef-ed, but also that you can spot at build time what\u0026rsquo;s included or not on a target host, which can raise eyebrows right away if you see arc4random.c built on a system which provides arc4random() for instance.\nThis is already in place in the portable branch, so this is what you get on an OpenBSD system:\nlaptop$ make gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -I/usr/local/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wno-pointer-sign -fno-strict-aliasing -fno-builtin-memset -c -o empty.o empty.c rm -f libopenbsd-compat.a /usr/bin/ar cru libopenbsd-compat.a empty.o ranlib libopenbsd-compat.a laptop$ and on a Linux system (Archlinux, glibc, LibreSSL):\n$ make gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o empty.o empty.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o closefrom.o closefrom.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o crypt_checkpass.o crypt_checkpass.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o errc.o errc.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o event_asr_run.o event_asr_run.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o fgetln.o fgetln.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o fmt_scaled.o fmt_scaled.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o fparseln.o fparseln.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o freezero.o freezero.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o getpeereid.o getpeereid.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o imsg.o imsg.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o imsg-buffer.o imsg-buffer.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o pidfile.o pidfile.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o recallocarray.o recallocarray.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o setproctitle.o setproctitle.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o strlcat.o strlcat.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o strlcpy.o strlcpy.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o strmode.o strmode.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o strtonum.o strtonum.c gcc -DHAVE_CONFIG_H -I. -I.. -I../smtpd -I../openbsd-compat -I../openbsd-compat/err_h -I/usr/include -g -O2 -fPIC -DPIC -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -fno-builtin-memset -I/usr/include/libressl -L/usr/lib/libressl -Wl,-rpath=/usr/lib/libressl -D_BSD_SOURCE -D_DEFAULT_SOURCE -c -o vis.o vis.c rm -f libopenbsd.a /usr/sbin/ar cru libopenbsd.a empty.o closefrom.o crypt_checkpass.o errc.o event_asr_run.o fgetln.o fmt_scaled.o fparseln.o freezero.o getpeereid.o imsg.o imsg-buffer.o pidfile.o recallocarray.o setproctitle.o strlcat.o strlcpy.o strmode.o strtonum.o vis.o /usr/sbin/ar: `u\u0026#39; modifier ignored since `D\u0026#39; is the default (see `U\u0026#39;) ranlib libopenbsd.a $ There is still a lot of work to do in the compat layer, the library was part of it but the handling of missing headers and such is also not too good. I have plans to keep working on this for the next few months until I\u0026rsquo;m comfident that running the portable branch on OpenBSD doesn\u0026rsquo;t have side-effects.\ntable API rework # Something I mentioned last month was that I wanted to turn the table API into something similar to filters, so people could easily write table backends in any language and rely on existing libraries.\nI have made very good progress on this front and I wrote a couple table backends using the new interface, however I\u0026rsquo;m not done yet and I will keep the details for another article.\nfilter-rspamd improved # My filter-rspamd got additional contributions from Reio Remma (@whataboutpereira), to extend the headers and have them include the scoring for the symbols they matched.\nThis results from a discussion and a previous pull request that I had rejected as I felt the header was way too verbose.\nThe 0.1.4 release of filter-rspamd would display headers as:\nX-Spam: yes X-Spam-Score: 8.789999 / 15 X-Spam-Symbols: MIME_TRACE, FROM_NEQ_ENVFROM, ARC_NA, RCPT_COUNT_ONE, URI_COUNT_ODD, DMARC_POLICY_ALLOW, R_DKIM_ALLOW, HAS_X_PRIO_THREE, MIME_GOOD, HAS_REPLYTO, R_SPF_ALLOW, FROM_HAS_DN, TO_MATCH_ENVRCPT_ALL, RCVD_TLS_ALL, FORGED_SENDER, MID_RHS_MATCH_FROM, HTML_SHORT_LINK_IMG_1, RCVD_IN_DNSWL_NONE, DKIM_TRACE, RSPAMD_URIBL, PREVIOUSLY_DELIVERED, HAS_LIST_UNSUB, BAD_REP_POLICIES, SUBJECT_ENDS_QUESTION, ASN, REPLYTO_DOM_NEQ_FROM_DOM, RCVD_COUNT_TWO, TO_DN_NONE In development, it now replaces the X-Spam-Symbols with X-Spam-Status and outputs as follows:\nX-Spam-Status: Yes, score=7.267 required=15.000 tests=[ARC_NA=0.000, BAYES_SPAM=5.100, DCC_BULK=0.667 FROM_EQ_ENVFROM=0.000, FROM_HAS_DN=0.000, MID_RHS_MATCH_FROM=0.000 MIME_BAD_ATTACHMENT=1.600, MIME_GOOD=-0.100, MIME_TRACE=0.000 PREVIOUSLY_DELIVERED=0.000, RCPT_COUNT_ONE=0.000 RCVD_COUNT_THREE=0.000, RCVD_TLS_LAST=0.000, RCVD_VIA_SMTP_AUTH=0.000 TO_DN_NONE=0.000, TO_MATCH_ENVRCPT_ALL=0.000] This is not only more compact but also avoids having to go look into /var/log/rspamd/rspamd.log for details about what scored how much.\nfilter-senderscore improved # In July, I wrote filter-senderscore which looks up the reputation for a source address in the SenderScore database, and allows blocking, junking and delaying sessions whose score falls below some threshold.\nThe filter works fairly well so I didn\u0026rsquo;t bother making any improvement to it.\nIn September, the junk action was introduced which allows a filter to tell that a session or transaction can be junked by OpenSMTPD. Since the filter was written before that feature, it did the junking itself by registering for the dataline phase and injecting a X-Spam: yes header, so I simplified by having the filter use the junk action instead.\nWhile I was there, I thought it would be nice to add an option to let user configure at which phase the client should be disconnected. Until now, if the -blockBelow option was provided and the reputation for an IP fell below the threshold, the session would be disconnected at the banner. The new -blockPhase option allows deferring the disconnect to later, making it possible to track the sender or recipient addresses before disconnecting if needed.\nfilter-greylist proof-of-concept # The SMTP protocol provides a mechanism for \u0026ldquo;temporary failures\u0026rdquo; which requests a sender to retry transfer of a message. A regular MTA like OpenSMTPD will honor that request but scripts and spam tools that are targeted at spamming en masse will not necessarily retry, as it would slow them and force them to keep track of which hosts need to be retried.\nThe idea behind greylisting is to trigger a temporary failure voluntarily when first contacted by an MX, then keep a reasonnable window of time within which the MX should perform a retry. If the MX retries in that timeframe then it is whitelisted, otherwise it is either blacklisted or kept in temporary-failure-land forever.\nMultiple greylisting solutions exist, including the spamd(8) (man page) daemon in OpenBSD. A lot of them are unusable as is with big mailers like Gmail / Yahoo / Hotmail because their retries do not come from the same MX host and they never get whitelisted.\nA few years ago, I introduced an spf walk utility which made things a bit better.\nWhat it did was to perform an SPF walk to catch as many IP addresses as it could from an SPF record to they could be whitelisted. Since all big mailers publish SPF records, it improved considerably the situation.\nIt wasn\u0026rsquo;t perfect either because some SPF records are context-dependant and can\u0026rsquo;t be looked up, others rely on hostnames which have geo-localized IP addresses preventing whitelists from being shared across countries, and finally whitelisting this way didn\u0026rsquo;t account for updates of SPF records\u0026hellip; so spf walk had to be ran frequently and there was always a possibility that an SPF change happened right after a run. A lot of people see spf walk as THE solution to the big mailer greylisting issue, but I don\u0026rsquo;t.\nI wrote filter-greylist as a proof-of-concept for an SPF-aware greylisting.\nWhat it does is consider that there are two kind of senders, those that do SPF and those that don\u0026rsquo;t. If a host doesn\u0026rsquo;t do SPF, then retries are expected to come from the same IP address. If a host does SPF, then the sending domain is remembered and ALL IP addresses part of the SPF record are considered as valid for the retry. This means that if Gmail contacts from an IP, then retries from another, this will be considered as a valid retry.\nI have been running with this filter for weeks and it works perfect, to the point where I had actually forgotten that it was running.\nThe filter has been published on Github but since this is a bit more risky than my other filters, test at your own risks.\n2 chapters of the book written # I have written two additional chapters to my OpenSMTPD book.\nI\u0026rsquo;m now focusing on the configuration examples which is probably the most interesting part for readers, but by far the most annoying part to write for me given how many features we support :-)\nWhat next ? # It\u0026rsquo;s unclear what I\u0026rsquo;ll do in December because I have multiple works in progress, the table API will be worked on for sure, the smtp-out reporting also.\nI\u0026rsquo;d like to improve my filter-greylist proof of concept but it\u0026rsquo;s not high priority for me, I\u0026rsquo;ll be happy to review pull requests ;-)\nI have started writing articles outside of my monthly reports, however they grow really big really fast, I need to find how to properly split.\n","date":"17 November 2019","permalink":"/posts/2019-11-17/november-2019-report-opensmtpd-6.6.1p1-filter-greylist-and-tons-of-portable-cleanup/","section":"Posts","summary":"TL;DR: - our CI was improved - a new OpenSMTPD release, 6.6.1p1 took place to deal with portable issues - multiple portability issues were adressed - new table API in the works - filter-rspamd and filter-senderscore were improved - filter-greylist proof-of-concept published - wrote 2 chapters for my book Got myself a pinebook pro # I got myself an arm64 pinebook pro:","title":"November 2019 report: OpenSMTPD 6.6.1p1, filter-greylist and tons of portable cleanup"},{"content":" TL;DR: - yay, surprise emergency hand surgery... - OpenSMTPD 6.6.0 was tagged and released, including portable version - Merged contributions to fix filter-rspamd bug with DKIM - Work resumed on 6.7.0 feature - An OpenSMTPD book is in the works How I landed in the emergencies # I accidentally cut my hand real bad early October and rushed to the nearest hospital. They had a look, told me it was superficial and required to be sutured, so I was out a few hours later with a few stitches and pain killers.\nA few days later, my hand was swell so I rushed to another hospital that was specialized in hand wounds, expecting a second advice and maybe some additional drugs.\nTurns out the cut was not superficial at all, it had hit joints and needed more than just a few stitches. The stitches also lacked propre protection so the wound had infected. I was told that I had to undergo immediate surgery because the infection was looking real bad, and that I was lucky I took this seriously and came right away.\nSo this is how my month of October was ruined. It took me two weeks to be able to type back on a keyboard, it is still painful and I\u0026rsquo;m still unable to practice music or martial arts as I can\u0026rsquo;t close my fingers entirely.\nLuckily, I should be able to recover fully, but\u0026hellip; it will take some time.\nMy work this month was obviously impacted, BUT I managed to pull the OpenSMTPD 6.6.0 release and do some thinking work around redesigns planned for development cycle that just started. I will discuss a bit my plan in this article.\nOpenSMTPD 6.6.0 was released # I have published and announced the OpenSMTPD 6.6.0 release today.\nIt is a very important release for us, probably one of the most important that happened in the last few years, as it brings the filter API and makes it possible to compete with other MTA by interfacing with a wide variety of third party tools.\nIt is now possible to do dnsbl, interface with antispam solutions such as Rspamd or SpamAssassin, perform DKIM signing and verifying at session time, rejecting sessions that do not behave as we expect, \u0026hellip;\nThe API was released as an undocumented experimental feature. This doesn\u0026rsquo;t mean it is unstable, the code has been used on the OpenSMTPD mail exchanger for over a year, used by developers for at least as long, and tested by users in the wild for several months. The only reason it is considered experimental is that we didn\u0026rsquo;t want to \u0026ldquo;release\u0026rdquo; a stable version of the API with easy access to documentation, without having a chance to obtain a first round of feedback over the course of a release with a limited set of filters and developers ;-)\nIn addition to filters, this is the first release that is considering LibreSSL as its target TLS library while providing support for OpenSSL. A lot of distributions remained with ancient OpenSMTPD release because of TLS library issues. This release is going to make every distro and system out there able to catch up and use the latest OpenSMTPD release.\nDKIM signing and verifying was fixed in filter-rspamd # Thanks to Reio Remma @whataboutpereira, a bug was fixed in filter-rspamd which would cause some mails to be incorrectly DKIM signed or verified.\nThe issue was that filters receive the raw SMTP lines during the DATA phase, and the filter didn\u0026rsquo;t account for the escaping of lines starting with a dot, leading to some mails being incorrectly signed or verified.\nHis fix was committed to github and prompted me to issue a new release, before updating the OpenBSD port as well.\nI have received an ok to commit the fix to stable packages and will do so this week-end.\nOpenSMTPD 6.7.x work # With my hand out of service, I spent time thinking about bits that needed a redesign.\nThe smtp-out reporting was almost completed and didn\u0026rsquo;t make it to this release, so I will make sure it is completed very soon so it doesn\u0026rsquo;t miss the next release.\nThe switch to libtls requires a standalone OpenSSL-based libtls interface to be written, I have started taking a look at it too. I already have a libtls-based OpenSMTPD in a branch, but I don\u0026rsquo;t want to switch to that until we know for sure that we\u0026rsquo;ll be able to provide a portable release, so hopefully for 6.7.x in six months and more realistically for 6.8.x in a year.\nThe MTA layer needs to be reworked and was discussed at lengths with Eric Faurot, this would be a change transparent to users but which would make the daemon MUCH simpler for developers. I hope we can pull that for 6.7.x but work has to start very soon to make this happen, so we\u0026rsquo;ll see in the next few weeks how this project evolves.\nFinally, I\u0026rsquo;d like to replace the current table API with one that\u0026rsquo;s similar to the filter API. The current API is a binary protocol relying on the imsg framework, requiring C plumbing and making the writing of table backends limited to skilled developers. I already discussed with Eric Faurot my feeling that it should be converted to a line-based protocol, similar to filters, which would allow backends to be written in any language, providing the same benefits as filters in terms of privilege separation and memory isolation. This would make the OpenSMTPD-extras repository deprecated, but given that I\u0026rsquo;m the author for pretty much every addon there, I think I can easily replace them with new interface counterparts.\nAn OpenSMTPD book is in the works # I had started writing a book about OpenSMTPD years ago but, because huge reworks were in progress in both configuration and filter areas, I kept delaying as I didn\u0026rsquo;t want to write configuration bits that would have to be rewritten from scratch, and I didn\u0026rsquo;t want the book to lack a chapter about spam filtering.\nWith all of the work that has gone through OpenSMTPD in the last two years, I feel that there\u0026rsquo;s no longer a gap to fill and a very complete and useful book can be written. I have therefore resumed work and currently have a PDF that\u0026rsquo;s well over a hundred pages when fully built, describing the history and design of OpenSMTPD, large chunks of how SMTP and DNS work, how to setup your MX, how to configure various setups, and much more.\nNeedless to say, this is a LOT of work.\nIt will take months to complete the book and publish something I\u0026rsquo;ll be happy with, so if you\u0026rsquo;re interested let me know and if you want to support me and help me get this out the sooner, then you know how to show your love and buy me time :-)\nWhat next ? # I have started writing a new article related to SMTP and reputation, I will try to complete it next week and provide early access to my patrons.\nDevelopment cycle for OpenSMTPD as resumed, I have a few tickets to tackle on Github and a few ideas that I have to dig, but the table API rework is a very high priority thing to do for me.\nI will also resume working on cleaning up the portability layer.\n","date":"26 October 2019","permalink":"/posts/2019-10-26/october-2019-report-opensmtpd-6.6.0-release-mostly/","section":"Posts","summary":"TL;DR: - yay, surprise emergency hand surgery... - OpenSMTPD 6.6.0 was tagged and released, including portable version - Merged contributions to fix filter-rspamd bug with DKIM - Work resumed on 6.","title":"October 2019 report: OpenSMTPD 6.6.0 release mostly"},{"content":" TL;DR: - I started writing this post a week ago but got interrupted by a baby, Jules - Spent MANY hours on writing OpenSMTPD-related articles - Enabled continuous integration in the OpenSMTPD portable repository - Managed to get rid of all the blocking issues for OpenSMTPD 6.6.0 release - Added some features and fixed a crash in filter-rspamd Welcome Jules, fork() completed # In my last report from August, I concluded not knowing if this report would be published on time.\nThe expected date for bringing our little monkey home was the 2nd of October, but while I started writing this article the 21st of September, he decided not to stick to the plan.\nThis is why I\u0026rsquo;m publishing today an article dated from the 21st, which is when I started writing it ;-)\nSo my biggest completed project for this September report was the little Jules, born Sunday the 22nd of September 2019 at 08:03 AM in the city of Nantes.\nHopefully a future OpenBSD hacker:\nThough he can be anything, including a unicorn if he wants. He cute af. The cutest.\nWrote two articles for the community # The first one, You should not run your mail server because mail is hard was intended to be an MTA-agnostic article, debunking some of the common claims that mail is hard.\nThe second one, Setting up a mail server with OpenSMTPD, Dovecot and Rspamd was intended both as a follow up to the previous article, to back the claims with a technical setup, but also as a mean to promote OpenSMTPD AND answer the \u0026ldquo;there\u0026rsquo;s not many OpenSMTPD tutorials out there\u0026rdquo; request.\nThe first article took me a couple hours to write, whereas the second one took me about ten hours because I wanted to explain everything I did and make it a reference tutorial for OpenSMTPD users.\nDeployed CI on our Github mirror repository # OpenSMTPD is developed on OpenBSD, so we\u0026rsquo;re guaranteed that it builds because we benefit from the HYAYFBTB1 continuous integration (CI) mechanism.\nThe portable branch is different. It is mainly maintained by myself and\u0026hellip; I only run OpenBSD and OSX daily. Most of the changes are merges from the OpenBSD branch which aren\u0026rsquo;t challenging to portability, but occasionally I need to rely on community feedback or spin a server for a specific distribution.\nSometimes, I mess up and the trivial merge from OpenBSD lacks an include or uses a function that I thought was portable, until someone reports the build is broken.\nGithub enabled CI on our account so I immediately set it up on the OpenSMTPD repository. It will still need improvements but at this point, every time we commit to the portable branch, a bootstrap, configure, make and make install are done, and members of the team all get an e-mail if the build is broken.\nWhile at it, I also added CI to the filter-rspamd and filter-senderscore repositories.\n1Hackers Yell At You For Breaking The Build\nIntroduced junk action for builtin and proc filters # Until now, builtin and proc filters had only 4 possible actions: proceed to let other filters handle the session, rewrite to man-in-the-middle rewrite parameters to the filter hook, reject to reject the phase with a custom SMTP message, and disconnect to reject AND disconnect the phase with a custom SMTP message.\nMost filters used reject or disconnect as a mean to kill spammers upfront, with builtin filters such as:\nfilter check_rdns phase connect match !rdns disconnect \u0026#34;550 GO AWAY\u0026#34; But I received a lot of feedback from people scared of rejecting legitimate mails by being too harsh, and asking if there was a way to junk the messages rather than throw them away.\nI introduced the junk action which flags a session or transaction so messages can proceed, but will get an X-Spam: yes header prepended. The following builtin filter would match the same session as above but, instead of killing the session, all messages from the session would be marked as spam:\nfilter check_rdns phase connect match !rdns junk With this new action, people can actually build confidence for a while before going stricter.\nAs a bonus, this makes it easier to write proc-filters that junk messages. Before this change, filters would have to keep state of junked sessions and register a callback for data-line phase, so they could craft the header and prepend it. With this change, they can simply junk the message right away and let OpenSMTPD prepend the header.\nImplemented Sender Rewriting Scheme support # OpenSMTPD lacked Sender Rewriting Scheme (SRS) support.\nLong story short, the Sender Policy Framework (SPF) was introduced to let a domain control which hosts can send mail on its behalf. This is very nice and all, but it doesn\u0026rsquo;t play well with mailing lists and forwarding hosts.\nFor example, if eric@faurot.net sends mail to misc@opensmtpd.org, the opensmtpd.org domain needs to send mail to all subscribers on behalf of faurot.net, but\u0026hellip; it is not allowed to do so, resulting in an SPF check failure.\nTo work around this, mailing list software encode the sender address in the protocol so it originates from the mailing list domain, and allows them to decode back upon replies:\nbb957901382a5d3f mta delivery evpid=25d3675b29721118 from=\u0026lt;misc+bounces-probe-eric=faurot.net@opensmtpd.org\u0026gt; to=\u0026lt;gilles@poolp.org\u0026gt; rcpt=\u0026lt;-\u0026gt; [...] The misc+bounces-probe-eric=faurot.net@opensmtpd.org address resolves to misc@opensmtpd.org once you remove the + subaddressing and the mailing list software handling that address uses the subaddressing value bounces-probe-eric=faurot.net to retrieve the original sender.\nSo far so good, but this only works for mailing lists. The openbsd.org domain is essentially a mail forwarder: sending mail to eric@openbsd.org really forwards to eric@faurot.net and sending mail to gilles@openbsd.orgreally forwards to gilles@poolp.org. In this case, no mailing list software encodes and decodes addresses, and this creates problems.\nIf eric@faurot.net sends mail to gilles@openbsd.org, the openbsd.org mail exchangers accepts the mail and forwards it to poolp.org. But the mail exchangers at poolp.org sees a mail originating from faurot.net coming from a mail exchanger that\u0026rsquo;s not the correct one\u0026hellip; the SPF check fails again.\nSRS comes into play and, once enabled, makes sure that any address forwarded is encoded in such a way that it passes SPF correctly and that replies can be traced back to the original sender. I won\u0026rsquo;t detail this much, there\u0026rsquo;s a specificiation that can be easily found on a search engine, but it\u0026rsquo;s now supported natively in OpenSMTPD.\nIf your mail exchanger is intended to forward mails on behalf of other domains, you can enable SRS using the following configuration:\nsrs key secret_key_to_prevent_people_from_impersonating_me [...] action \u0026#34;outgoing_mails\u0026#34; relay srs Resulting in:\nPlain and simple.\nLots of minor stuff here and there # Tons of minor stuff were committed here and there, ranging from new reporting events for developers, cleanups, stricter checks, documentation.\nI won\u0026rsquo;t list everything we did because you can check the history.\nOpenSMTPD 6.6.0 # OpenSMTPD 6.6.0 is now \u0026ldquo;feature-ready\u0026rdquo; and the only things we\u0026rsquo;ll commit until the release are bug fixes if we find any. The upcoming weeks will be stabilization only, though\u0026hellip; the last weeks were also stabilization mainly ;-)\nIf you hit issues with the portable branch, don\u0026rsquo;t worry, we tag the release when the OpenBSD version is stable and keep working on improving the portable version after, giving it its own tag. If you experience issues with portable at the time we tag 6.6.0, it doesn\u0026rsquo;t mean the issues will not be fixed before the portable version is released.\nVarious improvements to filter-rspamd # With the help of Reio Remma @whataboutpereira, filter-rspamd got a lot of new features.\nFirst of all, I fixed a concurrency-related crash experienced by two OpenBSD hackers. That was quite a mandatory feature. not crashing.\nThen, I added an X-Spam-Symbols header as we needed it to provide the same level of informations as the rspamc client, which is the alternate way of integrating Rspamd with OpenSMTPD.\nReio did a bit of cleanup, taught the filter how to notify Rspamd about the MX name, how to reuse Rspamd-provided SMTP messages in the reply to SMTP sessions, how to add / remove headers based on the Rspamd configuration. He basically made it closer to the Rspamd milter that\u0026rsquo;s available to other MTA.\nWhat next ? # I haven\u0026rsquo;t slept much this week so my mind is a bit blurry about what\u0026rsquo;s on my roadmap, but my goal is now to focus on the portable branch to improve it.\nWhen my next report is due, OpenSMTPD 6.6.0 may or may not have been released based on OpenBSD\u0026rsquo;s release plan.\nDepending on what happens, either I\u0026rsquo;ll spend the month working on the release or I\u0026rsquo;ll start working on the next big works, namely reworking the table layer to work similarly to filters, allowing tables to be written in any language.\nI may also be focusing on diapers.\n","date":"21 September 2019","permalink":"/posts/2019-09-21/september-2019-report-jules-opensmtpd-6.6.0-upcoming-release-and-related-things/","section":"Posts","summary":"TL;DR: - I started writing this post a week ago but got interrupted by a baby, Jules - Spent MANY hours on writing OpenSMTPD-related articles - Enabled continuous integration in the OpenSMTPD portable repository - Managed to get rid of all the blocking issues for OpenSMTPD 6.","title":"September 2019 report: Jules, OpenSMTPD 6.6.0 upcoming release and related things"},{"content":" TL;DR: - NO TL;DR: this time, I spent hours writing, you should spend minutes reading. - OK... I explain in WAY TOO MUCH details how to setup a mail server EDIT (2019-10-26) # OpenSMTPD 6.6.0 was released today, the article was updated to reflect that it applies to the current release and no longer a development version.\nEDIT (2019-12-15) # I have refactored the article to remove the political aspect behind self-hosting, which is now a standalone article, as well as a blurb on reputation since this will become part of a serie of articles.\nWow, that was unexpected # In a previous article, I explained why I think the \u0026ldquo;mail is hard\u0026rdquo; myth is unfounded and why I think people shouldn\u0026rsquo;t be discouraged from running their own mail services. I didn\u0026rsquo;t know this article would gather so much attention, reaching over 75k reads in three days (currently past 85k) when my next most read article reached 15k reads\u0026hellip; in over a year.\nI wish I had a soundcloud. I have a patreon though. I also wish I had entered more into details and not covered things so superficially.\nThis article is DENSE but this is because I will hold hands at an absurd level, removing all my blabber will only leave very few technical parts.\nThis is for techies and sysadmins # Self-hosting mail requires a minimum of knowledge and dedication, it is not a two-clicks thing, you need to have some basic knowledge and you have to be willing to pour some time into it. If you\u0026rsquo;ve never touched a DNS zone or if you think that taking an hour to setup something is hard, then this will not be easy and I don\u0026rsquo;t recommend you do it unless you WANT to learn a ton of stuff.\nIf you\u0026rsquo;re a sysadmin who is familiar with sysadmin work, then most of the sysadmin work you already do is considerably harder than setting up a mail instrastructure. I\u0026rsquo;ll reiterate it for you: mail is not hard.\nEHLO hypno.cat # For this article, we will setup a mail server for hypno.cat, a small website for my thriving (hypothetical) hypnotherapist activity. I registered the website years ago because I liked the name, but never done anything with it beyond hosting an awesome animated file.\nGoogle, Bing and Yahoo have somehow known about it for years and indexed that front page, so they won\u0026rsquo;t consider it as a domain that was just bought by a spammer to start sending mail, however it is virtually unknown to anyone on the internet because it has no content, doesn\u0026rsquo;t link to anywhere, isn\u0026rsquo;t linked from anywhere, and has never sent or received mails from or to anyone. It is an important detail because it shows how the age of a domain has a huge impact on your reputation.\nI\u0026rsquo;m going to set it up for real as I write the article, from the moment I have booted a brand new VPS to the moment I have received mail back and forth to my Gmail account. I have chosen that one because it is over-represented in the population and this explains why people complained the most about delivery issues there. I personally think that Outlook is worse in terms of interoperability, but lets keep that for a future article maybe.\nThe mail services for hypno.cat are going to provide TLS-secured incoming and outgoing e-mail. They are also going to allow me to submit messages from my smartphone\u0026rsquo;s IMAP client, the Gmail application, so my users don\u0026rsquo;t feel much of a change in habits. I could just as easily provide a webmail, with Rainloop or Roundcube, or setup Thunderbird or mutt. Any IMAP client will work just fine so I will not cover this part, there are many alternatives and plenty of tutorials on how to set them up already. There\u0026rsquo;s even techniques to help mail clients auto-discover their configuration, you can figure that out by yourselves.\nMy outgoing mail will pass the basic checks at Gmail, namely SPF, DKIM and DMARC, which are more than enough to get your mail through. We could do a lot more, but the goal here is to find the proper balance between doing enough work to look good and remaining simple.\nIncoming mail will be filtered to reduce the volume of spam, either by killing obvious bad senders at session time when they are detected, or by providing classification so Spam is in a dedicated folder to avoid cluttering the Inbox. I could stop here, since that would already be a fairly nice setup, but I\u0026rsquo;ll also throw in a bit of configuration to teach Dovecot how to train spam detection through moving mail from the Inbox to Spam and the other way around. That part is slighly more complex because it relies on Sieve which is not the best piece of engineering, feel free to discard if you don\u0026rsquo;t care, I did without it for over ten years. It\u0026rsquo;s a nice-to-have, not a must-have.\nRequirements # The very first requirement is to figure out where you are going to run your mail server.\nWith widepsread permanent connections through DSL or FTTH, a home connection may be tempting but it is not a good idea as the IP address spaces of ISP are often blacklisted or suffer from a bad reputation to start with. In addition, many ISP prevent outgoing SMTP traffic to avoid compromised desktops from becoming spamming bots. I find the best option to be renting a dedicated server or a VPS from a hosting company after making sure that SMTP traffic is allowed there.\nI have rented my dedicated servers at online.net for the last twelve years and am very happy with them. You\u0026rsquo;ll even find on this blog instructions to run OpenBSD on their servers as they don\u0026rsquo;t support it natively. They do not filter SMTP, which is good because you can run an SMTP service right away, however the IP addresses may have been previously used by bad senders and you will want to test them. If the IP that is automatically allocated to you isn\u0026rsquo;t good, you can still order an additional one and pick it in a different range after checking that it isn\u0026rsquo;t in a blacklist, or that it doesn\u0026rsquo;t have a bad reputation already.\nAlternatively, for the purpose of creating my own commercial mail services, I started building an infrastructure on vultr.com (that\u0026rsquo;s a referal link). I haven\u0026rsquo;t been there for long so I might still change my mind, but so far I\u0026rsquo;m very happy with it. They filter SMTP by default so you have to open a ticket to explain what you intend to do with mail. When I explained that I didn\u0026rsquo;t intend to become an ESP but rather provide hosting services, they were helpful and unfiltered the same day.\nIt does not matter much where you decide to run your mail server, but what you need to check is that:\nthe host doesn\u0026rsquo;t have a history of hosting spammers and allows you to do SMTP you have an IP address that is dedicated to your mail you have control of the reverse DNS for that IP address your IP address isn\u0026rsquo;t already in blacklists As long as these requirements are met, you should be fine.\nIt is a good plan to prepare for incidents and this is done by using a backup mail server to take over when the primary one is down. I won\u0026rsquo;t cover this as it\u0026rsquo;s not complex, just a matter of an additional DNS record and an additional mail server routing traffic to the primary one, you\u0026rsquo;ll figure this out by yourselves. HOWEVER, a lot of people mentioned that you can get blacklisted and lose mail, so I think now is the right time to recommend that you don\u0026rsquo;t host your different mail servers at the same place. You don\u0026rsquo;t want the two of them to go down if there\u0026rsquo;s a power or network shortage in your datacenter, and you also want to ensure that if your IP range gets blacklisted as collateral damage from an evil neighbor, the IP of your backup mail server is \u0026ldquo;far enough\u0026rdquo; to not be blacklisted too. This way, should your primary mail server be temporarily blocked from sending, it can reroute traffic through the backup mail server until the issue is solved.\nI have endured collateral damage on my own setup only once during the last ten years. It was annoying for sure because incidents are never fun, but I rerouted traffic to my secondary server to unbreak traffic, filled the form to delist my IP while providing proof that I was a legitimate sender, rerouted traffic back when issue was resolved. It was nowhere near the big deal people make out of it.\nNo stress for people who plan ahead.\nThe technical stack # I will spin a new VPS at vultr.com (that\u0026rsquo;s still a referal link), install the latest OpenBSD snapshot because that\u0026rsquo;s how I roll (this will not be covered in this article), and build my mail system on top of it. I will assume OpenBSD throughout the rest of my writing but besides system-specific commands to install packages the configuration should be similar from a system to another. If you\u0026rsquo;re tall enough to run a mail server, you should be tall enough to adapt pathnames accordingly or \u0026hellip; just use OpenBSD already !\nFor the SMTP layer, which is in charge of exchanging messages between hosts disregarding what users will use to access their mail, I will be using the OpenSMTPD software. It is the default SMTP daemon for the OpenBSD operating system and has a portable version which is available on Github with instructions on how to build. These instructions are for the 6.6.0 version or later, they will not work on earlier versions.\nFor the IMAP layer, which will allow users to retrieve messages they received and access them from their smartphones or webmails, I will use the latest version of Dovecot. I will also use the Dovecot-Pigeonhole package, which will allow us to train the antispam solution into learning Ham from Spam. If you intend to only use a console client over ssh, such as mutt for example, you can skip that part as OpenSMTPD can deliver in a local mailbox that mails clients can access directly. In this article, we setup IMAP because we acually want to read mails from a smartphone as that\u0026rsquo;s what the regular people do.\nFinally, for the spam filtering layer, I will use the excellent Rspamd daemon which is far more than just an antispam solution. It provides state of the art spam filtering methods, but also provides antivirus integration, dkim signing, and a ton of modules which you can chose to use or not to fine tune your setup. In this article, we\u0026rsquo;ll simply use it for its DKIM-signing and spam filtering features which is the bare minimum we need.\nMaking myself reachable # SMTP is very tighly coupled with DNS and other hosts rely on DNS lookups to find out which machine handle mail for your domain. This is done through the lookup of MX (Mail eXchanger) records, so the most minimal thing to do for SMTP to work is to declare an MX record for your domain. For hypno.cat, the zone contains the following:\n;; an A (and AAAA record for IPv6) record is declared to name your mail server mail.hypno.cat A 217.69.8.253 mail.hypno.cat AAAA 2001:19f0:6801:867:5400:01ff:fee7:7af7 ;; an MX record is declared to let the world know that mail.hypno.cat handles mail for hypno.cat ;; 0 is the highest preference, if we had a backup MX we\u0026#39;d set a higher number on it, ;; let\u0026#39;s keep that for later, shall we ? hypno.cat. MX 0 mail.hypno.cat. ;;hypno.cat. MX 10 mail-backup.hypno.cat. I can verify that everything is fine using the host and dig utilities to check that the mail server name resolves and that the MX lookup returns the mail server name:\n$ host mail.hypno.cat mail.hypno.cat has address 217.69.8.253 mail.hypno.cat has IPv6 address 2001:19f0:6801:867:5400:1ff:fee7:7af7 $ dig -t MX hypno.cat +short 0 mail.hypno.cat. At this point and with these DNS records, other mail servers can already lookup which MX is responsible for hypno.cat, and contact it when they want to send mail to any address for that domain.\nMaking myself look all fine and dandy # That\u0026rsquo;s not enough because nowadays you MUST have a reverse DNS (rDNS) AND Forward-Confirmed rDNS (FCrDNS).\nThere\u0026rsquo;s a reason for that. A large number of the hosts spamming the world are compromised machines, with a large share of them being home computers behind residential connections. Many of these residential connections don\u0026rsquo;t have rDNS, don\u0026rsquo;t have FCrDNS, or have an rDNS that matches a dynamically allocated IP pattern (ie: 123.123.123.123.dyn.adsl.example.com). Because they are individually compromised machines, and not regular servers under the spammers control, rDNS and FCrDNS can\u0026rsquo;t be configured to look nice\u0026hellip; so it became a proof-of-work to configure your rDNS and FCrDNS correctly while ensuring that it doesn\u0026rsquo;t look like a dynamically allocated IP pattern. In some of the Big Mailer Corps guidelines this is explicitely stated, for others you just discover that the hard way. Either way, it\u0026rsquo;s a BARE MINIMUM, make sure that your hostname looks like a REAL mail server (ie: mail.hypno.cat, not www.hypno.cat).\nrDNS is usually out of your control because it\u0026rsquo;s managed in a somewhat special \u0026ldquo;arpa\u0026rdquo; zone owned by the IP address owner. ISP usually don\u0026rsquo;t let you configure it but server providers can\u0026rsquo;t realistically expect their customers not to, so they generally provide a small form somewhere in their control panels to let you provide the rDNS you want for the IP address they allocated you:\nIf I setup my rDNS to be the same as the forward records I configured above, then I automatically pass the FCrDNS test. This can be easily verified by looking up the rDNS for an IP address:\n$ host 217.69.8.253 253.8.69.217.in-addr.arpa domain name pointer mail.hypno.cat. $ host 2001:19f0:6801:867:5400:1ff:fee7:7af7 7.f.a.7.7.e.e.f.f.f.1.0.0.0.4.5.7.6.8.0.1.0.8.6.0.f.9.1.1.0.0.2.ip6.arpa domain name pointer mail.hypno.cat. Then looking up the IP address for that rDNS:\n$ host mail.hypno.cat mail.hypno.cat has address 217.69.8.253 mail.hypno.cat has IPv6 address 2001:19f0:6801:867:5400:1ff:fee7:7af7 If they match both ways, then everything\u0026rsquo;s fine.\nAdvertise which machines are allowed to send mail on behalf of my domain # SPF is a mechanism which makes it possible for destination mail servers to determine if a machine was allowed to send mail on behalf of a domain.\nHow it works is very simple, basically you add a DNS record to the zone for your domain stating which servers will emit mails. When a destination mail server receives mail with a sender from your domain, it checks if the IP address of the server submitting the e-mail is part of the whitelist.\nSPF is not mandatory but it is one of the things that makes you look good. While it is only informative by default, not necessarily resulting in mails being junked if they fail the test, it does have an impact on computing the likeliness of a message being a spam when added to other factors. It is commonly accepted that lack of SPF has a very negative impact on the reputation of a domain, so given how simple it is to provide the record, there\u0026rsquo;s simply no excuse.\nThere are multiple ways to setup SPF, in this example I will simply set it up so ONLY my mail server can send mail on my behalf:\nhypno.cat. IN TXT \u0026#34;v=spf1 mx -all\u0026#34; Should hosts receive a mail with a sender @hypno.cat not originating from mail.hypno.cat, they will be able to assume, or even enforce, that it should be rejected.\nProve that I actually authorized the message # DKIM is an authentication mechanism by which you can cryptographically sign mails emitted by your mail server, proving that you saw them and took responsibility in letting them transit. Hosts receiving these mails can verify that you authorized them, by verifying the signature and detect possible forgery.\nLike SPF, this is informative by default and failing a DKIM signature or even lacking the mechanism doesn\u0026rsquo;t mean a mail can\u0026rsquo;t inbox, however it accounts in the bad points given to a sender and degrades the reputation. It is also commonly accepted that lack of DKIM has a very negative impact on the reputation of a domain, so again, just do it.\nIt is slightly more complex than SPF because, although it also relies on DNS, for DKIM you have to generate a keypair in order to publish the public key in a DNS record.\nWe\u0026rsquo;re in 2019, I will be generating a 1024 bits RSA key, I know.\n\u0026mdash; BEGIN DIGRESSION \u0026mdash;\n2048 bits RSA public keys do not fit in a DNS TXT record and must be truncated into two records, but not everyone is able to cope with these truncated keys so you may lose DKIM benefits by going 2048 if hosts fail to use your (partial) public key and assume you\u0026rsquo;ve failed signing. Given that we\u0026rsquo;re not looking into security more than trying to look good at this point, we\u0026rsquo;ll just look the other way while excusing the fact that there\u0026rsquo;s a very moderate risk of impersonation when used in conjunction with SPF. This is very bad thinking but DKIM also has other short-comings and given that virtually everyone does 1024 bits RSA, I suggest we argue about the 1024 vs \u0026gt;= 2048 key size when we\u0026rsquo;re already successful at exchanging mails with Big Mailer Corps, but I\u0026rsquo;ll just take a moment to stress out, without pointing fingers at any, that even some security companies that actually work in the mail industry have issues with 2048 bits RSA.\nCompanies that actually provide sender reputation scoring. Oh, the, irony.\n\u0026mdash; END DIGRESSION \u0026mdash;\nIf this genuinely worries you, you can still experiment truncating key into two records and see if it suits you, adapting my example is trivial, just make sure you have a TXT record with a part of the key and another TXT record with the remaining.\nI\u0026rsquo;ll create a directory to hold the keys:\n# mkdir /etc/mail/dkim Then, the following commands will generate the keypair and extract the public key out of the private key:\n# openssl genrsa -out /etc/mail/dkim/hypno.cat.key 1024 Generating RSA private key, 1024 bit long modulus ..............................++++++ ...............++++++ e is 65537 (0x10001) # openssl rsa -in /etc/mail/dkim/hypno.cat.key -pubout -out /etc/mail/dkim/hypno.cat.pub writing RSA key # cat /etc/mail/dkim/hypno.cat.pub -----BEGIN PUBLIC KEY----- MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDThHqiM610nwN1nmV8OMc7PaPO uJWVGRDz5bWj4cRjTTmQYjJQd02xrydRNrRhjEKBm2mMDArNWjoM3jvN04ZifqJx DmKr7X8jsYi+MPEHz6wZxB8mayDK6glYTCyx//zl1luUdvm26PutA38K8cgnb7iT kfVP2OqK6sHAdXjnowIDAQAB -----END PUBLIC KEY----- Finally, I can create the DNS TXT record by extracting the public key out of the armor delimiters and formatting the content so it displays as follows:\n20190913._domainkey.hypno.cat.\tIN TXT \u0026#34;v=DKIM1;k=rsa;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDThHqiM610nwN1nmV8OMc7PaPOuJWVGRDz5bWj4cRjTTmQYjJQd02xrydRNrRhjEKBm2mMDArNWjoM3jvN04ZifqJxDmKr7X8jsYi+MPEHz6wZxB8mayDK6glYTCyx//zl1luUdvm26PutA38K8cgnb7iTkfVP2OqK6sHAdXjnowIDAQAB;\u0026#34; The name of the record should be constructed using this pattern: \u0026lt;selector\u0026gt;._domainkey.\u0026lt;domain\u0026gt;., where \u0026lt;selector\u0026gt; is a name you chose and which allows multiple keys to coexist. I like to use the date at which I generated the keypair, but whatever you chose, write it down for later as we\u0026rsquo;ll need it when configuring DKIM signing.\nAlso, note that the public key doesn\u0026rsquo;t have to be kept in the directory once it\u0026rsquo;s published in DNS. It can always be retrieved from the private key should it be needed for some reason, however I like keeping it around for fast reference. And make sure the private key isn\u0026rsquo;t world-readable if you have local users on your system, for some reason OpenSSL thinks private keys should be created rw-r--r--.\nInstructing hosts what to do with SPF and DKIM failures with DMARC # SPF and DKIM are both fine, but if they\u0026rsquo;re informative then it also mean they don\u0026rsquo;t prevent abuses of your domain name.\nDMARC allows instructing a destination mail server what to do with senders that fail SPF and DKIM tests. Most notably it allows instructing them to reject such senders.\nThere\u0026rsquo;s no clear indicator that providing a reject policy brings any positive points to a domain over not doing anything, however there are multiple hints that having a DMARC record has a positive impact over having none, even if that record states that nothing should be done about SPF and DKIM failures.\nThe format of DMARC records goes beyond this article and you\u0026rsquo;ll find multiple examples in any search engine, but a very simple record will simply state a policy (p=) of none (reject if you want to be harsh, quarantine if you need more time to decide). The percentage field (pct=) declares how many of these mails should be subjected to the DMARC policy, and the Reporting URI of Aggregated field (rua=) is where you should receive DMARC reports should you want to analyze them (before switching from quarantine to reject for example).\nI\u0026rsquo;ll just setup the most dummy record, one that will make all Big Mailer Corps see that we care about DMARC even though we don\u0026rsquo;t know what to do with it:\n_dmarc.hypno.cat. IN TXT \u0026#34;v=DMARC1;p=none;pct=100;rua=mailto:postmaster@hypno.cat;\u0026#34; Getting yourself a certificate for TLS # There are multiple ways you can obtain a TLS certificate, and assuming that you are already familiar with hosting other services, I\u0026rsquo;ll just pretend that you already know how to obtain one from your registrar or from letsencrypt.\nIf you know how to deal with your certificates, you can just skip this section and go to the next one.\nSince I\u0026rsquo;m really setting up mail.hypno.cat, I won\u0026rsquo;t be able to continue this article without obtaining a certificate, so as a courtesy to OpenBSD users I\u0026rsquo;ll document how I will generate mine on a brand new OpenBSD install. acme-client is a utility that lets you request and renew certificates from the console. It relies on an HTTP challenge, so it needs to be able to write to a directory that is served over HTTP, but luckily OpenBSD also ships with an httpd daemon that we can use for the challenge. Because we\u0026rsquo;re EXTRA lucky, it also provides an example configuration file that we can use with almost no change.\nSimply copy the example configuration from /etc/example/httpd.conf to /etc/httpd.conf, replace example.com with mail.hypno.cat and remove the tls block since we only care about serving the challenge. The file should now read as follow:\nserver \u0026#34;mail.hypno.cat\u0026#34; { listen on * port 80 location \u0026#34;/.well-known/acme-challenge/*\u0026#34; { root \u0026#34;/acme\u0026#34; request strip 2 } location * { block return 302 \u0026#34;https://$HTTP_HOST$REQUEST_URI\u0026#34; } } The httpd daemon can be started with the following command:\n# rcctl -f start httpd httpd(ok) The acme-client itself is straighforward to configure, again simply copy the example configuration from /etc/example/acme-client.conf to /etc/acme-client.conf, replace example.com with mail.hypno.cat and you should end up with a block like this:\ndomain mail.hypno.cat { domain key \u0026#34;/etc/ssl/private/mail.hypno.cat.key\u0026#34; domain full chain certificate \u0026#34;/etc/ssl/mail.hypno.cat.fullchain.pem\u0026#34; sign with letsencrypt } at this point you can simply run acme-client on the domain:\n# acme-client -v mail.hypno.cat [...] acme-client: /etc/ssl/mail.hypno.cat.fullchain.pem: created The certificate and keys are created at the appropriate place, we\u0026rsquo;ll later adjust paths in OpenSMTPD and Dovecot to point to the proper files.\nYou can keep the httpd running for future renewals and call acme-client from a cron, you can shut it down and deal with renewals in another way. You sort out how you want to renew with the help of the acme-client(1) man page.\nInstalling and configuring Rspamd # On OpenBSD, Rspamd is packaged and can be installed with a single command. We also need to install Redis which is used to store Rspamd statistics and greylisting states (among other things), as well as filter-rspamd which is the piece of code that allows OpenSMTPD to work with Rspamd. Both of them are also packaged so they can be installed with a single command too.\nAs I\u0026rsquo;m writing this, filter-rspamd will only be available in the next OpenBSD release due in October, so if you want to play with it before then, you can simply obtain it from github and read the README instructing how to build. Both Redis and Rspamd are already available in OpenBSD packages so the instructions below are already valid for them.\nFirst, I\u0026rsquo;ll install these packages:\nmail$ doas pkg_add redis rspamd opensmtpd-filter-rspamd [...] redis-4.0.14: ok rspamd-1.9.0: ok opensmtpd-filter-rspamd-0.1.1: ok The following new rcscripts were installed: /etc/rc.d/redis /etc/rc.d/rspamd See rcctl(8) for details. Rspamd is highly configurable so if you want to do funky things, it\u0026rsquo;s up to you to go read the extensive documentation. For the purpose of this article I\u0026rsquo;ll keep it very basic, as is the case on my own machines.\nI could edit the configuration in /etc/rspamd/actions.conf to adjust junking thresholds to my liking, but the defaults suit me fine so this was just to show them to you:\nactions { reject = 15; # Reject when reaching this score add_header = 6; # Add header when reaching this score greylist = 4; # Apply greylisting when reaching this score (will emit `soft reject action`) [...] And what I really, really want is for Rspamd to handle my DKIM signing because this is one of the requirements to not look bad to the world.\nThis is done by providing the configuration to match the DKIM key I generated earlier in /etc/rspamd/local.d/dkim_signing.conf:\n# mkdir /etc/rspamd/local.d # cat \u0026lt;\u0026lt; EOF \u0026gt;/etc/rspamd/local.d/dkim_signing.conf allow_username_mismatch = true; domain { hypno.cat { path = \u0026#34;/etc/mail/dkim/hypno.cat.key\u0026#34;; selector = \u0026#34;20190913\u0026#34;; } } EOF The allow_username_mismatch configuration is needed here because Rspamd expects usernames to contain the domain name, but in this setup OpenSMTPD authenticates against simple usernames. Also make sure the dkim key file is readable by members of the _rspamd group.\nWe\u0026rsquo;re done here, Rspamd and Redis can be enabled so OpenBSD starts them at next reboot:\n# rcctl enable redis # rcctl enable rspamd And I can start them right away so we don\u0026rsquo;t have to wait until next reboot:\n# rcctl start redis redis(ok) # rcctl start rspamd rspamd(ok) Configuring OpenSMTPD # OpenSMTPD is installed by default on OpenBSD so there is no installing phase here. If you run a different operating system, either someone has packaged it there and you can install using your package manager, or you can build from source and install by getting code from Github mirror and following build instructions.\nAs stated earlier, these instructions are only valid for the 6.6.0 release and later, earlier versions don\u0026rsquo;t support filters and some other features described here.\nThe default configuration for OpenSMTPD is suitable for local mail server, not accepting connections from the outside, but able to let local users exchange messages between themselves and to a remote host. It looks like this:\n# $OpenBSD: smtpd.conf,v 1.11 2018/06/04 21:10:58 jmc Exp $ # This is the smtpd server system-wide configuration file. # See smtpd.conf(5) for more information. table aliases file:/etc/mail/aliases # To accept external mail, replace with: listen on all # listen on lo0 action \u0026#34;local_mail\u0026#34; mbox alias \u0026lt;aliases\u0026gt; action \u0026#34;outbound\u0026#34; relay # Uncomment the following to accept external mail for domain \u0026#34;example.org\u0026#34; # # match from any for domain \u0026#34;example.org\u0026#34; action \u0026#34;local_mail\u0026#34; match for local action \u0026#34;local_mail\u0026#34; match for any action \u0026#34;outbound\u0026#34; It uses sane defaults as to make it hard to misconfigure, so if you\u0026rsquo;re worried about the last line (match for any) turning your server into an open-relay, don\u0026rsquo;t worry too much as it has an implicit (from local). You would have to very explicitely tell your server to match from any for any in order to let spammers abuse you.\nI initially wanted to update the configuration file progressively to hold your hand, but this article grew a lot since my first version. I\u0026rsquo;ll just put the complete 16 configuration lines configuration file and comment it extensively afterwards:\npki mail.hypno.cat cert \u0026#34;/etc/ssl/mail.hypno.cat.fullchain.pem\u0026#34; pki mail.hypno.cat key \u0026#34;/etc/ssl/private/mail.hypno.cat.key\u0026#34; filter check_dyndns phase connect match rdns regex { \u0026#39;.*\\.dyn\\..*\u0026#39;, \u0026#39;.*\\.dsl\\..*\u0026#39; } \\ disconnect \u0026#34;550 no residential connections\u0026#34; filter check_rdns phase connect match !rdns \\ disconnect \u0026#34;550 no rDNS is so 80s\u0026#34; filter check_fcrdns phase connect match !fcrdns \\ disconnect \u0026#34;550 no FCrDNS is so 80s\u0026#34; filter senderscore \\ proc-exec \u0026#34;filter-senderscore -blockBelow 10 -junkBelow 70 -slowFactor 5000\u0026#34; filter rspamd proc-exec \u0026#34;filter-rspamd\u0026#34; table aliases file:/etc/mail/aliases listen on all tls pki mail.hypno.cat \\ filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } listen on all port submission tls-require pki mail.hypno.cat auth filter rspamd action \u0026#34;local_mail\u0026#34; maildir junk alias \u0026lt;aliases\u0026gt; action \u0026#34;outbound\u0026#34; relay helo mail.hypno.cat match from any for domain \u0026#34;hypno.cat\u0026#34; action \u0026#34;local_mail\u0026#34; match for local action \u0026#34;local_mail\u0026#34; match from any auth for any action \u0026#34;outbound\u0026#34; match for any action \u0026#34;outbound\u0026#34; That\u0026rsquo;s all is needed for our complete SMTP setup.\nThe first two pki lines declare that mail.hypno.cat will use the certificate and key we generated earlier for TLS.\nThe filter lines apply a set of filters to incoming connections, check_dyndns will filter if rDNS matches some patterns, check_rdns will filter if rDNS is missing, check_fcrdns will filter if FCrDNS is missing, these are builtin filters in OpenSMTPD that you can tweak to filter at different phases or using other criterias.\nsenderscore is a custom filter that you can install either through your package manager (pkg_add opensmtpd-filter-senderscore on OpenBSD), or obtain from Github. It is not mandatory but I find it particularly useful and will explain why in the next section. In this configuration, it will reject a sender if its senderscore is below 10, junk message (add X-Spam header) if score is below 70, and for my own personal pleasure, add a delay to each response that is proportional to how bad the reputation is so bad senders are slowed down.\nrspamd is also a custom filter that we installed in the previous section. There\u0026rsquo;s no configuration as it is fully controlled from the Rspamd daemon responses, all we need to do is declare it.\nIf you want to be more conservative and not reject mails to avoid false positives, you can assign a junk action rather than disconnect action to builtin filters and remove the -blockBelow option in the senderscore filter:\nfilter check_dyndns phase connect match rdns regex { \u0026#39;.*\\.dyn\\..*\u0026#39;, \u0026#39;.*\\.dsl\\..*\u0026#39; } junk filter check_rdns phase connect match !rdns junk filter check_fcrdns phase connect match !fcrdns junk filter senderscore proc-exec \u0026#34;filter-senderscore -junkBelow 70 -slowFactor 5000\u0026#34; This way instead of rejecting sessions, OpenSMTPD will simply junk them so they are in your Spam folder. By monitoring what ends in your junk folder, you can gain confidence and tune settings to match what you really want.\nThese simple filters, not even counting rspamd, are enough to cut most of my Spam turning hundreds of daily spam into a handful. The rdns regex one can be tuned as depending on your country, you may see patterns that are particularly repetitive and that would never appear in mine. To ease maintenance, you can store them in a file, one by line and use the following construct:\ntable \u0026lt;dyndns\u0026gt; file:/etc/mail/path-to-my-regex-file filter check_dyndns phase connect match rdns regex \u0026lt;dyndns\u0026gt; junk Let\u0026rsquo;s continue dissecting the configuration.\nThe table aliases points to a file with\u0026hellip; aliases in two columns. The left column has the user-part of an e-mail address you receive, the right column has the user to which it should be delivered (ie: root: gilles) It is recommended that you have aliases mapping root, abuse and postmaster to an account that you actually read. I didn\u0026rsquo;t do that because I\u0026rsquo;m going to trash the install after this article, but do it.\nThen comes the listen lines. The first one is the public endpoint that other mail servers can contact to send mails to hypno.cat, it is configured to offer tls using the certificate and key for mail.hypno.cat and filter incoming sessions through all my filters. The second one is the endpoint for my own users, on the submission port, it requires tls using the same certificates, requires auth using system users (the default, can be configured otherwise), and filters through the rspamd filter only so we can DKIM-sign our own messages.\nThe action lines define an action for local mail which is to deliver to a maildir, while classifying junk in a specific folder and resolving aliases, the other action is to relay mail and let other host know that we\u0026rsquo;re mail.hypno.cat when we greet ourselves. If a server had multiple IP addresses the relay action should also specify the source address to use (with src) to make sure the proper IP address is used, here I only have one IP address and it is used for mail so I don\u0026rsquo;t have to.\nFinally, the match lines are the ruleset. When an envelope enters the SMTP server through one of the listen endpoints, it is compared to each match line one after another until a matching action in found or the envelope is rejected. The conditions are fairly explicit so I won\u0026rsquo;t describe these four lines, you should be able to figure out by yourself what from any for domain \u0026quot;hypno.cat\u0026quot; means otherwise you shouldn\u0026rsquo;t be reading this in the first place.\nA few words about SenderScore # SenderScore is an IP reputation database which associates a score ranging between 0 and 100 to an IP address. The score is tied to the volume and behavior observed from these IP addresses, and while we can\u0026rsquo;t really know HOW these scores are computed because the methodology is not public, a bit of studying allows confirming that they didn\u0026rsquo;t make these numbers out of nowhere: they do correlate with delivery and error rates at various Big Mailer Corps and it is possible to influence the scoring with changes of behaviors that also correlate to changes in delivery and error rates.\nIt is also VERY obvious that to build their reputation score they have access to SMTP session informations that only a destination mail server can have, such as the volume of mails observed from an IP address, or the ratio of failed vs accepted recipients, and since the scoring correlates with delivery and error rates in my observations, this implies that they get this information from Big Mailer Corps. Do what you want with that bit of information.\nBased on MY understanding, it is a very interesting indicator to limit the amount of spam hitting you. I don\u0026rsquo;t trust it to be accurate when it comes to good reputations (because I kinda know how to work a bit around) but I do trust it to be HIGHLY accurate when it comes to bad reputations, because it takes a certain amount of doing bad things in high volumes to see a scoring degrade. If you don\u0026rsquo;t do bulk mailing, you should definitely either be unknown to them or have a scoring around 95 or above, otherwise you\u0026rsquo;re very likely doing something wrong.\nNot everyone is convinced about SenderScore and some delivery experts claim it\u0026rsquo;s bullshit. I personally ran months-long experiments graphing daily reputation and volumes, then comparing them to delivery graphs, and my opinion is quite the opposite: to me there is a very significant correlation that definitely helps classify senders. I think the best approach is to use the filter for junking, not blocking, and determining by yourself if you\u0026rsquo;re happy with it.\nSenderScore considers that hosts should have a reputation above 70, I would personally assume hosts below 80 to be good junking candidates, and hosts below 10 to be obvious rejects.\nInstalling and configuring Dovecot # OpenBSD also has a package for Dovecot which can be installed with a simple command:\nmail# pkg_add dovecot dovecot-2.3.7.2v0: ok The following new rcscripts were installed: /etc/rc.d/dovecot See rcctl(8) for details. New and changed readme(s): /usr/local/share/doc/pkg-readmes/dovecot [...] And, specifically for OpenBSD, following instructions from the readme, the /etc/login.conf file should contain a dovecot class to bump the resources allowed as Dovecot is very hungry for file descriptors:\ndovecot:\\ :openfiles-cur=1024:\\ :openfiles-max=2048:\\ :tc=daemon: Once that is done, there\u0026rsquo;s really not much to do on the Dovecot side except have it point to the proper TLS certificate we generated earlier.\nI do that by tweaking the ssl_cert and ssl_key configuration keys in the /etc/dovecot/conf.d/10-ssl.conf file so they read as follows:\nssl_cert = \u0026lt;/etc/ssl/mail.hypno.cat.fullchain.pem ssl_key = \u0026lt;/etc/ssl/private/mail.hypno.cat.key Then Dovecot must be told that mails should be looked up in the user ~/Maildir directory as this is where OpenSMTPD drops files. I do that by tweaking the mail_location configuration key in the /etc/dovecot/conf.d/10-mail.conf file so it reads as follows:\nmail_location = maildir:~/Maildir We\u0026rsquo;re all set, we can enable the daemon so it\u0026rsquo;s started at next boot, and start it right away:\nmail# rcctl enable dovecot mail# rcctl start dovecot dovecot(ok) At this point, you can already configure any mail client like mutt, thunderbird or even the gmail app on Android, so that it uses mail.hypno.cat both for incoming and outgoing mails.\nTeaching Dovecot to train Rspamd # You read until here, good.\nSo this is a bonus section, one that\u0026rsquo;s absolutely not mandatory in setting up your mail server, but one that I\u0026rsquo;d like to add because it lets users train the spam filtering the same way they\u0026rsquo;re used to already: moving a mail to spam, marking a mail as spam or reporting a mail as spam, whichever it is called on their interface. The good news is that it integrates with IMAP, so no matter if they use their smartphone app, a client on their desktop or a webmail, the action of reporting a spam will automatically train the filter.\nHow you do that is relatively simple in theory, it consists of plugging a couple scripts, one to handle moving a mail from Inbox to Spam and one to handle moving mail from Spam to elsewhere. In pratice, Big Mailer Corps filters also detect that you delete a mail without opening it and such, but let\u0026rsquo;s not get too excited and start with something that\u0026rsquo;s already good enough. How you do that in practice is a bit more complex, and that is because we will rely on Sieve which is\u0026hellip; how do you put it with diplomacy \u0026hellip; \u0026ldquo;overengineered\u0026rdquo;.\nGiven that this is not a critical piece of the setup, you are allowed to shut your brain off and follow blindly instructions. Worst than can happen at this point is that it will not train anything.\nTo enable imap sieve, the Pigeonhole package must be installed, and as has been the case throughout this article all I need is a simple command:\nmail# pkg_add dovecot-pigeonhole dovecot-pigeonhole-0.5.7.2v1: ok [...] And\u0026hellip; here comes the tricky part, you need to configure Dovecot to enable imap_sieve, and teach it what sieve scripts to use for what actions.\nFirst, I will need to enable imap_sieve in Dovecot by adding it to the mail plugins for IMAP. This is done by tweaking the mail_plugins configuration key in /etc/dovecot/conf.d/20-imap.conf:\nprotocol imap { [...] mail_plugins = $mail_plugins imap_sieve [...] } Then to teach Dovecot how to train, I\u0026rsquo;ll add the following bit to /etc/dovecot/conf.d/90-plugin.conf, telling it that there are sieve scripts to trigger when mail gets moved in and out of Junk mailbox:\nplugin { sieve_plugins = sieve_imapsieve sieve_extprograms sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment imapsieve_mailbox1_name = Junk imapsieve_mailbox1_causes = COPY APPEND imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve imapsieve_mailbox2_name = * imapsieve_mailbox2_from = Junk imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve imapsieve_mailbox3_name = Inbox imapsieve_mailbox3_causes = APPEND imapsieve_mailbox3_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve } Now that dovecot is ready, I\u0026rsquo;ll have to prepare the sieve part which relies on two sieve scripts to train spam and ham inside /usr/local/lib/dovecot/sieve:\n# cat report-ham.sieve require [\u0026#34;vnd.dovecot.pipe\u0026#34;, \u0026#34;copy\u0026#34;, \u0026#34;imapsieve\u0026#34;, \u0026#34;environment\u0026#34;, \u0026#34;variables\u0026#34;]; if environment :matches \u0026#34;imap.mailbox\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;mailbox\u0026#34; \u0026#34;${1}\u0026#34;; } if string \u0026#34;${mailbox}\u0026#34; \u0026#34;Trash\u0026#34; { stop; } if environment :matches \u0026#34;imap.user\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;username\u0026#34; \u0026#34;${1}\u0026#34;; } pipe :copy \u0026#34;sa-learn-ham.sh\u0026#34; [ \u0026#34;${username}\u0026#34; ]; # cat report-spam.sieve require [\u0026#34;vnd.dovecot.pipe\u0026#34;, \u0026#34;copy\u0026#34;, \u0026#34;imapsieve\u0026#34;, \u0026#34;environment\u0026#34;, \u0026#34;variables\u0026#34;]; if environment :matches \u0026#34;imap.user\u0026#34; \u0026#34;*\u0026#34; { set \u0026#34;username\u0026#34; \u0026#34;${1}\u0026#34;; } pipe :copy \u0026#34;sa-learn-spam.sh\u0026#34; [ \u0026#34;${username}\u0026#34; ]; Because both sieve scripts rely on shell scripts sa-learn-ham.sh and sa-learn-spam.sh, I also need to create these shell scripts in /usr/local/lib/dovecot/sieve:\n# cat sa-learn-ham.sh #!/bin/sh exec /usr/local/bin/rspamc -d \u0026#34;${1}\u0026#34; learn_ham # cat sa-learn-spam.sh #!/bin/sh exec /usr/local/bin/rspamc -d \u0026#34;${1}\u0026#34; learn_spam And finally, I need to compile the sieve scripts and make the shell scripts executable so that Dovecot can actually use them:\n# sievec report-ham.sieve # sievec report-spam.sieve # chmod 755 sa-learn-ham.sh # chmod 755 sa-learn-spam.sh That\u0026rsquo;s all, now moving mails from a folder to another I can see the following lines in the /var/log/rspamd/rspamd.log file:\n2019-09-13 23:59:46 #18598(controller) \u0026lt;1d44bd\u0026gt;; csession; rspamd_controller_learn_fin_task: \u0026lt;127.0.0.1\u0026gt; learned message as ham: CAHPtQbOxQxBCsVd7nUCP4podu74Pa-F6k28z+4BWfNeeqWYiAg@mail.gmail.com [...] 2019-09-14 00:01:57 #18598(controller) \u0026lt;b76e28\u0026gt;; csession; rspamd_controller_learn_fin_task: \u0026lt;127.0.0.1\u0026gt; learned message as spam: CAHPtQbOxQxBCsVd7nUCP4podu74Pa-F6k28z+4BWfNeeqWYiAg@mail.gmail.com [...] I\u0026rsquo;m sure these can be simplified, I\u0026rsquo;ll be honest and say that I don\u0026rsquo;t really care because it\u0026rsquo;s just a side-feature, I\u0026rsquo;m unlikely to touch the sieve scripts for the next decade, I don\u0026rsquo;t think anyone should touch sieve scripts, we should just come with something better that doesn\u0026rsquo;t remind me of m4 wankery.\nTesting it all # And now, we\u0026rsquo;ll just test that we can do a round-trip from hypno.cat to gmail.com.\nFirst, I\u0026rsquo;ll craft a mail from my hypno.cat account to my gmail.com account:\nAfter sending it, I\u0026rsquo;ll check it arrives at gmail.com:\nThen, I\u0026rsquo;ll check that it was sent over a TLS-secured channel:\nI\u0026rsquo;ll check that gmail.com is happy with our SPF declaration, our DKIM signature and that it saw we had a DMARC record. You do that by selecting \u0026ldquo;Show original\u0026rdquo; in the menu associated to each mail:\nAnd finally, I\u0026rsquo;ll reply to the mail just to be sure that it works both ways:\nYay, we\u0026rsquo;re done # So this article is dense, I wanted to explain why we do things, if you remove all my blabber and focus on the purely technical aspects you\u0026rsquo;ll realize that we built a working mail infra, providing TLS inbound and outbound services, with DKIM, SPF and DMARC compliant outgoing traffic, and spam-filtered incoming traffic. And we did this by:\nadding a couple A records, an MX record and three TXT records to our DNS zone making sure rDNS was properly set installing TLS certificates building a ~15 lines configuration file for an SMTP server modifying 3 lines in the default configuration of an IMAP server modifying 8 lines in the default configuration of an antispam solution And because we were in a very good mood and willing to take an extra step to help users train antispam, we implemented ham and spam training of Rspamd by:\nmodifying approximately 20 lines in the default configuration of an IMAP server copying 4 sieve scripts in a directory Agreed, it may take a new comer some efforts to come up with these without help, but none of these tasks qualify as hard or tricky. In addition, pretty much all of them are only one-time operations, you don\u0026rsquo;t edit the DNS zone or reset your rDNS every two days, just like you don\u0026rsquo;t touch your working configuration once in place unless you have a new use-case. It may happen that you have to do a maintenance, whatever that means, but I don\u0026rsquo;t see a scenario where such a maintenance would imply having to do \u0026ldquo;all\u0026rdquo; of that again.\nThat being said, after you do it a few times, it takes about ten minutes to replicate this setup from scratch as you\u0026rsquo;ll know what you\u0026rsquo;re doing, because\u0026hellip; you know, mail is not hard.\nSo what should you do next ? # What you should do next is setup redundancy to ensure that a failure of your mail server does not causes mail loss.\nIn practice, most mail exchangers will retry if their destination is down, so even if you endure a downtime, most of your mails will be buffered by your sender and resumed when your server is back.\nIn theory, you still want to do things right, and letting others buffer your own messages to cope with your downtimes is fairly impolite as they bear responsibility of your messages far longer than they should\u0026hellip; You should investigate how to setup a backup mail exchanger, arrange with friends you trust to backup each other, or even subscribe to a service that will offer backup mail to you. Anything as long as you know a secondary server will be able to cover for you if you have a downtime.\nThen, no matter which way you chose to do it, it will boil down to:\nadding an additional MX record to the DNS zone configuring a backup mail server so it forwards mail for your domains to your mail server Rocket, science.\nAre we done ? # I will write a serie of articles to discuss reputation, as well as some of the mechanisms in place at Big Mailer Corps that can cause people to fail delivery.\nIf these topics are of interest and I get positive feedback, I might just write more often about deliverability concepts, otherwise I\u0026rsquo;ll just resume what I was doing before: writing about my monthly opensource contributions. It\u0026rsquo;s up to you :-)\nFinally, in this article I have described a very simple setup but there are tons of interesting stuff you can do with mail infrastructures if you\u0026rsquo;re not scared of working a bit on it.\nI\u0026rsquo;m currently building an infrastructure for a future mail hosting service which spans over multiple data centers in multiple countries, which has a very high tolerance to failures, has backup mail servers to cope with very extreme failures of primary mail servers, can very easily scale to bursts of volumes, that decorrelates incoming and outgoing traffic so it can provide partial services to people, that can easily reroute traffic across machines to work around failures, while providing IMAP services to virtual accounts on a multitude of domains, all with daily backups of all mail directories. The infrastructure currently costs me\u0026hellip; less than 75 EUR a month. It is also fairly simple with most machines being default installs of OpenBSD with very few changes to the base system.\nIf reading about how you can build such setups is of any interest, I could also write about it to help people build MORE hosting alternatives which is ultimately what I hope for.\nThanks for reading me, I still don\u0026rsquo;t have a soundcloud, however I still have a patreon if you want to support me, or use the widgets right below the comment link to share this article on social medias.\nIf you find errors in this article, please let me know so I can fix them, I started writing this article late and continued half-way through the night so I might have let errors slip in despite multiple readings.\n","date":"14 September 2019","permalink":"/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/","section":"Posts","summary":"TL;DR: - NO TL;DR: this time, I spent hours writing, you should spend minutes reading. - OK... I explain in WAY TOO MUCH details how to setup a mail server EDIT (2019-10-26) # OpenSMTPD 6.","title":"Setting up a mail server with OpenSMTPD, Dovecot and Rspamd"},{"content":" TL;DR: - Mail is not hard: people keep repeating that because they read it, not because they tried it - Big Mailer Corps are quite happy with that myth, it keeps their userbase growing - Big Mailer Corps control a large percentage of the e-mail address space which is good for none of us - It's ok that people have their e-mails hosted at Big Mailer Corps as long as there's enough people outside too EDIT (2019-12-15) # A practical guide to set up a mail exchanger was published on this blog.\nDisclaimer # THIS IS FOR SYSADMINS WITH TECH KNOWLEDGE, WHO KNOW HOW TO HOST SERVICES. Self-hosting mail is not HARD but requires WORK, which are two different things. Setting up a mail infrastructures requires a lot of initial work, then basic long term maintenance.\nI work on an opensource SMTP server. I build both opensource and proprietary solutions related to mail. I will likely open a commercial mail service next year.\nIn this article, I will voluntarily use the term mail because it is vague enough to encompass protocols and software. This is not a very technical article and I don\u0026rsquo;t want to dive into protocols, I want people who have never worked with mail to understand all of it.\nI will also not explain how I achieve the tasks I describe as easy. I want this article to be about the \u0026ldquo;mail is hard\u0026rdquo; myth, disregarding what technical solution you use to implement it. I want people who read this to go read about Postfix, Notqmail, Exim and OpenSMTPD, and not go directly to OpenSMTPD because I provided examples.\nI will write a follow-up article, this time focusing on how I do things with OpenSMTPD. If people write similar articles for other solutions, please forward them to me and I\u0026rsquo;ll link some of them. it will be updated as time passes by to reflect changes in the ecosystem, come back and check again over time.\nFinally, the name Big Mailer Corps represents the major e-mail providers. I\u0026rsquo;m not targeting a specific one, you can basically replace Big Mailer Corps anywhere in this text with the name of any provider that holds several hundred of millions of recipient addresses. Keep in mind that some Big Mailer Corps allow hosting under your own domain name, so when I mention the e-mail address space, if you own a domain but it is hosted by a Big Mailer Corp, your domain and all e-mail addresses below your domain are part of their address space.\nOnce upon a time, the \u0026ldquo;mail is hard\u0026rdquo; myth # When you first look into becoming independant with your e-mails, as soon as you ask in a public tech medium for help setting up a \u0026ldquo;mail\u0026rdquo; server, people will invariably jump in the discussion to discourage you from attempting because \u0026ldquo;mail is hard\u0026rdquo;.\nNot only \u0026ldquo;mail is hard\u0026rdquo; but it also seems that \u0026ldquo;Big Mailer Corps have already won\u0026rdquo;, that \u0026ldquo;all mail you send will end up in your recipients\u0026rsquo; spam box\u0026rdquo;, and that \u0026ldquo;you will be flooded by spammers\u0026rdquo; who will \u0026ldquo;abuse your server to relay spam to the world\u0026rdquo;.\nWow, that\u0026rsquo;s overwhelming :-|\nYou just wanted to send and receive mail because it seemed like a good idea, and now this turned into the worst life decision. But is it ?\nSoftware is hard # Mail used to be hard. A long long time ago.\nIt was hard because software were hard to setup correctly. A mail server like Sendmail required a Ph.D in compilers theory to operate, and the elitist culture of postmasters who could read Sendmail\u0026rsquo;s so-called \u0026ldquo;configuration files\u0026rdquo; didn\u0026rsquo;t help create a user-friendly environment. Postmasters bragging about how hard Sendmail was, while disclosing their unconditional love for m4 was a pissing contest. I can easily imagine these people whipping themselves with fresh nettles as a hobby.\nPostfix is an alternative to Sendmail that has been around since 1998. While I wouldn\u0026rsquo;t consider it as \u0026ldquo;simple\u0026rdquo; by any stretch of the word, it is orders of magnitude simpler than Sendmail. With the help of a search engine, new comers will easily find tutorials to spot the three or four configuration options they need to tweak.\nOpenSMTPD is another alternative which was first released in 2013. It is also orders of magnitude simpler than Sendmail. The configuration reads almost as plain english and a usable configuration file can actually fit \u0026hellip; in a tweet.\n[full screen](/images/2019-08-30-tweet.png) I don\u0026rsquo;t have experience in other contenders, but most operating systems and distributions provide multiple alternatives, pre-packaged so you can install and run with a simple command.\nMail software is NOT hard. It was if you stopped looking in the 90\u0026rsquo;s.\nOK, software isn\u0026rsquo;t hard but dealing with SPAM is # The myth goes on by saying that mail is hard because as soon as you run your mail server, spammers are going to come in hordes and dealing with spam will become a daily nightmare. This could not be further from the truth.\nNow that we\u0026rsquo;ve established trust, I\u0026rsquo;m not going to lie to you: there are hordes of spammers.\nWhen you plug your server in to the internet, you\u0026rsquo;ll start seeing connections from random sources trying to get mail accross. You\u0026rsquo;ll see them coming from home connections, from remote countries, from IP addresses sharing the same range, there will be no end to how much amazement this will procure.\nBasically, they all fit in two categories:\nclients trying to abuse your server to use it as a relay to send spam to the world clients trying to send spam to you after having obtained your e-mail address somehow The ones from the first category are easy to deal with: IGNORE THEM. They search for misconfigured servers and try doing things that will be rejected on a properly configured server, or even try to authenticate with a dictionnary attack which is not going to succeed if you have good passwords. They are like mosquitoes on a summer evening, annoying but\u0026hellip; meh.\nc1a89cb774083905 smtp connected address=185.234.219.64 host=\u0026lt;unknown\u0026gt; c1a89cb774083905 smtp failed-command command=\u0026#34;AUTH LOGIN\u0026#34; result=\u0026#34;503 5.5.1 Invalid command: Command not supported\u0026#34; c1a89cb774083905 smtp disconnected reason=disconnect c1a89cb8c5b84cbf smtp connected address=193.32.160.143 host=\u0026lt;unknown\u0026gt; c1a89cb8c5b84cbf smtp bad-input result=\u0026#34;500 5.5.1 Invalid command: Pipelining not supported\u0026#34; c1a89cb8c5b84cbf smtp disconnected reason=quit c1a89cb9441966e7 smtp connected address=185.234.219.193 host=\u0026lt;unknown\u0026gt; c1a89cb9441966e7 smtp failed-command command=\u0026#34;AUTH LOGIN\u0026#34; result=\u0026#34;503 5.5.1 Invalid command: Command not supported\u0026#34; c1a89cb9441966e7 smtp disconnected reason=disconnect If they really, really bother you or you dislike logs full of such attempts, write a script that detects such patterns in logs and add them to your firewall. Out of pure laziness, I have never ever used scripts to deal with these in the twenty years I operated mail servers and I\u0026rsquo;m still here to talk about it. As long as you don\u0026rsquo;t see them succeeding anything, you can just disregard these.\nThe ones from the second category are slighly more annoying because if you ignore them, your mailbox becomes full of spam. Luckily, they are not so hard to filter through several simple means and it is very easy to reduce spam to a few, every now and then, properly classified in a Spam folder (also known as \u0026ldquo;junked\u0026rdquo; mails). I take absolutely no precaution hiding my e-mail address, gilles@poolp.org, and I sometimes get one or two spam e-mails per day in the junk folder. Not only is that not a daily nightmare, but it\u0026rsquo;s less than what I actually receive on my own Big Mailer Corps account, which I do not share as easily and which has an average of three to five daily junked mails.\nNote that some very simple filters you\u0026rsquo;d apply for the second category of spammers, are HIGHLY effective to also kill the spammers from the first category:\na56dece24dcac3d2 smtp failed-command command=\u0026#34;DATA\u0026#34; result=\u0026#34;550 message rejected\u0026#34; a56ded3dd6cda8c2 smtp failed-command command=\u0026#34;\u0026#34; result=\u0026#34;550 your IP reputation is too low for this MX\u0026#34; a56ded3f7db5b96c smtp failed-command command=\u0026#34;DATA\u0026#34; result=\u0026#34;550 message rejected\u0026#34; a56dec6ffdb2caef smtp failed-command command=\u0026#34;\u0026#34; result=\u0026#34;421 you must have rDNS to contact this MX\u0026#34; a56dec895475b9bf smtp failed-command command=\u0026#34;\u0026#34; result=\u0026#34;421 you must have FCrDNS to contact this MX\u0026#34; I\u0026rsquo;ll just state it as it is: You will never reach \u0026ldquo;absolute 0 spam\u0026rdquo;, it was proven mathematically in the 2000s, but the amount you\u0026rsquo;ll receive while self-hosted can be as low or lower as what you receive at Big Mailer Corps. Spam is not more of an issue self-hosted, no matter how much the marketing tries to tell you otherwise.\nTo illustrate this I emptied the Spam folder on my poolp.org account and my Big Mailer Corps account this morning, here\u0026rsquo;s the screenshot I took of both accounts tonight, poolp.org on the left and Big Mailer Corps on the right. Neither of them have spam in Inbox.\n[full screen](/images/2019-08-30-spambox.png) OK SPAM is not the issue but my mails will not reach my users at Big Mailer Corps # Another myth, with slightly more substance this time, is that sending e-mail to an address at a Big Mailer Corp will result in the e-mail being rejected or junked.\nLet me tell you a secret: Big Mailer Corps are not worried about you but are worried about big senders harassing their users. They do not care about your personal server sending a few mails, even if its in the thousands per months. What they care about is the infected computers or compromised servers flooding their users. What they care about are the marketing companies that are literally shitting over them, sending individually millions of commercial mails per day, trying to work-around spam filters, and that sometimes manage to go for a while without being rejected. Unless you are sending hundreds of thousands of mails to them on a daily basis, quite frankly and without trying to hurt your feelings, you fall waaaaaaaaaaaaaaay below the radars.\n[full screen](/images/2019-08-30-bigmailercorps.png) So why did I say this claim has more substance than the others ?\nBig Mailer Corps have introduced proof-of-work into mail exchanges, voluntarily or not.\nUnlike legitimate senders who want to reach a specific user, spammers want to reach a ton of users whoever they are, because statistically some of them are going to fall for whatever it is that they\u0026rsquo;re trying to sell. Among these spammers, we\u0026rsquo;ll also include marketing companies that buy lists of users from partners, sending indiscriminately to \u0026ldquo;activate\u0026rdquo; them in hope of reaching some percentage of openings. They don\u0026rsquo;t care who receives an e-mail as long as it opens, because you know, statistics. So the harder it is for them, the higher the chance they\u0026rsquo;ll switch to another target because it\u0026rsquo;s the only way to not fall behind statistically. Don\u0026rsquo;t think I\u0026rsquo;m making this up, you\u0026rsquo;d be amazed at what tricks and hacks are used to mechanically increase openings by a few percents, including lowering the number of recipients at a particular destination and increasing at another.\nIn opposition to this are legitimate users who can\u0026rsquo;t just switch to another target, they want to reach a specific recipient and, if work is needed to achieve that, well\u0026hellip; you got to do what you got to do. They\u0026rsquo;ll pour in the work needed to make it work.\nKnowing this, Big Mailer Corps came with sets of rules about what a Good Sender should do to be able to communicate with them. These lists are basically a proof of work: they do not guarantee that you\u0026rsquo;ll be able to send to them, they do not guarantee that you\u0026rsquo;ll hit the inbox and will not be junked, but they are the minimal set of things you should do to prove you actually care. And considering some spammers do their best to look good, if you don\u0026rsquo;t do it yourself it basically means you\u0026rsquo;re not even willing to do better than spammers.\nThese rules are not only here to annoy you, they are also very effective: some of the rules cannot be achieved for spammers who use compromised hosts, for example. This proof of work paradigm is annoying because it raises (in time) the cost of entry, but it can also be leveraged by others to kill spam. Since Good Senders most definitely want to be able to send to Big Mailer Corps, if you receive connections from clients that didn\u0026rsquo;t do the minimum to deliver there, you can shut them down yourself because they\u0026rsquo;re already cut from a big portion of the e-mail address space.\nThis is why I wrote this has more substance than the other claims. It\u0026rsquo;s true that IF you don\u0026rsquo;t even try to do the minimum work, THEN you\u0026rsquo;ll start with a penalty.\nIn practice, a notion of reputation is also into play. Some people don\u0026rsquo;t even try but they fall but fall sooooooo far below the radars\u0026hellip; that even without trying they\u0026rsquo;ll manage to send e-mails without issues. I often send mail to my Big Mailer Corp account from my development laptop, far from being properly configured, and they almost always reach inbox. A good reputation allows you to make some mistakes and go through without respecting all rules, while bad reputation increases the amount of work you need to do. Considering that a good reputation is earned from doing things right, you get the general idea: do things right.\nThe minimum set rules is VERY FAR from being hard. To quote someone on twitter:\n[full screen](/images/2019-08-30-tweet_2.png) I would add a few things to that list but this highlights that some people are already \u0026ldquo;golden\u0026rdquo;, just by setting up proper rDNS, SPF and DKIM. That last bit about handling incoming spam, we\u0026rsquo;ve already discussed it above ;-)\nOK, then why is everyone saying it\u0026rsquo;s hard ? # The first reason, is because no one claims otherwise and, since Big Mailer Corps benefits from this situation, they\u0026rsquo;re not going to contradict it either. Big Mailer Corps BENEFIT from the myth that mail is hard as this means more people rely on them, they control more of the e-mail address space, and this translates to more e-mails being analyzed for targeted advertisement. The more people are discouraged, the more people will eventually subscribe to their services, and since they already control a large share, they can make mail slighly more difficult by making their requirements higher (harder, not hard). This is not something they do in some kind of conspiracy, this is just the result of them obtaining more power because people stay away from self hosting.\nAnother reason is because it used to be hard a long time ago. People got traumatized by how hard it was to not screw up and never reevaluated the situation. Some people today genuinely discourage other people from running their mail server, citing the very real difficulties they faced over a decade ago, far before some of today\u0026rsquo;s tools even existed.\nAnd finally, another reason is that people keep repeating it without actually trying themselves. I know this for a fact because people have been telling me that mail is hard for the last ten years, and for the last ten years I asked what they found hard in order to try improving the situation. A VAST majority of these people confessed that they never actually tried: they read or heard that mail was hard, often from a source they trusted, then accepted that claim and started telling others that mail was hard. We can\u0026rsquo;t really blame them when that myth has been around for so long, if I had no previous knowledge of mail and did a quick search today to find out how to setup my server, I might just decide not to do it given how difficult it seems from reading others.\nWhat do we do from now ? # We need to reclaim mail. I\u0026rsquo;m not saying people shouldn\u0026rsquo;t be hosted at Big Mailer Corps, but these should not become the Pavlovian reaction to \u0026ldquo;where do I get an e-mail address ?\u0026rdquo;.\nAs long as there are enough mail hosts out there, Big Mailer Corps HAVE to remain friendly because their users can complain that their legitimate mail is junked, which results in a risk of them leaving for elsewhere, less e-mails, less targeted advertisement, less $$$. If the number of mail hosts shrinks to the point that trafic not coming from Big Mailer Corps is irrelevant, then it becomes their protocols and they can start making up rules that are not sustainable by anyone but them, because no one will notice when the insignificant number of e-mails not coming from there is junked. Again, not a conspiracy but a side effect of being the few relevant actors of a system: why bother abiding to standards that work for everyone and ensuring any sender can reach them\u0026hellip; if most of the trafic comes from a handful of actors.\nThis is already happening with one specifically requiring mails to be sent from another Big Mailer Corp to hit the inbox, or requiring that senders be added to the contacts for others. Any other sender will hit spambox unconditionnally for a while before being eventually upgraded to inbox.\nWe can\u0026rsquo;t let that happen: allowing e-mail to be fully controlled by a small set of cooperating multi-million users hosts is just accepting to be screwed.\nIt is therefore very important that we don\u0026rsquo;t let the myth propagate further. Our best interest is to have a WIDE variety of mail hosts and providers, small and big, commercial and not. We must not allow the number of mail hosts to shrink, they must increase so the e-mail address space out of the control of Big Mailer Corps remains significant.\nAnd by all means, we must not push everyone to use Big Mailer Corps, particularly because a lot of people simply read their e-mail from a smartphone and don\u0026rsquo;t see a difference in interface, so they could be using pretty much any provider and be just as happy.\nI hope my point gets accross, feel free to share this wherever you want and point people to this article.\nIn a few days I\u0026rsquo;ll publish a practical description of how to setup a host similar to mine, providing spam protection for incoming mail, and basic proof of work to make most Big Mailer Corps happy.\nIf you like my work, support me on patreon !\n","date":"30 August 2019","permalink":"/posts/2019-08-30/you-should-not-run-your-mail-server-because-mail-is-hard/","section":"Posts","summary":"TL;DR: - Mail is not hard: people keep repeating that because they read it, not because they tried it - Big Mailer Corps are quite happy with that myth, it keeps their userbase growing - Big Mailer Corps control a large percentage of the e-mail address space which is good for none of us - It's ok that people have their e-mails hosted at Big Mailer Corps as long as there's enough people outside too EDIT (2019-12-15) # A practical guide to set up a mail exchanger was published on this blog.","title":"You should not run your mail server because mail is hard"},{"content":" TL;DR: - small inprovements to the fion window manager - plakar is a backup utility I wrote a long time ago that I will share - tons of opensmtpd stuff, mostly filters and issues handling Shout outs to my patrons ! # As has become the habit, this report begins with a big thank you to my patrons, cited by contribution then alphabetical order.\nThis month has been sponsored by:\nJ. Derrick Wesley Mouedine Assaby Mischa Peters Diego Meseguer Edgar Pettijohn Jdelic Sean Geoff Hill Thomas Bleader Raton Nick Ryan Vegar Linge Halaand Igor Zinovik Paul Kelly Jan J C I have recently switched to a 75% part-time schedule at work so that I can spend a \u0026ldquo;free\u0026rdquo; week each month working on my own stuff, mostly opensource, without any kind of pressure: no one knows what I\u0026rsquo;ll be working on and no one but me gets to decide how I\u0026rsquo;ll spend this time. This comes at the cost of slashing a quarter of my wage, which is sustainable but not ideal, so while most of my \u0026ldquo;free\u0026rdquo; weeks will be spent on opensource, I\u0026rsquo;ll be doing some sponsored development or short contracts to cover some of the loss if needed.\nI opened a patreon account so that people who care about my work can sponsor it, allowing me to spend most (if not all) of these \u0026ldquo;free\u0026rdquo; weeks publishing code for the community. If you want me to spend more time doing this then you know what to do: become my patron !\nSo thanks again to my patrons who make this possible, I will list them in all of my monthly reports, they are the sponsors of the work I describe below.\nFLOSS Weekly #543 # This week, hosts Randal Schwartz and Jonathan Bennett invited me to discuss SMTP and OpenSMTPD in their show, FLOSS Weekly.\nThere\u0026rsquo;s not much to say besides that it was a pleasant experience, that I really enjoyed talking to them about OpenBSD and OpenSMTPD, and that you should go watch that episode and subscribe to the show :-)\nFion # I didn\u0026rsquo;t work much on the fion window manager, it\u0026rsquo;s obviously less of a priority to me than upcoming\u0026rsquo;s October OpenSMTPD release. Yet, I still managed to get a couple things in.\nI started by introducing a notion of \u0026ldquo;keyboard\u0026rdquo; mode to ease interactions with the window manager:\nFor debugging purposes, I had hardcoded a few keys as triggers for window management actions: w would create a workspace, d would destroy it, n and p would switch to next and previous ones. Then I worked on tiles, so I reassigned the n and p keys to switch between tiles. You get the idea, at some point you have to be able to use both workspaces and tiles at the same time :-)\nKeyboard modes allow some combinations of keys win+X to enter a mode specific to a fion concept. For instance, win+w enters workspace mode so the next key pressed applies to workspaces, win+t enters tile mode so the next key pressed applies to tiles. This makes it easier for me to remember the shortcuts because n becomes next in whatever mode you are, p becomes previous in whatever mode you are, etc\u0026hellip;\nThen, I worked on fixing one of the issues on the TODO because if I fix at least one issue each month, I might just be able to switch to fion by the end of the year :-p\nFion displays a status bar at the top of the screen which provides informations regarding the time, the current workspace and active tile. The status bar is updated by a call to a function called layout_update() which\u0026hellip; updates the layout on all screens. However, because I was learning XCB, I used the function xcb_wait_for_event() in my event loop, which caused fion to block between events and the layout_update() function to only be called when an event wakes up the loop. This meant that seconds in the displayed time would hang, then jump by several, it made things look laggy.\nI needed to switch to a multiplexed model and luckily, for once, it was not tricky to find how to do it as there\u0026rsquo;s a xcb_get_file_descriptor() function providing a descriptor to the connection. All it took was some rearranging of the event loop, using poll() to wake up on events happening on the connection, and a timeout to act as a \u0026rsquo;tick\u0026rsquo; so the layout can be updated disregarding if events occured or not.\nI committed these improvements, my next task will be to handle the superposition of tabs inside tiles so we can have multiple X clients sharing the same tile, then I\u0026rsquo;ll deal with killing tiles and resizing siblings, which is the main show stopper for switching to fion as far as i\u0026rsquo;m concerned.\nPlakar # I wasn\u0026rsquo;t sure if I was going to mention this yet but it\u0026rsquo;s useful enough.\nI\u0026rsquo;ve been self-hosted since ~1999 and along the years I\u0026rsquo;ve dealt with backups in various ways. I went from tar, to dump/restore, to incremental dump/restore, to rsync, to incremental rsync, and started writing my own tools for different use-cases I had.\nAs I\u0026rsquo;m doing some cleanup of my private repositories I found the following ones:\ndrwxr-xr-x 2 gilles gilles 512 Aug 3 2012 backup drwxr-xr-x 2 gilles gilles 512 Jul 23 2012 backuptool drwxr-xr-x 6 gilles gilles 512 Apr 1 2015 plakar Both backup and backuptool are going to bite the dust, they were basically experiments to come up with a one-file backup format, providing dedup and versionning of content. I won\u0026rsquo;t enter much details as they did work but were not that interesting and I didn\u0026rsquo;t use them for long, proof that they weren\u0026rsquo;t itching my scratch.\nOn another hand, plakar is far more interesting.\nIt is a utility that created a repository in ~/.plakar, and would then let you snapshot a filesystem, splitting each file into content-defined chunks and storing them into the repository. The chunks would be compressed and encrypted, hard-link games would ensure that the repository can maintain multiple snapshots of the same directories at virtually no cost, encryption would allow the repository to be pushed to a remote server since I tend to keep backups at two sites and on my google drive.\nAt this point, some of you will ask \u0026ldquo;Isn\u0026rsquo;t that what tarsnap do ?\u0026rdquo;, to which I\u0026rsquo;ll answer \u0026ldquo;It seems to but I don\u0026rsquo;t really know as I haven\u0026rsquo;t used it and tarsnap seems not to allow self-hosting\u0026rdquo;.\nAnyways, that repository contains a version written in Python which supports various operations: push to save a snapshot into the repository, pull to restore a snapshot, remove to kill a snapshot, list to list snapshots, diff to diff two snapshots and search to search for files and directories inside snapshots. In addition, operations can be done on portions of a snapshot, not requiring the extraction of an entire snapshot when only a few files or directories are needed. I had started an UI which allowed plakar to display a randomized password and launch a local webserver expecting that password, a web interface would then allow visualizing the snapshots and their contents.\nlaptop$ plakar list laptop$ plakar push ~/wip/OpenSMTPD snapshot ee30bf0b4a4f44b5bfe71c632424c4d3: dirs=47, files=426, errors=0 laptop$ plakar list 2019-08-25-05:34:01 ee30bf0b4a4f44b5bfe71c632424c4d3 dcnt=47 fcnt=426 size=28.4MB laptop$ plakar pull ee30 laptop$ ls ee30bf0b4a4f44b5bfe71c632424c4d3/home/gilles/wip/OpenSMTPD/ README THANKS regress smtpd smtpscript laptop$ ls /home/gilles/.plakar chunks pending purge resources snapshots laptop$ plakar ui To use UI, go to http://127.0.0.1:8080, and use password: 2f1f1b61350fdfbb Bottle v0.12.17 server starting up (using WSGIRefServer())... Listening on http://127.0.0.1:8080/ Hit Ctrl-C to quit. [full screen](2019-08-25-plakar_1.png) [full screen](2019-08-25-plakar_2.png) [full screen](2019-08-25-plakar_3.png) [full screen](2019-08-25-plakar_4.png) I don\u0026rsquo;t intend to release the Python code as I will not maintain it, however to learn Golang I rewrote plakar in Go this week-end and I have a version that works. It\u0026rsquo;s not yet finished but the push/pull/list/trash commands are working, I\u0026rsquo;m going to start using it on my machines and see how it goes for the next few weeks before I publish the code on Github.\nlaptop$ plakar list laptop$ plakar push ~/wip/OpenSMTPD 2b5bec66-1799-48a1-8418-42509347eac1 laptop$ plakar list 2b5bec66-1799-48a1-8418-42509347eac1 2019-08-25 05:38:58.912591955 +0200 CEST laptop$ plakar pull 2b5 Restoring directories: 48/48 Restoring files: 426/426 laptop$ plakar trash 2b5 laptop$ plakar list laptop$ laptop$ ls 2b5bec66-1799-48a1-8418-42509347eac1/home/gilles/wip/OpenSMTPD/ README THANKS regress smtpd smtpscript laptop$ ls /home/gilles/.plakar chunks objects purge snapshots transactions laptop$ OpenSMTPD # On the OpenSMTPD front, I\u0026rsquo;ve done quite a lot of work so I will only mention the big stuff.\nErrata published # A user reported a bug in OpenSMTPD causing the daemon to hit one of its own sanity checks and call fatal(), resulting in a denial of service.\nThe bug affected OpenSMTPD 6.5 and prior, so I had to investigate and prepare patches for OpenBSD 6.5, OpenBSD 6.4 as well as OpenSMTPD-portable.\nProxy-v2 support # I committed experimental support for proxy-v2 protocol, based on a diff contributed by Antoine Kaufmann and reworked a bit.\nWhat this does is allow OpenSMTPD to work with SMTP sessions proxied by haproxy, for example. The session begins with a header which contains informations regarding the source address, the proxy layer in OpenSMTPD will use that information to rewrite a struct sockaddr_storage for the SMTP engine.\nI have ran the code for my own testing but it is experimental because I have only done light testing, we need feedback from power users.\nHandled a ton of Github issues # We had been lagging a bit behind issues on Github, many of them being either easily fixable, already resolved or user mistakes.\nI took the time to go through about twenty of them, resolving half in a way or another and ensuring the other half is on the right track.\nSeveral issues are actually dependant of some projects we have, so they are stuck until we\u0026rsquo;re done with a project and will be closed in batches once this is done. This is for example the case for several tickets related to TLS which will not be handled until we complete the switch to libtls\u0026hellip; which provides out of the box support for these features or makes them trivial.\nfilter-rspamd # I have pre-released version 0.1.0 of my filter-rspamd on Github.\nIt has been running since last month on my machines without a single issue, so I guess it is fairly usable.\nI also committed a port to OpenBSD so that it\u0026rsquo;ll be possible to pkg_add filter-rspamd.\nfilter-senderscore # I have also pre-released version 0.1.0 of my filter-senderscore on Github.\nWhat it does is query the SenderScore DNS to obtain the reputation of a sender IP address, then allow blocking, flagging as Spam and/or applying a reputation-dependant delay to sessions.\nI committed a port to OpenBSD for that one too so it\u0026rsquo;ll be possible to pkg_add filter-senderscore.\nfilter-checksenderdomain # This one I don\u0026rsquo;t think I\u0026rsquo;ll release, but it\u0026rsquo;s available on Github.\nThe name should be quite obvious, it simply performs a DNS lookup on the MAIL FROM domain to check if it resolves. I wrote it to prove that some of our issues don\u0026rsquo;t belong in OpenSMTPD but can be solved as easily in a filter.\nsmtp-out reporting # I decided to postpone the smtp-out reporting because it requires a bit of rework on the IPC between SMTP engine and filtering layer, something not visible to people writing filters, but which I\u0026rsquo;m not comfortable doing at this point knowing that I may not be very available in September in case I break things.\nI will continue working on smtp-out reporting so it can get committed when OpenBSD 6.6 is out, until then I\u0026rsquo;ll work on stuff that are risk-free :-)\nWhat next ? # I will take next few weeks to stress test OpenSMTPD and ensure our next release is rock-solid, then I will enter some kind of code freeze until 6.6 is out because the changeset is already significant.\nWhen I write my next report in late September, I may or may not yet have fork()-ed a little human in my house, no guarantees that the September report will be on time !\nThat being said, I\u0026rsquo;m curious if the format of these reports is readable, if you have suggestions feel free to let me know. Should I provide more/less details, be more/less technical ?\nI intend to write some technical articles regarding the internals of Plakar, since there may be bits of interest to other projects, including a noSQL private repo I still have sleeping out there ;-)\nAm I readable ? Let me know.\nIf you like my work, support me on patreon !\n","date":"25 August 2019","permalink":"/posts/2019-08-25/august-2019-report-fion-plakar-and-opensmtpd/","section":"Posts","summary":"TL;DR: - small inprovements to the fion window manager - plakar is a backup utility I wrote a long time ago that I will share - tons of opensmtpd stuff, mostly filters and issues handling Shout outs to my patrons !","title":"August 2019 report: Fion, Plakar and OpenSMTPD"},{"content":"","date":"25 August 2019","permalink":"/tags/plakar/","section":"Tags","summary":"","title":"plakar"},{"content":" TL;DR: - not much work outside of OpenSMTPD this week - OpenSMTPD portable builds with OpenSSL 1.1.x again - smtp-out reporting is working correctly on my laptop - wrote two filters that I'm actually using as you read this Shout outs to my patrons ! # As will become the tradition hopefully, this report begins with a big thank you to my patrons, cited by contribution then alphabetical order.\nThis month has been sponsored by:\nJ. Derrick Mischa Peters Diego Meseguer Vegar Linge Halaand Bleader Raton Nick Ryan C Igor Zinovik Jan J I have recently switched to a 75% part-time schedule at work so that I can spend a \u0026ldquo;free\u0026rdquo; week each month working on my own stuff, mostly opensource, without any kind of pressure: no one knows what I\u0026rsquo;ll be working on and no one but me gets to decide how I\u0026rsquo;ll spend this time. This comes at the cost of slashing a quarter of my wage, which is sustainable but not ideal, so while most of my \u0026ldquo;free\u0026rdquo; weeks will be spent on opensource, I\u0026rsquo;ll be doing some sponsored development or short contracts to cover some of the loss if needed.\nI opened a patreon account so that people who care about my work can sponsor it, allowing me to spend most (if not all) of these \u0026ldquo;free\u0026rdquo; weeks publishing code for the community. If you want me to spend more time doing this then you know what to do: become my patron !\nSo thanks again to my patrons who make this possible, I will list them in all of my monthly reports, they are the sponsors of the work I describe below.\nWhat\u0026rsquo;s in it for you ?\nNot much beyond the monthly public shout out, the satisfaction of seeing some projects make progress thanks to you, as well as my gratefulness for helping me spend more time on something I love.\nThe adventurous may even get early beta access to some code or private repository when stated in the report ;-)\nOpenSMTPD portable builds again with OpenSSL 1.1.x # As I explained in a previous report in May, API changes that occured in OpenSSL after the LibreSSL fork caused enough divergence that it became impossible to build the same code for both. Since OpenSMTPD is developed on OpenBSD which ships with LibreSSL, this resulted in failures to build with OpenSSL 1.1.x and me trying to unfuck the situation through a maze of #ifdefs until my sanity began melting.\nIn May, I brought a few functions from OpenSSL to LibreSSL to help reduce the divergence in OpenSMTPD scope and as a result, with minimal work, I have been able to build a working OpenSMTPD on an Ubuntu system with OpenSSL 1.1.x again.\n$ uname -srm Linux 5.0.0-15-generic x86_64 $ openssl version OpenSSL 1.1.1b 26 Feb 2019 $ sudo smtpd -d [sudo] password for gilles: info: OpenSMTPD 6.5.0-portable starting ^C When I announced that I would target LibreSSL and no longer OpenSSL, a lot of people assumed that I was breaking OpenSSL support voluntarily which isn\u0026rsquo;t accurate. I just did not fix a breakage that was already there: the same code could not build on LibreSSL, OpenSSL 1.0.x and 1.1.x, but while LibreSSL was my daily target and would always build, OpenSSL 1.0.x was still very widespread and OpenSSL 1.1.x was not yet packaged everywhere.\nEven though it was pleasing to no longer deal with the differences between the three, I did not enjoy the situation as it shrank our community considerably, reduced the number of contributions we received, forced people into switching to another MTA or use old versions of OpenSMTPD with old versions of OpenSSL\u0026hellip; It was possible to build LibreSSL and build an OpenSMTPD with LibreSSL everywhere, I did it myself on various systems, but the fact is there aren\u0026rsquo;t many systems that provide an OpenSMTPD package depending on LibreSSL. Either they stopped providing OpenSMTPD, they packaged it with a ton of unsupported patches to make it build with OpenSSL, or they took the easy path and had an old OpenSMTPD depend on OpenSSL 1.0.x, causing many people to run very old versions of our code\u0026hellip;\nSo what has changed since then ?\nOpenSSL no longer supports most of 1.0.x so it\u0026rsquo;s no longer as widespread as it used to, only 1.0.2 will be supported until the end of this year, most systems have already swiched to either LibreSSL or OpenSSL 1.1.x. With that and the fact that I committed changes in LibreSSL to reduce the gap with OpenSSL 1.1.x, the cost of supporting OpenSSL 1.1.x while keeping LibreSSL my daily target has become minimal.\nWe used to have #ifdefs in the code to cope with differences because it was VERY DIFFICULT to do otherwise, but now the code can assume LibreSSL and the OpenSSL differences are handled in the compat layer outside of OpenSMTPD code itself. I can look at the smtpd code without having to think \u0026ldquo;is this for LibreSSL or OpenSSL ?\u0026rdquo;.\nThe only #ifdef that has creeped in the code is wether or not ECDSA server certificates are supported, because the API is radically different, but in this case it simply disables it if a function from LibreSSL could not be found. This is very very manageable to me, I can live with that one ifdef.\nThese changes were all committed to the portable branch, so the next OpenSMTPD release in November will support OpenSSL 1.1.x again and there will be no excuse not to run a recent version of OpenSMTPD :-)\nReporting API gets extended # The reporting API, a mechanism by which OpenSMTPD can notify an external process of internal events in real time, has been extended considerably.\nAs a reminder, the API is a building block of filters as they can rely on it to construct an internal state for sessions, but it can also be used to build a wide range of external tools that rely on analyzing events. Improvements to the API can simplify considerably how filters are written, just as it can make possible writing new tools that couldn\u0026rsquo;t be written before.\nThe first extension to the reporting API is the link-auth event. It is generated whenever an SMTP authentication is requested by a client, and it records the username as well as the result of the authentication request. By registering for this event, it is possible to determine if a session was authenticated or not, track which sessions were issued by which user, etc\u0026hellip;\nreport|1|1564213979.757908|smtp-in|link-auth|63dc9f08477410de|gilles|pass report|1|1564214009.115878|smtp-in|link-auth|63dc9f06465b1cb2|gilles|fail report|1|1564214037.439016|smtp-in|link-auth|63dc9f0732384d8d|gilles|error While working on that, I made sure that OpenSMTPD would obfuscate parameters to any event it reports when in an authentication phase. This ensures that one can\u0026rsquo;t accidentally leak credentials to a third party filter, a database or a log file:\nreport|1|1562975948.315256|smtp-in|protocol-client|4772092f35819fed|AUTH LOGIN report|1|1562975948.315572|smtp-in|protocol-server|4772092f35819fed|334 VXNlcm5hbWU6 report|1|1562975961.559697|smtp-in|protocol-client|4772092f35819fed|******** report|1|1562975961.559724|smtp-in|protocol-server|4772092f35819fed|334 UGFzc3dvcmQ6 report|1|1562975974.600669|smtp-in|protocol-client|4772092f35819fed|******** report|1|1562975975.325244|smtp-in|protocol-server|4772092f35819fed|535 Authentication failed report|1|1563215271.737448|smtp-in|protocol-client|3138586bbd3c217b|AUTH PLAIN report|1|1563215271.737705|smtp-in|protocol-server|3138586bbd3c217b|334 report|1|1563215272.493300|smtp-in|protocol-client|3138586bbd3c217b|AUTH PLAIN ******** report|1|1563215273.000374|smtp-in|protocol-server|3138586bbd3c217b|535 Authentication failed The second extension is the introduction of a tx-reset event which is generated whenever a transaction is reset. A transaction encompasses a sender, one or multiple recipients as well as a message, so filters that deal with transaction will usually register multiple tx-* events to build a state with the informations they need. Since multiple transactions can occur in a session, there needs to be a way to clear the transaction data when it is no longer relevant. Until now, to do this filters had to register for events that initiate or terminate a transaction tx-begin, tx-commit and tx-rollback, to make sure that a cleanup function was called from each of these. With tx-reset, filters no longer need to register these events if they don\u0026rsquo;t intend to do anything with them, they can register a tx-reset handler to cleanup transaction with the guarantee that when a transaction is no longer relevant the event will be generated.\nreport|1|1564191006.311117|smtp-in|tx-commit|f6369c781fea802b|fac552ff|2806 report|1|1564191006.311120|smtp-in|tx-reset|f6369c781fea802b|fac552ff report|1|1564191100.066978|smtp-in|tx-rollback|f6369c8a16ddbcb8|8a4f9aad report|1|1564191100.066982|smtp-in|tx-reset|f6369c8a16ddbcb8|8a4f9aad Finally, until now the reporting API \u0026ldquo;only\u0026rdquo; generated events for smtp-in which is incoming SMTP trafic, but I have started teaching smtp-out, which is outgoing SMTP trafic, to also report all events. This work is not fully done but it works enough that I have deployed it on my own MX:\nreport|1|1564191030.737128|smtp-out|link-connect|f6369c7b41a82d01|localhost|pass|45.76.46.201:40938|127.0.0.1:25 report|1|1564191030.738602|smtp-in|link-connect|f6369c7cc2cb4dfe|out.mailbrix.mx|fail|45.76.46.201:40938|127.0.0.1:25 report|1|1564191038.043809|smtp-in|link-connect|f6369c7d5157f9d8|mail.mailbrix.mx|pass|95.179.226.54:39341|45.76.46.201:25 report|1|1564191052.947431|smtp-out|link-connect|f6369c85f7a894d6|localhost|pass|45.76.46.201:4502|127.0.0.1:25 link-auth and tx-reset have already been committed, the smtp-out work should be committed this month when I have built enough confidence it doesn\u0026rsquo;t break anything.\nHere comes my filter-rspamd # I had already written a proof of concept rspamd filter in Python a few months ago, but it was meant to prove the API allowed such filters, not to be usable for real so I didn\u0026rsquo;t really care about it being robust or not.\nMany people have asked me if I intended to improve and release it, so I decided to actually take time and write a proper filter-rspamd that I would use myself. I wrote the first lines of the filter yesterday early in the morning and by noon I had it deployed on my own MX. I decided to write it in Golang because I didn\u0026rsquo;t (and still don\u0026rsquo;t) know that language, so if I managed to write a useful filter in a language I don\u0026rsquo;t know in just a few hours I guess the goal for a simple and useful API is reached ;-)\nSo how does it work ?\nFirst the filter is built:\n$ cd filter-rspamd $ go build Then it is installed:\n$ doas cp filter-rspamd /usr/libexec $ The smtpd.conf file needs to be updated:\n[...] filter rspamd proc-exec \u0026#34;/usr/libexec/filter-rspamd\u0026#34; listen on all filter rspamd [...] And that\u0026rsquo;s all, what\u0026rsquo;s left to do is rspamd configuration outside of my scope :-)\nSince I\u0026rsquo;m not confident in my Golang skills, I wrote the filter in such a way that ANY error happening for pretty much any reason will flush message unprocessed as if the filter had been bypassed. No matter if you forgot to start rspamd or if I made an error that causes my filter to fail parsing rspamd response in some situations, any error will just output the mail as it was received: fail-safe by default.\nAs of today, the filter already does a lot of useful things.\nAdding X-Spam{,-Action,-Score} headers:\n$ cat /var/mail/gilles From gilles@laptop.home Fri Jul 26 08:54:35 2019 Return-Path: \u0026lt;gilles@laptop.home\u0026gt; Delivered-To: gilles@laptop.home X-Spam-Action: add header X-Spam: yes X-Spam-Score: 9 / 15 Received: from localhost (laptop.home [local]) by laptop.home (OpenSMTPD) with ESMTPA id f5918d8b for \u0026lt;gilles@laptop.home\u0026gt;; Fri, 26 Jul 2019 08:54:35 +0200 (CEST) From: \u0026lt;gilles@laptop.home\u0026gt; Date: Fri, 26 Jul 2019 08:54:35 +0200 (CEST) To: gilles@laptop.home Subject: test Message-ID: \u0026lt;04eabdcea50bc883@laptop.home\u0026gt; test Rewrite Subject line:\n$ cat /var/mail/gilles From gilles@laptop.home Sat Jul 27 10:43:34 2019 Return-Path: \u0026lt;gilles@laptop.home\u0026gt; Delivered-To: gilles@laptop.home X-Spam-Action: rewrite subject Received: from localhost (laptop.home [local]) by laptop.home (OpenSMTPD) with ESMTPA id 97b0006c for \u0026lt;gilles@laptop.home\u0026gt;; Sat, 27 Jul 2019 10:43:34 +0200 (CEST) From: \u0026lt;gilles@laptop.home\u0026gt; Date: Sat, 27 Jul 2019 10:43:34 +0200 (CEST) To: gilles@laptop.home Subject: *** SPAM *** test Message-ID: \u0026lt;7e41c7e558f77a3e@laptop.home\u0026gt; test Reject content, greylist a session or soft reject a transaction:\n$ echo \u0026#39;XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X\u0026#39; | mail -s test gilles 550 message rejected $ echo greylist-test | mail -s test gilles 421 Try again later $ echo softfail-test | mail -s test gilles 451 Try again later Finally, in addition to this, filter-rspamd supports adding the DKIM-signature header provided by rspamd if any, so by properly configuring rspamd to perform DKIM signing, e-mails will be automagically DKIM signed when filter-rspamd is installed. NO MORE FUCKING DKIMPROXY.\n[...] DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=poolp.org; s=opensmtpd; t=1564217527; h=from:from:reply-to:subject📅date:message-id:message-id:to:to:cc; bh=g3zLYH4xKxcPrHOD18z9YfpQcnk/GaJedfustWU5uGs=; b=bPRnicsNRhVi+SdeF6UYQw7ZfViIloc460vRm+5qljYgNtBKmYSnda0JBhSKm1Kryn8Zde hRShh6QPHChc2RtCsh33U1il6MSE12OmC6vnI6TxDa0t2hXbYFFihx/4UNkFe7J1MYnR4i 0hilMVclvtxknWwJf2sMIuVYFijl4b8= X-Spam-Action: add header X-Spam: yes X-Spam-Score: 6.3 / 15 Received: from smtp-out.poolp.org (out.mailbrix.mx [212.83.129.132]) by out.mailbrix.mx (OpenSMTPD) with ESMTP id a1b7fac1 for \u0026lt;gilles.chehade@gmail.com\u0026gt;; Sat, 27 Jul 2019 08:52:06 +0000 (UTC) Received: from localhost (poolp.org [local]) by poolp.org (OpenSMTPD) with ESMTPA id c097ac23 for \u0026lt;gilles.chehade@gmail.com\u0026gt;; Sat, 27 Jul 2019 10:52:06 +0200 (CEST) From: gilles \u0026lt;gilles@poolp.org\u0026gt; Date: Sat, 27 Jul 2019 10:52:06 +0200 (CEST) To: gilles.chehade@gmail.com Subject: test Message-ID: \u0026lt;63dca0f89904c9b1@poolp.org\u0026gt; test All of these behaviors and thresholds are controlled from the rspamd configuration, so filter-rspamd doesn\u0026rsquo;t actually have any configuration itself.\nThe filter is still a work in progress, so the code is not released yet ( patrons may get an early copy), but it is definitely useable as not only I\u0026rsquo;m running it on two MX but another OpenBSD hacker is also using it. I\u0026rsquo;ll likely release soon.\nHere also comes my filter-jsonlog # I worked for years at a company dealing with large volumes of mails and know the importance of analytics and dashboards. Until recently, the only way to push data for analytics was to parse the mail log, extract patterns of information from lines to push them into a database. The log format was designed to make this easier but it was not optimal as it was also meant to not overwhelm humans.\nPeople have managed to create OpenSMTPD dashboard using the mail log, but also through parsing the output of smtpctl show stats which both work but are respectively unfriendly and hackish. We can do better !\nWith reporting API came a format that was designed solely for applications, so the mail log format was simplified and trimmed from redundant informations, whereas the reporting API would publish a lot of informations humans don\u0026rsquo;t really care about when looking at logs.\nI wrote a filter-eventlog last month which would simply write the reporting stream to a file:\n[...] report|1|1564217526.405901|smtp-in|protocol-client|63dca0f760bad4af|MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; report|1|1564217526.406175|smtp-in|filter-response|63dca0f760bad4af|mail-from|proceed report|1|1564217526.408265|smtp-in|tx-begin|63dca0f760bad4af|c097ac23 report|1|1564217526.408278|smtp-in|tx-mail|63dca0f760bad4af|c097ac23|\u0026lt;gilles@poolp.org\u0026gt;|ok report|1|1564217526.408292|smtp-in|protocol-server|63dca0f760bad4af|250 2.0.0: Ok report|1|1564217526.408414|smtp-in|protocol-client|63dca0f760bad4af|RCPT TO:\u0026lt;gilles.chehade@gmail.com\u0026gt; report|1|1564217526.408630|smtp-in|filter-response|63dca0f760bad4af|rcpt-to|proceed report|1|1564217526.410380|smtp-in|tx-envelope|63dca0f760bad4af|c097ac23|c097ac2360bfa130 report|1|1564217526.410400|smtp-in|tx-rcpt|63dca0f760bad4af|c097ac23|\u0026lt;gilles.chehade@gmail.com\u0026gt;|ok report|1|1564217526.410417|smtp-in|protocol-server|63dca0f760bad4af|250 2.1.5 Destination address valid: Recipient ok [...] After discussing with a friend well versed in ELK he suggested I adopt a key-value format so it could be injected more easily, so I added a command line option so the filter would output in the following format:\n[...] timestamp=1564217526.405901 subsystem=smtp-in evt=protocol-client session=63dca0f760bad4af line=\u0026#34;MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; \u0026#34; timestamp=1564217526.406175 subsystem=smtp-in evt=filter-response session=63dca0f760bad4af phase=mail-from response=proceed timestamp=1564217526.408265 subsystem=smtp-in evt=tx-begin session=63dca0f760bad4af msgid=c097ac23 timestamp=1564217526.408278 subsystem=smtp-in evt=tx-mail session=63dca0f760bad4af msgid=c097ac23 address=\u0026lt;gilles@poolp.org\u0026gt; status=ok timestamp=1564217526.408292 subsystem=smtp-in evt=protocol-server session=63dca0f760bad4af line=\u0026#34;250 2.0.0: Ok\u0026#34; timestamp=1564217526.408414 subsystem=smtp-in evt=protocol-client session=63dca0f760bad4af line=\u0026#34;RCPT TO:\u0026lt;gilles.chehade@gmail.com\u0026gt; \u0026#34; timestamp=1564217526.408630 subsystem=smtp-in evt=filter-response session=63dca0f760bad4af phase=rcpt-to response=proceed timestamp=1564217526.410380 subsystem=smtp-in evt=tx-envelope session=63dca0f760bad4af msgid=c097ac23 evpid=c097ac2360bfa130 timestamp=1564217526.410400 subsystem=smtp-in evt=tx-rcpt session=63dca0f760bad4af msgid=c097ac23 address=\u0026lt;gilles.chehade@gmail.com\u0026gt; status=ok timestamp=1564217526.410417 subsystem=smtp-in evt=protocol-server session=63dca0f760bad4af line=\u0026#34;250 2.1.5 Destination address valid: Recipient ok\u0026#34; [...] This was better as I could basically pipe this output into elasticsearch without post-processing and see these events in a Kibana dashboard, however it only eased creating dashboards to compare different records for a similar event, a much much better format would be to dump the entire session state for each event which would allow displaying ANY information for ANY event.\nI wrote filter-jsonlog which is another golang filter that is slightly more complex than filter-eventlog. Instead of just reading events from the stream and outputing them in a different format, it actually builds an internal representation of the state of SMTP sessions as it receives events, and for each event it outputs the state of the correspond SMTP session in json format since that format is easily injectable in various databases.\nIf we take the previous example, while it would be very easy to create a piechart of top recipients because you\u0026rsquo;d be comparing data associated to a similar event, it would be very hard if not impossible to create a piechart of TLS vs non-TLS sessions for a specific recipient. With filter-jsonlog, because any event has its entire session state associated, visualzation can be created about ANY session information occuring at ANY event.\nThe output is logged to syslog so that syslog configuration can be used to determine in which file it should land, and newsyslog can be used to determine the rotation policy, while I use filebeat to inject the log to an elasticsearch node. The setup for this is really trivial :-)\nHere are two screenshots: full screen\n[full screen](2019-07-27-kibana.jpeg) Unfortunately I suck at creating dashboards myself, so if you want to help me get things in shape to improve the jsonlog output and provide dashboard templates for the community, get in touch with me !\nThis filter is also still a work in progress and the code is not released yet ( patrons may get an early copy), but it does work kinda.\nWhat about the Fion window manager ? # I haven\u0026rsquo;t talked about the Fion window manager because I didn\u0026rsquo;t spend much time on it, I was stuck trying to handle keyboard shortcuts correctly and I didn\u0026rsquo;t want to waste my week on this, however I got unstuck late last night so I will resume my work on it ;-)\nWhat next ? # The smtp-out reporting is very important to me so it needs to be completed and committed. I will also try to package filter-rspamd so it\u0026rsquo;s one pkg_add away from a fresh install. Hopefully people will help with filter-jsonlog and dashboards so I can move that faster than I would alone.\nThere\u0026rsquo;s still work needed in the filters area but it\u0026rsquo;s becoming good enough that I will start documenting, making it easier for people to actually start playing with it.\nIf you like my work, support me on patreon !\n","date":"27 July 2019","permalink":"/posts/2019-07-27/july-2019-report-tons-of-smtpd-work-mostly/","section":"Posts","summary":"TL;DR: - not much work outside of OpenSMTPD this week - OpenSMTPD portable builds with OpenSSL 1.1.x again - smtp-out reporting is working correctly on my laptop - wrote two filters that I'm actually using as you read this Shout outs to my patrons !","title":"July 2019 report: tons of smtpd work mostly"},{"content":" TL;DR: - started working on FION, a static tile window manager - revived BPG, a PGP parser - converted OpenSMTPD to libtls - wrote a library to make writing of native C OpenSMTPD filters easy - started writing a filter-rspamd Thanks to my patrons ! # First of all, a huge thanks to my first patrons:\nBleader Raton Diego Meseguer Mischa Peters Vegar Linge Halaand I have recently switched to a 75% part-time schedule at work so that I can spend a \u0026ldquo;free\u0026rdquo; week each month working on my own stuff, mostly opensource, without any kind of pressure: no one knows what I\u0026rsquo;ll be working on and no one but me gets to decide how I\u0026rsquo;ll spend this time.\nThis comes at the cost of slashing a quarter of my wage, which is sustainable but not ideal, so while most of my \u0026ldquo;free\u0026rdquo; weeks will be spent on opensource, I\u0026rsquo;ll be doing some sponsored development or short contracts to cover some of the loss if needed.\nI opened a patreon account so that people who care about my work can sponsor it, allowing me to spend most (if not all) of these \u0026ldquo;free\u0026rdquo; weeks publishing code for the community. If you want me to spend more time doing this then you know what to do: become my patron !\nSo thanks again to my patrons who make this possible, I will list them in all of my monthly reports, they are the sponsors of the work I describe below.\nFION: a featureless ion # I\u0026rsquo;m not too much of a graphical interface person, so when I was shown the ion window manager in 2003 I became addicted and, ever since, it\u0026rsquo;s been very hard for me to work efficiently using anything else. This is not an elitist thing, it is really me being easily distracted by stuff like windows not being properly aligned and similar stupid concerns that actually slow me considerably. I call this the \u0026ldquo;oh-a-butterfy\u0026rdquo; effect.\nUnfortunately, the last ion release dates from 2009 and is unmaintained. This leads to multiple problems, one being the lack of modern features, like multi-screen, which is essential for me considering the amount of time I spend reading code and specs side-to-side. Another problem is the fact that it\u0026rsquo;s been removed from most repositories, and I keep having to build it from source here and there. Recently it\u0026rsquo;s been a bit more difficult because on some distro, the dependencies for ion were no longer available and I had to start patching it here and there so it could build.\nI fear the day this happens on OpenBSD because it will mean that I have to spend time maintaining a piece of code for which there\u0026rsquo;s no upstream anymore \u0026hellip; or be back to working partly in console, partly in a maximized tmux when I really need a browser visible.\nI know this will lead people to tell me \u0026ldquo;but there\u0026rsquo;s notion, ion\u0026rsquo;s fork\u0026rdquo; or \u0026ldquo;but you can use awesome/dwm/i3/wmii/\u0026rdquo; but honestly I have tried a lot of different window managers along the years to try getting away from ion as it was abandoned and they never quite fit for me. I can only work if I have a wm that does tabs within static tiles that I can h-split/v-split at will and resize however I want. Notion came short because at least on OpenBSD it was laggy.\nI might as well spend a few hours / days working on my own, learn how they work, and release something I\u0026rsquo;m willing to maintain on the long run as it scratches my own itches.\nFION\u0026rsquo;s feature # I didn\u0026rsquo;t care that much for many of ion\u0026rsquo;s feature, never used the lua binding or the floating workspaces.\nFION will simply implement what I intend to use, so at the time being, this is:\npledge()-ed multiple screens of different sizes multiple workspaces per screen workspaces may be split in multiple horizontal and vertical tiles each tile may hold multiple X clients in tabs tiles may be resized screens / workspaces / tiles / tabs may be controlled through keyboard As of today, I\u0026rsquo;d say I\u0026rsquo;m halfway through, FION will detect screens and setup the workspaces to match screen size. It can create as many workspaces as you want on every single screen. It can h-split and v-split as many tiles as you want on the current workspace. It can iterate through workspaces and tiles. It can start the X clients and attach them, properly sized, to the current tile. It can give the focus to a tile through keyboard-triggered iteration of tiles or by moving mouse over a tile.\nCan you try it yet ? nope.\nFION currently does not have the logic to remove tiles, you can create and split but not kill tiles, which is not hard to do but I just ran of time this week ;-)\nAlso, keyboard events are not properly handled yet so I had to map features to a simple key like \u0026rsquo;n\u0026rsquo; or \u0026lsquo;p\u0026rsquo; to test them. Until I have more time for figuring out how I can implement my keyboard shortcuts so that they use control sequences, it will not be usable for daily use.\nCode is already committed on Github but again \u0026hellip; not usable, so stay tuned, the first usable version will be announced on twitter when it\u0026rsquo;s available.\nObligatory screenshots # I must have spent about six hours worth of work on it and the result is quite ok so I have good hope (no promise !) to make a first beta release during my \u0026ldquo;free\u0026rdquo; week in July.\nHere are a couple screenshots:\nFION ran on my laptop a few days ago before X clients were resized to match tiles:\n[full screen](2019-06-30-fion_1.png) FION ran on a bigger screen this morning:\n[full screen](2019-06-30-fion_2.png) And because a project doesn\u0026rsquo;t exist until it has a logo:\n[full screen](2019-06-30-fion_logo.jpg) BPG: BSD Privacy Guard # While looking at old private repositories to decide if I should axe them or not, I ran into an interesting one: bpg.\nLong story short, while I was a student in 2002 or 2003, I was doing night shifts at a security company to monitor firewalls and since it was essentially a passive job because I had nothing to do until the screens went all red, I started working on BPG a BSD-licensed PGP implementation.\nThen, NetBSD came up with a similar project for a Google Summer of code and they had someone already working on it, so I gave up as I had only a few weekly hours available for this and my work would surely be less good than someone working on it full-time for NetBSD.\nAnyways, I spent a while looking at the code to see what could be done with it, and it is actually not bad at all.\nI have a parser that can read all PGP packets in both old \u0026amp; new format, map them to appropriate structures that can actually be used: it can essentially read the keyrings and packets generated from gpg2, though it will not decrypt/verify.\nThe parser code was quite clean and I didn\u0026rsquo;t make changes to it except remove an allocator to replace it with malloc_conceal(3). I did a bit of code cleanup on other parts, like armoring which was ugly, but that\u0026rsquo;s about it.\nI don\u0026rsquo;t know what to do about that code, I have no idea how usable is the NetBSD implementation and how interesting it is for people to have another PGP utility. I will release the parser for sure since the code is already here and maybe it\u0026rsquo;s going to be useful to someone but I\u0026rsquo;m unsure if I should move project forward at this point. Oh and I need a new name anyways because bpg is already taken and it is certainly not going to be OpenBPG to avoid confusion with OpenBGP :-)\nI have quickly written code to generate RSA keypairs just for fun:\nlaptop$ bpg key gen rsa 2048 \u0026gt; /tmp/rsa2048.bpg ..........................................+++...................+++. laptop$ bpg debug parse /tmp/rsa2048.bpg ## 05 : New format Secret-Key packet (920 bytes) version: 4 algorithm: RSA (encrypt or sign) keyid: e60e44654c3409be fingerprint: 7bfd297855a9c0d22a492ba2e60e44654c3409be ## 0d : New format User ID packet (16 bytes) content: test@example.com laptop$ cat /tmp/rsa2048.bpg -----BEGIN PGP PRIVATE KEY BLOCK----- Version: BPG v0.1 (OpenBSD) xcLYBF0YccABCACzay1zxLQzD9WvOaF5DZXKIx5nmwP70+ff4XceepFnylwe7iKzTUF6W4kv QxOF++z9SQ5mF0melyj0+TkJXUSs7MzNjuwb4OhDLiM9uVJ2J1wxA3KivY0pVjvJPtPTlMpZ D7Ebb1CnagP75DEcgpdzWHDSzBbgHN89ct6M7fPZ756iDxHdnhO8syr9sB4GOJyj8hC2dC4y qJvn7G7/BdQwDov23FUGcddWXIyPMs4GGbE+mpV1U6jqZbNUkGPl+Kh8CfRU15D2/60PsrgD 3OSDo3J/DwGd2/NiVRRWVSOOtpzm4sc4j1J76nwrUJvvWRlH+AMOfDxoDu03bY2Jb/NtABEB AAEAB/9Pw8ZhQYIbcV6+mBCBkNiXFSXfSbtrqbncfpBGrJcYXY628YfbzuzdSPSkXl2/o1Cp CmGsYY4JQ4qh3mrNDvoJJv2mJXQysLqRo2Fnf4x5muYRpEbCsyKezgemYJgr6GpNTfyfBc4F n8xFoB11X1mVniwKi1FgMXXOC9OFNATFTlK2EjiBCBN1g+AbO31ykqDyhuVjAisQXlrpvRSr Jlk3zwNDgyu8jd9NyqCj3wUbp5GTj9/N+JVqm4lxqRAXUJk804w9q4Vn1x5f+lxyOw70m6vV 9xKZqKB/g2CmqFNVn8nzparvZQ45FSwKFkCz8Z0XrCJG8HvxkS0fBewu2BLBBADvAzcqblpX 9ygSDsOF/JDt9nhduLcVDgFzOagfZvh9eAny0YGQZ7rue2Vd3DKbMRBkybpn3jKpRbJC8+96 6AVaaxtoaHt7jKxZMHdjlxRPe6f9rFqETgL4a0+xyGzGd72t2OZG9pXNbydrwJPo0PSybkvR hkyd7vP3886X1EkdrwQAwCupzj3qSKf06ATAy+31rKoIUgsHD7OuyrNl7I1FZIMWNttaEl1C Z0bYaKHaTZbtoNk9EsISxSIXf3HB7nisKmGDe+LfXD06q9Bk1Tpjoj+zMXhJAm/NOQfbWXgJ 3/1b6EbI63o1JcBlZDXbrGv5IOY+c7gZqbOc3hD8EThHA6MD/0fQC4TuqH9nkJVyiDp0G+48 nq2oFDiDEfJ58TiSFlws/K+nuy2uGGO91iJLA6go5B87b0ihvl6A1Z4XFXuopRMcVp87n9gH qqRLhvoBqhMdVlfPnYr8JZYu0HkzmFtCnwuQOIA8EH8FQ65ImrP5PopazIHiUwhunrTzoAWp /4JOulbNEHRlc3RAZXhhbXBsZS5jb20= =0fuz -----END PGP PRIVATE KEY BLOCK----- laptop$ They are not valid, generating a valid key that can be imported to gnupg requires adding a signature packet which means I have to write the signing code to do that.\nSo there we go, either the project ends up as a PGP parsing library that others may use freely in their PGP implementations or it can evolve into a PGP library and utility, it depends on how needed and wanted this is by the community.\nI will commit what I have during July.\nOpenSMTPD: # What should come as no surprise: I spent most my week working on OpenSMTPD.\nI have tackled two topics:\nmigration from the OpenSSL to the libtls API for TLS support filters OpenSMTPD: OpenSSL -\u0026gt; libtls migration # I have started working on the migration from the OpenSSL API to the libtls API for TLS support in OpenSMTPD.\nAnd by \u0026ldquo;I have started\u0026rdquo;, I really mean \u0026ldquo;I\u0026rsquo;m almost done\u0026rdquo;.\nThe server side is mostly finished as I just need to rewrite the SNI support in a slightly different way. The client side is also mostly finished but needs a bit of cleanup.\nThis wasn\u0026rsquo;t as straightforward as I thought it would be, mostly due to the fact that OpenSMTPD uses a low-level \u0026ldquo;io\u0026rdquo; API for all its network io and that it was tricky to work on a progressive migration. I eventually found a way that allowed me to migrate the client-side first, then focus on the server-side, then use solely libtls. When this will be committed to OpenBSD, it will have to be an atomic switch, I won\u0026rsquo;t be able to progressively bring server and client side migrations.\nI also ran into some issues that were not easy to tackle, like the fact that we used ex data in RSA / EC_KEY objects to pass pki name to the privsep crypto engine for key lookups \u0026hellip; but libtls already used ex data to pass a checksum. I could have worked-around but it made sense leaving the checksum in ex data, as it\u0026rsquo;ll be useful in the future, so a bit of refactoring was require here and there.\nSo libtls-enabled OpenSMTPD is now a thing, when its finally commited I\u0026rsquo;ll be able to close multiple feature-requests that will be trivial to implement or already solved by sife-effect of the mgiration.\nDoes this mean OpenSMTPD will become LibreSSL only ?\nNOPE. This means that instead of having OpenSMTPD detect and deal with the differences between OpenSSL and LibreSSL as well as different OpenSSL versions, it will just depend on the libtls interface. We can then work on having a portable libtls for use as a wrapper around OpenSSL. This will then solve the LibreSSL vs OpenSSL vs different versions of OpenSSL issue for good.\nOpenSMTPD: Filters # I have resumed working on filters and this resulted in a few interesting bits.\ntimestamps in filter events # Until now, filter events reported the time at which an event was generated using a time_t unix timestamp accounting seconds since epoch.\nI had already converted reporting events to use a timeval for sub-second precision, so I also converted filter events to use a timeval too.\nfilter strderr mapped to log_warn() # I okayed a diff from martijn@ to map the stderr file-descriptor in filters to an endpoint in smtpd which logs all input with log_warn().\nThis means that a filter calling warnx() or a shell script doing an echo bleh \u0026gt;\u0026amp;2 will be logged in maillog, making it simpler to see output during development and in case of issues in production.\nfilter-eventlog # I have committed to github my first native filter, filter-eventlog which is a filter written in C that reports ALL smtp-in events to an append-only file, and creating a new file every day to hold the events of the day.\nThis is useful for filter developers as it allows them to check what events are generated by the daemon and which ones they should listen to in their filters.\nWith the lines below in your config:\n[...] filter mylogs proc-exec \u0026#34;/usr/libexec/filter-eventlog /tmp/logs\u0026#34; listen on socket filter mylogs [...] Sending a mail will result in the following being written to files within /tmp/logs:\nreport|1|1561888072.457091|smtp-in|link-connect|d812eceafa0ae39e|laptop.home|pass|local:0|local:0 report|1|1561888072.457781|smtp-in|filter-response|d812eceafa0ae39e|connected|proceed report|1|1561888072.457800|smtp-in|protocol-server|d812eceafa0ae39e|220 laptop.home ESMTP OpenSMTPD report|1|1561888072.458290|smtp-in|protocol-client|d812eceafa0ae39e|EHLO localhost report|1|1561888072.458773|smtp-in|filter-response|d812eceafa0ae39e|ehlo|proceed report|1|1561888072.458778|smtp-in|link-identify|d812eceafa0ae39e|localhost report|1|1561888072.458784|smtp-in|protocol-server|d812eceafa0ae39e|250-laptop.home Hello localhost [local], pleased to meet you report|1|1561888072.458787|smtp-in|protocol-server|d812eceafa0ae39e|250-8BITMIME report|1|1561888072.458790|smtp-in|protocol-server|d812eceafa0ae39e|250-ENHANCEDSTATUSCODES report|1|1561888072.458794|smtp-in|protocol-server|d812eceafa0ae39e|250-SIZE 36700160 report|1|1561888072.458797|smtp-in|protocol-server|d812eceafa0ae39e|250-DSN report|1|1561888072.458800|smtp-in|protocol-server|d812eceafa0ae39e|250 HELP report|1|1561888072.459268|smtp-in|protocol-client|d812eceafa0ae39e|MAIL FROM:\u0026lt;gilles@laptop.home\u0026gt; report|1|1561888072.459718|smtp-in|filter-response|d812eceafa0ae39e|mail-from|proceed report|1|1561888072.460764|smtp-in|tx-begin|d812eceafa0ae39e|754cbaa4 report|1|1561888072.460770|smtp-in|tx-mail|d812eceafa0ae39e|754cbaa4|\u0026lt;gilles@laptop.home\u0026gt; |ok report|1|1561888072.460774|smtp-in|protocol-server|d812eceafa0ae39e|250 2.0.0: Ok report|1|1561888072.460995|smtp-in|protocol-client|d812eceafa0ae39e|RCPT TO:\u0026lt;gilles@laptop.home\u0026gt; report|1|1561888072.461131|smtp-in|filter-response|d812eceafa0ae39e|rcpt-to|proceed report|1|1561888072.462172|smtp-in|tx-envelope|d812eceafa0ae39e|754cbaa4|754cbaa41a1afada report|1|1561888072.462180|smtp-in|tx-rcpt|d812eceafa0ae39e|754cbaa4|\u0026lt;gilles@laptop.home\u0026gt; |ok report|1|1561888072.462184|smtp-in|protocol-server|d812eceafa0ae39e|250 2.1.5 Destination address valid: Recipient ok report|1|1561888072.462496|smtp-in|protocol-client|d812eceafa0ae39e|DATA report|1|1561888072.462638|smtp-in|filter-response|d812eceafa0ae39e|data|proceed report|1|1561888072.463142|smtp-in|tx-data|d812eceafa0ae39e|754cbaa4|ok report|1|1561888072.463147|smtp-in|protocol-server|d812eceafa0ae39e|354 Enter mail, end with \u0026#34;.\u0026#34; on a line by itself report|1|1561888072.463866|smtp-in|protocol-client|d812eceafa0ae39e|. report|1|1561888072.463997|smtp-in|filter-response|d812eceafa0ae39e|commit|proceed report|1|1561888072.464034|smtp-in|tx-commit|d812eceafa0ae39e|754cbaa4|477 report|1|1561888072.465786|smtp-in|protocol-server|d812eceafa0ae39e|250 2.0.0: 754cbaa4 Message accepted for delivery report|1|1561888072.466065|smtp-in|protocol-client|d812eceafa0ae39e|QUIT report|1|1561888072.466294|smtp-in|filter-response|d812eceafa0ae39e|quit|proceed report|1|1561888072.466300|smtp-in|protocol-server|d812eceafa0ae39e|221 2.0.0: Bye report|1|1561888072.466481|smtp-in|link-disconnect|d812eceafa0ae39e libopensmtpd # The filters protocol is a very simple line-based protocol that I already explained in previous blog posts, but while writing filters in various scripting languages is simple, writing a filter in C was still a bit tedious.\nI started writing a libopensmtpd to provide a straightforward method for writing native C filters. It is not ready yet for publishing but it is already functional and works enough to trigger all hooks.\nAll it takes is:\n$ cc -o /tmp/filter-debug debug.c -lopensmtpd $ To make the following filter usable:\n/* * Copyright (c) 2019 Gilles Chehade \u0026lt;gilles@poolp.org\u0026gt; * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED \u0026#34;AS IS\u0026#34; AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ #include \u0026lt;err.h\u0026gt; #include \u0026lt;inttypes.h\u0026gt; #include \u0026lt;stdlib.h\u0026gt; #include \u0026#34;opensmtpd.h\u0026#34; extern char *__progname; /* reporting */ static void report_link_connect(opensmtpd_ctx_t *ctx, const char *rdns, const char *fcrdns, const char *src, const char *dest) { warnx(\u0026#34;report_link_connect: rdns=%s, fcrdns=%s, src=%s, dest=%s\u0026#34;, rdns, fcrdns, src, dest); } static void report_link_disconnect(opensmtpd_ctx_t *ctx) { warnx(\u0026#34;report_link_disconnect\u0026#34;); } static void report_link_identify(opensmtpd_ctx_t *ctx, const char *helo_name) { warnx(\u0026#34;report_link_identify: helo_name=%s\u0026#34;, helo_name); } static void report_link_tls(opensmtpd_ctx_t *ctx, const char *tls_line) { warnx(\u0026#34;report_link_tls: tls_line=%s\u0026#34;, tls_line); } static void report_tx_begin(opensmtpd_ctx_t *ctx, uint32_t msgid) { warnx(\u0026#34;report_tx_begin: msgid=%08x\u0026#34;, msgid); } static void report_tx_mail(opensmtpd_ctx_t *ctx, uint32_t msgid, const char *mail_from, const char *result) { warnx(\u0026#34;report_tx_mail: msgid=%08x, mail_from=%s, result=%s\u0026#34;, msgid, mail_from, result); } static void report_tx_rcpt(opensmtpd_ctx_t *ctx, uint32_t msgid, const char *rcpt_to, const char *result) { warnx(\u0026#34;report_tx_rcpt: msgid=%08x, mail_from=%s, result=%s\u0026#34;, msgid, rcpt_to, result); } static void report_tx_envelope(opensmtpd_ctx_t *ctx, uint32_t msgid, uint64_t evpid) { warnx(\u0026#34;report_tx_envelope: msgid=%08x, evpid=%016\u0026#34;PRIx64\u0026#34;\u0026#34;, msgid, evpid); } static void report_tx_commit(opensmtpd_ctx_t *ctx, uint32_t msgid) { warnx(\u0026#34;report_tx_commit: msgid=%08x\u0026#34;, msgid); } static void report_tx_data(opensmtpd_ctx_t *ctx, uint32_t msgid, size_t msg_size) { warnx(\u0026#34;report_tx_data: msgid=%08x, msg_size=%zd\u0026#34;, msgid, msg_size); } static void report_tx_rollback(opensmtpd_ctx_t *ctx, uint32_t msgid) { warnx(\u0026#34;report_tx_rollback: msgid=%08x\u0026#34;, msgid); } static void report_protocol_client(opensmtpd_ctx_t *ctx, const char *line) { warnx(\u0026#34;report_protocol_client: line=%s\u0026#34;, line); } static void report_protocol_server(opensmtpd_ctx_t *ctx, const char *line) { warnx(\u0026#34;report_protocol_server: line=%s\u0026#34;, line); } static void report_filter_response(opensmtpd_ctx_t *ctx, const char *phase, const char *response) { warnx(\u0026#34;report_filter_response: phase=%s, response=%s\u0026#34;, phase, response); } static void report_timeout(opensmtpd_ctx_t *ctx) { warnx(\u0026#34;report_timeout\u0026#34;); } /* filtering */ static void filter_connect(opensmtpd_ctx_t *ctx, const char *rdns, const char *arg) { warnx(\u0026#34;filter_connect: rdns=%s, arg=%s\u0026#34;, rdns, arg); proceed(ctx); } static void filter_helo(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_helo: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_ehlo(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_ehlo: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_starttls(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_starttls: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_auth(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_auth: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_mail_from(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_mail_from: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_rcpt_to(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_rcpt_to: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_data(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_data: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_data_line(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_data_line: arg=%s\u0026#34;, arg); dataline(ctx, arg); } static void filter_rset(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_rset: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_quit(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_quit: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_noop(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_noop: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_help(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_help: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_wiz(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_wiz: arg=%s\u0026#34;, arg); proceed(ctx); } static void filter_commit(opensmtpd_ctx_t *ctx, const char *arg) { warnx(\u0026#34;filter_commit: arg=%s\u0026#34;, arg); proceed(ctx); } int main(int argc, char *argv[]) { opensmtpd_ctx_t *ctx; if (argc != 1) errx(1, \u0026#34;usage: %s\u0026#34;, __progname); smtpd_init(\u0026amp;ctx); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;link-connect\u0026#34;, report_link_connect); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;link-disconnect\u0026#34;, report_link_disconnect); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;link-identify\u0026#34;, report_link_identify); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;link-tls\u0026#34;, report_link_tls); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-begin\u0026#34;, report_tx_begin); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-mail\u0026#34;, report_tx_mail); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-rcpt\u0026#34;, report_tx_rcpt); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-envelope\u0026#34;, report_tx_envelope); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-data\u0026#34;, report_tx_data); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-commit\u0026#34;, report_tx_commit); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;tx-rollback\u0026#34;, report_tx_rollback); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;protocol-client\u0026#34;, report_protocol_client); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;protocol-server\u0026#34;, report_protocol_server); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;filter-response\u0026#34;, report_filter_response); smtpd_on_report(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;timeout\u0026#34;, report_timeout); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;connect\u0026#34;, filter_connect); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;helo\u0026#34;, filter_helo); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;ehlo\u0026#34;, filter_ehlo); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;starttls\u0026#34;, filter_starttls); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;auth\u0026#34;, filter_auth); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;mail-from\u0026#34;, filter_mail_from); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;rcpt-to\u0026#34;, filter_rcpt_to); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;data\u0026#34;, filter_data); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;rset\u0026#34;, filter_rset); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;quit\u0026#34;, filter_quit); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;noop\u0026#34;, filter_noop); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;help\u0026#34;, filter_help); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;wiz\u0026#34;, filter_wiz); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;commit\u0026#34;, filter_commit); smtpd_on_filter(ctx, \u0026#34;smtp-in\u0026#34;, \u0026#34;data-line\u0026#34;, filter_data_line); if (! smtpd_event_loop(ctx)) return 1; return 0; } and register all hooks:\nlaptop$ /tmp/filter-debug register|report|smtp-in|filter-response register|report|smtp-in|link-connect register|report|smtp-in|link-disconnect register|report|smtp-in|link-identify register|report|smtp-in|link-tls register|report|smtp-in|protocol-client register|report|smtp-in|protocol-server register|report|smtp-in|timeout register|report|smtp-in|tx-begin register|report|smtp-in|tx-commit register|report|smtp-in|tx-data register|report|smtp-in|tx-envelope register|report|smtp-in|tx-mail register|report|smtp-in|tx-rcpt register|report|smtp-in|tx-rollback register|filter|smtp-in|auth register|filter|smtp-in|commit register|filter|smtp-in|connect register|filter|smtp-in|data register|filter|smtp-in|data-line register|filter|smtp-in|ehlo register|filter|smtp-in|helo register|filter|smtp-in|help register|filter|smtp-in|mail-from register|filter|smtp-in|noop register|filter|smtp-in|quit register|filter|smtp-in|rcpt-to register|filter|smtp-in|rset register|filter|smtp-in|starttls register|filter|smtp-in|wiz register|ready ^C laptop$ In OpenSMTPD, with the following config:\n[...] filter foobar proc-exec \u0026#34;/tmp/filter-debug\u0026#34; listen on socket filter foobar it will result in the following output:\n[...] 86c0820a13ffee17 smtp connected address=local host=laptop.home \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_link_connect: rdns=laptop.home, fcrdns=pass, src=local:0, dest=local:0 \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_connect: rdns=laptop.home, arg=local \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=connected, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=220 laptop.home ESMTP OpenSMTPD \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=EHLO localhost \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_ehlo: arg=localhost \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=ehlo, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_link_identify: helo_name=localhost \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-laptop.home Hello localhost [local], pleased to meet you \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-8BITMIME \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-ENHANCEDSTATUSCODES \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-SIZE 36700160 \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250-DSN \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250 HELP \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=MAIL FROM:\u0026lt;gilles@laptop.home\u0026gt; \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_mail_from: arg=gilles@laptop.home \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=mail-from, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_begin: msgid=2b8bbe3b \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_mail: msgid=2b8bbe3b, mail_from=\u0026lt;gilles@laptop.home\u0026gt;, result=ok \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250 2.0.0: Ok \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=RCPT TO:\u0026lt;gilles@laptop.home\u0026gt; \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_rcpt_to: arg=gilles@laptop.home \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=rcpt-to, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_envelope: msgid=2b8bbe3b, evpid=2b8bbe3b60567a11 \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_rcpt: msgid=2b8bbe3b, mail_from=\u0026lt;gilles@laptop.home\u0026gt;, result=ok \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250 2.1.5 Destination address valid: Recipient ok \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=DATA \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data: arg= \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=data, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=Received: from localhost (laptop.home [local]) \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg= by laptop.home (OpenSMTPD) with ESMTPA id 2b8bbe3b \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg= for \u0026lt;gilles@laptop.home\u0026gt;; \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg= Sun, 30 Jun 2019 12:01:55 +0200 (CEST) \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_data: msgid=2b8bbe3b, msg_size=10034375646789 \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=354 Enter mail, end with \u0026#34;.\u0026#34; on a line by itself \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=From: \u0026lt;gilles@laptop.home\u0026gt; \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=Date: Sun, 30 Jun 2019 12:01:55 +0200 (CEST) \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=To: gilles \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg= \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=test \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_data_line: arg=. \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=. \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_commit: arg= \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_filter_response: phase=commit, response=proceed \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_tx_commit: msgid=2b8bbe3b \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_server: line=250 2.0.0: 2b8bbe3b Message accepted for delivery \u0026lt;dynproc:00000001\u0026gt;: filter-debug: report_protocol_client: line=QUIT \u0026lt;dynproc:00000001\u0026gt;: filter-debug: filter_quit: arg= [...] filter-rspamd in progress # I had written a filter-rspamd for OpenSMTPD in python already, however it was a proof-of-concept more than a real filter and I didn\u0026rsquo;t make it really robust.\nWith libopensmtpd at hands, I decided to write a native C filter-rspamd that I would officially maintain, one that would be robust enough to be used in production.\nI had most of the logic done in half an hour, unfortunately I ran out of time for this \u0026ldquo;free\u0026rdquo; week and didn\u0026rsquo;t complete the filter.\nBasically as it is right now, the filter will build everything needed for the call to rspamd, but I didn\u0026rsquo;t write the http request.\nI will surely finish that during my next \u0026ldquo;free\u0026rdquo; week.\nWhat next ? # Well, see you in a month for the July report.\nHopefully, by then I will have finished my filter-rspamd, I will have released an initial version of FION, and I will have released an initial version of libopensmtpd.\nWhen I\u0026rsquo;m done with FION and have decided what to to with BPG, I will have a look at some of my other private repositories to see what can be released :-)\nIf you like my work, support me on patreon !\n","date":"30 June 2019","permalink":"/posts/2019-06-30/june-2019-report-fion-bpg-and-smtpd/","section":"Posts","summary":"TL;DR: - started working on FION, a static tile window manager - revived BPG, a PGP parser - converted OpenSMTPD to libtls - wrote a library to make writing of native C OpenSMTPD filters easy - started writing a filter-rspamd Thanks to my patrons !","title":"June 2019 report: fion, bpg and smtpd"},{"content":" TL;DR: In this post I explain crudely how ca.c works and changes to OpenSMTPD related to ca.c I wrote an ECDSA privsep crypto engine I did some EEG work too This is the first report # I will now switch to a monthly report of my tech activities on this blog, and this is the first post in that new format.\nThe focus will be put on interesting topics, not necessarily every single commit and bug fix I do (ie: won\u0026rsquo;t mention LMTP bug fix here), this depends on the amount of slacking that ocurred during the month and how much I want to cover it up :-)\nDon\u0026rsquo;t forget to tip me on the link above if you like reading my reports, donations cover my random geek expenses and keep me motivated to write about what I do.\nAbout OpenSMTPD\u0026rsquo;s ca.c file # A few years ago, Neel Mehta blessed us with the Heartbleed bug in the OpenSSL library. A bug which ultimately resulted in the disclosure of sensitive data, including cryptographic private keys, that resided in the memory of OpenSSL\u0026rsquo;s custom allocator.\nA good approach at mitigating the issue was taken by the LibreSSL folks. Long story short, a custom allocator is a bad idea because it potentially defeats any security measure provided by the standard allocator. In OpenBSD, the multiple security mechanisms in place to detect memory corruption and invalid access were bypassed by the use of the OpenSSL custom allocator, and the first steps taken by LibreSSL were to ensure that functions such as OpenSSL_malloc and OpenSSL_free where REALLY calling malloc and free, and not something else trying to be \u0026ldquo;smart\u0026rdquo;.\nAnother approach was taken by reyk@ and consisted in assuming that private keys in a network facing process was too dangerous. To mitigate the risk, a good option would be to isolate the crypto operations that required access to the private key in a separate process, one that would be fairly isolated from untrusted users. Here comes ca.c, a process operating privileges separated crypto operations.\nHere\u0026rsquo;s how it works:\nUpon startup, OpenSMTPD declares a custom RSA engine which provides its own primitives for crypto operations. It then registers that custom engine as the default RSA engine, meaning that any use of the OpenSSL API to perform RSA operations will use the new primitives rather than the default ones.\nThe primitives provided by the RSA engine are actually of two kind: either they rely on the private key in which case they will perform an IPC to the CA (crypti agent) process which will do the operation itself through the default primitive and provide the result, or they don\u0026rsquo;t rely on the private key in which case the primitive simply falls back to the default one.\n[...] static int rsae_pub_dec(int flen,const unsigned char *from, unsigned char *to, RSA *rsa, int padding) { log_debug(\u0026#34;debug: %s: %s\u0026#34;, proc_name(smtpd_process), __func__); return (rsa_default-\u0026gt;rsa_pub_dec(flen, from, to, rsa, padding)); } static int rsae_priv_enc(int flen, const unsigned char *from, unsigned char *to, RSA *rsa, int padding) { log_debug(\u0026#34;debug: %s: %s\u0026#34;, proc_name(smtpd_process), __func__); if (RSA_get_ex_data(rsa, 0) != NULL) { return (rsae_send_imsg(flen, from, to, rsa, padding, IMSG_CA_PRIVENC)); } return (rsa_default-\u0026gt;rsa_priv_enc(flen, from, to, rsa, padding)); } [....] The idea is that if the network facing process is compromised, then no matter how bad the memory corruption and/or info leak is, the private keys are not available to that process.\nFix OpenSMTPD ca.c build with OpenSSL # That private key privsep relies on using the OpenSSL crypto engine API.\nEver since OpenSSL switched to 1.1.x, the crypto engine API changed because structures that were public and could be dereferenced would no longer be public. They would require a set of getters and setters accessors to make any use of the structure. The sad thing is that this is not the case with LibreSSL, the structures are still public and accessors are not available.\nThis results in the same code not being able to be built for both, either you use the opaque structures and build is broken on LibreSSL, or you use the public structures and build is broken on OpenSSL.\nI rewrote ca.c to pretend that the structure was opaque, not performing any direct deref, and added the missing accessors within an ifdef. This is not ideal but it reduces the delta between our native branch and portable, and as soon as accessors are available in LibreSSL we can simply remove the ifdef-ed part.\nI submitted the diff to my fellow OpenBSD hackers, it\u0026rsquo;s pending review at this point, should be committed soon.\nTurn most of OpenSMTPD\u0026rsquo;s SSL structures opaque # A lot of other structures that are also opaque did have accessors in LibreSSL, notably the EVP interface which we use for encrypted queue support.\nThe code used to rely on automatic variables and dereferences, I converted the code to heap allocated structures using the opaque structures interface.\nDiff was submitted, okayed and committed.\nAdd support for opaque RSA_METHOD engine in LibreSSL # As mentioned earlier, code in ca.c was broken due to the fact that LibreSSL considers RSA_METHOD as a public structure that can be dereferenced, whereas OpenSSL considers it as an opaque structure that requires accessors.\nEven considering that RSA_METHOD is a public structure, this doesn\u0026rsquo;t prevent us from providing accessors to allow a common idiom between LibreSSL and OpenSSL.\nI submitted a diff to the LibreSSL folks to bring the missing accessors, which will allow me to write code that builds for both LibreSSL (my target) and OpenSSL. This will not be an OpenSMTPD only thing because any OpenBSD daemon that relies or will rely on ca.c and wants to be portable will face the same issue.\nThe diff has been reviewed and okayd, it will require a LibreSSL (libcrypto, libssl and libtls) minor crank so I\u0026rsquo;m waiting for the appropriate time to do it. I\u0026rsquo;m also waiting for an ok from ports to make sure it doesn\u0026rsquo;t break ports as adding new symbols may have some configure scripts uncover other symbols we don\u0026rsquo;t have.\nWrite an ECDSA privsep engine for OpenSMTPD # The crypto privsep engine only supported RSA and this means that starting OpenSMTPD with an ECDSA results in a fatal():\ndebug: init private ssl-tree debug: SSL library error: ssl_ctx_create: error:06FFF07F:digital envelope routines:CRYPTO_internal:expecting an rsa key debug: SSL library error: ssl_ctx_create: error:140AE006:SSL routines:func(174):EVP lib pony express: ssl_ctx_create: could not fake private key: Invalid argument More and more people have complained that we didn\u0026rsquo;t support ECDSA certificates, but it was not a trivial one-liner as adding support meant writing a custom ECDSA crypto engine akin to what reyk@ did for RSA. It took me time to find the motivation, but ultimately I came around to implement such an engine this week.\ndebug: rsa_engine_init: using RSA privsep engine debug: ecdsa_engine_init: using ECDSA privsep engine The details of how I implemented the ECDSA crypto engine are not that interesting, they follow the same pattern as the RSA crypto engine:\nECDSA_SIG * ecdsae_do_sign(const unsigned char *dgst, int dgst_len, const BIGNUM *inv, const BIGNUM *rp, EC_KEY *eckey) { log_debug(\u0026#34;debug: %s: %s\u0026#34;, proc_name(smtpd_process), __func__); if (ECDSA_get_ex_data(eckey, 0) != NULL) return (ecdsae_send_enc_imsg(dgst, dgst_len, inv, rp, eckey)); return (ecdsa_default-\u0026gt;ecdsa_do_sign(dgst, dgst_len, inv, rp, eckey)); } The real challenge was to teach ca.c that multiple crypto engines could coexist, figuring out the primitives for ECDSA and doing the proper serialization in IPC.\nOn my laptop, OpenSMTPD can now load and work with both RSA and ECDSA certificates, accepting SMTP sessions on listeners presenting either one:\n9362d76af3ab2882 smtp connected address=127.0.0.1 host=localhost debug: looking up pki \u0026#34;localhost\u0026#34; debug: session_start_ssl: switching to SSL debug: pony: rsae_priv_enc 9362d76af3ab2882 smtp tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256 8c25253d498c0990 smtp connected address=127.0.0.1 host=localhost debug: looking up pki \u0026#34;localhost\u0026#34; debug: session_start_ssl: switching to SSL debug: pony: ecdsae_do_sign 8c25253d498c0990 smtp tls ciphers=TLSv1.2:ECDHE-ECDSA-AES256-GCM-SHA384:256 I have submitted the diff today, pending review, it should be committed shortly.\nTroubleshoot eegreader and write initial support for an eegviewer # A while ago, I got interested in reading electroencephalogram data to detect interesting patterns during self-hypnosis sessions.\nI got myself an OpenEEG device and realized there were no simple utilities to extract data from it, most were C++ bloated software that didn\u0026rsquo;t build on my laptop and required dependencies.\nI found out that the OpenEEG protocol was very easy to implement and wrote eegreader a utility to output an OpenEEG data stream to stdout, displaying sequence, version, the six channels and buttons values.\nThere seemed to be a bug because packet loss kept triggering my recovery code, skipping bits until finding the alignment for a new packet (eg: packets 53, 56, 58, 5a, 5c are missing in this sample):\n[...] 4f|2|512|511|505|499|493|487|7 50|2|532|511|505|500|493|487|7 51|2|513|65281|63745|62465|60673|59143|5 53|2|497|511|505|499|493|487|7 54|2|508|511|505|499|493|487|7 55|2|513|65281|63745|62465|60929|59143|5 57|2|514|511|505|500|494|487|5 59|2|506|511|505|500|494|487|5 5b|2|535|511|505|500|494|487|5 5d|2|499|511|505|500|493|487|5 [...] I investigated and realized that I didn\u0026rsquo;t have the issue on some other OSes, it seems to be an OpenBSD USB bug. Furthermore, with the latest OpenBSD snapshot, I seem to be hitting kernel panics in USB so some investigation is definitely needed. I pinged some of the USB gurus, we\u0026rsquo;ll see how I can help.\nI still managed to do something useful as I started the writing of eegviewer, which is a libxcb graphical interface displaying the eegreader input as a real-time graph. The idea is to simply pipe eegreader into eegviewer to obtain the graphs.\nThe eegreader utility has already been published a while ago on Github under the ISC license. I\u0026rsquo;ll publish eegviewer under the same license when I\u0026rsquo;m happy with the code, libxcb not being something I\u0026rsquo;m familiar with.\nFinally, I will very likely write an ISC-licensed libeeg which will provide an event-driven interface to OpenEEG data streams, allowing callbacks to be triggered upon certain events occuring on certain channels. I haven\u0026rsquo;t started that work yet but it is trivial so I might release that soon.\nOculus Quest # Thanks to my geek donations, I got myself an Oculus Quest device :-)\nI\u0026rsquo;m not a gamer but I\u0026rsquo;m genuinely curious what can be done with it, when time allows I\u0026rsquo;ll start digging the SDK.\nWhat next ? # See you in a month for the June report, very likely focusing around OpenSMTPD filters !\n","date":"2 June 2019","permalink":"/posts/2019-06-02/may-2019-report/","section":"Posts","summary":"TL;DR: In this post I explain crudely how ca.c works and changes to OpenSMTPD related to ca.c I wrote an ECDSA privsep crypto engine I did some EEG work too This is the first report # I will now switch to a monthly report of my tech activities on this blog, and this is the first post in that new format.","title":"May 2019 report"},{"content":" TL;DR: Way too many things happened in a six months timeframe. This post won't need a TL;DR as I'll keep it short. Generalized anxiety disorder and alexithymia. # Late 2018, I\u0026rsquo;ve been diagnosed with generalized anxiety disorder and alexithymia a couple weeks apart.\nThe anxiety disorder didn\u0026rsquo;t really come as a surprise as I know very well the traumatic experience that led to the disorder. It\u0026rsquo;ll still take a lot of hours of hypnosis to continue cleaning up trauma after trauma, but taking care of it and releasing some of the traumas has improved considerably my anxiety.\nI\u0026rsquo;ve also tried different approaches out of curiosity, including consumption of increasing doses of CBD in various forms, but didn\u0026rsquo;t observe noticeable improvements no matter if smoked, vaped, eaten, tinctured, \u0026hellip; it does help sleep earlier if I consume a large dose of CBD but only hypnosis relieves anxiety effectively for me.\nThe alexithymia diagnosis crushed me for a while. I didn\u0026rsquo;t know what it was but as the psychologist explained the disorder, all of the implications throughout the years hit me in the face very harshly.\nAlexithymia is an inability to identify and describe emotions in the self. I learnt at 38 that I\u0026rsquo;m mostly cut from emotions and that when people say \u0026ldquo;I feel happy\u0026rdquo; or \u0026ldquo;I feel sad\u0026rdquo; they actually feel it, whereas I will say that I feel them when I \u0026ldquo;know\u0026rdquo; I should feel them, not because I actually feel them.\nI won\u0026rsquo;t enter much more details as it still makes me uncomfortable at this point, but one of the side-effects of alexithymia is that it makes the anxiety disorder considerably worse: I cannot see attacks coming and can\u0026rsquo;t use any techniques to prevent them from unfolding. In most situations I will not even realize that I\u0026rsquo;m having an anxiety attack but will still suffer from all of its symptoms without being able to make a connection, nor understand what is happening.\nI wish I had read about this earlier and put some sense into it earlier, which is why I\u0026rsquo;m taking a few paragraphs to explain and will answer any question in the comments if you have some. I will probably write again about it in the future but I don\u0026rsquo;t think GAD or alexithymia characterize me, they haven\u0026rsquo;t prevented me from having a career or hobbies though they made it considerably harder, so I will not give them a bigger place than they already have in my every day life, it won\u0026rsquo;t be a recurrent topic.\nI\u0026rsquo;d say it\u0026rsquo;s been an emotional rollercoaster but now you know I don\u0026rsquo;t really grasp what it means :-)\nOther misc. personal stuff # In just a few months, I learnt my half was pregnant and we\u0026rsquo;re expecting a little guy in October \u0026lt;3, we moved to our own place that we\u0026rsquo;ve been waiting for two years to be built, my cat died after sharing sixteen years with me, I fully completed my final hypnotherapist training putting an end to years of training, and mostly finished my third year of studying occupational clinical psychology.\nI have also changed role at work and requested to switch to a 75% part-time job, spending three weeks at work for one week off to work on my projects, both open-source and hypno/psycho related.\nLife has been busy and kept me away from writing here.\nWhat next ? # Now that I will be able to spend one week off each month, I\u0026rsquo;ll write a monthly report of my activities on this blog, this will be an easier rythm than writing whenever I do something.\nThis post was not the May monthly report, which I\u0026rsquo;ll write right after and publish tonight, I just wanted to get the non-technical stuff out of the way.\nCheers, Gilles\n","date":"2 June 2019","permalink":"/posts/2019-06-02/happy-new-year-2019-a-personal-post/","section":"Posts","summary":"TL;DR: Way too many things happened in a six months timeframe. This post won't need a TL;DR as I'll keep it short. Generalized anxiety disorder and alexithymia.","title":"happy new year 2019, a personal post"},{"content":" TL;DR: regex table lookups were introduced for builtin filters. After a few weeks of working solely on filters, I wanted to work on something else. Using the same mechanism, all match criterias using tables can support regex. K_REGEX lookups # The table mechanism is used within OpenSMTPD to perform all kinds of lookups.\nRecently, while working on builtin filters, I introduced the K_REGEX lookup type allowing tables to serve regex(3) patterns.\nWhen a K_REGEX lookup is performed, the table API will use the lookup key as a string and iterate over entries in the table, compiling them with regcomp(3) and performing a simple regcomp(3) against the key.\nregex use in match criterias # In the new smtpd.conf, accept rules where replaced with match rules.\nThe match rules consist of matching criterias which describe how an envelope should look like to be accepted for a specific action. Among these criterias, you\u0026rsquo;ll find for example from src, for domain, mail-from or even rcpt-to, which can either take a table, or a string literal which \u0026hellip; is actually an inlined on-element table.\nThe new feature introduces a regex keyword that can be used to instruct OpenSMTPD that the table will use regex. This basically allows the following:\ntable regex_domains file:/etc/mail/list-of-regex.txt\u0026#34; match for domain regex \u0026#34;^.*\\.example.com$\u0026#34; action foobar match for domain regex \u0026lt;regex_domains\u0026gt; action foobar except that it also allows it for pretty much every criteria that can be used on a match rule and expect a parameter.\nWhat next ? # This was just committed a few minutes ago in -current and the feature is essentially complete, so there\u0026rsquo;s not much in that front.\nYou can already play with that feature which has been documented in the smtpd.conf(5) man page.\nSo \u0026hellip; NOW, I\u0026rsquo;m taking a break until next week because I did everything I wanted to do for this year ;-)\n","date":"21 December 2018","permalink":"/posts/2018-12-21/opensmtpd-now-supports-regex-in-match-rules/","section":"Posts","summary":"TL;DR: regex table lookups were introduced for builtin filters. After a few weeks of working solely on filters, I wanted to work on something else.","title":"OpenSMTPD now supports regex in match rules"},{"content":" TL;DR: Not this time, pal/gal, I took hours writing this post, you'll take a few minutes reading it all. Oh, and merry X-mas :-* A bit of short-sighted history # The filtering feature has been introduced only recently in OpenSMTPD, first presented on this blog a month ago.\nI had a working proof-of-concept running on my laptop and my plan was to start bringing the code to the OpenBSD tree, small chunks by small chunks, through a serie of diffs.\nI won\u0026rsquo;t cover the serie in details because it\u0026rsquo;s irrelevant, but the point is that we went from scratch to a working filter implementation in a few days, one that we knew would work but that had only ran on a one-interface laptop and was not polished.\nIn the last few weeks, as various proof-of-concepts for useful filters were written, the protocol and API improved and while there\u0026rsquo;s many more improvements to come before the next major release, it is safe to say that the API now allows full-featured filters.\nHow does filtering work in OpenSMTPD ? # During the lifetime of an SMTP session, whenever a client sends a command, the server checks if the command should be accepted or rejected then replies to the client.\nFirst it checks if the command is valid in terms of syntax and context (for example, AMIL FORM is an invalid command, MAIL FROM: \u0026gt;\u0026lt; is a valid command with an invalid parameter, and you can\u0026rsquo;t send RCPT TO before MAIL FROM). Then, it checks if the command is actually allowed by the ruleset. Sometimes, it may reject a valid command for issues related to lack of ressources, but let\u0026rsquo;s keep this corner case out.\nHow does filtering work then ?\nWell, right between the two types of checks, after the server has checked that the input was valid in terms of syntax and context, it pauses processing and requests the lookup process to take a decision based on the input. The lookup process passes that input to the filters sequentially and if any decides to take an action, the server will enforce that action instead of resuming its usual check.\nWhen it comes to the DATA part, things are trickier because the server doesn\u0026rsquo;t answer to the individual lines but to the whole message. Also, it is expected that the filters can actually make changes to the content of the message. As such, there are two kinds of filtering in OpenSMTPD: protocol filtering and data filtering. The former provides a simple query/reply mechanism to alter the decision taking in the SMTP engine, the latter provides a simple transformation mechanism consuming an input and producing an output.\nLet\u0026rsquo;s get a bit more technical # As mentionned in the blog post linked in my introduction, there are two mechanisms involved: reporting and filtering.\nEach of these have their own specific hooks, triggering at specific phases of the SMTP session, and allowing the SMTP engine to generate reporting events or filter queries at appropriate times.\nThe basic idea when I designed the filter API was that we don\u0026rsquo;t want complexity in these hooks. We do want filters to have as much flexibility as possible, a filter should be able to reject a session after the data phase because the source address used during the connect phase didn\u0026rsquo;t have a proper reverse DNS configured, but this should not come at the cost of having hooks with tons of parameters or huge structures encompassing each and everything.\nTo achieve this, the reporting and filtering mechanisms were designed with the following philosophy in mind:\nThe reporting mechanism should generate enough events that it is possible to replicate the state of an SMTP session, but these events should only contain the information related to the event itself. A filter that wants to reject a session after the data phase because of the reverse DNS doesn\u0026rsquo;t need to receive the reverse DNS during the data phase, instead it needs to receive it during the connect phase and keep it in a local state to check it at the data phase. This allowed making the reporting mechanism very simple and not limiting since a filter can essentially hook all reporting phases and have the exact same state as the SMTP session itself.\nThe filtering mechanism in turn should only take decisions based on the parameters of the phase itself \u0026hellip; and any local state. A filter registering a hook for the MAIL FROM phase will only receive the address of the sender. If it wants to take decisions based on anything more than that, then it needs to rely on a local state gathered from the reporting mechanism.\nThe data filtering is trickier because we want to be able to add, change or suppress lines. As a result, we can\u0026rsquo;t rely on the number of bytes we sent and the number of bytes we received, just like we can\u0026rsquo;t rely on a lines count. Furthermore, there can be multiple filters processing the data in sequence, so the first filter may receive 1 line, generate 4 headers, the second filter remove some of these headers, the third filter append a ton of data, etc\u0026hellip;\nThis is where eric@ had a very clever idea which is to consider the DATA part as a stream. The SMTP protocol ends DATA with a single . on a line by itself, so the lines are streamed to the filters which read them up to the . and output a new stream terminate by a . itself. Doing this allows the SMTP engine to stream to filters the DATA up to the client-submitted ., then read back a stream from the filters up to the filter-submitted ., not having to care if the stream was altered.\nSounds like a lot of details ?\nYou need not worry, in practice writing a filter is trivial.\nA case study: filter-rspamd # I wrote many filters to experiment and refine the API, but here\u0026rsquo;s an interesting one.\nLet\u0026rsquo;s have a look at how this whole reporting and filtering mechanisms come into play. I wrote the filter in python, it is fairly small but I will not copy paste the whole code because I wrote it as a PoC, and I\u0026rsquo;m ashamed of the quality.\nFirst of all, let\u0026rsquo;s define what we want to do with the filter.\nWe want a filter that will pass a DATA part to the rspamd daemon, with several session informations gathered from the connection up to the message itself so it can take a decision, and then alter the message to insert headers and possibly reject temporarily or permanently the message. Since I was in a good mood when I wrote the PoC, I also added DKIM signing but it\u0026rsquo;s out of scope ;-)\nRemember that this is a Python example, because I find it easier to understand for all, but filters can be written with any language really.\nsessions = {} class Rspamd(): def __init__(self): self.stream = smtp_in() self.stream.on_report(\u0026#39;link-connect\u0026#39;, link_connect, None) self.stream.on_report(\u0026#39;link-disconnect\u0026#39;, link_disconnect, None) self.stream.on_report(\u0026#39;link-identify\u0026#39;, link_identify, None) self.stream.on_report(\u0026#39;tx-begin\u0026#39;, tx_begin, None) self.stream.on_report(\u0026#39;tx-mail\u0026#39;, tx_mail, None) self.stream.on_report(\u0026#39;tx-rcpt\u0026#39;, tx_rcpt, None) self.stream.on_report(\u0026#39;tx-data\u0026#39;, tx_data, None) self.stream.on_report(\u0026#39;tx-commit\u0026#39;, tx_cleanup, None) self.stream.on_report(\u0026#39;tx-rollback\u0026#39;, tx_cleanup, None) self.stream.on_filter(\u0026#39;commit\u0026#39;, filter_commit, None) self.stream.on_filter(\u0026#39;data-line\u0026#39;, filter_data_line, None) def run(self): self.stream.run() Based on this excerpt alone, without looking at the actual implementation of the callback functions, here\u0026rsquo;s what you can understand:\nThe only times when we actually want to mess with a session somehow is when making changes to the data, which is handled by the data-line callback, and when we want to possibly reject temporarily or permanently a message, which is handled by the commit callback.\nAll of the on_report() calls are used solely to accumulate enough informations so the two filter hooks can work, and if we look at the implementation of one of these, you\u0026rsquo;ll realize that all it does is really store a particular bit of information in the local state for a session.\nFor example, the purpose of registering for the link-identify event is to store the helo name the client submitted in the local session state:\ndef link_identify(ctx, timestamp, session_id, args): helo = args[0] session = sessions[session_id] session.control[\u0026#39;Helo\u0026#39;] = helo So that the information can be sent to the rspamd daemon when the filter begins sending the message in the data-line callback.\nPer-listener filtering # In the first version of the filter feature I commited, filters were declared globally in the configuration, listeners would only enable/disable filtering, the filters would be applied in sequence:\nfilter smtp-in connect check-fcrdns reject \u0026#34;550 go away you punk\u0026#34; filter smtp-in connect check-rdns reject \u0026#34;550 go away you punk\u0026#34; listen on all filter\t# not filtered listen on socket\t# filtered This was a first step at plugging filters on and off, but real use cases rely on being able to plug different sets of filters on different interfaces, so that you can for example reject senders without a reverse DNS on an interface, while allowing them on the submission port where they authenticate.\nThis feature is now supported.\nFilter grammar changes # The filter declaration grammar I used to begin was easy to start playing right away, it couldn\u0026rsquo;t cover some of the use-cases I had in mind and planned for, but now that the plumbing has evolved enough we can move towards what\u0026rsquo;s going to look like the final grammar.\nFirst of all, let\u0026rsquo;s look how I would plug that filter-rspamd in my config:\nfilter rspamd proc-exec \u0026#34;/usr/local/bin/filter-rspamd\u0026#34; listen on all filter rspamd That\u0026rsquo;s all.\nI declared a filter named rspamd, instructed OpenSMTPD that it has to execute a proc filter from /usr/local/bin/filter-rspamd, and instructed the listener that all sessions handled through it should go through the rspamd filter.\nIf I didn\u0026rsquo;t want to use rspamd but a builtin filter to reject sessions without a forward-confirmed rDNS, I could have used:\nfilter fcrdns builtin connect check-fcrdns reject \u0026#34;550 go away you punk\u0026#34; listen on all filter fcrdns But what\u0026rsquo;s more interesting is the chaining of filters, which allows \u0026hellip; well, chaining filters:\nfilter fcrdns builtin connect check-fcrdns reject \u0026#34;550 go away you punk\u0026#34; filter rspamd proc-exec \u0026#34;/usr/local/bin/filter-rspamd\u0026#34; filter nazi_mode chain { fcrdns, rspamd } listen on all filter nazi_mode This, combined with the fact that the filters apply per-interface, allow for very flexible setups that could never be expressed on OpenSMTPD before.\nWhat next ? # The code and new grammar is working and committed in a branch, I intend to do some additional cleanup and code simplification before comitting to the OpenBSD tree hopefully this week.\nI will spend the remaining of the release cycle, until April, performing code cleanups, refining the reporting events and protocol, and maybe committing a few minor features (or maybe I have some other nice major features ready, who knows ;-)\nYou are HIGHLY encouraged to start playing with filters next week after my commit. There\u0026rsquo;s currently NO filter available, you can really be the first to do useful stuff for the community.\nStay tuned !\nOH AND MERRY X-MAS BECAUSE I WONT BE POSTING BEFORE THEN ;-)\n","date":"19 December 2018","permalink":"/posts/2018-12-19/more-on-opensmtpd-filters/","section":"Posts","summary":"TL;DR: Not this time, pal/gal, I took hours writing this post, you'll take a few minutes reading it all. Oh, and merry X-mas :-* A bit of short-sighted history # The filtering feature has been introduced only recently in OpenSMTPD, first presented on this blog a month ago.","title":"more on OpenSMTPD filters"},{"content":" TL;DR: I *FINALFUCKINGLY* commited proc filters support allowing full filtering in OpenSMTPD. eric@ implemented fc-rDNS lookups. fc-rDNS # fc-rDNS, or forward-confirmed reverse DNS, consists in performing a reverse DNS lookup to determine the hostname associated to an IP address\u0026hellip; then performing a DNS lookup on that hostname to check if it resolves back to the IP address.\nOn my request, eric@ implemented fc-rDNS lookups in our SMTP engine, causing OpenSMTPD to perform the double lookup upon clients connections.\nRight now, the fc-rDNS result is only passed to the reporting API but the idea is to allow using it in a builtin filter, something along the lines of filter smtp-in connect check-fcrdns disconnect \u0026quot;550 go away punk\u0026quot;.\nNote that this is particularly efficient at cutting spam, it essentially means that a spammer must control both the DNS and reverse DNS zone to pass the test, killing the bulk of zombified spam proxies living on infected home computers.\nThere is still some refining to do and the plugging of a builtin-filter, but the code is here and -current users will be able to test it soon\ntx-mail and tx-rcpt events # I have added two new reporting events for smtp-in and smtp-out: tx-mail and tx-rcpt.\nThey are generated whenever MAIL FROM or RCPT TO have received a result from the server:\nreport|1|1544130229|smtp-in|tx-mail|0f3004c08c82d33e|fc08ce7d|\u0026lt;owner-hackers+M85937=gilles=poolp.org@openbsd.org\u0026gt;|ok report|1|1544130229|smtp-in|tx-rcpt|0f3004c08c82d33e|fc08ce7d|\u0026lt;gilles@poolp.org\u0026gt;|ok A filter may listen for these events to determine which sender and recipients have been accepted in a transaction, something that would otherwise require keeping track of protocol-client and protocol-server events.\nproc filtering finally in # I have committed full proc filtering support today, allowing a standalone filter to perform all kind of filtering on every single phase of an SMTP session.\nThis means that with the code in -current, it is now possible to write a filter that talks to rspamd or that computes dkim signatures, something that was not doable until today without resorting to tricky setups.\nThe code is still a work in progress, we have things to clean up, improve and there are some known minor issues that we are working on. We will likely hit corner cases that will cause new issues\u0026hellip; but this is a fully working implementation, the real deal, not a PoC. Consider it as \u0026ldquo;in a process of stabilization to be production ready by April\u0026rdquo;.\nWhen we have stabilized the API, I\u0026rsquo;ll take time to write about how it works internally, for now let\u0026rsquo;s just celebrate because I FINALLY FUCKING COMMITED FILTERS.\npython bridge, proof of concept # As I wrote in a previous post, proc filters are programs reading on stdin and writing to stdout, making it possible write them in any language.\nI put my code where my mouth is:\nimport opensmtpd def link_connect(timestamp, session_id, args): rdns, fcrdns, laddr, raddr = args def link_disconnect(timestamp, session_id, args): _ = args def tx_begin(timestamp, session_id, args): tx_id = args[0] def tx_mail(timestamp, session_id, args): tx_id, address, status = args def tx_rcpt(timestamp, session_id, args): tx_id, address, status = args def tx_envelope(timestamp, session_id, args): tx_id, evp_id = args def tx_rollback(timestamp, session_id, args): tx_id = args[0] def tx_commit(timestamp, session_id, args): tx_id, nbytes = args def protocol_client(timestamp, session_id, args): line = args[0] if __name__ == \u0026#34;__main__\u0026#34; o = opensmtpd.SMTP_IN() o.on_report(\u0026#39;link-connect\u0026#39;, link_connect) o.on_report(\u0026#39;link-disconnect\u0026#39;, link_disconnect) o.on_report(\u0026#39;tx-begin\u0026#39;, tx_begin) o.on_report(\u0026#39;tx-mail\u0026#39;, tx_mail) o.on_report(\u0026#39;tx-rcpt\u0026#39;, tx_rcpt) o.on_report(\u0026#39;tx-envelope\u0026#39;, tx_envelope) o.on_report(\u0026#39;tx-commit\u0026#39;, tx_commit) o.on_report(\u0026#39;tx-rollback\u0026#39;, tx_rollback) o.on_report(\u0026#39;protocol-client\u0026#39;, protocol_client) o.on_report(\u0026#39;protocol-server\u0026#39;, protocol_server) o.run() This only covers the callback API for \u0026ldquo;report\u0026rdquo; events, but it does so thanks to an opensmtpd package that consists of less than 100 lines of code.\nI will be implementing the on_filter callback API probably next week, so I can start writing the filters I need in python.\nIf you want to discuss how to write an interface for the language of your choice, feel free to jump to our IRC channel, #opensmtpd @ freenode ;-)\nWhat next ? # Essentially code cleanup and simplification, API stabilization and writing filters to spot improvements required in the API.\nStay tuned !\n","date":"6 December 2018","permalink":"/posts/2018-12-06/opensmtpd-proc-filters-fc-rdns/","section":"Posts","summary":"TL;DR: I *FINALFUCKINGLY* commited proc filters support allowing full filtering in OpenSMTPD. eric@ implemented fc-rDNS lookups. fc-rDNS # fc-rDNS, or forward-confirmed reverse DNS, consists in performing a reverse DNS lookup to determine the hostname associated to an IP address\u0026hellip; then performing a DNS lookup on that hostname to check if it resolves back to the IP address.","title":"OpenSMTPD proc filters \u0026 fc-rDNS"},{"content":" TL;DR: The reporting mechanism has been described shortly in my previous article about both reporting and filters. Let's focus a bit more on the reporting bits this time. The format is improving further and has extended to outgoing trafic reporting. Reporting # In previous article, I described the events reporting mechanism that has been introduced in the development branch of OpenSMTPD.\nTo sum it up, you could now write an event processor as simple as a shell script reading its stdin on a loop:\n$ cat /tmp/reporting.sh #! /bin/sh # while read line; do echo $line \u0026gt;\u0026gt; /tmp/reporting.log done and configure your OpenSMTPD so it would report all incoming SMTP events:\n$ grep report /etc/mail/smtpd.conf proc reporting \u0026#34;/tmp/reporting.sh\u0026#34; report smtp on reporting which would then produce an events report log in /tmp/reporting.log containing entries similar to these:\nreport|smtp-in|link-connect|1541271219|3189ac6874354895|localhost|127.0.0.1:41564|127.0.0.1:25 report|smtp-in|protocol-server|1541271219|3189ac6874354895|220 poolp.org ESMTP OpenSMTPD report|smtp-in|protocol-client|1541271222|3189ac6874354895|helo localhost report|smtp-in|protocol-server|1541271222|3189ac6874354895|250 poolp.org Hello localhost [127.0.0.1], pleased to meet you report|smtp-in|link-disconnect|1541271224|3189ac6874354895 Improvements on the reports format # I made several improvements to the format described in the previous article.\nThe first improvement is that there is now a version embedded in each report. This allows event processors to be able to check if they know how to parse an event report, it allows them to easily support backward-compatible versions should we make changes to the format of an event report, but most importantly it allows you to just store these events somewhere then have tools post-process them months later without ambiguity with regard to the format of entries, even if there were OpenSMTPD updates in between.\nThe second improvement is that the timestamp which was appearing after the event type was moved in front of it. This doesn\u0026rsquo;t seem like an improvement but it eases reading and allows me to simplify some of the code :-)\nThe third improvement comes from adding some new events and adding information to some existing events. For instance, OpenSMTPD reported these transaction events:\nreport|smtp-in|tx-begin|1541271225|3189ac6874354895 report|smtp-in|tx-commit|1541271225|3189ac6874354895 report|smtp-in|tx-rollback|1541271225|3189ac6874354895 But reading from these, you could only obtain the session identifier, there was no way to find out the transaction identifier, how many envelopes were generated in the transaction or even the size of the message.\nTo solve these issues, the format of the events above has been extended so it would contain the transation identifier (aka. msgid), a tx-envelope event was introduced to report the generation of a new envelope in the transaction along with its envelope identifier (aka. evpid), and finally the size of the DATA part is reported on a tx-commit event.\nLast but not least, the DATA part begins with a DATA command issued by the client and ends with a single . on a line by itself. Despite the single . being sent within the DATA phase, it is not really part of the DATA itself and must be considered as a commit request.\nThis doesn\u0026rsquo;t seem like much but the devil is in the details. Not reporting that commit request as a protocol-client command means that we go straight from DATA command to a tx-commit event, without allowing a filter to actually refuse the commit request. If we generate this commit request event, a filter may decide that it wants to reject it which will then produce a tx-rollback that was not possible before.\nA pattern emerges that tx-* events should appear in between protocol-* events otherwise they cannot be filtered.\nHere is a sample curated event report log from my own server as of today:\n$ cat /tmp/reporting.log report|1|1541750432|smtp-in|link-connect|c73c0aff0dfb6250|poolp.org|local:0|local:0 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|220 poolp.org ESMTP OpenSMTPD report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|EHLO localhost report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250-poolp.org Hello localhost [local], pleased to meet you report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250-8BITMIME report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250-ENHANCEDSTATUSCODES report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250-SIZE 36700160 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250 HELP report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; report|1|1541750432|smtp-in|tx-begin|c73c0aff0dfb6250|f84306b3 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250 2.0.0: Ok report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|RCPT TO:\u0026lt;gilles@poolp.org\u0026gt; report|1|1541750432|smtp-in|tx-envelope|c73c0aff0dfb6250|f84306b3|f84306b34f388082 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250 2.1.5 Destination address valid: Recipient ok report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|DATA report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|354 Enter mail, end with \u0026#34;.\u0026#34; on a line by itself report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|. report|1|1541750432|smtp-in|tx-commit|c73c0aff0dfb6250|f84306b3|301 report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|250 2.0.0: f84306b3 Message accepted for delivery report|1|1541750432|smtp-in|protocol-client|c73c0aff0dfb6250|QUIT report|1|1541750432|smtp-in|protocol-server|c73c0aff0dfb6250|221 2.0.0: Bye report|1|1541750432|smtp-in|link-disconnect|c73c0aff0dfb6250 Introducing smtp-out # Obviously my plan is to be able to report and create dashboard for ALL trafic, not just incoming.\nI have worked on generating reports for smtp-out and I actually have something working in a branch, which I intend to commit next week.\nThe format is identical with the sole difference that smtp-in is replaced with smtp-out, the event types are the same, the parameters are the same, you just need to get your head around the fact that protocol-client is your peer when in smtp-in, whereas protocol-server is your peer when in smtp-out:\nreport|1|1541750707|smtp-out|link-connect|c73c0b206ad6c41c||:0|64.233.167.26:25 report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|220 mx.google.com ESMTP k132-v6si807155wma.16 - gsmtp report|1|1541750707|smtp-out|protocol-client|c73c0b206ad6c41c|EHLO out.mailbrix.mx report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-mx.google.com at your service, [212.83.129.132] report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-SIZE 157286400 report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-8BITMIME report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-STARTTLS report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-ENHANCEDSTATUSCODES report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-PIPELINING report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250-CHUNKING report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|250 SMTPUTF8 report|1|1541750707|smtp-out|protocol-client|c73c0b206ad6c41c|STARTTLS report|1|1541750707|smtp-out|protocol-server|c73c0b206ad6c41c|220 2.0.0 Ready to start TLS report|1|1541750707|smtp-out|link-tls|c73c0b206ad6c41c|version=TLSv1.2, cipher=ECDHE-RSA-CHACHA20-POLY1305, bits=256 report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|EHLO out.mailbrix.mx report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-mx.google.com at your service, [212.83.129.132] report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-SIZE 157286400 report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-8BITMIME report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-ENHANCEDSTATUSCODES report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-PIPELINING report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250-CHUNKING report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250 SMTPUTF8 report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|MAIL FROM:\u0026lt;gilles@poolp.org\u0026gt; report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250 2.1.0 OK k132-v6si807155wma.16 - gsmtp report|1|1541750708|smtp-out|tx-begin|c73c0b206ad6c41c|8ea46ad1 report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|RCPT TO:\u0026lt;gilles.chehade@gmail.com\u0026gt; report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250 2.1.5 OK k132-v6si807155wma.16 - gsmtp report|1|1541750708|smtp-out|tx-envelope|c73c0b206ad6c41c|8ea46ad1|8ea46ad1ccae4934 report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|DATA report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|354 Go ahead k132-v6si807155wma.16 - gsmtp report|1|1541750708|smtp-out|protocol-client|c73c0b206ad6c41c|. report|1|1541750708|smtp-out|protocol-server|c73c0b206ad6c41c|250 2.0.0 OK 1541750708 k132-v6si807155wma.16 - gsmtp report|1|1541750708|smtp-out|tx-commit|c73c0b206ad6c41c|8ea46ad1|818 report|1|1541750718|smtp-out|protocol-client|c73c0b206ad6c41c|QUIT report|1|1541750718|smtp-out|protocol-server|c73c0b206ad6c41c|221 2.0.0 closing connection k132-v6si807155wma.16 - gsmtp report|1|1541750718|smtp-out|link-disconnect|c73c0b206ad6c41c Because not everyone needs reporting and not everyone needs both incoming and outgoing reporting, I have added the smtp-in and smtp-out keywords to the grammar so that you can:\n$ grep report /etc/mail/smtpd.conf proc reporting \u0026#34;/tmp/reporting.sh\u0026#34; report smtp-in on reporting report smtp-out on reporting I\u0026rsquo;ll probably make report smtp on a shortcut for both smtp-in and smtp-out.\nThere is still work to be done on the smtp-out path because the SMTP engine is more complex than for the smtp-in path. For instance, it is currently not possible to have any of the transaction events generated between the protocol-client and protocol-server events due to how the state machine is written. Not really as much of a big deal as for smtp-in since smtp-out isn\u0026rsquo;t filtered and the order issues are less annoying, but to be really clean and consistent, the smtp-in and smtp-out reports should be very parallel in terms of order events. I should be able to look at the smtp-out reports from my laptop and the smtp-in reports from my server and see them appear in the same order.\nFinally, there is also an rDNS lookup that needs to be added so the report is identical to smtp-in, and we should be fine.\nWhat\u0026rsquo;s so good about this ? # Reporting is not JUST about being able to write dashboards, it is not just about being able to generate state for filters eithers.\nGenerating event reports logs that can be parsed by external tools open the way for many side applications ranging from tools to replay sessions when tracking issues, tools to analyze behavior of peers and feedback into pf or OpenSMTPD tables, and more interestingly for people who will developer filters\u0026hellip; it brings the ability to write and test a filter without a running OpenSMTPD instance, piping the event log directly into the filter.\nTo be very honest, I\u0026rsquo;m personally more excited by this new feature than the filters feature which might be more visible but would be far less powerful without the event logs.\nWhat next ? # More changes should happen to the format of entries in the next few weeks and months, this is a moving target as I wrote in previous article.\nBuiltin filters already require some of these lines to provide more informations and this is being worked on.\nMy next focus is the filtering of the DATA phase which is the requirement for us to provide support for dkim and antispam stuff without the need of proxies and reenqueuing. Work has already started but I will probably not commit any code related to this before the end of November.\n","date":"9 November 2018","permalink":"/posts/2018-11-09/opensmtpd-reporting-update/","section":"Posts","summary":"TL;DR: The reporting mechanism has been described shortly in my previous article about both reporting and filters. Let's focus a bit more on the reporting bits this time.","title":"OpenSMTPD reporting update"},{"content":" TL;DR: Filters have been a (the most ?) long awaited feature in OpenSMTPD. I finally committed most of the filters code to OpenBSD. There is still a bit of work required but the trickiest parts are done. This article describes how filters are implemented and what to expect. OpenSMTPD 6.4.0 was released ! # We have released OpenSMTPD 6.4.0 last week without filters.\nI won\u0026rsquo;t expand on the features in the 6.4 release as I already wrote about the configuration file changes, the issues that required it and the refactors involved, this was the one true major feature of the release.\nOne notable aspect though is that we dropped our support for OpenSSL in favor of LibreSSL, and THAT I should expand upon ;-)\nLibreSSL FTW ! # OpenSMTPD was started long before LibreSSL and from the very first portable version, we had to include ifdefs to accomodate the differences between the different OpenSSL versions across Linux distros: some had SNI, others don\u0026rsquo;t, some had GCM, others don\u0026rsquo;t.\nWhen LibreSSL was forked from OpenSSL after a large cleanup of the code, we thought we\u0026rsquo;d just depend on that because it was cleaner and also we knew that this version had all the features we relied on. Also, there was this plan for a libtls wrapper which would be much simpler to use and less error-prone than libssl.\nIt turned out that it wasn\u0026rsquo;t possible because OpenSSL and LibreSSL couldn\u0026rsquo;t coexist on many systems. The fact that they shared the same library names (libssl, libcrypto), and shared object versionning, caused issues with the confused runtime link editor. We decided to accomodate both, keeping checks in the configure layer to detect what was being used, etc\u0026hellip; It was horrible. It kept breaking every now and then due to either one making changes. I had to deal with these breakages. It was horrible.\nBefore the 6.4 release, I did the portability build passes and yet again it broke on new OpenSSL conflicts with our grammar. Three different people using different distros had sent me three different diffs which fixed the issues for them, but which I couldn\u0026rsquo;t merge without creating a delta with the native branch which was not acceptable for us. The more I tried fixing and the more I was irritated by this OpenSSL/LibreSSL hacks.\nI did a bit of investigation and it turns out the issues that prevented them from coexisting were no longer relevant. I verified by building, linking and running OpenSMTPD against LibreSSL on FreeBSD, Ubuntu, Debian, CentOS, ArchLinux and Fedora. It is technically possible to make OpenSMTPD depend on LibreSSL without forcing distros to switch from OpenSSL to LibreSSL, so there\u0026rsquo;s no reason for us not to depend on LibreSSL anymore.\nThis allowed me to start removing ifdefs from our code, removing useless tests from the configure.ac and assume that we have the features we need for all systems.\nThis doesn\u0026rsquo;t mean that people can\u0026rsquo;t link OpenSMTPD against OpenSSL, it just means that they get to handle the diffs to their own version, the extra work of maintaining OpenSSL goes out of our hands.\nFilters # We started working on filters years ago, I even wrote an article about them in 2014.\nA first implementation was written. After a while, it became clear that the design was wrong and causing unfixable bugs in some filters. I decided to pull the plug on that attempt and we removed all of the code.\nA year ago, during EuroBSDCon 2017, I mentionned the filtering daemon we had worked on and which solved these limitations. But as OpenBSD 6.4 release was getting closer, I started to dislike that idea because I believed it was not addressing the proper problem: we had came up with a working solution, not the best solution to the problem. The more I spent thinking about this, the more the problem looked different from what we initially modeled.\nI discussed my concerns with Eric, told him that I thought we had made a mistake in modeling filters, that by thinking them differently we could come up with a much simpler solution, and somehow managed to convince him I was right :-)\nGiven we took so much time already, it made sense delaying for a release so we would be confident about our technical choice.\nI worked on a proof of concept for the new model and a week later I had filters working on my laptop, a convinced eric and\u0026hellip; many months to make it bright and shiny because it was obviously not going to make it into 6.4\nEvent reports and filter request # The new filters are modeled around the notions of event reports and filter requests.\nDuring the lifetime of a session, an SMTP engine will report various events such as the beginning of a new connection, the negotiation of ciphers during the TLS handshake or even the beginning of an SMTP transaction with MAIL FROM. These events help the SMTP engine build a state for a session and determine what is acceptable or not from a client at a given time.\nSome filters may not need a state, they may operate on the parameter to a command and that\u0026rsquo;s all. Other filters however may need to build a state for a session, not necessarily the same exact and complete state as the SMTP engine, but a state that makes sense given the work the filter will do.\nThe event reports are informative messages, which a filter may decide to process or not, and that encompass ALL of the SMTP events a session goes through. It is possible to replay an entire SMTP session based on these reports.\nThe filter requests however are not informative. A filter will be configured to handle a particular filtering phase and will receive filter requests for that phase, containing the phase and the parameter to filter, it will then be REQUIRED to answer the request with an action to take.\nAgain, simple filters may only deal with filter requests, more complex filters may process the event reports to learn about sessions before dealing with filter requests.\nEvent processors # Event processors are standalone executables which \u0026hellip; process event reports and filter requests in an infinite loop.\nOpenSMTPD executes them at startup and sets up an environment so they behave like traditional Unix-filters: they read input from stdin and write output to stdout.\nBoth event reports and filter requests will consist of single lines, each consisting of multiple fields separated by a pipe symbol |. All lines will hold a generic set of fields such as event type and session identifier, as well as a set of event-specific fields such as an ip address for connection event, an email address for mail-from event, etc\u0026hellip;\nEvent reports will expect no response, meaning that for a line read on stdin there will be no line written on stdout, whereas filter requests will expect a response, meaning that for a line read on stdin there will be one line written to stdout.\nThe simplest processor to write is one reading events to log them:\n#! /bin/sh # while read line; do echo $line \u0026gt;\u0026gt; /tmp/output.log done If you have ever parsed OpenSMTPD logs to inject them into an ELK or similar, you should now be in love with the events reporting feature.\nFilters are not much more complex, the only difference is that $line needs to be split to extract the session identifier, and the processor needs to write back its decision.\nThe simplest filtering processor to write is one that accepts every filter-request:\n#! /bin/sh # while read line; do if echo $line | grep \u0026#39;^filter-request|\u0026#39;; then SESSION=`echo $line | cut -d\\| -f4` echo \u0026#34;filter-response|${SESSION}|proceed\u0026#34; fi done Note that while this example shows a synchronous protocol where a filter-request gets an immediate answer, the implementation does not impose that limitation. There should be a filter-response for each filter-request, but they do not need to happen right away and they do not need to happen in the same sequence order.\nNow that you\u0026rsquo;re all sold on this solution, here are the bonus advantages:\nOpenSMTPD only knows that it has to execute the processor and write to its stdin. There are no dependencies and no limitations to what language a processor is written in, a postmaster does not have to learn C to write a filter but can effectively hack something fast with a few lines of shell or python.\nThe events processors are forked: they do not share the same memory space as OpenSMTPD, they do not share the same memory space as each other, and on systems that provide randomized memory layout they do not even share the same memory layout. On OpenBSD, each processor could benefit from its own pledge() and its own unveil(). A compromise of either one does not imply the compromission of the others or of the daemon.\nIn addition, they can be executed with different privileges and including chrooted() to a particular directory:\nproc filter1 \u0026#34;/tmp/filter1.sh\u0026#34; user _filter1 proc filter2 \u0026#34;/tmp/filter2.sh\u0026#34; group _filter2 proc filter3 \u0026#34;/tmp/filter3.sh\u0026#34; user _filter3 group gilles proc filter4 \u0026#34;/filter4\u0026#34; user _filter4 chroot \u0026#34;/tmp\u0026#34; Event proc # Event processors can receive event reports for each events generated by the SMTP engine (for now).\nThis allows two things. First, it allows fine-grained reporting to specialized software that can extract metrics, generate fancy dashboards and such.\nUntil now, people had to try and extract this information from our log files which were meant to be consumed by humans primarily. The event logs provides all of the useful informations in a format that can be easily parsed by scripts. Converting these entries to the proper format for injecting in a timeseries database or similar becomes trivial.\nThen, some filters may be able to work without any context by looking solely at the current command, but in many cases a decision at a particular phase may be the consequence of decisions at earlier phases, and filters can consume these event reports to build their own view of the state of sessions.\nHere is a sample of event reports:\nreport|smtp-in|link-connect|1541271219|3189ac6874354895|localhost|127.0.0.1:41564|127.0.0.1:25 report|smtp-in|protocol-server|1541271219|3189ac6874354895|220 poolp.org ESMTP OpenSMTPD report|smtp-in|protocol-client|1541271222|3189ac6874354895|helo localhost report|smtp-in|protocol-server|1541271222|3189ac6874354895|250 poolp.org Hello localhost [127.0.0.1], pleased to meet you report|smtp-in|link-disconnect|1541271224|3189ac6874354895 And this is how you define an event proc in smtpd.conf:\nproc my_event_proc \u0026#34;/usr/local/bin/script.sh\u0026#34; report smtp on my_event_proc This gets OpenSMTPD to run the script at startup and to pipe the event reports to its stdin.\nFilter proc # Filter processors are very similar to event processors in how they work, excepted that they are registered for specific filtering phases:\nproc my_filter_proc \u0026#34;/usr/local/bin/script.sh\u0026#34; # line below can be uncommented if filters needs event reports to build a state # report smtp on my_filter_proc filter smtp helo on my_filter_proc filter smtp ehlo on my_filter_proc filter smtp mail-from on my_filter_proc filter smtp rcpt-to on my_filter_proc The filters will receive phase-specific parameters:\nfilter-request|smtp-in|ehlo|e68808ff61812a6f|laptop.home|localhost filter-request|smtp-in|mail-from|e68808ff61812a6f|\u0026lt;gilles@laptop.home\u0026gt; filter-request|smtp-in|rcpt-to|e68808ff61812a6f|\u0026lt;gilles@laptop.home\u0026gt; filter-request|smtp-in|ehlo|e688090368c275a3|laptop.home|localhost filter-request|smtp-in|mail-from|e688090368c275a3|\u0026lt;gilles@laptop.home\u0026gt; filter-request|smtp-in|rcpt-to|e688090368c275a3|\u0026lt;gilles@laptop.home\u0026gt; To which they will have to reply with a decision:\nfilter-response|e68808ff61812a6f|proceed filter-response|e68808ff61812a6f|rewrite|BLEH filter-response|e68808ff61812a6f|reject|550 go away filter-response|e68808ff61812a6f|disconnect|550 go away The only possible decisions are proceed, rewrite, reject or disconnect.\nBuiltin filters # Filter proc are very useful when a decision relies on anything else than the parameter to the command being filtered, when it depends on information gathered from previous events, or even when it depends on complex logic involving multiple lookups.\nIn many cases though, a decision to filter may rely solely on the current command and a simple table lookup. For instance, one may want to reject a HELO hostname that\u0026rsquo;s not part of a table, or reject a MAIL FROM that matches a regular expression.\nFor these cases, OpenSMTPD provides a small set of builtin filters which are fairly generic to be applied to all phases.\ntable helo-reject { \u0026#34;foobar\u0026#34;, \u0026#34;barbaz\u0026#34;, \u0026#34;bazqux\u0026#34; } table helo-reject-regex { \u0026#34;^f[oO]o[bB]ar$\u0026#34; } filter smtp helo check-table \u0026lt;helo-reject\u0026gt; reject \u0026#34;550 go away\u0026#34; filter smtp helo check-regex \u0026#34;^go\\-away$\u0026#34; reject \u0026#34;550 go away\u0026#34; filter smtp helo check-regex \u0026lt;helo-reject-regex\u0026gt; reject \u0026#34;550 go away\u0026#34; We will also implement some very limited additional phase-specific builtin filters that cover common use-cases. OpenSMTPD performs a reverse DNS lookup on connect, both connect and helo/ehlo filter phases commonly check reverse DNS, so we already provided a check-rdns for these phases:\nfilter smtp connect check-rdns reject \u0026#34;550 you need a reverse DNS\u0026#34; filter smtp ehlo check-rdns reject \u0026#34;550 your HELO hostname and rDNS mismatch\u0026#34; There shouldn\u0026rsquo;t be many builtin filters, we don\u0026rsquo;t expect more than a handful of them, but they do exist so there\u0026rsquo;s that ;-)\nWhat next ? # This code has been committed to OpenBSD and will be available in OpenSMTPD 6.5.0, which should happen sometime around April 2019.\nI haven\u0026rsquo;t documented anything yet because both configuration grammar and protocol are still going through lots of changes, I would strongly suggest against playing with event reports and filter procs before March, unless you\u0026rsquo;re a developer and happy with having to make changes to your code every few days.\nThe only part that I have not committed yet is the filtering of DATA, which still requires a bit of work but will be working and committed sometime in November.\nOnce I have this part done, I\u0026rsquo;ll implement a few filters for the use-cases I have such as filter-spf and filter-rspamd, then it will be ok as far as I\u0026rsquo;m concerned.\nMy main work for the release cycle to come is finishing and polishing filters, fixing the portable layer which has grown a monster, and ultimately doing a full release cycle of\u0026hellip; cosmethic cleanup so code is pleasing to read ;-)\n","date":"3 November 2018","permalink":"/posts/2018-11-03/opensmtpd-released-and-upcoming-filters-preview/","section":"Posts","summary":"TL;DR: Filters have been a (the most ?) long awaited feature in OpenSMTPD. I finally committed most of the filters code to OpenBSD. There is still a bit of work required but the trickiest parts are done.","title":"OpenSMTPD released and upcoming filters preview"},{"content":" TL;DR: Switching to new config is not too hard and can be done in minutes. The new config is also a new queue that is not backwards compatible. The easiest way is to flush the mail queue before switching. We came up with a solution to help maintainers of more complex setups. Switching from old config to new config # The new OpenSMTPD configuration grammar is slightly different from the current one, rules are no longer stated as single lines, but the conversion from previous ruleset to new ruleset isn\u0026rsquo;t that hard.\nLet\u0026rsquo;s do the exercise with poolp.org\u0026rsquo;s smtpd.conf which is a fairly complex ruleset, making use of several features including TLS, authentication, multi-domain hosting with primary and virtual domains, different aliases mappings, backup MX, relaying through a DKIM proxy, and more\u0026hellip;\npki mx1.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mx1.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; pki mail.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mail.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; table sources { 212.83.181.8 } table helonames { 212.83.181.8 = mx1.poolp.org } table aliases \u0026#34;/etc/mail/aliases\u0026#34; table opensmtpd-aliases \u0026#34;/etc/mail/aliases-opensmtpd.org\u0026#34; table pdomains \u0026#34;/etc/mail/primary-domains\u0026#34; table vdomains \u0026#34;/etc/mail/virtual-domains\u0026#34; table vusers \u0026#34;/etc/mail/virtual-users\u0026#34; table bdomains \u0026#34;/etc/mail/backup-domains\u0026#34; table shithole\t{ \u0026#34;@qq.com\u0026#34; } listen on lo0 listen on lo0 port 10028 tag DKIM listen on egress tls-require pki mx1.poolp.org hostnames { 212.83.181.7 = mail.poolp.org, 212.83.181.8 = mx1.poolp.org } listen on egress smtps pki mail.poolp.org auth hostname mail.poolp.org listen on egress port submission tls-require pki mail.poolp.org auth hostname mail.poolp.org reject from any sender \u0026lt;shithole\u0026gt; for any accept for local alias \u0026lt;aliases\u0026gt; deliver to maildir accept from any for domain \u0026lt;pdomains\u0026gt; alias \u0026lt;aliases\u0026gt; deliver to maildir accept from any for domain opensmtpd.org alias \u0026lt;opensmtpd-aliases\u0026gt; deliver to maildir accept from any for domain \u0026lt;vdomains\u0026gt; virtual \u0026lt;vusers\u0026gt; deliver to maildir accept from any for domain \u0026lt;bdomains\u0026gt; relay backup mx1.poolp.org accept tagged DKIM for any relay source \u0026lt;sources\u0026gt; hostnames \u0026lt;helonames\u0026gt; accept for any relay via smtp://127.0.0.1:10027 Fixing pki directives # First of all, the pki directives which contain the certificates and private keys are not really affected. They work exactly as with the old grammar, we only shortened the certificate keyword to cert by popular demand. This results in:\npki mx1.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mx1.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; pki mail.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mail.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; being rewritten as:\npki mx1.poolp.org cert \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mx1.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; pki mail.poolp.org cert \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mail.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; The ca directive which allows setting a custom CA certificate, and which is not used in this configuration file, is subjected to the same change so if you have a directive such as:\nca mail.poolp.org certificate \u0026#34;/etc/ssl/poolp.org.ca\u0026#34; all you have to do is replace certificate with cert:\nca mail.poolp.org cert \u0026#34;/etc/ssl/poolp.org.ca\u0026#34; Other pki options are unchanged and default to the sanest option.\nFixing table directives # No changes in table directives:\ntable sources { 212.83.181.8 } table helonames { 212.83.181.8 = mx1.poolp.org } table aliases \u0026#34;/etc/mail/aliases\u0026#34; table opensmtpd-aliases \u0026#34;/etc/mail/aliases-opensmtpd.org\u0026#34; table pdomains \u0026#34;/etc/mail/primary-domains\u0026#34; table vdomains \u0026#34;/etc/mail/virtual-domains\u0026#34; table vusers \u0026#34;/etc/mail/virtual-users\u0026#34; table bdomains \u0026#34;/etc/mail/backup-domains\u0026#34; table shithole\t{ \u0026#34;@qq.com\u0026#34; } Fixing listen directives # The listen directive works as before, however like for pki some keywords were shortened. For instance, mask-source, which is not used in my config, was shortened to the more compact versions mask-src.\nIn my smtpd.conf, there was no change to listen directives despite using a large subset of listen features:\nlisten on lo0 listen on lo0 port 10028 tag DKIM listen on egress tls-require pki mx1.poolp.org hostnames { 212.83.181.7 = mail.poolp.org, 212.83.181.8 = mx1.poolp.org } listen on egress smtps pki mail.poolp.org auth hostname mail.poolp.org listen on egress port submission tls-require pki mail.poolp.org auth hostname mail.poolp.org Fixing the ruleset # This leaves us with the complex part of the change, switching from the one-line rules that used to define a decision, an envelope matching pattern and an action as a whole, to the new two-line rules defining a set of valid actions and a distinct set of matching patterns referencing the actions.\nDon\u0026rsquo;t worry, be happy. The change is mechanical and doesn\u0026rsquo;t need to completely rethink how you used to do your configuration files. The rules are now split into two parts, the actions and the matching patterns. The actions must be declared before matching patterns but may be declared in any order. The matching patterns work like previous ruleset, they are listed in \u0026lsquo;first match wins\u0026rsquo; order and therefore require the most specific rules first as the ruleset is essentially cascading on mismatches. There is no change whatsoever to that logic, so converting a former ruleset to a new ruleset is basically\u0026hellip; splitting previous ruleset into action and match directives with the match directives following the exact same order as with current ruleset.\nIn my case, the following ruleset needs to be rewritten:\nreject from any sender \u0026lt;shithole\u0026gt; for any accept for local alias \u0026lt;aliases\u0026gt; deliver to maildir accept from any for domain \u0026lt;pdomains\u0026gt; alias \u0026lt;aliases\u0026gt; deliver to maildir accept from any for domain opensmtpd.org alias \u0026lt;opensmtpd-aliases\u0026gt; deliver to maildir accept from any for domain \u0026lt;vdomains\u0026gt; virtual \u0026lt;vusers\u0026gt; deliver to maildir accept from any for domain \u0026lt;bdomains\u0026gt; relay backup mx1.poolp.org accept tagged DKIM for any relay source \u0026lt;sources\u0026gt; hostnames \u0026lt;helonames\u0026gt; accept for any relay via smtp://127.0.0.1:10027 First of all, we need to identify what are the unique actions within that ruleset, and this boils down to \u0026ldquo;aliases/virtual/userbase and anything after relay, deliver to\u0026rdquo;:\nalias \u0026lt;aliases\u0026gt; deliver to maildir alias \u0026lt;opensmtpd-aliases\u0026gt; deliver to maildir virtual \u0026lt;vusers\u0026gt; deliver to maildir relay backup mx1.poolp.org relay source \u0026lt;sources\u0026gt; hostnames \u0026lt;helonames\u0026gt; relay via smtp://127.0.0.1:10027 The new grammar for these will result in:\naction act01 maildir alias \u0026lt;aliases\u0026gt; action act02 maildir alias \u0026lt;opensmtpd-aliases\u0026gt; action act03 maildir virtual \u0026lt;vusers\u0026gt; action act04 relay backup mx mx1.poolp.org action act05 relay src \u0026lt;sources\u0026gt; helo-names \u0026lt;helonames\u0026gt; action act06 relay host smtp://127.0.0.1:10027 Now that all actions have been defined, we need to identify what are the matching patterns within the former ruleset, rewrite them with new grammar and attach them to their respective action. The change is very mechanical but keywords were shortened and made less ambiguous:\nmatch from any mail-from \u0026lt;shithole\u0026gt; for any reject match for local action act01 match from any for domain \u0026lt;pdomains\u0026gt; action act01 match from any for domain opensmtpd.org action act02 match from any for domain \u0026lt;vdomains\u0026gt; action act03 match from any for domain \u0026lt;bdomains\u0026gt; action act04 match tag DKIM for any action act05 match for any action act06 There is exactly as many match rules as there used to be accept+reject rules, they are in the same order, they perform the same action, they are just expressed differently.\nPutting it all together # This is the resulting smtpd.conf for poolp.org which should be considerably more complex thant most setups, we use a large part of OpenSMTPD\u0026rsquo;s feature across all domains, most setups will have two or three actions at most:\npki mx1.poolp.org cert \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mx1.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; pki mail.poolp.org cert \u0026#34;/etc/ssl/poolp.org.fullchain.pem\u0026#34; pki mail.poolp.org key \u0026#34;/etc/ssl/private/poolp.org.key\u0026#34; table sources { 212.83.181.8 } table helonames { 212.83.181.8 = mx1.poolp.org } table aliases file:/etc/mail/aliases table opensmtpd-aliases file:/etc/mail/aliases-opensmtpd.org table pdomains file:/etc/mail/primary-domains table vdomains file:/etc/mail/virtual-domains table vusers file:/etc/mail/virtual-users table bdomains file:/etc/mail/backup-domains table shithole\t{ \u0026#34;@qq.com\u0026#34; } listen on lo0 listen on lo0 port 10028 tag DKIM listen on egress tls-require pki mx1.poolp.org hostnames { 212.83.181.7 = mail.poolp.org, 212.83.181.8 = mx1.poolp.org } listen on egress smtps pki mail.poolp.org auth hostname mail.poolp.org listen on egress port submission tls-require pki mail.poolp.org auth hostname mail.poolp.org action act01 maildir alias \u0026lt;aliases\u0026gt; action act02 maildir alias \u0026lt;opensmtpd-aliases\u0026gt; action act03 maildir virtual \u0026lt;vusers\u0026gt; action act04 relay backup mx mx1.poolp.org action act05 relay src \u0026lt;sources\u0026gt; helo-src \u0026lt;helonames\u0026gt; action act06 relay host smtp://127.0.0.1:10027 match from any mail-from \u0026lt;shithole\u0026gt; for any reject match for local action act01 match from any for domain \u0026lt;pdomains\u0026gt; action act01 match from any for domain opensmtpd.org action act02 match from any for domain \u0026lt;vdomains\u0026gt; action act03 match from any for domain \u0026lt;bdomains\u0026gt; action act04 match tag DKIM for any action act05 match for any action act06 Easing the conversion # The smtpd.conf.5 man page has been adapted but given that it is essentially a rewrite, it is possible that we missed stuff and we will do our best to bring them to the usual quality you expect from OpenBSD man pages.\nYou can find here a sample smtpd.conf that lists all directives the configuration parser supports. The file is not exhaustive as this is not doable given the flexibility of the pattern matching, but if you have a doubt on how to rewrite an old rule to a new rule this along the man page should be enough to help you.\nThe #OpenSMTPD channel on irc.freenode.net, as well as our mailing lists are also available to help you do the conversion.\nThe queue is not backward compatible # Just switching from old config to new config and restarting is not going to work.\nWhile the grammar change is very mechanical and seems to be very close to previous grammar, the underlying structures have drastically changed, the action resolving too and the result is that on-disk envelopes are not compatible and CAN\u0026rsquo;T be made compatible. There\u0026rsquo;s no way we can convert them because there\u0026rsquo;s runtime resolving logic that expects to find an information we can\u0026rsquo;t make up out of nowhere.\nSo there are two conversion paths.\nThe first one, the prefered of course, is to flush the mail queue before upgrading to new smtpd. You pause incoming sessions with smtpctl pause smtp so your primary MX stops accepting mail, which is never an issue because you have a secondary MX of course, and you issue a smtpctl schedule all so all pending mails are sent right away. When the mail queue is empty, which you can check with smtpctl show queue, you stop the daemon, upgrade and start the new one on your new config.\nThe second one is for busy mail hosts that can\u0026rsquo;t flush a mail queue. This is basically the case for a mailing list server which can almost never flush a queue empty due to unreachable hosts, remote host limits requiring the server to be down for an extended period of time, etc\u0026hellip; For these, we came up with the smtpd-salvage daemon.\nBasically, OpenSMTPD has envelopes versionning so a new config OpenSMTPD will not process old envelopes it can\u0026rsquo;t support, and an old config OpenSMTPD will not process the new envelopes it can\u0026rsquo;t support. If you can\u0026rsquo;t flush the queue, start the new config OpenSMTPD which will skip old envelopes and start creating new envelopes, then start smtpd-salvage with the older config. The smtpd-salvage daemon will then drain the queue from old messages. After a few days, all messages that weren\u0026rsquo;t delivered should be expired so smtpd-salvage can be stopped and deleted forever.\nUSE queue flush if you can.\n","date":"21 May 2018","permalink":"/posts/2018-05-21/switching-to-opensmtpd-new-config/","section":"Posts","summary":"TL;DR: Switching to new config is not too hard and can be done in minutes. The new config is also a new queue that is not backwards compatible.","title":"switching to OpenSMTPD new config"},{"content":" TL;DR: OpenBSD #p2k18 hackathon took place at Epitech in Nantes. I was organizing the hackathon but managed to make progress on OpenSMTPD. As mentionned at EuroBSDCon the one-line per rule config format was a design error. A new configuration grammar is almost ready and the underlying structures are simplified. Refactor removes ~750 lines of code and solves _many_ issues that were side-effects of the design error. New features are going to be unlocked thanks to this. Anatomy of a design error # OpenSMTPD started ten years ago out of dissatisfaction with other solutions, mainly because I considered them way too complex for me not to get things wrong from time to time.\nThe initial configuration format was very different, I was inspired by pyr@\u0026rsquo;s hoststated, which eventually became relayd, and designed my configuration format with blocks enclosed by brackets.\nWhen I first showed OpenSMTPD to pyr@, he convinced me that PF-like one-line rules would be awesome, and it was awesome indeed.\nIt helped us maintain our goal of simple configuration files, it helped fight feature creeping, it helped us gain popularity and become a relevant MTA, it helped us get where we are now 10 years later.\nThat being said, I believe this was a design error. A design error that could not have been predicted until we hit the wall to understand WHY this was an error. One-line rules are semantically wrong, they are SMTP wrong, they are wrong.\nOne-line rules are making the entire daemon more complex, preventing some features from being implemented, making others more complex than they should be, they no longer serve our goals.\nTo get to the point: we should move to two-line rules :-)\nSMTP is transactional protocol # SMTP is a transactional protocol which considers each message to be part of a transaction with one or many recipients.\nA message gets assigned a unique transaction identifier when committed to queue:\n250 2.0.0: 8a2e1208 Message accepted for delivery That identifier (supposedly) guarantees that the message has been written to disk, that the mail exchanger will take responsibility not to let it vanish, and that all recipients from the same transaction will share the same identifier.\nSimilarly, recipients may result in aliasing or may have ~/.forward files resulting in new recipients, and since all recipients from the same transaction must share the same identifier, this implies that all aliases and .forward expansions must be done before accepting a recipient.\nIf you must remember only one thing from this section, when the message identifier has been replied to the client, no new envelope will be generated for this transaction.\nThe problem with one-line rules # OpenSMTPD decides to accept or reject messages based on one-line rules such as:\naccept from any for domain poolp.org deliver to mbox Which can essentially be split into three units:\nthe decision: accept/reject the matching: from any for domain poolp.org the (default) action: deliver to mbox To ensure that we meet the requirements of the transactions, the matching must be performed during the SMTP transaction before we take a decision for the recipient.\nGiven that the rule is atomic, that it doesn\u0026rsquo;t have an identifier and that the action is part of it, the two only ways to make sure we can remember the action to take later on at delivery time is to either:\nsave the action in the envelope, which is what we do today evaluate the envelope again at delivery And this this where it gets tricky\u0026hellip; both solutions are NOT ok.\nThe first solution, which we\u0026rsquo;ve been using for a decade, was to save the action within the envelope and kind of carve it in stone. This works fine\u0026hellip; however it comes with the downsides that errors fixed in configuration files can\u0026rsquo;t be caught up by envelopes, that delivery action must be validated way ahead of time during the SMTP transaction which is much trickier, that the parsing of delivery methods takes place as the _smtpd user rather than the recipient user, and that envelope structures that are passed all over OpenSMTPD carry delivery-time informations, and more, and more, and more. The code becomes more complex in general, less safe in some particular places, and some areas are nightmarish to deal with because they have to deal with completely unrelated code that can\u0026rsquo;t be dealt with later in the code path.\nThe second solution can\u0026rsquo;t be done. An envelope may be the result of nested rules, for example an external client, hitting an alias, hitting a user with a .forward file resolving to a user. An envelope on disk may no longer match any rule or it may match a completely different rule If we could ensure that it matched the same rule, evaluating the ruleset may spawn new envelopes which would violate the transaction. Trying to imagine how we could work around this leads to more and more and more RFC violations, incoherent states, duplicate mails, etc\u0026hellip;\nThere is simply no way to deal with this with atomic rules, the matching and the action must be two separate units that are evaluated at two different times, failure to do so will necessarily imply that you\u0026rsquo;re either using our first solution and all its downsides, or that you are currently in a world of pain trying to figure out why everything is burning around you. The minute the action is written to an on-disk envelope, you have failed.\nA proper ruleset must define a set of matching patterns resolving to an action identifier that is carved in stone, AND a set of named action set that is resolved dynamically at delivery time.\n# this is resolved at delivery time action my_action mbox # this is resolved at SMTP transaction time match from any for local action my_action Action and Match # By splitting these, we get immediate benefits, I will only list a few of the improvements this allowed and why.\nFirst of all, the on-disk envelopes now save the name of the action, rather than the action, and resolve the action right before delivery.\nThe obvious benefit is that if I was wrong and wanted to convert to maildir, it would be enough to change the previous example to:\naction my_action maildir match from any for local action my_action After a restart, all envelopes would immediately catch up change without any need to rewrite anything. The same would apply if I got my action wrong:\naction my_action maildir \u0026ldquo;~/Maildirr\u0026rdquo;\nWhich would be unfixable today without going to edit envelopes on-disk manually, but would only require fixing smtpd.conf and restarting with the new configuration.\nThe other benefit is that code paths like aliases expansion required going down to the action to write the envelope, this would involve looking up users, expanding .forward variables, making sure that the delivery data is fully ready so that the delivery process just has to pass it to the MDA. The aliases expansion is already a complex code path by nature, throwing a ton of additional delivery-time data to process and craft into structures made some parts really hard to work with.\nThe structures involved would not only contain the envelope data, they would also contain the delivery data which were passed to all components including the SMTP server, the disk queue, the scheduler, the delivery and finally the mda, when the only processes really needing them were the deivery and mda processes.\nCode parsing user-supplied variables, which could be expanded by the user process right before executing the mda command needed to be parsed during the SMTP transaction within an OpenSMTPD owned process.\nI don\u0026rsquo;t know where to end, the refactor to move to his new grammar essentially simplified every single process within OpenSMTPD, removed over 750 lines of code from the daemon, automagically unlocked features that were previously not even doable, and makes it possible to move forward with other features which were tricky.\nNew features ? # Yes, there are new features which required the grammar change but they are not part of it and will come after.\nThe only features the new grammar bring is some matching patterns such as:\nmatch tls [...] # to match a session using TLS match auth [...] # to match an authenticated session match from socket [...] # to match sessions issued by the local enqueuer match helo foobar [...] # to match sessions that issues HELO foobar What\u0026rsquo;s the catch ? # The code is ready but I haven\u0026rsquo;t finished the man page update which is a pre-requisite to me sending this to OpenBSD developers to get their feedbak on the new grammar.\nIn addition, the change of behaviour and grammar is absolutely not backward-compatible and will make the older queue incompatible.\nBesides that, it has 100% benefits and no downside compared to previous grammar, it is cleaner, safer, saner and allows for a brighter future.\nIt is currently powering poolp.org and the misc@opensmtpd.org mailing list and shows no sign of regression.\nThis should hit OpenBSD -current in a few days/weeks and be part of the next OpenBSD (6.4) and OpenSMTPD major release.\n","date":"30 April 2018","permalink":"/posts/2018-04-30/opensmtpd-new-config/","section":"Posts","summary":"TL;DR: OpenBSD #p2k18 hackathon took place at Epitech in Nantes. I was organizing the hackathon but managed to make progress on OpenSMTPD. As mentionned at EuroBSDCon the one-line per rule config format was a design error.","title":"OpenSMTPD new config"},{"content":" TL;DR: I run several \u0026quot;dedibox\u0026quot; servers at online.net, all powered by OpenBSD. OpenBSD is not officially supported so you have to work-around. Running full-disk encrypted OpenBSD there is a piece of cake. As a bonus, my first steps within a brand new booted machine ;-) Step #0: choosing your server # OpenBSD is not officially supported, I can\u0026rsquo;t guarantee that this will work for you on any kind of server online.net provides, however I\u0026rsquo;ve been running https://poolp.org on OpenBSD there since 2008, only switching machines as they were getting a bit old and new offers came up.\nCurrently, I\u0026rsquo;m running two SC 2016 (SATA) and one XC 2016 (SSD) boxes, all three running OpenBSD reliably ever since I installed them.\nRecently I\u0026rsquo;ve been willing to reinstall the XC one after I did some experiments that turned it into a FrankenBSD, so this was the right occasion to document how I do it for future references.\nI wrote an article similar to this a few years ago relying on qemu to install to the disk, since then online.net provided access to a virtual serial console accessed within the browser, making it much more convenient to install without the qemu indirection which hid the NIC devices and disks duid and required tricks.\nThe method I currently use is a mix and adaptation from the techniques described in https://www.2f30.org/guides/openbsd-dedibox.html to boot the installer, and the technique described in https://geekyschmidt.com/2011/01/19/configuring-openbsd-softraid-fo-encryption.html to setup the crypto slice.\nStep #1: boot to rescue mode # The web console has a rescue mode which will essentially boot the server on a system running in RAM.\nThis usually allows you to unfuck a broken system by booting a Linux or FreBSD system, mounting disks, making appropriate changes to the disk, then rebooting back to the original system.\nWe will actually make use of this rescue mode to write an OpenBSD boot disk image at the beginning of the real disk, allowing us to reboot the server right into the OpenBSD intaller.\nlaptop$ ssh gilles@163.172.61.249 [...] gilles@163-172-61-249:~$ wget https://ftp.fr.openbsd.org/pub/OpenBSD/6.2/amd64/miniroot62.fs [...] gilles@163-172-61-249:~$ sudo dd if=miniroot62.fs of=/dev/sda [sudo] password for gilles: 9600+0 records in 9600+0 records out 4915200 bytes (4,9 MB, 4,7 MiB) copied, 0,116668 s, 42,1 MB/s gilles@163-172-61-249:~$ You can then reboot back to normal mode and activate the serial console from the web console.\nStep #2: boot to the installer # On the serial console, you will be greeted by the bootloader prompt. For some reason, every couple seconds a keystroke gets triggered by the interface causing the \u0026rsquo;n\u0026rsquo; character to be inserted. It takes a bit of synchronization but you should be able to set tty to com1, getting rid of the keystroke and allowing proper output to the terminal:\nboot\u0026gt; set tty com1 \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 boot\u0026gt; boot The bootloader will load a ramdisk kernel and drop you into the installer, which is our next step.\nStep #3: prepare softraid # Once the installer is started, drop immediately into a shell:\nWelcome to the OpenBSD/amd64 6.2 installation program. (I)nstall, (U)pgrade, (A)utoinstall or (S)hell? s # First of all, we need to rewrite the MBR after we messed it up with our miniroot trick:\n# fdisk -iy sd0 Writing MBR at offset 0. # Then, we can enter the disklabel to setup a RAID slice and a swap slice, keeping in mind that swap is already encrypted by default on OpenBSD. The RAID slice will be used to setup softraid with the crypto discipline.\n# disklabel -E sd0 Label editor (enter \u0026#39;?\u0026#39; for help at any prompt) \u0026gt; p M OpenBSD area: 64-500103450; size: 244191.1M; free: 244191.1M # size offset fstype [fsize bsize cpg] c: 244198.3M 0 unused \u0026gt; a partition: [a] offset: [64] size: [500103386] 240000M Rounding size to cylinder (16065 sectors): 491524676 FS type: [4.2BSD] RAID \u0026gt; a partition: [b] offset: [491524740] size: [8578710] FS type: [swap] \u0026gt; w \u0026gt; q No label changes. # Once this is done, we can use bioctl to setup the encrypted slice:\n# bioctl -c C -r auto -l /dev/sd0a softraid0 New passphrase: Re-type passphrase: sd1 at scsibus1 targ 1 lun 0: \u0026lt;OPENBSD, SR CRYPTO, 006\u0026gt; SCSI2 0/direct fixed sd1: 240002MB, 512 bytes/sector, 491524148 sectors softraid0: CRYPTO volume attached as sd1 # At this point, we\u0026rsquo;re ready to perform a regular install\u0026hellip; on sd1, not sd0, beware,:\n# install At any prompt except password prompts you can escape to a shell by typing \u0026#39;!\u0026#39;. Default answers are shown in []\u0026#39;s and are selected by pressing RETURN. You can exit this program at any time by pressing Control-C, but this can leave your system in an inconsistent state. Terminal type? [vt220] System hostname? (short form, e.g. \u0026#39;foo\u0026#39;) pocs Available network interfaces are: em0 em1 vlan0. Which network interface do you wish to configure? (or \u0026#39;done\u0026#39;) [em0] IPv4 address for em0? (or \u0026#39;dhcp\u0026#39; or \u0026#39;none\u0026#39;) [dhcp] em0: DHCPDISCOVER - interval 1 em0: DHCPOFFER from 163.172.61.1 (00:81:c4:f6:e9:17) em0: DHCPREQUEST to 255.255.255.255 em0: DHCPACK from 163.172.61.1 (00:81:c4:f6:e9:17) em0: bound to 163.172.61.249 -- renewal in 2147483647 seconds IPv6 address for em0? (or \u0026#39;autoconf\u0026#39; or \u0026#39;none\u0026#39;) [none] Available network interfaces are: em0 em1 vlan0. Which network interface do you wish to configure? (or \u0026#39;done\u0026#39;) [done] Default IPv4 route? (IPv4 address or none) [163.172.61.1] add net default: gateway 163.172.61.1 Using DNS domainname online.net Using DNS nameservers at 62.210.16.6 62.210.16.7 Password for root account? (will not echo) Password for root account? (again) Start sshd(8) by default? [yes] Change the default console to com1? [yes] Available speeds are: 9600 19200 38400 57600 115200. Which speed should com1 use? (or \u0026#39;done\u0026#39;) [9600] Setup a user? (enter a lower-case loginname, or \u0026#39;no\u0026#39;) [no] gilles Full name for user gilles? [gilles] Password for user gilles? (will not echo) Password for user gilles? (again) WARNING: root is targeted by password guessing attacks, pubkeys are safer. Allow root ssh login? (yes, no, prohibit-password) [no] What timezone are you in? (\u0026#39;?\u0026#39; for list) [Europe/Paris] I insist again, you want to write to the new softraid-backed disk:\nAvailable disks are: sd0 sd1. Which disk is the root disk? (\u0026#39;?\u0026#39; for details) [sd0] sd1 No valid MBR or GPT. Use (W)hole disk MBR, whole disk (G)PT or (E)dit? [whole] Setting OpenBSD MBR partition to whole sd1...done. The auto-allocated layout for sd1 is: # size offset fstype [fsize bsize cpg] a: 1.0G 64 4.2BSD 2048 16384 1 # / b: 16.2G 2097216 swap c: 234.4G 0 unused d: 4.0G 36067392 4.2BSD 2048 16384 1 # /tmp e: 29.5G 44455968 4.2BSD 2048 16384 1 # /var f: 2.0G 106342848 4.2BSD 2048 16384 1 # /usr g: 1.0G 110537152 4.2BSD 2048 16384 1 # /usr/X11R6 h: 10.0G 112634304 4.2BSD 2048 16384 1 # /usr/local i: 2.0G 133605824 4.2BSD 2048 16384 1 # /usr/src j: 6.0G 137800128 4.2BSD 2048 16384 1 # /usr/obj k: 162.7G 150383040 4.2BSD 4096 32768 1 # /home Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] Just for the purpose of simplifying, I will create a custom layout with only a root slice, note that this is insecure and, unless you know what you\u0026rsquo;re doing you should avoid that as it exposes your system to a denial of service.\nThe swap slice is redundant with the one we already create along the RAID slice in sd0, so the options are either to stick with the auto layout or to create a custom layout that\u0026rsquo;s similar but without swap.\nAt the very very least, you want to isolate / from /tmp, /var, /usr and /home, though isolating it from /usr/local is not a bad idea.\nUse (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] C Label editor (enter \u0026#39;?\u0026#39; for help at any prompt) \u0026gt; p OpenBSD area: 64-491508675; size: 491508611; free: 491508611 # size offset fstype [fsize bsize cpg] c: 491524148 0 unused \u0026gt; a partition: [a] offset: [64] size: [491508611] FS type: [4.2BSD] mount point: [none] / Rounding size to bsize (64 sectors): 491508608 \u0026gt; w \u0026gt; q No label changes. /dev/rsd1a: 239994.4MB in 491508608 sectors of 512 bytes 295 cylinder groups of 814.44MB, 26062 blocks, 52224 inodes each Available disks are: sd0. Which disk do you wish to initialize? (or \u0026#39;done\u0026#39;) [done] /dev/sd1a (448be85498897147.a) on /mnt type ffs (rw, asynchronous, local) Let\u0026#39;s install the sets! Location of sets? (disk http or \u0026#39;done\u0026#39;) [http] HTTP proxy URL? (e.g. \u0026#39;http://proxy:8080\u0026#39;, or \u0026#39;none\u0026#39;) [none] HTTP Server? (hostname, list#, \u0026#39;done\u0026#39; or \u0026#39;?\u0026#39;) [ftp.fr.openbsd.org] Server directory? [pub/OpenBSD/6.2/amd64] Select sets by entering a set name, a file name pattern or \u0026#39;all\u0026#39;. De-select sets by prepending a \u0026#39;-\u0026#39;, e.g.: \u0026#39;-game*\u0026#39;. Selected sets are labelled \u0026#39;[X]\u0026#39;. [X] bsd [X] base62.tgz [X] game62.tgz [X] xfont62.tgz [X] bsd.mp [X] comp62.tgz [X] xbase62.tgz [X] xserv62.tgz [X] bsd.rd [X] man62.tgz [X] xshare62.tgz Set name(s)? (or \u0026#39;abort\u0026#39; or \u0026#39;done\u0026#39;) [done] Get/Verify SHA256.sig 100% |**************************| 2152 00:00 Signature Verified Get/Verify bsd 100% |**************************| 12777 KB 00:08 Get/Verify bsd.mp 100% |**************************| 12858 KB 09:46 Get/Verify bsd.rd 100% |**************************| 9565 KB 00:05 Get/Verify base62.tgz 100% |**************************| 139 MB 01:24 Get/Verify comp62.tgz 100% |**************************| 75525 KB 50:08 Get/Verify man62.tgz 100% |**************************| 7008 KB 00:03 Get/Verify game62.tgz 100% |**************************| 2718 KB 01:56 Get/Verify xbase62.tgz 100% |**************************| 17964 KB 12:29 Get/Verify xshare62.tgz 100% |**************************| 4417 KB 02:43 Get/Verify xfont62.tgz 100% |**************************| 39342 KB 00:20 Get/Verify xserv62.tgz 100% |**************************| 12572 KB 00:06 Installing bsd 100% |**************************| 12777 KB 00:00 Installing bsd.mp 100% |**************************| 12858 KB 00:00 Installing bsd.rd 100% |**************************| 9565 KB 00:00 Installing base62.tgz 100% |**************************| 139 MB 00:10 Extracting etc.tgz 100% |**************************| 189 KB 00:00 Installing comp62.tgz 100% |**************************| 75525 KB 00:08 Installing man62.tgz 100% |**************************| 7008 KB 00:01 Installing game62.tgz 100% |**************************| 2718 KB 00:00 Installing xbase62.tgz 100% |**************************| 17964 KB 00:01 Extracting xetc.tgz 100% |**************************| 7036 00:00 Installing xshare62.tgz 100% |**************************| 4417 KB 00:01 Installing xfont62.tgz 100% |**************************| 39342 KB 00:02 Installing xserv62.tgz 100% |**************************| 12572 KB 00:01 Location of sets? (disk http or \u0026#39;done\u0026#39;) [done] Location of sets? (disk http or \u0026#39;done\u0026#39;) [done] Saving configuration files...done. Making all device nodes...done. Multiprocessor machine; using bsd.mp instead of bsd. Relinking to create unique kernel...done. CONGRATULATIONS! Your OpenBSD install has been successfully completed! To boot the new system, enter \u0026#39;reboot\u0026#39; at the command prompt. When you login to your new system the first time, please read your mail using the \u0026#39;mail\u0026#39; command. # Step #4: reboot to encrypted OpenBSD system # The root slice being encrypted, you\u0026rsquo;ll need to type your password every time you reboot.\nConnect to the serial console, you should see the boot prompt asking for a password. The \u0026rsquo;n\u0026rsquo; glitch is still around, so either you break out of password by typing enter so you can do the \u0026lsquo;set tty com1\u0026rsquo; trick, or you do as I do and synchronize with the \u0026rsquo;n\u0026rsquo; keystroke to delete it and type password really fast.\n# reboot syncing disks... done sd1 detached rebooting... [...] Using drive 0, partition 3. Loading...... probing: pc0 com0 com1 mem[632K 2009M 14336M a20=on] disk: hd0+ sr0* \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 Passphrase: boot\u0026gt; set tty com1 \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 boot\u0026gt; boot Passphrase: [...] OpenBSD/amd64 (pocs.online.net) (tty01) login: gilles Password: OpenBSD 6.2 (GENERIC.MP) #134: Tue Oct 3 21:22:29 MDT 2017 Welcome to OpenBSD: The proactively secure Unix-like operating system. Please use the sendbug(1) utility to report bugs in the system. Before reporting a bug, please try to reproduce it with the latest version of the code. With bug reports, please try to ensure that enough information to reproduce the problem is enclosed, and if a known fix for it exists, include that as well. You have new mail. $ As suggested by semarie@, once logged in you can take opportunity to fix the \u0026rsquo;n\u0026rsquo; glitch by setting tty in boot.conf:\n$ su Password: # echo set tty com1 \u0026gt; /etc/boot.conf Bonus: further tightening your system # These are the few steps I immediately do to tighten up my systems furthers:\nenable doas # The \u0026lsquo;gilles\u0026rsquo; account I created at install is part of the \u0026lsquo;wheel\u0026rsquo; group, which turns out to be exactly what the example doas.conf allows:\n$ su Password: # cat /etc/examples/doas.conf |tail -1 permit keepenv :wheel # cp /etc/examples/doas.conf /etc # exit $ doas sh doas (gilles@pocs.online.net) password: # disable the root account # Now that \u0026lsquo;gilles\u0026rsquo; can use doas we no longer ever need to authenticate as root, so disable it by setting the password to \u0026lsquo;*\u0026rsquo;. This will prevent \u0026lsquo;root\u0026rsquo; from being usable directly or through su, yet if really needed \u0026lsquo;gilles\u0026rsquo; can still doas su to obtain a shell running as user \u0026lsquo;root\u0026rsquo;:\n# usermod -p\u0026#39;*\u0026#39; root # update system with syspatch # The brand new system may require some patches to be applied, the syspatch command written by ajacoutot@ performs a binary patching of the system, then causes the kernel to be relinked using the KARL mechanism to shuffle objects order:\n# syspatch Get/Verify syspatch62-001_tcb_inv... 100% |*************| 465 KB 00:00 Installing patch 001_tcb_invalid Get/Verify syspatch62-002_fktrace... 100% |*************| 785 KB 00:21 Installing patch 002_fktrace Get/Verify syspatch62-003_mpls.tgz 100% |***************| 837 KB 00:00 Installing patch 003_mpls Get/Verify syspatch62-004_libssl.tgz 100% |*************| 2515 KB 00:01 Installing patch 004_libssl Relinking to create unique kernel... done. add my ssh public key to my ~/.ssh/authorized_keys # Password for accessing SSH are bad, copy the SSH public key generated on my laptop with ssh-keygen to the authorized_keys file of my account on the server:\n# echo \u0026#39;ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII19dxt4hf1t4Lo05TaPZQqrtwtszHHyAF7ctPYPRIvp gilles@debug.poolp.org\u0026#39;\u0026gt;\u0026gt;~gilles/.ssh/authorized_keys # disable password authentication within ssh # No reason to allow PasswordAuthentication anymore, disable in sshd_config and restart sshd:\n# echo PasswordAuthentication no \u0026gt;\u0026gt; /etc/ssh/sshd_config # /etc/rc.d/sshd restart sshd(ok) sshd(ok) # reboot so you boot on a brand new up-to-date system with latest stable kernel # # reboot syncing disks... done sd1 detached rebooting... [...] Using drive 0, partition 3. Loading...... probing: pc0 com0 com1 mem[632K 2009M 14336M a20=on] disk: hd0+ sr0* \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 Passphrase: boot\u0026gt; set tty com1 \u0026gt;\u0026gt; OpenBSD/amd64 BOOT 3.33 boot\u0026gt; boot Passphrase: [...] OpenBSD/amd64 (pocs.online.net) (tty01) login: Password: VOILA !\nWant to comment ? Open an issue on Github\n","date":"29 January 2018","permalink":"/posts/2018-01-29/install-openbsd-on-dedibox-with-full-disk-encryption/","section":"Posts","summary":"TL;DR: I run several \u0026quot;dedibox\u0026quot; servers at online.net, all powered by OpenBSD. OpenBSD is not officially supported so you have to work-around. Running full-disk encrypted OpenBSD there is a piece of cake.","title":"Install OpenBSD on dedibox with full-disk encryption"},{"content":" TL;DR: deraadt@ thought it would be nice to have a spf fetch utility in base. Aaron Poffenberger wrote a shell-based `spf_fetch` utility. I wrote a C-based `spfwalk` utility that's `pledge()`-ed. The `spfwalk` utility got merged to `smtpctl`. What\u0026rsquo;s SPF in a few words # SPF is the Sender Policy Framework, a standard to verify the domain name of an e-mail sender.\nLong story short, the SMTP protocol does not come with a way to authenticate a domain and, during an SMTP session, nothing really prevents a sender from pretending to come from any domain:\n$ nc localhost 25 220 poolp.org ESMTP OpenSMTPD HELO pussycat 250 poolp.org Hello pussycat [127.0.0.1], pleased to meet you MAIL FROM:\u0026lt;gilles@systemd.lol\u0026gt; 250 2.0.0: Ok ^C $ Note that this is a feature of the SMTP protocol, not a bug.\nIt turns out that the internet is a hostile place and people started abusing this so\u0026hellip; the solution, as usual when it comes to SMTP, was to shove data inside DNS records \\o/\nSPF allows the owner of a DNS zone to declare which machines are allowed to send mail on behalf of the domain in a TXT record. With this feature, whenever a spammer attempts to send mail on behalf of @google.com, the receiving MX can simply check that the client is allowed to send mail by the google.com zone.\nThis seems lovely, however it\u0026rsquo;s not a spam killer because it actually requires senders to create the record, destination nodes to actually check the TXT record and match the client address against it, and it doesn\u0026rsquo;t prevent spammers from adding SPF records to their own domains.\nWhat it does, when everything is setup correctly on both ends, is protect the destination node from spammers trying to impersonnate a sender domain\u0026hellip; as long as the spammer doesn\u0026rsquo;t control a machine declared in the TXT record [ it\u0026rsquo;s easy to whitelist a /16 ;) ].\nGreylisting # A technique that\u0026rsquo;s been used widely to reduce spam is to rely on greylisting.\nBasically, when a MX you don\u0026rsquo;t know connects to your node, it gets bounced with a temporary failure requesting a retry. All SMTP server know how to handle these retries, however in a spamming economy based on sending volumes of messages it\u0026rsquo;s often not worth it to retry on temporary failures.\nSpammers will come from many source addresses, keep being seen as new MX, keep being bounced, life is good.\nWhy am I talking about greylisting you ask ?\nWhen BIG MAILERS made it easier for spammers to annoy us # It\u0026rsquo;s impolite to point fingers so I won\u0026rsquo;t name them explicitely, you know who they are.\nGreylisting works lovely to waste spammers\u0026rsquo; time\u0026hellip; as long as you can assume legitimate hosts come from the same IP addresses.\nAt some point, BIG MAILERS decided that: nope, we\u0026rsquo;ll send from this host, then we\u0026rsquo;ll retry from this one, then another one, and then since we have hundreds of IP addresses available, well just fuck you, we\u0026rsquo;ll make sure we never hit you twice with the same.\nI don\u0026rsquo;t know if these were the exact words, but it essentially leads to that \u0026ldquo;fuck you\u0026rdquo; result so\u0026hellip;\nSince BIG MAILERS could not send from a small set of addresses and reuse the same ones upon retries, they started advertising their MX in SPF records. And by that I mean, they started whitelisting full ranges in records requiring recursive and cross-domains lookups because WHY NOT ?\nThis way, greylisting became essentially unusable to many who turned it off because they can\u0026rsquo;t easily whitelist BIG MAILERS.\nThis is how we get to spfwalk # We don\u0026rsquo;t want to NOT have greylisting, so we want a tool to actually harvest records enabled in BIG MAILERS\u0026rsquo; SPF record.\nderaadt@ ran into an issue with one of his MX which had BIG MAILERS unable to pass greylisting. He asked me if I could write a utility to fetch SPF records and insert them into a PF table to whitelist MX for particular domains.\nCoincidentally, Aaron Poffenberger announced the next day that he had been working on a tool called spf_fetch https://github.com/akpoff/spf_fetch to do just that. The idea was nice however it couldn\u0026rsquo;t be committed to OpenBSD as is because it is written as a set of shell scripts and we wanted a piece of code that could be pledge()-ed.\nI contacted Aaron and told him I was going to be working on a C version based on the asr asynchronous resolver, and told him I would appreciate if he contributed since he had already started a similar project. A few weeks later, we had a work in progress spfwalk utility commited to my repository https://github.com/poolpOrg/spfwalk.\nsmtpctl spf walk # Instead of having a new utility committed to base, we decided to make it a subcommand of the smtpctl utility.\nThe main idea is that smtpctl spf walk will read a list of domains from stdin and output a list of IP addresses and ranges to stdout, allowing it to be used in scripts, to generate file lists, or to be piped directly into pfctl to feed a table.\nThis is still a work in progress, but it allows you to cat /etc/mail/BIGMAILERS.txt | smtpctl spf walk | pfctl -t spf-white -T add -f -, which is quite an improvement over having nothing to do it to start with :-)\nFinal words # sunil@ merged my utility to smtpctl, I made sure it ran unprivileged, so now we have an spf fetching utility that runs unprivileged and pledge()-ed in the base system.\nThis was committed a few days ago to OpenBSD -current, it requires testing because I\u0026rsquo;m pretty sure it has bugs still.\nWant to comment ? Open an issue on Github\n","date":"8 January 2018","permalink":"/posts/2018-01-08/spfwalk/","section":"Posts","summary":"TL;DR: deraadt@ thought it would be nice to have a spf fetch utility in base. Aaron Poffenberger wrote a shell-based `spf_fetch` utility. I wrote a C-based `spfwalk` utility that's `pledge()`-ed.","title":"spfwalk"},{"content":" TL;DR: Came very close to a burn out late 2016, had to quit former employer. Started working at another company early 2017. Became a student again, currently halfway through second year in psychology. Haven't done much opensource since early 2017, slowly resuming. Happy new year ! # I wish you a very happy new year 2018 and hope that you succeed in whatever you attempt.\nIt feels odd to write again to this blog considering the last blog post dates from September 2016, over a year ago.\nI didn\u0026rsquo;t lack interest in writing but I almost burnt out at my former company, which killed all my motivation to touch a computer outside of dayjob whatsoever, and had to quit my former employer to begin a new job somewhere else in early 2017.\nThis change, in addition to my numerous psychology night classes and my many many many hypnosis sessions and trainings, has left me with almost no spare time to boot a laptop at home.\nI must admit that I enjoyed putting some distance between computers and myself for a change. It allowed me to focus on things a bit less virtual and to enjoy the few times I actually booted an OpenBSD machine to write some code.\nIt would be a lie to say I didn\u0026rsquo;t do a thing, I actually have plenty of code that has moved forward: 2018 will be the year I start sharing back code ;-)\nI\u0026rsquo;ll keep this blog post short, there are two posts coming shortly, so in the meantime\u0026hellip;\nHAPPY NEW YEAR, ENJOY YOURSELVES, 2018 WILL BE A GOOD YEAR !\n","date":"8 January 2018","permalink":"/posts/2018-01-08/news-from-the-front/","section":"Posts","summary":"TL;DR: Came very close to a burn out late 2016, had to quit former employer. Started working at another company early 2017. Became a student again, currently halfway through second year in psychology.","title":"News from the front !"},{"content":" TL;DR: We just released OpenSMTPD 6.0.0 and it's quite different from former releases. Turns out most of the changes are not visible. A featureless release # I managed to wrap the 6.0.0 release yesterday.\nUnlike most of our releases, it comes out with almost no new feature.\nThe changelog fits in less than 10 lines as follows:\n- new fork+reexec model so each process has its own randomized memory space - logging format has been reworked - a \u0026quot;multi-line response\u0026quot; bug in the LMTP delivery backend has been fixed - connections concurrency limits have been bumped - artificial delaying in remote sessions have been reduced - dhparams option has been removed - dhe option has been added, supporting auto and legacy modes - smtp engine has been simplified - various cosmethic changes, code cleanup and documentation improvement Seems like a very productive slacking, however some of these changes turn out to be very interesting in terms of code simplification and security.\nIn this article I\u0026rsquo;ll focus on the fork+(re)exec model change.\nMemory space randomization # OpenBSD provides each process with a randomized memory space where pretty much everything ending up in memory is going to have different locations each time you run a new instance. All memory allocations using either malloc() or mmap() are randomized, the dynamic loader is randomized and recent work was committed so that libc gets randomized at each and every system start, the goal being that not only the place where the symbols are mapped in memory changes at every run, but that the symbols themselves get shuffled so that they are randomized one relative to another.\nThis basically means that if we both run the same program, there\u0026rsquo;s about zero chances that we will have a similar memory layout and that functions will be sitting at the same addresses.\nIn terms of security this makes exploitation particularly hard as there aren\u0026rsquo;t so many things an attacker can guess regarding the memory layout. She can\u0026rsquo;t study her own system to determine where things are located on yours, and every time the instance is restarted, former knowledge has gone away so that anything learnt will no longer be true once the program restarts.\nThis coupled with the fact that OpenBSD daemons tend to exit rather than attempt to rexecute a crashed process, level up the barrier quite a bit.\nHere, a program will simply print \u0026ldquo;the same\u0026rdquo; addresses every time it is executed:\n$ ./a.out 0x7f7ffffd4514 address of a stack variable 0x1061902ffe40 address of a malloc() allocation 0x105eeb000a64 address of a function within the program 0x105eeb000a6a address of the main function 0x1060fcf67a90 address of a libc function $ ./a.out 0x7f7ffffbb714 address of a stack variable 0xf7cbac277c0 address of a malloc() allocation 0xf7a05d00a64 address of a function within the program 0xf7a05d00a6a address of the main function 0xf7c34ba9a90 address of a libc function $ ./a.out 0x7f7ffffeab74 address of a stack variable 0x15b5c11cc5c0 address of a malloc() allocation 0x15b354800a64 address of a function within the program 0x15b354800a6a address of the main function 0x15b5ab116a90 address of a libc function $ The OpenSMTPD bootstrap # The idea for fork+exec in OpenSMTPD came from a discussion between deraadt@ and eric@ during a hackathon I hosted in Nantes mid-May this year.\nThe OpenSMTPD bootstrap process was quite simple:\nUpon executation, the parent process would read configuration, build a memory representation of it and would then create a bunch of socketpair() before fork()-ing all of its child processes.\nBy the virtue of fork(), all children would inherit the configuration as well as all the sockets necessary for IPC. They would only have to zero and free the bits of configuration they don\u0026rsquo;t need and close sockets meant to be used by the other children and be set.\nThis was nice and all, but could be improved.\nWhen a process fork()s, the child process gets an exact copy of the parent memory space. Any symbol in the parent address space is present at the same address in the child and only new allocations will cause the parent and child to diverge.\nSo everytime you run OpenSMTPD, the processes in your instance are different from the last time you ran it, however within a single instance the processes share a lot of addresses together. In the following example, note how all of the symbols share the same address in parent and child, except for the malloc() call that happened after fork():\n$ ./a.out === in parent 0x7f7ffffd3c44 address of a stack variable 0x8e8a457c040 address of a malloc() allocation BEFORE fork 0x8e8181631c0 address of a malloc() allocation AFTER fork 0x8e5ce100b34 address of a function within the program 0x8e5ce100c20 address of the main function 0x8e88231aa90 address of a libc function === in child 0x7f7ffffd3c44 address of a stack variable 0x8e8a457c040 address of a malloc() allocation BEFORE fork 0x8e8ad0311c0 address of a malloc() allocation AFTER fork 0x8e5ce100b34 address of a function within the program 0x8e5ce100c20 address of the main function 0x8e88231aa90 address of a libc function $ ./a.out === in parent 0x7f7ffffc0684 address of a stack variable 0x1fc99fc64100 address of a malloc() allocation BEFORE fork 0x1fc9443a6280 address of a malloc() allocation AFTER fork 0x1fc704400b34 address of a function within the program 0x1fc704400c20 address of the main function 0x1fc97c85aa90 address of a libc function === in child 0x7f7ffffc0684 address of a stack variable 0x1fc99fc64100 address of a malloc() allocation BEFORE fork 0x1fc90bb1e280 address of a malloc() allocation AFTER fork 0x1fc704400b34 address of a function within the program 0x1fc704400c20 address of the main function 0x1fc97c85aa90 address of a libc function $ While all the OpenSMTPD processes end up diverging by a great lot when it comes to locally allocated addresses, the problem is that if a process is compromised then it gives immediate information regarding all of the other processes.\nKnowing the memory address of the configuration in a child process will immediately tell an attacker where the configuration is located in all of the other processes, and if the proper amount of bugs is found this can prove useful.\nThe new fork()+(re)exec model # The exec family function will transform an existing process into a new process by basically reconstructing it from the new program.\nThis is what happens when your shell stops being a shell and becomes a ls instance for example. It forgets (pretty much) everything it knew about the former process and starts with its own brand new randomized memory space.\nSo deraadt@ suggested that if OpenSMTPD would not just fork() children but instead fork() them and reexecute the smtpd binary, then each of the children would have its own randomized memory space.\nThe idea itself is neat, however not so trivial to implement because when we reexec the whole \u0026ldquo;inherit configuration and descriptors\u0026rdquo; part goes away. It\u0026rsquo;s not just fork and exec, it\u0026rsquo;s fork and exec and figure a way for the parent to pass back all the information and descriptors back to the new post-fork instance so it is the new instance that allocates memory and decides where the information goes.\n$ ./a.out === in parent 0x7f7fffff6984 address of a stack variable 0x10a78e9cdc0 address of a malloc() allocation 0x10873500b84 address of a function within the program 0x10873500c57 address of the main function 0x10b6b748a90 address of a libc function === in child 0x7f7fffff8bc4 address of a stack variable 0x142a9fe228c0 address of a malloc() allocation 0x14289e900b84 address of a function within the program 0x14289e900c57 address of the main function 0x142ad94e0a90 address of a libc function $ ./a.out === in parent 0x7f7ffffbdee4 address of a stack variable 0x1f673b59d700 address of a malloc() allocation 0x1f6482500b84 address of a function within the program 0x1f6482500c57 address of the main function 0x1f66b1b71a90 address of a libc function === in child 0x7f7ffffdcc74 address of a stack variable 0xe10ddc66e40 address of a malloc() allocation 0xe0ec9f00b84 address of a function within the program 0xe0ec9f00c57 address of the main function 0xe10d3a5fa90 address of a libc function $ Eric reworked the entire bootstrap process so the parent reads configuration, creates all of the sockets, forks \u0026amp; execs all children in a special mode\u0026hellip;\n\u0026hellip; then have the parent pass each children only the bit of configuration they need and the descriptors they require for IPC.\nThis was really hard work and I\u0026rsquo;m amazed he managed to wrap it between the hackathon and the release freeze, this makes OpenSMTPD far more resistant to a whole lot new range of attacks.\nI\u0026rsquo;ll write about the smtp engine improvements in a further article, stay tuned.\n","date":"12 September 2016","permalink":"/posts/2016-09-12/opensmtpd-6.0.0-is-released/","section":"Posts","summary":"TL;DR: We just released OpenSMTPD 6.0.0 and it's quite different from former releases. Turns out most of the changes are not visible. A featureless release # I managed to wrap the 6.","title":"OpenSMTPD 6.0.0 is released !"},{"content":" TL;DR: OpenSMTPD on github no longer diverges from OpenBSD. Since last episode # Sometime this year, we released OpenSMTPD 5.7.1 which was the first version that shipped with the long-awaited experimental filters support. The result of over a year and a half of very very deep refactor in our IO layer and the addition of a very elegant but also very tricky new API.\nThis refactor took place outside the OpenBSD tree because it required breaking a lot of code for extended period of times, which spanned over 3 OpenBSD releases. It was near impossible to work in tree as we could not break the base daemon and leave it in a weird state for too long, and the changes were so invasive that it was also impossible to backout parts of it when a regression was spotted.\nWe eventually reached a point where it was very stable but also diverged greatly from the daemon in base. We made a \u0026ldquo;checkpoint\u0026rdquo; by releasing 5.7.1 separately of OpenBSD and the goal was now to bring the daemon in tree up to date.\nThe release was quickly followed by 5.7.2 and 5.7.3, fixing security issues that were uncovered by Qualys during a coordinated audit and an issue spotted by an independant user.\nBringing back OpenSMTPD to the OpenBSD tree # Working off-tree came with a lot of burden, first because it cut us off from the other OpenBSD hackers but also because it came with extra work.\nFor instance, during an internal audit we spotted a way to trigger a denial with our control socket. This applied to both the version in OpenBSD and Github. Both versions diverged quite a lot so the fix to one would not apply to the other and of course it took place at worst point in time, when OpenBSD was so close to its new release that 3 stable versions were tagged forcing us to test the fix on all stable branches + current branch + github\u0026rsquo;s master + github\u0026rsquo;s portable.\nNot to mention that every time someone submitted a pull request on Github we had to adapt it to apply on the older version, and every time an OpenBSD hacker made a cleanup to the older version, we had to adapt and apply to Github. This was no fun at all, many of the contributions not applying cleanly and increasing a gap.\nOn another hand, it was very difficult to bring the changes back to OpenBSD as a HUGE part of it was a refactor that could not be split into reviewable minidiffs for other to look at it. Many small features could be split, but given that they were written on top of the refactor, this implied that even taken independantly, they had to be adapted to apply to the old code. This was considerable work that caused us to defer, defer, defer, \u0026hellip;\nFinally, we reached a point where it was no longer possible to defer and I dived back into this with help from sunil@, jung@ and millert@. I took a few days off, spent full-time on this, and we managed to make huge progress that allowed us to fully bring OpenBSD\u0026rsquo;s smtpd up-to-date after a two-months effort of spliting the delta into small reviewable chunks.\nI owe all of them a very large amount of beer.\nOpenBSD\u0026rsquo;s pledge() # One of the most notable change that has taken place in the last months is that we fully integrated the new OpenBSD pledge() system-call.\nBut what is pledge() ?\npledge() is a new system call that will appear with OpenBSD 5.9. What it does is allow a program to promise it will only use certain features and allow the system to terminate the process if it doesn\u0026rsquo;t stick to them.\nFor example, let\u0026rsquo;s imagine the following program which creates a directory:\n#include \u0026lt;sys/stat.h\u0026gt; #include \u0026lt;unistd.h\u0026gt; int main(int argc, char *argv[]) { mkdir(\u0026#34;/tmp/test\u0026#34;, 0700); return 0; } If you execute it, the process creates the directory as expected:\n$ cc test.c $ ./a.out $ ls -l /tmp|grep test drwx------ 2 gilles wheel 512 Dec 22 10:44 test $ But now, let\u0026rsquo;s imagine that the program actually makes a promise that it will only use stdio features, essentially IO on previously allocated descriptors and memory allocation so the libc basic stuff works:\n#include \u0026lt;sys/stat.h\u0026gt; #include \u0026lt;unistd.h\u0026gt; int main(int argc, char *argv[]) { pledge(\u0026#34;stdio\u0026#34;, NULL); mkdir(\u0026#34;/tmp/test\u0026#34;, 0700); return 0; } We run the program again:\n$ cc test.c $ ./a.out Abort trap (core dumped) $ But this time, when the process calls mkdir() the kernel detects that it has violated the promises made in the program and aborts execution.\nSo pledge() is a security feature ? # At first it looks like pledge() is just a security feature to mitigate risks and it DOES bring a very effective mitigation mechanism: an attacker who wants to run arbitrary code in a compromised process still needs to respect promises made by the program\u0026hellip; in that arbitrary code\u0026hellip;\nIn practice, it\u0026rsquo;s far from being JUST a security feature, it is also a quality feature.\nWhen a developer converts a program to pledge(), he needs to understand what features are being used, why they are being used, when they are being used and where they are being used. If there\u0026rsquo;s a wrong assumption, the process dies. In addition, the promises can be shortened at runtime to restrict further.\nIn a project like OpenSMTPD, this is particularly interesting because it comes with a multi-process design where different processes do specific things, each being able to declare its own list of very reduced promises.\nThe daemon starts with a huge promise list to be able to setup its sockets and processes, chroot, change privileges, do some filesystem stuff, then processes rapidly reduce their promises down to what they will really use at runtime: in many cases stdio.\nIntuitively, having worked on this code base for so long, I had a pretty clear idea of what the processes were doing at runtime so I thought it would be very straightforward and would \u0026ldquo;just work (c)\u0026rdquo;.\nIn practice, pledge() pinpointed a few cases where my promises were not held and had me investigate WHY a component needed a particular feature at runtime, when it was not part of the promises I intuitively thought off.\nMost cases were really simple to solve and a matter of doing a call just a bit too late in the code path, but in some cases it really helped refactor code to get rid of a feature that wasn\u0026rsquo;t really needed.\npledge() helps you write better code by getting a very clear feedback that a piece of code is doing more than you think it does.\nWhat\u0026rsquo;s next ? # The code in Github is now a clone of the OpenBSD CVS tree and the only delta is a version check so that it can build on OpenBSD releases that do not have the pledge() system call.\nThe portable branch has been sync-ed with the CVS tree but it currently does not build as it requires a bit of plumbing. We are currently working on this so we can resume publishing snapshots.\nWe\u0026rsquo;re going to synchronize our releases with OpenBSD again so you can expect OpenSMTPD 5.9.1 to be released when OpenBSD 5.9 is released, though there is a chance we release an intermediate version so people can run a version that benefits from all the cleanups, improvements and safer \u0026ldquo;enqueuer\u0026rdquo; before the 5.9.1 release in May.\n","date":"22 December 2015","permalink":"/posts/2015-12-22/home-sweet-home/","section":"Posts","summary":"TL;DR: OpenSMTPD on github no longer diverges from OpenBSD. Since last episode # Sometime this year, we released OpenSMTPD 5.7.1 which was the first version that shipped with the long-awaited experimental filters support.","title":"Home, sweet home"},{"content":" TL;DR: yeeeees, filters are coming. don't believe us ? here's an example. be patient. On my death bed # On my death bed, when my life flashes before my eyes and I start recalling what people have told me during my (hopefully long) lifetime, these sentences will single out:\n\u0026ldquo;When will OpenSMTPD support filters ? I need it.\u0026rdquo;\nNot that it carries a philosophical meaning that will have taken me a lifetime of thinking, but because for the last three years I have been hearing this every time I met someone IRL and discussed OpenSMTPD, I have read it on our Github issues tracker, on GTalk, on IRC, on Twitter, on Facebook, on random forums, and just this week three times in my mailbox. I can honestly say that during these three years, OpenSMTPD filters have been mentioned to me hundreds of times without exaggerating. Yup. That\u0026rsquo;s what I\u0026rsquo;ll remember :-/\nA short reminder # The SMTP server accepts connections from clients, it establishes sessions and processes SMTP transactions which then eventually result in a mail being accepted for delivery or rejected. This decision isn\u0026rsquo;t a very smart process, the server looks at a ruleset that basically lists which domains are to be accepted or rejected, and it tries to locate a user under these domains. That\u0026rsquo;s it, more or less.\nSo what are filters useful for ? # Filters are small pieces of code that can be used to modify the behavior of an SMTP session and decide to accept, reject or modify the client input based on decisions that are not related to that ruleset. For instance, a filter may decide that besides the ruleset, it also wants to check if a domain is not part of a public blacklist. It may want to decide that a particular client should not be able to send mail because according to some database, it has exceeded a quota. It may simply want to check at the content of the DATA phase and try to detect some keywords typical of spam or even alter it to remove (bad bad bad) or append some headers.\nTechnically these behavior could be implemented in the daemon itself, but providing an API for filters allows this code to remain outside of the daemon code, keeping it simple, and allowing anyone to customize it for their needs without needing us to cooperate.\nFilters goals # Filters are a feature available in various MTA, this is not an OpenSMTPD thing and that\u0026rsquo;s why so many people have requested it. They expect the software to be extensible through filters because there are many tools out there that they could plug somehow to make it fit their use case. These tools are usable with the other MTA, so they should be usable with ours.\nThere are different ways of achieving a filter API, some relying on forked processes with a common convention to read on standard input and write on standard output, others providing a set of functions to do the work. Some as standalone filters, others as shared objects or libraries importing the features inside the daemon code base.\nThe most commonly used is the milters API which was written for a competitor software and adapted to other competitors with varying degrees of success and completeness. It is mature, has a wide range of filters written for it, and allows doing pretty much anything. When we started thinking about how we\u0026rsquo;d integrate filters, this was the first API we looked at as we could have a reference implementation to compare ours to and we would benefit from the plethora of filters that were already available.\nThen, we changed our mind: it doesn\u0026rsquo;t do what we want and it doesn\u0026rsquo;t fit well in our asynchronous model, it\u0026rsquo;s going to be a pain in the ass. We don\u0026rsquo;t want to grab an existing API and modify our design to have it fit it, we want an API that just fits in our design.\nWhat we need is an API that\u0026rsquo;s fully asynchronous, that\u0026rsquo;s highly flexible while not bloated and that\u0026rsquo;s dead easy. It needs to be so flexible that we can implement bindings to other languages than C, or even a milter interface to translate milter calls to ours, and it needs to be so easy that it doesn\u0026rsquo;t require hours to get a basic filter working even if using the API for the first time.\nFilters design # So, I did a bit of research on how we would implement this API and after a while came up with these requirements:\nfilters should not be libraries or shared objects, they should be standalone executables running in their own memory space; they shouldn\u0026rsquo;t be fork()-ed at runtime because it would crush performances; they should be able to be individually chroot()-ed; they should be able to run with their very own privileges, isolated one from another if needed; whatever their dependencies, the daemon should not have to care, it should not know anything besides its filter protocol; Long story short, while it would be considerably easier for us, we don\u0026rsquo;t want filters to be plugged as shared objects. The problem with that approach is that the filter would then share the daemon memory space and that\u0026rsquo;s something we don\u0026rsquo;t want to happen. The filters should be isolated in their own memory space and they should know nothing about the server except what the server hands them. That way, if a filter suffers from a bug allowing leak of random memory areas, it will not be able to leak information that is part of the server. This concern predates by far the Heartbleed catastrophe but since some people back then thought it was not a real problem, now they have a perfect example of why it was ;-)\nAnyways, to respect these requirements and given that the daemon itself drops privileges at startup, filters had to become daemons of their own started with the server and communicating with it during its lifetime. Fortunately this is something we\u0026rsquo;re quite familiar with, it is basically OpenSMTPD\u0026rsquo;s model: parent starts, sets up communication channels with imsg framework and starts other processes.\nWe decided to build the filter API on that model, then we realized that while we were familiar with it, it would be far too complex for users to deal with this whole fork() / imsg thing. So we wrote a \u0026ldquo;filter glue\u0026rdquo; which takes care of setting everything up and running so that a user can write a filter by focusing only on its own functions.\nstatic int on_connect(uint64_t id, struct filter_connect *conn) { log_warnx(\u0026#34;filter-stub: OHAI, SOMEONE CONNECTED !\u0026#34;); return (1); } int main(int argc, char **argv) { int\tch; log_init(-1); while ((ch = getopt(argc, argv, \u0026#34;\u0026#34;)) != -1) { switch (ch) { default: log_warnx(\u0026#34;warn: filter-stub: bad option\u0026#34;); return (1); } /* NOTREACHED */ } argc -= optind; argv += optind; /* register a callback for \u0026#34;on_connect\u0026#34; event */ filter_api_on_connect(on_connect); /* start doing stuff */ filter_api_loop(); log_warnx(\u0026#34;warn: filter-stub: exiting\u0026#34;); return (1); } As shown in this example, the main function only calls filter_api_on_connect() to let it know that it wants on_connect() to be called when there\u0026rsquo;s a connection. Then it calls filter_api_loop() to actually start processing events.\nThe whole \u0026ldquo;setup a channel with the daemon, fork and maintain state\u0026rdquo; is completely hidden within the API layer and transparent to the user. The filter API exposes a unique session identifier and per-hook structures filled with the data available in that phase, the filter is free to maintain its own data structures using the identifier to pass data from a hook to another.\nIt doesn\u0026rsquo;t get much simpler.\nBut that\u0026rsquo;s C ?! # Of course, it\u0026rsquo;s C, what else ?\nThe API is written and exposed in C allowing C programmers to write native filters. We want to make it possible to write them in other languages but our language of choice and the one we will use for any filter that may be shipped with the daemon remains C.\nSo from there, if we want to allow filters to be written in different languages, we have two ways:\neither we provide a binding, which is basically a reimplementation of our API in that language; or we provide bridge filters, which are basically proxies that will convert calls in the end-user language to C calls;\nBoth are slightly different, they achieve the same, but it seemed to me a better approach to write bridge filters that keeps a binding of only the few exposed API calls, than having to reimplement the entire API glue. Also, if we provide a \u0026ldquo;kind of official\u0026rdquo; bridge, it will benefit everyone ultimately.\nSo how hard is it to write a bridge ? # It really depends on what the target language is. For instance, I had a working filter-python bridge in less than an hour without having ever used the python API before. The filter-perl bridge took a few hours. The code to make the bridge itself is rather short, it is simply a matter of mapping a callback to a translation function. For example, if we look at our filter-python and how it implements on_connect(), we would see something like this (just focusing on the interesting parts):\nstatic int on_connect(uint64_t id, struct filter_connect conn) { PyObject\tpy_args; PyObject\tpy_ret; PyObject\tpy_id; PyObject\tpy_local; PyObject\tpy_remote; PyObject *py_hostname; /* we translate our parameters to types that python can understand */ py_args = PyTuple_New(4); py_id = PyLong_FromUnsignedLongLong(id); py_local = PyString_FromString(filter_api_sockaddr_to_text((struct sockaddr *)\u0026amp;conn-\u0026gt;local)); py_remote = PyString_FromString(filter_api_sockaddr_to_text((struct sockaddr *)\u0026amp;conn-\u0026gt;remote)); py_hostname = PyString_FromString(conn-\u0026gt;hostname); /* we prepare for calling the on_connect function inside the python script */ PyTuple_SetItem(py_args, 0, py_id); PyTuple_SetItem(py_args, 1, py_local); PyTuple_SetItem(py_args, 2, py_remote); PyTuple_SetItem(py_args, 3, py_hostname); /* we let python do the work and give us back a return value */ py_ret = PyObject_CallObject(py_on_connect, py_args); Py_DECREF(py_args); if (py_ret == NULL) { PyErr_Print(); log_warnx(\u0026#34;warn: filter-python: call to on_connect handler failed\u0026#34;); exit(1); } return (1); } int main(int argc, char *argv[]) { /* we don\u0026#39;t care what\u0026#39;s above */ /* we look if there\u0026#39;s a symbol \u0026#34;on_connect\u0026#34; defined in the end-user filter */ py_on_connect = PyObject_GetAttrString(module, \u0026#34;on_connect\u0026#34;); /* if there\u0026#39;s one and it\u0026#39;s a function, we register our translating callback */ if (py_on_connect \u0026amp;\u0026amp; PyCallable_Check(py_on_connect)) filter_api_on_connect(on_connect); /* we don\u0026#39;t care what\u0026#39;s below */ } Just for the purpose of comparing with another bridge, here\u0026rsquo;s the filter-perl bridge, doing the exact same thing:\nstatic int on_connect(uint64_t id, struct filter_connect *conn) { const char\t*local; const char\t*remote; local = filter_api_sockaddr_to_text((struct sockaddr *)\u0026amp;conn-\u0026gt;local); remote = filter_api_sockaddr_to_text((struct sockaddr *)\u0026amp;conn-\u0026gt;remote); /* looks simpler than python ? I hid the gross parts in call_sub_sv() ;-) */ call_sub_sv((SV *)pl_on_connect, \u0026#34;%i%s%s%s\u0026#34;, id, local, remote, conn-\u0026gt;hostname); return filter_api_accept(id); } int main(int argc, char *argv[]) { /* we don\u0026#39;t care what\u0026#39;s above */ /* we look for a function \u0026#34;on_connect\u0026#34; defined in the end-user filter and register a callback */ if ((pl_on_connect = perl_get_cv(\u0026#34;on_connect\u0026#34;, FALSE))) filter_api_on_connect(on_connect); /* we don\u0026#39;t care what\u0026#39;s below */ } As you can see, adding support for writing filters in another language is not really a big challenge, I could write two bridges in an afternoon without having played with either one of the C/API for these languages. I\u0026rsquo;m sure someone else could have completed them faster than I did.\nNow, anyone willing to write a filter in perl or python can simply install the bridges then focus on writing the filter script itself in their language of choice. Here\u0026rsquo;s an example for Python:\nimport filter def on_connect(session, local_addr, remote_addr, hostname): print \u0026#34;on_connect:\u0026#34;, hex(session), local_addr, remote_addr, hostname return filter.accept(id) And here\u0026rsquo;s one for Perl:\nuse strict; use warnings; sub on_connect { my ($id, $l, $r, $h) = @_; return smtpd::filter_api_accept $id; } The nice part is that this comes with a very very very low overhead, the interpreter is started only once and the performances should be near native no matter which language is used.\nWhat about milters ? # Well, it should be possible to implement a milter bridge that can translate milter API calls to our API using the same technique as for translating python and perl calls to our API. There may be some changes required in our API, but we\u0026rsquo;re not opposed to that and if someone stepped up to write a milter bridge, then I would definitely help by adapting the API where needed\u0026hellip; as long as it remains simple and clean.\nNow, as far as I am concerned\u0026hellip;\nI have absolutely no interest in a milter interface myself as I will be using the native one, so clearly this is not even going to hit my todo list. In my opinion, having native filters is a much better goal than trying to make filters written for another MTA work with ours, I suspect there will always be corner cases that will turn this into a nightmare when the same effort could be put in writing quality native filters.\nIf someone wants to spend time trying to make milters run on OpenSMTPD, well, good luck and may you find entertainment in that journey !\nSo, what\u0026rsquo;s the state of filters ? # The filter API works. It is not done, it still has bugs, it still lacks features, but it\u0026rsquo;s in a state where you can actually write a filter and have it process all kinds of events and tell the SMTP server to reject or accept based on filter decision. I wrote tiny test scripts to verify that I could write filters to limit the number of connections from a same user, reject upon some keywords, alter content, skip headers, etc\u0026hellip; all of this works.\nJust as an example, here\u0026rsquo;s a filter I wrote this morning, all it does is declare callbacks for every hook and print the parameters it receives. I wrote it in Python because it\u0026rsquo;s easier for everyone to figure out what it does, but you could do the same in C or Perl:\nimport filter def on_connect(id, local_ip, remote_ip, hostname): print \u0026#34;on_connect:\u0026#34;, hex(id), local_ip, remote_ip, hostname return filter.accept(id) def on_helo(id, helo): print \u0026#34;on_helo:\u0026#34;, hex(id), helo return filter.accept(id) def on_mail(id, mail): print \u0026#34;on_mail:\u0026#34;, hex(id), mail return filter.accept(id) def on_rcpt(id, rcpt): print \u0026#34;on_rcpt:\u0026#34;, hex(id), rcpt return filter.accept(id) def on_data(id): print \u0026#34;on_data:\u0026#34;, hex(id) return filter.accept(id) def on_dataline(id, line): print \u0026#34;on_dataline:\u0026#34;, hex(id), line return filter.writeln(id, line) def on_eom(id, size): print \u0026#34;on_eom:\u0026#34;, hex(id), size return filter.accept(id) def on_commit(id): print \u0026#34;on_commit:\u0026#34;, hex(id) return filter.accept(id) def on_rollback(id): print \u0026#34;on_rollback:\u0026#34;, hex(id) return filter.accept(id) def on_disconnect(id): print \u0026#34;on_disconnect:\u0026#34;, hex(id) With this filter plugged in, the following SMTP session:\n220 debug.poolp.org ESMTP OpenSMTPD EHLO myself 250-debug.poolp.org Hello myself [127.0.0.1], pleased to meet you 250-8BITMIME 250-ENHANCEDSTATUSCODES 250-SIZE 36700160 250-DSN 250 HELP MAIL FROM:\u0026lt;gilles\u0026gt; 250 2.0.0: Ok RCPT TO:\u0026lt;gilles\u0026gt; 250 2.1.5 Destination address valid: Recipient ok DATA 354 Enter mail, end with \u0026quot;.\u0026quot; on a line by itself Subject: ohai ! test . 250 2.0.0: 2fb4c3b7 Message accepted for delivery Results in the following log on the server running in debug mode, with all filter callbacks triggered:\ndebug: smtp: new client on listener: 0x1c9bbad31000 smtp-in: New session a7bc78b6177f09b5 from host localhost [127.0.0.1] on_connect: 0xa7bc78b6177f09b5L 127.0.0.1 127.0.0.1 localhost on_helo: 0xa7bc78b6177f09b5L myself on_mail: 0xa7bc78b6177f09b5L gilles@debug.poolp.org on_rcpt: 0xa7bc78b6177f09b5L gilles@debug.poolp.org on_data: 0xa7bc78b6177f09b5L smtp: 0x1c9c37af3000: fd 5 from queue smtp: 0x1c9c37af3000: fd 7 from filter on_dataline: 0xa7bc78b6177f09b5L Received: from myself (localhost [127.0.0.1]); on_dataline: 0xa7bc78b6177f09b5L by debug.poolp.org (OpenSMTPD) with ESMTP id 2fb4c3b7; on_dataline: 0xa7bc78b6177f09b5L for \u0026lt;gilles@debug.poolp.org\u0026gt;; on_dataline: 0xa7bc78b6177f09b5L Fri, 12 Dec 2014 15:48:42 +0100 (CET) on_dataline: 0xa7bc78b6177f09b5L Subject: ohai ! on_dataline: 0xa7bc78b6177f09b5L on_dataline: 0xa7bc78b6177f09b5L test debug: smtp: 0x1c9c37af3000: data io done (195 bytes) on_eom: 0xa7bc78b6177f09b5L 195 filter: eom not received yet debug: 0x1c9c37af3000: end of message, msgflags=0x0000 debug: scheduler: evp:2fb4c3b7f8d5ee1d scheduled (mda) smtp-in: Accepted message 2fb4c3b7 on session a7bc78b6177f09b5: from=\u0026lt;gilles@debug.poolp.org\u0026gt;, to=\u0026lt;gilles@debug.poolp.org\u0026gt;, size=195, ndest=1, proto=ESMTP on_commit: 0xa7bc78b6177f09b5L CAN I HAZ IT FOR REAL !!? # Filters are not stable yet, just this morning I have fixed a bug that lead to a crash and another user has found a corner case which causes the filter glue to crash when a filter forks. They\u0026rsquo;re not just disabled, but also removed from our stable release. Unlike a while ago, there\u0026rsquo;s no way to activate them in a stable release. Gone. Bye-bye. Ma3 salam.\nFilters support is only available in our development master and portable branches which are mirrored on Github. It is also available in our snapshots, which we publish every now and then from these branches. However\u0026hellip; they are not meant for general use.\nPeople keep complaining that we didn\u0026rsquo;t document the API, but what they fail to grasp is that:\nwe don\u0026rsquo;t write documentation before actual code, we write code then document; we don\u0026rsquo;t document code that is subject to heavy changes, it\u0026rsquo;s not meant to be used, it should be ignored, you should pretend it doesn\u0026rsquo;t exist; we have a working API but it\u0026rsquo;s not done, it has issues, we don\u0026rsquo;t want people to use it;\nAt the time being, we leave it undocumented because it makes it harder for people who can\u0026rsquo;t read code to actually play with it, hit issues that aren\u0026rsquo;t bugs and confuse us about what is a real issue and what is a user bug.\nWe want people to help us squash the bugs in this API, however we can\u0026rsquo;t hold hands of everyone willing to give this API a try and write a tiny filter. So we decided to keep the API undocumented for now and only help those that can read code and figure things out on their own, we want to keep the noise ratio to the lowest. If you don\u0026rsquo;t know C, I would discourage you to play with the API yet, even if you intend to write filters using a bridge. You will be on your own and unlikely to go much far.\nI wanted to show code today because so many people keep requesting a filter API and we\u0026rsquo;ve been saying that \u0026ldquo;it\u0026rsquo;s a work in progress\u0026rdquo; for so long that it looked like vaporware. I hope this post clarifies that we actually HAVE an API, a usable one on top of that, but it\u0026rsquo;s just not ready and we\u0026rsquo;re too close to having what we want that we\u0026rsquo;re not going to ruin it all by rushing to deliver something half-baked.\nNote that I didn\u0026rsquo;t show how you could actually plug it, this was done on purpose. If you manage to find by yourself, feel free to help us move the API forward by starting to write your own filters. If you need help to activate, you\u0026rsquo;ve failed the easiest test, by all means stay away from this API until it has stabilized.\nAWWWWW :-( # Don\u0026rsquo;t be sad :-(\nWe\u0026rsquo;re working precisely on getting this API stable enough that we can ship it with our next major release. It will be considered experimental, but it will be shipped and you will be able to play with it.\nBefore you ask, no we don\u0026rsquo;t know when that release would be. Unlike other releases which we can plan on a regular schedule, this one is so invasive and the feature is so big that we will delay for as long as it takes before planning the release. The only thing delaying the major release is the filter code.\nAnd don\u0026rsquo;t get confused, there will be an OpenSMTPD-5.4.4 release soon\u0026hellip; but this is a minor release, not the one with filters.\n","date":"12 December 2014","permalink":"/posts/2014-12-12/the-state-of-filters/","section":"Posts","summary":"TL;DR: yeeeees, filters are coming. don't believe us ? here's an example. be patient. On my death bed # On my death bed, when my life flashes before my eyes and I start recalling what people have told me during my (hopefully long) lifetime, these sentences will single out:","title":"The state of filters"},{"content":"EHLO,\nThis is the third post of a series about OpenSMTPD improvements that have taken place since this summer.\nContent altering # For a long time, we have developed OpenSMTPD with a strict rule that the daemon should not alter DATA (as in the DATA SMTP phase) in any way.\nThe rationale was that by enforcing that rule, the message writing was simplified as the smtp process would simply read data from a client and write it, without any post-processing, to a file descriptor. We didn\u0026rsquo;t want to prevent the writing of data modifiers, but we wanted to move this feature out of the daemon and into the filtering API which would allow any kind of rewriting outside the daemon memory space and with different privileges if we wanted to.\nThe other rationale was that by avoiding adding hooks to alter content we were receiving, we made it harder on us and contributors to go lazy and implement every feature out there in the daemon just because it was easier through the hooks. When we wanted to implement a feature, we would immediately ask ourselves if it could be done outside the daemon scope before eventually adding it.\nBesides the occasional complaints that we did not support address masquerading (yet), this did not really cause any issue with our users running a wide range of MUA and we decided to maintain that rule, convince people that masquerading was really a filter thing and that it would become available as soon as our filter API had stabilized.\nThen we became default MTA on OpenBSD # We started getting complaints from hackers that, although without a value, the BCC header was being emitted by OpenSMTPD.\nIt seemed rather strange because clients typically strip them from the DATA phase and we never actually receive them. It turned out that some older MUA did not do that stripping and I had to add a basic check to skip BCC line(s) if any appeared in the DATA phase before the message content. I was not too happy about that change, and a few other hackers/people either as it turns out, but it was not really altering content, simply discarding content that we were not meant to receive in the first place so\u0026hellip;\nThen came another complaint: some hackers were receiving mails from some other hackers \u0026hellip; but the mail had a From header with the local domain instead of the sender domain.\nThis also seemed strange because the \u0026ldquo;local domain\u0026rdquo; is not necessarily the \u0026ldquo;sender domain\u0026rdquo;. For instance, if you mail me at @opensmtpd.org, the destination domain will be @opensmtpd.org but the local domain will be poolp.org as that\u0026rsquo;s the local domain of the MX machine. This could only mean that the domain had been inserted by the receiving MX because it was missing.\nNow, when a mail goes out, it can take two paths:\nit can go through a SMTP connection to localhost; or it can use the local enqueuer, send-mail or sendmail, which are actually both mapped to smtpctl working in enqueue mode; Many MUA support both modes, but usually rely on the second mode by default when ran locally except for a few modern ones that need to be configured explicitly to do so.\nWhat bothered me was that modern MUA usually append a domain so they could pretty much all be ruled out, and older MUA tend to rely on the local enqueuer which has code to insert the local domain if it is missing. In these two cases, the mail would have not been emitted without a domain.\nSo the logical explanation was that the sender used a MUA that both bypassed the local enqueuer AND did emit a From without a domain. After a bit of testing, this proved to be the case and it brought hell on me.\nAddresses Parsing # It became clear that if we wanted to continue supporting these older MUA, some basic rewriting was required in the daemon and were no longer just a feature but a mandatory part of the session processing. Without this, an OpenSMTPD server could send mails to another MTA such as Sendmail or Postfix and the recipient would get unexpected headers due to the local rewriting performed by them.\nAs a quick fix, I wrote a small addresses parser that would extract From, To and Cc fields. It would then attempt to parse the addresses in them, check if a domain was missing and append the local domain in that case. This proved to be a very very bad idea because unlike email addresses in the SMTP protocol, the email addresses inside a message can be written in many ways and are a nightmare to parse. Addresses can span on multiple lines, have comments inside them, they can have brackets or not, they can be surrounded by components, parts of the representation may use different encodings and we even ran into representations such as \u0026ldquo;Gilles Chehade\u0026rdquo; which goes beyond mind-fuckery.\nThe issue was not just the parsing, but also that once appended the address had to be rendered back so not only I had to find a way to parse the way-too-many fucked up representations into a structure I could work on, but I had to be able to render the fucked up representation back with the domain inserted in them.\nI took a first approach of not doing the replacing in place, but rather constructing a list of sanitized addresses that would be output as a multi-line header. The addresses would be parsed into a common structure and rendered so that (superfluous spaces intended):\nFrom: \u0026quot;Gilles Chehade\u0026quot; gilles, \u0026quot;Eric Faurot\u0026quot; eric@poolp.org, \u0026quot;Charles Longeau\u0026quot; \u0026lt; chl \u0026gt; would turn into:\nFrom: \u0026quot;Gilles Chehade\u0026quot; \u0026lt;gilles@poolp.org\u0026gt;, \u0026quot;Eric Faurot\u0026quot; \u0026lt;eric@poolp.org\u0026gt;, \u0026quot;Charles Longeau\u0026quot; \u0026lt;chl@poolp.org\u0026gt; But this lead to another round of complaints that the rewriting was altering the message structure, not only did it expand multi-line (which is RFC compliant) but \u0026lsquo;\u0026lt;\u0026rsquo; and \u0026lsquo;\u0026gt;\u0026rsquo; appeared surrounding the addresses even if they were lacking in the original header.\nIt became really clear at this point that any attempt at taking a \u0026ldquo;parse-then-render\u0026rdquo; approach was not going to work.\nMessage Parser # After a discussion with eric@, I decided to take a different approach and split the problem in two smaller problems.\nThe first one is locating the specific parts of a message that need to be altered, in this case some specific headers. The second one is to actually do the appending somehow.\nLocating the specific parts of a message is a seemingly simple issue, however it raises some concerns of its own. The DATA arrives line by line and the headers may span on multiple lines, therefore the processing can\u0026rsquo;t be done as lines arrive but requires a bit of context and knowing which line is the last one \u0026hellip; when this information is actually carried by the next one which has not yet been received.\nThe simplest way to deal with the problem is to take a full message parser, such as the one we have in the enqueuer, grab the part that deals with the headers and feed it with the full headers for the session in progress.\nTo achieve this, we could either buffer the headers in memory during the session then work on that buffer or, since we\u0026rsquo;re writing the full message to a file anyway, have the message parser work on that file before it is committed to the queue. Unfortunately, neither of these works with us:\nKeeping full headers in memory paves the road for resources starvation attacks. On a default install, a message can be up to 35MB with many people bumping that value further. Depending on your operating system, OpenSMTPD may accept several thousands of concurrent clients. Now, if we went that path, an evil human being could simply craft very large messages consisting solely of headers and have thousands of clients emitting them without sending the final \u0026ldquo;.\u0026rdquo; but waiting for the server timeout to trigger instead. You get the idea why this is not a good idea.\nWorking with the file we saved the message to is slightly better, however two issues pop-up.\nFirst, we can\u0026rsquo;t do in-place editing so we will basically be rewriting the entire content to another file. For large messages, the performances penalty will be very visible. Then, we have an atomicity requirement, we can\u0026rsquo;t acknowledge the client that we have accepted the message until it has been committed to queue, so between his final \u0026ldquo;.\u0026rdquo; and our acknowledgement, he will have to wait for the entire message to be processed, copied to the other file and committed to queue\u0026hellip; during that time, he will be consuming a session slot preventing another client from being handled. If he gets lucky, he may even have a chance to hit a timeout causing his disconnect while the server will abort processing of his message and trash it.\nSo, nope, working with full message or full headers is not the way\u0026hellip; data has to be processed as a stream.\nStream Message Parser # I discussed the idea with eric@ a bit and then came up with a prototype for a stream message parser that we would plug in the SMTP session.\nvoid rfc2822_parser_init(struct rfc2822_parser *); int rfc2822_parser_feed(struct rfc2822_parser *, const char *); void rfc2822_parser_reset(struct rfc2822_parser *); void rfc2822_parser_release(struct rfc2822_parser *); int rfc2822_header_callback(struct rfc2822_parser *, const char *, void (*)(const struct rfc2822_header *, void *), void *); void rfc2822_header_default_callback(struct rfc2822_parser *, void (*)(const struct rfc2822_header *, void *), void *); void rfc2822_body_callback(struct rfc2822_parser *, void (*)(const char *, void *), void *); Before, the workflow would basically go like \u0026ldquo;read a line from this client, write that line to this file\u0026rdquo;.\nNow, the SMTP server will associate a parser context to every session and register a set of callbacks. The default callbacks will simply receive the line as parameter and write it to the file, doing exactly what was done before. However, they now get a chance to modify it somehow before writing it.\nFor example, before the parser was integrated, the BCC remove code I had quicked-fix required adding code to the session reading function to detect if we were still in headers, if there was a continuation line, as well as explicit strcasecmp() checks to bypass the writing. It was not much, but it made it clear that the function would become a big pile of crap once we started adding other special cases.\nNow, it is as simple as registering a callback for that header:\nrfc2822_header_callback(\u0026amp;s-\u0026gt;rfc2822_parser, \u0026#34;bcc\u0026#34;, header_bcc_callback, s); And writing a callback that \u0026hellip; does nothing:\nstatic void header_bcc_callback(const struct rfc2822_header *hdr, void *arg) { } During a session, the lines read from the clients are passed to the rfc2822_parser_feed function which then takes care of all context handling, buffering and calling registered callbacks when needed. The callbacks will receive headers as a rfc2822_header structure holding a list of lines, and the lines can then either be written as is to a file or modified at will. A message crafted to exhaust resources by providing a single header with an insanely large number of lines will cause the parser to generate an error that the server will properly handle, so all cases solved.\nWith this in, the code in the SMTP session remains clear with no special cases and no \u0026ldquo;state\u0026rdquo; variables all over the place.\nSo how do we deal with addresses ? # Let\u0026rsquo;s get back to the main issue: fixing addresses.\nSo, the message parser has made it possible to extract specific headers and we could now do the following without having to kludge the session reading code:\nrfc2822_header_callback(\u0026amp;s-\u0026gt;rfc2822_parser, \u0026#34;from\u0026#34;, header_masquerade_callback, s); rfc2822_header_callback(\u0026amp;s-\u0026gt;rfc2822_parser, \u0026#34;to\u0026#34;, header_masquerade_callback, s); rfc2822_header_callback(\u0026amp;s-\u0026gt;rfc2822_parser, \u0026#34;cc\u0026#34;, header_masquerade_callback, s); But how do we implement header_masquerade_callback ?\nInstead of using the previous \u0026ldquo;parse-and-render\u0026rdquo; approach, I decided to give a try at another way: locate insert points.\nWhile it is hard to parse the header into a series of per-address structures that I can work with and render back, it is not too hard to parse the header into a series of buffers in which I can check if a domain is missing and locate where it should have been. This technique has the advantage that once the processing has been done, the buffers can be written back in sequence preserving the structure of the original message.\nFor instance:\nFrom: \u0026quot;Gilles Chehade\u0026quot; gilles, \u0026quot;Eric Faurot\u0026quot; eric@poolp.org, \u0026quot;Charles Longeau\u0026quot; \u0026lt; chl \u0026gt; Would now be parsed into three buffers containing (note that the various superfluous spaces were preserved):\n\u0026quot;Gilles Chehade\u0026quot; gilles \u0026quot;Eric Faurot\u0026quot; eric@poolp.org \u0026quot;Charles Longeau\u0026quot; \u0026lt; chl \u0026gt; The domain-appending logic would then go through the three buffers and determine:\nis a domain appending necessary ? if it is, where in the buffer should it be inserted ? The second buffer would be unchanged while the first and third buffers would result in an insert point right after gilles and right after chl. The code would then simply copy the buffer to another one up to that insert point, insert the domain, then copy the remaining. Once all buffers have been processed, they are written sequentially resulting in:\nFrom: \u0026quot;Gilles Chehade\u0026quot; gilles@poolp.org, \u0026quot;Eric Faurot\u0026quot; eric@poolp.org, \u0026quot;Charles Longeau\u0026quot; \u0026lt; chl@poolp.org \u0026gt; Again note that structure has been preserved, header has not be rendered multi-line and superfluous spaces in the original header are left as is.\nSo\u0026hellip; we\u0026rsquo;re all done ? # This has been committed several weeks ago and there\u0026rsquo;s been no complaints so I guess the solution was correct.\nWe\u0026rsquo;ve discussed about masquerading a bit and maybe we will reevaluate our position that it would be better off in a filter, after all \u0026hellip; the code to do the domain append is masquerading so if we\u0026rsquo;re going to do it for the incoming case we might as well make it handle the outgoing case too.\nHowever, our stance has not changed and while the message parser offers a lot of ease for in-daemon content altering, we still believe that the bulk of content altering is filters material. If only to benefit from privileges separation while parsing untrusted content.\n","date":"10 December 2014","permalink":"/posts/2014-12-10/some-opensmtpd-overview-part-3/","section":"Posts","summary":"EHLO,\nThis is the third post of a series about OpenSMTPD improvements that have taken place since this summer.\nContent altering # For a long time, we have developed OpenSMTPD with a strict rule that the daemon should not alter DATA (as in the DATA SMTP phase) in any way.","title":"Some OpenSMTPD overview, part 3"},{"content":" Why we killed the MFA (filter) process # For as long as I can remember, a process called MFA was created by OpenSMTPD at start time.\nMFA stood for \u0026ldquo;Mail Filter Agent\u0026rdquo; and the goal of that process was initially to take care of all filtering tasks ranging from filtering senders based on the ruleset matching to starting and controlling filters.\nAs time passed by, we figured the lookup process was better suited for ruleset matching and the MFA process became mostly idle, we renamed it to \u0026ldquo;filter\u0026rdquo; process since the only thing it had to do was take care of starting filters and making sure they had the proper environment.\nAt this point, we should have realized that this was unnecessary but this process had been with us for so long that it seemed just right. After all, we gave it a name, how do you kill something you named \u0026hellip;\nAnyways, as we improved the filter API design it became clear that the filter process kept getting in the way of us doing things in a simple way. We had all informations available at a place, yet we had to make an indirection and pass data back and forth just so the process could get a chance to acknowledge it didn\u0026rsquo;t care. If the filter process is no longer needed after starting the filters, why not have them started by another process which actually does something useful afterwards instead of staying idle ?\nEric offered to take the process down and while it was heart-breaking, I encouraged him to kill it with fire.\nThe pony process # There\u0026rsquo;s been complaints that the daemon was starting too many processes and that probably some could be merged.\nAfter a bit of brainstorming, we figured that the smtp process in charge of accepting incoming sessions, the mta process in charge of starting outgoing sessions and the delivery process in charge of delivering mail locally were a bit related: they were all unprivileged, running as _smtpd, chrooted to /var/empty and far from being busy even in scenarios where the server was busy itself.\nIn addition, they all shared a common design, they all had a notion of sessions following the same pattern and we figured that without too much efforts we could merge them into a single process.\nI did the initial work to merge them three into a single process and eric@ did another pass to cleanup the names of the IMSG we were passing to improve further.\nPretty neat, hu ? Indeed. However we faced another challenge: this change proved to be controversial.\nWhile pretty much everyone agrees that this was the way to go (actually I have no idea, I didn\u0026rsquo;t run a poll), some people were not too receptive to the name we gave that process: \u0026ldquo;pony\u0026rdquo;. It is apparently unprofessional and should we leave it that way, gosh so much $$$ are we going to lose.\nActually \u0026ldquo;pony\u0026rdquo; was not meant to stay, it had nothing to do with the pony trend on internet and the name was chosen because we couldn\u0026rsquo;t agree on a proper name for that process which did smtp, mta and delivery. Instead of spending two hours trying to find a name, I suggested we name it \u0026ldquo;pony express\u0026rdquo; as an hommage to the courageous messengers of the far west. If someone came up with a better name, we could just switch, I just didn\u0026rsquo;t want to be stuck on that meaningless task. Until now, the challenge remains unsolved.\nAlso, I promised blambert@ a pony during a hackathon in Budapest so \u0026hellip; one stone, two birds.\nThe klondike process # As you may have seen, this year was a terrible shot for the OpenSSL project. The Heartbleed vulnerability has shaken the internet and it even came out with a logo.\nA few months earlier, for unrelated reasons I had started working on reducing the amount of time a key is exposed in the memory-space of a network facing process. I didn\u0026rsquo;t solve the issue completely but tried to improve it slightly. My improvements would have helped in some cases but not with Heartbleed.\nIn the wake of the catastrophe, reyk@ implemented a privilege-separated RSA engine which allowed the RSA private parts (huhu) to remain COMPLETELY isolated from the network facing processes. His solution wasn\u0026rsquo;t a fix to Heartbleed but a proper design which would have made it inoperant.\nOf course, I immediately asked him if he could help integrate in OpenSMTPD which he did\u0026hellip; in a process named \u0026ldquo;klondike\u0026rdquo;\u0026hellip; because we\u0026rsquo;re all \u0026ldquo;pony express\u0026rdquo; and unprofesionnal :-p\nThe OpenSMTPD-extras # For a long time, we shipped countless pieces of code that didn\u0026rsquo;t really belong in our repository.\nNot because they\u0026rsquo;re not related to OpenSMTPD but because they are not really useful in the stock daemon, they came with some dependencies that we didn\u0026rsquo;t want to have or because too few people use them that we want to maintain them as part of the official release.\nSo I lobbied and lobbied and lobbied to convince eric@ and chl@ that we needed an OpenSMTPD-extras repository where we could move table_ldap, table_mysql, table_postgres, table_redis and table_whateverwecomeupwith to avoid keeping too many things in our main repository.\nSome work was done to split the code between these two repositories and make sure that the content of OpenSMTPD-extras would build by itself. This work was mainly done by chl@ and myself, and turned out not to be too painful, it only took us a couple hours to finish the split.\nThe idea behind this repository is that the OpenSMTPD repository is essentially read-only, only developers can commit and the commits must follow strict rules with regard to dependencies and what they bring to the server. We don\u0026rsquo;t want to ship mysql or posgres support with the daemon bringing their dependencies, and that\u0026rsquo;s even more relevant now that they can be standalone backends that can be packaged separately.\nBy providing a separate repository, we can remove these restrictions: the OpenSMTPD repository remains strict, the OpenSMTPD-extras repository is a set of unrequired plugins that can ship with whatever dependencies they need and we won\u0026rsquo;t be strict about that. Of course that doesn\u0026rsquo;t mean we will allow bad code to creep in, but we will be less strict as these will be optional user contributions, kind of what \u0026ldquo;ports\u0026rdquo; are for the BSD.\nThe OpenSMTPD-extras repository is now packaged in OpenBSD ports at least.\n","date":"1 December 2014","permalink":"/posts/2014-12-01/some-opensmtpd-overview-part-2/","section":"Posts","summary":"Why we killed the MFA (filter) process # For as long as I can remember, a process called MFA was created by OpenSMTPD at start time.","title":"Some OpenSMTPD overview, part 2"},{"content":" EHLO world ! # Yesterday I thought I\u0026rsquo;d write a first OpenSMTPD-related post to sum up the changes that have happened since this summer but it turned out to be painful as they amount to quite a lot. Instead, I think a better strategy is to split this into a serie of smaller posts focused on the specific changes ;-)\nSo, in June I have organized a mini hackathon at my brand new place and invited some of the French Connection at home. The goal was to work a bit and find an excuse to watch the French team take over the world cup before an eventual surrender once everyone had left. Since it was an OpenSMTPD hackathon, eric@ and chl@ were present, but we also invited miod@ who had been hosting me several times at his hackathons and jca@ who turned out to live close to my place.\nEric and I planned to finish work on the filter API which would allow plugging third-party filters to the daemon for session-time and data post-processing, Charles had a few things to cleanup in the portable branch, and we all had a few other things we wanted to deal with.\nToday, I will talk about the filter API and pluggable backends.\nThe filter API # Filters have been long awaited in OpenSMTPD, it\u0026rsquo;s been a constant feature-request for at least two years.\nIn fact, filters have existed in OpenSMTPD for a long time as an (unsatisfying) unexposed prototype, but we had strong ideas about where we wanted to go with this subsystem and this was just not it.\nWe decided to start from scratch and only retain one idea from the prototype, the fact that a filter is not a shared object. The idea behind that was that we wanted filters to run outside of the daemon memory space and be able to provide per-filter privilege separation and chrooting. Of course, we didn\u0026rsquo;t want to execute the filter for every session as the overhead would be ridiculous, so a filter would effectively be a daemon communicating with the server through IPC.\nNow, our other goal was to make filters dead simple for both users and developers and having to write a daemon with IPC was kind of raising the bar too high. So the biggest challenge was to overcome this and make sure that neither of them would need to actually see that.\nWe came up with an API which turned the filters into event machines, a filter writer would call a filter_init() function that would take care of all the daemon/imsg part, then call a set of functions to register callbacks on specific events and finally call a filter_loop() function to make the filter start processing requests. These ideas were already part of the first prototype, but had to be revamped to allow for privilege separation and chrooting to be doable.\nAt the same time, eric@ completely rethought filter chaining to allow filters to pass data one to another, it was a much more complex task than it looked as the tricky parts are not getting it to work, but getting it to fail gracefully if a filter suddenly craps itself.\nBy the end of the hackathon, we had the subsystem mostly done and functional to a point where I could write both a python bridge filter and a php bridge filter so filters could be written in these languages, and write a few dummy filters to alter content, turn it into l33tsp34k, reject mails containing viagra and what not.\nShortly after, we received our first filter contribution from a community member who wrote a DKIM signer using our API, another feature long awaited :-)\nOur next major release should come with this API enabled, the only thing blocking us from doing so at the moment is the lack of feedback and confidence that it is rock solid. People have been requesting this API for ages yet it is very very hard to actually get people to use and comment on it.\nPluggable backends # As I already mentionned in previous posts, we had done a lot of work on abstracting the queue API to make it \u0026ldquo;pluggable\u0026rdquo; in the sense that one can write a queue in a standalone file, expose it through a structure and have a pointer point to that structure to start using it instead of our stock queue. This has been done a long time ago to make it easier for us to experiment, revert and experiment again with confidence we didn\u0026rsquo;t break something else in the way.\nFrom OpenSMTPD\u0026rsquo;s perspective, a queue is basically an iterable key-value store and all it has to know is that messages and envelopes have identifiers that it can pass to the queue operations which will know what to do with them. It no longer has any knowledge of the internals and you can basically move a queue to a database, a cloud or your own filesystem-based storage and the server will not notice a difference. As a matter of fact, the first prototype we wrote to prove this worked was to store a queue on the Google cloud, and let either one of eric\u0026rsquo;s or my laptop schedule the message without having it locally.\nAt some point, I convinced eric@ that we should have a similar interface for the scheduler if only to make sure that the daemon did not make any wrong assumption on scheduling. I came up with a prototype while at an OpenBSD hackathon in Budapest two years ago and once it started working, eric@ took over, fixed some API layer violations and improved it further. All over last year, we had to do many changes to the scheduler while running OpenSMTPD in a high-volume environment and this has proved to be a very useful change.\nSo\u0026hellip; if we have to play games with pointers and rebuild, that\u0026rsquo;s not really \u0026ldquo;pluggable\u0026rdquo; is it ?\nWhen I first considered splitting the subsystems into their own isolated API, what I had in mind was not just to make it easier to test things. I wanted to be able, just like filters, to provide an API that would allow plugging custom queues and schedulers without having to modify something in the server. People have tons of different use-cases and we want to keep the OpenSMTPD code base as simple, clean and dependency-free as possible. So by providing the proper API we can ensure that people can deal with their use-cases without having us to cooperate, they\u0026rsquo;re free to use whatever dependency and we will not have to even care.\nDuring our little hackathon, I asked eric@ what was needed to avoid the pointer juggling and just make it possible to set it from our configuration file. He worked on it for a few hours and we ended up with a version that would allow me to build a queue as an external executable that could be plugged and communicate with the daemon like filters. Since I was in a pythonic mood, I wrote a queue-python which allowed me to write a queue backend in python and used it to write a prototype queue_mysql in python. This was not really intended to be a serious project, just a proof of concept that this kind of things can be achieved, but it only took a couple hours to complete these and the python implementation of a queue is so small that it paves the road to interesting experiments :-)\nEric also made the scheduler pluggable, I didn\u0026rsquo;t write one as I got context-switched to something else, but you get the idea of what can be achieved now.\nWhat\u0026rsquo;s coming in next post ? # I\u0026rsquo;ll be discussing the new OpenSMTPD-extras repository which holds all pluggable contents, as well as the smtp/mta processes merge, the mfa process removal and reyk@\u0026rsquo;s RSA privsep engine.\nEric will be talking about asr, the asynchronous resolver, and how we pushed him into making it a standalone library.\nI\u0026rsquo;ll convince Charles to talk a bit about portable :-)\n","date":"26 November 2014","permalink":"/posts/2014-11-26/some-opensmtpd-overview-part-1/","section":"Posts","summary":"EHLO world ! # Yesterday I thought I\u0026rsquo;d write a first OpenSMTPD-related post to sum up the changes that have happened since this summer but it turned out to be painful as they amount to quite a lot.","title":"Some OpenSMTPD overview, part 1"},{"content":"Good news everyone !\nWith last post dating from almost a year, we can all agree that I\u0026rsquo;ve outpassed my slacking skills by a great margin and I probably deserve an award of some kind ;-)\nAnyways, there\u0026rsquo;s been a few complaints during these 12 months because this blog was the main source of information for the OpenSMTPD project and several posts which used to describe features and configuration samples had suddenly disappeared. I\u0026rsquo;m sorry about that, I guess an explanation is in order.\nI didn\u0026rsquo;t close it out of rage, nor trip on a cable, nor lose content in a database crash or whatever. I made a bad tech decision at a very bad time.\nWhat happened is that this blog relies on another project of mine for storage, and that project needed some love, so I had taken the decision to simply put the blog in pause for the week-end until I could finish what I wanted to finish on that other project.\nWhile I had taken it down for a few days, life bit me. Family and health issues I was dealing with at that time had worsened, then while coping with that an opportunity to leave Paris and move to the city of Nantes had popped-up overflowing me with much higher priority things to deal with like preparing the moving out for one. As you can imagine, the blog has plummeted at the booooooottom of a very high list of things I had to take care of.\nNow, I\u0026rsquo;ve moved in to Nantes for a few months already, things have eventually settled down for the better or the worse, and I finally managed to find some time to work back on fixing my other project to bring this blog back up. I\u0026rsquo;m not done, but I don\u0026rsquo;t intend to take it down again so I hacked up a shit-tons of indirections here and there to let me improve the storage without taking the entire universe down with me.\nI\u0026rsquo;ll resume posting about OpenSMTPD and technical topics starting next week, I still need to finish a few unrelated things first.\nNote that this is not an official OpenSMTPD website, just the personal blog of one of the developers so unlike before I will be posting more and more stuff that are unrelated and implement categories so that you don\u0026rsquo;t have to go through my personnal / political / music / martial arts / pro-Palestinian ramblings ;-)\nBe back in a few days !\n","date":"24 November 2014","permalink":"/posts/2014-11-24/what-the-fsck-did-just-happen/","section":"Posts","summary":"Good news everyone !\nWith last post dating from almost a year, we can all agree that I\u0026rsquo;ve outpassed my slacking skills by a great margin and I probably deserve an award of some kind ;-)","title":"what the fsck did just happen"},{"content":"EHLO readers,\nThis blog post is the first since a few months, I\u0026rsquo;ve been busy and struggling with some personal health and familial issues. I won\u0026rsquo;t share them here as its not really something anyone can help with, so\u0026hellip; let\u0026rsquo;s focus on OpenSMTPD !\nWhat happened since last post\nWhen I wrote the last blog post, we had just released 5.3.2 which was a minor release that fixed a few non-critical bugs that were reported to us since the first major release a few months earlier.\nA while later, we released another minor release, 5.3.3, that also fixed minor bugs and brought some new non-invasive features to deal with common use-cases reported by our increasing user base.\nOpenSMTPD 5.3.3 was very stable, it\u0026rsquo;s been running on busy servers at work and we did not experience any bug with it while accepting and routing millions of daily messages with remote hosts on several machines.\nIt was a nice release for what it\u0026rsquo;s worth :-)\nWhat now ?\nWell, we didn\u0026rsquo;t stop hacking on OpenSMTPD and since 5.3.3 we have gone through lots of simplifications and adding new features. There are actually so many changes that a blog post can\u0026rsquo;t possibly go through all of it but I\u0026rsquo;ll discuss some of the most important and visible ones.\nWe have released new major version 5.4.1 a few days ago, and the features that are described below are all part of it. It is a very good release IMO and you should definitely take time to switch your 5.3.x setups to this new one.\nsmtpd.conf improvements\nFirst of all, despite being already very simple our configuration file has been simplified further and new features were introduced.\nFor example, certificate configuration had always been a bit tricky because we inherited an old behaviour were we would infer file names from a certificate name. This imposed certificates to be in a specific location /etc/mail/certs and to follow a naming conventions so that a certificate foobar would expect /etc/mail/certs/foobar.key and /etc/mail/certs/foobar.crt to be found.\nI was heavily bothered with this because I store my certificates and keys in /etc/ssl and /etc/ssl/private respectively and didn\u0026rsquo;t like having the /etc/mail/certs path imposed because that\u0026rsquo;s the way we had it with Sendmail. For years I used symbolic links but that was a hack around fixing this broken behavior.\nAlso, some people were confused because not only they could provide the path to their certificates but the listen directive expected parameters to be provided in a specific order which meant that:\nlisten on all tls hostname example.org certificate example would fail while:\nlisten on all tls certificate example hostname example.org would work the way they expected. We therefore went into a grammar cleanup to ensure that parameters were swappable where it made sense and to come up with a much simpler way to deal with certificates. The pki directive was introduced which was no longer tied to a listen statement but declared a database of certificates and keys associated to a hostname:\npki example.org certificate \u0026ldquo;/etc/ssl/example.org.crt\u0026rdquo; pki example.org key \u0026ldquo;/etc/ssl/private/example.org.key\u0026rdquo; With this in place, it became possible to simplify the configuration by referencing the hostname in places where it made sense:\nlisten on all tls pki example.org accept for any relay pki example.org and with parameters swappable, it was now also possible to use either one of these:\nlisten on all tls pki example.org listen on all pki example.org tls with the same expected result.\nWhile there, we decided to introduce a set of new features to our grammar so that use-cases that previously required several rules could be simplified. For example, we introduced the possibility to use negations:\naccept from ! local [\u0026hellip;] reject for ! domain We also added the possibily to filter senders and recipients at the ruleset level:\naccept from any sender \u0026ldquo;@poolp.org\u0026rdquo; [\u0026hellip;] accept from any sender \u0026ldquo; gilles@poolp.org\u0026rdquo; [\u0026hellip;] accept for domain poolp.org recipient \u0026ldquo;@poolp.org\u0026rdquo; [\u0026hellip;] accept for domain poolp.org recipient \u0026ldquo; gilles@poolp.org\u0026rdquo; [\u0026hellip;] accept from any sender [\u0026hellip;] And pushed it further by accepting wildcards in domain parts:\naccept for domain poolp.org recipient \u0026ldquo;gilles@*.poolp.org\u0026rdquo; [\u0026hellip;] Eric thought it would be a good idea to be able to accept mail for domains but require that the mail be forwarded elsewhere. We therefore introduced the forward-only keyword which allows you to:\naccept for domain poolp.org foward-only Accepting mails for local recipients ONLY if they are redirected to an external address either through the use of a ~/.forward file or by an alias.\nWith these the configuration file became quite more powerful while keeping the intended simplicity. But \u0026hellip; we did more ;-)\nUntil now, a listener had to declare it\u0026rsquo;s hostname:\nlisten on all hostname myhostname But this meant that if you had a single interface accepting mails for various domains, you had to list them all by their IP addresses:\nlisten on 192.168.1.1 hostname mx1.poolp.org listen on 192.168.1.2 hostname mx1.pool.ps Eric introduced the hostnames table to perform address to hostname lookups, allowing for a single listen line to use a hostnames table:\nlisten on all hostnames and reduce considerably the size of a multi-domain setup.\nMore features were added later but they are not part of the latest major release so I will discuss them in a post next week ;-)\nTLS improvements\nThe new pki stuff in smtpd.conf allowed us to simplify a bit of code internally and it allowed us to implement new features. Some where not mature enough to make it to the major release (I\u0026rsquo;ll talk about them next week), however there\u0026rsquo;s been a couple improvements that did make it to the release.\nWe have introduced support for TLS Perfect Forward Secrecy. We already had support for ephemeral key exchange for years, but after a suggestion in the OpenBSD hackers list to add EC support to another project, we came up with a diff to bring support for ECDHE in OpenSMTPD. It was such a simple change that it was committed the same day and we saw the result almost immediately when reading headers from mails we received.\nWe then brough back a feature I had implemented a long time ago but disabled for some reason I can\u0026rsquo;t recall. It required a bit of work but when declaring a PKI, a custom CA certificate may be provided allowing a listener to use a private CA certificate to authenticate users from an organization\u0026hellip; as simple as:\npki mx1.poolp.org ca \u0026ldquo;/etc/ssl/myca.pem\u0026rdquo; For a long time, certificates were verified but would simply alter the headers, but with this feature came the need to be able to prevent sessions from establishing if verification failed. We introduced the \u0026ldquo;verify\u0026rdquo; keyword:\na user MUST provide a certificate that can be verified with our system bundle if no \u0026ldquo;ca\u0026rdquo; is part of our pki entry, or with our \u0026ldquo;ca\u0026rdquo; if we declared one. listen on all tls-require verify pki mx1.poolp.org We pushed the feature a bit further, by allowing a relay rule to only work if it can establish a TLS session \u0026hellip; and eventually verify the remote certificate:\nONLY relay through TLS accept [\u0026hellip;] relay tls\nONLY relay through TLS \u0026hellip; if remote certificate is valid ! accept [\u0026hellip;] relay tls verify\nONLY relay through our route if remote certificate is valid ! accept [\u0026hellip;] relay via tls://mysmarthost verify To increase the security of the entire SMTP network, when relaying OpenSMTPD will always look for a pki entry matching its hostname and present a client certificate. This works transparently and takes into account hostnames table, source address and other tweaks.\nmta improvements\nOur MTA now always attempts TLS before falling back to plaintext. As stated above, it can also require remote hosts to present valid certificates and it will always attempt to present one itself.\nWe implemented support for AUTH LOGIN which was missing and made a lot of improvements to the logic used to optimize transfers. The mechanisms range from detecting hosts which we can\u0026rsquo;t establish a valid session with, to keeping a connection alive a few seconds to avoid a round-trip if an envelope gets scheduled for that same host just a few seconds later.\nTo summarize, we tried to make our MTA smarter so it takes less resources locally while not beeing seen as too agressive from the other side. There are many tricky details about this, if there\u0026rsquo;s demand we\u0026rsquo;ll post a more in-depth view of the internals ;-)\nsmtp improvements\nWe added some features that were requested by users, such as the possibily to create a listener that only listens to inet4 or inet6 rather than both.\nWe made it possible for listeners to require valid certificates, but also to select the banner name from a table dynamically based on the local address the client has connected to.\nA common feature was to allow the Received line to hide the From part which is not really a requirement for us to display and which allows hiding details from internal network. This is done by simply indicating on the listener that it should mask source:\nlisten on all mask-source Finally, we did a lot of internal rework since we are currently working on the filter API and while it is disabled, we wanted the code path to enter the filter loop to detect that there was no regressions when no filters are plugged.\nqueue improvements\nAn envelope cache was introduced to improve disk-IO pattern.\nOpenSMTPD is very strict about commiting changes to disk and keeping a coherent disk-based queue that it can start from if we were to crash or suffer a power down at the \u0026ldquo;wrong\u0026rdquo; moment.\nThe entire queue API has been designed around that constraint and this security comes with a price with regard to performances. There\u0026rsquo;s not so much we can do about disk-writes, however a common pattern is for OpenSMTPD to transmit an envelope to queue which writes it to disk and notifies the scheduler \u0026hellip; which requests the queue to load that envelope so that it can be delivered.\nBy introducing an envelope cache, the envelope is written to disk but retained in queue memory so that when the scheduler requests it back we avoid a disk read which we know will happen almost right away.\nThe queue cache will be improved with time but this first version already had a positive impact.\nWe also made many improvements that I won\u0026rsquo;t describe here as they are very technical and require a knowledge of the interactions between queue, mta, mda and scheduler. If people are interested, I\u0026rsquo;ll go into details in a more specific post.\nscheduler improvement\nLike for the queue, we made TONS of improvements and I won\u0026rsquo;t go into too many details either. But two of them are interesting enough that they need to be mentionned \u0026hellip; also they kept Eric and I working hours on investigating and fixing so just for that reason they deserve a mention.\nFirst improvement was the breaking of envelope batches into smaller batches. For optimization reasons, the scheduler tries to pack sets of envelopes to send to the mta/mda however when there are many many envelopes going to the same destination, the batches may grow very large.\nDue to our design, an event handler should never take a long time to operate otherwise it blocks the process from handling other events. In a situation, we had about 10k bounces from a major ISP and this caused the scheduler to send to the queue these 10k bounces in a row. The scheduler became hogged, it would not schedule anything until done with that task, and the queue was hogged as it would be processing this huge amount of envelopes without the scheduler letting it handle any other event. We broke the batch into small pieces allowing other events to interlace and this caused us to have much more efficient scheduling.\nThe other problem was an algorithmic issue\u0026hellip; We mainly work with trees but the scheduler had one sorted-list which it used for a single purpose. While this was no problem when working with a small set of envelopes (small being below 100K), as we grew trafic and were facing larger sets of envelopes, our O(n) inserts became so consuming that it caused the scheduler to eat all CPU for a few seconds \u0026hellip; every few minutes. In our very busy environement, this led to late that cumulated and cumulated until it was no longer possible to deliver in time. We ended up figuring the issue and replaced with a nice tree which allowed us to deal with a much larger volume and a very relaxed smtpd. OpenSMTPD has no problem working with queues that are several hundreds of thousands messages larges, but in theory we could go much higher we just don\u0026rsquo;t deal with enough volume to prove it ;-)\nsmtpcl improvements\nThe smtpctl utility which is the admin interface to the daemon has been rewritten to be much simpler and many new features were added.\nIt can now be used to show routing information, state of remote hosts, it can pause and resume individual envelopes and messages, and it transparently supports showing envelopes and messages even if the queue is encrypted and/or compressed.\nWe\u0026rsquo;re planning on many new improvements to it but it\u0026rsquo;s quite sexy as is ;-)\ndocumentation improvements\nWe introduced a table(5) man page to describe the format of tables, something that has bothered many users and someone contributed a sendmail(8) page to document our sendmail-like interface.\ntables improvements\nWe introduced table_passwd to provide user info and credentials in a passwd like table that is compatible with software such as Dovecot. Users can now store their recipients in a shared file and avoid duplicating information.\nWe made sure table_sqlite worked for all kinds of lookups, it is used in one of my side-project where an OpenSMTPD is used for a completely virtual setup where there\u0026rsquo;s not a single real user.\ntable_mysql has been improved and is pretty much production-ready, we use it in production ourselves and do not face issues with it.\nportable\nI reworked the entire autotools layout to reduce the delta between master and portable branches. Every component is being dealt as a module and while I\u0026rsquo;m not done, this is already much more clean and will allow us to reduce the inheritance of unrequired dependencies.\nI\u0026rsquo;ll keep working on it until it reaches perfection :-)\nA few words on performances and volumes\nI work at a company that handles a lot of mails, and by a lot I\u0026rsquo;m talking in matters of several millions per day both incoming and outgoing.\nWe initially relied on other opensource software deployed on many servers and as timed passed, progressively switched many servers to OpenSMTPD 5.3.3.\nAfter the many improvements done to the queue and scheduler, the number of servers could shrink and it is now technically possible to handle the same amount of mails (we\u0026rsquo;re still talking several millions a day) on a single instance of OpenSMTPD 5.4.1 that\u0026rsquo;s nowhere near busy.\nI know for a fact other companies have started using OpenSMTPD with high volumes too, so people who wonder if we will be efficient enough for their uses should simply ask themselves: do I need to send and receive several millions of mails each day. If that\u0026rsquo;s not the case, then you can\u0026rsquo;t possibly go wrong with us\u0026hellip; and if that\u0026rsquo;s the case, well, we do that and much more efficiently than with the older software, so you probably can rely on us.\nI will ask my manager if I can put out some graphs and numbers of real-life trafic as I don\u0026rsquo;t believe in oriented benchmarks ;-)\nThat\u0026rsquo;s all folks !\nThat\u0026rsquo;s all for this first post since a long time, I will make sure to post every weeks about updates since we\u0026rsquo;re very active on development.\nOh and the RSS feed will be back shortly, no need to tell me ;-)\n","date":"14 December 2013","permalink":"/posts/2013-12-14/news-from-the-front/","section":"Posts","summary":"EHLO readers,\nThis blog post is the first since a few months, I\u0026rsquo;ve been busy and struggling with some personal health and familial issues. I won\u0026rsquo;t share them here as its not really something anyone can help with, so\u0026hellip; let\u0026rsquo;s focus on OpenSMTPD !","title":"News-from-the-front"},{"content":"Hey,\nNo OpenSMTPD news since almost a month\u0026hellip; time to break the trend !\nAs far as I\u0026rsquo;m concerned, I\u0026rsquo;ve been busy with other work but still a git log shows that between Eric and I we have accumulated quite a few things during this month. Below is a short non-exhaustive summary of changes and news.\nAll of it is already pushed to our Github mirror and part of the snapshots that were published yesterday. I\u0026rsquo;d like to stress out that if you can run bleeding edge stuff, you REALLY want to test the latest snapshot as it has some very invasive improvements which we want to make sure are \u0026ldquo;bug-free\u0026rdquo; before the next big release in a few months.\nWe have also fixed some minor bugs that were reported to us since 5.3.1 and discovered/smashed some other bugs on our own. The bugfixes are part of the snapshots and were backported to the 5.3.2 stable release that should take place next week.\nMTA improvements\nOpenSMTPD has various \u0026ldquo;logical\u0026rdquo; optimizations to try and optimize its transfers. It will create several, but not too many, connections to a destination; it will detect that two domains share a same set of MX; it will spread load across MX of same priority; it will group messages on connections and group envelopes for messages; proceed to quadratic increase of delay upon temporary failures, and much more \u0026hellip;\nThis all performs pretty well with regular use-cases but we have discovered a use-case that was sub-optimal: dealing with a large list of single recipient messages going to the same destination. For example, the misc@opensmtpd.org list uses DKIM-proxy and signs all outgoing mail for each single recipient. Then, our code will optimize that out by grouping these recipients so that their mail is delivered through the same connection. So far, so good.\nProblem occurs if the other side uses grey-listing and rejects the recipients with 451 instead of dropping the connection with a 421 at first recipient: we keep reusing the connection to submit envelopes which we know are likely to fail. With a small list like ours, that\u0026rsquo;s not a problem as we will fail a few dozens envelopes and the quadratic delay will cause things to sort themselves after a few bounces but \u0026hellip;\n\u0026hellip; now imagine your list is that of a company with hundreds of thousands of recipients and the same happens. By the time you\u0026rsquo;re done submitting your list of recipients, it\u0026rsquo;s already time to try out the first of the list. The delay between your last recipient and the retry of first one causes you to fail to pass the grey-listing again, and you enter another loop of failures\u0026hellip; It eventually sorts itself out when the quadratic delay is so large that the delay between your last submitted recipient and the next retry of the first recipient respects the grey-listing imposed delay.\nMost people will never hit this case, but those who hit it will see their mails delivered after much more time than they really should (ie: your customers could receive their mail 12 hours later when they could be delivered after 4 hours.\nEric and I came up with two distinct mechanisms to improve the situation:\nFirst of all, instead of starting several concurrent connections, OpenSMTPD will first try to establish one and ensure that it is working before creating new ones progressively. If an issue prevents connections from being established and sessions from being started, the MTA layer will penalize the route and prevent it from being used for a small period. If new envelopes arrive for that destination, they are delayed until the route is considered usable again.\nThen, another mechanism kicks in. If too many recipients are rejected in a row, instead of trying the entire list OpenSMTPD will consider that they are all temporarily failed and mark the route temporarily unusable. This will cause the deliveries to be reattempted later, just like when a grey-listing mechanism is in place, without hammering the host with tons of recipients that are most likely going to fail. The ones that were rejected are penalized and their delay is slightly increased compared to those that OpenSMTPD failed. With this, in most situations people will not hit the case; in case we really face the problem, we will avoid hammering the server that kindly asked us to try again later several times; and if this was just a coincidence, legitimate envelopes are only deferred for 400 seconds.\nFinally, Eric spotted a bug that could cause a wrong error to be detected and imposing a long delay on some deliveries to hosts that advertise both IPv4 and IPv6 MX records.\nsmtpctl show routes\nWith all this routing logic added, we needed a way to be able to better understand what is happening at the MTA level.\nLog files have been extended and they provide a precise history, however it\u0026rsquo;s always nice to have a tool that provides real-time display of the state of something you\u0026rsquo;re investigating. So Eric added a \u0026ldquo;smtpctl show routes\u0026rdquo; subcommand that display the routing informations that are currently in used by MTA. It displays active routes, routes that have been penalized and when their penalty is going to expire and the route be usable again. This is a shortened output of the command right when I sent the \u0026ldquo;new snapshot published\u0026rdquo; mail to our mailing list, it displays the route in use, the state of the route, the number of connections currently active to that route, the penalty level and the timeout before a route is usable again after we detected a failure:\nsmtpctl show routes 88.190.237.114 \u0026lt;-\u0026gt; 144.76.32.53 (arati.lostca.se) \u0026mdash;- nconn=1 penalty=0 timeout=- 88.190.237.114 \u0026lt;-\u0026gt; 213.41.253.7 (gw.zefyris.com) \u0026mdash;- nconn=1 penalty=0 timeout=- 88.190.237.114 \u0026lt;-\u0026gt; 213.165.67.115 (mx01.gmx.net) \u0026ndash;Q- nconn=0 penalty=0 timeout=2s 88.190.237.114 \u0026lt;-\u0026gt; 213.165.67.97 (mx01.gmx.net) \u0026ndash;Q- nconn=0 penalty=0 timeout=1s 88.190.237.114 \u0026lt;-\u0026gt; 213.165.67.99 (mx00.gmx.net) \u0026ndash;Q- nconn=0 penalty=0 timeout=2s [\u0026hellip;] about a hundred entries [\u0026hellip;] 88.190.237.114 \u0026lt;-\u0026gt; 72.249.41.52 (lavabit.com) \u0026mdash;- nconn=1 penalty=0 timeout=- 88.190.237.114 \u0026lt;-\u0026gt; 87.98.163.156 (ns1.huynguyen.org) N\u0026mdash; nconn=1 penalty=0 timeout=- 88.190.237.114 \u0026lt;-\u0026gt; 95.130.12.24 (host1.trois-doubles.net) \u0026ndash;Q- nconn=0 penalty=0 timeout=1s\nTruly perfect for troubleshooting remote delivery issues !\nSSL issues\nEric ran into an issue where the daemon would crash when accepting a SSL connection from a client that didn\u0026rsquo;t do what we expected.\nI tried to reproduce but for some reason, it would not happen on OpenBSD, only on Linux. I spent a while troubleshooting and eventually found a subtle issue with our handling of I/O events.\nDuring a non-SSL session, OpenSMTPD enables and disables events as the session progresses since a SMTP session is a succession of command/responses where it is not expected to read a command when it has to send a response, and not expected to send a response when it is waiting for a client command. This works fine.\nDuring a SSL session, however, renegociation can take place and expect the server to write while the server is in read mode, and the other way around. To cope with this, the code wrongfully enabled both read/write events, causing the daemon to eventually end up in a weird state leading to either a hang until the client disconnects or a fatal() being hit.\nIn addition, we never set the client socket non-blocking, even though the logic around expected it, because it was assumed that we didn\u0026rsquo;t really need to do so due to the event-based approach. However, I found ways to cause SSL_accept() to be triggered BUT block with a misbehaving client that didn\u0026rsquo;t complete the SSL \u0026ldquo;handshake\u0026rdquo;. Other related issues could cause a SSL client to \u0026ldquo;block\u0026rdquo; other concurrent sessions until it timedout.\nAll these were fixed by making the client socket non-blocking and by being more strict with our event handling, none of the bugs could be reproduce on Linux after the fixes while no regressions were observed on OpenBSD.\nPortable improvements\nAn OpenBSD hacker complained that on Linux authentication didn\u0026rsquo;t work.\nIn fact, it would only work when built with PAM (which most Linux users have been doing so far) as the default authentication method in -portable is getpwnam(3) which doesn\u0026rsquo;t return the passwd field on Linux even when called as root. He suggested we implement a getspnam(3) method and provided the code which we merged.\nI spent a while tracking a crash on FreeBSD-STABLE that didn\u0026rsquo;t trigger on FreeBSD-RELEASE and which was caused by their import of NetBSD\u0026rsquo;s strnvis(3) which has \u0026hellip; swapped parameters compared to OpenBSD\u0026rsquo;s strnvis(3). This caused an invalid read that lead to a segv. We marked strnvis(3) as broken on NetBSD and FreeBSD to ensure that -portable builds with the compat glue until we deal with it differently.\nCharles has also worked on some fixes for OSX which we broke with all our new code :-)\nsmtpd -dvn improvements\nThe -d option was supposed to start it in foreground, while the -v option was supposed to start it verbose. In practice, this was not the case as -d started it in foreground AND verbose, while -v was essentially no-op. Oh and due to the log.[ch] code we inherited, it was not possible to start in foreground without verbose, and it was not possible for us to activate traces without being verbose too.\nSome people have been willing to start it in foreground without the verbose mode because they want to run it with a process manager and let it handle god knows what. Since it\u0026rsquo;s a common request and our behaviour was not consistent with the documentation, I decided to rework that and make it behave like expected.\nWhile at it, I fixed the requirement for verbose mode to enable traces so that now it is possible to:\nsmtpd -d -\u0026gt; start smtpd in foreground but not verbose smtpd -v -\u0026gt; start smtpd in background but verbose smtpd -dv -\u0026gt; foreground and verbose\nand no matter if -d and/or -v is specified, \u0026ldquo;smtpctl trace \u0026quot; will activate traces at runtime which will either appear at foreground or be logged to syslog as mail.debug.\nI also received two complaints the same day that \u0026lsquo;smtpd -n\u0026rsquo; checked the configuration file for validity but didn\u0026rsquo;t detect SSL certs errors as they are loaded after the configuration file parsing (on purpose). I slightly reworked the code so that we still load them after the configuration is parsed but to detect errors in the \u0026lsquo;smtpd -n\u0026rsquo; case.\nAssorted bugfixes and improvements\nTable lookups where not always folding key to lowercase which could lead to failed lookups for existing entries.\nThe \u0026ldquo;as\u0026rdquo; feature allows a rule to override the sender at SMTP-level (not in the message itself) and a missing check would lead it to override the sender when generating a bounce. Since the override can be partial (ie: @domain, or user without domain) and the sender for a bounce is an empty address, this could lead to invalid addresses (ie: MAIL FROM: , MAIL FROM: \u0026lt;@poolp.org\u0026gt;) causing a reject of the bounce by remote host.\nAlso, on the bounce side, the delivery_mbox backend didn\u0026rsquo;t assume that a sender could be empty in the bounce case and it would call the mailer.local third-party utility with an empty sender. That utility would write to the mbox with the usual delimiter \u0026ldquo;From\n\u0026quot; but with an empty address causing the following delimiter to be generated \u0026ldquo;From \u0026quot; (two spaces before date). Some MUA that do not check for \u0026ldquo;From \u0026quot; but for \u0026ldquo;From \u0026quot; as a delimiter would fail to detect the bounces in the mbox. As mentionned before, the table API and queue API now allow people to write standalone backends that are started by OpenSMTPD. They can have their own dependencies and be maintained out of the official tree, just as any other tool. To help with this, various APIs have been improved and changed slightly to ease this mechanism.\nA user has improved local LMTP delivery by adding support for unix sockets so that OpenSMTPD can talk LMTP to a unix socket setup by another daemon (Dovecot seems to be the trend ;-)\nAnother user reported a hang after a while and Eric traced it back to a missing check that could lead to an EOF not being detected. The mda/mta sessions would be in a broken state for a session and fail to deliver the message, causing it to stay in queue indefinitely. It was rare enough that we never hit it as it was \u0026hellip; specific to the size of a message \u0026hellip;\nA third user reported a \u0026ldquo;reject\u0026rdquo; rule not working as expected and we spotted a typo in the config file parser causing \u0026ldquo;reject\u0026rdquo; rules to fail when used with the \u0026ldquo;sender\u0026rdquo; parameter.\nA poolp user mentionned the \u0026lsquo;+\u0026rsquo; delimiter not working for one of his domains while it worked with his @poolp.org mail account. I realized that the \u0026lsquo;+\u0026rsquo; delimiter handling was not implemented for virtual domains, only for primary domains\u0026hellip; so I added support for it.\nFinally, an OpenBSD hacker reported that he had issues with his ~/.forward file causing recipient to be rejected when the file is empty. This was indeed the case and anoter user provided a diff to fix it.\nThat\u0026rsquo;s all folks !\nFor now ;-)\n","date":"14 December 2013","permalink":"/posts/2013-12-14/opensmtpd-improvements-summary/","section":"Posts","summary":"Hey,\nNo OpenSMTPD news since almost a month\u0026hellip; time to break the trend !\nAs far as I\u0026rsquo;m concerned, I\u0026rsquo;ve been busy with other work but still a git log shows that between Eric and I we have accumulated quite a few things during this month.","title":"OpenSMTPD improvements summary"},{"content":"Yop,\nLes services de poolp.org sont hébergés sur plusieurs serveurs.\nLe serveur principal à été basculé d\u0026rsquo;un petit hébergeur US \u0026ldquo;OpenBSD-friendly\u0026rdquo; vers Online.net depuis 3 ans maintenant. J\u0026rsquo;ai rusé à l\u0026rsquo;époque pour installer OpenBSD sur la Dedibox \u0026ldquo;DC\u0026rdquo; en utilisant une vieille version de Yaifo et en faisant des upgrades successives du système d\u0026rsquo;exploitation. Mis à part 3 petits downtimes dont 1 de ma responsabilité, je n\u0026rsquo;ai jamais eu à me plaindre de leurs services et de leur matériel.\nLe serveur secondaire, qui ne fournit que de l\u0026rsquo;espace disque pour les backups et un service de DNS et de SMTP secondaire en cas de panne du serveur primaire, est actuellement un serveur Kimsufi de chez OVH sur lequel j\u0026rsquo;ai installé OpenBSD via le vKVM. Le serveur est sous-utilisé, recevant un trafic anecdotique en dehors des backups journaliers en provenance du serveur primaire, mais je suis globalement content du service qui n\u0026rsquo;a pas subi de downtime depuis plus d\u0026rsquo;un an en ce qui me concerne.\nDes fois, pour les besoins d\u0026rsquo;un projet, je prends un serveur supplémentaire pour une durée indéterminée et je le laisse expirer à la fin du projet, ou bien je bascule le serveur secondaire dessus si à prix équivalent le matériel est plus confortable.\nCe week-end, je me suis rendu compte de la disponibilité des nouveaux serveurs Dedibox Classic+ Gen2 qui avaient l\u0026rsquo;air pas mal sexy. J\u0026rsquo;allais justement renouveller un serveur un peu moins confortable au même tarif, j\u0026rsquo;ai donc décidé de prendre un nouveau serveur chez Online et de laisser expirer celui que j\u0026rsquo;allais renouveller chez OVH.\nLe Classic+ Gen2 avait l\u0026rsquo;air assez sympa et la dispo d\u0026rsquo;un KVM/IP laissait présager qu\u0026rsquo;installer OpenBSD serait une partie de plaisir.\nCa, c\u0026rsquo;était avant d\u0026rsquo;essayer pour de vrai.\nLet\u0026rsquo;s go !\nMon serveur est commandé et je recoit peu après un mail m\u0026rsquo;indiquant sa disponibilité. Ni une, ni deux, je me loggue et ne vois pas de menu pour avoir accès au KVM, juste un bouton \u0026ldquo;Install\u0026rdquo;.\nComme il me faudra de toutes facons connaitre un peu le materiel avant d\u0026rsquo;installer OpenBSD par KVM, je prends le premier OS de la liste et lance une install pour pouvoir recuperer quelques infos du dmesg.\nL\u0026rsquo;install est rapide, il faut attendre un délai d\u0026rsquo;une heure avant de réussir à se connecter à la machine par SSH. Ca laisse le temps de chercher comment on accède au KVM depuis la console Online.\niDRAC\nL\u0026rsquo;Install de Debian finie, des menus supplémentaires apparaissent dans la console Online, dont le menu iDrac qui permets de lancer une console virtuelle.\nPas de bol, il faut un OS \u0026ldquo;java\u0026rdquo;-compliant, je viens de vendre mon laptop et je n\u0026rsquo;ai rien qui fasse l\u0026rsquo;affaire. Après moultes galères, je mets la main sur un Windows et je lance la console virtuelle, choisis l\u0026rsquo;ISO d\u0026rsquo;OpenBSD et tente un boot.\nPendant le boot, perte de connexion, impossible de relancer la console virtuelle, impossible de rebooter le serveur depuis la console Online ou bien simplement de faire un ssh sur la machine (normal, elle est en plein boot de l\u0026rsquo;installeur OpenBSD\u0026hellip;).\nSur le canal IRC d\u0026rsquo;#online, on m\u0026rsquo;indique que l\u0026rsquo;iDRAC est dans le caniveau, une intervention est planifiee.\nJe retente de differentes facons, mais le crash de l\u0026rsquo;iDRAC est systematique et me fait perdre la main completement sur le serveur sans aucune action possible à part quémander un redémarrage sur IRC et/ou ouvrir un ticket pour planifier une intervention.\nA la cinquième tentative, mes reves d\u0026rsquo;install par KVM/IP s\u0026rsquo;envolent, on me suggère de faire tourner une VM (ESXi est proposé comme OS), mais je suis pas trop fan donc\u0026hellip;\nQEMU à la rescousse\nJe cherche du coté de Yaifo vu qu\u0026rsquo;il m\u0026rsquo;a sauvé la vie la première fois, mais il n\u0026rsquo;est plus maintenu depuis un bail. J\u0026rsquo;envisage de le mettre à jour, mais pas trop envie de rester sur le problème 40 ans et l\u0026rsquo;idée de me retapper des heures d\u0026rsquo;attentes entre chaque intervention sur mes tentatives de tests échoué m\u0026rsquo;enchante pas des masses.\nSur IRC, Enjolras (#gcu) me fait savoir qu\u0026rsquo;il a installé DragonFlyBSD sur un Kimsufi en se servant d\u0026rsquo;un FreeBSD en mode rescue et d\u0026rsquo;unionfs pour faire l\u0026rsquo;install sur le disque physique depuis un QEMU lancé en curses.\nIdée séduisante, en rescue sur une Dedibox même pas besoin d\u0026rsquo;unionfs\u0026hellip;\nCi-dessous, voici donc la méthode pour installer le plus simplement possible un OpenBSD sur une Dedibox. Elle devrait marcher pour n\u0026rsquo;importe quel OS qui supporte le matériel avec un minimum de changements. Merci Enjolras !\nRécupérer les infos utiles\nIl faut tout d\u0026rsquo;abord booter sur le Linux installé et récupérer une copie du dmesg.\nLa raison est toute simple, QEMU va simuler le matériel, l\u0026rsquo;installation verra un disque et une interface réseau différentes de ceux du serveur physique. Lorsque l\u0026rsquo;installation sera terminée, si on reboot sur le serveur physique sans avoir fait quelques modifications, le système cherchera une interface et un disque inexistants avec le résultat attendu\u0026hellip; un ticket pour une intervention dans quelques heures ;-)\nLes infos qui nous intéressent sont éventuellement le DUID pour identifier le disque dans /etc/fstab et la description des interfaces réseau pour retrouver le nom du driver qui les prendra en charge et ainsi pouvoir créer les fichiers de conf réseau.\nSur le Classic+, j\u0026rsquo;ai décidé de ne pas me servir du DUID vu que c\u0026rsquo;est sympa mais pas obligatoire et pas d\u0026rsquo;un intérêt fou dans mon cas. En revanche, un petit grep eth a mis en évidence deux interfaces réseau \u0026ldquo;Broadcom BCM5716\u0026rdquo; et un petit coup de man bnx à confirmé que le driver bnx(4) les prends en charge sous OpenBSD. QEMU proposera une interface Realtek prise en charge par le driver re(4), donc ca n\u0026rsquo;ira pas.\nLe disque dur est monté sur /dev/sda, avec OpenBSD dans QEMU il sera visible en tant que /dev/wd0c, et avec OpenBSD sur le serveur physique il sera visible en tant que /dev/sd0c.\nDe plus, il va nous falloir les infos pour la configuration réseau:\nadresse IP broadcast netmask gateway On peut les récupérer depuis l\u0026rsquo;interface Online ou depuis ifconfig et route, en ce qui me concerne:\nadresse : 88.191.185.XXX broadcast : 88.191.185.255 netmask : 255.255.255.0 gateway : 88.191.185.1 Il faudra aussi l\u0026rsquo;adresse d\u0026rsquo;un serveur de nom pour éviter de passer 4 plombes sur le reverse lookup à la première connexion ssh, personnellement j\u0026rsquo;ai utilisé le 8.8.8.8 de Google pour mon premier boot puisque j\u0026rsquo;installe un serveur de nom local par la suite.\nLet\u0026rsquo;s Go Pour De Vrai!\nPremière étape: installer QEMU ;-) % sudo apt-get update % sudo apt-get install qemu\nSeconde étape: télécharger l\u0026rsquo;ISO d\u0026rsquo;install % wget ftp://ftp.fr.openbsd.org/pub/OpenBSD/5.3/amd64/install53.iso\nTroisième étape: lancer l\u0026rsquo;install. % sudo qemu-system-x86_64 -curses -boot -d -cdrom install53.iso -hda /dev/sda\nA ce stade, qemu boot sur l\u0026rsquo;installer et on fait face a une install tout a fait normale d\u0026rsquo;OpenBSD. Je le lance en mono-cpu, je prefere une install qui boot sur un kernel monoproc la première fois et faire une install du kernel MP après le premier boot, chacun est libre de faire comme il veut ;-)\nJe ne vais pas documenter l\u0026rsquo;installation, il y a moultes informations à ce sujet, par contre je vais juste faire une petite remarque utile pour vous éviter de perdre du temps inutilement:\nen mode \u0026ldquo;auto\u0026rdquo;, disklabel ne consommera pas tout le disque, il faut faire un découpage custom\nIl vaut mieux s\u0026rsquo;en rendre compte pendant l\u0026rsquo;install que de se galérer à faire les resize après \u0026hellip; croyez moi.\nInstallation finie ?\nAvant de quitter le mode rescue et de booter sur le serveur physique, quelques petites modifications.\nOn configure\u0026hellip;\n\u0026hellip; la gateway:\necho 88.191.185.1 \u0026gt; /etc/mygate\n\u0026hellip; l\u0026rsquo;interface réseau:\necho inet 88.191.185.XXX 255.255.255.0 88.191.185.255 \u0026gt; /etc/hostname.bnx0\n\u0026hellip; le resolver:\ncat \u0026lt; /etc/resolv.conf nameserver 8.8.8.8 search file bind EOF\n\u0026hellip; le fstab (attention à pas se rater):\nsed \u0026rsquo;s/wd0/sd0/g\u0026rsquo; \u0026lt; /etc/fstab \u0026gt; /tmp/fstab \u0026amp;\u0026amp; mv /tmp/fstab /etc/fstab\n\u0026hellip; et HOP on reboot\nreboot\nPremier reboot\u0026hellip;\nLe premier boot est long. Tres long. En fait pas si long que ca, mais l\u0026rsquo;anxiete de devoir refaire un ticket pour une intervention et attendre 4h pour refaire une tentative donne l\u0026rsquo;impression que des heures s\u0026rsquo;ecoulent avant la premiere reponse aux ping ;-)\nUne fois ce stade, on doit pouvoir se SSH sur la machine dans un OpenBSD avec un kernel monoproc. Si c\u0026rsquo;est bon, il ne reste plus qu\u0026rsquo;à installer le kernel MP:\ncp /bsd /bsd.sp ftp ftp://ftp.fr.openbsd.org/pub/OpenBSD/5.3/amd64/bsd.mp cp bsd.mp /bsd sync reboot\nAu reboot on est sur un OpenBSD 5.3 GENERIC#MP \\o/\nOpenBSD 5.3 (GENERIC.MP) #62: Tue Mar 12 18:21:20 MDT 2013 deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8557342720 (8160MB) avail mem = 8307052544 (7922MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe6da0 (57 entries) bios0: vendor Dell Inc. version \u0026ldquo;2.2.3\u0026rdquo; date 10/25/2012 bios0: Dell Inc. PowerEdge R210 II acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP SPMI DMAR ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR HEST ERST BERT EINJ acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) XHC_(S4) PXSX(S4) RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5) PXSX(S4) RP06(S5) PXSX(S4) RP07(S5) PXSX(S4) RP08(S5) PEG0(S5) PEGP(S5) PEG1(S5) PEG2(S5) PEG3(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3093.38 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV, PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL, DTES64,MWAIT,DSCPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1, SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF, PERF,ITSC,FSGSBASE,SMEP,ERMS cpu0: 256KB 64b/line 8-way L2 cache cpu0: apic clock running at 99MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3092.98 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH, DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL, VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT, DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS cpu1: 256KB 64b/line 8-way L2 cache cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3092.98 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH, DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL, VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT, DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS cpu2: 256KB 64b/line 8-way L2 cache cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz, 3092.98 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH, DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL, VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT, DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS cpu3: 256KB 64b/line 8-way L2 cache ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 3 (P0P1) acpiprt2 at acpi0: bus 2 (RP01) acpiprt3 at acpi0: bus -1 (RP02) acpiprt4 at acpi0: bus -1 (RP03) acpiprt5 at acpi0: bus -1 (RP04) acpiprt6 at acpi0: bus -1 (RP05) acpiprt7 at acpi0: bus -1 (RP06) acpiprt8 at acpi0: bus -1 (RP07) acpiprt9 at acpi0: bus -1 (RP08) acpiprt10 at acpi0: bus 1 (PEG0) acpiprt11 at acpi0: bus -1 (PEG1) acpiprt12 at acpi0: bus -1 (PEG2) acpiprt13 at acpi0: bus -1 (PEG3) acpicpu0 at acpi0: C3, C2, C1, PSS acpicpu1 at acpi0: C3, C2, C1, PSS acpicpu2 at acpi0: C3, C2, C1, PSS acpicpu3 at acpi0: C3, C2, C1, PSS acpipwrres0 at acpi0: FN00 acpipwrres1 at acpi0: FN01 acpipwrres2 at acpi0: FN02 acpipwrres3 at acpi0: FN03 acpipwrres4 at acpi0: FN04 acpitz0 at acpi0: critical temperature is 106 degC ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 3093 MHz: speeds: 3101, 3100, 3000, 2900, 2800, 2700, 2600, 2500, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 vendor \u0026ldquo;Intel\u0026rdquo;, unknown product 0x0158 rev 0x09 ppb0 at pci0 dev 1 function 0 \u0026ldquo;Intel Xeon E3-1200v2 PCIE\u0026rdquo; rev 0x09: msi pci1 at ppb0 bus 1 mpii0 at pci1 dev 0 function 0 \u0026ldquo;Symbios Logic SAS2008\u0026rdquo; rev 0x03: msi mpii0: PERC H200A, firmware 7.15.8.0 IR, MPI 2.0 scsibus0 at mpii0: 42 targets sd0 at scsibus0 targ 1 lun 0: SCSI4 0/direct fixed naa.600508e0000000003cd970ac3a16d00c sd0: 953344MB, 512 bytes/sector, 1952448512 sectors ehci0 at pci0 dev 26 function 0 \u0026ldquo;Intel 6 Series USB\u0026rdquo; rev 0x04: apic 0 int 20 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 \u0026ldquo;Intel EHCI root hub\u0026rdquo; rev 2.00/1.00 addr 1 ppb1 at pci0 dev 28 function 0 \u0026ldquo;Intel 6 Series PCIE\u0026rdquo; rev 0xb4: msi pci2 at ppb1 bus 2 bnx0 at pci2 dev 0 function 0 \u0026ldquo;Broadcom BCM5716\u0026rdquo; rev 0x20: apic 0 int 16 bnx1 at pci2 dev 0 function 1 \u0026ldquo;Broadcom BCM5716\u0026rdquo; rev 0x20: apic 0 int 17 ehci1 at pci0 dev 29 function 0 \u0026ldquo;Intel 6 Series USB\u0026rdquo; rev 0x04: apic 0 int 23 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 \u0026ldquo;Intel EHCI root hub\u0026rdquo; rev 2.00/1.00 addr 1 ppb2 at pci0 dev 30 function 0 \u0026ldquo;Intel 82801BA Hub-to-PCI\u0026rdquo; rev 0xa4 pci3 at ppb2 bus 3 vga1 at pci3 dev 3 function 0 \u0026ldquo;Matrox MGA G200eW\u0026rdquo; rev 0x0a wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) pcib0 at pci0 dev 31 function 0 \u0026ldquo;Intel C202 LPC\u0026rdquo; rev 0x04 ahci0 at pci0 dev 31 function 2 \u0026ldquo;Intel 6 Series AHCI\u0026rdquo; rev 0x04: msi, AHCI 1.3 scsibus1 at ahci0: 32 targets ichiic0 at pci0 dev 31 function 3 \u0026ldquo;Intel 6 Series SMBus\u0026rdquo; rev 0x04: apic 0 int 19 iic0 at ichiic0 sdtemp0 at iic0 addr 0x19: mcp98243 spdmem0 at iic0 addr 0x51: 8GB DDR3 SDRAM ECC PC3-10600 with thermal sensor isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 kbc: cmd word write error pcppi0 at isa0 port 0x61 spkr0 at pcppi0 mtrr: Pentium Pro MTRR support uhub2 at uhub0 port 1 \u0026ldquo;Intel Rate Matching Hub\u0026rdquo; rev 2.00/0.00 addr 2 uhidev0 at uhub2 port 1 configuration 1 interface 0 \u0026ldquo;Avocent USB Composite Device-0\u0026rdquo; rev 1.10/0.00 addr 3 uhidev0: iclass 3/1 ukbd0 at uhidev0: 8 variable keys, 6 key codes wskbd0 at ukbd0 mux 1 wskbd0: connecting to wsdisplay0 uhidev1 at uhub2 port 1 configuration 1 interface 1 \u0026ldquo;Avocent USB Composite Device-0\u0026rdquo; rev 1.10/0.00 addr 3 uhidev1: iclass 3/1 ums0 at uhidev1: 3 buttons, Z dir wsmouse0 at ums0 mux 0 uhub3 at uhub1 port 1 \u0026ldquo;Intel Rate Matching Hub\u0026rdquo; rev 2.00/0.00 addr 2 uhub4 at uhub3 port 5 \u0026ldquo;Standard Microsystems product 0x2514\u0026rdquo; rev 2.00/0.00 addr 3 vscsi0 at root scsibus2 at vscsi0: 256 targets softraid0 at root scsibus3 at softraid0: 256 targets root on sd0a (f32d1747471a3037.a) swap on sd0b dump on sd0b bnx0: address d4:ae:52:c8:01:89 brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 bnx1: address d4:ae:52:c8:01:8a brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8\n","date":"8 May 2013","permalink":"/posts/2013-05-08/installer-openbsd-sur-une-dedibox-classic-gen2/","section":"Posts","summary":"Yop,\nLes services de poolp.org sont hébergés sur plusieurs serveurs.\nLe serveur principal à été basculé d\u0026rsquo;un petit hébergeur US \u0026ldquo;OpenBSD-friendly\u0026rdquo; vers Online.net depuis 3 ans maintenant.","title":"Installer OpenBSD sur une dedibox Classic+ Gen2"},{"content":"Yop,\nThis week has been very productive with several tickets closed and more about to be closed.\nEric has done an amazing work as you will soon realize ;-)\nI will push to our Github mirror and publish a snapshot this week-end very likely, until then don\u0026rsquo;t look for the features, only we have them !\nCompressed and Encrypted queue\nWhile working on bringing back encrypted queue support, I ran into a couple issues.\nFirst, I could use a compressed queue or an encrypted queue, but activating both would fail. This was strange because each one worked fine and their output was correct but somehow the combination led to errors. I tracked it down and figured that the reuse of the same buffer led to an overlap which didn\u0026rsquo;t happen before the switch to aes-256-gcm but would completely corrupt decryption now. Using two separate buffers for the compression and encryption was enough to solve this.\nThen, I realized that while everything was working flawlessly at runtime, OpenSMTPD would fail to load envelope at startup. After some tracking I isolated the bug to the queue_fsqueue.c backend which was violating API layers by trying to validate the envelope content while it should be done by the upper layer after decryption/decompression. The fix seemed tricky at first but was really about 10 lines to end up with.\nAs of my sandbox, I currently have an OpenSMTPD that runs with the following configuration file: listen on lo0\nqueue compression\ngenerate key with: openssl rand -hex 16 queue encryption key 95ca5f8053fc2baca7c390bea62bd9e2\naccept from any for domain poolp.org deliver to maildir accept for any relay All it lacks is better smtpctl integration so that we can \u0026ldquo;smtpctl show envelope\u0026rdquo; on an encrypted envelope :-)\ntable_proc API\nA feature I have wanted for many months is the ability to externalize table backends so that we can write very funky backends with dependencies that are never going to make it to the OpenBSD tree.\nFor example, people are going to want to store their aliases in mysql or pgsql and while the feature is easy to achieve thanks to the table API, we can\u0026rsquo;t provide it out of the box because of the dependencies AND people can\u0026rsquo;t contribute it easily because tables are part of the daemon and will require rebuilding after some patching.\nSo what I wanted was to have table backends work like our filters: they are standalone programs that are executed at startup by OpenSMTPD and communication takes place through the imsg(3) framework.\nEric took that idea, improved it and made a working implementation. So now, one can write a custom backend without having us do anything on our side, with whatever dependencies wanted, and OpenSMTPD can make use of it by providing the path in the configuration file:\ntable aliases \u0026ldquo;proc:/usr/libexec/smtpd/backend-table-sqlite -f /etc/mail/sqlite.conf /etc/mail/sqlite-aliases.db\u0026rdquo;\nlisten on lo0\naccept from any for domain poolp.org alias deliver to maildir accept for any relay\nWriting a custom backend implementation is very simple, fill in the blanks: /* * Copyright (c) 2013 Eric Faurot eric@openbsd.org * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED \u0026ldquo;AS IS\u0026rdquo; AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */\ninclude include include include \u0026ldquo;smtpd-defines.h\u0026rdquo; include \u0026ldquo;smtpd-api.h\u0026rdquo; static int table_stub_update(void); static int table_stub_check(int, const char *); static int table_stub_lookup(int, const char *, char *, size_t); static int table_stub_fetch(int, char *, size_t);\nint main(int argc, char **argv) { int ch;\nwhile ((ch = getopt(argc, argv, \u0026ldquo;f:\u0026rdquo;)) != -1) { switch (ch) { default: errx(1, \u0026ldquo;bad option\u0026rdquo;); /* NOTREACHED */ } } argc -= optind; argv += optind;\ntable_api_on_update(table_stub_update); table_api_on_check(table_stub_check); table_api_on_lookup(table_stub_lookup); table_api_on_fetch(table_stub_fetch); table_api_dispatch(); return (0); } static int table_stub_update(void) { return (-1); }\nstatic int table_stub_check(int service, const char *key) { return (-1); }\nstatic int table_stub_lookup(int service, const char *key, char *dst, size_t sz) { return (-1); }\nstatic int table_stub_fetch(int service, char *dst, size_t sz) { return (-1); }\nSexy hu ? :-)\nqueue_proc API\nSince he was already deep into the proc-ification of stuff, he also proc-ified the queue which was another feature I have wanted real bad for a very long time.\nSo it is now possible to write a queue backend with whatever dependency you want and have OpenSMTPD use it through a single configuration line.\nAnd just like for tables, writing a custom queue is pretty easy, just fill in the blanks: /* * Copyright (c) 2013 Eric Faurot eric@openbsd.org * * Permission to use, copy, modify, and distribute this software for any * purpose with or without fee is hereby granted, provided that the above * copyright notice and this permission notice appear in all copies. * * THE SOFTWARE IS PROVIDED \u0026ldquo;AS IS\u0026rdquo; AND THE AUTHOR DISCLAIMS ALL WARRANTIES * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */\ninclude include include include \u0026ldquo;smtpd-defines.h\u0026rdquo; include \u0026ldquo;smtpd-api.h\u0026rdquo; include \u0026ldquo;log.h\u0026rdquo; static int queue_stub_message_create(uint32_t *msgid) { return (0); }\nstatic int queue_stub_message_commit(uint32_t msgid) { return (0); }\nstatic int queue_stub_message_delete(uint32_t msgid) { return (0); }\nstatic int queue_stub_message_fd_r(uint32_t msgid) { return (-1); }\nstatic int queue_stub_message_fd_w(uint32_t msgid) { return (-1); }\nstatic int queue_stub_message_corrupt(uint32_t msgid) { return (0); }\nstatic int queue_stub_envelope_create(uint32_t msgid, const char *buf, size_t len, uint64_t *evpid) { return (0); }\nstatic int queue_stub_envelope_delete(uint64_t evpid) { return (0); }\nstatic int queue_stub_envelope_update(uint64_t evpid, const char *buf, size_t len) { return (0); }\nstatic int queue_stub_envelope_load(uint64_t evpid, char *buf, size_t len) { return (0); }\nstatic int queue_stub_envelope_walk(uint64_t *evpid) { return (0); }\nint main(int argc, char **argv) { int ch;\nwhile ((ch = getopt(argc, argv, \u0026ldquo;f:\u0026rdquo;)) != -1) { switch (ch) { default: log_warnx(\u0026ldquo;warn: backend-queue-stub: bad option\u0026rdquo;); exit(1); /* NOTREACHED */ } } argc -= optind; argv += optind;\nqueue_api_on_message_create(queue_stub_message_create); queue_api_on_message_commit(queue_stub_message_commit); queue_api_on_message_delete(queue_stub_message_delete); queue_api_on_message_fd_r(queue_stub_message_fd_r); queue_api_on_message_fd_w(queue_stub_message_fd_w); queue_api_on_message_corrupt(queue_stub_message_corrupt); queue_api_on_envelope_create(queue_stub_envelope_create); queue_api_on_envelope_delete(queue_stub_envelope_delete); queue_api_on_envelope_update(queue_stub_envelope_update); queue_api_on_envelope_load(queue_stub_envelope_load); queue_api_on_envelope_walk(queue_stub_envelope_walk); queue_api_dispatch(); return (0); } The awesome part is that this layer considers the queue as a key-value store and unless it WANTS to inspect content, it can consider it as a blob\u0026hellip; and since this layer is below the encryption and compression layer, a backend that doesn\u0026rsquo;t need to inspect content will magically support the compression and encryption feature out of the box ;-)\nIf someone is interested in writing a queue_torrent/queue_git or queue_whateverdecentralizedtechnology, I will help as I have ideas around decentralized encrypted queues\u0026hellip;\nscheduler_proc\nNo, just kidding, he didn\u0026rsquo;t do that much. scheduler_proc should be next ;-)\nassorted improvements\nThere\u0026rsquo;s been various improvements here and there on my part.\nI have improved logging so that relaying displays the source address as well as the relay session.\nI have a pending diff that is waiting for an OK and which ensures that OpenSMTPD copes with corrupted envelopes at runtime by moving them to the /corrupt queue. Currently, if an admin edits manually an envelope and fucks up, the daemon will abort.\nI also found a missing check within the imsg API that can cause OpenSMTPD and possibly other daemons to fatal() under some situations. The diff is ready and waiting for okays from other OpenBSD hackers.\nThat\u0026rsquo;s about all I think ;-)\nWant to make us happy ?\nI will take the opportunity to let you know, in case you missed it, that after a lot of hesitation and feeling bad about it, Eric has asked for a laptop or small donations to help him get one. So far he hasn\u0026rsquo;t received much which is a shame considering how shitty his laptop is and how much time he spends working on it to write tricky DNS and SMTP code ;-)\nConsider making him a small paypal donation ( eric@openbsd.org) if you like his work, he really deserves not to work on that trashpile.\nAs for me, feel free to Flattr or head your spare bitcoins my way (13tTchonwhjvLRo2NcUwyqfKrTvKMLBWRi) as it automagically converts into much needed beers ;-)\nThe OpenSMTPD book\nThe OpenSMTPD book initially planned for late May will be delayed slightly as my health issues (nothing worrisome, just annoying stuff) caused me to stop writing for weeks. I will resume by next week but I don\u0026rsquo;t want to rush just to make it to the deadline I imposed myself \u0026hellip; so it\u0026rsquo;ll probably be released sometime this Summer.\n# I hope, you\u0026rsquo;ll enjoy it because it\u0026rsquo;s proved to be much more efforts than I assumed ;)\nMore next week !\n","date":"26 April 2013","permalink":"/posts/2013-04-26/opensmtpd-table_proc-queue_proc-crypto-queue-and-other-stuff/","section":"Posts","summary":"Yop,\nThis week has been very productive with several tickets closed and more about to be closed.\nEric has done an amazing work as you will soon realize ;-)","title":"OpenSMTPD: table_proc, queue_proc, crypto queue and other stuff"},{"content":"OHAI,\nI have not updated this blog since the release of OpenSMTPD 5.3, over a month ago. In the meantime, some minor bugs have been fixed and we have released OpenSMTPD 5.3.1.\nThe main reason for the silence is that I was busy with completely unrelated day-time work, unrelated opensource code, the OpenSMTPD book in progress and some minor health issues that have caused endless side-effects keeping my mind away from this blog.\nAnyway, starting next week I should be actively blogging about developments in progress as we have resumed working on new features that I\u0026rsquo;m looking forward to see committed to master. For the time being, here\u0026rsquo;s a few features that have been committed or that are being worked on actively at the moment:\nhelo no longer requires source The \u0026ldquo;helo\u0026rdquo; features allows OpenSMTPD to advertise a particular helo hostname when relaying to another MX. It does so by looking in a table for a helo entry matching the IP address that it has used to connect to the MX. The feature required that both a source table AND a helo table be used for the helo feature to work, but this was an unrequired constraint and prevented some setups to be possible as reported by Todd Fries, so I simply removed the constraint. This has been committed to master and will be part of new snapshots.\n\u0026ldquo;error\u0026rdquo; alias type When writing code, we often have to test how be behave when facing a success, permanent failure or temporary failure. To achieve this, we usually create mail accounts that discard content, that have broken ~/.forward files (temporary failure) or we simply mail non-existent accounts (permanent failure). Todd had another use case where he wanted a virtual domain to have accounts that returned errors. I implemented a new alias type, \u0026ldquo;error\u0026rdquo;, which allows an alias to expand to an error code and message:\ntable aliases { foobar = \u0026ldquo;error:550 you\u0026rsquo;re not allowed to mail foobar\u0026rdquo;, barbaz = \u0026ldquo;error:450 woops, temporary failure\u0026rdquo; }\nThe error alias type can be used for regular aliases, virtual domains as well as ~/.forward files if a user temporarily wants to bounce mails to his account. It only supports 5xx and 4xx codes. It has been committed to master and will be part of new snapshots.\nOffline queue is broken Charles Longeau had spotted a strange issue where mail submitted while the daemon was offline would not be enqueued correctly when the daemon would start. I had never seen this but my OpenSMTPD instance is almost never down so the offline enqueuer is barely used. I tried to reproduce without success and eventually managed to hit the issue.\nIt turns out that this was only a permission problem and that mails submitted by the root user would manage to be enqueued, while mails submitted by other users would fail and remain in the offline queue. The issue has been fixed, committed and upcoming snapshots will have the fix.\nQueue spin Elbarto, a FreeBSD user, has reported that after changing permissions of files in the queue, he got the queue process to go crazy and spin. This looked strange because OpenSMTPD checks permissions at startup and should warn the admin if they are inappropriate. It turns out that the issue happened when the admin would change permissions of some directories in the spooler AFTER startup of the daemon.\nThis is an uncommon case and people should never hit in regular use, but the daemon should cope with these basic errors and not go crazy. It took some time to reproduce but I eventually found the culprit and came up with a fix that was committed. It will be part of upcoming snapshots.\nLog envelope source A user reported that we didn\u0026rsquo;t log the IP address that was used during a relaying when we log the status of the envelope. In setups that make use of multiple IP addresses, it makes it harder to track the path of an envelope while adding the source address is trivial. The log format for envelopes now contain an additional \u0026ldquo;source\u0026rdquo; field with the IP address that was used for the relaying. It will be part of new snapshots.\nIPv6 over IPv4 preference ftigeot@ from DragonFlyBSD reported strange behaviour where IPv6 was tried AFTER IPv4 which is inappropriate on that system. This behaviour is caused by OpenBSD\u0026rsquo;s default IPv4 before IPv6 search strategy, which turns out to be quite the opposite of what other systems assume. We decided to have portable OpenSMTPD use IPv6 before IPv4 strategy since that is what most people expect. Charles committed the fix to ASR with Eric\u0026rsquo;s and my approval.\nK_ADDRNAME not supported by db backend Todd Fries reported that he wasn\u0026rsquo;t able to use a db(3)-backed table as a helo table. It turns out that the db backend didn\u0026rsquo;t support the K_ADDRNAME service. The fix was trivial and committed shortly after. New snapshots will support it.\nEncrypted queue I have started working again on encrypted queue and the implementation is currently being discussed with other OpenBSD hackers. I\u0026rsquo;m expecting it to be committed within the next two weeks. It will allow people to have their mail queue transparently encrypted and protected against tampering. It has absolutely no use in a self-hosted environment, but will allow users to host their own mail servers while delegating the hosting of the queue to a company they trust for hosting but not for privacy (ie: I\u0026rsquo;d trust Google to store my queue safely, but I don\u0026rsquo;t trust them not to look at it).\nQueue encryption is compatible with queue compression and can be enabled with:\nqueue encryption key \u0026ldquo;foobarbaz\u0026rdquo;\nThe key will be expanded using 100.000 iterations of pkcs5_pbkdf2 and a random salt for each envelope and message. Each envelope and message will be encrypted with AES-256 in GCM mode, providing encryption and integrity, with a random IV for each. I\u0026rsquo;m looking forward to that feature :-)\nWhat\u0026rsquo;s coming next ? A big surprise. Eric is working on a feature that I\u0026rsquo;ve been wanting for MONTHS. It will be huge.\nStay tuned ;-)\n","date":"21 April 2013","permalink":"/posts/2013-04-21/news-from-the-front/","section":"Posts","summary":"OHAI,\nI have not updated this blog since the release of OpenSMTPD 5.3, over a month ago. In the meantime, some minor bugs have been fixed and we have released OpenSMTPD 5.","title":"News from the front"},{"content":"OHAI !\nI\u0026rsquo;ve stayed silent for the last month as we got ready for the big thing\u0026hellip;\nCharles, Eric and I are proud to announce the release of OpenSMTPD 5.3, our first official production-ready release !\nOpenSMTPD has been in use at poolp.org since 2008 and these last five years have seen a considerable amount of work. At various times we have been contemplating a release then delaying to add that one thing that seemed essential. Today, the code we release is still not perfect but we are very happy with it and it has proved to do the work pretty fine. It will continue to improve and mature from now on, and we hope you will enjoy it ;-)\nBelow is a copy of the release mail, see you soon for updates on new features ;-)\nOpenSMTPD 5.3 has just been released.\nOpenSMTPD is a FREE implementation of the SMTP protocol with some common extensions. It allows ordinary machines to exchange e-mails with systems speaking the SMTP protocol. It implements a fairly large part of RFC5321 and can already cover a large range of use-cases.\nOpenSMTPD has been under development for a long time now and many people are already using it in production, but this is our first stable release and if you were waiting for a GO! from us, here it is ;-)\nThe archives are now available from the main site at www.OpenSMTPD.org\nWe would like to thank the OpenSMTPD community for their help in testing the snapshots, reporting bugs, contributing code and packaging for other systems.\nFeatures: HUMAN READABLE CONFIGURATION IPv4 and IPv6 support STARTTLS and SMTPS support for both incoming and outgoing sessions AUTH support: bsd_auth(3) and crypt(3) SIZE support: limit the size of client-submitted messages Listener-specific banner hostname Listener-specific sessions tagging Support for global and per-domain expiry for messages Support for customizable delays for bounces Support for primary and virtual domains Support for alternate user database: db(3), file or smtpd.conf Support for aliases and ~/.forward mappings Delivery to mbox, maildir or third-party MDA Support for LMTP relaying Support for smarthost Support for sending certificate when connecting to remote host Support for backup MX Support for relay source address override Support for relay HELO override Support for SMTP-level sender override Support for connections reuse and optimization Support for queue backends: filesystem and ram Support for lookup backends: db(3), static Run-time statistics through \u0026ldquo;smtpctl show stats\u0026rdquo; Run-time tracing through \u0026ldquo;smtpctl trace \u0026quot; Run-time monitoring through \u0026ldquo;smtpctl monitor\u0026rdquo; Experimental:\nSQLite lookup backend LDAP lookup backend Portable:\nSupport for PAM authentication Known to build and work on FreeBSD, NetBSD, DragonFlyBSD and Linux Limitations:\nNo filters support yet (work in progress) No masquerading or address rewrite yet Checksums: SHA256 (opensmtpd-5.3.tar.gz) = 05efe80755e7fa01e79e6bba1a4e89244849406acb1152995d2c1da5e9e3a596\nSHA256 (opensmtpd-5.3p1.tar.gz) = 618092f1f0b5aba5f8d4c933536a76d3a5a8e45c28b599a6420321cd4478f3d9\nSupport: You are encouraged to register to our general purpose mailing-list: http://www.opensmtpd.org/list.html\nThe \u0026ldquo;Official\u0026rdquo; IRC channel for the project is at: #OpenSMTPD @ irc.freenode.net\nReporting Bugs: Please read http://www.opensmtpd.org/report.html Security bugs should be reported directly to security@opensmtpd.org Other bugs may be reported to bugs@opensmtpd.org\nOpenSMTPD is brought to you by Gilles Chehade, Eric Faurot and Charles Longeau.\n","date":"17 March 2013","permalink":"/posts/2013-03-17/opensmtpd-5.3-released/","section":"Posts","summary":"OHAI !\nI\u0026rsquo;ve stayed silent for the last month as we got ready for the big thing\u0026hellip;\nCharles, Eric and I are proud to announce the release of OpenSMTPD 5.","title":"OpenSMTPD 5.3 released"},{"content":"OHAI,\nWe had planned to slow down a bit to prepare for release: no more feature, no more invasive changes until we tag something out of which we can publish a release. I then discussed with Bret Lambert (blambert@) and he asked me if we used some kind of code scanner to detect defects in our code.\nWe do use the clang static analyzer every now and then, the last time being a couple week ago, and we spend a few minutes cleaning up the defects we believe to be real issues.\nBret then asked me if I had ever tried the scanner at Coverity and told me I should try to get our codebase scanned with it. I recalled seeing a summary of issues uncovered by their scanner on Apache a while ago and it was very impressive, so I asked to register OpenSMTPD and was happy when they confirmed I could use their scanner without having to sign any NDA, terms of services or random lawyer documents [rare and appreciable] :-)\nSo here\u0026rsquo;s a summary of this week\u0026rsquo;s work, all of it has been committed and merged in the OpenBSD tree and Github mirror. The latest snapshot is up to date.\nTRACE_LOOKUP\nA user reported on IRC that he had an issue with aliases. The issue turned out to be a user error but it prompted me into improving the aliases expansion logging.\nUntil now, to follow the aliases expansion logic, one had to run in debug mode to get to see the tons of log_debug() we have there. I introduced TRACE_LOOKUP and replaced the log_debug() with log_trace() which allow to turn the logging on or off at runtime with the command smtpctl trace lookup\nMTA logging improvement\nEric and I thought that we didn\u0026rsquo;t log enough information when relaying. The incoming path logs many details about sessions, but the outgoing path logged almost exclusively SSL-related details. So Eric spent some time adding proper logging to te MTA process, which should make it much easier for an admin to understand what happens by reading logs ;-)\nImproved memory usage\nEric introduced the mta_envelope to represent an envelope within the mta process. This allows keeping in memory ONLY the information needed by the mta and shrinks a lot the memory usage when the daemon is under a lot of stress.\nHe also introduced the mda_envelope structure to achieve the same for the mda process and raised limits to allow more concurrent local deliveries to take place.\nBug fixes\nFinally, most of the work this week has been spent on fixing bugs reported by the code scanner provided by Coverity.\nIt doesn\u0026rsquo;t run on OpenBSD but since it runs on Linux and our portable version is very close to the native version, we could run the test on Linux and fix on OpenBSD.\nWe didn\u0026rsquo;t find anything really scary, probably due to the fact we already cleaned and removed scary code detected by Clang, but we did find lots of little issues that we wouldn\u0026rsquo;t spot just from reading, even several times.\nAmongst the issues, we found a descriptor leak, several small memory leaks, possible double-free, possible double-close, possible double-fclose, use-after-free, various missing bzero, dead code and null-derefs. Many of these would lead to a crash on OpenBSD but the reason we didn\u0026rsquo;t experience them is that they happened in error code paths for the most part. We have fixed about 40 issues, there are 40 still, with several part of external code (mail.local, aldap, etc) and some we believe are false positives but require further reading with a clear mind. The scanner manages to find errors which makes you wonder if they really are, until you realize that it does a far better job at unrolling the code ;-)\nThis teaches us two lessons: find a way to better test error paths, run the scanner from Coverity regularly !\nAnyways, all related commits are available from Github in case you\u0026rsquo;re interested, as usual I\u0026rsquo;ll document funky bugs here so stay tuned.\n","date":"2 February 2013","permalink":"/posts/2013-02-02/opensmtpd-minor-fixes--preparing-release/","section":"Posts","summary":"OHAI,\nWe had planned to slow down a bit to prepare for release: no more feature, no more invasive changes until we tag something out of which we can publish a release.","title":"OpenSMTPD: minor fixes + preparing release"},{"content":"OHAI,\nLately we have stabilized and decided that we\u0026rsquo;re okay with the features we have for our first release, this time for good, so we focused on making sure that we didn\u0026rsquo;t introduce regressions and that we were rock solid.\nWe stressed incoming and outgoing path and found areas of improvements. This will be a long-standing task, but we have already improved memory usage considerably under high-load, we\u0026rsquo;re pretty happy with the result.\nAll of the following has been committed, we published snapshots and we sync-ed the OpenBSD tree with our own repository so that OpenBSD-current now has all of the features, improvements and bug fixes we came up with in the last few weeks (or months for some stuff we didn\u0026rsquo;t backport).\nAnyway, here goes a summary and I\u0026rsquo;m off to bed as I\u0026rsquo;m sick as shit.\nfix a descriptor leak\nWe were told of a crash which appeared to be coming from a descriptor leak. On Linux, lsof would display that message files were removed while the transfer process still had a descriptor opened. We tracked the issue and spotted that it would happen because of a missing condition in the transfer process where the message file would not be closed if recipients were rejected and the session was reused to send another message. Two liner fix.\nSSL code cleanup\nAn OpenBSD user reported a problem with ldapd\u0026rsquo;s ssl.c where the prime used for DH parameters was 512 bits which is short enough by today\u0026rsquo;s standards to be rejected by OpenLDAP\u0026rsquo;s client. OpenSMTPD\u0026rsquo;s ssl.c has been changed a long time ago to bump this prime to a 1024 bits prime, but ldapd\u0026rsquo;s ssl.c was actually a copy of OpenSMTPD\u0026rsquo;s ssl.c from two years ago.\nThe desynchronization of ssl.c accross OpenBSD daemons has annoyed me for a long time but it occured to me that maintaining this gap would be causing further divergences in the future as reyk@ moves OpenIKED and relayd forward. After a quick chat with him I started creating a daemon-agnostic version of ssl.c.\nThere is still work to do in that area, but OpenSMTPD now comes with a ssl_privsep.c that is equivalent to that of relayd; a ssl.c file that no longer knows of any smtpd specific structures and which can be shared with other daemons; a ssl_smtpd.c that contains the smtpd-specific bits.\nNote that ssl.c doesn\u0026rsquo;t contain new code, this was only a rework of the interfaces to allow it to be shareable with different daemons.\nRuntime tracing and profiling\nI have added a feature that I\u0026rsquo;ve been wanting for a long time: activating traces at runtime without a daemon restart.\nSay a user suddenly observes that connections to a remote host fails, he can now simply type: smtpctl trace transfer\nTo obtain a real-time view of the sessions as they take place with remote hosts.\nThere are many trace subsystems, for incoming connections, outgoing connections, msg exchanges accross processes, etc, etc, \u0026hellip; read the man page ;-)\nWhile at it, if you need to verify bottlenecks OpenSMTPD also supports real-time profiling of imsg and queue using: smtpctl profile imsg and smtpctl profile queue\nBoth tracing and profiling can be turned on and off at runtime.\nImprove memory use\nEric cleaned up some code to avoid passing envelopes when we could pass evpid instead. This prevents OpenSMTPD from accumulating data in the inter-process buffers (an evpid is 64bits, an envelope structure is several kbytes).\nFor the places where the envelope really needs to be passed, he suggested that we send a compressed version. I proposed that we use the ASCII envelope conversion as we know it works given that it\u0026rsquo;s already used for disk-based envelopes. The ascii envelopes allow compressing the envelope quite a lot as instead of passing a large datastructure with partially used large fields, we pass the ascii representation that can be as small as a hundred bytes.\nWe should now be able to cope better in very heavily loaded situations ;-)\n","date":"25 January 2013","permalink":"/posts/2013-01-25/opensmtpd-gentlemen-fasten-your-seatbelt/","section":"Posts","summary":"OHAI,\nLately we have stabilized and decided that we\u0026rsquo;re okay with the features we have for our first release, this time for good, so we focused on making sure that we didn\u0026rsquo;t introduce regressions and that we were rock solid.","title":"OpenSMTPD: gentlemen, fasten your seatbelt"},{"content":"\u0026lt;p\u0026gt;Hey,\u0026lt;/p\u0026gt; I wanted to publish something last Friday but I unfortunately got my hands on a copy of Super Nintendo's Zelda which led to an instant loss of motivation.\nHere's a quick summary of what happened recently in OpenSMTPD-land, it's not an exhaustive description and to be honest I'm keeping a lot of information private at this point and probably until release ;-)\nA snapshot should be published tomorrow.\nStressing the daemon\nFirst of all, we have done a lot of stress testing lately.\nWe have tested the incoming path, accepting hundreds of thousands of mails from hundreds of sessions, arriving in chaotic order and with random data. We're quite confident that our incoming path is rock solid now, we will be performing our final test very soon with one billion messages.\nWe have also tested our outgoing path, first in a confined environment which revealed no bugs, and then in a live environment which revealed minor bugs that where fixed in the process. There is still one \"memory usage\"-related issue but it applies to very special and stressed setups, not something people would usually experience, and we happen to have a diff for it which requires some additional work before being in a state proper for commit.\nDuring these stress tests, we have gathered a lot of information to prove some of our theories right and wrong. We now know where we stand with regard to other MTA and we have built tools which allow us to have a very precise understanding of our areas of improvements. Clearly, we have no reason to be ashamed, far from it.\noptimization\nOur queue code has a design that allows for very efficient queues to be written. We know that because we have already written several backends, we know the time spent in the API, the time spent in the backends and for disk-based backend the exact cost of the disk IO.\nYou'd assume our queue would be very fast, but the default backend we ship is sub-optimal by design because the admin that is in me wants the queue to provide some features that are incompatible with performances.\nOne of these features is to provide per-envelope files as well as a locality between envelopes and messages so that SMTP transactions can be backed up and restored easily on another machine. Stuff like that.\nThis user-friendliness means that we can't rely on tricks to avoid hitting the disk too often, we have as many open()-fsync()-close() as there are envelopes; we have as many mkdir()/rename() as there are messages; and you can add many more file-system related calls used to deal with the atomicity of our queue commits. A lot of this is unnecessary for OpenSMTPD and could be handled in a much more efficient way... but is really only done so that a human can inspect and manipulate the queue more easily.\nA queue that allowed for envelopes to be written sequentially in a single binary file that would require only one fsync() could increase drastically our performances, and we will surely do that at some point, but our default queue will always be the user-friendly one even at the cost of a slower incoming path.\nThat being said, we still want our queue to operate fast and limit the impact of our design. Ideally, our queue shouldn't be more than 10% slower than the other software with their optimized queues. So...\nI spent some time tracing our queue code and spotted that the system calls pattern was not really matching what I'd expect. Our queue logic is very simple and has a very \"linear\" pattern, several system calls should have a matching number of calls. I tracked and fixed until a kdump output displayed the optimal number of calls for each system calls according to the numbers I had on paper. I added a couple functions to help profile every single queue operation, Eric came up with a better interface for these functions and we now have an invaluable tool for queue backend development :-)\nEric also spent time improving our memory usage and removing some pressure from inter-process IO by coming up with an API that allows to encode/decode the data more efficiently. Until now we passed structures, which could contain huge buffers for which we only consumed a few bytes; now the new API not only passes only the required data but also provides type checking which allows us to make sure we don't pass the wrong data by mistake. As a bonus, the API can use the types to know the average size of the data and only reallocate in cases where we exceed that size.\nWhen we were done with this, we were very good with the outgoing path, and we were pretty much equivalent to the other MTA with regard to the incoming path. There is still a lot of room for improvement, but given the constraints we imposed ourselves I'm really glad that we're not twice as slow as the slowest MTA out there :-)\nMore resistant fs-queue\nOur default queue had a design that was very strict with \"strange\" situations.\nAny time an unexpected situation happened, the daemon would fatal. Since unexpected situations are not supposed to happen, this shouldn't be a problem right ?\nNo. Not right. On a regular setup this never happens, but sometimes a human does something as innocent as a chmod on a file or directory... and OpenSMTPD figures something is not normal and commits suicide.\nTo be honest not only did I not receive a report of a queue fatal in months, but I also don't recall ever hitting any of these on poolp.org... until the stress.\nI hit a couple fatal() which turned out to be related to an error in our use of the fts(3) API which for some reason didn't trigger until the stress. I fixed the issue then decided to go track every single fatal() in the queue code and try to convert it into a temporary failure condition so that even if a failure happened, the daemon would deal with it gracefully.\nIt turned out to be much simpler than I assumed and our fs-queue is now capable of coping with an admin messing with the queue. Of course, an admin should never tweak with the queue, but being able to not fatal() on a chmod or mv felt like essential ;-)\nPer-listener hostname\nOur smtpd.conf file had a \"hostname\" directive which allowed setting the hostname to display on the greeting banner.\nThe directive was removed and it is now possible to specify the hostname for each listener:\nlisten on lo0 listen on 192.168.1.1 hostname mx1.int.poolp.org listen on 192.168.2.1 hostname mx2.int.poolp.org Not specifying one will use the machine's hostname.\nPer-source HELO\nWhen relaying mail, smtpd.conf allows for overriding the source address using a table containing an address or a list of addresses:\ntable myaddrs { 192.168.1.2, 192.168.1.3 }\naccept for all relay source The above causes the relaying to bind one of these addresses a the source address. However, during a SMTP session, our MTA has to advertise itself at the HELO/EHLO stage and tell its hostname. The hostname is sometimes checked and if it doesn't match the source address used, the MTA is rejected.\nSo we needed a way to have our MTA provide the remote host with a HELO/EHLO parameter that corresponds with the source address used. We had ideas that involved performing a DNS lookup from MTA but it would not work due to NAT.\nI suggested that we use a new lookup service K_ADDRNAME which allows for a mapping of an IP address to a name:\ntable myaddrs { 192.168.1.2, 192.168.1.3 } table myhelo { 192.168.1.2 =\u003e mx1.poolp.org, 192.168.1.3 =\u003e mx2.poolp.org }\naccept for all relay source helo With the following, MTA will use a source from the table myaddrs and, at HELO/EHLO time, will use the name from the table myhelo that matched the address it used to connect.\nSender filtering\nAnother feature that people have been requesting very frequently is the ability to use a sender email address as a matching condition in the ruleset.\nUntil recently, the matching of a rule was done by looking at the client address and the destination domain. It was not possible to express something like \"accept all mail coming with sender gilles@poolp.org\" or \"reject all mail coming with sender @redhat.com\".\nI have introduced the \"sender\" filtering which can apply to a full email address or to a domain, both in accept and reject rules. It works as follow:\naccept from any source, if sender has domain @openbsd.org [...] accept from any sender \"@openbsd.org\" for any relay\naccept from localhost, if sender has domain @poolp.org [...] accept sender \"@poolp.org\" for any relay\naccept from any source, only if sender is gilles@poolp.org [...] accept sender gilles@poolp.org for any relay It can apply to relay or deliver rules, and allows the use of tables to apply different relay rules to different domains or users coming from different networks.\ntable hackers { \"@opensmtpd.org\", \"@poolp.org\" } table slackers { richard@foot-cheese.org, lennart@thepig.org }\naccept from 192.168.1.0/24 sender relay accept from 192.168.2.0/24 sender relay via smtp://example.org Various little fixes\nYou have no idea.\nWe have fixed various little bugs that triggered in very specific cases which you just can't hit out of a live test.\nWe also fixed/improves/rework minor things, like making the \"relay backup\" parameter optional by picking the machine hostname, changing the API for queue remove to take an evpid instead of a full envelope, removing userinfo from a structure that wasn't using it, switching the queue code to use the first two bytes of a msgid instead of the last two bytes to create the bucket, fixing a segfault with a specific configuration file, allowing authentication to fail temporarily, etc, etc, etc ...\nOh and on a completely useless and unrelated note:\nI installed OpenSMTPD on a raspberry, so here's a picture of my mailserver at home:\n","date":"18 January 2013","permalink":"/posts/2013-01-18/opensmtpd-stress-and-features/","section":"Posts","summary":"\u0026lt;p\u0026gt;Hey,\u0026lt;/p\u0026gt; I wanted to publish something last Friday but I unfortunately got my hands on a copy of Super Nintendo's Zelda which led to an instant loss of motivation.","title":"OpenSMTPD: stress and features"},{"content":"YO!\nI know I should have posted a \u0026ldquo;Happy New Year\u0026rdquo; note earlier but you know how it is: new year eve\u0026rsquo;s aftermath, days to recover, work resumes, mood slacks, bleh.\nAnyway, today\u0026rsquo;s note is simply about that: wishing you and me an awesome year 2013 and hoping you will not turn it into a fucking mess. After all, you\u0026rsquo;re in charge so we know who to blame ;-)\nAs far as I\u0026rsquo;m concerned, year 2012 wasn\u0026rsquo;t \u0026ldquo;great\u0026rdquo;, it was not the absolute suck either, it was even quite decent. I resumed music more seriously; started sponsoring two little kids in Lebanon and Palestine; moved to a nicer place in Paris; played some metal with my friends of Spinach Blast; visited my family and friend in Lebanon; hacked and shared burgers with fellow OpenBSD hackers in Hungary; ate a fair share of ice-creams in Nice, southern France; visited buddies in Nantes, western France; sadly left great co-workers and friends at a company \u0026hellip; to join a fun crew at a new company and work on my own project for a few months.\nOk, despite some down times, 2012 was a very decent year actually\u0026hellip; I probably missed a lot of things too, it was a busy year ;-)\n2013 will be an even better year I hope.\nI\u0026rsquo;m supposed to spend the next few months working full-time on OpenSMTPD to make it fit the needs of my employer; by then, I will be done with paying back my car which will effectively set me free from bank slavery; this will of course happen right before I move back to Nantes from where I\u0026rsquo;ll work remotely for my employer on new challenging tasks that are less OpenSMTPD-centric; and being away from Paris will allow me to save enough money by the end of the year to try and convince my banker to enslave me for a couple decades so I can buy an appartment with a view on the Erdre river. Pffffiou\u0026hellip;\nThat\u0026rsquo;s the plan at least and I guess we\u0026rsquo;ll see next year how it went ;-)\nMeanwhile, Poolp.org enters its 6th year of providing free services based on open-source software to its little community of users. No changes planned on that front besides maybe more services and a more active blog that\u0026rsquo;s a bit more diverse than my OpenSMTPD hacking summaries.\nFor the time being, you g{al,uy}s have fun and I\u0026rsquo;ll just zZzZz\n","date":"8 January 2013","permalink":"/posts/2013-01-08/welcome-to-you-2013/","section":"Posts","summary":"YO!\nI know I should have posted a \u0026ldquo;Happy New Year\u0026rdquo; note earlier but you know how it is: new year eve\u0026rsquo;s aftermath, days to recover, work resumes, mood slacks, bleh.","title":"Welcome to you, 2013"},{"content":"OHAI,\nI was unable to write a story last week as my desk broke into pieces for no apparent reason but my foot hitting it accidentally.\nI still don\u0026rsquo;t have a desk so I will make this short by not telling about the queue profiling and SQL log conversion I wrote, and not even writing about the filter work done by Eric, he\u0026rsquo;ll write about it if he wants to or I\u0026rsquo;ll do that next time.\nYou guys enjoy New Year\u0026rsquo;s Eve, don\u0026rsquo;t get too drunk and remember to send tons of mails to those not around using an awesome piece of software :-p\nSSL verification and separation\nOpenSMTPD has had code to deal with establishing secure channels for a very long time now, however what it didn\u0026rsquo;t do was to perform certificate chain verification in both server and client modes. In server mode, it would never request a client certificate and in client mode it would never check that the certificate handed by the server was valid.\nSo I started adding support and hit the first issue: access to the CA bundle from within the chroot. After a discussion with reyk@ over what was the best way to deal with it, he convinced me that we could really improve our design by moving certs and keys from network-facing processes to a separate process and performing on-demand requests.\nDealing with OpenSSL was as nice as usual, it almost seemed like sharing a tasty dinner with Richard Stallman, but I eventually saw the light and got it to work as expected. The good part is that the client and server modes have symmetric operations which means that the code is identical for the most part.\nWe now have the sensitive stuff isolated in the lookup process. The smtp and mta processes use the imsg framework to request them and to pass over chains for verifications. It all fits in a very few lines of code which is cool because the less OpenSSL code I have to deal with, the better.\nFor now, we don\u0026rsquo;t do a full verification of the X509 attributes so OpenSMTPD lies about the verification and pretends it didn\u0026rsquo;t do it in the Received lines, however the admin can see the verification take place in the logs. I\u0026rsquo;ll fix the Received line to tell the truth when I\u0026rsquo;m confident the verification is 100% accurate.\nNow, before I switch subject, a couple related ideas for the future:\nIf we were to provide a K_CERT table service, we could fetch the certificates and chains from custom backends (ldap, sql, you name it, \u0026hellip;), this would take approximately one hour of work. The only reason I\u0026rsquo;m not doing it is that I don\u0026rsquo;t have a need for that at the moment ;-)\nAlso, the certificate verification reply to mta and smtp is a simple status that is either success or failure. This means that implementing a certificate-based authentication is now trivial, probably an hour of work too. Guess why I didn\u0026rsquo;t implement it yet ?\nFix relaying logic\nWhile working on the SSL code, I spotted strange behaviour when relaying between my primary and secondary MX.\nAfter some testing, I realized that we lost the \u0026ldquo;backup\u0026rdquo; flag somewhere. After a quick chat with eric@, I convinced him we should introduce the backup:// schema and get rid of the flag in the envelope. This way, we could make it obvious that mx2.poolp.org is a backup MX by having backup://mx2.poolp.org as a relay URL.\nThen, I spotted some strange issues where I didn\u0026rsquo;t request TLS and it would attempt it, or where I requested TLS but it would fallback to plaintext. It became obvious there was something fishy with the semantic of our relay URL schemas and that they needed to be more clearly defined.\nAfter going through every possible schema, we defined them as follow:\nsmtp+tls:// -\u0026gt; attempt TLS, then fallback to plaintext, this is the default smtp:// -\u0026gt; plain text ONLY, no encryption tls:// -\u0026gt; STARTTLS only, encrypted channel guaranteed smtps:// -\u0026gt; SMTPS only, encrypted channel guaranteed ssl:// -\u0026gt; STARTTLS, fallback to SMTPS, encrypted channel guaranteed\nWith this, unless smtp+tls:// is specified, we never fallback to plaintext if an encrypted channel is requested and we never break the user assumption that a relaying will be secure by sending the data over a plaintext channel.\nI guess that\u0026rsquo;s all for this time, I really need to zZzZ\n","date":"28 December 2012","permalink":"/posts/2012-12-28/opensmtpd-ssl-relay-stuff/","section":"Posts","summary":"OHAI,\nI was unable to write a story last week as my desk broke into pieces for no apparent reason but my foot hitting it accidentally.","title":"OpenSMTPD: SSL \u0026 relay stuff"},{"content":"\u0026lt;p\u0026gt;OHAI,\u0026lt;/p\u0026gt; This week has been crazy, my brain melted a little as I worked on filters, it melted a little as I worked on LDAP and it melted a little more as I tried to trick Eric into working on filters with me (yes, that actually worked :-).\nAnyway, as usual, this is just going to be a summary of what we did as tons of little stuff have been committed here and there. You can keep track of the changes by checking the commit log of our Github mirror, or by joining our little gang on our IRC channel: #OpenSMTPD @ freenode.\nSnapshots containing all the following features should be published tomorrow.\nLet the fun begin !\nLDAP backend\nI had started working on LDAP support for OpenSMTPD a long time ago but for some reason the support was never finished and ended up rotting in my sandbox.\nA few months ago, I brought the bits back to a git branch so that I would keep running into it every few days as a reminder that I should not slack. But since I'm not too much of a LDAP fan, or a LDAP user for what it's worth, I made the branch public in hope someone would pick it up and move it forward.\nA poolp user had started bringing the bits up to date and getting a working support in shape for aliases lookup. Resuming from there I simplified the code further and added support for almost all kinds of lookups making OpenSMTPD capable of using LDAP as a backend for the most common use-cases.\nHere's a configuration file to authenticate local users, lookup a domain and perform aliases lookups against LDAP:\n/etc/mail/smtpd.conf table myldaptable ldap:/etc/mail/ldapd.conf\nlisten on egress tls auth accept for domain alias deliver to maildir accept for any relay and here's the table configuration:\n/etc/mail/ldapd.conf url ldap://127.0.0.1 username cn=admin,dc=opensmtpd,dc=org password thisbemypasswd basedn dc=opensmtpd,dc=org\naliases lookup alias_filter (\u0026(objectClass=courierMailAlias)(uid=%s)) alias_attributes maildrop\ncredentials lookup credentials_filter (\u0026(objectClass=posixAccount)(uid=%s)) credentials_attributes uid,userPassword\ndomains lookup domain_filter (\u0026(objectClass=rFC822localPart)(dc=%s)) domain_attributes dc The support is functional but it needs to be improved further as it currently has two drawbacks: the backend does not reconnect to the LDAP server should it lose the connection, and it uses the aldap synchronous API which means that queries that take time to complete will be heavy on the lookup process.\nAlso, I only tested with OpenBSD's ldapd(8) as it was dead simple and I didn't want to add more pain than required on my shoulders. Turns out, it did make my experiment far more enjoyable that I would have assumed ;-)\nOh, and I ordered a LDAP book to get more familiar with the service as I suspect I'll be getting questions regarding LDAP every now and then given how many times it's been requested in the past. I might as well know what I'm talking about :-)\nSource address selection\nEric has plugged the K_SOURCE lookup service to relay rules, this allows OpenSMTPD to perform a lookup of the source address it should use when the transfer process establishes an outgoing connection to a relay.\nUntil now OpenSMTPD could not force the IP address it used for outgoing trafic without relying on a hardcoded hack that was committed to the poolp branch. It was done this way on purpose and we delayed this feature until the other parts were rewritten appropriately for the puzzle to fit right.\nIt is now possible to force an address using the source keyword:\ntable myaddrs { 88.190.237.114, 91.121.164.52 }\naccept for any relay source If multiple addresses are provided, they will be cycled through, and the mta code will detect which ones are no longer usable.\nIntermediate bounces\nA feature we had a long time ago and which disappeared during a cleanup was the support of intermediate bounces.\nWhen OpenSMTPD fails to deliver a mail it has to notify the sender that the message was never delivered. It sometimes happens immediately, but sometimes the failure may be temporary and the daemon keeps the message and tries to deliver it every now and then (ok, the logic is slightly more complex, but you get the idea). In such cases, the bounce will not be sent before OpenSMTPD gives up on trying after 4 days by default.\nObviously, getting a mail 4 days later to tell you that no one read yours when you assumed it was already in the recipients mailbox for a while is quite irritating. The intermediate bounce will instead notify the sender that an error occured after a few temporarily failed deliveries, and let him know that the daemon will keep trying to deliver for a while.\nAfter discussions, Eric reimplemented intermediate bounces in OpenSMTPD but did it in a slightly different way than with other daemons. By default, an intermediate bounce will be sent after a mail has been sitting in the queue for over 4 hours without being delivered ... but in addition, a set of delays may be provided in smtpd.conf to send multiple intermediate bounces. For example, if I wanted intermediate bounces to be sent after 4 hours, after a day and after two days, I could simple use:\nbounce-warn 4h, 1d, 2d The keyword may change, but the idea and code is here and working.\nTags \u0026 DKIM example\nI've implemented tagging of sessions a very long time ago, I think it was actually already there when OpenSMTPD was not yet OpenSMTPD but still a poolp project :-)\nThe feature was hidden and undocumented, it had uses but so limited that I did not want users to start using it in random situations that I would have to cope with later. Basically, a listener may tag all sessions initiated through it; then rules may apply to specific tags allowing some rules to apply only to some sessions.\nEric realized that this was perfect to deal with one of our use-cases: DKIM signatures.\nWe want DKIM signatures but we don't necessarily want to write a filter for that as there are already tools that do the work. So we need to accept a message, pass it to a DKIM signing tool, which will in turn pass it back to us, so that we can send it where it needs to be sent.\nA tool to do this is DKIMproxy. I won't enter the details behind DKIMproxy, I'll actually post a description of how to setup OpenSMTPD with it very soon on this blog. But the idea is that using tags we can determine which sessions we want forwarded to DKIMproxy, and which sessions are coming back from it and need to be relayed to the final destination:\nlisten on all interfaces that are attached to the default route listen on egress\nlisten on loopback interface on port 10028 and tag DKIM listen on lo0 port 10029 tag DKIM\nonly accept to relay the sessions that are tagged DKIM accept tagged DKIM for any relay\nthis is reached by the sessions that are NOT tagged and will cause OpenSMTPD to relay to the DKIMproxy accept for any relay via smtp://127.0.0.1:10028 Now if I were to send mail to my gmail account, I would connect to the daemon, my session would not be tagged so it would match the last rule causing the message to be send to DKIMproxy. DKIMproxy would then relay back the mail to my loopback interface on port 10029 which would have the new session tagged DKIM causing the first rule to be matched. 4 lines. ridiculous.\nOpenSMTPD goodies\nHere it is, OpenSMTPD goodies, not for sale, limited to friends and a few spare I will give away or sell at low price depending on how many are left.\nIf you were kind enough to contribute and finish the FAQ section of the OpenSMTPD website, I might just send you a mug and a tshirt signed by Charles, Eric and I ;-)\nTime to go zZz, stay tuned for more news !\n","date":"14 December 2012","permalink":"/posts/2012-12-14/opensmtpd-ldap-support-selectable-source-dkim-and-goodies/","section":"Posts","summary":"\u0026lt;p\u0026gt;OHAI,\u0026lt;/p\u0026gt; This week has been crazy, my brain melted a little as I worked on filters, it melted a little as I worked on LDAP and it melted a little more as I tried to trick Eric into working on filters with me (yes, that actually worked :-).","title":"OpenSMTPD: LDAP support, selectable source, DKIM and Goodies"},{"content":"OHAI,\nThis week I had intended to work on filters. After 3 days of pain and swearing, even though it did move forward, I finally decided to step back for a few days and work on something else to preserve my sanity.\nAs you may have guessed, this week has not been too productive\u0026hellip; Oh wait, yes it has ! It\u0026rsquo;s actually been incredibly productive :-)\nJust as usual, I will only write a summary for this week, the complete changelog can be retrieved from our mirror on Github.\nK_SOURCEADDR table service\nOpenSMTPD has been taught how to fetch a source address from a table, but does not make use of it yet. This will, for example, allow users to force a source address for their outgoing mail when they get blacklisted by the monkeys at spamhaus.\nThe K_SOURCEADDR service performs a cyclic lookup returning each address of a table one after the other so that a table holding multiple addresses will cycle through them.\nImproved DNS API\nEric has cleaned and improved the DNS API: dnsquery*() functions now have a more logical ordering of their arguments; struct dns has been killed and we now use two small imsg-specific structures; dns_query_mx_preference() has been introduced to retrieve the preference level of a specific MX for a domain; and finally all MX addresses are looked up in parallel instead of sequentially.\nMTA improvements\nEric has also improved the MTA internal operations so that it uses better abstractions for relay domains, hosts, sources and routes. A MTA session now operates on a given route and reports errors on that route only.\nThe relay tries to open as many routes as possible, within various limits, and drains all mails dispatching them. Oh and it is ready to use the K_SOURCEADDR lookup service but doesn\u0026rsquo;t do it yet, this will probably be part of next week\u0026rsquo;s milestone.\nTable API improvements and new SQLite backend\nThe table API now provides a simple mechanism for backends to support a configuration file without having to deal with the parsing. The backend can be declared in smtpd.conf using:\ntable foobar mybackend:/etc/mail/mybackend.conf\nThen the backend may simply do:\nstatic int table_mybackend_config(struct table *t, const char *configfile) { void *cf;\ncf = table_config_create(); if (! table_config_parse(cf, configfile, T_HASH)) { table_config_destroy(cf); return 0; }\ntable_set_configuration(t, cfg); return 1; }\nTo have the /etc/mail/mybackend.conf file parsed into a key/value table. It can then fetch the values from any other handler using:\nstatic void * table_mybackend_open(struct table *t) { void *cf = table_get_configuration(t);\nif (table_config_get(cf, \u0026ldquo;key\u0026rdquo;) == NULL) { log_warnx(\u0026ldquo;table_mybackend: open: missing key from config file\u0026rdquo;); return NULL; }\nreturn t; }\nSince I needed a use case, I added support for SQLite as a table backend allowing the use of SQLite for any kind of lookup. I had already done it in the past as a proof of concept when we were still using the map API for lookups, but this time it\u0026rsquo;s the real deal.\nSQLite support is achieved using the same approach as that of Postfix where the schema is not imposed but rather the user provides the queries themselves to allow as much flexibiliy as possible.\nTo show you how it can be setup, here\u0026rsquo;s a sample smtpd.conf:\nsmtpd.conf\ntable mytbl sqlite:/etc/mail/sqlite.conf\ni could have another one configured differently\ntable mytbl2 sqlite:/etc/mail/sqlite-other.conf\nand i can have the same one serve different kinds of lookups ;-)\naccept for domain alias deliver to mbox\nand here\u0026rsquo;s the sample sqlite.conf that goes with it\nPath to database\ndbpath /tmp/sqlite.db\nAlias lookup query\nrows \u0026gt;= 0\nfields == 1 (user varchar)\nquery_alias select value from aliases where key=?;\nDomain lookup query\nrows == 1\nfields == 1 (domain varchar)\nquery_domain select value from domains where key=?;\nOf course, you may have multiple smtpd tables using sqlite backends, they may use different configuration files, you can have two aliases databases for two different domains or use the same database table to hold all information for all lookups. It\u0026rsquo;s as flexible as it gets \u0026hellip; ALL lookup services are supported by the SQLite backend so it can be used to store anything used by OpenSMTPD.\nK_USERINFO lookup service\nOpenSMTPD uses the table API for every lookups but there was still one kind of lookups that were performed using a different API: users informations.\nThe table API expects lookups to be done asynchronously but OpenSMTPD had some code that looked up users synchronously, like right before a delivery or to find the home directory of a user for a ~/.forward check.\nI introduced a new lookup service, K_USERINFO, which allows processes to lookup for informations regarding a username, such as its uid, gid and home directory. I then reworked the ~/.forward check and the delivery code to ensure the user information lookup is performed asynchronously through the K_USERINFO service rather than through the user_lookup() API which was synchronous.\nThe only backend to implement K_USERINFO was table_getpwnam which was essentially doing the same as before, just asynchronously, and at that point, user_lookup() bit the dust.\nREALLY VIRTUAL users\nA feature that has been requested for a long time and which was very heard to implement was support for virtual users.\nOpenSMTPD required that the end user be a real system-user that could be looked up using getpwnam(). The K_USERINFO lookup service changed this slightly by having OpenSMTPD require that the end user be a user that could be looked up using table_lookup().\nAnd since we can write lookup services using any backend, I wrote K_USERINFO handlers for table_static, table_db and table_sqlite. I then added a new keyword to smtpd.conf to allow rules to specify a user table:\ntable bleh1 { vuser =\u0026gt; vuser:10💯/tmp/vuser } table bleh2 { vuser =\u0026gt; vuser:20:200:/tmp2/vuser }\naccept for domain poolp.org users deliver to maildir accept for domain opensmtpd.org users deliver to maildir accept for domain pool.ps deliver to maildir\nWith this, OpenSMTPD will accept mail for domain poolp.org but will only find users if they are part of the table bleh1. The domain opensmtpd.org has a different users database, that shares a username but does not share the uid, gid and homedir. In this example, I used static tables, but it could really be sqlite, db or whatever ;-)\nThe domain pool.ps has no users table and defaults to the system database which is what most users will expect.\nRelay URL update and K_CREDENTIALS lookup service\nSince several months, smtpd.conf supports a \u0026ldquo;relay URL\u0026rdquo; format to define relays we want to route via:\ntable creds { mail.poolp.org =\u0026gt; gilles:mypasswd } accept for any relay via tls+auth://mail.poolp.org:31337 auth\nWhen sending mail, the creds table will search for an entry matching the domain name of the relay and find the credentials there. This has annoyed me for a while because it meant that it was not possible to share credentials between multiple relays, it was not possible to have different credentials for two relays operating under the same name, etc, etc \u0026hellip;\nAlso, it annoyed me that outgoing authentication would use K_CREDENTIALS while incoming authentication would not use the table API but the auth_backend API instead.\nI convinced Eric that it would be nice to provide a new mechanism in relay URL so that we could have a label, like tls+auth://label@mail.poolp.org:31337 and it would be used as the key for the credentials lookup.\nThis would allow multiple relays to refer to the same label, or different relays under the same hostname to refer to different labels. It would also allow two nice tricks: first, since the labels are looked up in a different service then we can update the creds table live and MTA will pick up the change; then, if for incoming authentication we assume the username to be the label, then K_CREDENTIALS can be use as THE mechanism to authenticate both in and out sessions and we can kill the auth_backend API.\nSo \u0026hellip; I wrote it and we can now do:\ntable in_auth { gilles =\u0026gt; gilles:encryptedpasswd } table out_auth { bleh =\u0026gt; gilles:cleartextpasswd }\nlisten on all tls auth\naccept for domain poolp.org deliver to maildir accept for any relay via tls+auth://bleh@mail.poolp.org auth\nAnd this closes the last issue with regard to assuming any locality of the users for any purpose. An OpenSMTPD instance no longer assumes users to be local, or to really exists, for any purpose whatsoever.\nNext week should see a long awaited feature, I will not say much and leave it as a surprise. Meanwhile, you guys have a nice week-end, I need a zZzZ ;-)\n","date":"7 December 2012","permalink":"/posts/2012-12-07/opensmtpd-fully-virtual-setups-updated-dns-mta-code-sqlite-support/","section":"Posts","summary":"OHAI,\nThis week I had intended to work on filters. After 3 days of pain and swearing, even though it did move forward, I finally decided to step back for a few days and work on something else to preserve my sanity.","title":"OpenSMTPD: fully virtual setups, updated DNS \u0026 MTA code, SQLite support"},{"content":"OHAI,\nFirst of all, on a completely unrelated note, I\u0026rsquo;d like to emphasize that this blog post is being written from my text editor launched by a little shell and pushed to poolp using the Ocean API. I can be web2.0 from console and emacs, and that\u0026rsquo;s definitely worth a mention ;-)\nAs has become a habit, a lot of work has been poured in OpenSMTPD this week, here is an incomplete summary of the most notable changes. For the complete changelog feel free to check the commit log on our Github mirror.\nA snapshot should be published tomorrow but meanwhile enjoy the reading, some paragraphs have guest-starred eric ;-)\nSimplify parse.y grammar\nI have spent many hours working on a simplification of the parse.y grammar which worked fine for the correct configurations but which could allow very twisted configurations to parse valid.\nWhile fixing an ambiguous case, I realized that the parsing had inherited some complex logic from a feature that was desired a long time ago but which we never used as the rules could become ambiguous. So I began shooting down various parts which were not worth implementing and ended up removing no longer required lists and structures, making the use of tables more coherent accross the daemon as sometimes they were refered by table pointer and sometimes by table id.\nThe end result is a much less bloated grammar, which is semantically more right, and which is much cleaner.\nMore virtual simplification\nBuilding on the foundations from last week and the cleaned up parse.y, I have brought a new feature which was not possible before: using \u0026ldquo;for local\u0026rdquo; or \u0026ldquo;for any\u0026rdquo; as the destination of a virtual domain. Indeed, you could \u0026ldquo;accept for any\u0026rdquo; or \u0026ldquo;accept for local\u0026rdquo; but then OpenSMTPD would assume a primary domain and would perform a system user lookup for the user part. This was actually a side-effet of a parse.y ambiguity where ANY, LOCAL, DOMAIN and VIRTUAL were at the same level and VIRTUAL being considered as a special case.\nSince last week, a virtual domain is simply a regular domain which has defined a virtual mapping. The same logic has been applied to ANY and LOCAL meaning that you can \u0026ldquo;accept for any\u0026rdquo; or \u0026ldquo;accept for any virtual [\u0026hellip;]\u0026rdquo; and it will work as expected, instead of the syntax I demonstrated last week: accept for domain \u0026ldquo;*\u0026rdquo; virtual [\u0026hellip;], which is still valid but will work in a different way internally.\nA user had opened a ticket to ask if we could turn OpenSMTPD into a sink where it would accept mail for any destination and deliver to a single account. With the \u0026ldquo;accept for any virtual [\u0026hellip;]\u0026rdquo; improvement it became simpler as it only required adding a global catch-all that isn\u0026rsquo;t domain aware.\nVirtual now support a global catch-all \u0026ldquo;@\u0026rdquo; which allows a static mapping to handle the catch-alls for multiple domains:\naccept for any virtual { \u0026ldquo;@\u0026rdquo; =\u0026gt; gilles } deliver to maildir\nThe above will have any user of any domain delivered to local account gilles.\nGeneral cleanup\nWe have spent a large amount of time working on a general cleanup of the code base. Amongst other things, we removed some fields from the global struct smtpd to statically isolate them to the specific files that were using them. This helped ensure that we didn\u0026rsquo;t violate API layers.\nThen we spent a great deal of time killing a monster structure, struct submit_status, which was used for all kind of inter-process exchanges. We came up with several lighter structures, carrying only the required information and being tied to a particular process to process exchange. This has required a bit of rework in various processes but the end result is less confusion, code that\u0026rsquo;s easier to maintain and read for new comers.\nMFA rework\nThe MFA process, in charge of filtering the different stages of a SMTP session has been considerably simplified. It no longer knows about SMTP states, which was an API layer violation, and has had a lot of code removed while providing the same service. It shrank by over 100 lines and has become a very small piece of code whose only purpose is to serve as the entry point to the filters evaluation.\nThis was a pre-requisite to the filter work which is already complex enough that we didn\u0026rsquo;t want to add unneeded complexity upfront.\nMTA rework\nA lot of work has been done in the MTA engine which was almost completely rewritten. MTA now knows how to share MX, PTR and credentials for a route. This means that a single DNS request to the lookup daemon can be performed to deal with multiple messages heading to multiple domains.\nIt also deals much better with MX problems: When too many sessions fail to establish a link with a specific MX, the MX is marked as broken and not tried any further by new sessions.\nIt is also capable of dispatching connections on various MX of same priority: When sending many messages, the MTA will spawn sessions against the different MXs with the lowest priority within the default connection limit.\nIf all MXs at a given priority have errors, it moves to the next level to reach backup MXs. If no MX can be reached for a route, a temporary error is triggered.\nI should mention that several kinds of error that used to trigger a permanent error on messages will now simply mark MX on which they occur as broken, giving the mails a chance to be routed through another MX.\nThe logs have been improved too, especially with SSL-related errors. All problems on MXs are now logged (as \u0026ldquo;smtp-out:\u0026rdquo;) to help the administrator diagnose with relaying.\nSMTP rework\nThe SMTP engine has been reworked as a pre-requisite for filters. It is now running on poolp.org and powers the OpenSMTPD mailing list.\nThe idea was to make the code generally simpler to follow and extend.\nFirst, most of the smtp specific structures and defines have been isolated into smtp_session.c, which makes the smtpd.h file a bit less bloated. It was also the opportunity to finally get rid of the horrible submit_status structure.\nThe dispatching of user command is more straightforward, and the imsg dispatching code now makes use of very specific message structures.\nThe rewrite was painful, mostly because the former code was not easy to grasp, and because we slacked too much on the regression suite, which got improved in the process.\nFurthermore, the interaction with the MFA got more complicated with the recent update. Basically we need to de-couple the forwarding of the message data to the MFA from the receveing of filtered data.\nTo sum up, the new code should be a much better ground to implement features like proper filtering and pipelining on the SMTP side.\nimsgproc and filter work\nOpenSMTPD has had filters for over a year now, but disabled as the API is not stable and there were more important stuff to deal with.\nThe design for filters has been discussed on this blog already but as a quick reminder, filters in OpenSMTPD are standalone programs that are linked against a lib we provide to demonize and provide an event-based callback mechanism.\nThe filters can be very very easy and an example of a working filter could be as simple as:\ndefine SMTPD_FILTER\ninclude \u0026ldquo;smtpd-api.h\u0026rdquo;\nvoid mail_cb(uint64_t id, struct filter_mail p, void *arg) { / block idiots */ if (! strcmp(p-\u0026gt;domain, \u0026ldquo;0pointer.net\u0026rdquo;)) { filter_api_reject(id, 530, \u0026ldquo;You\u0026rsquo;re not welcome, go away !\u0026rdquo;); return; }\nfilter_api_accept(id); }\nint main(int argc, char argv[]) { / init the lib and setup daemon and imsg framework */ filter_api_init();\n/* register callbacks */ filter_api_register_mail_callback(mail_cb, NULL);\n/* event loop */ filter_api_loop();\n/* never reached */ return 0; }\nI have spent time in the glue code to change a bit how it worked and have it rely on a new api imsgproc which generalizes the setup imsg / fork / exec operation which we\u0026rsquo;ll end up using for all filters but also possibly for other pluggable backends. It works fine and I\u0026rsquo;ve successfully compiled about a dozen different filters working at different hooks and combinations of hooks.\nThe API is broken at this point due to the late commit of the smtp rework which didn\u0026rsquo;t leave me much time to fix before the week-end but surely it will get better next week.\nMonkey Branch\nYesterday morning I was waiting for Eric to merge a blocking part for my current work so I decided to switch to something radically different.\nSo I tackled a new issue: how do we ensure that our error code path is correct for the errors that you cannot reproduce easily because they are so rare. A typical example of such an error is a getpwnam() failure because a descriptor was not available or because we received an EIO. In such cases, we want OpenSMTPD to correctly handle the error as transient and not reject it permanently which would lead to a lost mail.\nI recalled this Chaos Monkey tool at Netflix that voluntarily produced errors at random to let them ensure their high-availability really works, and I came up with a set of MONKEY_* macros to randomly produce chaos at strategic places. The monkeys will provoke latency in imsg handling and will cause some places to report a temporary failure at random.\nIn the ten minutes that followed my initial testing, they helped spot two very subtle bugs which we would have probably not run into in years. We now know for sure that these code paths are correct and we need to spread more monkeys ;-)\nThis is done in a separate branch which I mirrored on github and which is synched with master. To test, simply checkout the monkey branch (lol no ?) and setup env CFLAGS=-DUSE_MONKEY before building. Then, when sending mail to yourself you should start seeing:\n$ echo test | mail -s \u0026rsquo;test\u0026rsquo; gilles $ echo test | mail -s \u0026rsquo;test\u0026rsquo; gilles send-mail: command failed: 421 Temporary failure $ echo test | mail -s \u0026rsquo;test\u0026rsquo; gilles send-mail: command failed: 421 Temporary failure $ echo test | mail -s \u0026rsquo;test\u0026rsquo; gilles $\nIdeally, OpenSMTPD should NEVER return a 5xx error while running in monkey mode that it doesn\u0026rsquo;t return when not running in monkey mode.\nFun with Google Charts ;-)\nThat has nothing to do with OpenSMTPD itself but rather with me playing with a Google API to generate graphs based on the output of smtpctl show stats:\n\u0026lt;img src=\u0026quot;https://www.poolp.org/~gilles/graph.png\u0026quot;\u0026gt; \u0026lt;img src=\u0026quot;https://www.poolp.org/~gilles/graph2.png\u0026quot;\u0026gt; I will try to spend some time making sure the smtpctl tool can be used to build other tools upon for administrators to take full advantage of the real-time statistics, profiling, scheduler-provided mail queue info, etc\u0026hellip;\nStay tuned, more goodies next week !\n","date":"30 November 2012","permalink":"/posts/2012-11-30/opensmtpd-more-features-more-cleanup-more-more/","section":"Posts","summary":"OHAI,\nFirst of all, on a completely unrelated note, I\u0026rsquo;d like to emphasize that this blog post is being written from my text editor launched by a little shell and pushed to poolp using the Ocean API.","title":"OpenSMTPD: more features, more cleanup, more more"},{"content":"Almost a month since my last post\u0026hellip; I know, I know.\nSo I left my position at Scality two weeks ago. I had been there for a year and a half working on a very interesting and tricky project. I will miss my colleagues and the discussions we had on all kinds of topic, I hope them the best future and that we will be able to share beers again ;-)\nI should also emphasize that they spoiled me for my leave with a ton of gifts including most of the books from my amazon wishlist (Cryptography Engineering, Implementing SSL/TLS, Theory of music) as well as candies, chocolate, rubber ducks see rubber duck debugging, boxers (for when I\u0026rsquo;ll become a homeworker) and, last but not least, a book about the inner details of former president Sarkozy occupation of the Elysee. I suspect the book to be a reminder that if even criminals with disgusting ass-faces manage to succeed in life, I shouldn\u0026rsquo;t worry about new challenges. Thanks.\nObligatory OpenSMTPD section of my blog post ;-)\nThe last two weeks have been spent working very hard on improving OpenSMTPD and we\u0026rsquo;re making progress at an alarming rate. I won\u0026rsquo;t go into the details of each and every improvement we made but here\u0026rsquo;s a quick summary:\nThe logging format has been improved, we have done a lot of rewording and changes to provide the most information using a concise format that can be easily understood by humans and that can easily be parsed or grepped.\nSupport for a monitor command has been added to the smtpctl utility. It allows an administrator to easily monitor a running instance of OpenSMTPD and what it is doing in real-time, displaying states every second.\nFirst shot at a regress suite with a utility that allows the scripting of SMTP sessions scenarios. In a near future, we will be writing various scenarios which will allow us to verify that we don\u0026rsquo;t introduce regressions with new features and bugfixes.\nImproved scheduler API by removing the very annoying and tricky to implement Qwalk API and replacing it with a new queue operation Q_LEARN.\nMailq now supports an \u0026ldquo;online\u0026rdquo; mode which provides more information than the offline mode. The online mode will query the scheduler getting reliable real-time information. Offline mode is equivalent to what we had before.\nImproved format for expansion strings in smtpd.conf and ~/.forward files. We used to support one char formats like %a, %u, and it was both confusing and hard to extend to support new formats while still making sense at first sight. We now support a clearer format: %{rcpt.domain}, %{user.directory}, etc\u0026hellip; The are various supported formats documented in smtpd.conf(5) and each support partial expansion using optional begin and end positive and negative offsets: with user \u0026ldquo;gilles\u0026rdquo;, %{user.username[1:5]} will expand \u0026ldquo;ille\u0026rdquo;.\nAdded a RAM queue_backend, mostly useful for debugging at this point or if you don\u0026rsquo;t care about losing mail when you shutdown the daemon ;-)\nImproved the grammar of our configuration file by removing a lot of stuff that was no longer relevant and changing the syntax to remove all ambiguities. Sure, it breaks existing configuration files but given that none should exceed about 10 lines, it should be straightforward to fix: the poolp.org smtpd.conf which voluntarily exhibits a complex setup (aliases, primary domains, backup MX, virtual domains, relaying, \u0026hellip;) was switched in two minutes.\nReplace the map API with a new table API that provides much simpler and saner semantics. Tables are simpler to declare than maps from a smtpd.conf point of view; they have types so that OpenSMTPD can detect tables used in an inappropriate context at smtpd.conf parse time; and they provide lookup services which allows backends to support only a few kinds of lookups and smtpd to spot that at smtpd.conf parse time too. The new table API simplifies a lot of things, the backends are simpler to write and the end-result is much more reliable.\nWith new table API I could get rid of the user_backend API and replace it with a table lookup service. It is now possible to write table backends to lookup system users instead of relying on getpwnam\u0026hellip; but at the moment OpenSMTPD hardcodes the use of internal table so there\u0026rsquo;s still work to do.\nWe no longer support \u0026ldquo;file\u0026rdquo; as a lookup backend. Whenever a smtpd.conf refers to a file for a lookup, it will internally convert it to a static table which will achieve the exact same result while removing duplicate code. The change is not visible to the users.\nNew dict* API, akin to the tree* API but with char* keys will allow us to simplify a lot of code with regard to how tables are handled and managed by OpenSMTPD.\nTONS of KNF cleanup (several hundreds), removal of old defines, refactors to remove structures that are used as unnecessary indirections, simplifications of equivalent code, etc, etc \u0026hellip;\nVarious bug fixes including one causing some envelopes to possibly be skipped and triggering a start-time crash.\nThat\u0026rsquo;s a SUMMARY, the complete updates can be checked on the commit logs of our github mirror if you\u0026rsquo;re interested. Keep in mind that this is the changelog of the past two weeks only, we have the same kind of changes coming in the next few days and we\u0026rsquo;re definitely not slowing down ;-)\nStay tuned !\n","date":"17 November 2012","permalink":"/posts/2012-11-17/news-from-the-front/","section":"Posts","summary":"Almost a month since my last post\u0026hellip; I know, I know.\nSo I left my position at Scality two weeks ago. I had been there for a year and a half working on a very interesting and tricky project.","title":"News from the front"},{"content":"The crypto backend that has been committed a couple days ago started a little discussion and it was decided to switch the default cipher from Blowfish to AES-128 and make it a default.\nEncryption can now be enabled as simply as:\nqueue encryption key \u0026ldquo;foobar\u0026rdquo;\nThat being said, Charles cleaned up a bit the compression code to make it use FILE* instead of file descriptors. This allows us to ensure we don\u0026rsquo;t have to deal with any buffering and interruption handling code.\nThe fread/fwrite buffer sizes have been increased to make compression and encryption more efficient and have them spend last time in IO.\nFinally, Mathieu (naabed- on #opensmtpd) has played puzzle with some old code of mine to bring LDAP support and provide a map_ldap backend. It\u0026rsquo;s not ready yet, but we imported code in the tree so that it can be worked on more efficiently.\nStay tuned !\n","date":"31 August 2012","permalink":"/posts/2012-08-31/opensmtpd-crypto/compress-fixes-and-import-initial-stab-at-ldap/","section":"Posts","summary":"The crypto backend that has been committed a couple days ago started a little discussion and it was decided to switch the default cipher from Blowfish to AES-128 and make it a default.","title":"OpenSMTPD: crypto/compress fixes and import initial stab at LDAP"},{"content":"A few days ago, Charles committed the compress_backend API which allowed transparent deflation/inflation of envelopes and messages as they hit the queue.\nThe compression code is executed before the queue_backend gets the data so that any queue_backend can benefit transparently from compression without having to add any code to handle it. Due to our design, this also means that if the queue_backend stores envelopes and messages remotely, then the compression will take place before sending and after fetching, the remote end only ever sees compressed data.\nThe idea with the compress_backend was to not only to save disk-space on the queue, which is a very worthy improvement for hosts that deal with very large queues; but also to prove a point: the queue_backends can be made agnostic of the envelopes and messages content, effectively handling blobs.\nAnd a side-effect of that is that since queue_backends don\u0026rsquo;t need to inspect envelopes and messages, and since they can store data remotely, we could provide full queue encryption and ensure that the remote end only ever sees encrypted data.\nThe crypto_backend provides a transparent encryption/decryption service that can be used in any part of OpenSMTPD. As of now, the queue is the only consumer but we already have other use-cases and probably some ideas we haven\u0026rsquo;t come up with yet will pop-up months from now ;-)\nOf course, encryption is tricky so the configuration is hard \u0026hellip; NOT !\nqueue encryption key \u0026ldquo;foobar\u0026rdquo; This will enable transparent encryption of the queue using Blowfish in CBC mode with a random IV and will internally expand the key \u0026ldquo;foobar\u0026rdquo; using sha256. This is just to provide sane defaults, one could override the cipher and digest using the following:\nqueue encryption key \u0026ldquo;foobar\u0026rdquo; cipher aes-128-cbc digest sha512 The IV is randomly chosen for every envelope and message, it is then encrypted and prepended to the message. This ensures that the same key used to encrypt the same envelope will produce different output.\nThings will still be improved further but in my opinion it is already covering the use-cases of many who would need an encrypted queue, and to be honest with a quick look on the intertubez I couldn\u0026rsquo;t find a competitor to compare to on that particular feature so we\u0026rsquo;ll move it forward as we hit use-cases.\nOh\u0026hellip; and of course this does work along the compression support so you can have both encryption and compression enabled.\nThis should hit the OpenBSD tree pretty shortly, this evening for sure.\n","date":"29 August 2012","permalink":"/posts/2012-08-29/opensmtpd-crypto_backend-and-encrypted-queue/","section":"Posts","summary":"A few days ago, Charles committed the compress_backend API which allowed transparent deflation/inflation of envelopes and messages as they hit the queue.\nThe compression code is executed before the queue_backend gets the data so that any queue_backend can benefit transparently from compression without having to add any code to handle it.","title":"OpenSMTPD: crypto_backend and encrypted queue"},{"content":"Howdie,\nEric and I had planned a loooooong time ago to have a hackathon this week-end. However, some asshole piece of shit ran into my car while it was parked, made it unable to take the road and flee without leaving a note. Long story made short, I had to be at the repair shop this week-end to be able to get my car before end of September.\nWe did get our hackathon though and while it was probably much less productive than if we had met in the same room, it was still very very interesting with tons of improvements and many great features written :-p\nAdd support for types and improve increment/decrement in stat_backend I recently committed the stat_backend API which allows us to provide different backends for statistics storage. Until now these stats were necessarily integer counters that supported being incremented, decremented and set.\nI added support for types allowing us to store more precise/accurate statistics in places where integers weren\u0026rsquo;t the best way to represent a value. It is currently possible to store counters, unix timestamp, timeval, timespec, etc \u0026hellip; We can easily extend types as we ran into uses for them.\nWhile at it, Eric asked me if I could allow increments/decrements by values to avoid having to call stat-\u0026gt;increment() or stat-\u0026gt;decrement() multiple times when a batch of increments/decrements are happening.\nDone, committed.\nIntroduce TRACE_PROFILING and profile stats A while ago, Eric has written profiling support which we have used to pinpoint some bottlenecks and fix them pretty successfully. For some reason, it was never committed but the idea was to fetch clock value before and after an imsg callback, allowing us to know precisely the time spent in each events.\nI merged the profiling support to our -current tree since it required some rework and committed it; then I took advantage of the new stats types support to also support pushing profiling information into the stats API.\nWith the ramstat backend it has limited value, but with a persistent storage backend for statistics you can easily graph your events profiling in real-time with about no impact on performances. This doesn\u0026rsquo;t sound too sexy to users but believe me it is. If you run OpenSMTPD on a very busy machine and you think it is not working as efficiently as it should, you can get a LIVE insight at which operations are taking time and should be optimized.\nStart removing user backend We currently have a user_backend API which is supposed to allow us to bring support for virtual users. That API needed to provide getbyname() and getbyuid() handlers to allow users lookups by name AND uid, preventing us from simply using the maps API.\nTurns out that the only use for getbyuid() was in the offline enqueue code path which can safely use getpwuid() as it will necessarily come from a local user. With getbyuid() out of the way, we\u0026rsquo;re almost ready to start removing user_backend API and start providing support for completely virtual users using maps and the supported backends.\nScheduler improvement Eric did quite a bit of work to improve the scheduler, fix some bugs in it and think about some of the improvements we want to bring to it.\nWe still have a disagreement on one algorithm to use internally to boost performances further but it only affects a small part of the scheduler and we will sort that out by experimenting both as it is really nitpicking at this point and only a live test with billions of envelopes in the queue will prove either theory to be right/wrong.\nImproved logging Todd (toddf@) improved IPv6 addresses logging by also printing ports when needed. He also changed the enqueuing logging to display both server and clients exchanges, instead of the old method that only displayed server answers. While there Eric added logging for smtpd pauses and resumes as well as envelopes removals.\nEvents race condition at startup We discovered a race condition that could cause OpenSMTPD to crash at startup if some processes start sending imsg to others before they are ready to service them.\nEric fixed the following bugs by disabling some events and enabling them back when the process is ready to serve. This has been a long-standing issue fixed with a 6 liners diff :-p\nFD exhaustion handling in smtpd and control process Until now, if OpenSMTPD had reached system-imposed limits on descriptors, it would simply fatal. I spent some time making it more reliable by defining a descriptor reserve under which it would stop accepting clients on the network or through smtpctl until the exhaustion situation is resorbed.\nExperimenting showed that it works pretty fine, there is still probably some improvements to do in that area performances-wise but at least it is stable in such situations and sessions run smooth without performances degradation.\nQueue compression Charles committed his queue compression feature which allows OpenSMTPD to deflate/inflate envelopes and messages transparently when they hit or are read from the queue.\nIt worked pretty fine but when I tried to inflate an envelope, I realized that the envelopes and messages didn\u0026rsquo;t use the same compression algorithm. I reworked a bit the compression backend to have it use gzip algorithm consistently.\nThis is awesome, it works out of the box by adding to smtpd.conf:\nqueue compress\nWith this, the size required to store envelopes is halved and a huge gain can be made on messages at the cost of some CPU time \u0026hellip; and from an administration point of view, the queue can be inspected using gzcat instead of cat \u0026hellip; piece of cake !\nQwalk rewrite We received a bug report where envelopes would appear several times in a mailq output.\nEric started investigating the issue and pinpointed the qwalk() API as the culprit. He rewrote it in a saner way and got the issue fixed.\nWhile he was doing it, another OpenBSD hacker sent a different bug report which turned out to be also fixed with the new qwalk(). That\u0026rsquo;s what we call proactive bug fixing :-p\nQueue encryption Finally, one of the features I\u0026rsquo;ve been waiting for a long time to implement: encrypted queue.\nI built upon Charles\u0026rsquo; queue compression diff to provide queue encryption as well. It works in a transparent way and out of the box by simply adding to smtpd.conf:\nqueue encrypt \u0026ldquo;mysecretkey\u0026rdquo;\nWith this, envelopes and messages are encrypted using AES-128 in ECB mode with \u0026ldquo;mysecretkey\u0026rdquo; as a key. I have plans to make the algorithm and mode selectable but I wanted it to work first before I start improving it. This feature is compatible with the compression, so you can enable both and it will compress AND encrypt (in proper order ;-) transparently.\nWhat\u0026rsquo;s even nicer is that our handling of envelopes/messages is fully isolated OUT of queue backends so that if you write a queue backend you never deal with envelopes and message contents. The side effect is that you can write a queue backend to store on a remote untrusted storage \u0026hellip; and people with control of that storage can never inspect content if encryption is enabled :-p\nI didn\u0026rsquo;t commit it yet but it should hit the tree this week.\nStay tuned for more awesomeness !\n","date":"26 August 2012","permalink":"/posts/2012-08-26/opensmtpd-i-have-no-idea-for-that-title-sorry/","section":"Posts","summary":"Howdie,\nEric and I had planned a loooooong time ago to have a hackathon this week-end. However, some asshole piece of shit ran into my car while it was parked, made it unable to take the road and flee without leaving a note.","title":"OpenSMTPD: I have no idea for that title, sorry"},{"content":"It\u0026rsquo;s been over a month since my last post but I have an excuse: I was super busy, then I was super on vacation \u0026hellip; but I\u0026rsquo;m back again ;-)\nSo my life was pretty intense these last few weeks, both professionnally and on the opensource front. I\u0026rsquo;ll explain later, for now let\u0026rsquo;s just focus on the OpenSMTPD goodies.\nOpenSMTPD portable to a new system First of all, I\u0026rsquo;d like to stress out that Rune Lynge and Charles Longeau did an amazing job at porting OpenSMTPD to yet another system \u0026hellip; MacOSX. While I don\u0026rsquo;t know anyone running a mail server on MacOSX, nor do I know anyone who knows anyone doing it, the fact that we can do it fills my heart with joy. We can officially be a free and very expensive mail server at once ;)\nThe compressed/encrypted queue experiment Charles also worked on a proof-of-concept to compress the message and envelopes in the queue. It is not committed yet but there is discussion around it in private as it will very likely hit the tree soon. This is a very important move forward as it also proves that compressing/encrypting the queue does not prevent OpenSMTPD from working which in turn proves that we will be able to store an encrypted queue in an untrusted storage (ie: public cloud provider) and still be able to deliver mail while retaining privacy. Believe me, we\u0026rsquo;re not done on that front and you\u0026rsquo;ll hear about it again in the future.\nScheduler/Queue layer separations On my end, not much done since the hackathon. While there, we had discussed a scheduler and mta refactor with Eric Faurot. The design was not optimal, there was a layer violation with the scheduler taking looks at the queue during initialization, etc \u0026hellip;\nDuring the hackathon I moved the queue initialization to the queue process. I didn\u0026rsquo;t commit it but Eric cloned my repository, improved it further and committed the result a few days ago. I then realized that since the scheduler no longer had to look at the queue, it could be chrooted to /var/empty instead of /var/spool/smtpd to further restrict it in case of a catastrophe.\nTransactional scheduling At the end of the hackathon I also started working on a new scheduler logic, it started with just a few simplifications but then I realized that making the scheduler transactionnal would help us reach the design we had discussed.\nThe scheduler initially consisted of a set of trees and a list sorted by schedule time. An envelope would be part of the trees allowing fast lookups for internal purposes, and the scheduler would just have to look at the first envelope of the list to see if something could be scheduled.\nThis had a side effect that as soon as pushing an envelope in the scheduler, it could be scheduled. That is not always desireable as sometimes we have multiple recipients for a same message / destination and we start scheduling before we get a chance to check that other recipients can be part of the same SMTP transaction. This would lead to inefficient MTA sessions as we would potentially connect multiple times to the same host to deliver the same message for a handful of different recipients.\nThe transactional scheduler uses a set of tree and linear queue BUT when it receives envelopes it doesn\u0026rsquo;t add them to these structures. Instead it adds them to a temporary set of structures, the changeset, which can then be atomically committed to be merged to the scheduling structures, or rollbacked depending on errors we encounter. That ensures that whenever we commit and make envelopes scheduleable, we are able to send as many envelopes as possible in one go to optimize MTA sessions.\nWhen I left the hackathon this was partly done but still required lots of work and improvements which Eric completed by himself and finished just a few days ago. It works perfect, it is a HUGE step forward which will make OpenSMTPD far more efficient.\nNew MTA logic While he was at it, he also reworked entirely the MTA logic. I\u0026rsquo;m not familiar with the new code yet but what I can tell from it is that it\u0026rsquo;s been simplified, cleaned and that it now supports some long awaited features like connection-caching to avoid the overhead of round-trips by reusing existing connections when different messages are heading to the same host.\nActually, it is a bit more complex than that because we no longer deal with hosts but rather with \u0026ldquo;routes\u0026rdquo; which allows reusing the same connection when talking to a MX that is shared by multiple domains (ie: poolp.org / opensmptd.org share mx1.poolp.org, so if we have a connection to mx1.poolp.org already opened, mails for both domains will take advantage of it).\nThere is still work to do in that area but this is also another HUGE step forward on a bit of code that we had planned to rework for months.\nStats backends Statistics used to be stored as size_t fields of fixed structures residing in a chunk of shared mmap-ed memory.\nThis was highly annoying as adding new counters would require changing structures, while the shared memory was in complete opposition with our separated processes design. To make things more annoying even, some of the statistics were used to alter logic when they were supposed to be informative only \u0026hellip; and the statistics were necessarily non-persistent as they were kept in RAM.\nI came up with the stat_backend API to allow writing custom statistic backends and implement the ramstat backend to provide the same feature as before.\nThe main difference lies in the fact that the stats are pushed and never fetched so the logic can never use them; the keys are dynamic so we can generate very precise statistics by creating keys as dynamic strings containing msgid, evpid, etc \u0026hellip; and since they are only pushed, we can centralize them to a single process, have them pushed by imsg and remove the need for shared memory.\nI also wrote a sqlitestat backend which is not committed but that does work and that allows providing persistent statistics that survive accross restarts so that admins can generate graphs and whatnot.\nIdeas float to write stat backends to send to collecting daemons, snmpd, etc \u0026hellip; but the next plan for the stat API is to extend it to provide different type of statistics so we can have ratios, times, etc\u0026hellip;\nBounces grouping OpenSMPTD tries to avoid bounces by rejecting early at session time. Sometimes this is not doable as we accept a recipient and the delivery fails later, after a ~/.forward for example, and we have to generate a bounce for a failed recipient.\nInitially OpenSMTPD grouped all failed recipients coming from same sender as part of the same bounce message so that sender would only receive one report with all failed recipients \u0026hellip; but that feature got broken at some point in time leading to as many bounces as failed recipients.\nIt was conceptually simple to implement but it could not easily be done until queue/scheduler interaction logic was rewritten. Now the queue handles the bounce re-enqueueing. The grouping is simply achieved by delaying the actual bounce a bit in case another bounce for the same message arrives shortly after. Guess who designed and implemented it ? Yup, Eric !\nLoop detection Loop detection was performed by the scheduler, which was a layer violation and which went in the way of Charles\u0026rsquo; attempt at the compression experiment.\nWe got rid of it altogether and reimplemented it at the proper place. I added a Delivered-To loop detection at the MDA level while Eric plugged the Received loop detection at the MTA level.\nBackup MX Oh and since Eric was bored with all these \u0026ldquo;little\u0026rdquo; reworks, he implemented today the also long awaited \u0026ldquo;backup MX\u0026rdquo; features.\nUntil now, OpenSMTPD could not operate like a real backup MX, we could trick it into doing a similar job by using a set of \u0026ldquo;relay via\u0026rdquo; rules like I did on mx2.poolp.org:\naccept for domain \u0026ldquo;poolp.org\u0026rdquo; relay via \u0026ldquo;mx1.poolp.org\u0026rdquo;\nBut this would be a pain to maintain when many backup MX with different priorities are declared in the zone and it would not honour the priorities.\nIt is now possible to do:\naccept for domain \u0026ldquo;poolp.org\u0026rdquo; relay backup \u0026ldquo;mx2.poolp.org\u0026rdquo;\nwhich will declare ourselves as the mx2.poolp.org MX backup for domain poolp.org and allow OpenSMTPD to perform a MX lookup and try to find a MX with a higher priority to hand the mail over. The diff is so small that it\u0026rsquo;s a shame we didn\u0026rsquo;t have this before :-p\nGetting closer to release OpenSMTPD is now in an almost production-ready state, in a better shape than ever, and we\u0026rsquo;re looking forward to receive as many tests and bug reports as possible to ensure that it\u0026rsquo;s rock solid.\nYou can submit your bug reports on our github tracker, feel free to join us on IRC at #OpenSMTPD @ freenode and subscribe to our mailing list misc@opensmtpd.org by sending a mail with subject [misc] subscribe.\nCheers !\n","date":"21 August 2012","permalink":"/posts/2012-08-21/opensmtpd-plenty-of-news/","section":"Posts","summary":"It\u0026rsquo;s been over a month since my last post but I have an excuse: I was super busy, then I was super on vacation \u0026hellip; but I\u0026rsquo;m back again ;-)","title":"OpenSMTPD: plenty of news"},{"content":"Yesterday, I came back from Budapest and was too tired to write anything, but here\u0026rsquo;s a quick summary of all the features that have been implemented with regard to OpenSMTPD by Eric, Charles and I.\nFirst of all, Eric [eric@] has moved ASR, his asynchronous resolver, to OpenBSD\u0026rsquo;s libc. This is a huge step forward that will allow OpenSMTPD to contain less code, while providing a better coverage of the resolver code. I really hope it catches on other systems which can easily get the resolver from OpenBSD and provide saner asynchronous resolving.\nI cleaned up the scheduler code a bit by introducing a new file to separate the scheduler code from the scheduler API. This brings the scheduler code closer to the logic we have for other parts of OpenSMTPD where we support custom backends.\nWhen OpenSMTPD receives a mail for a local address with a \u0026lsquo;+\u0026rsquo; in it, it ignores the second part so that for example \u0026lsquo; gilles+foo@poolp.org\u0026rsquo; is really delivered to \u0026lsquo;gilles\u0026rsquo;. Charles [chl@] added support for Maildir tagging, a very nice feature that had been requested by a user recently and which allows OpenSMTPD to automatically create directories within a Maildir to categorize incoming mails. So now, if you setup OpenSMTPD to deliver to a maildir AND OpenSMTPD receives a mail for a local address with a \u0026lsquo;+\u0026rsquo; in it, it will extract the second part and use it as a subdirectory of the Maildir.\nI finally plugged the relay URLS in parse.y allowing simpler smtpd.conf syntax when using \u0026ldquo;relay via\u0026rdquo; rules, but also preparing the way for upcoming relay maps. I had discussed this on this blog a while ago, but the idea is that you can now express relays using URL like \u0026lsquo;smtp://mx1.poolp.org\u0026rsquo;, \u0026rsquo;tls://mail.poolp.org\u0026rsquo;, etc \u0026hellip; and soon you will be able to provide a map with multiple URL and let OpenSMTPD deliver from either.\nOpenSMTPD initially supported multiple queues and as time passed by, we removed them and came up with a simpler design that was more effective. The code to differentiate between queues was still all over the place and both Eric and Charles worked on getting rid of it. Finally, Charles managed to remove the last pieces and kill the latest bits of it.\nWith Eric, we decided to reduce the number of buckets from 0xfff to 0xff as we observed performances degradations during enqueuing when the queue had thousands of buckets. It was not a matter of envelopes in the buckets but really a degradation caused by the number of entries in the first level directory.\nI simplified the scheduler loop logic to make it much much simpler than before. It cannot be much simpler than now: check next envelope schedule time -\u0026gt; schedule / sleep until schedulable. This may sound too simple for real world, but it is actually because we moved the smartness outside of the scheduling loop and it will deserve a post by itself. I have only committed the scheduler loop simplification for now, the rest is coming soon ;-)\nThere was a problem in the scheduler logic loop which was caused by an optimization I wrote and the fact that our signals are handled asynchronously. This would not be visible most of the time, but if you ever had a very busy queue with tons of messages being schedulable, then the scheduler could not be interrupted with a signal. Sending CTRL-C to OpenSMTPD would kill all processes excepted the scheduler which would keep on as long it had a schedulable envelope before catching the signal. This has been fixed by forcing the scheduler to exit the scheduling loop after each envelope but setting a new scheduling loop call right away. It allows OpenSMTPD to handle the signal and resume its loop.\nI added a TRACE_SCHEDULER trace so that we can easily log scheduler actions using log_trace() and enable/disable them at will without rebuilding. This was prompted by Eric asking me if I still had to keep the ugly output or if I was done with debugging the scheduler ;-)\nCharles added support for literal inet addresses, allowing a user to send mail from or to a user at a specific internet address (gilles@[192.168.1.2], for example). This was requested a while ago, not to hard to accomplish but it was still sitting in our bug tracker.\nHe also restricted the character set allowed in an email address. We\u0026rsquo;re supposed to handle a shitload of characters including [!#$\u0026amp;\u0026rsquo;*/=?^`{|}] as well as any other one inside double quotes. This is so completely fucked up that we decided we don\u0026rsquo;t want to support them and deal with being scared that one could leverage a very smart attack exploiting a user part in a shell filter. If your email address contains one of the forbidden characters, either come up with a good rationale, or use another MTA. For other people, you will probably not notice because no one sane would use \u0026lsquo;#\u0026rsquo; or \u0026lsquo;{\u0026rsquo; in an email address.\nFinally, the hackathon has been much more productive as Eric and I shared the same room and got to discuss some refactors during hours after our hacking sessions at the hackroom. We have designed the new scheduler and MTA refactors which are going to be very very impressive, I can\u0026rsquo;t wait to write about it when it\u0026rsquo;s done [yes I have started working on it, and Eric cloned my branch and worked on it in the plane] ;-)\nOff to sleep, I have to recover now and wake up for my real work in a few hours !\n","date":"15 July 2012","permalink":"/posts/2012-07-15/g2k12-openbsd-hackathon-part-ii/","section":"Posts","summary":"Yesterday, I came back from Budapest and was too tired to write anything, but here\u0026rsquo;s a quick summary of all the features that have been implemented with regard to OpenSMTPD by Eric, Charles and I.","title":"g2k12: OpenBSD hackathon - part II"},{"content":"Yesterday, after nearly missing my plane by 5 minutes, I finally managed to make it to Budapest for g2k12, the OpenBSD general hackathon.\nWhile this is not my first hackathon, this is the first general one with many hackers working on all kinds of subsystems ranging from ports to network and kernel. It\u0026rsquo;s fun to see many people focused on improving different areas which you use daily, sometimes without even noticing, and doing it with as much fun as you are having fixing your own area ;-)\nThere are way too many changes happening to mention them all, and they will probably be part of an Undeadly series of article, so I'll just focus on what Eric, Charles and I did. I apologize in advance for the low redactional quality of this post, but it's late, I'm tired and it's about hotter here than on the sun so I will probably be too lazy to read myself. Anyway, this is just a summary of what happened since yesterday:\nFirst of all, I scratched the Github repository for OpenSMTPD and reimported it using 'git cvsimport' to retain the 4 years of history. Worked like a charm, there are still a few glitches in the workflow but they should be sorted soon.\nIn the past, OpenSMTPD supported multiple queues but it turned out to be a useless features which made the queue API slightly more complex. Eric had started simplifying and removing the code that was superfluous but we still had this queue_kind thing that allowed the logic to determine which queue it was dealing with. Charles got rid of it to simplify our queue code further.\nMeanwhile, Eric (finally) committed his asynchronous resolver in the libc. This allowed him to get immediate feedback from other hackers who confirmed that it was pretty impressive and improved drastically some ports. There are still some shortcomings but this is a very nice move forward and allows to remove the asynchronous resolver from OpenSMTPD to keep the code base small.\nI then plugged the text_to_relayhost() code which allows expressing relays as an URL. This means that: accept for domain foobar.org relay via mx1.poolp.org \\ port 666 tls auth becomes: accept for domain foobar.org relay via \"tls+auth://mx1.poolp.org:666\" It is nicer as when relay maps are finished, one rule will be able to refer to multiple relays that do not necessarily share the same protocol and options. Also, I think that it looks much better and is much easier to remember ;-)\nRecently, Eric and I did a quick benchmark of OpenSMTPD to figure out what were the bottlenecks. The benchmark was quite interesting and I think it deserves a post by itself so I won't go too much into the details. Point is, after a quick discussion yesterday evening I removed the /envelopes/ subdirectory of each message to store envelopes at the same level as the message they are carrying. This saves an extra mkdir() call per message, not much but saves 3 seconds on the enqueuing of 1000 mails. Also, Eric noted that the use of 24 bits for our buckets caused performances to degrade when our queue is full because of the readdir() iterations on a dir entry with 4096 entries. We reduced to 16 bits, which allowed to save a few more seconds. There's still lots of room for improvements but the performances area I will really need to write a full post about, we have fancy graphs and stuff to show how much we understand the bottlenecks and how we can be confident we will be damn fast ;-)\nCharles then adapted portable OpenSMTPD to the removal of ASR and got a build that would link against it externally. It was committed to Github, no snapshot generated yet but we will generate one tomorrow for sure.\nI started simplifying the runner code. The runner is actually a scheduler, it was called \"runner\" because it used to iterate over runqueues which we no longer have. Since it is a scheduler, I renamed it and made sure everything that refered to it would refer to it with the appropraite name. Also, since the scheduler code is tricky we need to be able to log and Eric was annoyed by the very verbose logging. I added support for a \"scheduler\" trace which allows to toggle the very verbose logging using '-T scheduler'. While at it, since the scheduler logic was tricky, I started working on making it much simpler by heavily commenting it and reducing it to about 20 lines of very very simple logic.\nThere's still code to be written for the scheduling because we discussed of a new strategy for scheduling envelopes. It will (in theory) make our lives a lot simpler and allow some very nice feature we had in the pipe for long. I've started but at this point I got too drunk to keep writing that kind of invasive code.\nMost important of all, we spent a lot of time talking, designing and thinking about what will be done this week and in the weeks and months that follow. These discussions are probably worth more in terms of productivity than all of the code we've written these last few hours...\nFinally, as I'm writing this, Charles is sitting at a table with me in the hall of the hotel working on closing one of our \"known issues\"... but I'll leave that for tomorrow ;-)\n","date":"9 July 2012","permalink":"/posts/2012-07-09/g2k12-openbsd-hackathon/","section":"Posts","summary":"Yesterday, after nearly missing my plane by 5 minutes, I finally managed to make it to Budapest for g2k12, the OpenBSD general hackathon.\nWhile this is not my first hackathon, this is the first general one with many hackers working on all kinds of subsystems ranging from ports to network and kernel.","title":"g2k12: OpenBSD hackathon"},{"content":"It\u0026rsquo;s been a while since my last post about OpenSMTPD.\nI don\u0026rsquo;t know what Eric and Charles have been up to, I was quite busy with my daytime job and couldn\u0026rsquo;t really spend much time on the hobby ;-)\nAnyways, one of the most requested feature of OpenSMTPD is to provide users with a mean to report bugs and let them contribute more easily. It\u0026rsquo;s been requested two times this week, it\u0026rsquo;s been requested countless times in the last few weeks and months, and as a matter of fact it\u0026rsquo;s also been a nuisance for us as until now we\u0026rsquo;ve kept track of bugs and feature requests in my mailbox. Bug reports have been lost, others have been forgotten, it\u0026rsquo;s not productive.\nI decided to take a shot at fixing that issue and proposed to my fellow coworkers that we create a Github project, just like what jasper@ did for ports. Charles was already a big fan of Github (\u0026ldquo;I \u0026lt;3 Github\u0026rdquo;) and Eric approved and created an account. So here we are ;-)\nWhy Github ?\nI initially thought about writing a small tool at poolp.org to keep track of bugs and features, and it wouldn\u0026rsquo;t be too hard, but it requires time and that is a very scarce resource these days.\nI\u0026rsquo;ve had very good feedback of Github, it has a nice interface, is easy to use and has the bug tracker, comments and annotations I want. I don\u0026rsquo;t see the point in reinventing the wheel (at this point) so as far as I\u0026rsquo;m concerned it will be just fine.\nWe don\u0026rsquo;t intend to use the repository there as a development repository but instead as a mean to let users know what\u0026rsquo;s in the works by exposing upcoming features.\nSee, OpenSMTPD has several repositories:\nThe OpenBSD repository, which is our primary repository and where OpenBSD users should grab the latest version of OpenSMTPD. It is as close to production-ready as it can get and no other tree (that I know of :-) holds a \u0026ldquo;better\u0026rdquo; version.\nThe portable repository, which is maintained by Charles Longeau at Github, and which is basically a checkout of the OpenBSD tree with the portability goo required to build on other operating systems.\nThen, at poolp.org, we have a private repository where Charles, Eric and I create branches to work on specific new features that can possibly break OpenSMTPD, bring their share of regressions, or that are simply not meant to hit the OpenBSD tree as-is (the so-called \u0026ldquo;experiments\u0026rdquo; :)\nThe repository at poolp.org contains at least 2 branches:\nThe master branch which is synchronized with the OpenBSD tree and from which we derive when we implement new features; the poolp branch which is an experimental branch we use to run code live and test that it is reliable before it is committed to OpenBSD; and eventually several minor branches that come and go.\nThe workflow goes more or less like this:\ncreate a branch X from master for feature X; work on feature X until reasonnably stable; merge feature X in branch poolp; test on mx1.poolp.org that branch poolp is stable; commit feature on OpenBSD; synchronize master branch with OpenBSD; The repository at Github would clone our master branch and eventually other branches, making it visible to the community.\n","date":"19 June 2012","permalink":"/posts/2012-06-19/opensmtpd-development-bug-tracker-github/","section":"Posts","summary":"It\u0026rsquo;s been a while since my last post about OpenSMTPD.\nI don\u0026rsquo;t know what Eric and Charles have been up to, I was quite busy with my daytime job and couldn\u0026rsquo;t really spend much time on the hobby ;-)","title":"OpenSMTPD: development, bug tracker \u0026 GitHub"},{"content":"This is the first post of a series to illustrate and describe a \u0026ldquo;proof of concept\u0026rdquo; code by Charles, Eric and I. I will describe the features as they are implemented.\nSince mid-November 2011, OpenSMTPD offers support for an extensible queue API. The queue_backend API allows a developer to write a custom storage driver by implementing a small set of functions that take care of storing, updating and removing envelopes and messages. The details behind these functions aren\u0026rsquo;t exposed to the daemon which only manipulates message and envelope identifiers.\nWhen the feature was committed, I said that \u0026ldquo;we should know be able to store the queue anywhere\u0026rdquo; to which an OpenBSD hacker joked about being webscale with a MongoDB backend. Little did he know that using a cloud was (one of) the real motivations to that change ;-)\nOpenSMTPD ships with a single queue backend, namely the queue_fsqueue backend, that implements a file-system store using a hashed layout. Technically, we SHOULD be able to use ANY backend as long as it allows key/value storage and linear walk (thought linear walk of a subset is preferable ;-).\nNow, what would we gain from storing the queue in a cloud ?\nSMTP servers tend to store their queue locally. When my primary MX is down, another server, called a \u0026ldquo;backup MX\u0026rdquo; is going to have to accept the mail and store it locally, so that when my primary MX is up again that backup MX sends the mail so my primary MX can store it locally. After what it can finally decide to deliver it, either locally or by relaying to another MX.\nWe actually have a hierarchy of prioritized servers, each accepting mail for a domain and keeping it locally until a server with a higher priority is available again. That sounds safe, but it is actually rather fragile as it relies on EVERY MX doing the proper work to ensure nothing gets lost. Technically, it shouldn\u0026rsquo;t receive much mail, but in practice my secondary MX never has an empty queue and if this is true for a site as small as poolp.org, I\u0026rsquo;d guess secondary MX at bigger sites should also have mails they don\u0026rsquo;t want to lose on their secondary MX.\nTo be really safe, EACH MX server be it primary or secondary should ensure queue redundancy and replication to another machine so that a mail never gets lost if the machine goes down temporarily or permanently. Of course, I don\u0026rsquo;t think I know anyone doing that. Of course, I don\u0026rsquo;t even do that. In place, we rely on statistics that it is unlikely to happen, or that it happens so rarely that it\u0026rsquo;s not worth the trouble.\nNowadays, there are plenty of solutions to store mails on distributed storages. This could be an opensource solution deployed on a set of servers you own, or a service provided by Google, Amazon or even that awesome company called Scality. By sharing a queue that provides high-availability and replication, we can ensure that the burden is no longer on the MX. As far as incoming mail is concerned, ALL MX are equivalent and can be cheap and low end machines with small disks, that don\u0026rsquo;t provide replication and backups. If a mail is accepted by a specific MX, it enters THE SAME QUEUE as the primary MX in charge of delivering the content.\nTechnically this means we can remove the priorities and create a pool of many MX with identical configuration that all accept incoming mail.\nIf we face a peak and need to scale \u0026hellip; we just add clones to the pool !\nNice theory, how long before it is reality ?\nI told Charles and Eric that it would be very nice if we had a REST backend. This would allow people to write small frontends to their storage backend, and let OpenSMTPD communicate with these frontends. With this, we could let OpenSMTPD use any cloud instead of showing a preference to one (even though SCALITY offers the best of all worlds, but that\u0026rsquo;s just a totally biased opinion ;-)).\nSince I couldn\u0026rsquo;t provide access to a scality RING to Eric and I was already familiar with Google\u0026rsquo;s AppEngine API, I installed the SDK and started writing the frontend to their datastore. The beauty of our API is that if it works for ONE, we know it will work for EITHER ONE. I thought it would take a few days to achieve the result I wanted but after about an hour I had a working REST server and had sent Eric a mail with the description of the API. Basically, the goal of the front-end is just to map an URL to a storage operation, so it was really about writing a few dozens lines of code.\nThe next day, Eric told me that he had cloned the API on his machine to ease development and that he had started writing the backend. Yes. He had rewritten a frontend just to ease development because it was much easier than using the remote frontend. That speaks a lot about how easy it is to write one for your own custom storage backends.\nHe asked me to make somes changes to the frontend API that would make things easier and more efficient for him, and after about another hour of doing things right I left him with the latest version of the REST server deployed on Google\u0026rsquo;s cloud.\nWhen I woke up the next day, I had a mail in my mailbox saying that \u0026ldquo;his queue is web-scale\u0026rdquo;.\nEric\u0026rsquo;s backend uses an asynchronous HTTP client he had written and crafts the queries that are expected by the frontend. The code is not in a committable state yet, but it does work and paves the road for many many more interesting features to come.\nWith his backend, OpenSMTPD no longer stores mails locally, it has its queue deported to Google AppEngine and fetches/commits there. I can start another instance of OpenSMTPD with the same configuration on another machine that sees the exact same queue.\nIn one word, it\u0026rsquo;s \u0026hellip; AWESOME ;-)\nWhere do I get the code ?\nYou don\u0026rsquo;t, the PoC is not finished yet, we have three steps missing\u0026hellip; each as funky as this one ;-)\nWe will very likely make the REST backends part of the official distribution as they are independent of the storage solution, however we won\u0026rsquo;t ship the frontends as they are tied to the storage solution (and commercial companies) and it\u0026rsquo;s not acceptable with regard to OpenBSD rules.\nThe frontends will be distributed unofficially, we\u0026rsquo;ll sort out how in time.\nWhat\u0026rsquo;s next ?\nWAIT AND SEE ;-)\nYou can join us on #OpenSMTPD @ irc.freenode.net to discuss and help.\nWe have a mailing list available at misc@opensmtpd.org where we can discuss OpenSMTPD related stuff that is not OpenBSD-specific. To subscribe, just send a mail with subject: [misc] subscribe\nOH, and just for the record, NO I have not been paid by scality to place their product here, I just like what we do there ;-)\n","date":"6 June 2012","permalink":"/posts/2012-06-06/opensmtpd-rest-queue/","section":"Posts","summary":"This is the first post of a series to illustrate and describe a \u0026ldquo;proof of concept\u0026rdquo; code by Charles, Eric and I. I will describe the features as they are implemented.","title":"OpenSMTPD REST queue"},{"content":"First of all, I\u0026rsquo;d like to \u0026ldquo;thank\u0026rdquo; The Spamhaus project which forced me into hacking this feature in a hurry.\nSee, I rent a server at online.net which I use for OpenSMTPD live testing and to run some experimental code before it hits the OpenBSD tree.\nYesterday I received a mail from a user complaining that he couldn\u0026rsquo;t send mail to a friend because we were listed on spamhaus. That was very strange because ALL users at poolp.org are trusted and I have SMTP logs displayed at all times precisely because we run experimental code. There is no way spam would be sent from that box without us noticing.\nAfter investigating, it turns out that this very smart project decided that blacklisting a full /23 range was acceptable to block 2 spammers in a netblock. I thought it was a mistake but it turns out that it\u0026rsquo;s part of their \u0026ldquo;escalation\u0026rdquo; process to break mail systems and have users suffer mail loss when they are in a conflict with a provider.\nAfter a quick exchange with a couple people defending their position, I got bored of dealing with idiots and decided to start working on the src-address map right away.\nSo here it is source address maps:\nuse 192.168.1.1 as source address accept for relay src-address \u0026ldquo;192.168.1.1\u0026rdquo; # and\nmap \u0026ldquo;srcmap\u0026rdquo; source plain \u0026ldquo;/etc/mail/srcmap.txt\u0026rdquo;\nuse an address from the \u0026ldquo;srcmap\u0026rdquo; mapping accept for relay src-address map \u0026ldquo;srcmap\u0026rdquo; # Of course, the srcmap mapping allows dynamic changes so it can be updated at runtime without a daemon restart.\nThis has not been committed to the tree yet, I\u0026rsquo;d like to make some changes to the map API first to support selection policies. Hopefully it can be done by the end of this week-end.\nMeanwhile, I\u0026rsquo;d like to take a few seconds to discourage STRONGLY the use of blacklists managed by irresponsable people that do not provide a FREE method for unlisting (a paying method is called racket in my book). Also, I\u0026rsquo;d like to stress out that using Spamhaus will cause you to lose legitimate mails whenever they feel it is in their interest to pressure a provider.\nIf you are a Spamhaus user and you don\u0026rsquo;t agree with me, I\u0026rsquo;d like to send you a mail to prove my point, but I can\u0026rsquo;t, which proves my point.\n","date":"17 May 2012","permalink":"/posts/2012-05-17/opensmtpd-src-address-maps-and-spamhaus/","section":"Posts","summary":"First of all, I\u0026rsquo;d like to \u0026ldquo;thank\u0026rdquo; The Spamhaus project which forced me into hacking this feature in a hurry.\nSee, I rent a server at online.","title":"OpenSMTPD: src-address maps and spamhaus"},{"content":"When we first started working on OpenSMTPD, we planned for future features but we did not necessarily start integrating them right away as the goal was to bootstrap the project first with basic features.\nAmongst these features was the ability to use mappings for outgoing MX servers. Currently, OpenSMTPD knows two ways of going out:\nUsing the DNS system to find MX records: accept for [\u0026hellip;] relay\nUsing the provided mail exchanger: accept for [\u0026hellip;] relay via \u0026ldquo;my-mx.poolp.org\u0026rdquo; [\u0026hellip;]\nThe first method is straightforward, nothing can really be tweaked about it without tweaking the DNS system itself.\nThe second method is much more flexible as it allows providing a port, a transport method (plaintext ? starttls ? smtps ?), a credentials map for authentication to remote relays, a certificate, a SMTP-level FROM overriding, and possibly more as time passes.\nIn some situations, you can end up with a rule like: accept for all relay via \u0026ldquo;my-mx.poolp.org\u0026rdquo; tls port 25 certificate \u0026ldquo;foobar\u0026rdquo; auth \u0026ldquo;mycredsmap\u0026rdquo; as \u0026ldquo; foo@bar.org\u0026rdquo;\nNot too complicated, but not too nice either. It becomes annoying when we start considering implementation of mappings at the relay level:\nmap \u0026ldquo;relaymap\u0026rdquo; source plain \u0026ldquo;/etc/mail/relaymap\u0026rdquo;\naccept for all relay via map \u0026ldquo;relaymap\u0026rdquo; tls port 25 certificate \u0026ldquo;foobar\u0026rdquo; auth \u0026ldquo;mycredsmap\u0026rdquo; as \u0026ldquo; foo@bar.org\u0026rdquo;\nHow do you specifiy different ports, different ssl options and enable/disable auth on a specific MX ?\nWell you can\u0026rsquo;t because tls, port and auth is tied to the rule and not the MX referenced by the rule\u0026hellip;\nSo I came up with a prototype for a new syntax:\naccept for all relay via \u0026ldquo;[schema://]host[:port]\u0026rdquo;\nWhere schema can be smtp://, smtps://, tls://, ssl://, smtps+auth://, tls+auth:// or ssl+auth://.\nAs usual we default to the sanest behaviour so if you specify (or use the default) smtp://, like:\naccept for all relay via \u0026ldquo;mx.poolp.org\u0026rdquo;\nOpenSMTPD will attempt to use STARTTLS if possible before falling back to a plaintext session; whereas:\naccept for all relay via \u0026ldquo;tls://mx.poolp.org\u0026rdquo;\nOpenSMTPD will make it mandatory to use a STARTTLS session, refusing to deliver the message otherwise.\nSince the MX options are now tied to the MX itself, it becomes possible to store them in a map:\n% cat /etc/mail/mxmap.txt ssl://mx1.poolp.org ssl://mx2.poolp.org ssl://mx3.poolp.org\n% cat /etc/mail/smtpd.conf map \u0026ldquo;mxmap\u0026rdquo; source plain \u0026ldquo;/etc/mail/mxmap.txt\u0026rdquo;\naccept for all relay via map \u0026ldquo;mxmap\u0026rdquo;\nOf course, this will become more interesting when the implementation for relay maps is done and allows looking policies (round-robin, random, ratio, \u0026hellip;) and that you can use them from other backends like SQL or db, as this will allow changing exchangers at runtime, a feature that offers plennnnnnty of possibilities :-)\nAnyways, I have it mostly working on my sandbox, it still needs a couple hours of work I think, I\u0026rsquo;ll get to it by this week-end if time permits.\nStay tuned !\n","date":"14 May 2012","permalink":"/posts/2012-05-14/opensmtpd-relay-maps-new-url-syntax/","section":"Posts","summary":"When we first started working on OpenSMTPD, we planned for future features but we did not necessarily start integrating them right away as the goal was to bootstrap the project first with basic features.","title":"OpenSMTPD relay maps \u0026 new url syntax"},{"content":"Ok, I already posted about the new SQLite map_backend earlier today but it turns out that I\u0026rsquo;ve been very productive, and it was worth another post ;-)\nSo I decided to start improving our maps support and make them usable from places where they weren\u0026rsquo;t.\nprimary domains\nOpenSMTPD supports two kinds of domains: primary domains and virtual domains. The primary domains do not require providing a list of recipients as it will use system accounts; whereas virtual domains require a list of recipients and/or a fallback address to be provided within a map.\nUntil now, primary domains were static, they had to be listed one by rule:\naccept for domain \u0026ldquo;opensmtpd.org\u0026rdquo; deliver to mbox accept for domain \u0026ldquo;poolp.org\u0026rdquo; deliver to mbox accept for domain \u0026ldquo;pool.ps\u0026rdquo; deliver to mbox\nI fixed a few things so that maps can be used and while you can still list one rule for each domain, you can simplify your ruleset by referencing a map:\nmap \u0026ldquo;pdoms\u0026rdquo; source plain \u0026ldquo;/etc/mail/pdoms.txt\u0026rdquo;\naccept for domain map \u0026ldquo;pdoms\u0026rdquo; deliver to mbox\nWith this new configuration, OpenSMTPD does not need to be restarted to add or remove a primary domain, they can be changed within the map directly.\nmap_compare()\nThe map_backend API is very simple and consists in 3 calls: map_open() to open the map, map_lookup() to find a value associated to a key, map_close() to cose the map.\nThe map_lookup() performs an exact match on the key so it can determine if x@foobar.org exists, but it cannot determine if any address ends with that domain. For a particular feature I needed to be able to check if a key was a subset of any key in a map.\nI came up with map_compare() which is a function that will take a map, a key, a map kind and a function to compare the key with each of the keys sequentially. The comparison ends as soon as the function succeeds so that the sequential scan can be aborted early if necessary.\nThe API is very simple but inefficient as it should be reserved for those cases were you HAVE to iterate over relatively small sets. You\u0026rsquo;ll understand in a minute ;-)\nK_NETADDR map kind\nUntil now, OpenSMTPD used inet addresses in only one place: as a parameter to from:\naccept from \u0026ldquo;127.0.0.1\u0026rdquo; [\u0026hellip;]\nThis will change soon so it became necessary to have a map kind to represent inet addresses. I introduced the K_NETADDR map kind which can be used to represent either an address or a netmask for both AF_INET and AF_INET6 (though AF_INET6 netmasks are unsupported on OpenBSD yet [I have a diff]).\nWith this new map kind I decided to go and simplify the parse.y file which was doing much more than it should be doing as you can see.\nNow, not only our parse.y file got much cleaner and easier to read, but the text -\u0026gt; inet conversion has been isolated to util.c, and the ruleset matching has been improved to support \u0026hellip; maps. It is now possible to:\nmap \u0026ldquo;src\u0026rdquo; source plain \u0026ldquo;/etc/mail/trusted\u0026rdquo;\naccept from map \u0026ldquo;src\u0026rdquo; for all relay\nWith /etc/mail/trusted containing a set of addresses and netmasks:\n127.0.0.1 ::1 192.168.1.0/24\nNow the need to iterate over all entries of a map becomes more clear :-)\nI have other goodies but I\u0026rsquo;m exhausted so it\u0026rsquo;ll have to wait\u0026hellip;\nStay tuned !\n","date":"13 May 2012","permalink":"/posts/2012-05-13/opensmtpd-map_compare-and-k_netaddr/","section":"Posts","summary":"Ok, I already posted about the new SQLite map_backend earlier today but it turns out that I\u0026rsquo;ve been very productive, and it was worth another post ;-)","title":"OpenSMTPD, map_compare() and K_NETADDR"},{"content":"During the r2k12 hackathon in Paris, Marc Espie committed SQLite to OpenBSD\u0026rsquo;s base system.\nThis has the side effect that OpenSMTPD can start using it and while we agreed that we did not want it as a strong dependency, the various backends API allow us to make it a soft dependency that can be removed without breaking the daemon if someone really does not want SQLite linked.\nToday I decided to give it a try and implement a SQLite backend to the map API. About ten minutes later (yes, really ten minutes !), I had a working prototype that was suboptimal and that didn\u0026rsquo;t make use of SQL capabilities.\nAn hour later, I have a SQLite backend that will use multiple tables with different structures and that can be used to lookup aliases, virtual domains and credentials for authenticated relaying.\nFirst you create a database with the following schema.sql:\n\u0026ndash; TABLES REQUIRED BY THE MAPS BACKEND\nCREATE TABLE IF NOT EXISTS aliases ( id INTEGER PRIMARY KEY AUTOINCREMENT, name VARCHAR(255) NOT NULL, address VARCHAR(255) NOT NULL );\nCREATE TABLE IF NOT EXISTS secrets ( id INTEGER PRIMARY KEY AUTOINCREMENT, relay VARCHAR(255) UNIQUE NOT NULL, username VARCHAR(255) NOT NULL, password VARCHAR(255) NOT NULL );\nCREATE TABLE IF NOT EXISTS virtual ( id INTEGER PRIMARY KEY AUTOINCREMENT, name VARCHAR(255) NOT NULL, address VARCHAR(255) NOT NULL );\nThen you declare your map with source \u0026ldquo;sqlite\u0026rdquo;:\nmap \u0026ldquo;aliases\u0026rdquo; { source sqlite \u0026ldquo;/etc/mail/sqlite.db\u0026rdquo; } map \u0026ldquo;virtmap\u0026rdquo; { source sqlite \u0026ldquo;/etc/mail/sqlite.db\u0026rdquo; } map \u0026ldquo;secrets\u0026rdquo; { source sqlite \u0026ldquo;/etc/mail/sqlite.db\u0026rdquo; }\naccept for local alias aliases deliver to mbox accept for virtual virtmap deliver to maildir accept for all relay via \u0026ldquo;mail.poolp.org\u0026rdquo; tls auth \u0026ldquo;secrets\u0026rdquo;\nAnd voila ! The lookups are performed at runtime, as usual, which means that you can add virtual domains, aliases or new credentials through SQL queries to the sqlite.db database.\nThe diff will only apply to OpenSMTPD for OpenBSD -current, it will not work as is on -portable but it should be committed pretty soon.\n","date":"12 May 2012","permalink":"/posts/2012-05-12/opensmtpd-meets-sqlite/","section":"Posts","summary":"During the r2k12 hackathon in Paris, Marc Espie committed SQLite to OpenBSD\u0026rsquo;s base system.\nThis has the side effect that OpenSMTPD can start using it and while we agreed that we did not want it as a strong dependency, the various backends API allow us to make it a soft dependency that can be removed without breaking the daemon if someone really does not want SQLite linked.","title":"OpenSMTPD meets SQLite"},{"content":"It\u0026rsquo;s been a whiiiiiiile since my last OpenSMTPD related post. Time to break the slacking, blog-wise.\nUnlike what would appear from the outside, there\u0026rsquo;s been a lot of activity, discussions and planning, I\u0026rsquo;ll address some here and will discuss it further soon.\nFirst of all, chl@ has committed to a private branch a major improvement to the filter API: async DNS lookups. Due to the design of OpenSMTPD, it is not acceptable for a filter to perform a blocking call as it will prevent it from processing other sessions. To solve this, one would have to link the filter against eric@\u0026rsquo;s resolver or fork a process per lookup and add complex code to simulate async behaviour.\nWhen we discussed it, chl@ or I (can\u0026rsquo;t recall who) proposed that queries be rerouted through the OpenSMTPD\u0026rsquo;s LKA process so that it performs the lookup on behalf of the filter. I suggested that filters register a callback upon return of the DNS query and chl@ implemented it. He came up with a PoC working DNSBL filter which proves the solution works and we\u0026rsquo;ll provide a real implementation later.\neric@ committed ASR to OpenBSD\u0026rsquo;s libc, which will allow us to cleanup the code base for OpenSMTPD as it will no longer require shipping an async resolver. He also made various fixes and cleanups here and there.\nSince r2k12, we\u0026rsquo;ve had discussions on design improvements and refactoring of some parts of OpenSMTPD. I\u0026rsquo;ve planned some changes in the scheduler API and a PoC SQLite backend for maps/scheduler/queue as SQLite is now in base and it will allow us to ensure the various backends API are well designed. I also have plans to completely refactor the statistics collecting by providing a backend API, removing the shared mmap-ed page and turning the static counters to a more dynamic structure allowing statistics to be collected at runtime (ie: collect all deliveries for a virtual host, something not doable currently). Also, I have a plan to allow the use of mappings in several contexts where it\u0026rsquo;s not doable yet, like source address and MX relays, which would allow us some very nice features.\neric@ provided me with a doc describing the refactors he plans on MTA after discussions we had, which will allow OpenSMTPD to perform much more efficient and smarter remote deliveries.\nStay tuned, awesome stuff coming soon \u0026hellip;\n","date":"9 May 2012","permalink":"/posts/2012-05-09/some-opensmtpd-news/","section":"Posts","summary":"It\u0026rsquo;s been a whiiiiiiile since my last OpenSMTPD related post. Time to break the slacking, blog-wise.\nUnlike what would appear from the outside, there\u0026rsquo;s been a lot of activity, discussions and planning, I\u0026rsquo;ll address some here and will discuss it further soon.","title":"Some OpenSMTPD news"},{"content":"Je sais pas pourquoi, mais je suis epuise alors que j\u0026rsquo;ai dormi pres de 14h cette nuit. J\u0026rsquo;ai envie de faire une pause, de me prendre une annee de vacances au bord de la plage les doigts de pieds dans le sable fin blanc pendant qu\u0026rsquo;on me sert du sorbet a la noix de coco sur fond de Gojira alors que le soleil me rechauffe le visage mais juste un peu, pas trop, voila, comme ca, merci. L\u0026rsquo;odeur des langoustes grilles me chatouille les narines et, alors que je sirote une pinte de Guinness(c) (la premire d\u0026rsquo;une longue serie ?), j\u0026rsquo;entends le doux murmure des vagues wuuuuuush \u0026hellip;. wuuuuuuush \u0026hellip; couvert par le cri des mouettes t\u0026rsquo;as un syscall a finiiiiir \u0026hellip; t\u0026rsquo;as un syscall a finiiiiiiiiir \u0026hellip;\nEn rouvrant les yeux, un peu triste et du, j\u0026rsquo;entrevois deux lignes \u0026hellip;\n$ cc -c -I/usr/src/sys -D_KERNEL sys_whore.c $ \u0026hellip; qui me ramene la dure realite\u0026hellip; j\u0026rsquo;ai un syscall a terminer.\nMeme dans mes songes, je passe vraiment des vacances de merde \u0026hellip;\n[Ajout] Bon alors voila, j\u0026rsquo;ai perdu plusieurs heures dessus donc je vais donner une astuce qui va peut-etre epargner de nombreuses heures de recherche a mes camarades tech4 qui codent leur syscall pour BSD.\nint sys_whore(struct proc *p, void *v, register_t *retval) { return (0); } Mais a quoi peut bien servir retval alors qu\u0026rsquo;on a un return() dans le code, et comment faire pour faire retourner autre chose qu\u0026rsquo;un entier a notre syscall ?\nApres m\u0026rsquo;etre cogne pres d\u0026rsquo;une dizaine de syscalls sans en voir un seul qui utilisait le retval, j\u0026rsquo;ai demande sur le channel #netbsd ou on m\u0026rsquo;a dit de la merde.\nUn peu plus tard sur #openbsd, on m\u0026rsquo;a conseill la lecture de /usr/src/sys/i386/i386/trap.c et la tout devient evident a la lecture de la fonction syscall().\nEn realite, que se passe-t\u0026rsquo;il lorsqu\u0026rsquo;un programme demande l\u0026rsquo;execution de notre syscall ?\nLes parametres sont places en ordre inverse dans la stack Le numro de syscall est place dans le registre EAX L\u0026rsquo;interruption 0x80 fait passer en mode kernel qui execute\u0026hellip; la fonction syscall() definie dans le trap.c Quand on regarde ce qu\u0026rsquo;elle fait, on comprends mieux. Quelques morceaux choisis:\nregister_t code, args[8], rval[2]; [...] rval[0] = 0; /* valeur de retour de l\u0026#39;appel systeme */ [...] /* appel; on stock le return */ orig_error = error = (*callp-\u0026gt;sy_call)(p, args, rval); [...] switch (error) { [...] default: /* return different de zero (et de cas particuliers) */ bad: if (p-\u0026gt;p_emul-\u0026gt;e_errno) error = p-\u0026gt;p_emul-\u0026gt;e_errno[error]; frame.tf_eax = error; frame.tf_eflags |= PSL_C; /* carry bit */ break; } Explications: Le premier element de rval est la valeur de retour de notre syscall, a ne pas confondre avec le return. En fait, le return dans le code de notre syscall sert uniquement a determiner s\u0026rsquo;il s\u0026rsquo;est bien deroule, il renvoie 0 en cas de succes quoi qu\u0026rsquo;il arrive, et une valeur d\u0026rsquo;errno dans le cas contraire. Mais alors \u0026hellip; c\u0026rsquo;est pour ca que retval n\u0026rsquo;est presque jamais utilise dans les syscalls !\nEn effet, si le syscall rate son return sera un code errno et la fonction wrapper n\u0026rsquo;aura pas besoin de verifier retval puisque la valeur qu\u0026rsquo;elle devra renvoyer sera -1 ou NULL, donc pas besoin d\u0026rsquo;y toucher. De meme, si syscall reussi, comme generalement il est sense renvoyer 0, il n\u0026rsquo;a pas besoin d\u0026rsquo;y toucher non plus puisque retval[0] est initialise a 0 avant le call. Le seul cas ou retval est modifie alors c\u0026rsquo;est quand le syscall renvoie une valeur differente de 0 en cas de success\u0026hellip; la lecture de getpid() et de ses soeurs le confirme\u0026hellip; YOUPI !\nCa vous rappelle rien ? Dans ktrace en tech3 on avait eu le meme probleme, lorsqu\u0026rsquo;un syscall ratait il renvoyait non pas -1 mais une valeur differente de 0 et il fallait ruser avec EAX pour determiner s\u0026rsquo;il s\u0026rsquo;agissait d\u0026rsquo;un ratage ou pas pour afficher nous meme -1 comme etant la valeur de retour.\nUn mystere qui m\u0026rsquo;a occupe une partie de la journe est resolu ;)\n[Ajout #2] Si vous vous demandiez pourquoi retval est un tableau de deux entiers, la reponse m\u0026rsquo;est apparue tout a l\u0026rsquo;heure, je sais pas, un clair de lucidite. Comme son nom l\u0026rsquo;indique, c\u0026rsquo;est un tableau de registre (register_t), et si le premier element correspond au registre EAX (bah oui puisque la valeur de retour est placee dans ce registre), le fait que syscall() associe le registre EDX au second element peut sembler un peu bizarre.\nEt bien nan, en fait c\u0026rsquo;est pour resoudre un cas bien particulier, celui de sys_fork() puisqu\u0026rsquo;il renvoie DEUX valeurs de retour en cas de succes. Le pere recupere la valeur de retour; le pid, dans EAX et le fils recupere la valeur de retour, 0, dans EDX. Magique non ?\n","date":"21 May 2005","permalink":"/posts/2005-05-21/fatigue-et-envie-de-vacances.../","section":"Posts","summary":"Je sais pas pourquoi, mais je suis epuise alors que j\u0026rsquo;ai dormi pres de 14h cette nuit. J\u0026rsquo;ai envie de faire une pause, de me prendre une annee de vacances au bord de la plage les doigts de pieds dans le sable fin blanc pendant qu\u0026rsquo;on me sert du sorbet a la noix de coco sur fond de Gojira alors que le soleil me rechauffe le visage mais juste un peu, pas trop, voila, comme ca, merci.","title":"fatigue et envie de vacances..."},{"content":" Identification # This website is owned and operated by Gilles CHEHADE, registered as an auto-entrepreneur under the SIRET number 515 053 908 00031, located at 30 Boulevard de l’estuaire, 44200 Nantes, France.\nIf you need to contact Gilles CHEHADE, you can send an email to: gilles@poolp.org.\nInsurance # Gilles CHEHADE\u0026rsquo;s activities are covered by an insurance contract with AIG, under the contract number RD01416644U.\nWebsite Hosting # This website is self-managed by Gilles CHEHADE and is hosted by Vultr.com.\nThe physical servers of the site are located in the various data centers of Vultr.com. In case of a complaint regarding the site or its content, Gilles CHEHADE can be contacted via the email mentioned above.\nData Retention Policy # We respect your privacy. Our website does not use cookies or trackers. We do not collect any personal data about our visitors, except for IP addresses, timestamps, and pages visited. This information is collected in accordance with French law and is only used for the purpose of security and analytics. This data is not shared with any third parties and is not used for any other purposes.\nAny changes to this privacy policy will be posted on this page. Please check back frequently to see any updates or changes to our privacy policy.\nThis policy was last updated on July 18, 2023.\n","date":"1 January 0001","permalink":"/legal/","section":"poolp.org","summary":"Identification # This website is owned and operated by Gilles CHEHADE, registered as an auto-entrepreneur under the SIRET number 515 053 908 00031, located at 30 Boulevard de l’estuaire, 44200 Nantes, France.","title":"Legal Notice"}]
\ No newline at end of file
diff --git a/index.xml b/index.xml
index afe2de517..d1cc2596e 100644
--- a/index.xml
+++ b/index.xml
@@ -30,4 +30,4 @@ While this is not my first hackathon, this is the first general one with many ha
 I don&amp;rsquo;t know what Eric and Charles have been up to, I was quite busy with my daytime job and couldn&amp;rsquo;t really spend much time on the hobby ;-)</description></item><item><title>OpenSMTPD REST queue</title><link>https://poolp.org/posts/2012-06-06/opensmtpd-rest-queue/</link><pubDate>Wed, 06 Jun 2012 14:44:10 +0000</pubDate><guid>https://poolp.org/posts/2012-06-06/opensmtpd-rest-queue/</guid><description>This is the first post of a series to illustrate and describe a &amp;ldquo;proof of concept&amp;rdquo; code by Charles, Eric and I. I will describe the features as they are implemented.</description></item><item><title>OpenSMTPD: src-address maps and spamhaus</title><link>https://poolp.org/posts/2012-05-17/opensmtpd-src-address-maps-and-spamhaus/</link><pubDate>Thu, 17 May 2012 17:23:09 +0000</pubDate><guid>https://poolp.org/posts/2012-05-17/opensmtpd-src-address-maps-and-spamhaus/</guid><description>First of all, I&amp;rsquo;d like to &amp;ldquo;thank&amp;rdquo; The Spamhaus project which forced me into hacking this feature in a hurry.
 See, I rent a server at online.</description></item><item><title>OpenSMTPD relay maps &amp; new url syntax</title><link>https://poolp.org/posts/2012-05-14/opensmtpd-relay-maps-new-url-syntax/</link><pubDate>Mon, 14 May 2012 09:55:52 +0000</pubDate><guid>https://poolp.org/posts/2012-05-14/opensmtpd-relay-maps-new-url-syntax/</guid><description>When we first started working on OpenSMTPD, we planned for future features but we did not necessarily start integrating them right away as the goal was to bootstrap the project first with basic features.</description></item><item><title>OpenSMTPD, map_compare() and K_NETADDR</title><link>https://poolp.org/posts/2012-05-13/opensmtpd-map_compare-and-k_netaddr/</link><pubDate>Sun, 13 May 2012 03:09:16 +0000</pubDate><guid>https://poolp.org/posts/2012-05-13/opensmtpd-map_compare-and-k_netaddr/</guid><description>Ok, I already posted about the new SQLite map_backend earlier today but it turns out that I&amp;rsquo;ve been very productive, and it was worth another post ;-)</description></item><item><title>OpenSMTPD meets SQLite</title><link>https://poolp.org/posts/2012-05-12/opensmtpd-meets-sqlite/</link><pubDate>Sat, 12 May 2012 14:23:54 +0000</pubDate><guid>https://poolp.org/posts/2012-05-12/opensmtpd-meets-sqlite/</guid><description>During the r2k12 hackathon in Paris, Marc Espie committed SQLite to OpenBSD&amp;rsquo;s base system.
 This has the side effect that OpenSMTPD can start using it and while we agreed that we did not want it as a strong dependency, the various backends API allow us to make it a soft dependency that can be removed without breaking the daemon if someone really does not want SQLite linked.</description></item><item><title>Some OpenSMTPD news</title><link>https://poolp.org/posts/2012-05-09/some-opensmtpd-news/</link><pubDate>Wed, 09 May 2012 23:03:36 +0000</pubDate><guid>https://poolp.org/posts/2012-05-09/some-opensmtpd-news/</guid><description>It&amp;rsquo;s been a whiiiiiiile since my last OpenSMTPD related post. Time to break the slacking, blog-wise.
-Unlike what would appear from the outside, there&amp;rsquo;s been a lot of activity, discussions and planning, I&amp;rsquo;ll address some here and will discuss it further soon.</description></item><item><title>fatigue et envie de vacances...</title><link>https://poolp.org/posts/2005-05-21/fatigue-et-envie-de-vacances.../</link><pubDate>Sat, 21 May 2005 00:00:00 +0000</pubDate><guid>https://poolp.org/posts/2005-05-21/fatigue-et-envie-de-vacances.../</guid><description>Je sais pas pourquoi, mais je suis epuise alors que j&amp;rsquo;ai dormi pres de 14h cette nuit. J&amp;rsquo;ai envie de faire une pause, de me prendre une annee de vacances au bord de la plage les doigts de pieds dans le sable fin blanc pendant qu&amp;rsquo;on me sert du sorbet a la noix de coco sur fond de Gojira alors que le soleil me rechauffe le visage mais juste un peu, pas trop, voila, comme ca, merci.</description></item></channel></rss>
\ No newline at end of file
+Unlike what would appear from the outside, there&amp;rsquo;s been a lot of activity, discussions and planning, I&amp;rsquo;ll address some here and will discuss it further soon.</description></item><item><title>fatigue et envie de vacances...</title><link>https://poolp.org/posts/2005-05-21/fatigue-et-envie-de-vacances.../</link><pubDate>Sat, 21 May 2005 00:00:00 +0000</pubDate><guid>https://poolp.org/posts/2005-05-21/fatigue-et-envie-de-vacances.../</guid><description>Je sais pas pourquoi, mais je suis epuise alors que j&amp;rsquo;ai dormi pres de 14h cette nuit. J&amp;rsquo;ai envie de faire une pause, de me prendre une annee de vacances au bord de la plage les doigts de pieds dans le sable fin blanc pendant qu&amp;rsquo;on me sert du sorbet a la noix de coco sur fond de Gojira alors que le soleil me rechauffe le visage mais juste un peu, pas trop, voila, comme ca, merci.</description></item><item><title>Legal Notice</title><link>https://poolp.org/legal/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://poolp.org/legal/</guid><description>Identification # This website is owned and operated by Gilles CHEHADE, registered as an auto-entrepreneur under the SIRET number 515 053 908 00031, located at 30 Boulevard de l’estuaire, 44200 Nantes, France.</description></item></channel></rss>
\ No newline at end of file
diff --git a/legal/index.html b/legal/index.html
new file mode 100644
index 000000000..69bd332ce
--- /dev/null
+++ b/legal/index.html
@@ -0,0 +1,18 @@
+<!doctype html><html lang=en dir=ltr class=scroll-smooth data-default-appearance=light data-auto-appearance=true><head><meta charset=utf-8><meta http-equiv=content-language content="en"><meta name=viewport content="width=device-width,initial-scale=1"><meta http-equiv=x-ua-compatible content="ie=edge"><title>Legal Notice &#183; poolp.org</title><meta name=title content="Legal Notice &#183; poolp.org"><meta name=description content="Legal notice"><link rel=canonical href=https://poolp.org/legal/><link type=text/css rel=stylesheet href=/css/main.bundle.min.a541a0fe3bf78817235f39ab69ca97b5b6eed7dd6221f8c6cc58aa90096f4da91424aeb15d985a623c2524fa176998883a5c35b9f73065a7a907d982e43d8efb.css integrity="sha512-pUGg/jv3iBcjXzmracqXtbbu191iIfjGzFiqkAlvTakUJK6xXZhaYjwlJPoXaZiIOlw1ufcwZaepB9mC5D2O+w=="><script type=text/javascript src=/js/appearance.min.516a16745bea5a9bd011138d254cc0fd3973cd55ce6e15f3dec763e7c7c2c7448f8fe7b54cca811cb821b0c7e12cd161caace1dd794ac3d34d40937cbcc9ee12.js integrity="sha512-UWoWdFvqWpvQERONJUzA/TlzzVXObhXz3sdj58fCx0SPj+e1TMqBHLghsMfhLNFhyqzh3XlKw9NNQJN8vMnuEg=="></script>
+<script defer type=text/javascript id=script-bundle src=/js/main.bundle.min.3ddcf04f5b0d4c32f72e726d3c12eebccd8c3c9f9daa9a13414808895de0ae1e1cdddda6849680d5bd96dbb60d0b1e0b24c29f5c83e5631e55e7e6bc02015490.js integrity="sha512-PdzwT1sNTDL3LnJtPBLuvM2MPJ+dqpoTQUgIiV3grh4c3d2mhJaA1b2W27YNCx4LJMKfXIPlYx5V5+a8AgFUkA==" data-copy data-copied></script>
+<script src=/js/zoom.min.js></script>
+<link rel=apple-touch-icon sizes=180x180 href=/apple-touch-icon.png><link rel=icon type=image/png sizes=32x32 href=/favicon-32x32.png><link rel=icon type=image/png sizes=16x16 href=/favicon-16x16.png><link rel=manifest href=/site.webmanifest><meta property="og:title" content="Legal Notice"><meta property="og:description" content="Legal notice"><meta property="og:type" content="article"><meta property="og:url" content="https://poolp.org/legal/"><meta property="article:section" content><meta property="og:site_name" content="poolp.org"><meta name=twitter:card content="summary"><meta name=twitter:title content="Legal Notice"><meta name=twitter:description content="Legal notice"><script type=application/ld+json>[{"@context":"https://schema.org","@type":"Article","articleSection":"poolp.org","name":"Legal Notice","headline":"Legal Notice","description":"Legal notice","abstract":"Identification # This website is owned and operated by Gilles CHEHADE, registered as an auto-entrepreneur under the SIRET number 515 053 908 00031, located at 30 Boulevard de l’estuaire, 44200 Nantes, France.","inLanguage":"en","url":"https:\/\/poolp.org\/legal\/","author":{"@type":"Person","name":""},"mainEntityOfPage":"true","wordCount":"221"}]</script><script src=/lib/jquery/jquery.slim.min.js integrity></script>
+<link rel=stylesheet href=/css/syntax.css><link rel=stylesheet href=/css/custom.css><script>for(var elements=document.querySelectorAll(".discord-dialog__close-button, .discord-dialog"),i=0;i<elements.length;i++)elements[i].addEventListener("click",function(){toggle(".discord-dialog")})</script><style>@import "https://fonts.cdnfonts.com/css/whitney-2";.discord-dialog *,.discord-dialog :after,.discord-dialog :before{box-sizing:inherit}.discord-dialog a{text-decoration:none}@keyframes discord-dialog__transition{0%{opacity:0;transform:translateY(16px)}100%{opacity:1;transform:translateY(0)}}.discord-dialog{-webkit-tap-highlight-color:transparent;box-sizing:border-box;display:none;z-index:2147483647;position:fixed;max-width:288px;right:0;bottom:0;margin:18px;border-radius:10px;font-family:Whitney,sans-serif;background:#2e3136;color:#72767d;box-shadow:0 16px 24px 2px rgba(0,0,0,.14),0 6px 30px 5px rgba(0,0,0,.12),0 8px 10px -5px rgba(0,0,0,.4);transition:opacity .3s cubic-bezier(.25,.8,.25,1),transform .3s cubic-bezier(.25,.8,.25,1);animation:discord-dialog__transition .2s}.discord-dialog--active{display:block}.discord-dialog__container{font-size:14px;line-height:1.2;color:#99aab5;padding:8px 28px 28px}.discord-dialog__logo{display:none;width:auto;margin-top:5px;height:24px}.discord-dialog__logo path{fill:#fff}.discord-dialog__close-button{position:absolute;right:0;padding:16px;cursor:pointer}.discord-dialog__close-button svg{height:20px;width:20px}.discord-dialog__close-button svg path{fill:#ccc}.discord-dialog__close-button:hover{opacity:.8}.discord-dialog__close-button:active{opacity:.4}.discord-dialog__footer{padding:10px 20px}.discord-dialog__guild-icon{border-radius:50%;width:64px;display:block;height:auto;margin:16px 0}.discord-dialog__guild{font-size:24px;line-height:30px;margin-bottom:12px;font-weight:700;color:#fff}.discord-dialog__btn{background:#7289da;color:#fff;text-align:center;display:block;border-radius:1px;margin:8px;font-size:16px;line-height:24px;border-radius:3px;text-align:center;padding:12px 16px;position:relative;top:-6px;box-shadow:0 2px 4px rgba(0,0,0,.3);transition:box-shadow .3s cubic-bezier(.25,.8,.25,1),background .3s cubic-bezier(.25,.8,.25,1)}.discord-dialog__btn:hover{background:#8095de}.discord-dialog__btn:active{background:#677bc4}.discord-dialog__explainer{display:block;margin:2px 8px;color:#7289da;font-size:12px;text-align:right;top:-6px}.discord-dialog__explainer:hover{opacity:.8}.discord-dialog__explainer:active{opacity:.6}.discord-button{-webkit-tap-highlight-color:transparent;box-sizing:border-box;box-shadow:0 1px 3px rgba(0,0,0,.12),0 1px 2px rgba(0,0,0,.24);position:fixed;bottom:0;right:0;z-index:50;margin:23.5px;padding:15.5px;border-radius:50%;background:#7289da;cursor:pointer;transition:box-shadow .3s cubic-bezier(.25,.8,.25,1),background .3s cubic-bezier(.25,.8,.25,1);display:flex;text-align:center;align-items:center;justify-content:center}.discord-button svg{height:28px;width:28px;transition:background .3s cubic-bezier(.25,.8,.25,1)}.discord-button svg path{fill:#fff}.discord-button:hover{box-shadow:0 3px 6px rgba(0,0,0,.16),0 3px 6px rgba(0,0,0,.23)}.discord-button:hover svg{opacity:.6}.discord-button:active{box-shadow:0 10px 20px rgba(0,0,0,.19),0 6px 6px rgba(0,0,0,.23)}.discord-button:active svg{opacity:.4}</style><meta name=theme-color></head><body class="flex flex-col h-screen px-6 m-auto text-lg leading-7 max-w-7xl bg-neutral text-neutral-900 dark:bg-neutral-800 dark:text-neutral sm:px-14 md:px-24 lg:px-32"><div id=the-top class="absolute flex self-center"><a class="px-3 py-1 text-sm -translate-y-8 rounded-b-lg bg-primary-200 focus:translate-y-0 dark:bg-neutral-600" href=#main-content><span class="font-bold text-primary-600 ltr:pr-2 rtl:pl-2 dark:text-primary-400">&darr;</span>Skip to main content</a></div><div class=min-h-[148px]></div><div class="fixed inset-x-0 pl-[24px] pr-[24px]" style=z-index:100><div id=menu-blur class="absolute opacity-0 inset-x-0 top-0 h-full single_hero_background nozoom bg-neutral dark:bg-neutral-800"></div><div class="relative max-w-[64rem] ml-auto mr-auto"><div style=padding-left:0;padding-right:0;padding-top:2px;padding-bottom:3px class="main-menu flex items-center justify-between px-4 py-6 sm:px-6 md:justify-start space-x-3"><div><a href=/ class=flex><span class=sr-only>poolp.org</span>
+<img src=/img/poolp_org.png width=150 height=51 class="logo max-h-[5rem] max-w-[5rem] object-scale-down object-left nozoom" alt=poolp.org></a></div><div class="flex flex-1 items-center justify-between"><nav class="flex space-x-3"></nav><nav class="hidden md:flex items-center space-x-5 md:ml-12 h-12"><a href=https://github.com/poolpOrg/ target=_blank class="flex items-center"><p class="text-base font-medium text-gray-500 hover:text-gray-900" title>Github</p></a><button id=search-button aria-label=Search class="text-base hover:text-primary-600 dark:hover:text-primary-400 h-12" title>
+<span class="relative block icon"><svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="search" class="svg-inline--fa fa-search fa-w-16" role="img" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M505 442.7 405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9.0 208 0S0 93.1.0 208s93.1 208 208 208c48.3.0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9.0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7.0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7.0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></span></button><div class="ltr:mr-14 rtl:ml-14 cursor-pointer text-sm text-neutral-700 hover:text-primary-600 dark:text-neutral dark:hover:text-primary-400"><button id=appearance-switcher aria-label="Dark mode switcher" type=button><div class="flex items-center justify-center h-12 dark:hidden"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M32 256C32 132.2 132.3 32 255.8 32c11.36.0 29.7 1.668 40.9 3.746 9.616 1.777 11.75 14.63 3.279 19.44C245 86.5 211.2 144.6 211.2 207.8c0 109.7 99.71 193 208.3 172.3 9.561-1.805 16.28 9.324 10.11 16.95C387.9 448.6 324.8 480 255.8 480 132.1 480 32 379.6 32 256z"/></svg></span></div><div class="items-center justify-center hidden h-12 dark:flex"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M256 159.1c-53.02.0-95.1 42.98-95.1 95.1s41.2 96.9 95.1 96.9 95.1-42.98 95.1-95.1S309 159.1 256 159.1zM509.3 347l-63.2-91.9 63.15-91.01c6.332-9.125 1.104-21.74-9.826-23.72l-109-19.7-19.7-109c-1.975-10.93-14.59-16.16-23.72-9.824L256 65.89 164.1 2.736c-9.125-6.332-21.74-1.107-23.72 9.824L121.6 121.6 12.56 141.3C1.633 143.2-3.596 155.9 2.736 164.1L65.89 256 2.74 347.01c-6.332 9.125-1.105 21.74 9.824 23.72l109 19.7 19.7 109c1.975 10.93 14.59 16.16 23.72 9.824L256 446.1l91.01 63.15c9.127 6.334 21.75 1.107 23.72-9.822l19.7-109 109-19.7C510.4 368.8 515.6 356.1 509.3 347zM256 383.1c-70.69.0-127.1-57.31-127.1-127.1.0-70.69 57.31-127.1 127.1-127.1S383.1 186.2 383.1 256c0 70.7-56.4 127.1-127.1 127.1z"/></svg></span></div></button></div></nav><div class="flex md:hidden items-center space-x-5 md:ml-12"><span></span>
+<button id=search-button-mobile aria-label=Search class="text-base hover:text-primary-600 dark:hover:text-primary-400" title>
+<span class="relative block icon"><svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="search" class="svg-inline--fa fa-search fa-w-16" role="img" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M505 442.7 405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9.0 208 0S0 93.1.0 208s93.1 208 208 208c48.3.0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9.0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7.0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7.0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></span></button>
+<button id=appearance-switcher-mobile aria-label="Dark mode switcher" type=button style=margin-right:5px><div class="flex items-center justify-center h-12 dark:hidden"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M32 256C32 132.2 132.3 32 255.8 32c11.36.0 29.7 1.668 40.9 3.746 9.616 1.777 11.75 14.63 3.279 19.44C245 86.5 211.2 144.6 211.2 207.8c0 109.7 99.71 193 208.3 172.3 9.561-1.805 16.28 9.324 10.11 16.95C387.9 448.6 324.8 480 255.8 480 132.1 480 32 379.6 32 256z"/></svg></span></div><div class="items-center justify-center hidden h-12 dark:flex"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M256 159.1c-53.02.0-95.1 42.98-95.1 95.1s41.2 96.9 95.1 96.9 95.1-42.98 95.1-95.1S309 159.1 256 159.1zM509.3 347l-63.2-91.9 63.15-91.01c6.332-9.125 1.104-21.74-9.826-23.72l-109-19.7-19.7-109c-1.975-10.93-14.59-16.16-23.72-9.824L256 65.89 164.1 2.736c-9.125-6.332-21.74-1.107-23.72 9.824L121.6 121.6 12.56 141.3C1.633 143.2-3.596 155.9 2.736 164.1L65.89 256 2.74 347.01c-6.332 9.125-1.105 21.74 9.824 23.72l109 19.7 19.7 109c1.975 10.93 14.59 16.16 23.72 9.824L256 446.1l91.01 63.15c9.127 6.334 21.75 1.107 23.72-9.822l19.7-109 109-19.7C510.4 368.8 515.6 356.1 509.3 347zM256 383.1c-70.69.0-127.1-57.31-127.1-127.1.0-70.69 57.31-127.1 127.1-127.1S383.1 186.2 383.1 256c0 70.7-56.4 127.1-127.1 127.1z"/></svg></span></div></button></div></div><div class="-my-2 -mr-2 md:hidden"><label id=menu-button for=menu-controller class=block><input type=checkbox id=menu-controller class=hidden><div class="cursor-pointer hover:text-primary-600 dark:hover:text-primary-400"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path fill="currentcolor" d="M0 96C0 78.33 14.33 64 32 64H416c17.7.0 32 14.33 32 32 0 17.7-14.3 32-32 32H32C14.33 128 0 113.7.0 96zM0 256c0-17.7 14.33-32 32-32H416c17.7.0 32 14.3 32 32s-14.3 32-32 32H32c-17.67.0-32-14.3-32-32zM416 448H32c-17.67.0-32-14.3-32-32s14.33-32 32-32H416c17.7.0 32 14.3 32 32s-14.3 32-32 32z"/></svg></span></div><div id=menu-wrapper style=padding-top:5px class="fixed inset-0 z-30 invisible w-screen h-screen m-auto overflow-auto transition-opacity opacity-0 cursor-default bg-neutral-100/50 backdrop-blur-sm dark:bg-neutral-900/50"><ul class="flex space-y-2 mt-3 flex-col items-end w-full px-6 py-6 mx-auto overflow-visible list-none ltr:text-right rtl:text-left max-w-7xl"><li><span class="cursor-pointer inline-block align-text-bottom hover:text-primary-600 dark:hover:text-primary-400"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 320 512"><path fill="currentcolor" d="M310.6 361.4c12.5 12.5 12.5 32.75.0 45.25C304.4 412.9 296.2 416 288 416s-16.38-3.125-22.62-9.375L160 301.3 54.63 406.6C48.38 412.9 40.19 416 32 416S15.63 412.9 9.375 406.6c-12.5-12.5-12.5-32.75.0-45.25l105.4-105.4L9.375 150.6c-12.5-12.5-12.5-32.75.0-45.25s32.75-12.5 45.25.0L160 210.8l105.4-105.4c12.5-12.5 32.75-12.5 45.25.0s12.5 32.75.0 45.25l-105.4 105.4L310.6 361.4z"/></svg></span></span></li><li class=mt-1><a href=https://github.com/poolpOrg/ target=_blank class="flex items-center"><p class="text-bg font-bg text-gray-500 hover:text-gray-900" title>Github</p></a></li></ul><hr><ul class="flex mt-4 flex-col items-end w-full px-6 py-6 mx-auto overflow-visible list-none ltr:text-right rtl:text-left max-w-7xl"><li class=mb-1><a href=/categories/technology/ class="flex items-center"><p class="text-sm font-sm text-gray-500 hover:text-gray-900" title=technology>Technology</p></a></li><li class=mb-1><a href=/categories/music/ class="flex items-center"><p class="text-sm font-sm text-gray-500 hover:text-gray-900" title=music>Music</p></a></li><li class=mb-1><a href=/categories/personal/ class="flex items-center"><p class="text-sm font-sm text-gray-500 hover:text-gray-900" title=personal>Personal</p></a></li></ul></div></label></div></div><div class="main-menu flex pb-3 flex-col items-end justify-between md:justify-start space-x-3" style=margin-top:-15px><div class="hidden md:flex items-center space-x-5"><a href=/categories/technology/ class="flex items-center"><p class="text-xs font-light text-gray-500 hover:text-gray-900" title=technology>Technology</p></a><a href=/categories/music/ class="flex items-center"><p class="text-xs font-light text-gray-500 hover:text-gray-900" title=music>Music</p></a><a href=/categories/personal/ class="flex items-center"><p class="text-xs font-light text-gray-500 hover:text-gray-900" title=personal>Personal</p></a></div></div></div></div><script>window.addEventListener("scroll",function(){var t=window.pageYOffset||document.documentElement.scrollTop||document.body.scrollTop||0,n=document.getElementById("menu-blur");n.style.opacity=t/300})</script><div class="relative flex flex-col grow"><main id=main-content class=grow><article class=max-w-full><header><ol class="text-sm text-neutral-500 dark:text-neutral-400 print:hidden"><li class="inline hidden"><a class="hover:underline decoration-neutral-300 dark:underline-neutral-600" href=/>poolp.org</a><span class="px-1 text-primary-500">/</span></li><li class="inline hidden"><a class="hover:underline decoration-neutral-300 dark:underline-neutral-600" href=/legal/>Legal Notice</a><span class="px-1 text-primary-500">/</span></li></ol><h1 class="mt-0 text-4xl font-extrabold text-neutral-900 dark:text-neutral">Legal Notice</h1></header><section class="max-w-full mt-6 prose dark:prose-invert"><div id=identification class=anchor></div><h1 class="relative group">Identification
+<span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#identification aria-label=Anchor>#</a></span></h1><p>This website is owned and operated by Gilles CHEHADE, registered as an auto-entrepreneur under the SIRET number 515 053 908 00031, located at 30 Boulevard de l’estuaire, 44200 Nantes, France.</p><p>If you need to contact Gilles CHEHADE, you can send an email to: <a href=mailto:gilles@poolp.org>gilles@poolp.org</a>.</p><div id=insurance class=anchor></div><h1 class="relative group">Insurance
+<span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#insurance aria-label=Anchor>#</a></span></h1><p>Gilles CHEHADE&rsquo;s activities are covered by an insurance contract with AIG, under the contract number RD01416644U.</p><div id=website-hosting class=anchor></div><h1 class="relative group">Website Hosting
+<span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#website-hosting aria-label=Anchor>#</a></span></h1><p>This website is self-managed by Gilles CHEHADE and is hosted by Vultr.com.</p><p>The physical servers of the site are located in the various data centers of Vultr.com. In case of a complaint regarding the site or its content, Gilles CHEHADE can be contacted via the email mentioned above.</p><div id=data-retention-policy class=anchor></div><h1 class="relative group">Data Retention Policy
+<span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#data-retention-policy aria-label=Anchor>#</a></span></h1><p>We respect your privacy. Our website does not use cookies or trackers. We do not collect any personal data about our visitors, except for IP addresses, timestamps, and pages visited. This information is collected in accordance with French law and is only used for the purpose of security and analytics. This data is not shared with any third parties and is not used for any other purposes.</p><p>Any changes to this privacy policy will be posted on this page. Please check back frequently to see any updates or changes to our privacy policy.</p><hr><p>This policy was last updated on July 18, 2023.</p></section><footer class=pt-8><section class="flex flex-row flex-wrap justify-center pt-4 text-xl"><a target=_blank class="m-1 rounded bg-neutral-300 p-1.5 text-neutral-700 hover:bg-primary-500 hover:text-neutral dark:bg-neutral-700 dark:text-neutral-300 dark:hover:bg-primary-400 dark:hover:text-neutral-800" href="https://www.linkedin.com/shareArticle?mini=true&amp;url=https://poolp.org/legal/&amp;title=Legal%20Notice" title="Share on LinkedIn" aria-label="Share on LinkedIn"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path fill="currentcolor" d="M416 32H31.9C14.3 32 0 46.5.0 64.3v383.4C0 465.5 14.3 480 31.9 480H416c17.6.0 32-14.5 32-32.3V64.3c0-17.8-14.4-32.3-32-32.3zM135.4 416H69V202.2h66.5V416zm-33.2-243c-21.3.0-38.5-17.3-38.5-38.5S80.9 96 102.2 96c21.2.0 38.5 17.3 38.5 38.5.0 21.3-17.2 38.5-38.5 38.5zm282.1 243h-66.4V312c0-24.8-.5-56.7-34.5-56.7-34.6.0-39.9 27-39.9 54.9V416h-66.4V202.2h63.7v29.2h.9c8.9-16.8 30.6-34.5 62.9-34.5 67.2.0 79.7 44.3 79.7 101.9V416z"/></svg></span></a><a target=_blank class="m-1 rounded bg-neutral-300 p-1.5 text-neutral-700 hover:bg-primary-500 hover:text-neutral dark:bg-neutral-700 dark:text-neutral-300 dark:hover:bg-primary-400 dark:hover:text-neutral-800" href="https://twitter.com/intent/tweet/?url=https://poolp.org/legal/&amp;text=Legal%20Notice" title="Tweet on Twitter" aria-label="Tweet on Twitter"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M459.37 151.716c.325 4.548.325 9.097.325 13.645.0 138.72-105.583 298.558-298.558 298.558-59.452.0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055.0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421.0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391.0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04.0-57.828 46.782-104.934 104.934-104.934 30.213.0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg></span></a><a target=_blank class="m-1 rounded bg-neutral-300 p-1.5 text-neutral-700 hover:bg-primary-500 hover:text-neutral dark:bg-neutral-700 dark:text-neutral-300 dark:hover:bg-primary-400 dark:hover:text-neutral-800" href="https://reddit.com/submit/?url=https://poolp.org/legal/&amp;resubmit=true&amp;title=Legal%20Notice" title="Submit to Reddit" aria-label="Submit to Reddit"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M201.5 305.5c-13.8.0-24.9-11.1-24.9-24.6.0-13.8 11.1-24.9 24.9-24.9 13.6.0 24.6 11.1 24.6 24.9.0 13.6-11.1 24.6-24.6 24.6zM504 256c0 137-111 248-248 248S8 393 8 256 119 8 256 8s248 111 248 248zm-132.3-41.2c-9.4.0-17.7 3.9-23.8 10-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3 55.4 12.5c0 13.6 11.1 24.6 24.6 24.6 13.8.0 24.9-11.3 24.9-24.9s-11.1-24.9-24.9-24.9c-9.7.0-18 5.8-22.1 13.8l-61.2-13.6c-3-.8-6.1 1.4-6.9 4.4l-19.1 86.4c-33.2 1.4-63.1 11.3-85.5 26.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9.0-46.3 46.9-14.4 62.8-1.1 5-1.7 10.2-1.7 15.5.0 52.6 59.2 95.2 132 95.2 73.1.0 132.3-42.6 132.3-95.2.0-5.3-.6-10.8-1.9-15.8 31.3-16 19.8-62.5-14.9-62.5zM302.8 331c-18.2 18.2-76.1 17.9-93.6.0-2.2-2.2-6.1-2.2-8.3.0-2.5 2.5-2.5 6.4.0 8.6 22.8 22.8 87.3 22.8 110.2.0 2.5-2.2 2.5-6.1.0-8.6-2.2-2.2-6.1-2.2-8.3.0zm7.7-75c-13.6.0-24.6 11.1-24.6 24.9.0 13.6 11.1 24.6 24.6 24.6 13.8.0 24.9-11.1 24.9-24.6.0-13.8-11-24.9-24.9-24.9z"/></svg></span></a><a target=_blank class="m-1 rounded bg-neutral-300 p-1.5 text-neutral-700 hover:bg-primary-500 hover:text-neutral dark:bg-neutral-700 dark:text-neutral-300 dark:hover:bg-primary-400 dark:hover:text-neutral-800" href="https://pinterest.com/pin/create/bookmarklet/?url=https://poolp.org/legal/&amp;description=Legal%20Notice" title="Pin on Pinterest" aria-label="Pin on Pinterest"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 496 512"><path fill="currentcolor" d="M496 256c0 137-111 248-248 248-25.6.0-50.2-3.9-73.4-11.1 10.1-16.5 25.2-43.5 30.8-65 3-11.6 15.4-59 15.4-59 8.1 15.4 31.7 28.5 56.8 28.5 74.8.0 128.7-68.8 128.7-154.3.0-81.9-66.9-143.2-152.9-143.2-107 0-163.9 71.8-163.9 150.1.0 36.4 19.4 81.7 50.3 96.1 4.7 2.2 7.2 1.2 8.3-3.3.8-3.4 5-20.3 6.9-28.1.6-2.5.3-4.7-1.7-7.1-10.1-12.5-18.3-35.3-18.3-56.6.0-54.7 41.4-107.6 112-107.6 60.9.0 103.6 41.5 103.6 100.9.0 67.1-33.9 113.6-78 113.6-24.3.0-42.6-20.1-36.7-44.8 7-29.5 20.5-61.3 20.5-82.6.0-19-10.2-34.9-31.4-34.9-24.9.0-44.9 25.7-44.9 60.2.0 22 7.4 36.8 7.4 36.8s-24.5 103.8-29 123.2c-5 21.4-3 51.6-.9 71.2C65.4 450.9.0 361.1.0 256 0 119 111 8 248 8s248 111 248 248z"/></svg></span></a><a target=_blank class="m-1 rounded bg-neutral-300 p-1.5 text-neutral-700 hover:bg-primary-500 hover:text-neutral dark:bg-neutral-700 dark:text-neutral-300 dark:hover:bg-primary-400 dark:hover:text-neutral-800" href="https://www.facebook.com/sharer/sharer.php?u=https://poolp.org/legal/&amp;quote=Legal%20Notice" title="Share on Facebook" aria-label="Share on Facebook"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M504 256C504 119 393 8 256 8S8 119 8 256c0 123.78 90.69 226.38 209.25 245V327.69h-63V256h63v-54.64c0-62.15 37-96.48 93.67-96.48 27.14.0 55.52 4.84 55.52 4.84v61h-31.28c-30.8.0-40.41 19.12-40.41 38.73V256h68.78l-11 71.69h-57.78V501C413.31 482.38 504 379.78 504 256z"/></svg></span></a><a target=_blank class="m-1 rounded bg-neutral-300 p-1.5 text-neutral-700 hover:bg-primary-500 hover:text-neutral dark:bg-neutral-700 dark:text-neutral-300 dark:hover:bg-primary-400 dark:hover:text-neutral-800" href="mailto:?body=https://poolp.org/legal/&amp;subject=Legal%20Notice" title="Send via email" aria-label="Send via email"><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M207.8 20.73c-93.45 18.32-168.7 93.66-187 187.1-27.64 140.9 68.65 266.2 199.1 285.1 19.01 2.888 36.17-12.26 36.17-31.49l1e-4-.6631c0-15.74-11.44-28.88-26.84-31.24-84.35-12.98-149.2-86.13-149.2-174.2.0-102.9 88.61-185.5 193.4-175.4 91.54 8.869 158.6 91.25 158.6 183.2v16.16c0 22.09-17.94 40.05-40 40.05s-40.01-17.96-40.01-40.05v-120.1c0-8.847-7.161-16.02-16.01-16.02l-31.98.0036c-7.299.0-13.2 4.992-15.12 11.68-24.85-12.15-54.24-16.38-86.06-5.106-38.75 13.73-68.12 48.91-73.72 89.64-9.483 69.01 43.81 128 110.9 128 26.44.0 50.43-9.544 69.59-24.88 24 31.3 65.23 48.69 109.4 37.49C465.2 369.3 496 324.1 495.1 277.2V256.3c0-149.2-133.9-265.632-287.3-235.57zM239.1 304.3c-26.47.0-48-21.56-48-48.05s21.53-48.05 48-48.05 48 21.56 48 48.05-20.6 48.05-48 48.05z"/></svg></span></a><a target=_blank class="m-1 rounded bg-neutral-300 p-1.5 text-neutral-700 hover:bg-primary-500 hover:text-neutral dark:bg-neutral-700 dark:text-neutral-300 dark:hover:bg-primary-400 dark:hover:text-neutral-800" href="https://api.whatsapp.com/send?text=https://poolp.org/legal/&amp;resubmit=true&amp;title=Legal%20Notice" title aria-label><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path fill="currentcolor" d="M380.9 97.1C339 55.1 283.2 32 223.9 32c-122.4.0-222 99.6-222 222 0 39.1 10.2 77.3 29.6 111L0 480l117.7-30.9c32.4 17.7 68.9 27 106.1 27h.1c122.3.0 224.1-99.6 224.1-222 0-59.3-25.2-115-67.1-157zm-157 341.6c-33.2.0-65.7-8.9-94-25.7l-6.7-4-69.8 18.3L72 359.2l-4.4-7c-18.5-29.4-28.2-63.3-28.2-98.2.0-101.7 82.8-184.5 184.6-184.5 49.3.0 95.6 19.2 130.4 54.1 34.8 34.9 56.2 81.2 56.1 130.5.0 101.8-84.9 184.6-186.6 184.6zm101.2-138.2c-5.5-2.8-32.8-16.2-37.9-18-5.1-1.9-8.8-2.8-12.5 2.8s-14.3 18-17.6 21.8c-3.2 3.7-6.5 4.2-12 1.4-32.6-16.3-54-29.1-75.5-66-5.7-9.8 5.7-9.1 16.3-30.3 1.8-3.7.9-6.9-.5-9.7-1.4-2.8-12.5-30.1-17.1-41.2-4.5-10.8-9.1-9.3-12.5-9.5-3.2-.2-6.9-.2-10.6-.2s-9.7 1.4-14.8 6.9c-5.1 5.6-19.4 19-19.4 46.3.0 27.3 19.9 53.7 22.6 57.4 2.8 3.7 39.1 59.7 94.8 83.8 35.2 15.2 49 16.5 66.6 13.9 10.7-1.6 32.8-13.4 37.4-26.4 4.6-13 4.6-24.1 3.2-26.4-1.3-2.5-5-3.9-10.5-6.6z"/></svg></span></a><a target=_blank class="m-1 rounded bg-neutral-300 p-1.5 text-neutral-700 hover:bg-primary-500 hover:text-neutral dark:bg-neutral-700 dark:text-neutral-300 dark:hover:bg-primary-400 dark:hover:text-neutral-800" href="https://t.me/share/url?url=https://poolp.org/legal/&amp;resubmit=true&amp;title=Legal%20Notice" title aria-label><span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 496 512"><path fill="currentcolor" d="M248 8C111.033 8 0 119.033.0 256S111.033 504 248 504 496 392.967 496 256 384.967 8 248 8zM362.952 176.66c-3.732 39.215-19.881 134.378-28.1 178.3-3.476 18.584-10.322 24.816-16.948 25.425-14.4 1.326-25.338-9.517-39.287-18.661-21.827-14.308-34.158-23.215-55.346-37.177-24.485-16.135-8.612-25 5.342-39.5 3.652-3.793 67.107-61.51 68.335-66.746.153-.655.3-3.1-1.154-4.384s-3.59-.849-5.135-.5q-3.283.746-104.608 69.142-14.845 10.194-26.894 9.934c-8.855-.191-25.888-5.006-38.551-9.123-15.531-5.048-27.875-7.717-26.8-16.291q.84-6.7 18.45-13.7 108.446-47.248 144.628-62.3c68.872-28.647 83.183-33.623 92.511-33.789 2.052-.034 6.639.474 9.61 2.885a10.452 10.452.0 013.53 6.716A43.765 43.765.0 01362.952 176.66z"/></svg></span></a></section></footer></article><div id=top-scroller class="pointer-events-none absolute top-[110vh] bottom-0 w-12 ltr:right-0 rtl:left-0"><a href=#the-top class="pointer-events-auto sticky top-[calc(100vh-5.5rem)] flex h-12 w-12 mb-16 items-center justify-center rounded-full bg-neutral/50 text-xl text-neutral-700 backdrop-blur hover:text-primary-600 dark:bg-neutral-800/50 dark:text-neutral dark:hover:text-primary-400" aria-label="Scroll to top" title="Scroll to top">&uarr;</a></div></main><footer id=site-footer class="py-10 print:hidden"><div class="flex items-center justify-between"><p class="text-sm text-neutral-500 dark:text-neutral-400">Copyright © Gilles Chehade - poolp.org
+-
+<u><a href=/legal/>Legal notice</a></u></p><p class="text-xs text-neutral-500 dark:text-neutral-400">Powered by <a class="hover:underline hover:decoration-primary-400 hover:text-primary-500" href=https://gohugo.io/ target=_blank rel="noopener noreferrer">Hugo</a> & <a class="hover:underline hover:decoration-primary-400 hover:text-primary-500" href=https://blowfish.page/ target=_blank rel="noopener noreferrer">Blowfish</a></p></div><script>mediumZoom(document.querySelectorAll("img:not(.nozoom)"),{margin:24,background:"rgba(0,0,0,0.5)",scrollOffset:0})</script><script type=text/javascript src=/js/process.min.35c1113bcc16c5a59bf031082f9e63822aa95280423881a7847a7ff33a16e6299ce6a840d9ef4e10d947e030a18f3f20359afb2ec0f35967484b9a9360ac3145.js integrity="sha512-NcERO8wWxaWb8DEIL55jgiqpUoBCOIGnhHp/8zoW5imc5qhA2e9OENlH4DChjz8gNZr7LsDzWWdIS5qTYKwxRQ=="></script></footer><div id=search-wrapper class="invisible fixed inset-0 flex h-screen w-screen cursor-default flex-col bg-neutral-500/50 p-4 backdrop-blur-sm dark:bg-neutral-900/50 sm:p-6 md:p-[10vh] lg:p-[12vh]" data-url=https://poolp.org style=z-index:500><div id=search-modal class="flex flex-col w-full max-w-3xl min-h-0 mx-auto border rounded-md shadow-lg top-20 border-neutral-200 bg-neutral dark:border-neutral-700 dark:bg-neutral-800"><header class="relative z-10 flex items-center justify-between flex-none px-2"><form class="flex items-center flex-auto min-w-0"><div class="flex items-center justify-center w-8 h-8 text-neutral-400"><span class="relative block icon"><svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="search" class="svg-inline--fa fa-search fa-w-16" role="img" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentcolor" d="M505 442.7 405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9.0 208 0S0 93.1.0 208s93.1 208 208 208c48.3.0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9.0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7.0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7.0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></span></div><input type=search id=search-query class="flex flex-auto h-12 mx-1 bg-transparent appearance-none focus:outline-dotted focus:outline-2 focus:outline-transparent" placeholder=Search tabindex=0></form><button id=close-search-button class="flex items-center justify-center w-8 h-8 text-neutral-700 hover:text-primary-600 dark:text-neutral dark:hover:text-primary-400" title="Close (Esc)">
+<span class="relative block icon"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 320 512"><path fill="currentcolor" d="M310.6 361.4c12.5 12.5 12.5 32.75.0 45.25C304.4 412.9 296.2 416 288 416s-16.38-3.125-22.62-9.375L160 301.3 54.63 406.6C48.38 412.9 40.19 416 32 416S15.63 412.9 9.375 406.6c-12.5-12.5-12.5-32.75.0-45.25l105.4-105.4L9.375 150.6c-12.5-12.5-12.5-32.75.0-45.25s32.75-12.5 45.25.0L160 210.8l105.4-105.4c12.5-12.5 32.75-12.5 45.25.0s12.5 32.75.0 45.25l-105.4 105.4L310.6 361.4z"/></svg></span></button></header><section class="flex-auto px-2 overflow-auto"><ul id=search-results></ul></section></div></div></div></body><script data-name=BMC-Widget data-cfasync=false src=https://cdnjs.buymeacoffee.com/1.0.0/widget.prod.min.js data-id=poolporg data-description="Support me on Buy me a coffee!" data-message="Support me !" data-color=#FFDD00 data-position=left data-x_margin=18 data-y_margin=18></script><div class=discord-button onclick='document.querySelector(".discord-dialog").className="discord-dialog discord-dialog--active"'><svg class="discord-button__icon" viewBox="0 0 24 24"><path d="M22 24l-5.25-5L17.38 21H4.5A2.5 2.5.0 012 18.5V3.5A2.5 2.5.0 014.5 1h15A2.5 2.5.0 0122 3.5V24M12 6.8c-2.68.0-4.56 1.15-4.56 1.15 1.03-.92 2.83-1.45 2.83-1.45l-.17-.17C8.41 6.36 6.88 7.53 6.88 7.53c-1.72 3.59-1.61 6.69-1.61 6.69 1.4 1.81 3.48 1.68 3.48 1.68L9.46 15c-1.25-.27-2.04-1.38-2.04-1.38S9.3 14.9 12 14.9s4.58-1.28 4.58-1.28S15.79 14.73 14.54 15L15.25 15.9S17.33 16.03 18.73 14.22C18.73 14.22 18.84 11.12 17.12 7.53c0 0-1.53-1.17-3.22-1.2l-.17.17s1.8.53 2.83 1.45c0 0-1.88-1.15-4.56-1.15M9.93 10.59c.65.0 1.18.57 1.17 1.27C11.1 12.55 10.58 13.13 9.93 13.13 9.29 13.13 8.77 12.55 8.77 11.86 8.77 11.16 9.28 10.59 9.93 10.59m4.17.0c.65.0 1.17.57 1.17 1.27C15.27 12.55 14.75 13.13 14.1 13.13 13.46 13.13 12.94 12.55 12.94 11.86 12.94 11.16 13.45 10.59 14.1 10.59z"/></svg></div><div class=discord-dialog><div class=discord-dialog__header><a href=https://discordapp.com><svg class="discord-dialog__logo" viewBox="0 0 532 130"><path d="M53.2 20.3H20v37.6l22.1 20V41.4H54c7.5.0 11.2 3.7 11.2 9.5v27.9c0 5.8-3.5 9.7-11.2 9.7H20v21.2h33.2c17.8.1 34.5-8.8 34.5-29.4V50.2c0-20.8-16.7-29.9-34.5-29.9zM227.3 80.4V49.6c0-11.1 19.8-13.7 25.8-2.5l18.3-7.5C264.3 23.7 251.1 19 240.2 19c-17.8.0-35.4 10.4-35.4 30.6v30.8c0 20.3 17.6 30.6 35 30.6 11.2.0 24.6-5.6 32-20.1l-19.6-9.1C247.4 94.2 227.3 91.2 227.3 80.4zM166.7 53.8c-6.9-1.5-11.5-4-11.8-8.3.4-10.4 16.3-10.7 25.6-.8l14.7-11.4C186 22 175.6 19 164.8 19c-16.3.0-32.1 9.2-32.1 26.8.0 17.1 13 26.2 27.3 28.4 7.3 1 15.4 3.9 15.2 9-.6 9.6-20.2 9.1-29.1-1.8L132 94.8c8.3 10.7 19.6 16.2 30.2 16.2 16.3.0 34.4-9.5 35.1-26.8C198.3 62.2 182.5 56.7 166.7 53.8zM99.8 109.7h22.4V20.3H99.8v89.4zM477.5 20.3h-33.2v37.6l22.1 20V41.4h11.8c7.5.0 11.2 3.7 11.2 9.5v27.9c0 5.8-3.5 9.7-11.2 9.7h-34v21.2h33.2c17.8.1 34.5-8.8 34.5-29.4V50.2C512 29.4 495.3 20.3 477.5 20.3zM314.6 19c-18.4.0-36.7 10.1-36.7 30.7v30.6c0 20.5 18.4 30.7 36.9 30.7 18.4.0 36.7-10.2 36.7-30.7V49.7C351.5 29.2 333 19 314.6 19zM329 80.3c0 6.4-7.2 9.7-14.3 9.7-7.2.0-14.4-3.2-14.4-9.7V49.7c0-6.6 7-10.1 14-10.1 7.3.0 14.7 3.2 14.7 10.1V80.3zM431.8 49.7c-.5-21-14.7-29.4-33-29.4h-35.5v89.5H386V81.3h4l20.6 28.4h28L414.4 79C425.2 75.6 431.8 66.3 431.8 49.7zM399.2 61.8H386V41.4h13.2c14.2.0 14.2 20.4.0 20.4z"/></svg></a></div><div class=discord-dialog__close-button onclick='document.querySelector(".discord-dialog").className="discord-dialog"'><svg viewBox="0 0 24 24"><path fill="#000" d="M13.46 12 19 17.54V19H17.54L12 13.46 6.46 19H5V17.54L10.54 12 5 6.46V5H6.46L12 10.54 17.54 5H19V6.46L13.46 12z"/></svg></div><div class=discord-dialog__container><div class=discord-dialog__guild>You're invited to join my Discord server</div>This is a chat server where I hang out, discuss my projects and sometimes screencast as I work on them.<br><br>Feel free to hop in, talk about your own projects, share your thoughts:
+<b>this is a virtual coworking room for anyone to join</b>.</div><div class=discord-dialog__footer><a href=https://discord.gg/YC6j4rbvSk class=discord-dialog__btn>Join the server now</a></div></div></html>
\ No newline at end of file
diff --git a/sitemap.xml b/sitemap.xml
index c3ccbcdde..f82a3f1a1 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>https://poolp.org/posts/2023-07-05/implementing-a-system-call-for-openbsd/</loc><lastmod>2023-07-05T23:14:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/</loc><lastmod>2023-07-05T23:14:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/</loc><lastmod>2023-07-05T23:14:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-07-05/meet-chatgpt-the-new-ai-sidekick-helping-gilles/</loc><lastmod>2023-07-05T12:29:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-07-04/building-my-own-guitar-part-3/</loc><lastmod>2023-07-04T12:31:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-06-30/2023-q2-music-unity-xgb-jujulang-setdb-and-opensmtpd/</loc><lastmod>2023-06-30T18:21:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-06-29/building-my-own-guitar-part-2/</loc><lastmod>2023-06-29T13:22:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-06-28/building-my-own-guitar-part-1/</loc><lastmod>2023-06-28T13:42:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-06-25/a-new-section-appeared/</loc><lastmod>2023-06-25T17:29:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-04-02/march-2023-melodya-mhl-midi-csv-and-more.../</loc><lastmod>2023-04-02T17:29:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-03-01/i-love-you/</loc><lastmod>2023-03-01T12:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-12-30/december-2022-some-more-earmuff-and-go-harmony/</loc><lastmod>2022-12-30T12:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-11-30/november-2022-feedchain-go-harmony-dart-harmony-and-harmonee/</loc><lastmod>2022-11-30T00:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-11-20/feedchain-is-a-standalone-news-feed-project/</loc><lastmod>2022-11-20T12:01:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-10-18/october-2022-blog-comments-a-bit-of-plakar-and-the-streamchain-project/</loc><lastmod>2022-10-18T11:11:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-06-14/june-2022-go-harmony-and-earmuff/</loc><lastmod>2022-06-14T12:27:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-05-22/may-2022-yet-again-tons-of-plakar-stuff/</loc><lastmod>2022-05-22T13:27:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-04-24/april-2022-plakar.io-plakar-refactor-and-ssh-support/</loc><lastmod>2022-04-24T17:24:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-04-01/april-2022-plakar-keys-and-ui-stuff/</loc><lastmod>2022-04-01T01:52:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-03-06/march-2022-plakar-clone-and-plakar-sync/</loc><lastmod>2022-03-06T00:51:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-12-30/farewell-2021-welcome-2022-a-personal-post/</loc><lastmod>2021-12-30T08:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-12-13/december-2021-a-bit-of-plakar-a-bit-of-go-fastcdc-and-some-useless-stuff/</loc><lastmod>2021-12-13T07:40:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-10-26/november-2021-a-bit-of-go-ipcmsg-a-bit-of-go-privsep-and-a-ton-of-plakar/</loc><lastmod>2021-10-26T07:40:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-10-26/october-2021-mostly-plakar/</loc><lastmod>2021-10-26T07:40:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-08-13/august-2021-taking-a-break/</loc><lastmod>2021-08-13T08:53:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-05-02/</loc><lastmod>2021-05-17T22:52:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-05-01/</loc><lastmod>2021-05-12T00:47:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-04-30/april-2021-opensmtpd-plakar-ipcmsg-privsep-and-a-small-hypnosis-talk/</loc><lastmod>2021-04-30T21:12:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-03-26/march-2021-backups-with-plakar/</loc><lastmod>2021-03-26T16:52:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-02-28/february-2021-noosmtpd-libtls-conversion-ciphers-curves-and-protocols/</loc><lastmod>2021-02-28T23:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-01-31/january-2021-opensmtpd-libtls-conversion-and-unix-domain-sockets-support-noosmtpd/</loc><lastmod>2021-01-31T00:18:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-12-29/writing-a-custom-mail-delivery-agent/</loc><lastmod>2020-12-29T01:01:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-12-24/december-2020-opensmtpd-6.8.0p1-released-fixed-several-bugs-proposed-several-diffs-book-is-on-github/</loc><lastmod>2020-12-24T02:38:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-11-22/november-2020-i-wasnt-slacking-but-no-opensource-work/</loc><lastmod>2020-11-22T21:26:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-07-31/july-2020-webmail-custom-mda-and-python-framework-work/</loc><lastmod>2020-07-31T22:47:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-06-27/june-2020-poolp.org-folder-pinning-and-webmail-work/</loc><lastmod>2020-06-27T23:13:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-05-28/may-2020-opensmtpd-6.7.1p1-release-table-procexec-and-many-pocs/</loc><lastmod>2020-05-28T01:57:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-05-01/april-2020-worked-on-a-webmail-and-a-bit-of-opensmtpd-too/</loc><lastmod>2020-05-01T11:37:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-03-30/anxiety-openbsd-break-covid-19-and-resuming-work/</loc><lastmod>2020-03-30T14:01:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-01-30/opensmtpd-advisory-dissected/</loc><lastmod>2020-01-30T23:36:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-01-22/january-2020-opensmtpd-work-libasr-and-libtls/</loc><lastmod>2020-01-22T06:51:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-26/spf-aware-greylisting-et-filter-greylist/</loc><lastmod>2019-12-26T14:15:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-24/december-2019-opensmtpd-and-filters-work-articles-and-goodies/</loc><lastmod>2019-12-24T09:31:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-23/mettre-en-place-un-serveur-de-mail-avec-opensmtpd-dovecot-et-rspamd/</loc><lastmod>2019-12-23T07:57:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-15/decentralisons-smtp-pour-le-bien-commun/</loc><lastmod>2019-12-15T13:05:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-15/decentralised-smtp-is-for-the-greater-good/</loc><lastmod>2019-12-15T07:29:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-01/spf-aware-greylisting-and-filter-greylist/</loc><lastmod>2019-12-01T10:11:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-11-17/november-2019-report-opensmtpd-6.6.1p1-filter-greylist-and-tons-of-portable-cleanup/</loc><lastmod>2019-11-17T22:45:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-10-26/october-2019-report-opensmtpd-6.6.0-release-mostly/</loc><lastmod>2019-10-26T16:13:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-09-21/september-2019-report-jules-opensmtpd-6.6.0-upcoming-release-and-related-things/</loc><lastmod>2019-09-21T07:08:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/</loc><lastmod>2019-09-14T05:37:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-08-30/you-should-not-run-your-mail-server-because-mail-is-hard/</loc><lastmod>2019-08-30T12:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-08-25/august-2019-report-fion-plakar-and-opensmtpd/</loc><lastmod>2019-08-25T04:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-07-27/july-2019-report-tons-of-smtpd-work-mostly/</loc><lastmod>2019-07-27T07:27:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-06-30/june-2019-report-fion-bpg-and-smtpd/</loc><lastmod>2019-06-30T07:42:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-06-02/may-2019-report/</loc><lastmod>2019-06-02T14:57:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-06-02/happy-new-year-2019-a-personal-post/</loc><lastmod>2019-06-02T13:42:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-12-21/opensmtpd-now-supports-regex-in-match-rules/</loc><lastmod>2018-12-21T22:36:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-12-19/more-on-opensmtpd-filters/</loc><lastmod>2018-12-19T18:21:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-12-06/opensmtpd-proc-filters-fc-rdns/</loc><lastmod>2018-12-06T21:31:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-11-09/opensmtpd-reporting-update/</loc><lastmod>2018-11-09T08:27:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-11-03/opensmtpd-released-and-upcoming-filters-preview/</loc><lastmod>2018-11-03T17:27:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-05-21/switching-to-opensmtpd-new-config/</loc><lastmod>2018-05-21T18:32:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-04-30/opensmtpd-new-config/</loc><lastmod>2018-04-30T12:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-01-29/install-openbsd-on-dedibox-with-full-disk-encryption/</loc><lastmod>2018-01-29T23:28:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-01-08/spfwalk/</loc><lastmod>2018-01-08T17:19:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-01-08/news-from-the-front/</loc><lastmod>2018-01-08T12:11:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2016-09-12/opensmtpd-6.0.0-is-released/</loc><lastmod>2016-09-12T08:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2015-12-22/home-sweet-home/</loc><lastmod>2015-12-22T09:53:01+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-12-12/the-state-of-filters/</loc><lastmod>2014-12-12T16:56:24+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-12-10/some-opensmtpd-overview-part-3/</loc><lastmod>2014-12-10T12:38:26+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-12-01/some-opensmtpd-overview-part-2/</loc><lastmod>2014-12-01T09:04:11+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-11-26/some-opensmtpd-overview-part-1/</loc><lastmod>2014-11-26T14:25:12+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-11-24/what-the-fsck-did-just-happen/</loc><lastmod>2014-11-24T18:01:51+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-12-14/news-from-the-front/</loc><lastmod>2013-12-14T21:36:40+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-12-14/opensmtpd-improvements-summary/</loc><lastmod>2013-12-14T21:36:40+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-05-08/installer-openbsd-sur-une-dedibox-classic-gen2/</loc><lastmod>2013-05-08T14:23:43+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-04-26/opensmtpd-table_proc-queue_proc-crypto-queue-and-other-stuff/</loc><lastmod>2013-04-26T18:42:36+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-04-21/news-from-the-front/</loc><lastmod>2013-04-21T17:15:21+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-03-17/opensmtpd-5.3-released/</loc><lastmod>2013-03-17T19:08:55+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-02-02/opensmtpd-minor-fixes--preparing-release/</loc><lastmod>2013-02-02T12:54:36+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-01-25/opensmtpd-gentlemen-fasten-your-seatbelt/</loc><lastmod>2013-01-25T23:01:47+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-01-18/opensmtpd-stress-and-features/</loc><lastmod>2013-01-18T22:11:07+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-01-08/welcome-to-you-2013/</loc><lastmod>2013-01-08T23:16:43+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-12-28/opensmtpd-ssl-relay-stuff/</loc><lastmod>2012-12-28T20:11:23+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-12-14/opensmtpd-ldap-support-selectable-source-dkim-and-goodies/</loc><lastmod>2012-12-14T21:43:44+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-12-07/opensmtpd-fully-virtual-setups-updated-dns-mta-code-sqlite-support/</loc><lastmod>2012-12-07T21:17:57+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-11-30/opensmtpd-more-features-more-cleanup-more-more/</loc><lastmod>2012-11-30T22:13:25+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-11-17/news-from-the-front/</loc><lastmod>2012-11-17T13:28:58+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-08-31/opensmtpd-crypto/compress-fixes-and-import-initial-stab-at-ldap/</loc><lastmod>2012-08-31T15:21:39+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-08-29/opensmtpd-crypto_backend-and-encrypted-queue/</loc><lastmod>2012-08-29T10:21:59+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-08-26/opensmtpd-i-have-no-idea-for-that-title-sorry/</loc><lastmod>2012-08-26T22:53:03+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-08-21/opensmtpd-plenty-of-news/</loc><lastmod>2012-08-21T22:22:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-07-15/g2k12-openbsd-hackathon-part-ii/</loc><lastmod>2012-07-15T21:43:58+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-07-09/g2k12-openbsd-hackathon/</loc><lastmod>2012-07-09T23:52:31+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-06-19/opensmtpd-development-bug-tracker-github/</loc><lastmod>2012-06-19T19:57:37+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-06-06/opensmtpd-rest-queue/</loc><lastmod>2012-06-06T14:44:10+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-17/opensmtpd-src-address-maps-and-spamhaus/</loc><lastmod>2012-05-17T17:23:09+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-14/opensmtpd-relay-maps-new-url-syntax/</loc><lastmod>2012-05-14T09:55:52+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-13/opensmtpd-map_compare-and-k_netaddr/</loc><lastmod>2012-05-13T03:09:16+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-12/opensmtpd-meets-sqlite/</loc><lastmod>2012-05-12T14:23:54+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-09/some-opensmtpd-news/</loc><lastmod>2012-05-09T23:03:36+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2005-05-21/fatigue-et-envie-de-vacances.../</loc><lastmod>2005-05-21T00:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url></urlset>
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>https://poolp.org/posts/2023-07-05/implementing-a-system-call-for-openbsd/</loc><lastmod>2023-07-05T23:14:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/</loc><lastmod>2023-07-05T23:14:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/</loc><lastmod>2023-07-05T23:14:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-07-05/meet-chatgpt-the-new-ai-sidekick-helping-gilles/</loc><lastmod>2023-07-05T12:29:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-07-04/building-my-own-guitar-part-3/</loc><lastmod>2023-07-04T12:31:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-06-30/2023-q2-music-unity-xgb-jujulang-setdb-and-opensmtpd/</loc><lastmod>2023-06-30T18:21:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-06-29/building-my-own-guitar-part-2/</loc><lastmod>2023-06-29T13:22:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-06-28/building-my-own-guitar-part-1/</loc><lastmod>2023-06-28T13:42:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-06-25/a-new-section-appeared/</loc><lastmod>2023-06-25T17:29:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-04-02/march-2023-melodya-mhl-midi-csv-and-more.../</loc><lastmod>2023-04-02T17:29:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2023-03-01/i-love-you/</loc><lastmod>2023-03-01T12:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-12-30/december-2022-some-more-earmuff-and-go-harmony/</loc><lastmod>2022-12-30T12:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-11-30/november-2022-feedchain-go-harmony-dart-harmony-and-harmonee/</loc><lastmod>2022-11-30T00:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-11-20/feedchain-is-a-standalone-news-feed-project/</loc><lastmod>2022-11-20T12:01:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-10-18/october-2022-blog-comments-a-bit-of-plakar-and-the-streamchain-project/</loc><lastmod>2022-10-18T11:11:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-06-14/june-2022-go-harmony-and-earmuff/</loc><lastmod>2022-06-14T12:27:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-05-22/may-2022-yet-again-tons-of-plakar-stuff/</loc><lastmod>2022-05-22T13:27:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-04-24/april-2022-plakar.io-plakar-refactor-and-ssh-support/</loc><lastmod>2022-04-24T17:24:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-04-01/april-2022-plakar-keys-and-ui-stuff/</loc><lastmod>2022-04-01T01:52:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2022-03-06/march-2022-plakar-clone-and-plakar-sync/</loc><lastmod>2022-03-06T00:51:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-12-30/farewell-2021-welcome-2022-a-personal-post/</loc><lastmod>2021-12-30T08:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-12-13/december-2021-a-bit-of-plakar-a-bit-of-go-fastcdc-and-some-useless-stuff/</loc><lastmod>2021-12-13T07:40:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-10-26/november-2021-a-bit-of-go-ipcmsg-a-bit-of-go-privsep-and-a-ton-of-plakar/</loc><lastmod>2021-10-26T07:40:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-10-26/october-2021-mostly-plakar/</loc><lastmod>2021-10-26T07:40:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-08-13/august-2021-taking-a-break/</loc><lastmod>2021-08-13T08:53:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-05-02/</loc><lastmod>2021-05-17T22:52:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-05-01/</loc><lastmod>2021-05-12T00:47:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-04-30/april-2021-opensmtpd-plakar-ipcmsg-privsep-and-a-small-hypnosis-talk/</loc><lastmod>2021-04-30T21:12:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-03-26/march-2021-backups-with-plakar/</loc><lastmod>2021-03-26T16:52:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-02-28/february-2021-noosmtpd-libtls-conversion-ciphers-curves-and-protocols/</loc><lastmod>2021-02-28T23:00:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2021-01-31/january-2021-opensmtpd-libtls-conversion-and-unix-domain-sockets-support-noosmtpd/</loc><lastmod>2021-01-31T00:18:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-12-29/writing-a-custom-mail-delivery-agent/</loc><lastmod>2020-12-29T01:01:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-12-24/december-2020-opensmtpd-6.8.0p1-released-fixed-several-bugs-proposed-several-diffs-book-is-on-github/</loc><lastmod>2020-12-24T02:38:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-11-22/november-2020-i-wasnt-slacking-but-no-opensource-work/</loc><lastmod>2020-11-22T21:26:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-07-31/july-2020-webmail-custom-mda-and-python-framework-work/</loc><lastmod>2020-07-31T22:47:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-06-27/june-2020-poolp.org-folder-pinning-and-webmail-work/</loc><lastmod>2020-06-27T23:13:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-05-28/may-2020-opensmtpd-6.7.1p1-release-table-procexec-and-many-pocs/</loc><lastmod>2020-05-28T01:57:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-05-01/april-2020-worked-on-a-webmail-and-a-bit-of-opensmtpd-too/</loc><lastmod>2020-05-01T11:37:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-03-30/anxiety-openbsd-break-covid-19-and-resuming-work/</loc><lastmod>2020-03-30T14:01:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-01-30/opensmtpd-advisory-dissected/</loc><lastmod>2020-01-30T23:36:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2020-01-22/january-2020-opensmtpd-work-libasr-and-libtls/</loc><lastmod>2020-01-22T06:51:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-26/spf-aware-greylisting-et-filter-greylist/</loc><lastmod>2019-12-26T14:15:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-24/december-2019-opensmtpd-and-filters-work-articles-and-goodies/</loc><lastmod>2019-12-24T09:31:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-23/mettre-en-place-un-serveur-de-mail-avec-opensmtpd-dovecot-et-rspamd/</loc><lastmod>2019-12-23T07:57:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-15/decentralisons-smtp-pour-le-bien-commun/</loc><lastmod>2019-12-15T13:05:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-15/decentralised-smtp-is-for-the-greater-good/</loc><lastmod>2019-12-15T07:29:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-12-01/spf-aware-greylisting-and-filter-greylist/</loc><lastmod>2019-12-01T10:11:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-11-17/november-2019-report-opensmtpd-6.6.1p1-filter-greylist-and-tons-of-portable-cleanup/</loc><lastmod>2019-11-17T22:45:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-10-26/october-2019-report-opensmtpd-6.6.0-release-mostly/</loc><lastmod>2019-10-26T16:13:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-09-21/september-2019-report-jules-opensmtpd-6.6.0-upcoming-release-and-related-things/</loc><lastmod>2019-09-21T07:08:00+02:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/</loc><lastmod>2019-09-14T05:37:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-08-30/you-should-not-run-your-mail-server-because-mail-is-hard/</loc><lastmod>2019-08-30T12:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-08-25/august-2019-report-fion-plakar-and-opensmtpd/</loc><lastmod>2019-08-25T04:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-07-27/july-2019-report-tons-of-smtpd-work-mostly/</loc><lastmod>2019-07-27T07:27:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-06-30/june-2019-report-fion-bpg-and-smtpd/</loc><lastmod>2019-06-30T07:42:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-06-02/may-2019-report/</loc><lastmod>2019-06-02T14:57:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2019-06-02/happy-new-year-2019-a-personal-post/</loc><lastmod>2019-06-02T13:42:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-12-21/opensmtpd-now-supports-regex-in-match-rules/</loc><lastmod>2018-12-21T22:36:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-12-19/more-on-opensmtpd-filters/</loc><lastmod>2018-12-19T18:21:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-12-06/opensmtpd-proc-filters-fc-rdns/</loc><lastmod>2018-12-06T21:31:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-11-09/opensmtpd-reporting-update/</loc><lastmod>2018-11-09T08:27:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-11-03/opensmtpd-released-and-upcoming-filters-preview/</loc><lastmod>2018-11-03T17:27:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-05-21/switching-to-opensmtpd-new-config/</loc><lastmod>2018-05-21T18:32:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-04-30/opensmtpd-new-config/</loc><lastmod>2018-04-30T12:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-01-29/install-openbsd-on-dedibox-with-full-disk-encryption/</loc><lastmod>2018-01-29T23:28:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-01-08/spfwalk/</loc><lastmod>2018-01-08T17:19:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2018-01-08/news-from-the-front/</loc><lastmod>2018-01-08T12:11:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2016-09-12/opensmtpd-6.0.0-is-released/</loc><lastmod>2016-09-12T08:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2015-12-22/home-sweet-home/</loc><lastmod>2015-12-22T09:53:01+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-12-12/the-state-of-filters/</loc><lastmod>2014-12-12T16:56:24+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-12-10/some-opensmtpd-overview-part-3/</loc><lastmod>2014-12-10T12:38:26+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-12-01/some-opensmtpd-overview-part-2/</loc><lastmod>2014-12-01T09:04:11+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-11-26/some-opensmtpd-overview-part-1/</loc><lastmod>2014-11-26T14:25:12+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2014-11-24/what-the-fsck-did-just-happen/</loc><lastmod>2014-11-24T18:01:51+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-12-14/news-from-the-front/</loc><lastmod>2013-12-14T21:36:40+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-12-14/opensmtpd-improvements-summary/</loc><lastmod>2013-12-14T21:36:40+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-05-08/installer-openbsd-sur-une-dedibox-classic-gen2/</loc><lastmod>2013-05-08T14:23:43+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-04-26/opensmtpd-table_proc-queue_proc-crypto-queue-and-other-stuff/</loc><lastmod>2013-04-26T18:42:36+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-04-21/news-from-the-front/</loc><lastmod>2013-04-21T17:15:21+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-03-17/opensmtpd-5.3-released/</loc><lastmod>2013-03-17T19:08:55+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-02-02/opensmtpd-minor-fixes--preparing-release/</loc><lastmod>2013-02-02T12:54:36+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-01-25/opensmtpd-gentlemen-fasten-your-seatbelt/</loc><lastmod>2013-01-25T23:01:47+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-01-18/opensmtpd-stress-and-features/</loc><lastmod>2013-01-18T22:11:07+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2013-01-08/welcome-to-you-2013/</loc><lastmod>2013-01-08T23:16:43+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-12-28/opensmtpd-ssl-relay-stuff/</loc><lastmod>2012-12-28T20:11:23+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-12-14/opensmtpd-ldap-support-selectable-source-dkim-and-goodies/</loc><lastmod>2012-12-14T21:43:44+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-12-07/opensmtpd-fully-virtual-setups-updated-dns-mta-code-sqlite-support/</loc><lastmod>2012-12-07T21:17:57+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-11-30/opensmtpd-more-features-more-cleanup-more-more/</loc><lastmod>2012-11-30T22:13:25+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-11-17/news-from-the-front/</loc><lastmod>2012-11-17T13:28:58+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-08-31/opensmtpd-crypto/compress-fixes-and-import-initial-stab-at-ldap/</loc><lastmod>2012-08-31T15:21:39+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-08-29/opensmtpd-crypto_backend-and-encrypted-queue/</loc><lastmod>2012-08-29T10:21:59+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-08-26/opensmtpd-i-have-no-idea-for-that-title-sorry/</loc><lastmod>2012-08-26T22:53:03+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-08-21/opensmtpd-plenty-of-news/</loc><lastmod>2012-08-21T22:22:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-07-15/g2k12-openbsd-hackathon-part-ii/</loc><lastmod>2012-07-15T21:43:58+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-07-09/g2k12-openbsd-hackathon/</loc><lastmod>2012-07-09T23:52:31+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-06-19/opensmtpd-development-bug-tracker-github/</loc><lastmod>2012-06-19T19:57:37+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-06-06/opensmtpd-rest-queue/</loc><lastmod>2012-06-06T14:44:10+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-17/opensmtpd-src-address-maps-and-spamhaus/</loc><lastmod>2012-05-17T17:23:09+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-14/opensmtpd-relay-maps-new-url-syntax/</loc><lastmod>2012-05-14T09:55:52+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-13/opensmtpd-map_compare-and-k_netaddr/</loc><lastmod>2012-05-13T03:09:16+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-12/opensmtpd-meets-sqlite/</loc><lastmod>2012-05-12T14:23:54+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2012-05-09/some-opensmtpd-news/</loc><lastmod>2012-05-09T23:03:36+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/posts/2005-05-21/fatigue-et-envie-de-vacances.../</loc><lastmod>2005-05-21T00:00:00+00:00</lastmod><changefreq>daily</changefreq><priority>0.5</priority></url><url><loc>https://poolp.org/legal/</loc><changefreq>daily</changefreq><priority>0.5</priority></url></urlset>
\ No newline at end of file