<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://osmnvaslan.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://osmnvaslan.github.io/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-11-15T13:49:49+00:00</updated><id>https://osmnvaslan.github.io/feed.xml</id><title type="html">Aslan Osmanov</title><subtitle>Notes on the parts of a system that fail quietly. Python, messaging, LLM plumbing.</subtitle><author><name>Aslan Osmanov</name></author><entry><title type="html">now() is not the insert time</title><link href="https://osmnvaslan.github.io/now-is-not-the-insert-time/" rel="alternate" type="text/html" title="now() is not the insert time" /><published>2026-11-15T00:00:00+00:00</published><updated>2026-11-15T00:00:00+00:00</updated><id>https://osmnvaslan.github.io/now-is-not-the-insert-time</id><content type="html" xml:base="https://osmnvaslan.github.io/now-is-not-the-insert-time/"><![CDATA[<p><a href="https://github.com/OsmnvAslan/txoutbox">txoutbox</a> 0.2.3 fixed a bug where the Postgres
adapter could publish two messages with the same key in the wrong order. The feature that
allowed it was the one meant to prevent reordering. This is how it happened.</p>

<h2 id="the-feature">The feature</h2>

<p>An outbox table gets its ids from a sequence, and a sequence hands out values at <code class="language-plaintext highlighter-rouge">INSERT</code>,
not at <code class="language-plaintext highlighter-rouge">COMMIT</code>. A transaction that took id 1 can commit after a transaction that took id 2.
A relay that reads in id order sees id 2 first, publishes it, and only later finds id 1.</p>

<p><code class="language-plaintext highlighter-rouge">PostgresStorage(visibility_delay=...)</code> exists for this. The relay only claims rows older
than the delay. If the delay is longer than your slowest producing transaction, every lower
id has committed by the time a higher one becomes visible.</p>

<p>“Older” was measured from <code class="language-plaintext highlighter-rouge">created_at</code>, and the schema declared it like this:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">created_at</span> <span class="n">TIMESTAMPTZ</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">DEFAULT</span> <span class="n">now</span><span class="p">()</span>
</code></pre></div></div>

<h2 id="the-bug">The bug</h2>

<p>In Postgres <code class="language-plaintext highlighter-rouge">now()</code> is the start time of the current transaction, not the current time.
Every row a transaction inserts gets the same <code class="language-plaintext highlighter-rouge">created_at</code>, however long the transaction
has been running.</p>

<p>So a lower id can look younger than a higher one. Take a delay of 2 seconds:</p>

<table>
  <thead>
    <tr>
      <th>Time</th>
      <th>Transaction X</th>
      <th>Transaction Y</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>0.0 s</td>
      <td><code class="language-plaintext highlighter-rouge">BEGIN</code></td>
      <td> </td>
    </tr>
    <tr>
      <td>3.0 s</td>
      <td> </td>
      <td><code class="language-plaintext highlighter-rouge">BEGIN</code>, insert id 1 (<code class="language-plaintext highlighter-rouge">created_at</code> = 3.0)</td>
    </tr>
    <tr>
      <td>4.0 s</td>
      <td>insert id 2 (<code class="language-plaintext highlighter-rouge">created_at</code> = 0.0)</td>
      <td> </td>
    </tr>
    <tr>
      <td>4.5 s</td>
      <td><code class="language-plaintext highlighter-rouge">COMMIT</code></td>
      <td><code class="language-plaintext highlighter-rouge">COMMIT</code></td>
    </tr>
  </tbody>
</table>

<p>At 4.5 s the relay asks for rows older than 2 seconds. Id 2 claims to be 4.5 seconds old
and passes. Id 1 is 1.5 seconds old and does not. Both rows have the same key, and id 2
goes out first.</p>

<p>Strict ordering did not help. Its predicate held a row back only while an earlier row with
the same key was leased or waiting for a retry. An earlier row that was merely too young
was not a reason to wait.</p>

<h2 id="the-fix">The fix</h2>

<p>Two changes, and both are needed.</p>

<p>The schema now uses the real insert time:</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">created_at</span> <span class="n">TIMESTAMPTZ</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">DEFAULT</span> <span class="n">clock_timestamp</span><span class="p">()</span>
</code></pre></div></div>

<p>And with strict ordering, the claim query holds a row back while an earlier pending row
with its key is still inside the delay (simplified):</p>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">AND</span> <span class="k">NOT</span> <span class="k">EXISTS</span> <span class="p">(</span>
  <span class="k">SELECT</span> <span class="mi">1</span> <span class="k">FROM</span> <span class="n">outbox</span> <span class="n">p</span>
  <span class="k">WHERE</span> <span class="n">p</span><span class="p">.</span><span class="k">key</span> <span class="o">=</span> <span class="n">o</span><span class="p">.</span><span class="k">key</span> <span class="k">AND</span> <span class="n">p</span><span class="p">.</span><span class="n">id</span> <span class="o">&lt;</span> <span class="n">o</span><span class="p">.</span><span class="n">id</span> <span class="k">AND</span> <span class="n">p</span><span class="p">.</span><span class="n">status</span> <span class="o">=</span> <span class="s1">'pending'</span>
    <span class="k">AND</span> <span class="p">(</span><span class="n">p</span><span class="p">.</span><span class="n">retry_at</span> <span class="o">&gt;</span> <span class="n">now</span><span class="p">()</span> <span class="k">OR</span> <span class="n">p</span><span class="p">.</span><span class="n">lease_until</span> <span class="o">&gt;</span> <span class="n">now</span><span class="p">()</span>
         <span class="k">OR</span> <span class="n">p</span><span class="p">.</span><span class="n">created_at</span> <span class="o">&gt;</span> <span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="err">$</span><span class="mi">4</span><span class="p">::</span><span class="n">interval</span><span class="p">)</span>
<span class="p">)</span>
</code></pre></div></div>

<p>The predicate alone is not enough for old tables. A row whose transaction has not committed
is invisible to the check, so with the <code class="language-plaintext highlighter-rouge">now()</code> default the delay would have to exceed twice
your longest producer transaction. <code class="language-plaintext highlighter-rouge">create_schema()</code> now switches the default on existing
tables; it is a metadata-only <code class="language-plaintext highlighter-rouge">ALTER TABLE</code>.</p>

<h2 id="the-test">The test</h2>

<p>The regression test inserts a young row and then an older row with the same key, the older
one backdated by five seconds:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">assert</span> <span class="nb">list</span><span class="p">(</span><span class="k">await</span> <span class="n">storage</span><span class="p">.</span><span class="n">claim</span><span class="p">(</span><span class="n">batch_size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">lease</span><span class="o">=</span><span class="n">LEASE</span><span class="p">,</span> <span class="n">worker_id</span><span class="o">=</span><span class="s">"w"</span><span class="p">))</span> <span class="o">==</span> <span class="p">[]</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">1.1</span><span class="p">)</span>
<span class="n">got</span> <span class="o">=</span> <span class="k">await</span> <span class="n">storage</span><span class="p">.</span><span class="n">claim</span><span class="p">(</span><span class="n">batch_size</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">lease</span><span class="o">=</span><span class="n">LEASE</span><span class="p">,</span> <span class="n">worker_id</span><span class="o">=</span><span class="s">"w"</span><span class="p">)</span>
<span class="k">assert</span> <span class="p">[</span><span class="n">m</span><span class="p">.</span><span class="n">payload</span> <span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="n">got</span><span class="p">]</span> <span class="o">==</span> <span class="p">[</span><span class="sa">b</span><span class="s">"young"</span><span class="p">,</span> <span class="sa">b</span><span class="s">"old"</span><span class="p">]</span>
</code></pre></div></div>

<p>Before the fix the first claim returned the old row.</p>

<h2 id="what-i-took-from-it">What I took from it</h2>

<p>A timestamp column answers one question, and it is worth writing down which. “When did this
transaction start” and “when was this row written” are different facts, and <code class="language-plaintext highlighter-rouge">DEFAULT now()</code>
silently picks the first. If anything downstream compares the column with the wall clock,
it wants <code class="language-plaintext highlighter-rouge">clock_timestamp()</code>.</p>]]></content><author><name>Aslan Osmanov</name></author><summary type="html"><![CDATA[How a DEFAULT now() column let an outbox relay publish two events for one key in the wrong order.]]></summary></entry></feed>