Itamar Turner-Trauring

Articles on Python performance (with a focus on data science / scientific computing) and Docker packaging at https://pythonspeed.com (click through and sign up for my newsletter for weekly articles in your inbox!).

Public Key

npub1l3hcw0upjp59wljh5l5e70uzter9kez5t4d6vvl657w4u685alfs52qxlu

NIP-05 Address

itamarst@hachyderm-io.mostr.pub

Profile Code

nprofile1qqs0cmu887qeq6zh0et606vl87p9u3jmv3296kaxx0a20827dr6wl5cpz3mhxue69uhhyetvv9ujuerpd46hxtnfduqs6amnwvaz7tmwdaejumr0ds6wmcvj

Publishing to

offchain.pub

nos.lol

relay.damus.io

Author Public Key

npub1l3hcw0upjp59wljh5l5e70uzter9kez5t4d6vvl657w4u685alfs52qxlu

Show more details

Last Notes

2026-10-22 14:42:56 UTC

Security reminder:
If you are using Docker layer caching, that can cache the base Linux distribution's packages.
I.e. you will not get security updates even if you e.g. "apt-get update && apt-get upgrade" in your Dockerfile.
That means your TLS certificate authority public key database won't get updated, which this week is Very Bad.
So you want to rebuild all Docker images without caching, on a regular basis, e.g. weekly. And this week you want to rebuild them without caching ASAP.
Long version: https://pythonspeed.com/articles/docker-cache-insecure-images/

2026-10-17 21:56:42 UTC

I find the notion that LLMs writing your code for you are just like having interns or junior engineers to be rather offensive.
Interns aren't there to do your menial work, they're there to learn new skills. Junior engineers are hired with the idea that they will become senior engineers, which requires you and your organization to invest in teaching them. Humans learn! They can get better!
But LLMs don't learn. So the comparison to LLMs is insulting to interns, juniors, and other people with less experience. And therefore a rather depressing view into software engineering culture.
(Separately, I also find it baffling that anyone would want to have a pseudo-conversation that involves no learning on the other side, but apparently that is just a personal quirk of mine.)

2026-04-28 19:09:11 UTC

Reading intro C book and oh boy.
1. Bug caused by integer wraparound made Boeing 787 engines all turn off at once.
2. You should check for integer wraparound.
3. But! It's very easy to make checks that don't actually work.
Extra 😱 in the age of LLMs.
Programmers: You don't need language safety features, just be careful.
Also programmers: No need to read code anymore!

2026-04-23 16:53:07 UTC

"AI" is going to make getting software engineering jobs harder over the next year... but not because "AI" will replace humans, but rather because of the interaction of accounting and the need to pacify stock market investors.
1. Megacorps have spent massively on AI datacenters. This is a capital expenditure which means the expense isn't booked immediately.
2. Instead, the expense gets turned into deprecation over the next few years, i.e. reported expenses are going to be much higher for the next few years.
3. Higher expenses shrink reported profits, so these companies will start doing layoffs.
4. So far there is no evidence of offsetting profits.
Notice this effect happens _even if all AI datacenter spending stops today_. It's baked in as depreciation.
So far we've had layoffs at Amazon, Meta, and Oracle (much more dire straits financially than the others). Today Microsoft is doing voluntary worker buyouts: https://www.bloomberg.com/news/articles/2026-04-23/microsoft-offers-voluntary-retirement-to-about-7-of-us-workers
I've seen rumors of Google doing buyouts too. And when these layoffs hit software engineers, that's going to mean a lot more competition for any open jobs. These companies have also hired a lot, and will be hiring less.
So that means more applicants for fewer jobs.

2026-02-28 22:43:47 UTC

To expand this point: software is composed of code, organizational processes, and people. A surprisingly large number of software developers are narrowly focused on generating code and miss all the surrounding processes and people (skills, knowledge, understanding of goals) that make that code usable.
"It's so good at generating code" doesn't address the costs in other categories:
- Degradation of processes. For example, you get denial of service on code reviews means code reviews go away, "it passes tests, it's fine!"
- Degradation of people's skills and knowledge. If it breaks, no one will know how it works because no one really understands the code. And an experienced engineer can only control an agent because they have experience _writing code_. If you stop writing code, that experience degrades, new people don't get it, and control of the agent degrades.
For this narrow focus on artifacts, and not the surrounding requirements, I highly recommend the book "What Machines Can't Do: Politics and Technology in the Industrial Enterprise", by Robert J. Thomas. (It has nothing to do with AI, it's about manufacturing.) But you see the failure modes from what he describes as having only "an aesthetic of product", and no "aesthetic of process"; similar failure modes result in the original DevOps movement, which was a political movement, not a job title.
#nevent1q…rmha

2026-02-26 15:01:05 UTC

You want your code to be faster, or at least not to get slower.
Step 1. Add benchmarks to CI. This will catch accidental regressions.
Step 2. You spend a day adding a feature, submit a PR... and discover you accidentally made an important code path slower. You don't know which particular commit was at fault, though.
Step 3. Add unit tests that will catch changes to speed, so you can catch potential regressions while coding. Or, that's the theory anyway.
Here's some thoughts on how to do step 3: https://pythonspeed.com/articles/speed-unit-tests/

2026-01-07 20:19:15 UTC

We unit test code's correctness—so why not unit test performance as well?
Having thought about the problem a little, here's my suggestion for a first step: testing big-O scalability.
https://pythonspeed.com/articles/big-o-tests/
#python

2025-10-08 15:12:43 UTC

I am releasing software!
And by releasing I mean:
1. Doing a release.
2. It fails.
3. I realize I forgot to update version in that one place.
4. Updating the version.
5. Doing a release.
6. Realizing PyPI isn't showing the description.
7. Trying to fix that.
8. Doing a release.
9. Writing this while I wait to see if this time PyPI will show the description.

2025-06-11 15:24:25 UTC

- reply

@nprofile…6m2n
The scoping of what to teach is a tricky problem yeah.
I think my success criteria (and there are other ones I'm sure) would be "are you able to read Kerrisk's The Linux Programming Interface book" which probably does a involve a bunch of additional syntax, just opening a random page (e.g. setting flags with |= operator).
The other question is... what can you assume the people you're writing for already know?
Someone with decent knowledge of Python you'd need to teach some syntax and surface semantics (straightforward, mostly), data types (straightforward, mostly), and pointers (complicated).
But "getting memory management right" is unnecessary as you said, and "here's how you make .h files for your library" is unnecessary, "what does static and friends mean, what are compilation units etc" is almost certainly unnecessary.

2025-06-11 14:51:51 UTC

- reply

@nprofile…6m2n This knowledge really useful in a bunch of places, yes!
- Operating system APIs are all documented as C/C++ APIs
- All (or almost all?) books on low-level software performance assume you can read C
I'm thinking about the latter a lot at the moment since I'm trying to write a book that -doesn't- require that, but would be nice to have something I can point people at.

2025-06-10 14:14:28 UTC

- reply

@nprofile…3kq5 If you're in continental US, https://cortico.ai/careers/software-engineer-backend-2/

2025-06-02 15:31:14 UTC

Job posting for a non-profit using #Python; remote work for continental US timezones, must have right to work in the US. You'll get to work with some great, welcoming people!
------
Cortico is a non-profit committed to fostering constructive conversations and surfacing underheard voices. We believe that intentionally designed and intentionally deployed technology can enable communities to achieve these aims in ways that are more inclusive and scalable.
Our platform brings people together in recorded small-group conversations. Through a powerful combination of AI and human listening, this platform enables organizations to make sense of the conversations they collect, and surfaces community voices into public dialogue and decision making.
We are looking for a backend-focused Software Engineer to join our small engineering team.
The primary responsibilities of this role include:
- Designing, building, and maintaining scalable backend services to enable our partners to hold conversations and to process, analyze, and store the resulting audio data and transcripts
- Working with our MIT collaborators to integrate AI/ML models into our production systems
- Building and enhancing APIs that connect our mobile and web applications
- Contributing to architecture decisions that shape our platform’s future
- Writing clean, well-tested, and documented code
- Participating in code reviews and helping maintain high code quality standards
- Collaborating cross-functionally with product, design, and research teams
- Mentoring junior engineers and sharing knowledge with the broader team
At Cortico, we look at a number of things when setting compensation—like your experience, skills, certifications, training, and what the role needs. Where you land on our range depends on your unique background and experience, as well as where you live. For this role, the estimated pay range is $105,000 – $125,000.
Full listing here: https://cortico.ai/careers/software-engineer-backend-2/
#FediHired

2025-05-30 19:03:56 UTC

Apparently 82% of Israeli Jews support ethnic cleansing, per this survey by an Israeli newspaper.
For some historical context, when I was a kid living in Israel, political parties supporting ethnic cleansing were illegal.
(The fact they're surveying Jews specifically is because there are also ~20% of population who are Palestinian citizens of Israel, who tend to have very different views and also a lot less influence on outcomes. These latter are not to be confused with Palestinians in Gaza and West Bank, who are not citizens of Israel.)
https://archive.is/85C4v

2025-05-29 14:40:22 UTC

So I have a Take:
Imagine if LLM coding assistants had come out when programming required explicit manual memory management.
Everyone is writing all this C code with malloc() and free(). It's a pain, and repetitive, and why spend time thinking about this?
So all the early adopters of LLMs are saying "this is amazing, I don't have to write all this boilerplate malloc() and free() and multiplication of pointer sizes by array length, it auto-generates that for me, This Is The Future! You will All Be Left Behind if you don't adopt this!"
(I am actually skeptical this is something LLMs would do reliably, but let's just pretend they can.)
And maybe that approach would actually win, and no one would have created garbage-collected (or equivalent) languages, because that's silly, you have a LLM to generate that code for you.
Never mind that garbage collection is vastly superior to LLM-generated-malloc():
* The code is _way_ shorter and therefore easier to reason about the parts you actually care about (the logic)
* You don't have to worry about the LLM generating garbage one time out of N
* Less segfaults and memory corruption, etc..
Back to our actual present: a lot of what I hear people saying about LLMs is "look, I don't have to write as much boilerplate or repetitive code" and I'm sorry but that's not a benefit, that's just doubling down on a liability. All things being equal (if it's just as understandable, just as fast, etc), you want to solve a problem with the fewest lines of code as possible.
If you have to write the same thing over and over again, that is a failure in your tooling, and you should build better tooling. A better library, a better abstraction, even a better programming language.
And to be fair sometimes this better tooling requires significant investment, and those resources that are available for R&D are being piled into LLMs instead of into reducing how much code we write.
But sometimes better tooling and libraries is just a matter of _thinking_ about a problem. And can you even think if your boss wants you to hit the deadline, "and AI should make you twice as productive, right?"
But also: why think, when you can double down on a bad status quo and have an LLM poop out some more code for you?

2025-05-13 17:03:36 UTC

Your occasional reminder that if you have an ancestor who was a Canadian citizen (could be really far back, doesn't have to be recent!) you can get a Canadian citizenship grant right now.
You have to apply to qualify, and this is TEMPORARY MEASURE. There will be a new law eventually which might make you a citizen, or might take away this option altogether. Don't wait.
https://www.reddit.com/r/Canadiancitizenship/comments/1keq0uz/start_here_the_can_i_get_canadian_citizenship_faq/
#trans

2025-04-24 21:58:47 UTC

Does anyone know of a good patient advocate, ideally based in California? Or have experience with some of the non-profits that do this? For someone with very complex medical issues.
#california #disability

2025-03-22 13:05:10 UTC

Thinking about major international conferences that tend to be in the US only. E.g. PyCon US this year is allowing people from other countries to cancel and get a refund, but what about the future?
One can hold the conference in:
1. The US. This is going to mean huge drop in non-US attendees who are worried about being turned back or even arrested and detained for weeks.
2. In Canada. This works for non-US residents, with the usual exceptions for some countries. But US permanent residents (aka Green Card) are getting targeted and may be detained, arrested, and mistreated when they try to return home to the US. So they won't be able to go.
4. Hybrid US/online, or Canada/online. Everyone can attend, but very inconsistent experience. EuroPython has been doing hybrid conferences, though, including speakers that can be both in person or remote.
5. Online only. Consistent experience for everyone.
Hybrid and online versions do have the benefit of being more environmentally friendly, since there is long-haul travel involved.

2024-11-26 15:42:12 UTC

The Unix Philosophy is all about building Rube Goldberg machines: single-use, brittle, untestable, utterly ridiculous, but somehow actually working.
An example:
https://github.com/ShigShag/asus_x670e_ethernet_fix?tab=readme-ov-file#check_and_reset_ethernetsh

2024-11-22 22:30:28 UTC

- reply

3/3 In batch data processing, one big CO2 emissions cost comes from _running the wrong thing_. Your software crashes... after 5 hours. Or maybe after two weeks of running jobs you discover a semantic bug and have to throw away all the results.
This isn't quite blockchain level of environmental wastefulness—at least the goal is good—but still, this is still something you want to minimize.
Which means as a library author, you want to catch users' mistakes _as early as possible_.

2024-11-22 22:24:55 UTC

- reply

2/N I talked to someone at... MariaDB maybe? and they explained they had a group that looked into reducing carbon emissions, and they came to the conclusion that the biggest problem in their database was things like users forgetting to add an index.
So one thing to do as a library/app maintainer is to make it easy for users to have speed by default, since faster computing means lower emissions. But we can take it a step further if we focus on batch data processing—

2024-11-22 22:23:01 UTC

1/N Starting to write an outline of article about how open source maintainers (especially of data science and scientific computing libraries) can reduce carbon emissions, and realized I might get to talk about my favorite ACM Computing Survey paper, "Systems Approaches to Tackling Configuration Errors: A Survey" (https://tianyin.github.io/pub/csur.pdf) and related research by Tianyin Xu and collaborators.
This is just one point out of many, but it's maybe the most surprising—

2024-11-22 17:21:47 UTC

One of my favorite things about compilers is how often telling them to target your current CPU ("-march=native") actually makes the resulting code run slower.
At a guess the latency and performance models compilers have for modern CPUs aren't very accurate, and don't get fixed because almost no one compiles this way? But that's just a guess.

2024-11-12 17:30:00 UTC

I previously wrote about how SIMD can speed up your code, using Mandelbrot as an example. The code I used in that article required unstable Rust though, which is not ideal.
In this follow-up I show how you can write SIMD with stable Rust, using the `wide` crate:
https://pythonspeed.com/articles/simd-stable-rust/
#rust

2024-11-01 14:23:00 UTC

It is ludicrously easy to write buggy numeric code, so whenever you refactor for readability or speed you want to at validate you are still getting the same result.
My book-in-progress on optimizing numeric code has this after every optimization step:
```
assert np.array_equal(version1(data), version2(data))
```
While tweaking a function for readability I've just gone through literally 10+ different iterations before I got that to pass!
https://pythonspeed.com/products/lowlevelcode/
#python #numpy

2024-07-12 16:07:25 UTC

For those not aware: you should not be using Eventlet (and probably not Gevent either). The fundamental approach of monkey-patching the Python standard library is untenable without massive resources, resources that don't exist.
Eventlet has a migration path off via asyncio integration, not sure about Gevent.
#python

2024-07-11 00:30:16 UTC

I am about to run cmake send help

2024-07-08 14:45:07 UTC

Oh, cool, Python exceptions now have as of 3.11 an `add_note()` function that lets you add extra notes to an exception. They get displayed with the traceback.
https://docs.python.org/3/library/exceptions.html#BaseException.add_note
#python

2024-06-28 14:33:19 UTC

PyO3 0.22 is out, so if you have Rust-based packages you can now start releasing Python 3.13 wheels.
#python

2024-06-20 19:03:38 UTC

Working on upgrading some open source Python packages to support 3.13, and it's kinda neat how much Rust support is becoming a blocker for progress, because it means Rust is being used in key packages.
(Upcoming PyO3 0.22 will add 3.13 support, so this blocker will hopefully be gone in a short while.)

2024-05-07 21:58:36 UTC

The hardest problem in computer science is linking (the thing that happens after compilation and breaks everything, not the thing websites do).

2024-04-23 19:36:19 UTC

TFW the code is doing something ridiculously inefficient but you can't fix it because it's an accidentally public API in a library.

2023-12-07 02:21:14 UTC

- reply

@npub1pfe…e2dm There's also https://old.reddit.com/domain/glyph.im/

2023-07-18 15:27:30 UTC

New major release of Cython means a bunch of #Python packages using Cython that didn't pin their build dependencies are now broken. And if there's no wheels, now that's your problem 😭