How we combined real-time video streaming, GPU shaders, and questionable design choices to create conference magic
When you work at a software agency that builds its own products, at some point in time there must come a moment when someone asks: “What’s the most ridiculous thing we could build with the tech we develop?” For us, at Software Mansion, that question led us to creating Pufferfish, a game where players become fish, control their puffiness with lip movements, and battle in a real-time aquarium streamed to their phones.
This wasn’t just an experiment in absurdity — Pufferfish served a purpose.
It was a real tech demo designed to showcase our in-house tools at Demuxed, one of the premier video technology conferences worldwide. Here’s how we turned faces into fish and made it technically impressive while doing so.
The concept: beautifully stupid
By design, the rules of Pufferfish are simple. Players transform into fish swimming mindlessly around a shared aquarium. Each player’s face, captured by their phone’s camera, appears as a fish on the main screen. By making a kissing face — puckering their lips — players can puff up their fish.
Puffed fish can pop smaller, unpuffed fish. They also bounce off other puffed fish in glorious, physics-defying collisions. The entire aquarium is displayed on a large TV screen while simultaneously streaming to each player’s phone with minimal latency.
It’s the kind of game concept that makes perfect sense towards the end of a prolonged brainstorming session and somehow still sounds good the next day.
What made Pufferfish special wasn’t the gameplay — it was the technology powering it. We deliberately built the demo to showcase three of our in-house products working in concert:
Smelter: real-time video composition
The aquarium view isn’t a simple video feed. It’s a real-time composition of multiple video streams merged into a single cohesive scene. This is where Smelter — our tool for composing multiple video streams in real time — came in handy.
Smelter takes each player’s camera feed, positions their face on their corresponding fish, layers in the background, adds other players’ fish, and outputs a single, synchronized stream. All of this happens live, with no pre-rendering.
If that sounds interesting, check out the Smelter demo!
Fishjam: low-latency media distribution
At the heart of Pufferfish sits Fishjam, our API for low-latency video conferencing and live streaming. Fishjam handles the real-time distribution of video streams from the game server to both the players’ phones and the large display screen.
In a game where timing matters — when you need to puff up to avoid being popped — latency is everything. Fishjam ensures that what players see on their screens matches the server state with minimal delay.
TypeGPU: shaders in TypeScript
The visual magic of Pufferfish — the fish’s appearance, the puffing animations, the aquarium aesthetics — all comes from GPU shaders. But instead of writing these shaders in traditional GLSL or HLSL, we used TypeGPU, our TypeScript library that lets developers write shaders in TypeScript.
For example, here’s how the position of a spike on a pufferfish is calculated using the golden angle. The following TypeScript code is then converted to shader language.
import tgpu from "typegpu";
import * as d from "typegpu/data";
import { cos, sin, sqrt } from "typegpu/std";
const SPIKES_NUM = 60;
// Golden angle
const phi = Math.PI * (sqrt(5.0) - 1.0);
const fibonacciSphereDistribution = (index: number): d.v3f => {
"use gpu";
const y = 1.0 - (d.f32(index) / d.f32(SPIKES_NUM - 1)) * 2.0;
const radius = sqrt(1.0 - y * y);
const theta = phi * d.f32(index);
const x = cos(theta) * radius;
const z = sin(theta) * radius;
return d.vec3f(x, y, z);
};For a team primarily working in TypeScript, this removes the context-switching cost of learning shader languages while still delivering GPU-accelerated graphics. The fish in Pufferfish are rendered with TypeGPU shaders, proving the library works for real-time, interactive applications.
Why build something this silly?
Tech demos often fall into two camps: either they’re dry, technical proofs-of-concept that fail to capture imagination, or they’re polished but generic examples that don’t stand out. We wanted something different.
Pufferfish was designed to be memorable. At a conference like Demuxed, attendees see countless demos. We needed something that would make people stop, laugh, and then realize, “Wait, this is actually technically impressive.”
By choosing a very goofy and abstract concept — turning people’s faces into competitive puffing fish — we created a demo that:
- Attracted attention through sheer novelty
- Demonstrated real technical capabilities under the hood
- Showed our personality as a company that takes technology seriously but doesn’t take itself too seriously
- Created a shared experience that conference attendees would remember and talk about
The conference experience
At Demuxed London, Pufferfish did exactly what we hoped. Attendees gathered around the screen, pulled out their phones, and within seconds were making ridiculous faces to puff up their digital fish. The mixture of laughter and genuine technical interest was exactly what we were after.
The demo sparked conversations about low-latency streaming, real-time video composition, and writing shaders in Typescript — but it started with “why is that person making a duck face at their phone?”
Beyond the aquarium
Pufferfish might be silly, but the technology behind it is solving real problems. Fishjam powers applications requiring low-latency video communication. Smelter enables real-time video composition for everything from virtual events to interactive streaming. TypeGPU makes GPU programming accessible to web and mobile developers.
Sometimes the best way to demonstrate serious technology is through profoundly unserious applications. We turned faces into fish, and in doing so, showed what’s possible when real-time video streaming, GPU-accelerated graphics, and creative engineering come together.
Want to try Pufferfish yourself? Head over to puffer.fishjam.io to turn your face into a fish and start puffing.
If you’re at the next conference and someone offers to turn you into a pufferfish, say yes. Your face muscles, tired from the ridiculous mix of smiling and puffing, will most definitely thank you.
