When the network card crashes the router, we fix the driver — not just reboot the box.
Why a network-card driver decides whether your router stays up
A modern carrier router handles millions of packets a second on 10, 40 and 100 gigabit network cards. To keep up, packets aren't copied around politely — the card writes each one straight into a shared pool of memory buffers, and the kernel's high-speed packet path (XDP) processes it in place, before the rest of the operating system even wakes up. It is the fastest, leanest path in the whole system — and the network-card driver is the code that manages that pool of buffers: handing a fresh one to the card, taking back a used one, and recycling it for the next packet.
That recycling is a bookkeeping problem. Every packet buffer has a lifecycle — given to the card → filled → processed → returned to the pool. Get the bookkeeping exactly right and the router runs for years. Get one edge case wrong — return a buffer but keep pointing at it, or forget to return one at all — and eventually the driver reaches for a packet that is no longer there. At that moment the kernel does the only safe thing it can: it stops the whole machine. To your NOC, the router simply rebooted itself.
What actually went wrong — a kernel upgrade moved risk into the driver
Kernels are upgraded for good reasons: security fixes, new hardware, better performance. This upgrade also refactored the network-card driver onto a new shared buffer library used by several cards at once. That kind of consolidation is healthy long-term — but it rewrote the most delicate code in the system, and the rewrite left a cluster of bugs in exactly the places that are hardest to test: the error paths and shutdown paths that only run when something rare happens.
- Fine in the lab; crashes only with real traffic and real memory pressure.
- No bad config, no obvious trigger — just an occasional hard reset.
- Each reset drops every subscriber on the box for the time it takes to come back.
- Impossible to reproduce on demand, so impossible to "just tune away".
- Some paths forgot to return a buffer or undo a hardware mapping.
- Nothing crashes — memory and mapping tables just creep upward.
- Days later the box is sluggish or wedged, with no single event to blame.
- The classic "it was fine last week" ticket that never gets root-caused.
The BNGSOFT difference: we own the driver, so we fix the driver root-cause not reboot
Most operators are stuck with whatever the card vendor ships, so their options are limited to reboot, roll back, or file a ticket and wait. Because BNGSOFT builds and controls its own kernel and data plane, we treat a driver crash the same way we'd treat a bug in our own code — and we followed it all the way down.
The class of defect we hardened — in plain terms
All nine fixes share one theme: the driver's bookkeeping about which packet buffers are live. None require you to trust a marketing claim — each is a specific, concrete edge case that real traffic eventually hits.
Touching a buffer that was already returned
The original crash. Under memory pressure a slot was freed but still pointed at, so a later packet reached into memory that was gone. Now the driver checks first and skips a slot with nothing live in it.
Leaving a "ghost" pointer behind
Some paths returned a buffer but didn't clear the slot, leaving a dangling reference that could be re-used by mistake. Every such path now clears the slot as it releases it.
Forgetting to release on the error path
When something rare failed mid-way, a buffer or a hardware mapping could be left un-returned — a slow leak. The error paths now unwind cleanly and give everything back.
Cleaning up in the wrong order
On a card reconfigure, tearing things down in the wrong sequence could leak the entire buffer pool. The teardown now recycles live buffers before releasing the pool.
Book-keeping a packet that never sent
On a full transmit ring the driver could mark work as done that was never actually queued. It now only updates its records when a packet was genuinely sent.
Recovering a packet instead of dropping it
A specific hardware condition was treated as a fatal error and the packet discarded, when the data was actually intact and recoverable. It's now recovered — fewer needless drops.
One driver family, three cards — we audited all of them 10 / 40 / 100G
Finding the crash on one card was the start, not the end. The same kernel change touched a whole family of network-card drivers, so we went card by card — because a bug of this kind rarely lives alone. Different cards use different internal designs, and that turned out to matter:
- Uses the new shared buffer library the upgrade introduced.
- Source of the original self-reboot, plus a cluster of related edge-case leaks.
- Root-caused and hardened across receive, transmit, fast-path and shutdown.
- Different buffer design, but shares the risky habit of holding packet state across cycles.
- Carried the same kind of "touch a freed packet" defect in multi-part receive, plus fast-path transmit leaks.
- Found by looking — before it ever crashed a box — and fixed the same way.
- Uses an older, self-contained buffer design the upgrade did not reshape.
- Read the whole receive / transmit / fast-path lifecycle: the bug class structurally cannot occur here.
- We report that plainly — no change, no invented "fix" to pad a list.
Before & after — the same kernel upgrade, two responses
- Reboot the box when it crashes and hope it holds until the next one.
- Downgrade the kernel — losing the security and hardware fixes that upgrade brought.
- Or file a vendor ticket and wait months for an upstream release.
- The silent leaks go unnoticed and keep degrading boxes in the background.
- You're depending on a component you can't see into and can't fix.
- Crash read and root-caused to a single line from the live failure.
- Driver patched at the source — kernel kept current, no downgrade.
- Whole driver audited; nine defects hardened, crashes and leaks alike.
- Kernel image rebuilt and validated with the fast path still at line rate.
- You get a platform that's more reliable than stock, not just back to baseline.
How the approaches compare
Framed by response strategy rather than by vendor — what each choice actually costs you when a network-card driver misbehaves after a kernel change.
| Reboot & hope | Downgrade the kernel | Wait for upstream fix | BNGSOFT: own the driver | |
|---|---|---|---|---|
| Stops the crash | Temporarily | Usually | Eventually | Yes — at the source line |
| Fixes the silent leaks too | No | Maybe | Only if reported | Yes — whole class audited |
| Keeps kernel security / HW fixes | Yes | No — you lose them | Yes | Yes — stay current |
| Time to resolution | Never — recurs | Hours + regression risk | Weeks to months | Days, in-house |
| Impact on packet speed | None | None | None | None — line rate preserved |
| Depends on an outside party | — | — | Yes — their schedule | No — we control the stack |
What it means for you — and for your subscribers
- Fewer unexplained reboots — the crash class that causes them is engineered out, not patched over.
- No slow-degradation mystery tickets — the silent leaks are fixed alongside the loud crash.
- Stay on a current kernel — keep the security and hardware improvements; no forced downgrade.
- No waiting on a vendor — when the driver misbehaves, the people who can fix it are the people who built your platform.
- Reliability with no speed penalty — the fast path is untouched and still runs at line rate.
- A vendor who reads crash logs — support that goes to the source, not a script.
- The connection just stays up — no random drop-outs when a distant router resets itself.
- Consistent performance — no gradual slow-down as a box quietly leaks over a week.
- Fewer outages at peak — the failures were worst exactly when the network was busiest.
- Invisible, which is the point — the most reliable infrastructure is the kind nobody notices.
We don't reboot around problems. We fix them at the source.
Because BNGSOFT builds its own kernel and data plane, a defect anywhere in the stack — even deep inside a network-card driver — is ours to diagnose and ours to fix. That's the difference between a platform you rent and one that's genuinely engineered for you: when something breaks in the hardest place to reach, it still gets fixed, fast, at the source, with no loss of speed.
Ask us how we handle kernel and driver reliability across your fleet — sales@bngsoft.com.